A method, recording medium and system for calibrating the images taken by a camera with respect to the coordinate system of a scanner

Abstract

The present application belongs to the technical field of map positioning and marking, and particularly relates to a camera image positioning and marking method relative to a scanner coordinate system. The method comprises the following steps: extracting frame structure edge data in an image, calculating vanishing point positions and mapping to a normalized plane; according to the intersection relationship of each plane of the frame structure in the point cloud, extracting edge and corner point information and mapping to the normalized plane; calculating camera external parameters by using the position relationship of the same object on the normalized plane and optimizing the camera external parameters; and using the optimized camera external parameters to complete the calibration. The method can easily position and mark the image captured by the camera in the scanner reference coordinate corresponding to the point cloud, realizes efficient and accurate joint calibration of the camera and the scanner, and is particularly suitable for a city building scene. The present application also provides a non-transitory readable recording medium storing the program of the method and a system comprising the medium. The program can be called by a processing circuit to execute the above method.

Description

Technical Field

[0001] This invention belongs to the field of map positioning and annotation technology, and discloses a method, recording medium and system for calibrating camera-captured images relative to the scanner coordinate system. Background Technology

[0002] UAV LiDAR (Light Detection and Ranging) systems are highly integrated, typically fusing scanners and camera sensors to acquire multi-source geospatial data. They have been widely applied in resource inventory, ecological environment monitoring, geological disaster detection, and power line inspection. Scanners acquire geometric information and echo intensity of object surfaces, offering advantages such as fast 3D data acquisition and short update cycles. Cameras, on the other hand, acquire texture and spectral information of object surfaces, complementing each other's strengths. Joint calibration of scanners and cameras is a prerequisite for multi-sensor data fusion, aiming to obtain accurate joint calibration extrinsic parameters and provide the geographic information industry with more data fusion solutions.

[0003] Currently, the development of lightweight unmanned aerial vehicle (UAV) platforms is on a large scale, requiring a high degree of integration of sensor payloads. However, highly integrated multi-sensor coordinate systems face problems such as inconsistent coordinate references and low sensor data alignment accuracy. Therefore, joint calibration of multiple sensors is necessary to achieve correlation between different sensor data. Although the placement parameters between sensors are usually provided through factory calibration, achieving pixel-level accuracy, more precise extrinsic parameters are still needed through joint calibration to meet the higher data fusion requirements of the industry.

[0004] Currently, joint calibration methods using calibration boards as calibration objects are widely used due to their convenient feature extraction. However, their feature constraints are not robust, often requiring additional feature constraints from the calibration board, the acquisition of large amounts of pose data to solve for accurate calibration extrinsic parameters, or the creation of more complex calibration board objects. In building scenes, there are many structured objects with strong stability in their edge features, eliminating the need for specific calibration board objects during calibration. However, in large-scale scenes, the feature search and matching process is complex, the accuracy of model constraints is low, and the model is overly complex, leading to inaccurate extrinsic parameters, poor point cloud-image mapping, and difficulty in locating and labeling camera-captured images in the corresponding scanner reference coordinates. Summary of the Invention

[0005] To address the above problems, this invention provides a method for calibrating camera-captured images relative to a scanner coordinate system, the specific solution including the following steps:

[0006] Step 1: Extract the straight line segment data of the frame structure edge in the image acquired by the camera, calculate the vanishing point position and map it to the normalized plane;

[0007] Step 2: Based on the intersection relationship of each plane of the frame structure in the point cloud obtained by the scanner, extract the edge and corner point information, map it to the normalized plane, and establish the positional relationship between it and the same object calculated by the vanishing point position of the image and mapped to the normalized plane.

[0008] Step 3: Based on their relative positions, calculate the angle difference and displacement difference mapped to the normalized plane when the camera and scanner photograph the same object;

[0009] Step 4: Adjust the camera pose. After multiple calculations of angle and displacement differences and nonlinear optimization iterations, obtain the optimal external parameters of the camera. Evaluate the accuracy of the optimal external parameters. If the preset accuracy requirements are met, use the optimal external parameters to complete the calibration of the camera image relative to the scanner coordinate system.

[0010] Preferably, the ratio of the number of pixels of the frame structure edge envelope in the image acquired by the camera to the camera resolution is within a set range.

[0011] Preferably, in step 1, the vanishing point position of the image is calculated by classifying the line segment data according to the vanishing point direction, and then calculating the vanishing point position of the classified line segment data using a least squares adjustment model.

[0012] The LSD (Line Segment Detector) line detection algorithm is used to improve line detection efficiency and effectively filter irrelevant data such as lawns and trees. By constructing a conditional least squares adjustment model to calculate the pixel coordinates of the vanishing point, the detection accuracy of the vanishing point is improved, as is the extraction accuracy of the edge vector features corresponding to the vanishing point.

[0013] Preferably, in step 2, the extraction is to directly extract the straight line segment data of the frame structure edge using a straight line detection algorithm.

[0014] Preferably, in step 4, the Levenberg-Marquardt algorithm is used to perform nonlinear optimization iterative calculations.

[0015] Preferably, in step 4, the accuracy assessment of the optimal extrinsic parameters involves projecting the feature points onto a normalized plane using the optimized extrinsic parameters and verifying whether the error between the pixel coordinates of the projected points and the pixel coordinates of the feature points in the real image is within a preset range.

[0016] Preferably, the normalized plane coincides with the plane of the original two-dimensional image captured by the camera.

[0017] The error is calculated using the following function:

[0018]

[0019] In the formula, R and t are the rotation matrix and translation vector, respectively, and x... i and X i K represents the intersection of a two-dimensional straight line in the image of the wall column edge and a three-dimensional straight line in the point cloud acquired by the scanner, where K is the camera intrinsic parameter matrix.

[0020] Another aspect of the present invention is to provide a non-transient readable recording medium for storing one or more programs containing multiple instructions, which, when executed, cause the processing circuit to perform the above-described method for calibrating a camera-captured image relative to a scanner coordinate system.

[0021] Another aspect of the present invention provides a calibration system for camera-captured images relative to a scanner coordinate system, comprising a processing circuit and a memory electrically coupled thereto, the memory being configured to store at least one program, the program containing multiple instructions, the processing circuit running the program being able to execute the aforementioned calibration method for camera-captured images relative to a scanner coordinate system.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] Compared to existing LiDAR and camera joint calibration methods for building scenes, this invention introduces the vanishing point attributes from computer vision into the calibration scene. This allows for direct construction of the correspondence between point clouds and feature lines in image data, avoiding the complex problems of traversing feature searches and matching. Frame structures are ubiquitous in cities. For point cloud data of frame structures, edge vectors and corner points can be easily obtained with the existence of three intersecting planes. Using the method of this invention, the image captured by the camera can be easily located and labeled in the scanner's reference coordinates corresponding to the point cloud. The LSD algorithm directly extracts building edge lines and sets a length threshold to filter straight lines without obvious structure. Compared to existing methods that extract contour lines first and then straight lines, this improves the efficiency of the feature detection process, increasing feature line extraction efficiency by more than 25%. The method uses parameterized least squares modulus... The method improves the accuracy of vanishing point coordinates by solving for them. Using the factory calibration parameters as a reference, the optimized calibration parameters of this invention show a significant improvement in rotation angle error. The rotation angle error in the X-axis direction is about 0.67°, the rotation angle error in the Y-axis direction is less than 0.1°, and the rotation angle error in the Z-axis direction is 0.16°. The translation error in each direction is reduced by about 1 cm compared to the initial value. The optimization results of the extrinsic parameters of the calibration are evaluated using reprojection error. The reprojection error of this invention is about 3 pixels, which is about one pixel higher than that of existing methods. Attached Figure Description

[0024] Figure 1These are original photographs of the building structure in embodiments of the present invention;

[0025] Figure 2 This is a comparison diagram of building straight line detection and filtering in an embodiment of the present invention;

[0026] Figure 3 This is a point cloud of a building in different poses and an extracted image of the wall and column edges in an embodiment of the present invention;

[0027] Figure 4 This is a schematic diagram of point-line constraints for building walls and columns in an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be described below with reference to the accompanying drawings. The described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without innovative effort are within the scope of protection of the present invention.

[0029] An embodiment of a transistor structure provided by the present invention is as follows:

[0030] A specific implementation example is the underground parking garage on the second basement level of a building in a certain district of a certain city. Figure 1 As shown, the experimental subjects are mainly wall columns, which have parallel line characteristics in different directions.

[0031] First, line segment detection and building edge vector extraction are performed on the image data and point cloud collected by the camera, respectively. The line segments are classified taking into account the vanishing point direction. The vanishing point coordinates of the classified lines are solved by the least squares adjustment model.

[0032] The first step in vanishing point detection is line extraction;

[0033] This solution uses LSD (Line Detection Algorithm) to detect building edges and straight lines with large gray-level gradient changes. Objects in building scenes are mostly structured representations. Considering the large number of straight lines in building scenes, the advantage of using the LSD algorithm is that it can directly detect straight line segments in the image, reflecting more realistic edges, without needing to perform contour extraction first and then line segment detection. Furthermore, by setting a length threshold for straight line segments (determined based on the camera's shooting distance), line segments smaller than the threshold are filtered out. This effectively filters out short line segments without obvious structural features while preserving the linear features of the buildings, thus avoiding the loss of excessive edge features. A comparison of the straight line detection results with those of filtered-out short line segments is provided. Figure 2 As shown.

[0034] The RANSAC (Random Sample Consensus) algorithm is used to detect vanishing points on the filtered lines, and the line segments to which the vanishing points belong are classified.

[0035] Considering the specific nature of the joint calibration experiment scenario in this invention, feature lines of buildings are extracted using LSD (Line Detection Algorithm) to form a line segment dataset. Then, the RANSAC method is used for vanishing point detection and line classification. Two line segments are randomly selected from the line segment dataset, and the initial vanishing point is estimated by simple line intersection. The distance from the vanishing point to other line segments in the dataset is estimated, a distance threshold is set, and line segments with distances less than the threshold are added to the set of line segments with estimated vanishing points. The intersection positions of the line segments in the set are estimated again, and the average coordinates of the intersection points are taken as the vanishing point position. The above steps are repeated, and finally, the set of line segments with the most vanishing points and the smallest model error is stored. The vanishing point position is calculated using the least squares algorithm. Each vanishing point and its corresponding line classification set are calculated. From the remaining line segments, the RANSAC algorithm is used again to obtain the next line set and solve for the vanishing point.

[0036] The vanishing points of the sets of lines in each direction can be determined by their intersection. Three vanishing points can be obtained from the perspective projection of an image. Typically, building images contain two horizontal vanishing points and one vertical vanishing point, corresponding to three sets of parallel lines. Errors are introduced during image processing, feature extraction, and calculation, causing the projections of these parallel lines in 3D space onto the image to not completely intersect at the same location. For cases where the intersection points of a single set of line segments do not converge at a single point, a conditional adjustment model with parameters is constructed, and the vanishing point coordinates are calculated using redundant observations.

[0037] Assuming the two endpoints of a line segment in the set of parallel lines are denoted as i and j, and the vanishing point of the line is denoted as v, construct the equation for the three points to be collinear:

[0038] (y v -y i (x) j -x i )-(x v -x i )(y j -y i ) = 0

[0039] In the formula, (x, y) represents the image coordinates of the endpoints. The linear equation expression after differentiating the parameters is:

[0040] [(y j -y v )v xi +(x v -x j )v yi +(y v -yi )v xj +(x i -x v )v yj +(y i -y j )v x +(x j -x i )v y +d0=0

[0041] d0=(y v -y i (x) j -x i )-(x v -x i )(y j -y i )

[0042] Based on all the lines in the classified set of line segments, an adjustment model is constructed using redundant observations:

[0043] A n×4n V 4n×1 +B n×2 x 2×1 +W n×1 =0

[0044] In the formula, n is the number of lines in the set of parallel lines with the same vanishing point direction, and V = [v xi v yi v xj v yj ] T Let x be the observation vector, and x = [v] x v y ] T Let A and B be the coefficient matrices of the observation vector and the unknowns, respectively, and W be the closure error matrix. The adjustment model is a conditional adjustment model with parameters, and its solution takes the form shown below:

[0045]

[0046]

[0047] In the formula, N aa =AP -1 A T , Finally, the coordinates of the vanishing point after adjustment are obtained.

[0048] Secondly, point cloud and image data were collected using the lidar and camera sensors in the LiDAR system. The data acquisition of the sensor equipment was controlled by a computer. The middle few frames of the point cloud analysis for each pose were used as the raw data for the experiment, and the distortion-free images after distortion correction were used as the raw image data.

[0049] Finally, the vanishing point information is fused to provide constraints on edge vector pairs, thereby constructing a constraint solution for the rotation matrix. The translation amount is solved based on the point-line feature constraints. Finally, the optimal extrinsic parameters are solved by nonlinear optimization LM algorithm based on the established minimum reprojection function.

[0050] Vanishing points can serve as rotational angular constraints for rigid transformations between the camera and scanner coordinate systems. When the projected plane of a building intersects all three coordinate axes in space, according to the principles of perspective imaging, three vanishing points will exist in the image: two horizontal vanishing points and one vertical vanishing point. Vanishing point coordinates can represent the vector direction of wall and column edges. The coordinates of the three vanishing points and the three edge vector directions of the building's corners in the scanner coordinate system provide a constraint relationship for the rotation matrix.

[0051] like Figure 3 As shown, in the point cloud of the building wall, there are cases where the vertical side intersects with the ground. By calculating the plane normal through the relationship of the intersection of the building plane, the edge vectors in three directions of the building corner are solved. These correspond to the three edge line directions corresponding to the vanishing point of the image. The vector pairs of the three edge directions all point to the plane at infinity. Thus, the rotation parameters between the scanner coordinate system and the camera coordinate system can be calculated.

[0052] Based on the known rotation angle, the translation amount is estimated using point-line constraints, such as... Figure 4 As shown, assuming both the camera and the scanner can observe a straight line L in three-dimensional space, let l be the coordinates of the three-dimensional line projected onto the camera's imaging plane. In this case, the optical center, the two-dimensional image line, and the three-dimensional line lie on the same plane, where plane I is a plane with Z... C The normalized image plane with coordinate 1. Two-dimensional image lines are represented in the camera coordinate system as:

[0053] ax + by + c = 0

[0054] In the formula, n = (a, b, c) is defined as the normal vector of the normalized plane, and (x, y, 1) are the coordinates of the image point. Considering the Z-axis of a line in the actual image plane... C Since the coordinates are determined by the camera's inherent parameters, the two-dimensional image line can be represented in the pixel coordinate system as ku + v + b = 0. Assuming the pixel coordinates of a point on the image plane are (u, v), and its coordinates in the normalized image plane are (x, y), the transformation relationship between the two coordinate systems and the formula for normal vector transformation are as follows:

[0055] (uv 1) T =K·(xy 1) T

[0056] By transforming pixel coordinates to image space coordinates, we can derive the formula for transforming the normal vector:

[0057]

[0058] In the formula, K is the intrinsic parameter matrix of the camera, m i =(k i 1 b i ) T .

[0059] In the scanner coordinate system, the direction vector d and point p can be represented as Rd and PR+t in the camera coordinate system. The rotation matrix R and the translation vector t describe the rigid transformation between the lidar coordinate system and the camera coordinate system. Since both vectors are on the normalized plane, the following constraint relationship can be obtained:

[0060]

[0061] In the formula, R and t are the rotation and translation matrices for the rigid transformation between the lidar coordinate system and the camera coordinate system, respectively, and d i Let p be the direction vector of the i-th line in the scanner coordinate system. i Let n be a point along the direction of the i-th straight line. i It is the normal vector of the normalized plane.

[0062] Each 2D-3D point-line pair in each pose provides two constraints. Typically, at least two pose datasets are required. From these datasets, any three non-intersecting line pairs are selected, and the translation amount containing three unknowns is solved. After solving the rotation matrix and translation vector, the joint calibration extrinsic parameters can be obtained.

[0063] When more pose data is acquired, more accurate rotation matrices and translation vectors are found through certain constraints. Initial pose transformation parameters are calculated using datasets from two poses. 3D feature points are back-projected onto the image plane using these initial parameters. There is a reprojection error between the pixel coordinates of the projected points and the true pixel coordinates of the image feature points. The constructed error function is as follows:

[0064]

[0065] In the formula, R and t are the rotation matrix and translation vector, respectively, and x... i and X i K represents the intersection of a two-dimensional line in the image and a three-dimensional line in the point cloud at the edge of the wall column, and K is the camera intrinsic parameter matrix.

[0066] The goal of optimization is to minimize the reprojection error. The intersection of the two-dimensional and three-dimensional lines at the edges of the walls and columns in the dataset is defined as x. i and X i Before establishing the error function, initial extrinsic parameters with low accuracy are treated as outliers and filtered out before optimization. A threshold for reprojection error is set to distinguish the initial solution and eliminate the influence of noise as much as possible. Based on feature point pairs, the LM nonlinear optimization algorithm is used to iteratively solve for the joint calibration extrinsic parameters to minimize the reprojection error.

[0067] Accuracy evaluation after nonlinear optimization;

[0068] The accuracy evaluation experiment is divided into qualitative evaluation and quantitative evaluation. Since the true external parameters of the scanner and camera in the joint calibration experiment are unknown, the back projection error function is constructed using the coordinates of the corner points of the wall columns as the coordinates of the feature points. The back projection error is used for quantitative evaluation, and the visualization mapping between the image and the point cloud is used for qualitative evaluation.

[0069] The rotation angle error is significantly improved by LM optimization. The optimized extrinsic parameters are used to project the feature points onto the pixel plane. The error between the pixel coordinates of the projected points and the pixel coordinates of the corners of the real image is compared. The extrinsic parameters have a smaller error.

[0070] Based on the collected corner feature point pairs, the error between the pixel coordinates of the projected points and the pixel coordinates of the corners in the real image is compared. Five pose data points are randomly selected, totaling 15 feature points, to observe the error. The maximum reprojection error of the selected key feature points does not exceed 6.5 pixels, the minimum is not less than 1.7 pixels, and the average reprojection error is 3.4 pixels, which meets the preset optimal extrinsic parameter accuracy requirements. Therefore, this optimal extrinsic parameter is used to complete the calibration of the camera-captured image relative to the scanner coordinate system.

[0071] This solution effectively solves the problem of inaccurate and inefficient acquisition of joint calibration extrinsic parameters of scanners and cameras during building scene calibration, and resolves the issues of inaccuracy in feature extraction and low robustness of feature constraints. It enables convenient and accurate acquisition of joint calibration extrinsic parameters.

[0072] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computers or available storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0073] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0074] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0075] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0076] The technical solution of the present invention is "a non-transitory readable recording medium" by assembling the above method steps into a program and storing it on a hard disk or other non-transitory storage medium; and the technical solution of the present invention is "a calibration system for camera images relative to scanner coordinate system" by electrically connecting the storage medium to a computer processor and calibrating the camera-captured image relative to the scanner coordinate system through data processing.

[0077] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for calibrating a camera-captured image relative to a scanner coordinate system, characterized in that... Includes the following steps: Step 1: Extract the line segment data of the frame structure edge in the image acquired by the camera, calculate the vanishing point position and map it to the normalized plane; the line segment data and vanishing points are obtained by: extracting the feature lines of the building through a line detection algorithm to form a line segment dataset, and then using the RANSAC method to perform vanishing point detection and line classification. Step 2: Based on the intersection relationship of each plane of the frame structure in the point cloud obtained by the scanner, extract the edge and corner point information, map it to the normalized plane, and establish the positional relationship between it and the same object calculated by the vanishing point position of the image and mapped to the normalized plane. Step 3: Based on their relative positions, calculate the angle difference and displacement difference mapped to the normalized plane when the camera and scanner photograph the same object; Step 4: Adjust the camera pose. After multiple calculations of angle and displacement differences and nonlinear optimization iterations, obtain the optimal external parameters of the camera. Evaluate the accuracy of the optimal external parameters. If the preset accuracy requirements are met, use the optimal external parameters to complete the calibration of the camera image relative to the scanner coordinate system.

2. The method for calibrating a camera-captured image relative to a scanner coordinate system according to claim 1, characterized in that, The ratio of the number of pixels encompassing the edge of the frame structure in the image acquired by the camera to the camera resolution is within a set range.

3. The method for calibrating a camera-captured image relative to a scanner coordinate system according to claim 2, characterized in that, In step 1, the vanishing point position of the image is calculated by classifying the line segment data according to the vanishing point direction, and then calculating the vanishing point position of the classified line segment data using a least squares adjustment model.

4. The method for calibrating a camera-captured image relative to a scanner coordinate system according to claim 3, characterized in that, In step 2, the extraction is to directly extract the straight line segment data of the frame structure edge using a straight line detection algorithm.

5. The method for calibrating a camera-captured image relative to a scanner coordinate system according to claim 4, characterized in that, In step 4, the Levenberg-Marquardt algorithm is used for nonlinear optimization iterative calculations.

6. The method for calibrating a camera-captured image relative to a scanner coordinate system according to claim 5, characterized in that, In step 4, the accuracy assessment of the optimal extrinsic parameters involves projecting the feature points onto a normalized plane using the optimized extrinsic parameters and verifying whether the error between the pixel coordinates of the projected points and the pixel coordinates of the feature points in the real image is within a preset range.

7. The method for calibrating a camera-captured image relative to a scanner coordinate system according to claim 6, characterized in that, The normalized plane coincides with the plane of the original two-dimensional image captured by the camera. The error is calculated using the following function: In the formula, R and t are the rotation matrix and translation vector, respectively, and x... i and X i K represents the intersection of a two-dimensional straight line in the image of the wall column edge and a three-dimensional straight line in the point cloud acquired by the scanner, where K is the camera intrinsic parameter matrix.

8. A non-transitory readable recording medium for storing one or more programs containing multiple instructions, characterized in that, When the instruction is executed, the processing circuit will perform a calibration method for a camera-captured image relative to a scanner coordinate system, as described in any one of claims 1-7.

9. A calibration system for camera-captured images relative to a scanner coordinate system, comprising a processing circuit and a memory electrically coupled thereto, characterized in that, The memory is configured to store at least one program, the program containing multiple instructions, and the processing circuit runs the program to perform a calibration method for a camera-captured image relative to a scanner coordinate system, as described in any one of claims 1-7.

Images