A method, system and device for calibrating equipment in a map surveying and mapping system

Abstract

This invention discloses a method, apparatus, and system for calibrating equipment in a map surveying system. The method includes: extracting linear elements from a ground image captured by an imaging device; extracting the outer contour points of the ground elements from acquired point cloud data; projecting the outer contour points onto the ground image to obtain a matching relationship between the outer contour points and the linear elements; and determining imaging device extrinsic calibration parameters based on the matching relationship and the pose of an inertial navigation device that minimize the distance from the outer contour points to the image plane containing the matching linear elements. The image plane is the plane formed by the linear element matching the outer contour points and the optical center of the imaging device, and the imaging device extrinsic calibration parameters include the spatial transformation relationship between the imaging device and the inertial navigation device. This method achieves spatial relationship calibration between devices without the need for a dedicated calibration device, provides high calibration accuracy, and eliminates the need for manual operation, enabling rapid and efficient automated calibration.

Description

Technical Field

[0001] The present invention relates to the field of map surveying and mapping technology, and in particular to a method, system and device for calibrating equipment in a map surveying and mapping system. Background Art

[0002] In order to update the vector elements in the HD map (such as lane lines, traffic signs, etc.), it is necessary to use the images and trajectories collected by the Advanced Driving Assistant System (ADAS) as data sources to generate and update the vector elements. After ADAS collects the data, it usually needs to go through processes such as hand-eye calibration, image positioning, and 3D reconstruction to finally generate the vector elements in the HD map.

[0003] Hand-eye calibration can be used to calibrate the spatial relationship of each device in a mapping system such as ADAS, so as to determine the relative position relationship of each device in the mapping system in space. For example, the spatial relationship between the imaging device and the inertial navigation device in the ADAS can be calibrated to obtain the spatial relationship between the external sensors (such as cameras, radars, etc.) in the ADAS and the inertial navigation device or the vehicle body, which can be characterized by the relative translation or rotation relationship between the coordinate systems of each other. According to the calibrated spatial relationship between each device, the data collected by different devices can be converted to a unified coordinate system to facilitate the fusion of the data collected by each device.

[0004] Traditional hand-eye calibration generally involves setting up special calibration equipment on the ADAS, and manually operating the calibration equipment to calibrate the various devices in the ADAS. This method requires a lot of human resource costs, is inconvenient to operate, cannot achieve automated calibration, and the accuracy of the calibration operation is relatively low. It is limited by the experience and operational proficiency of the calibrators, and cannot complete hand-eye calibration quickly and efficiently. Summary of the invention

[0005] In view of the above problems, the present invention is proposed to provide a method, system and apparatus for calibrating equipment in a map surveying and mapping system that overcomes the above problems or at least partially solves the above problems.

[0006] An embodiment of the present invention provides a device calibration method in a map surveying and mapping system, comprising:

[0007] Extracting linear elements from ground images taken by imaging devices;

[0008] Extracting outer contour points of ground elements from the acquired point cloud data;

[0009] Projecting the outer contour points into the ground image to obtain a matching relationship between the outer contour points and the linear elements;

[0010] According to the matching relationship and the posture of the inertial navigation device, the extrinsic calibration parameters of the imaging device that minimize the distance from the outer contour point to the image plane where the matching linear element is located are determined; the image plane is a plane formed by the linear element matching the outer contour point and the optical center of the imaging device, and the extrinsic calibration parameters of the imaging device include the spatial transformation relationship between the imaging device and the inertial navigation device.

[0011] In some optional embodiments, extracting linear elements from a ground image captured by an imaging device includes:

[0012] Extracting linear elements from a ground image captured by an imaging device by using a selected straight line extraction algorithm and obtaining the gray intensity gradient direction of the linear elements; or

[0013] The road elements in the ground image taken by the imaging device are segmented, and the linear elements included in the ground image are fitted through the machine learning algorithm to obtain the grayscale intensity gradient information of the linear elements.

[0014] In some optional embodiments, extracting outer contour points of ground elements from the acquired point cloud data includes:

[0015] Extracting point cloud data of a selected range from point cloud data acquired by a point cloud device according to the geographical area covered by the ground image;

[0016] According to the connectivity of the reflectivity of the laser points in the point cloud data, the connected laser points are obtained to obtain the outer contour points of the ground elements and the corresponding reflectivity and grayscale intensity gradient information.

[0017] In some optional embodiments, projecting the outer contour points into the ground image to obtain a matching relationship between the outer contour points and the linear elements includes:

[0018] Projecting the outer contour points into the ground image according to the external parameters of the imaging device and the position and posture of the inertial navigation device;

[0019] According to the grayscale intensity gradient information of the outer contour point and the grayscale intensity gradient information of the linear element, the nearest linear element having the same direction as the grayscale intensity gradient of the outer contour point is determined as the linear element matching the outer contour point.

[0020] In some optional embodiments, projecting the outer contour points into the ground image according to the external parameters of the imaging device and the position and posture of the inertial navigation device includes:

[0021] The product of the coordinates of the outer contour point and the external parameters of the imaging device, the posture of the inertial navigation device, and the internal parameters of the imaging device is calculated to obtain the projection coordinates of the outer contour point in the ground image.

[0022] In some optional embodiments, the determining of the nearest linear element having a consistent direction with the grayscale intensity gradient of the outer contour point comprises:

[0023] The distance from the outer contour point in the point cloud data to the image plane where the matching linear element is located is minimized by descending the grayscale intensity gradient, so as to obtain a linear element with the same grayscale intensity gradient direction as the outer contour point and the smallest distance.

[0024] In some optional embodiments, determining the imaging device extrinsic calibration parameters that minimize the distance from the outer contour point to the image plane according to the matching relationship and the inertial navigation device posture includes:

[0025] Presetting an imaging device external parameter calibration parameter, wherein the imaging device external parameter calibration parameter includes a spatial conversion relationship between the imaging device and the inertial navigation device;

[0026] According to the imaging device extrinsic calibration parameters, the inertial navigation device posture, the corresponding outer contour points and linear elements included in the matching relationship, construct a distance expression from the outer contour point to the plane formed by the matching linear element and the optical center of the imaging device;

[0027] For each projection, an increment of the extrinsic calibration parameter of the imaging device is obtained according to the distance expression. If the increment does not meet the preset conditions, the extrinsic calibration parameter of the imaging device is updated according to the increment, and then the outer contour point is re-projected into the ground image to obtain the matching relationship between the outer contour point and the linear element, and the increment is re-determined until the increment that meets the preset conditions is obtained or the number of iterations reaches a preset threshold number.

[0028] In some optional embodiments, the imaging device external parameter calibration parameters also include a system deviation between the inertial navigation device and the point cloud device, and the position and posture of the inertial navigation device used for distance calculation is adjusted according to the system deviation.

[0029] In some optional embodiments, the position and posture of the inertial navigation device is calculated based on the GNSS positioning information and the positioning base station information.

[0030] The embodiment of the present invention further provides a device calibration apparatus in a map surveying and mapping system, comprising:

[0031] A linear element extraction module, used for extracting linear elements from a ground image taken by an imaging device;

[0032] A contour extraction module is used to extract the outer contour points of ground elements from the acquired point cloud data;

[0033] A matching module, used for projecting the outer contour points into the ground image to obtain a matching relationship between the outer contour points and the linear elements;

[0034] A calibration module is used to determine the imaging device extrinsic calibration parameters that minimize the distance from the outer contour point to the image plane where the matching linear element is located according to the matching relationship and the inertial navigation device posture; the image plane is a plane formed by the linear element matching the outer contour point and the optical center of the imaging device, and the imaging device extrinsic calibration parameters include the spatial transformation relationship between the imaging device and the inertial navigation device.

[0035] The embodiment of the present invention further provides a map surveying and mapping system, characterized in that it comprises: an imaging device, a point cloud device, an inertial navigation device, and a device calibration device in the above-mentioned map surveying and mapping system;

[0036] The equipment calibration device in the map surveying and mapping system determines the external calibration parameters of the imaging device based on the ground image taken by the imaging device, the point cloud data acquired by the point cloud device and the position and posture of the inertial navigation device.

[0037] The beneficial effects of the above technical solution provided by the embodiment of the present invention include at least:

[0038] Through the matching relationship between the linear elements in the ground image taken by the imaging device and the outer contour points of the ground elements in the point cloud data, combined with the position and posture of the inertial navigation device, the calibration parameters that minimize the distance from the outer contour points of the ground elements in the point cloud data to the image plane of the imaging device are solved, that is, the spatial transformation relationship of the imaging device relative to the inertial navigation device, so as to accurately calibrate the spatial position of the imaging device relative to the inertial navigation device, so that the ground image taken by the imaging device and the coordinate system of the map surveying and mapping equipment can be integrated, conveniently realize data conversion, and facilitate unification of data into the same coordinate system for calculation and processing. This calibration method does not require additional calibration plates or calibration equipment to realize calibration, does not require manual operation, is easy to implement, has a high degree of automation, and has high calibration accuracy.

[0039] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0040] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0042] Figure 1 This is a flow chart of a device calibration method in a map surveying and mapping system in Embodiment 1 of the present invention;

[0043] Figure 2 This is a block diagram of the implementation principle of the device calibration method in the map surveying and mapping system in the second embodiment of the present invention;

[0044] Figure 3 A schematic diagram of extracting contour points of point cloud data in Embodiment 2 of the present invention;

[0045] Figure 4 A schematic diagram of extracting linear elements from a ground image in Embodiment 2 of the present invention;

[0046] Figure 5 A schematic diagram of matching contour points and linear elements in the second embodiment of the present invention;

[0047] Figure 6 A schematic diagram of minimizing distance to achieve external parameter calibration in the second embodiment of the present invention;

[0048] Figure 7 It is a schematic diagram of the device calibration structure in the map surveying and mapping system in an embodiment of the present invention;

[0049] Figure 8 It is a schematic diagram of the structure of the equipment calibration device in the map surveying and mapping system in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0051] In a map surveying and mapping system, there are usually imaging devices such as cameras, point cloud devices for acquiring point cloud data, inertial navigation devices, etc. In order to unify the data acquired by each device into the same coordinate system for integration and use, it is generally necessary to know the spatial relationship between the devices, such as the rotation and translation relationship between each other's coordinate systems. Therefore, it is necessary to calibrate the external parameters of the imaging device in the surveying and mapping system in order to know the spatial transformation relationship between the imaging device and other devices.

[0052] In the prior art, the external parameter calibration of surveying and mapping system equipment used for high-precision maps requires manual calibration with the help of calibration equipment such as calibration plates, which has low accuracy and inconvenient operation. Therefore, an embodiment of the present invention provides a device calibration method in a map surveying and mapping system, which can automatically complete the equipment calibration without the help of additional calibration equipment, with high calibration accuracy, fast and efficient.

[0053] The following is a detailed description through specific embodiments.

[0054] Embodiment 1

[0055] The device calibration method in the map surveying and mapping system provided in the first embodiment of the present invention has the following process: Figure 1 As shown, the following steps are included:

[0056] S11: extracting linear elements from a ground image captured by an imaging device.

[0057] The imaging device can capture ground images, and the captured ground images will contain various ground elements. For example, for a road, it may contain at least one of lanes, lane lines, lane center lines, lane side lines, road boundaries, traffic lights, sidewalks, traffic sign lines, and objects around the road. After capturing the ground image, the ground elements in the ground image can be extracted, for example, by extracting linear elements in the image to obtain ground elements through linear element extraction.

[0058] The main goal of linear element extraction is to extract straight line features. The linear elements can be extracted from the ground image taken by the imaging device through the selected Line Segment Detector (LSD) algorithm and the gray intensity gradient direction of the linear elements can be obtained.

[0059] Optionally, fine segmentation can be performed by deep learning methods, and then the line features of the edge can be fitted. That is, the road elements in the ground image taken by the imaging device are segmented, and the linear elements included in the ground image are fitted by a machine learning algorithm and the grayscale intensity gradient information of the linear elements is obtained. The grayscale intensity gradient information may include the grayscale intensity gradient direction.

[0060] S12: Extracting outer contour points of ground elements from the acquired point cloud data.

[0061] Point cloud data acquired by point cloud devices are generally presented in the form of countless laser points, such as the point cloud scanned by the Riegl laser radar currently used. Since different objects have different reflectivities, for laser points in point cloud data, if the reflectivities of laser points in a certain neighborhood are also similar, they are considered to be connected.

[0062] In this step, the outer contour points of the ground elements in the point cloud data are extracted to realize the extraction of the ground elements. The preferred approach is to obtain the required point cloud data according to the geographical area involved in the ground image in order to match it with the ground image taken by the above imaging device, and obtain point cloud data with roughly the same area.

[0063] Therefore, this step first extracts the point cloud data of the selected range from the point cloud data obtained by the point cloud device according to the geographical area involved in the ground image; then, according to the connectivity of the reflectivity of the laser points in the point cloud data, the laser points with connectivity are obtained to obtain the outer contour points of the ground elements and the corresponding reflectivity and grayscale intensity gradient information. Among them, the grayscale intensity gradient information may include the direction of the grayscale intensity gradient. For example, the reflectivity of the laser points can be used to find the laser points that are interconnected within a certain range to extract the edge of a ground element, that is, the outer contour point. The ground elements can be one or more of arrows, text, zebra crossings, guide strips, etc.

[0064] Optionally, the outer contour points of the ground elements can be extracted by a more general method, such as through space, based on features such as contour curvature and normal components of laser points.

[0065] S13: Projecting the outer contour points of the extracted ground elements into the ground image to obtain a matching relationship between the outer contour points of the ground elements in the point cloud data and the linear elements in the ground image.

[0066] When matching, the outer contour points in the point cloud data are projected into the ground image in order to match the same elements in the point cloud data and the ground image: according to the external parameters of the imaging device and the position and posture of the inertial navigation device, the outer contour points of the ground elements in the point cloud data are projected into the ground image; according to the grayscale intensity gradient information of the outer contour points and the grayscale intensity gradient information of the linear elements, the nearest linear element with the same direction as the grayscale intensity gradient of the outer contour points is determined as the linear element matching the outer contour points.

[0067] During the first projection, the outer contour points of the ground elements in the point cloud data are projected into the ground image according to the initial external parameters of the imaging device and the position and posture of the inertial navigation device; if the increment of the calibration parameters of the external parameters of the imaging device determined in the subsequent steps does not meet the preset conditions and needs to be projected again according to the adjusted external parameters of the imaging device, the outer contour points of the ground elements in the point cloud data are projected into the ground image according to the adjusted external parameters of the imaging device and the position and posture of the inertial navigation device, and then matching is performed.

[0068] When projecting the outer contour points of the ground elements in the point cloud data into the ground image, the coordinates of the outer contour points are multiplied by the external parameters of the imaging device, the posture of the inertial navigation device, and the internal parameters of the imaging device to obtain the projection coordinates of the outer contour points in the ground image. That is, the projection coordinates of the outer contour points in the ground image are equal to the product of the coordinates of the outer contour points and the external parameters of the imaging device, the posture of the inertial navigation device, and the internal parameters of the imaging device. For example, for the three-dimensional point coordinates Q in the point cloud data, the posture matrix of the inertial navigation device is known. Imaging device extrinsic matrix The camera intrinsic parameter matrix K, then the corresponding pixel coordinates of the three-dimensional point Q after being projected into the ground image are:

[0069] In one example, the external parameters of the imaging device, also known as the imaging device posture, may include: 1 / dx, 1 / dy, r, u0, v0, f; dx and dy represent: how many length units a pixel occupies in the x-direction and y-direction, respectively, that is, the size of the actual physical value represented by a pixel, which is the key to realizing the conversion between the image physical coordinate system and the pixel coordinate system. u0, v0 represent the horizontal and vertical pixel numbers between the center pixel coordinate of the image and the pixel coordinate of the origin of the image, f is the focal length of the imaging device, and r can be a constant of 1.

[0070] The external parameters of the imaging device may include T and R, where R represents the rotation parameter. The rotation parameters of the three axes of the imaging device coordinate system are ω, δ, and θ, respectively. The 3*3 rotation matrices of each axis are combined, that is, the matrices are multiplied first to obtain the rotation parameter R that combines the rotation information of the three axes, and its size is still 3*3; T represents the translation parameters of the three axes (Tx, Ty, Tz).

[0071] The position and attitude of the inertial navigation device can be settled based on the information of GNSS or differential base stations. The position and attitude of the inertial navigation device includes the spatial coordinates, pitch angle, yaw angle, roll angle, acceleration increment, and angular velocity increment of the inertial navigation device.

[0072] The above-mentioned determination of the nearest linear element having the same direction as the grayscale intensity gradient of the outer contour point includes minimizing the distance from the outer contour point in the point cloud data to the image plane where the matching linear element is located by descending the grayscale intensity gradient, thereby obtaining the linear element having the same direction as the grayscale intensity gradient of the outer contour point and the smallest distance.

[0073] This step realizes point cloud-image feature matching. The initial external parameters and inertial navigation posture of the imaging device are used to project the point cloud contour points to the image plane, and the nearest neighbor straight line features with consistent gradient directions are searched in the image plane as the matching results of the point cloud contour points. The image plane is a spatial plane composed of the linear features in the image and the optical center of the imaging device. The distance between the point cloud contour points and the image plane is minimized by gradient descent to optimize the posture of the imaging device in the point cloud coordinate system. The above steps are repeated until the algorithm converges and the best matching relationship between the point cloud and image features is found.

[0074] S14: According to the obtained matching relationship and the inertial navigation device posture, determine the imaging device extrinsic calibration parameters that minimize the distance from the outer contour point to the image plane where the matching linear element is located. The image plane is a plane formed by the linear element matching the outer contour point and the optical center of the imaging device, and the imaging device extrinsic calibration parameters include the spatial transformation relationship of the imaging device relative to the inertial navigation device.

[0075] After completing the search and matching of the point features of the nearest point cloud data and the line features of the image data, a hand-eye calibration variable (X) of the imaging device can be set, and a function D = f (X, V) can be constructed based on the matching result (V), where X = argmin x (D) By minimizing the distance D between points, lines or planes, the variable X is solved and the optimal solution of X is obtained through iteration, and the external calibration parameters of the imaging device, i.e., the posture, can be obtained.

[0076] The process of obtaining the external calibration parameters of the imaging device includes:

[0077] Preset calibration parameters, wherein the calibration parameters include a spatial transformation relationship between the imaging device and the inertial navigation device; construct a distance expression from the outer contour point to the plane formed by the matching linear element and the optical center of the imaging device according to the calibration parameters, the inertial navigation device posture, and the corresponding outer contour points and linear elements included in the matching relationship;

[0078] For each projection, the increment of the calibration parameter is obtained according to the distance expression. If the increment does not meet the preset conditions, the calibration parameter is updated according to the increment, and the outer contour point is re-projected into the ground image to obtain the matching relationship between the outer contour point and the linear element, and the increment is re-determined until the increment that meets the preset conditions is obtained or the number of iterations reaches the preset threshold.

[0079] In an optional embodiment, the preset calibration parameters also include the system deviation between the inertial navigation device and the point cloud device, and the inertial navigation device posture used for distance calculation is adjusted according to the system deviation. When performing the first projection, the projection can be performed based on the initial external parameters of the imaging device and the calculated posture of the inertial navigation device. If it is determined that the increment does not meet the requirements later, the projection can be performed according to the updated calibration parameters, that is, the new external parameters of the imaging device. At the same time, in order to improve the calibration accuracy, the posture of the inertial navigation device can be further corrected. Therefore, the system deviation can be included in the set calibration parameters. Since the posture of the inertial navigation device is basically consistent with that of the point cloud device, the deviation is very small. Therefore, in the first projection, the calculated posture of the inertial navigation device is directly used for projection calculation, but in the future, it can be further corrected according to the system deviation to make the posture of the inertial navigation device and the system posture of the point cloud device more consistent.

[0080] In an optional embodiment, the position and posture of the inertial navigation device may be calculated based on Global Navigation Satellite System (GNSS) positioning information and positioning base station information.

[0081] The above method can realize the automatic matching of point cloud data and images, solve the on-site calibration of the inertial navigation-imaging equipment of the surveying and mapping system in scenes with a priori high-precision point cloud maps, without adding additional calibration facilities. Through the joint optimization method of the external parameter matrix of the inertial navigation-imaging equipment, the degree of automation of data processing and the calibration accuracy are improved.

[0082] Embodiment 2

[0083] Embodiment 2 of the present invention provides a specific implementation example of device calibration in the above-mentioned map surveying and mapping system, and its implementation principle is as follows: Figure 2 shown.

[0084] First, point cloud features and image features are extracted, see step S11 and step S12 of the first embodiment.

[0085] See the example diagram of point cloud feature extraction for details. Figure 3 , extract the contours of each ground element from the point cloud data, such as the lane lines, turn arrows, pedestrian crossings, etc. shown in the figure, and obtain the outer contour points of the ground elements in the point cloud data.

[0086] See the example diagram of image feature extraction for details. Figure 4 , extract the ground elements in the ground image of the same ground environment, such as lane lines, turning corners, pedestrian crossings, and even cars and buildings, and obtain linear elements in the ground image, such as Figure 4 The white lines in the

[0087] like Figure 2 As shown in the figure, the extracted features are matched to match the same ground elements in the ground image and the point cloud data. The matched examples are as follows Figure 5 As shown, the black line is Figure 3 The outer contour points of the extracted ground elements shown in the figure can be matched with Figure 4 For example, lanes, lane lines, turn arrows and pedestrian crossings at the same position, see step S13 in Example 1. When matching, the contour points in the point cloud data are projected into the ground image. When projecting, it is necessary to perform the projection according to the coordinate system corresponding to the point cloud data, the coordinate system of the imaging device, and the relative translation and rotation relationship between them (i.e., the external parameters of the imaging device).

[0088] The imaging device coordinate system is a bridge connecting the image physical coordinate system and the world coordinate system, where the coordinate origin of the imaging device coordinate system is the optical center of the lens, which is also the projection center in the imaging device coordinate system. The coordinate system of the point cloud data can be approximately equal to the coordinate system of the inertial navigation device (INS) of the point cloud device system, so the inertial navigation device position and posture can be used as calculation data. In subsequent calculations, corrections can be made based on the system deviation between the inertial navigation device coordinate system and the point cloud device system coordinate system to obtain more accurate results.

[0089] The inertial navigation posture can be solved by combining GNSS and base station information. Using a differential GNSS reference station with known accurate three-dimensional coordinates, such as a base station, the pseudo-range correction, position correction or phase correction is obtained, and then this correction is sent to the GNSS receiver in real time or afterwards to correct the measurement data to improve the positioning accuracy of the GNSS or combined inertial navigation system.

[0090] Then, if Figure 2 As shown in the figure, based on the matching relationship and inertial navigation pose, the external parameter matrix of the inertial navigation-imaging device and the system deviation of the inertial navigation trajectory and the point cloud coordinate system are solved through the joint nonlinear optimization of multiple image features. The inertial navigation pose is adjusted based on the system deviation, that is, the post-solution of the inertial navigation pose.

[0091] The specific optimization process can be achieved by solving a least squares problem, such as

[0092] (1) The extrinsic calibration parameter of the imaging device to be solved is defined as X.

[0093] The defined imaging device extrinsic calibration parameter X may include the imaging device extrinsic parameter, that is, the rotation and / or translation relationship of the imaging device relative to the point cloud coordinate system, and optionally, may also include a system deviation.

[0094] (2) The inertial navigation posture and matching relationship obtained above are input into the equation.

[0095] The inertial navigation posture can be expressed as P = (P1, P2, ... Pi ..., Pn), where Pi is the inertial navigation posture corresponding to the i-th picture, including the three-axis coordinates xyz of the inertial navigation and six variables such as roll angle (roll,), pitch angle (pitch), and yaw angle (yaw). Among them, the roll angle is the angle of rotation around the X axis, the pitch angle is the angle of rotation around the Y axis, and the yaw angle is the angle of rotation around the Z axis.

[0096] The matching relationship between the outer contour points in the point cloud data and the linear features in the image is expressed as a feature pair V = (V1, V2, ..., Vi, ..., Vn), where Vi is the linear feature corresponding to the i-th image and the point cloud contour point matching it.

[0097] (3) Use the optimization equation to perform optimization iterations to obtain the external calibration parameters of the imaging device

[0098] For the i-th image, after forming the image plane in space according to the linear elements of the image, the distance di=dist(X, Pi, Vi) from the outer contour point in the point cloud edge data to the image plane is calculated. The principle of distance calculation can be found in Figure 6 As shown, for the linear element L in the image, it forms an image plane with the optical center C of the camera coordinate system, and the contour point in the point cloud data is Figure 6 The point-to-surface distance of each point in the image plane can be calculated based on the rotation and translation (R, t) relationship between the world coordinate system and the camera coordinate system. The world coordinate system can be understood as the point cloud coordinate system, which can be expressed by the coordinate system corresponding to the inertial navigation posture.

[0099] Then for the 1st, 2nd, ..., i, ..., nth images, the sum of the distances di can be expressed as:

[0100] Where X = argmin x (D)

[0101] (4) Solving the optimization equation When solving the above equation, the Gauss-Newton method can be used to solve it, including the following steps:

[0102] 4.1) Initialize the external calibration parameter X of the imaging device to X0

[0103] 4.2) Based on X k The initial external parameters of the imaging device included in the projection are projected, and the outer contour points of the extracted ground elements are projected into the ground image to obtain the matching relationship between the outer contour points of the ground elements in the point cloud data and the linear elements in the ground image. The matching relationship between the outer contour points in the point cloud data and the linear elements in the image obtained by the first projection is used to calculate the Jacobian J, where Calculate the increment ΔX.

[0104] For the first projection, X k =X0, for the subsequent projections X k The imaging device extrinsic calibration parameter X obtained by the previous iteration update.

[0105] 4.3) Determine whether ΔX is small enough. If so, stop the iterative operation to obtain the final imaging device external parameter calibration parameter X, which is used to calibrate the imaging device;

[0106] If not, update X so that X k+1 =X k +ΔX, return and continue to execute 4.2) until ΔX meets the set conditions, that is, it is small enough.

[0107] If not, further determine whether the number of iterations reaches the set number threshold. If so, stop the iteration operation. If not, update X so that X k+1 =X k +ΔX, return and continue to execute 4.2) until ΔX meets the set conditions, that is, it is small enough.

[0108] The above method uses a priori high-precision point cloud maps to assist in the external parameter calibration between imaging devices and inertial navigation. It can be applied to environments with a priori high-precision point cloud maps, and surveying and mapping systems, such as mobile surveying and mapping platforms or equipment, can accurately self-position through GNSS and differential information. By extracting image features from external sensors of the surveying and mapping system (such as lidar, millimeter-wave radar, cameras and other imaging devices), they are matched with point cloud features in the high-precision point cloud. Through the matched features, the translation and rotation relationship between the coordinate system of the external sensor and the coordinate system of the surveying and mapping system is calibrated together with the self-positioning information of the surveying and mapping system. After calibration, the spatial conversion and data fusion between the sensors in the surveying and mapping system can be performed more accurately.

[0109] Based on the same inventive concept, the embodiment of the present invention also provides a map surveying and mapping system, whose structure is as follows: Figure 7 As shown, it includes: an imaging device 1, a point cloud device 2, an inertial navigation device 3 and a device calibration device 4.

[0110] The imaging device 1 is used to capture ground images;

[0111] Point cloud device 2 is used to obtain point cloud data;

[0112] The device calibration device 3 determines the extrinsic calibration parameters of the imaging device for the imaging device 1 based on the ground image taken by the imaging device 1, the point cloud data acquired by the point cloud device and the position and posture of the inertial navigation device 3. Specifically: the device calibration device 3 is used to extract linear elements from the ground image taken by the imaging device; extract the outer contour points of the ground elements from the acquired point cloud data; project the extracted outer contour points into the ground image to obtain the matching relationship between the outer contour points in the point cloud data and the linear elements in the ground image; according to the determined matching relationship and the position and posture of the inertial navigation device, determine the extrinsic calibration parameters of the imaging device that minimize the distance from the outer contour points in the point cloud data to the image plane where the matching linear elements are located; wherein the image plane is the plane formed by the linear elements matching the outer contour points and the optical center of the imaging device, and the extrinsic calibration parameters of the imaging device include the spatial transformation relationship of the imaging device relative to the inertial navigation device.

[0113] The embodiment of the present invention also provides a device calibration device in a map surveying and mapping system, the structure of which is as follows: Figure 8 As shown, it includes: a linear element extraction module 11, a contour extraction module 12, a matching module 13 and a calibration module 14. Among them:

[0114] A linear element extraction module 11 is used to extract linear elements from a ground image captured by an imaging device;

[0115] The contour extraction module 12 is used to extract the outer contour points of the ground elements from the acquired point cloud data;

[0116] A matching module 13 is used to project the extracted outer contour points into the ground image to obtain a matching relationship between the outer contour points in the point cloud data and the linear elements in the ground image;

[0117] The calibration module 14 is used to determine the imaging device extrinsic calibration parameters that minimize the distance from the outer contour point in the point cloud data to the image plane where the matching linear element is located based on the determined matching relationship and the inertial navigation device posture; wherein the image plane is a plane formed by the linear element matching the outer contour point and the optical center of the imaging device, and the imaging device extrinsic calibration parameters include the spatial transformation relationship between the imaging device and the inertial navigation device.

[0118] The linear element extraction module 11 is used to extract linear elements from the ground image taken by the imaging device, including: extracting linear elements from the ground image taken by the imaging device through a selected straight line extraction algorithm and obtaining the grayscale intensity gradient direction of the linear elements; or performing element segmentation on the road elements in the ground image taken by the imaging device, fitting the linear elements included in the ground image through a machine learning algorithm and obtaining the grayscale intensity gradient information of the linear elements.

[0119] The contour extraction module 12 is used to extract the outer contour points of the ground elements from the acquired point cloud data, including: extracting the point cloud data of a selected range from the point cloud data acquired by the point cloud device according to the geographical area involved in the ground image; according to the connectivity of the reflectivity of the laser points in the point cloud data, acquiring the outer contour points of the ground elements and the corresponding reflectivity and grayscale intensity gradient information by acquiring the connected laser points.

[0120] The matching module 13 is used to project the outer contour points of the extracted ground elements in the point cloud data into the ground image to obtain the matching relationship between the outer contour points and the linear elements, including: projecting the outer contour points of the extracted ground elements in the point cloud data into the ground image according to the external parameters of the imaging device and the posture of the inertial navigation device; and determining the nearest linear element with the same direction as the grayscale intensity gradient of the outer contour point as the linear element matching the outer contour point according to the grayscale intensity gradient information of the outer contour point and the grayscale intensity gradient information of the linear element.

[0121] The matching module 13 projects the outer contour points into the ground image according to the external parameters of the imaging device and the posture of the inertial navigation device, including: calculating the product of the coordinates of the outer contour points and the external parameters of the imaging device, the posture of the inertial navigation device, and the internal parameters of the imaging device to obtain the projection coordinates of the outer contour points in the ground image.

[0122] The matching module 13 is used to determine the nearest linear element that is consistent with the direction of the grayscale intensity gradient of the outer contour point, including: minimizing the distance from the outer contour point in the point cloud data to the image plane where the matching linear element is located by descending the grayscale intensity gradient, and obtaining the linear element that is consistent with the direction of the grayscale intensity gradient of the outer contour point and has the smallest distance.

[0123] The calibration module 14 is used to determine the imaging device extrinsic calibration parameters that minimize the distance from the outer contour point to the image plane according to the matching relationship and the inertial navigation device posture, including:

[0124] Presetting calibration parameters, wherein the calibration parameters include a spatial conversion relationship between the imaging device and the inertial navigation device;

[0125] According to the imaging device external parameter calibration parameters, the inertial navigation device posture, the corresponding outer contour points and linear elements included in the matching relationship, construct a distance expression from the outer contour point to the plane formed by the matching linear element and the imaging device optical center;

[0126] For each projection, the increment of the calibration parameter is obtained according to the constructed distance expression. If the increment does not meet the preset conditions, the calibration parameters are updated according to the increment, and the outer contour points are re-projected into the ground image to obtain the matching relationship between the outer contour points and the linear elements, and the increment is re-determined until the increment that meets the preset conditions is obtained or the number of iterations reaches the preset threshold.

[0127] The above calibration parameters also include the system deviation between the inertial navigation device and the point cloud device, and the position and posture of the inertial navigation device used for distance calculation are adjusted according to the system deviation.

[0128] The position and attitude of the inertial navigation device are calculated based on the GNSS positioning information and positioning base station information.

[0129] Since the principles of the problems solved by these devices and systems are similar to those of the aforementioned methods, the implementation of the devices and systems can refer to the implementation of the aforementioned methods, and the repeated parts will not be repeated.

[0130] An embodiment of the present invention further provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the device calibration method in the above-mentioned map surveying and mapping system is implemented.

[0131] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned device calibration method in the map surveying and mapping system when executing the stored program.

[0132] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0133] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0134] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0135] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0136] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A device calibration method in a map surveying and mapping system, characterized in that: include: Extracting linear elements from ground images taken by imaging devices; Extracting outer contour points of ground elements from the acquired point cloud data; Projecting the outer contour points into the ground image according to the external parameters of the imaging device and the position and posture of the inertial navigation device; According to the grayscale intensity gradient information of the outer contour point and the grayscale intensity gradient information of the linear element, determine the nearest linear element having the same direction as the grayscale intensity gradient of the outer contour point as the linear element matching the outer contour point, so as to obtain a matching relationship between the outer contour point and the linear element; Determine, according to the matching relationship and the inertial navigation device posture, an imaging device extrinsic calibration parameter that minimizes the distance from the outer contour point to the image plane where the matching linear element is located; The image plane is a plane formed by the linear elements matching the outer contour points and the optical center of the imaging device, and the imaging device extrinsic calibration parameters include the spatial transformation relationship between the imaging device and the inertial navigation device.

2. The method according to claim 1, characterized in that Extract linear elements from ground images taken by imaging devices, including: Extracting linear elements from a ground image captured by an imaging device by using a selected straight line extraction algorithm and obtaining the gray intensity gradient direction of the linear elements; or The road elements in the ground image taken by the imaging device are segmented, and the linear elements included in the ground image are fitted through the machine learning algorithm to obtain the grayscale intensity gradient information of the linear elements.

3. The method according to claim 1, characterized in that Extract the outer contour points of ground elements from the acquired point cloud data, including: Extracting point cloud data of a selected range from point cloud data acquired by a point cloud device according to the geographical area covered by the ground image; According to the connectivity of the reflectivity of the laser points in the point cloud data, the connected laser points are obtained to obtain the outer contour points of the ground elements and the corresponding reflectivity and grayscale intensity gradient information.

4. The method according to claim 1, characterized in that The projecting the outer contour points into the ground image according to the external parameters of the imaging device and the position and posture of the inertial navigation device includes: The product of the coordinates of the outer contour point and the external parameters of the imaging device, the posture of the inertial navigation device, and the internal parameters of the imaging device is calculated to obtain the projection coordinates of the outer contour point in the ground image.

5. The method according to claim 1, characterized in that The determining of the nearest linear element having the same direction as the grayscale intensity gradient of the outer contour point comprises: The distance from the outer contour point in the point cloud data to the image plane where the matching linear element is located is minimized by descending the grayscale intensity gradient, so as to obtain a linear element with the same grayscale intensity gradient direction as the outer contour point and the smallest distance.

6. The method according to claim 1, characterized in that Determining the imaging device extrinsic calibration parameters that minimize the distance from the outer contour point to the image plane according to the matching relationship and the inertial navigation device posture includes: Presetting an imaging device external parameter calibration parameter, wherein the imaging device external parameter calibration parameter includes a spatial conversion relationship between the imaging device and the inertial navigation device; According to the imaging device extrinsic calibration parameters, the inertial navigation device posture, the corresponding outer contour points and linear elements included in the matching relationship, construct a distance expression from the outer contour point to the plane formed by the matching linear element and the optical center of the imaging device; For each projection, an increment of the extrinsic calibration parameter of the imaging device is obtained according to the distance expression. If the increment does not meet the preset conditions, the extrinsic calibration parameter of the imaging device is updated according to the increment, and then the outer contour point is re-projected into the ground image to obtain the matching relationship between the outer contour point and the linear element, and the increment is re-determined until the increment that meets the preset conditions is obtained or the number of iterations reaches a preset threshold number.

7. The method according to claim 6, characterized in that The imaging device external parameter calibration parameters also include a system deviation between the inertial navigation device and the point cloud device, and the inertial navigation device posture used for distance calculation is adjusted according to the system deviation.

8. The method according to any one of claims 1 to 7, characterized in that: The position and attitude of the inertial navigation device are calculated based on the GNSS positioning information and positioning base station information.

9. A device calibration device in a map surveying and mapping system, characterized in that: include: A linear element extraction module, used for extracting linear elements from a ground image taken by an imaging device; A contour extraction module is used to extract the outer contour points of ground elements from the acquired point cloud data; A matching module, used for projecting the outer contour points into the ground image according to the external parameters of the imaging device and the position and posture of the inertial navigation device; According to the grayscale intensity gradient information of the outer contour point and the grayscale intensity gradient information of the linear element, determine the nearest linear element having the same direction as the grayscale intensity gradient of the outer contour point as the linear element matching the outer contour point, so as to obtain a matching relationship between the outer contour point and the linear element; A calibration module is used to determine the imaging device extrinsic calibration parameters that minimize the distance from the outer contour point to the image plane where the matching linear element is located according to the matching relationship and the inertial navigation device posture; the image plane is a plane formed by the linear element matching the outer contour point and the optical center of the imaging device, and the imaging device extrinsic calibration parameters include the spatial transformation relationship between the imaging device and the inertial navigation device.

10. A map surveying and mapping system, characterized in that: include: Imaging equipment, point cloud equipment, inertial navigation equipment and equipment calibration device in a map surveying and mapping system as claimed in claim 9; The equipment calibration device in the map surveying and mapping system determines the external calibration parameters of the imaging device based on the ground image taken by the imaging device, the point cloud data acquired by the point cloud device and the position and posture of the inertial navigation device.

Images