Camera extrinsic parameter calibration method, device and system in vehicle-mounted mobile measurement system

Abstract

The present invention discloses a camera extrinsic calibration method, device and system in a vehicle-mounted mobile measurement system. The method comprises: matching the same-name elements of at least two images in the point cloud data and the image data to obtain a same-name element combination, and matching the same-name element point pairs included in each image from the same-name element combination; for the same-name element point pairs, according to the set posture of the camera relative to the inertial navigation and the posture of the inertial navigation in the world coordinate system when shooting the image, the first element point is back-projected into the point cloud data to obtain the back-projection point, and the back-projection point is projected onto the image where the second element point is located to obtain the projection point, and the distance between the projection point and the second element point is determined; it is determined whether the average value of the distance is lower than the error threshold; if not, the set posture is changed according to the preset rules until the average value is lower than the error threshold, and the current set posture is used as the calibration result of the camera extrinsic parameter. It can realize the automatic calibration of the camera extrinsic parameters in the vehicle-mounted mobile measurement system, and the calibration accuracy is high.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic map data acquisition, and in particular to a camera extrinsic parameter calibration method, device and system in a vehicle-mounted mobile measurement system. Background Art

[0002] In the process of electronic map production, in addition to using point cloud data obtained by laser scanners to produce various map elements, point clouds are usually colored using images taken by cameras to improve the efficiency of electronic map production. The accuracy of point cloud coloring depends on the relative posture between the laser scanner and the camera, that is, the accuracy of the camera's external parameter calibration.

[0003] Currently, the main method for calibrating camera extrinsics is to manually select the corresponding points or boundaries of the same name between the point cloud and the image, and then perform manual calibration. This method has very high requirements on the quality of the point cloud and the calibration personnel. The calibration results are easily affected by noise, prone to matching errors, and time-consuming. Summary of the Invention

[0004] In view of the above problems, the present invention is proposed to provide a camera extrinsic parameter calibration method, device and system in a vehicle-mounted mobile measurement system that overcomes the above problems or at least partially solves the above problems.

[0005] In a first aspect, an embodiment of the present invention provides a camera extrinsic parameter calibration method in a vehicle-mounted mobile measurement system, comprising:

[0006] Matching the elements of the same name between the collected point cloud data and at least two images in the image data to obtain a combination of elements of the same name, and matching the element point pairs of the same name included in each of the two images from the combination of elements of the same name;

[0007] For pairs of element points with the same name, based on the set pose of the camera relative to the inertial navigation system and the pose of the inertial navigation system in the world coordinate system when the image was captured, back-project the first element point in the pair onto the point cloud data to obtain a back-projected point, project the back-projected point onto the image containing the second element point in the pair to obtain a projected point, and determine the distance between the projected point and the second element point;

[0008] Determining whether the average value of the distances is lower than an error threshold;

[0009] If not, the set posture is changed according to a preset rule until the average value of the re-determined distance is lower than the error threshold, and the current set posture is used as the calibration result of the camera extrinsic parameter.

[0010] In some optional embodiments, based on the set position of the camera relative to the inertial navigation system and the position of the inertial navigation system in the world coordinate system when capturing the image, back-projecting the first element point in the pair of element points with the same name into the point cloud data to obtain the back-projected point specifically includes:

[0011] Determine the position of the inertial navigation system in the world coordinate system when the image was taken based on the image's timestamp and inertial navigation trajectory data;

[0012] According to the set position of the camera relative to the inertial navigation system and the determined position of the inertial navigation system in the world coordinate system, the first element point in the pair of element points with the same name is back-projected into the point cloud data to obtain a back-projected point.

[0013] In some optional embodiments, determining the position and posture of the inertial navigation system in the world coordinate system when the image was captured based on the timestamp and inertial navigation trajectory data of the image specifically includes:

[0014] Interpolate the inertial navigation trajectory data according to the timestamp of the image to obtain the interpolation point;

[0015] The position and posture of the inertial navigation system included in the interpolation point in the world coordinate system is determined as the position and posture of the inertial navigation system in the world coordinate system when the image is captured.

[0016] In some optional embodiments, based on the set position of the camera relative to the inertial navigation system and the position of the inertial navigation system in the world coordinate system when capturing the image, back-projecting the first element point in the pair of element points with the same name into the point cloud data to obtain the back-projected point specifically includes:

[0017] According to the camera intrinsic parameters and distortion parameters, the pixel coordinates of the first element point in the pair of element points with the same name are converted into camera coordinates, the camera coordinates of the first element point are converted into inertial navigation coordinates according to the set position of the camera relative to the inertial navigation, and the inertial navigation coordinates of the first element point are converted into world coordinates according to the position of the inertial navigation in the world coordinate system when the image where the first element point is located is captured, to obtain a world coordinate point set, and the intersection of the trajectory of the world coordinate point set and the plane where the point cloud data is located is used as the back projection point of the first element point; correspondingly, the back projection point is projected onto the image where the second element point in the pair of element points with the same name is located to obtain a projection point, specifically including:

[0018] The world coordinates of the back-projection point are converted into inertial navigation coordinates according to the posture of the inertial navigation system in the world coordinate system when the image of the second element point in the pair of element points with the same name is captured. The inertial navigation coordinates of the back-projection point are converted into camera coordinates according to the set posture of the camera relative to the inertial navigation system. The camera coordinates of the back-projection point are converted into pixel coordinates according to the camera intrinsic parameters and distortion parameters. The point in the image where the second element point is located, determined according to the pixel coordinates of the back-projection point, is used as the projection point of the back-projection point.

[0019] In some optional embodiments, determining whether the average value of the distance is lower than an error threshold specifically includes:

[0020] It is determined whether an average value of distances greater than a first distance threshold among the distances is lower than an error threshold.

[0021] In some optional embodiments, determining whether the average value of the distance is lower than an error threshold further includes:

[0022] Determine a first average value of distances greater than a first distance threshold among corresponding distances of pairs of feature points of the same name in the feature combination, and determine whether an average value of the first average values ​​greater than a second distance threshold is lower than an error threshold; or,

[0023] A second average value of distances greater than a first distance threshold among corresponding distances of pairs of feature points with the same name in the two images is determined, and it is determined whether an average value of the second average values ​​greater than a third distance threshold is lower than an error threshold.

[0024] In some optional embodiments, determining the distance between the projection point and the second element point includes:

[0025] The Mahalanobis distance between the projection point and the second element point is determined as the distance between the two.

[0026] In a second aspect, an embodiment of the present invention provides a camera extrinsic parameter calibration device in a vehicle-mounted mobile measurement system, comprising:

[0027] A matching module is used to match the elements of the same name in the collected point cloud data and at least two images in the image data to obtain a combination of elements of the same name, and to match the element point pairs of the same name included in each of the two images from the combination of elements of the same name;

[0028] a determination module for, for the element point pairs with the same name matched by the matching module, back-projecting a first element point in the element point pair with the same name onto point cloud data to obtain a back-projection point based on a set position of the camera relative to the inertial navigation system and a position of the inertial navigation system in a world coordinate system when the image was captured, projecting the back-projection point onto the image containing the second element point in the element point pair with the same name to obtain a projection point, and determining a distance between the projection point and the second element point;

[0029] a judging module, configured to judge whether the average value of the distances determined by the determining module is lower than an error threshold;

[0030] When the judgment module determines that the result is no, the determination module is used to change the set posture according to a preset rule and re-determine the distance between the projection point and the second element point;

[0031] The calibration module is used to use the current set posture as the calibration result of the camera external parameter when the judgment module judges that the camera is correct.

[0032] In a third aspect, an embodiment of the present invention provides a vehicle-mounted mobile measurement system, comprising an inertial navigation system, a laser scanner, a camera, and a controller, wherein the controller is provided with the above-mentioned camera extrinsic parameter calibration device;

[0033] The camera extrinsic parameter calibration device is used to calibrate the extrinsic parameters of the camera based on the inertial navigation trajectory data collected by the inertial navigation, the point cloud data collected by the laser scanner and the image data collected by the camera.

[0034] In a fourth aspect, an embodiment of the present invention provides a server comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the camera extrinsic parameter calibration method in the above-mentioned vehicle-mounted mobile measurement system.

[0035] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implements the camera extrinsic parameter calibration method in the above-mentioned vehicle-mounted mobile measurement system.

[0036] The beneficial effects of the above technical solutions provided by the embodiments of the present invention include at least:

[0037] 1. The camera extrinsic parameter calibration method in a vehicle-mounted mobile measurement system provided by an embodiment of the present invention matches the same-name elements in at least two images of the point cloud data and image data to obtain a same-name element combination, and matches the same-name element point pairs included in each image from the same-name element combination; for the same-name element point pairs, based on the set position of the camera relative to the inertial navigation system and the position of the inertial navigation system in the world coordinate system when the image was captured, the first element point is back-projected into the point cloud data to obtain a back-projection point, the back-projection point is projected onto the image where the second element point is located to obtain a projection point, and the distance between the projection point and the second element point is determined; it is determined whether the average value of the distance is lower than an error threshold; if not, the set position is changed according to a preset rule until the average value is lower than the error threshold, and the current set position is used as the calibration result of the camera extrinsic parameter. The entire calibration process is completed automatically without human intervention, achieving automatic calibration, saving labor costs, and improving calibration efficiency.

[0038] 2. For the feature point pairs with the same name, the first feature point is back-projected into the point cloud data to obtain the back-projection point, and the back-projection point is projected onto the image where the second feature point is located to obtain the projection point. The distance between the projection point and the second feature point is determined. The feature point in the point cloud data is directly projected onto the image where the second feature point is located, and the feature points in the point cloud data are corrected, thereby avoiding the calibration error caused by the accuracy of the point cloud data itself. Therefore, the quality requirements for the point cloud data can be reduced, the adaptability range is expanded, and the calibration accuracy is high.

[0039] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purposes and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0040] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0042] Figure 1 This is a flow chart of a camera extrinsic parameter calibration method in a vehicle-mounted mobile measurement system according to the first embodiment of the present invention;

[0043] Figure 2 Schematic diagram of point cloud data, images, and inertial navigation trajectories in Example 1 of the present invention;

[0044] Figure 3 This is a schematic diagram of a combination of elements with the same name in the first embodiment of the present invention;

[0045] Figure 4 A specific implementation flowchart for back-projecting the first feature point into point cloud data;

[0046] Figure 5 A flowchart for the specific implementation of projecting the back-projected point onto the image where the second feature point is located;

[0047] Figure 6A This is an example diagram of a method for determining an average value in Example 1 of the present invention;

[0048] Figure 6B This is an example diagram of another method for determining an average value in the first embodiment of the present invention;

[0049] Figure 7 This is a specific implementation flow chart of camera extrinsic parameter calibration in the vehicle-mounted mobile measurement system in the second embodiment of the present invention; Figure 8 Schematic diagram of the structure of a camera extrinsic parameter calibration device in a vehicle-mounted mobile measurement system according to an embodiment of the present invention;

[0050] Figure 9 Schematic diagram of the structure of the vehicle-mounted mobile measurement system in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0052] To address the existing issues in the art of manually calibrating camera extrinsic parameters, which impose high requirements on point cloud quality and the number of calibrators, as well as large calibration errors and a long time consumption, the present invention provides a method, device, and system for calibrating camera extrinsic parameters in a vehicle-mounted mobile measurement system. These methods can automatically calibrate camera extrinsic parameters in a vehicle-mounted mobile measurement system with high calibration efficiency and accuracy.

[0053] A vehicle-mounted mobile measurement system refers to a comprehensive measurement system composed of a positioning and orientation system (POS) (inertial navigation), a control system, and measurement sensors (laser scanners, cameras, etc.) integrated on a vehicle-mounted platform. The vehicle-mounted mobile measurement system used in the technical solution of this application includes an inertial navigation system, a control system, a laser scanner, and a camera.

[0054] First, a vehicle-mounted mobile measurement system is used for data acquisition. The data acquisition process is carried out while the vehicle equipped with the vehicle-mounted mobile measurement system is moving. The 3D laser scanner is used to obtain point cloud data; the inertial navigation is used to obtain inertial navigation pose information (the position and pose information of the inertial navigation in the world coordinate system) and obtain inertial navigation trajectory data, including the trajectory of the inertial navigation position information and the time relationship, and the pose trajectory of the inertial navigation pose information and the time relationship in the world coordinate system; and the camera is used to obtain images within the shooting field of view.

[0055] Example 1

[0056] The first embodiment of the present invention provides a camera extrinsic parameter calibration method in a vehicle-mounted mobile measurement system, the process of which is as follows: Figure 1 As shown, the following steps are included:

[0057] Step S11: performing matching of elements with the same name on the collected point cloud data and at least two images in the image data to obtain a combination of elements with the same name, and matching pairs of elements with the same name included in each of the two images from the combination of elements with the same name.

[0058] The vehicle-mounted mobile measurement system acquires the following data during the vehicle's driving: image data acquired by the camera, point cloud data acquired by the laser scanner, and inertial navigation trajectory data acquired by the inertial navigation system. The inertial navigation trajectory data includes the trajectory of the inertial navigation system's position information over time and the trajectory of the inertial navigation system's pose information over time in the world coordinate system.

[0059] Specifically, the obtained point cloud data and inertial navigation trajectory data are solved, or the original point cloud data and inertial navigation trajectory data are first obtained, and then the corresponding solving method is used to complete the data solving.

[0060] The original point cloud data is in the laser scanner's own coordinate system (SOCS). It is necessary to use relevant solution methods and combine it with the inertial navigation pose information (the inertial navigation pose in the world coordinate system can be used as the laser scanner's pose in the world coordinate system) to solve the panoramic point cloud data in the world coordinate system. The position information of the track points in the collected original inertial navigation trajectory data is longitude and latitude, which needs to be solved into the world coordinate system.

[0061] Optionally, the calculated point cloud data and inertial navigation trajectory data can be converted to a local world coordinate system. This can reduce the size of the coordinate values ​​in the data, reduce the amount of calculation, and increase the readability of the data.

[0062] Reference Figure 2 As shown in the figure, the left side is the solved point cloud data; the middle side is the acquired image; the right side is the solved inertial navigation trajectory data, including the inertial navigation trajectory and posture trajectory.

[0063] The collected point cloud data and at least two images in the image data are matched with the elements of the same name to obtain a combination of elements of the same name. Specifically, it can include first determining the elements in the point cloud data, such as a road sign, a lane line or a ground sign; for each element in the point cloud data, using a deep learning method, matching the elements in at least two images from the image data to obtain a combination of elements of the same name, and referring to Figure 3 As shown, the point cloud data is the point cloud data corresponding to a certain road sign. The road sign in the three images and the road sign corresponding to the point cloud data are the same road sign, that is, they are elements with the same name.

[0064] Match pairs of same-name feature points between any two images in the same-name feature combination. This means performing same-name point registration on the same-name features in any two images within the same-name feature combination. For example, if images A, B, C, D, and E, taken sequentially within a 50-meter radius, all contain the same street sign, then the street sign in image A can be registered with the street sign in image B, the street sign in image B with the street sign in image C, the street sign in image C with the street sign in image D, and the street sign in image D with the street sign in image E. This means performing same-name point registration on the same-name features in two images taken at adjacent times to obtain same-name feature point pairs.

[0065] The above-mentioned pairs of feature points with the same name are the image points of the same point on the object in different images.

[0066] Step S12: For the element point pairs with the same name, based on the set position of the camera relative to the inertial navigation and the position of the inertial navigation in the world coordinate system when the image is captured, the first element point in the element point pair with the same name is back-projected into the point cloud data to obtain the back-projection point, and the back-projection point is projected onto the image where the second element point in the element point pair with the same name is located to obtain the projection point, and the distance between the projection point and the second element point is determined.

[0067] The initial value of the set position of the camera relative to the inertial navigation system can be the known position of the camera at the initial moment of the camera extrinsic calibration. That is, for a camera that has never undergone extrinsic calibration, its set position can be the initial position of the camera relative to the inertial navigation system when the vehicle-mounted mobile measurement system is installed. For a camera that has undergone extrinsic calibration, its set position can be the most recent camera extrinsic calibration result.

[0068] Specifically, the above-mentioned camera external parameters are the position parameters of the camera relative to the inertial navigation system: X pano-pos 、Y pano-pos and Z pano-pos , and the camera's attitude angle relative to the inertial navigation: Yaw pano-pos Pitch pano-pos and roll angle ROll pano-pos .

[0069] In one embodiment, referring to Figure 4 As shown, back-projecting the first feature point in the pair of feature points with the same name onto the point cloud data to obtain the back-projected point may include the following steps:

[0070] Step S41: converting the pixel coordinates of the first feature point in the pair of feature points with the same name into camera coordinates according to the camera intrinsic parameters and distortion parameters.

[0071] Step S42: converting the camera coordinates of the first feature point into inertial navigation coordinates according to the set position of the camera relative to the inertial navigation.

[0072] Step S43: according to the position and posture of the inertial navigation system in the world coordinate system when the image where the first feature point is located is captured, the inertial navigation coordinates of the first feature point are converted into world coordinates to obtain a world coordinate point set.

[0073] Step S44: The intersection of the trajectory of the world coordinate point set and the plane of the point cloud data is used as the back-projection point of the first feature point.

[0074] Projecting the back-projected point onto the image where the second element point in the pair of element points with the same name is located to obtain the projection point is the inverse process of back-projecting the first element point onto the point cloud data to obtain the back-projected point. Figure 5 As shown, the following steps may be included:

[0075] Step S51: converting the world coordinates of the back-projected point into inertial navigation coordinates according to the posture of the inertial navigation in the world coordinate system when capturing the image of the second element point in the pair of element points with the same name.

[0076] Step S52: converting the inertial navigation coordinates of the back-projected point into camera coordinates according to the set position of the camera relative to the inertial navigation.

[0077] Specifically, it can also be directly based on the following formula (1) through the transformation matrix from the world coordinate system to the inertial navigation coordinate system: And the transformation matrix from the inertial navigation coordinate system to the camera coordinate system Get the transformation matrix from the world coordinate system to the camera coordinate system

[0078]

[0079] Then according to the transformation matrix from the world coordinate system to the camera coordinate system Posture transformation in and position transformation Use formula (2) to convert the world coordinates P of the feature point W Convert to camera coordinates P C :

[0080]

[0081] Step S53: Convert the camera coordinates of the back-projected point into pixel coordinates according to the camera intrinsic parameters and distortion parameters.

[0082] Specifically, you can first transpose the horizontal and vertical coordinates in the camera coordinates:

[0083]

[0084] Then according to the horizontal coordinate of the transposed camera coordinate and the vertical axis The preset camera intrinsic parameters and distortion parameters are used to convert the camera coordinates of the back-projected points into pixel coordinates using formulas (4) to (7):

[0085]

[0086]

[0087]

[0088]

[0089] In formula (4)-formula (7), u and v are pixel coordinates, f x 、f y 、c x and cy is the intrinsic parameter of the camera, k1, k2, k3, p1 and p2 are the distortion parameters of the camera.

[0090] Step S54: using the point in the image where the second feature point is located, which is determined according to the pixel coordinates of the back-projection point, as the projection point of the back-projection point.

[0091] In one embodiment, the above-mentioned determination of the distance between the projection point and the second element point may be to determine the Mahalanobis distance between the projection point and the second element point as the difference between the two.

[0092] Step S13: Determine whether the average value of the distance is lower than the error threshold.

[0093] Specifically, the average value of the distance may be determined by the following three methods.

[0094] Method 1: Directly determine the average value of the distance.

[0095] Summarizes the distances determined for pairs of features with the same name and determines the average of those distances that are greater than a first distance threshold.

[0096] If the distance between the projection point and the second feature point is greater than the first distance threshold, it can be considered that there is a problem with the first feature point and / or the first feature point data in the corresponding pair of feature points with the same name, so it is deleted and the average value of the filtered distance is determined, so that the obtained average value is closer to reality and a large amount of interference information is eliminated.

[0097] Method 2: Determine the average distance using the combination of features with the same name as the unit.

[0098] For each feature combination with the same name, determine a first average value of distances greater than a first distance threshold among corresponding distances of feature point pairs with the same name in the feature combination with the same name; and determine an average value of the first average values ​​greater than a second distance threshold.

[0099] The average value of the distance is determined using the combination of elements with the same name as the unit. If the first average value of the distance corresponding to the combination of elements with the same name is greater than the second distance threshold, it means that the corresponding elements may be blocked or interfered with during the data collection process, resulting in errors in the collected data. Therefore, they are deleted and the average value of the filtered distance is determined, so that the obtained average value is closer to reality and eliminates interference information.

[0100] For example, refer to Figure 6AAs shown, taking a set of homonymous feature combinations as an example, images A and D have homonymous features. After homonymous point registration is completed for homonymous feature pairs AB between images A and B, BC between images B and C, and CD between images C and D, distances AB-1, AB-2, AB-3, ... are obtained for the homonymous feature pairs in homonymous feature pair AB between images A and B. Distances BC-1, BC-2, BC-3, ... are obtained for the homonymous feature pair BC between images B and C. Distances CD-1, CD-2, CD-3, ... are obtained for the homonymous feature pair CD between images C and D. A first average value of all distances obtained for the homonymous feature combinations that are greater than a first distance threshold is determined, and the average value of the first average values ​​that are greater than a second distance threshold is determined.

[0101] Method 3: Determine the average value of the distance using the image as the unit.

[0102] A second average value of distances greater than a first distance threshold among corresponding distances of pairs of feature points with the same name in the two images is determined, and an average value of the second average values ​​greater than a third distance threshold is determined.

[0103] The average value of the distance is determined based on the image as a unit. If the second average value of the distance determined based on all pairs of feature points with the same name in the two images is greater than the third distance threshold, it means that there may be problems with the data of one or both images. Therefore, it is deleted and the average value of the filtered distance is determined. This makes the obtained average value closer to reality and eliminates interference information.

[0104] Reference Figure 6BAs shown, the element pairs with the same name obtained by matching in image A and image B include element pair AB1, element pair AB2, element pair AB3, element pair AB4, element pair AB5 and element pair AB6. For the aligned same-name feature point pairs in the same-name feature pair AB1, the distances AB1-1, AB1-2, AB1-3... are obtained; for the aligned same-name feature point pairs in the same-name feature pair AB2, the distances AB2-1, AB2-2, AB2-3... are obtained; for the aligned same-name feature point pairs in the same-name feature pair AB3, the distances AB3-1, AB3-2, AB3-3... are obtained; for the aligned same-name feature point pairs in the same-name feature pair AB4, the distances AB4-1, AB4-2, AB4-3... are obtained; for the aligned same-name feature point pairs in the same-name feature pair AB5, the distances AB5-1, AB5-2, AB5-3... are obtained; for the aligned same-name feature point pairs in the same-name feature pair AB6, the distances AB6-1, AB6-2, AB6-3... are obtained. Determine a second average value of distances greater than the first distance threshold among the distances obtained for the pairs of feature points with the same name in the above-mentioned image A and image B; and determine an average value of the second average values ​​obtained that are greater than the third distance threshold.

[0105] Optionally, other methods for determining the average value of the distance may be selected according to actual needs, and this embodiment does not limit the specific method for determining the average value of the distance.

[0106] When the judgment result of step S13 is yes, step S15 is executed; when the judgment result of step S13 is no, step S14 is executed.

[0107] Step S14: changing the set position of the camera relative to the inertial navigation according to a preset rule.

[0108] When it is determined that the average value of the distances corresponding to the pairs of feature points with the same name is not less than the error threshold, the set position of the camera relative to the inertial navigation is changed according to a preset rule, and step S12 is re-executed to determine a new distance average value.

[0109] Step S15: Use the current set pose as the calibration result of the camera extrinsic parameters.

[0110] When the average value of the distances corresponding to the pairs of feature points with the same name is judged to be lower than the error threshold, the set position of the current camera relative to the inertial navigation is used as the calibration result of the camera extrinsic parameter.

[0111] The entire camera extrinsic parameter calibration process in the above embodiment 1 is completed automatically without manual intervention, thus achieving automatic calibration, saving labor costs, and improving calibration efficiency.

[0112] For feature point pairs with the same name, the first feature point is back-projected into the point cloud data to obtain the back-projection point, and the back-projection point is projected onto the image where the second feature point is located to obtain the projection point. The distance between the projection point and the second feature point is determined. The feature point in the point cloud data is directly projected onto the image where the second feature point is located, and the feature points in the point cloud data are corrected, avoiding the calibration error caused by the accuracy of the point cloud data itself. Therefore, the quality requirements for the point cloud data can be reduced, the adaptability range is expanded, and the calibration accuracy is high.

[0113] Example 2

[0114] The second embodiment of the present invention provides a specific implementation method for camera extrinsic parameter calibration in a vehicle-mounted mobile measurement system, and the process is as follows: Figure 7 As shown, the following steps are included:

[0115] Step S71: Acquire image data, point cloud data, and inertial navigation trajectory data collected by the vehicle-mounted mobile measurement system during the driving process of the vehicle.

[0116] Specifically, the acquired point cloud data and inertial navigation trajectory data are the data of the local world coordinate system after the solution. This can reduce the size of the coordinate values ​​in the data, reduce the amount of calculation, and increase the readability of the data.

[0117] Step S72: Identify the elements in the point cloud data. For each element, use a deep learning method to determine the elements with the same name in at least two images in the image data, obtain a combination of elements with the same name, and match the element point pairs with the same name included in each image from the combination of elements with the same name.

[0118] For each pair of feature points with the same name, execute the following steps S73-S75 respectively.

[0119] Step S73: interpolating the inertial navigation trajectory data according to the timestamp of the image where the first feature point in the pair of feature points with the same name is located to obtain interpolation points, and determining the position of the inertial navigation in the world coordinate system included in the interpolation points as the position of the inertial navigation in the world coordinate system when the image was captured.

[0120] For a clearer description, the image where the first feature point is located will be described as the first image, and the position and posture of the inertial navigation system in the world coordinate system when the first image is taken will be described as the first position and posture of the inertial navigation system in the world coordinate system; the image where the second feature point in the pair of feature points with the same name is located will be described as the second image, and the position and posture of the inertial navigation system in the world coordinate system when the second image is taken will be described as the second position and posture of the inertial navigation system in the world coordinate system.

[0121] Step S74: According to the set position of the camera relative to the inertial navigation system and the first position of the inertial navigation system in the world coordinate system, the first feature point is back-projected into the point cloud data to obtain a back-projection point.

[0122] Step S75: According to the set position of the camera relative to the inertial navigation system and the second position of the inertial navigation system in the world coordinate system, the back-projected point is projected onto the second image to obtain the projection point, and the Mahalanobis distance between the projection point and the second feature point is determined as the distance between the two.

[0123] Taking the same-name feature combination consisting of the same road sign in image A, image B, and point cloud data F as an example, the same-name feature combination determines a pair of same-name feature points A1 B1 in image A and image B. Point A1 in image A is back-projected onto point cloud data F to obtain back-projected point A2; then, back-projected point A2 is projected onto image B to obtain projected point A3. The Mahalanobis distance between the obtained projected point A3 and point B1 is determined to be the distance between the two.

[0124] The Mahalanobis distance is independent of the measurement scale and is therefore not affected by the dimension. It can take into account the relationship between various characteristics and eliminate the interference of correlation between variables. Therefore, the Mahalanobis distance can better reflect the distance between two points.

[0125] Step S76: Determine whether the average value of the distances greater than the first distance threshold is lower than the error threshold.

[0126] When the judgment result of step S76 is yes, execute step S78; when the judgment result of step S76 is no, execute step S77.

[0127] Step S77: changing the set position of the camera relative to the inertial navigation according to a preset rule.

[0128] When it is determined that the average value of the distances greater than the first distance threshold is not lower than the error threshold, the set position of the camera relative to the inertial navigation is changed according to a preset rule, and steps S74 and S75 are re-executed.

[0129] Step S78: The current camera's set position relative to the inertial navigation is used as the calibration result of the camera's extrinsic parameters.

[0130] When the average value of the distances greater than the first distance threshold in the judgment distance is lower than the error threshold, the set position of the current camera relative to the inertial navigation is used as the calibration result of the camera extrinsic parameter.

[0131] Based on the inventive concept of the present invention, the embodiment of the present invention further provides a camera extrinsic parameter calibration device in a vehicle-mounted mobile measurement system, whose structure is as follows: Figure 8 As shown, including:

[0132] The matching module 81 is used to match the elements with the same name in the collected point cloud data and at least two images in the image data to obtain a combination of elements with the same name, and match the element point pairs with the same name included in each of the two images from the combination of elements with the same name;

[0133] a determination module 82 for back-projecting the first element point in the pair of element points with the same name matched by the matching module 81 onto the point cloud data to obtain a back-projected point, projecting the back-projected point onto the image containing the second element point in the pair of element points with the same name to obtain a projected point, and determining the distance between the projected point and the second element point;

[0134] A judging module 83 is configured to judge whether the average value of the distance determined by the determining module 82 is lower than an error threshold;

[0135] When the judgment module 83 judges as no, the determination module 82 is used to change the set posture according to a preset rule and re-determine the distance between the projection point and the second element point;

[0136] The calibration module 84 is configured to use the current set posture as the calibration result of the camera external parameters when the judgment module 83 judges that the answer is yes.

[0137] In some embodiments, the determination module 82 back-projects the first element point of the same-name element point pair onto the point cloud data to obtain a back-projected point based on the set position of the camera relative to the inertial navigation system and the position of the inertial navigation system in the world coordinate system when the image is captured, specifically for:

[0138] The method includes: determining the position of the inertial navigation system in the world coordinate system when the image is captured based on the timestamp and inertial navigation trajectory data of the image; and back-projecting the first element point of the same-name element point pair into the point cloud data to obtain a back-projection point based on the set position of the camera relative to the inertial navigation system and the determined position of the inertial navigation system in the world coordinate system.

[0139] In some embodiments, the determination module 82 determines the position of the inertial navigation system in the world coordinate system when the image was captured based on the timestamp of the image and the inertial navigation trajectory data, specifically for:

[0140] Interpolation points are obtained by interpolating the inertial navigation trajectory data according to the timestamp of the image; and the position and posture of the inertial navigation in the world coordinate system included in the interpolation points are determined as the position and posture of the inertial navigation in the world coordinate system when the image is captured.

[0141] In some embodiments, the determination module 82 back-projects the first element point of the same-name element point pair onto the point cloud data to obtain a back-projected point based on the set position of the camera relative to the inertial navigation system and the position of the inertial navigation system in the world coordinate system when the image is captured, specifically for:

[0142] According to the camera intrinsic parameters and distortion parameters, the pixel coordinates of the first element point in the pair of element points with the same name are converted into camera coordinates, the camera coordinates of the first element point are converted into inertial navigation coordinates according to the set position of the camera relative to the inertial navigation, and the inertial navigation coordinates of the first element point are converted into world coordinates according to the position of the inertial navigation in the world coordinate system when the image where the first element point is located is captured, to obtain a world coordinate point set, and the intersection of the trajectory of the world coordinate point set and the plane where the point cloud data is located is used as the back projection point of the first element point; correspondingly, the back projection point is projected onto the image where the second element point in the pair of element points with the same name is located to obtain a projection point, which is specifically used for:

[0143] The world coordinates of the back-projection point are converted into inertial navigation coordinates according to the posture of the inertial navigation system in the world coordinate system when the image of the second element point in the pair of element points with the same name is captured. The inertial navigation coordinates of the back-projection point are converted into camera coordinates according to the set posture of the camera relative to the inertial navigation system. The camera coordinates of the back-projection point are converted into pixel coordinates according to the camera intrinsic parameters and distortion parameters. The point in the image where the second element point is located, determined according to the pixel coordinates of the back-projection point, is used as the projection point of the back-projection point.

[0144] In some embodiments, the determination module 83 is specifically configured to:

[0145] It is determined whether an average value of distances greater than a first distance threshold among the distances is lower than an error threshold.

[0146] In some embodiments, the determination module 83 is further configured to:

[0147] Determine a first average value of distances greater than a first distance threshold among corresponding distances of pairs of feature points of the same name in the combination of features of the same name, and determine whether an average value of the first average values ​​greater than a second distance threshold is lower than an error threshold; or determine a second average value of distances greater than the first distance threshold among corresponding distances of pairs of feature points of the same name in the two images, and determine whether an average value of the second average values ​​greater than a third distance threshold is lower than an error threshold.

[0148] In some embodiments, the determination module 83 is specifically configured to:

[0149] The Mahalanobis distance between the projection point and the second element point is determined as the distance between the two.

[0150] Based on the inventive concept of the present invention, the embodiment of the present invention further provides a mobile measurement system, the structure of which is as follows: Figure 9 As shown, it includes an inertial navigation system 91, a laser scanner 92, a camera 93 and a controller 94. The controller 94 is provided with the above-mentioned camera extrinsic parameter calibration device;

[0151] The camera extrinsic parameter calibration device is used to calibrate the extrinsic parameters of the camera 93 based on the inertial navigation trajectory data collected by the inertial navigation 91, the point cloud data collected by the laser scanner 92 and the image data collected by the camera 93.

[0152] Regarding the apparatus and system in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiment of the method and will not be elaborated here.

[0153] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium having computer instructions stored thereon. When the instructions are executed by a processor, the camera extrinsic parameter calibration in the above-mentioned vehicle-mounted mobile measurement system is implemented.

[0154] Based on the same inventive concept, an embodiment of the present invention also provides a server, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the camera extrinsic parameter calibration in the above-mentioned vehicle-mounted mobile measurement system is implemented.

[0155] Unless otherwise specifically stated, terms such as process, calculate, compute, determine, display, and the like may refer to the actions and / or processes of one or more processing or computing systems, or similar devices, that manipulate and convert data represented as physical (e.g., electronic) quantities within registers or memories of a processing system into other data similarly represented as physical quantities within the memories, registers, or other such information storage, transmission, or display devices of the processing system. Information and signals may be represented using any of a variety of different techniques and methods. For example, data, instructions, commands, information, signals, bits, symbols, and chips referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, light fields or particles, or any combination thereof.

[0156] It should be understood that the specific order or hierarchy of steps in the disclosed processes is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of the present disclosure. The accompanying method claims present elements of the various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.

[0157] In the foregoing detailed description, various features are grouped together in a single embodiment to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that embodiments of the claimed subject matter require more features than are expressly recited in each claim. On the contrary, as reflected in the appended claims, the invention comprises less than all the features of any individual disclosed embodiment. The appended claims are therefore hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.

[0158] Those skilled in the art will also appreciate that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the embodiments herein may be implemented as electronic hardware, computer software, or a combination thereof. In order to clearly illustrate the interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps described above are generally described around their functions. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. A skilled person may implement the described functions in an adaptable manner for each specific application, but such implementation decisions should not be interpreted as departing from the scope of protection of this disclosure.

[0159] The steps of the methods or algorithms described in conjunction with the embodiments herein may be directly embodied as hardware, software modules executed by a processor, or a combination thereof. The software module may be located in a RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, register, hard disk, removable disk, CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is connected to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium may also be an integral part of the processor. The processor and storage medium may be located in an ASIC. The ASIC may be located in a user terminal. Of course, the processor and storage medium may also be present in a user terminal as discrete components.

[0160] For software implementation, the techniques described in this application can be implemented using modules (e.g., procedures, functions, etc.) that perform the functions described in this application. These software codes can be stored in a memory unit and executed by a processor. The memory unit can be implemented within the processor or external to the processor. In the latter case, it is communicatively coupled to the processor via various means, which are well known in the art.

[0161] The foregoing description includes examples of one or more embodiments. Of course, it is not possible to describe all possible combinations of components or methods for the purposes of describing the above embodiments, but one of ordinary skill in the art will recognize that the various embodiments may be further combined and arranged. Therefore, the embodiments described herein are intended to encompass all such changes, modifications and variations that fall within the scope of the appended claims. Furthermore, to the extent the term "comprising" is used in the specification or claims, the term is intended to be encompassed in a manner similar to the term "including," as explained in terms of "including," used as a transitional word in the claims. Furthermore, any use of the term "or" in the specification of the claims is intended to mean a "non-exclusive or."

Claims

1. Camera extrinsic calibration method in vehicle-mounted mobile measurement system, where: include: Matching the elements with the same name in the collected point cloud data and at least two images in the image data to obtain a combination of elements with the same name, and matching the element point pairs with the same name included in each of the two images from the combination of elements with the same name; For the element point pairs with the same name, according to the set position of the camera relative to the inertial navigation and the position of the inertial navigation in the world coordinate system when taking the image, the first element point in the element point pair with the same name is back-projected into the point cloud data to obtain a back-projected point, the back-projected point is projected onto the image where the second element point in the element point pair with the same name is located to obtain a projection point, and the distance between the projection point and the second element point is determined; Determining whether the average value of the distance is lower than an error threshold; If not, the set posture is changed according to a preset rule until the average value of the re-determined distance is lower than the error threshold, and the current set posture is used as the calibration result of the camera extrinsic parameter.

2. The method of claim 1, wherein: According to the set position of the camera relative to the inertial navigation system and the position of the inertial navigation system in the world coordinate system when the image is captured, the first element point in the element point pair with the same name is back-projected onto the point cloud data to obtain the back-projection point, specifically including: According to the timestamp and inertial navigation trajectory data of the image, determine the position and posture of the inertial navigation in the world coordinate system when the image was taken; According to the set position and posture of the camera relative to the inertial navigation system and the determined position and posture of the inertial navigation system in the world coordinate system, the first element point in the pair of element points with the same name is back-projected into the point cloud data to obtain a back-projection point.

3. The method of claim 2, wherein: Determining the position and posture of the inertial navigation in the world coordinate system when the image is captured based on the timestamp and inertial navigation trajectory data of the image specifically includes: Interpolate the inertial navigation trajectory data according to the timestamp of the image to obtain the interpolation point; The position and posture of the inertial navigation system included in the interpolation point in the world coordinate system is determined as the position and posture of the inertial navigation system in the world coordinate system when the image is captured.

4. The method of claim 1, wherein: According to the set position of the camera relative to the inertial navigation system and the position of the inertial navigation system in the world coordinate system when the image is captured, the first element point in the element point pair with the same name is back-projected onto the point cloud data to obtain the back-projection point, specifically including: According to the camera intrinsic parameters and distortion parameters, the pixel coordinates of the first element point in the same-name element point pair are converted into camera coordinates, the camera coordinates of the first element point are converted into inertial navigation coordinates according to the set position of the camera relative to the inertial navigation, and the inertial navigation coordinates of the first element point are converted into world coordinates according to the position of the inertial navigation in the world coordinate system when the image where the first element point is located is taken, to obtain a world coordinate point set, and the intersection of the trajectory where the world coordinate point set is located and the plane where the point cloud data is located is used as the back-projection point of the first element point; correspondingly, the back-projection point is projected onto the image where the second element point in the same-name element point pair is located to obtain a projection point, specifically including: According to the posture of the inertial navigation in the world coordinate system when the image where the second element point in the pair of element points with the same name is located is taken, the world coordinates of the back-projection point are converted into inertial navigation coordinates, the inertial navigation coordinates of the back-projection point are converted into camera coordinates according to the set posture of the camera relative to the inertial navigation, the camera coordinates of the back-projection point are converted into pixel coordinates according to the camera intrinsic parameters and distortion parameters, and the point in the image where the second element point is located, determined according to the pixel coordinates of the back-projection point, is used as the projection point of the back-projection point.

5. The method of claim 1, wherein: The determining whether the average value of the distance is lower than the error threshold specifically includes: It is determined whether an average value of the distances greater than the first distance threshold among the distances is lower than an error threshold.

6. The method of claim 5, wherein: The determining whether the average value of the distance is lower than an error threshold further includes: Determine a first average value of distances greater than a first distance threshold among corresponding distances of pairs of feature points of the same name in the feature combination of the same name, and determine whether an average value of the first average values ​​greater than a second distance threshold is lower than an error threshold; or, A second average value of the distances greater than the first distance threshold among the corresponding distances of the feature point pairs with the same name in the two images is determined, and it is determined whether an average value of the second average values ​​greater than the third distance threshold is lower than an error threshold.

7. The method according to any one of claims 1 to 6, wherein: Determining the distance between the projection point and the second element point includes: The Mahalanobis distance between the projection point and the second element point is determined as the distance between the two.

8. A camera extrinsic calibration device in a vehicle-mounted mobile measurement system, wherein: include: A matching module is used to match the elements of the same name between the collected point cloud data and at least two images in the image data to obtain a combination of elements of the same name, and to match the element point pairs of the same name included in each pair of images from the combination of elements of the same name; a determination module, for back-projecting the first element point in the same-name element point pair matched by the matching module to obtain a back-projection point according to the set position of the camera relative to the inertial navigation and the position of the inertial navigation in the world coordinate system when the image is captured, and projecting the back-projection point to the image where the second element point in the same-name element point pair is located to obtain a projection point, and determining the distance between the projection point and the second element point; A judging module, used to judge whether the average value of the distance determined by the determining module is lower than an error threshold; When the judgment module judges as no, the determination module is used to change the set posture according to a preset rule and re-determine the distance between the projection point and the second element point; The calibration module is used to use the current set posture as the calibration result of the camera external parameter when the judgment module judges that the current set posture is yes.

9. A vehicle-mounted mobile measurement system, wherein: It comprises an inertial navigation system, a laser scanner, a camera and a controller, wherein the controller is provided with the camera extrinsic parameter calibration device as claimed in claim 8; The camera extrinsic parameter calibration device is used to calibrate the extrinsic parameters of the camera according to the inertial navigation trajectory data collected by the inertial navigation, the point cloud data collected by the laser scanner and the image data collected by the camera.

10. A computer-readable storage medium having computer instructions stored thereon, wherein: When the instruction is executed by the processor, the camera extrinsic parameter calibration method in the vehicle-mounted mobile measurement system described in any one of claims 1-7 is implemented.

Images