Camera extrinsic parameter calibration method, device and system in vehicle-mounted mobile measurement system

Abstract

The present invention discloses a camera extrinsic calibration method, device and system in a vehicle-mounted mobile measurement system. The method comprises: matching the same-name elements of point cloud data and image data to obtain a same-name element combination, aligning the element points included in the same-name element pairs in the two images to obtain the same-name element point pairs; for the same-name element point pairs in the same-name element pairs, according to the set posture of the camera relative to the inertial navigation and the posture of the inertial navigation in the world coordinate system when shooting the image, the first element point is back-projected into the point cloud data to obtain the back-projection point, the back-projection point is projected onto the image where the second element point is located to obtain the projection point, and the ratio of the so-called area of ​​the second element point to the so-called area of ​​the projection point is determined; it is judged whether the average value of the ratio is lower than the error threshold; if not, the set posture is changed until it is judged to be yes, and the current set posture is used as the calibration result of the camera extrinsic parameters. It can realize the automatic calibration of the camera extrinsic parameters in the vehicle-mounted mobile measurement system, and the calibration accuracy is high.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic map data acquisition, and in particular to a camera extrinsic parameter calibration method, device and system in a vehicle-mounted mobile measurement system. Background Art

[0002] In the process of electronic map production, in addition to using point cloud data obtained by laser scanners to produce various map elements, point clouds are usually colored using images taken by cameras to improve the efficiency of electronic map production. The accuracy of point cloud coloring depends on the relative posture between the laser scanner and the camera, that is, the accuracy of the camera's external parameter calibration.

[0003] Currently, the main method for calibrating camera extrinsics is to manually select the corresponding points or boundaries of the same name between the point cloud and the image, and then perform manual calibration. This method has very high requirements on the quality of the point cloud and the calibration personnel. The calibration results are easily affected by noise, prone to matching errors, and time-consuming. Summary of the invention

[0004] In view of the above problems, the present invention is proposed to provide a camera extrinsic parameter calibration method, device and system in a vehicle-mounted mobile measurement system that overcomes the above problems or at least partially solves the above problems.

[0005] In a first aspect, an embodiment of the present invention provides a camera extrinsic parameter calibration method in a vehicle-mounted mobile measurement system, comprising:

[0006] Matching the elements of the same name in the collected point cloud data and at least two images in the image data to obtain an element combination of the same name, and registering the element points included in the element pairs of the same name in the two images in the element combination of the same name to obtain multiple pairs of element points of the same name;

[0007] For the same-name element point pair in the same-name element pair, according to the set position of the camera relative to the inertial navigation and the position of the inertial navigation in the world coordinate system when taking the image, back-project the first element point in the same-name element point pair to the point cloud data to obtain a back-projection point, project the back-projection point to the image where the second element point in the same-name element point pair is located to obtain a projection point, and determine the ratio of the so-called area of ​​the second element point to the so-called area of ​​the projection point;

[0008] Determining whether the average value of the ratio is lower than an error threshold;

[0009] If not, the set posture is changed according to a preset rule until the average value of the re-determined ratio is lower than the error threshold, and the current set posture is used as the calibration result of the camera extrinsic parameter.

[0010] In some optional embodiments, according to the set posture of the camera relative to the inertial navigation system and the posture of the inertial navigation system in the world coordinate system when the image is captured, the first element point in the same-name element point pair is back-projected into the point cloud data to obtain the back-projection point, specifically including:

[0011] According to the timestamp and inertial navigation trajectory data of the image, determine the position and posture of the inertial navigation in the world coordinate system when the image was taken;

[0012] According to the set position and posture of the camera relative to the inertial navigation system and the determined position and posture of the inertial navigation system in the world coordinate system, the first element point in the pair of element points with the same name is back-projected into the point cloud data to obtain a back-projection point.

[0013] In some optional embodiments, determining the position and posture of the inertial navigation system in the world coordinate system when the image is captured based on the timestamp and inertial navigation trajectory data of the image specifically includes:

[0014] Interpolate the inertial navigation trajectory data according to the timestamp of the image to obtain the interpolation point;

[0015] The position and posture of the inertial navigation system included in the interpolation point in the world coordinate system is determined as the position and posture of the inertial navigation system in the world coordinate system when the image is captured.

[0016] In some optional embodiments, according to the set posture of the camera relative to the inertial navigation system and the posture of the inertial navigation system in the world coordinate system when the image is captured, the first element point in the same-name element point pair is back-projected into the point cloud data to obtain the back-projection point, specifically including:

[0017] According to the camera intrinsic parameters and distortion parameters, the pixel coordinates of the first element point in the same-name element point pair are converted into camera coordinates, the camera coordinates of the first element point are converted into inertial navigation coordinates according to the set position of the camera relative to the inertial navigation, and the inertial navigation coordinates of the first element point are converted into world coordinates according to the position of the inertial navigation in the world coordinate system when the image where the first element point is located is taken, to obtain a world coordinate point set, and the intersection of the trajectory where the world coordinate point set is located and the plane where the point cloud data is located is used as the back-projection point of the first element point; correspondingly, the back-projection point is projected onto the image where the second element point in the same-name element point pair is located to obtain a projection point, specifically including:

[0018] According to the posture of the inertial navigation in the world coordinate system when the image where the second element point in the pair of element points with the same name is located is taken, the world coordinates of the back-projection point are converted into inertial navigation coordinates, the inertial navigation coordinates of the back-projection point are converted into camera coordinates according to the set posture of the camera relative to the inertial navigation, the camera coordinates of the back-projection point are converted into pixel coordinates according to the camera intrinsic parameters and distortion parameters, and the point in the image where the second element point is located, determined according to the pixel coordinates of the back-projection point, is used as the projection point of the back-projection point.

[0019] In some optional embodiments, determining whether the average value of the ratio is lower than an error threshold specifically includes:

[0020] It is determined whether an average value of the ratios greater than the first ratio threshold value among the ratios is lower than an error threshold value.

[0021] In some optional embodiments, the determining whether the average value of the ratio is lower than an error threshold further includes:

[0022] A first average value of the ratios of the corresponding ratios of the pairs of elements with the same name in the element combination with the same name that are greater than a first ratio threshold is determined, and it is determined whether an average value of the first average values ​​that are greater than a second ratio threshold is lower than an error threshold.

[0023] In some optional embodiments, the determining whether the average value of the ratio is lower than an error threshold further includes:

[0024] A second average value of the ratios of the corresponding ratios of the pairs of elements with the same name in the two images that are greater than the first ratio threshold is determined, and it is determined whether an average value of the second average values ​​that are greater than the third ratio threshold is lower than an error threshold.

[0025] In a second aspect, an embodiment of the present invention provides a camera extrinsic parameter calibration device in a vehicle-mounted mobile measurement system, comprising:

[0026] A matching module is used to match the elements of the same name in the collected point cloud data and at least two images in the image data to obtain a combination of elements of the same name, and to register the element points included in the element pairs of the same name in the two images in the combination of elements of the same name to obtain multiple pairs of element points of the same name;

[0027] a determination module, for back-projecting the first element point in the same-name element point pair matched by the matching module to the point cloud data to obtain a back-projection point, projecting the back-projection point to the image where the second element point in the same-name element point pair is located to obtain a projection point, and determining a ratio of the so-called area of ​​the second element point to the so-called area of ​​the projection point, according to the set position of the camera relative to the inertial navigation and the position of the inertial navigation in the world coordinate system when shooting the image;

[0028] A judging module, used to judge whether the average value of the ratio determined by the determining module is lower than an error threshold;

[0029] When the judgment module judges as no, the determination module is used to change the set posture according to a preset rule, and re-determine the ratio of the area enclosed by the second feature point to the area enclosed by the projection point;

[0030] The calibration module is used to use the current set posture as the calibration result of the camera external parameter when the judgment module judges that the current set posture is yes.

[0031] In a third aspect, an embodiment of the present invention provides a vehicle-mounted mobile measurement system, including an inertial navigation system, a laser scanner, a camera, and a controller, wherein the controller is provided with the above-mentioned camera extrinsic parameter calibration device;

[0032] The camera extrinsic parameter calibration device is used to calibrate the extrinsic parameters of the camera according to the inertial navigation trajectory data collected by the inertial navigation, the point cloud data collected by the laser scanner and the image data collected by the camera.

[0033] In a fourth aspect, an embodiment of the present invention provides a server, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the camera extrinsic parameter calibration method in the above-mentioned vehicle-mounted mobile measurement system.

[0034] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implements the camera extrinsic parameter calibration method in the above-mentioned vehicle-mounted mobile measurement system.

[0035] The beneficial effects of the above technical solution provided by the embodiment of the present invention include at least:

[0036] 1. The camera extrinsic calibration method in the vehicle-mounted mobile measurement system provided by the embodiment of the present invention is to match the same-name elements in the point cloud data and at least two images in the image data to obtain the same-name element combination, and align the element points included in the same-name element pairs in the two images to obtain a number of same-name element point pairs; for the same-name element point pairs in the same-name element pairs, according to the set posture of the camera relative to the inertial navigation and the posture of the inertial navigation in the world coordinate system when shooting the image, the first element point is back-projected into the point cloud data to obtain the back-projection point, and the back-projection point is projected onto the image where the second element point is located to obtain the projection point, and the ratio of the so-called area of ​​the second element point to the so-called area of ​​the projection point is determined; it is judged whether the average value of the ratio is lower than the error threshold; if not, the set posture is changed according to the preset rules until the average value is lower than the error threshold, and the current set posture is used as the calibration result of the camera extrinsic parameter. The entire calibration process is completed automatically without manual participation, realizing automatic calibration, saving labor costs, and improving calibration efficiency.

[0037] 2. For the pairs of feature points with the same name, the first feature point is back-projected into the point cloud data to obtain the back-projection point, and the back-projection point is projected onto the image where the second feature point is located to obtain the projection point. The ratio of the so-called area of ​​the second feature point to the so-called area of ​​the projection point is determined. The feature points in the point cloud data are directly projected onto the image where the second feature point is located, and the feature points in the point cloud data are corrected, thereby avoiding the calibration error caused by the accuracy of the point cloud data itself. Therefore, the quality requirements for the point cloud data can be reduced, the adaptability range is expanded, and the calibration accuracy is high.

[0038] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0039] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0041] Figure 1 This is a flow chart of a camera extrinsic parameter calibration method in a vehicle-mounted mobile measurement system in Embodiment 1 of the present invention;

[0042] Figure 2 Schematic diagram of point cloud data, images and inertial navigation trajectory in Embodiment 1 of the present invention;

[0043] Figure 3 This is a schematic diagram of a combination of elements with the same name in Embodiment 1 of the present invention;

[0044] Figure 4 A specific implementation flow chart for back-projecting the first feature point into point cloud data;

[0045] Figure 5 A specific implementation flow chart for projecting the back-projected point onto the image where the second feature point is located;

[0046] Fig. 6A This is an example diagram of a method for determining an average value in Embodiment 1 of the present invention;

[0047] Figure 6B This is an example diagram of another method for determining an average value in Embodiment 1 of the present invention;

[0048] Figure 7 This is a specific implementation flow chart for camera extrinsic calibration in a vehicle-mounted mobile measurement system;

[0049] Figure 8 Schematic diagram of the structure of a camera extrinsic parameter calibration device in a vehicle-mounted mobile measurement system according to an embodiment of the present invention;

[0050] Fig. 9 Schematic diagram of the structure of the vehicle-mounted mobile measurement system in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0052] In order to solve the problems in the prior art that manual calibration of camera extrinsic parameters has high requirements on point cloud quality and calibration personnel, and has large calibration errors and a long time consumption, the embodiments of the present invention provide a camera extrinsic parameter calibration method, device and system in a vehicle-mounted mobile measurement system, which can realize automatic calibration of camera extrinsic parameters in a vehicle-mounted mobile measurement system with high calibration efficiency and accuracy.

[0053] The vehicle-mounted mobile measurement system refers to a comprehensive measurement system composed of an integrated positioning and orientation system (POS) (referred to as inertial navigation), a control system, and a measurement sensor (laser scanner, camera, etc.) on a vehicle-mounted platform. The vehicle-mounted mobile measurement system used in the technical solution of the present application includes an inertial navigation, a control system, a laser scanner, and a camera.

[0054] First, a vehicle-mounted mobile measurement system is used to collect data. The data collection process is carried out while the vehicle equipped with the vehicle-mounted mobile measurement system is moving. The 3D laser scanner is used to obtain point cloud data; the inertial navigation is used to obtain the inertial navigation posture information (the position and posture information of the inertial navigation in the world coordinate system) and obtain the inertial navigation trajectory data, including the trajectory of the relationship between the inertial navigation position information and time and the posture trajectory of the relationship between the position and posture information of the inertial navigation in the world coordinate system and time; and the camera is used to obtain images within the shooting angle of view.

[0055] Embodiment 1

[0056] Embodiment 1 of the present invention provides a camera extrinsic parameter calibration method in a vehicle-mounted mobile measurement system, and the process is as follows: Figure 1 As shown, the following steps are included:

[0057] Step S11: matching the elements of the same name in the collected point cloud data and at least two images in the image data to obtain an element combination of the same name, and aligning the element points included in the element pairs of the same name in each pair of images in the element combination of the same name to obtain multiple pairs of element points of the same name.

[0058] The vehicle-mounted mobile measurement system acquires the following data during the driving process: image data acquired by the camera, point cloud data acquired by the laser scanner, and inertial navigation trajectory data acquired by the inertial navigation. The inertial navigation trajectory data includes: the trajectory of the relationship between the position information of the inertial navigation and time, and the posture trajectory of the relationship between the posture information of the inertial navigation in the world coordinate system and time.

[0059] Specifically, the obtained point cloud data and inertial navigation trajectory data are solved, or the original point cloud data and inertial navigation trajectory data are first obtained, and then the corresponding solving method is used to complete the data solving.

[0060] The original point cloud data is the point cloud data under the laser scanner's own coordinate system (SOCS). It is necessary to use relevant solution methods and combine the inertial navigation posture information (the inertial navigation posture in the world coordinate system can be used as the laser scanner's posture in the world coordinate system) to solve the panoramic point cloud data of the world coordinate system; the position information of the trajectory points in the collected original inertial navigation trajectory data is longitude and latitude, which needs to be solved into the world coordinate system.

[0061] Optionally, the solved point cloud data and inertial navigation trajectory data can also be in a local world coordinate system, which can reduce the size of the coordinate values ​​in the data, reduce the amount of calculation, and increase the readability of the data.

[0062] Reference Figure 2 As shown, the left side is the solved point cloud data; the middle is the acquired image; the right side is the solved inertial navigation trajectory data, including the inertial navigation trajectory and posture trajectory.

[0063] Matching the elements with the same name in the collected point cloud data and at least two images in the image data to obtain a combination of elements with the same name, specifically, it may include first determining the elements in the point cloud data, such as a road sign, a lane line, or a ground sign, etc.; for each element in the point cloud data, using a deep learning method, matching the elements in at least two images from the image data to obtain a combination of elements with the same name, and referring to Figure 3 As shown, the point cloud data is the point cloud data corresponding to a certain road sign. The road sign in the three images and the road sign corresponding to the point cloud data are the same road sign, that is, they are elements with the same name.

[0064] Match the same-name feature point pairs included in each pair of images from the same-name feature combination, that is, perform same-name point registration on the same-name features in each pair of images in the same-name feature combination. For example, if images A, B, C, D, and E taken sequentially within a range of 50 meters all contain the same road sign, then the road sign in image A can be registered with the road sign in image B, the road sign in image B can be registered with the road sign in image C, the road sign in image C can be registered with the road sign in image D, and the road sign in image D can be registered with the road sign in image E. That is, perform same-name point registration on the same-name features in two images taken at adjacent times to obtain same-name feature point pairs.

[0065] The above-mentioned pairs of feature points with the same name are the image points of the same point on the object in different images.

[0066] Step S12: For the same-name element point pairs in the same-name element pairs, according to the set position of the camera relative to the inertial navigation and the position of the inertial navigation in the world coordinate system when taking the image, the first element point in the same-name element point pair is back-projected into the point cloud data to obtain the back-projection point, and the back-projection point is projected onto the image where the second element point in the same-name element point pair is located to obtain the projection point, and the ratio of the so-called area of ​​the second element point to the so-called area of ​​the projection point is determined.

[0067] The initial value of the set position and posture of the above-mentioned camera relative to the inertial navigation system can be the known position and posture of the camera at the initial moment of the camera extrinsic calibration. That is, for a camera that has never undergone extrinsic calibration, its set position and posture can be the initial position and posture of the camera relative to the inertial navigation system when the vehicle-mounted mobile measurement system is installed; for a camera that has undergone extrinsic calibration, its set position and posture can be the most recent camera extrinsic calibration result.

[0068] Specifically, the above-mentioned camera external parameters, namely the position of the camera relative to the inertial navigation, refer to the position parameters of the camera relative to the inertial navigation: X pano-pos , Y pano-pos and Z pano-pos , and the attitude angle of the camera relative to the inertial navigation: Yaw pano-pos , Pitch pano-pos and roll angle ROll pano-pos .

[0069] In one embodiment, referring to Figure 4 As shown, back-projecting the first feature point in the feature point pair with the same name to the point cloud data to obtain the back-projected point may include the following steps:

[0070] Step S41: converting the pixel coordinates of the first feature point in the feature point pair with the same name into camera coordinates according to the camera intrinsic parameters and distortion parameters.

[0071] Step S42: Convert the camera coordinates of the first feature point into inertial navigation coordinates according to the set position of the camera relative to the inertial navigation.

[0072] Step S43: according to the position and posture of the inertial navigation in the world coordinate system when the image where the first feature point is located is captured, the inertial navigation coordinates of the first feature point are converted into world coordinates to obtain a world coordinate point set.

[0073] Step S44: taking the intersection of the trajectory of the world coordinate point set and the plane of the point cloud data as the back-projection point of the first feature point.

[0074] Projecting the back-projected point to the image where the second element point in the same-name element point pair is located to obtain the projection point is the inverse process of back-projecting the first element point to the point cloud data to obtain the back-projected point. Figure 5 As shown, the following steps may be included:

[0075] Step S51: converting the world coordinates of the back-projected point into inertial navigation coordinates according to the position and posture of the inertial navigation in the world coordinate system when the image of the second element point in the same-name element point pair is captured.

[0076] Step S52: converting the inertial navigation coordinates of the back-projected point into camera coordinates according to the set position of the camera relative to the inertial navigation.

[0077] Specifically, it can also be directly based on the following formula (1), through the transformation matrix from the world coordinate system to the inertial navigation coordinate system: And the transformation matrix from the inertial navigation coordinate system to the camera coordinate system Get the transformation matrix from the world coordinate system to the camera coordinate system

[0078]

[0079] Then according to the transformation matrix from the world coordinate system to the camera coordinate system Posture changes in and position transformation Using formula (2), the world coordinates P of the feature point W Convert to camera coordinates P C :

[0080]

[0081] Step S53: Convert the camera coordinates of the back-projected point into pixel coordinates according to the camera intrinsic parameters and distortion parameters.

[0082] Specifically, you can first transpose the horizontal and vertical coordinates in the camera coordinates:

[0083]

[0084] Then according to the horizontal coordinate of the transposed camera coordinates and the vertical coordinate The preset camera intrinsic parameters and distortion parameters are used to convert the camera coordinates of the back-projected points into pixel coordinates using formulas (4) to (7):

[0085]

[0086] K=1+K1r 2 +K2r 4 +K3r 6 Formula (5)

[0087]

[0088]

[0089] In formula (4) to formula (7), u and v are pixel coordinates, f x 、f y 、c x and c y is the intrinsic parameter of the camera, k1, k2, k3, p1 and p2 are the distortion parameters of the camera.

[0090] Step S54: taking the point in the image where the second feature point is located, which is determined according to the pixel coordinates of the back-projection point, as the projection point of the back-projection point.

[0091] For the pair of element points with the same name in the pair of elements with the same name, after determining the projection point of the first element point in the image where the second element point is located, determine the area of ​​the range enclosed by the second element point, that is, the area of ​​the corresponding element with the same name; determine the area of ​​the range enclosed by the projection points; and then determine the ratio of the so-called area of ​​the second element point to the so-called area of ​​the projection point.

[0092] Step S13: Determine whether the average value of the ratio is lower than the error threshold.

[0093] Specifically, the average value of the ratio may be determined by the following three methods.

[0094] Method 1: Directly determine the average value of the ratio.

[0095] The ratios determined based on the pairs of features with the same name are summarized to determine an average of the ratios that are greater than a first ratio threshold.

[0096] If the ratio of the so-called area of ​​the second element point to the so-called area of ​​the projection point is greater than the first ratio threshold, it can be considered that there is a problem with the first element and / or first element data of the corresponding pair of elements with the same name, so it is deleted and the average value of the filtered ratio is determined, so that the obtained average value is closer to reality and a large amount of interference information is eliminated.

[0097] Method 2: Determine the average value of the ratio using the combination of elements with the same name as the unit.

[0098] For each combination of elements with the same name, determine a first average value of the ratios of the corresponding ratios of the pairs of elements with the same name in the combination of elements with the same name that are greater than a first ratio threshold; and determine an average value of the first average values ​​that are greater than a second ratio threshold.

[0099] The average value of the ratio is determined using the combination of elements with the same name as a unit. If the first average value of the ratio corresponding to the combination of elements with the same name is greater than the second ratio threshold, it means that the corresponding elements may be blocked or interfered in the process of data collection, resulting in errors in the collected data. Therefore, they are deleted and the average value of the filtered ratio is determined, so that the obtained average value is closer to reality and interference information is eliminated.

[0100] For example, refer to Fig. 6A As shown, taking a group of combinations of elements with the same name as an example, image A-image D have elements with the same name, after the same name point alignment is completed for the element pair AB in image A and image B, the element pair BC in image B and image C, and the element pair CD in image C and image D, the ratio AB corresponding to the element pair AB, the ratio BC corresponding to the element pair BC, and the ratio CD corresponding to the element pair CD are determined using the method of step S12, and then the first average value of the ratios greater than the first ratio threshold among all the ratios obtained for the above-mentioned element pairs with the same name is determined; and the average value of the first average values ​​greater than the second ratio threshold is determined.

[0101] Method 3: Determine the average value of the ratio using the image as a unit.

[0102] A second average value of the ratios of the corresponding ratios of the pairs of elements with the same name in the two images that are greater than the first ratio threshold is determined, and an average value of the second average values ​​that are greater than the third ratio threshold is determined.

[0103] The average value of the ratio is determined based on the image as a unit. If the second average value of the ratio determined based on all pairs of elements with the same name in the two images is greater than the third ratio threshold, it means that there may be problems with the data of one or both images, so they are deleted and the average value of the filtered ratio is determined, so that the obtained average value is closer to reality and interference information is eliminated.

[0104] Reference Figure 6B As shown, the same-name element pairs matched in image A and image B include same-name element pair AB1, same-name element pair AB2, same-name element pair AB3, same-name element pair AB4, same-name element pair AB5 and same-name element pair AB6. For the same-name element point pairs after registration in the same-name element pair AB1, the method of step S12 is used to obtain the ratio AB1; for the same-name element point pairs after registration in the same-name element pair AB2, the method of step S12 is used to obtain the ratio AB2; for the same-name element point pairs after registration in the same-name element pair AB3, the method of step S12 is used to obtain the ratio AB3; for the same-name element point pairs after registration in the same-name element pair AB4, the method of step S12 is used to obtain the ratio AB4; for the same-name element point pairs after registration in the same-name element pair AB5, the method of step S12 is used to obtain the ratio AB5; for the same-name element point pairs after registration in the same-name element pair AB6, the method of step S12 is used to obtain the ratio AB6. Determine a second average value of the ratios of the corresponding ratios of the pairs of elements with the same name in the above-mentioned image A and image B that are greater than the first ratio threshold; and determine an average value of the obtained second average values ​​that are greater than the third ratio threshold.

[0105] Optionally, other methods for determining the average value of the ratio may be selected according to actual needs, and this embodiment does not limit the specific method for determining the average value of the ratio.

[0106] When the judgment result of step S13 is yes, step S15 is executed; when the judgment result of step S13 is no, step S14 is executed.

[0107] Step S14: changing the set position and posture of the camera relative to the inertial navigation according to a preset rule.

[0108] When it is determined that the average value of the corresponding ratios of the pairs of elements with the same name is not lower than the error threshold, the set position and posture of the camera relative to the inertial navigation is changed according to a preset rule, and step S12 is re-executed to determine a new average value of the ratios.

[0109] Step S15: Use the current set posture as the calibration result of the camera external parameters.

[0110] When it is judged that the average value of the corresponding ratios of each pair of elements with the same name is lower than the error threshold, the set position and posture of the current camera relative to the inertial navigation is used as the calibration result of the camera extrinsic parameter.

[0111] The entire camera extrinsic parameter calibration process in the above-mentioned embodiment 1 is completed automatically without manual intervention, thus realizing automatic calibration, saving labor costs, and improving calibration efficiency.

[0112] For the feature point pairs with the same name, the first feature point is back-projected into the point cloud data to obtain the back-projection point, and the back-projection point is projected onto the image where the second feature point is located to obtain the projection point. The ratio of the so-called area of ​​the second feature point to the so-called area of ​​the projection point is determined. It is more direct to project the feature points in the point cloud data onto the image where the second feature point is located, and the feature points in the point cloud data are corrected, thereby avoiding the calibration error caused by the accuracy of the point cloud data itself. Therefore, the quality requirements for the point cloud data can be reduced, the adaptability range is expanded, and the calibration accuracy is high.

[0113] Embodiment 2

[0114] Embodiment 2 of the present invention provides a specific implementation method for camera extrinsic calibration in a vehicle-mounted mobile measurement system, and the process is as follows: Figure 7 As shown, the following steps are included:

[0115] Step S71: acquiring image data, point cloud data and inertial navigation trajectory data collected by the vehicle-mounted mobile measurement system during the driving process of the vehicle.

[0116] Specifically, the acquired point cloud data and inertial navigation trajectory data are the data of the local world coordinate system after the solution. This can reduce the size of the coordinate values ​​in the data, reduce the amount of calculation, and increase the readability of the data.

[0117] Step S72: Identify the elements in the point cloud data. For each element, use a deep learning method to determine the elements with the same name in at least two images in the image data to obtain a combination of elements with the same name. Align the element points included in the pairs of elements with the same name in each pair of images in the combination of elements with the same name to obtain multiple pairs of element points with the same name.

[0118] For each pair of elements with the same name, the following steps S73 to S75 are executed respectively.

[0119] Step S73: interpolating the inertial navigation trajectory data according to the timestamp of the image where the first element point in the same-name element point pair is located to obtain interpolation points, and determining the position and posture of the inertial navigation in the world coordinate system included in the interpolation points as the position and posture of the inertial navigation in the world coordinate system when the image is captured.

[0120] For a clearer description, the image where the first feature point is located is described as the first image, and the position and posture of the inertial navigation system in the world coordinate system when the first image is taken is described as the first position and posture of the inertial navigation system in the world coordinate system; the image where the second feature point in the pair of feature points with the same name is located is described as the second image, and the position and posture of the inertial navigation system in the world coordinate system when the second image is taken is described as the second position and posture of the inertial navigation system in the world coordinate system.

[0121] Step S74: According to the set position of the camera relative to the inertial navigation and the first position of the inertial navigation in the world coordinate system, the first element point is back-projected into the point cloud data to obtain a back-projection point.

[0122] Step S75: According to the set position of the camera relative to the inertial navigation system and the second position of the inertial navigation system in the world coordinate system, the back-projection point is projected onto the second image to obtain the projection point, and the ratio of the area enclosed by the second element point to the area enclosed by the projection point is determined.

[0123] Through the above steps S73-S75, for a pair of elements with the same name in each pair of images, the corresponding area ratio is determined. After the corresponding ratios of all pairs of elements with the same name are determined, step S76 is executed.

[0124] Step S76: Determine whether the average value of the ratios greater than the first ratio threshold value among the ratios is lower than the error threshold value.

[0125] When the judgment of step S76 is yes, execute step S78; when the judgment of step S76 is no, execute step S77.

[0126] Step S77: changing the set position and posture of the camera relative to the inertial navigation according to a preset rule.

[0127] When it is determined that the average value of the ratios greater than the first ratio threshold in the ratios is not lower than the error threshold, the set position and posture of the camera relative to the inertial navigation is changed according to a preset rule, and step S74 and step S75 are re-executed.

[0128] Step S78: Using the current camera's set position relative to the inertial navigation as the calibration result of the camera's external parameters.

[0129] When the average value of the ratios greater than the first ratio threshold in the judgment ratios is lower than the error threshold, the set position and posture of the current camera relative to the inertial navigation is used as the calibration result of the camera extrinsic parameter.

[0130] Based on the inventive concept of the present invention, an embodiment of the present invention further provides a camera extrinsic parameter calibration device in a vehicle-mounted mobile measurement system, whose structure is as follows: Figure 8 As shown, including:

[0131] A matching module 81 is used to match the elements of the same name in the collected point cloud data and at least two images in the image data to obtain an element combination of the same name, and to register the element points included in the element pairs of the same name in the two images in the element combination of the same name to obtain multiple pairs of element points of the same name;

[0132] The determination module 82 is used for, for the same-name element point pairs in the same-name element point pairs matched by the matching module 81, according to the set position of the camera relative to the inertial navigation and the position of the inertial navigation in the world coordinate system when shooting the image, back-projecting the first element point in the same-name element point pair to the point cloud data to obtain a back-projection point, projecting the back-projection point to the image where the second element point in the same-name element point pair is located to obtain a projection point, and determining the ratio of the so-called area of ​​the second element point to the so-called area of ​​the projection point;

[0133] A judging module 83, used to judge whether the average value of the ratio determined by the determining module 82 is lower than an error threshold;

[0134] When the judgment module 83 judges as no, the determination module 82 is used to change the set posture according to a preset rule, and re-determine the ratio of the area enclosed by the second feature point to the area enclosed by the projection point;

[0135] The calibration module 84 is used to use the current set posture as the calibration result of the camera external parameters when the judgment module 83 judges to be yes.

[0136] In some embodiments, the determination module 82 back-projects the first element point in the same-name element point pair onto the point cloud data to obtain a back-projection point according to the set position of the camera relative to the inertial navigation system and the position of the inertial navigation system in the world coordinate system when the image is captured, specifically for:

[0137] According to the timestamp of the image and the inertial navigation trajectory data, the position and posture of the inertial navigation in the world coordinate system when the image is taken is determined; according to the set position and posture of the camera relative to the inertial navigation and the determined position and posture of the inertial navigation in the world coordinate system, the first element point in the same-name element point pair is back-projected into the point cloud data to obtain a back-projection point.

[0138] In some embodiments, the determination module 82 determines the position and posture of the inertial navigation in the world coordinate system when the image is captured based on the timestamp of the image and the inertial navigation trajectory data, specifically for:

[0139] The inertial navigation trajectory data is interpolated according to the timestamp of the image to obtain interpolation points; and the position and posture of the inertial navigation in the world coordinate system included in the interpolation points is determined as the position and posture of the inertial navigation in the world coordinate system when the image is taken.

[0140] In some embodiments, the determination module 82 back-projects the first element point in the same-name element point pair onto the point cloud data to obtain a back-projection point according to the set position of the camera relative to the inertial navigation system and the position of the inertial navigation system in the world coordinate system when the image is captured, specifically for:

[0141] According to the camera intrinsic parameters and distortion parameters, the pixel coordinates of the first element point in the same-name element point pair are converted into camera coordinates, the camera coordinates of the first element point are converted into inertial navigation coordinates according to the set position of the camera relative to the inertial navigation, and the inertial navigation coordinates of the first element point are converted into world coordinates according to the position of the inertial navigation in the world coordinate system when the image where the first element point is located is taken, to obtain a world coordinate point set, and the intersection of the trajectory where the world coordinate point set is located and the plane where the point cloud data is located is used as the back-projection point of the first element point; correspondingly, the back-projection point is projected onto the image where the second element point in the same-name element point pair is located to obtain a projection point, which is specifically used for:

[0142] According to the posture of the inertial navigation in the world coordinate system when the image where the second element point in the pair of element points with the same name is located is taken, the world coordinates of the back-projection point are converted into inertial navigation coordinates, the inertial navigation coordinates of the back-projection point are converted into camera coordinates according to the set posture of the camera relative to the inertial navigation, the camera coordinates of the back-projection point are converted into pixel coordinates according to the camera intrinsic parameters and distortion parameters, and the point in the image where the second element point is located, determined according to the pixel coordinates of the back-projection point, is used as the projection point of the back-projection point.

[0143] In some embodiments, the determination module 83 is specifically configured to:

[0144] It is determined whether an average value of the ratios greater than the first ratio threshold value among the ratios is lower than an error threshold value.

[0145] In some embodiments, the determination module 83 is further configured to:

[0146] A first average value of the ratios of the corresponding ratios of the pairs of elements with the same name in the element combination with the same name that are greater than a first ratio threshold is determined, and it is determined whether an average value of the first average values ​​that are greater than a second ratio threshold is lower than an error threshold.

[0147] In some embodiments, the determination module 83 is further configured to:

[0148] A second average value of the ratios of the corresponding ratios of the pairs of elements with the same name in the two images that are greater than the first ratio threshold is determined, and it is determined whether an average value of the second average values ​​that are greater than the third ratio threshold is lower than an error threshold.

[0149] Based on the inventive concept of the present invention, an embodiment of the present invention further provides a load-carrying mobile measurement system, whose structure is as follows: Fig. 9 As shown, it includes an inertial navigation system 91, a laser scanner 92, a camera 93 and a controller 94, and the controller 94 is provided with the above-mentioned camera external parameter calibration device;

[0150] The camera extrinsic parameter calibration device is used to calibrate the extrinsic parameters of the camera 93 according to the inertial navigation trajectory data collected by the inertial navigation 91, the point cloud data collected by the laser scanner 92 and the image data collected by the camera 93.

[0151] Regarding the apparatus and system in the above-mentioned embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments of the method, and will not be elaborated here.

[0152] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the camera extrinsic parameter calibration in the above-mentioned vehicle-mounted mobile measurement system is implemented.

[0153] Based on the same inventive concept, an embodiment of the present invention also provides a server, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the camera extrinsic parameter calibration in the above-mentioned vehicle-mounted mobile measurement system is implemented.

[0154] Unless otherwise specifically stated, terms such as processing, computing, calculating, determining, displaying, etc. may refer to the actions and / or processes of one or more processing or computing systems, or similar devices, which operate and convert data represented as physical (e.g., electronic) quantities within registers or memories of a processing system into other data similarly represented as physical quantities within memories, registers, or other such information storage, transmission, or display devices of the processing system. Information and signals may be represented using any of a variety of different techniques and methods. For example, data, instructions, commands, information, signals, bits, symbols, and chips mentioned throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, light fields or particles, or any combination thereof.

[0155] It should be understood that the specific order or hierarchy of steps in the disclosed process is an example of an exemplary method. Based on design preferences, it should be understood that the specific order or hierarchy of steps in the process can be rearranged without departing from the scope of protection of the present disclosure. The attached method claims present the elements of the various steps in an exemplary order and are not intended to be limited to the specific order or hierarchy described.

[0156] In the above detailed description, various features are grouped together in a single embodiment to simplify the disclosure. This method of disclosure should not be interpreted as reflecting an intention that the embodiments of the claimed subject matter require more features than are clearly stated in each claim. On the contrary, as reflected in the appended claims, the invention is in a state of having less than all the features of the disclosed individual embodiments. Therefore, the appended claims are hereby expressly incorporated into the detailed description, with each claim standing on its own as a separate preferred embodiment of the invention.

[0157] Those skilled in the art will also appreciate that the various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the embodiments herein can all be implemented as electronic hardware, computer software, or a combination thereof. In order to clearly illustrate the interchangeability between hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above around their functions. Whether such functions are implemented as hardware or software depends on specific applications and the design constraints imposed on the entire system. A skilled person can implement the described functions in an alternative manner for each specific application, but such implementation decisions should not be interpreted as departing from the scope of protection of the present disclosure.

[0158] The steps of the method or algorithm described in conjunction with the embodiments herein may be directly embodied as hardware, a software module executed by a processor, or a combination thereof. The software module may be located in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a mobile disk, a CD-ROM, or any other form of storage medium well known in the art. An exemplary storage medium is connected to the processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium may also be an integral part of the processor. The processor and the storage medium may be located in an ASIC. The ASIC may be located in a user terminal. Of course, the processor and the storage medium may also be present in a user terminal as discrete components.

[0159] For software implementation, the techniques described in this application can be implemented with modules (e.g., procedures, functions, etc.) that perform the functions described in this application. These software codes can be stored in a memory unit and executed by a processor. The memory unit can be implemented within the processor or outside the processor. In the latter case, it is coupled to the processor in a communication manner via various means, which are well known in the art.

[0160] The above description includes examples of one or more embodiments. Of course, it is impossible to describe all possible combinations of components or methods for the purpose of describing the above embodiments, but it should be recognized by those skilled in the art that the various embodiments may be further combined and arranged. Therefore, the embodiments described herein are intended to cover all such changes, modifications and variations that fall within the scope of protection of the appended claims. In addition, with respect to the term "comprising" used in the specification or claims, the word is covered in a manner similar to the term "including", just as "including," is explained as a transitional word in the claims. In addition, any term "or" used in the specification of the claims is intended to mean "non-exclusive or".

Claims

1. Camera extrinsic calibration method in vehicle-mounted mobile measurement system, where: include: Matching the elements of the same name in the collected point cloud data and at least two images in the image data to obtain an element combination of the same name, and registering the element points included in the element pairs of the same name in the two images in the element combination of the same name to obtain multiple pairs of element points of the same name; For the same-name element point pair in the same-name element pair, according to the set position of the camera relative to the inertial navigation and the position of the inertial navigation in the world coordinate system when taking the image, the first element point in the same-name element point pair is back-projected into the point cloud data to obtain a back-projection point, the back-projection point is projected onto the image where the second element point in the same-name element point pair is located to obtain a projection point, and the ratio of the area enclosed by the second element point to the area enclosed by the projection point is determined; Determining whether the average value of the ratio is lower than an error threshold; If not, the set posture is changed according to a preset rule until the average value of the re-determined ratio is lower than the error threshold, and the current set posture is used as the calibration result of the camera extrinsic parameter.

2. The method of claim 1, wherein: According to the set position of the camera relative to the inertial navigation system and the position of the inertial navigation system in the world coordinate system when the image is captured, the first element point in the element point pair with the same name is back-projected onto the point cloud data to obtain the back-projection point, specifically including: According to the timestamp and inertial navigation trajectory data of the image, determine the position and posture of the inertial navigation in the world coordinate system when the image was taken; According to the set position and posture of the camera relative to the inertial navigation system and the determined position and posture of the inertial navigation system in the world coordinate system, the first element point in the pair of element points with the same name is back-projected into the point cloud data to obtain a back-projection point.

3. The method of claim 2, wherein: Determining the position and posture of the inertial navigation in the world coordinate system when the image is captured based on the timestamp and inertial navigation trajectory data of the image specifically includes: Interpolate the inertial navigation trajectory data according to the timestamp of the image to obtain the interpolation point; The position and posture of the inertial navigation system included in the interpolation point in the world coordinate system is determined as the position and posture of the inertial navigation system in the world coordinate system when the image is captured.

4. The method of claim 1, wherein: According to the set position of the camera relative to the inertial navigation system and the position of the inertial navigation system in the world coordinate system when the image is captured, the first element point in the element point pair with the same name is back-projected onto the point cloud data to obtain the back-projection point, specifically including: According to the camera intrinsic parameters and distortion parameters, the pixel coordinates of the first element point in the same-name element point pair are converted into camera coordinates, the camera coordinates of the first element point are converted into inertial navigation coordinates according to the set position of the camera relative to the inertial navigation, and the inertial navigation coordinates of the first element point are converted into world coordinates according to the position of the inertial navigation in the world coordinate system when the image where the first element point is located is taken, to obtain a world coordinate point set, and the intersection of the trajectory where the world coordinate point set is located and the plane where the point cloud data is located is used as the back-projection point of the first element point; correspondingly, the back-projection point is projected onto the image where the second element point in the same-name element point pair is located to obtain a projection point, specifically including: According to the posture of the inertial navigation in the world coordinate system when the image where the second element point in the pair of element points with the same name is located is taken, the world coordinates of the back-projection point are converted into inertial navigation coordinates, the inertial navigation coordinates of the back-projection point are converted into camera coordinates according to the set posture of the camera relative to the inertial navigation, the camera coordinates of the back-projection point are converted into pixel coordinates according to the camera intrinsic parameters and distortion parameters, and the point in the image where the second element point is located, determined according to the pixel coordinates of the back-projection point, is used as the projection point of the back-projection point.

5. The method of claim 1, wherein: The determining whether the average value of the ratio is lower than the error threshold specifically includes: It is determined whether an average value of the ratios greater than the first ratio threshold value among the ratios is lower than an error threshold value.

6. The method of claim 5, wherein: The determining whether the average value of the ratio is lower than an error threshold further includes: A first average value of the ratios of the corresponding ratios of the pairs of elements with the same name in the element combination with the same name that are greater than a first ratio threshold is determined, and it is determined whether an average value of the first average values ​​that are greater than a second ratio threshold is lower than an error threshold.

7. The method of claim 6, wherein: The determining whether the average value of the ratio is lower than an error threshold further includes: A second average value of the ratios of the corresponding ratios of the pairs of elements with the same name in the two images that are greater than the first ratio threshold is determined, and it is determined whether an average value of the second average values ​​that are greater than the third ratio threshold is lower than an error threshold.

8. A camera extrinsic calibration device in a vehicle-mounted mobile measurement system, wherein: include: A matching module is used to match the elements of the same name in the collected point cloud data and at least two images in the image data to obtain a combination of elements of the same name, and to register the element points included in the element pairs of the same name in the two images in the combination of elements of the same name to obtain multiple pairs of element points of the same name; a determination module, for back-projecting the first element point in the same-name element point pair matched by the matching module to the point cloud data to obtain a back-projection point, projecting the back-projection point to the image where the second element point in the same-name element point pair is located to obtain a projection point, and determining a ratio of the so-called area of ​​the second element point to the so-called area of ​​the projection point, according to the set position of the camera relative to the inertial navigation and the position of the inertial navigation in the world coordinate system when shooting the image; A judging module, used to judge whether the average value of the ratio determined by the determining module is lower than an error threshold; When the judgment module judges as no, the determination module is used to change the set posture according to a preset rule, and re-determine the ratio of the area enclosed by the second element point to the area enclosed by the projection point; The calibration module is used to use the current set posture as the calibration result of the camera external parameter when the judgment module judges that the current set posture is yes.

9. A vehicle-mounted mobile measurement system, wherein: It comprises an inertial navigation system, a laser scanner, a camera and a controller, wherein the controller is provided with the camera extrinsic parameter calibration device as claimed in claim 8; The camera extrinsic parameter calibration device is used to calibrate the extrinsic parameters of the camera according to the inertial navigation trajectory data collected by the inertial navigation, the point cloud data collected by the laser scanner and the image data collected by the camera.

10. A computer-readable storage medium having computer instructions stored thereon, wherein: When the instruction is executed by the processor, the camera extrinsic parameter calibration method in the vehicle-mounted mobile measurement system described in any one of claims 1-7 is implemented.

Images