Camera extrinsic parameter calibration method and device, computer device and storage medium

Abstract

The application relates to a camera external parameter calibration method and device, computer equipment and a storage medium. The method comprises the following steps: extracting first lane line detection data and first positioning data from road test data of a vehicle where a target camera is located; the first lane line detection data comprises a plurality of detection results, and the first positioning data comprises positioning results corresponding to the detection results; performing specified detection on the first lane line detection data to obtain second lane line detection data; the second lane line detection data comprises detection results in the first lane line detection data that pass the specified detection; second positioning data is obtained, and the second positioning data comprises positioning results corresponding to the detection results in the second lane line detection data; and the target camera is calibrated according to the second lane line detection data, the second positioning data and a high-precision map. The application can quickly calibrate the camera external parameter in any area where a high-precision map exists, and high-precision calibration results can be obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of camera calibration, and particularly relates to a camera extrinsic parameter calibration method and device, computer equipment and a storage medium. BACKGROUND

[0002] The following statements are merely provided for background information related to the present application and do not necessarily constitute the prior art.

[0003] The camera extrinsic parameter is also called camera pose, which is composed of a rotation matrix and a translation matrix, and the camera extrinsic parameter calibration refers to determining a rotation matrix and a translation matrix to describe the conversion relationship between the camera coordinate system and other coordinate systems (such as the vehicle body coordinate system). In the field of computer vision application, the camera extrinsic parameter calibration is a key link, because the accuracy of calibration determines whether the computer vision system can effectively perform positioning, ranging, detection and other functions involving two-dimensional and three-dimensional interaction.

[0004] The inventor finds that, when calibrating the camera extrinsic parameter, an auxiliary such as a checkerboard, a high-precision sensor (such as a laser radar) or a high-precision ranging device built indoors is usually needed, although this can obtain high-precision calibration results, however, it is impossible to realize real-time and real-site calibration, and the operation is relatively cumbersome when calibrating by using the above-mentioned devices, for example, when calibrating the camera extrinsic parameter by using the laser radar, the camera extrinsic parameter to the laser radar needs to be calibrated by using the projection of the point cloud to the image first, and then the camera extrinsic parameter is obtained according to the transfer relationship, the above-mentioned method cannot directly obtain the camera extrinsic parameter, and on the other hand, the error is accumulated in the transfer process, which leads to the decline of the calibration accuracy. SUMMARY

[0005] In view of the above-mentioned defects or shortcomings, the present application provides a camera extrinsic parameter calibration method, device, computer equipment and storage medium, the embodiments of the present application can quickly complete the calibration of the camera extrinsic parameter in any area where there is a high-precision map, and high-precision calibration results can be obtained.

[0006] According to the first aspect, the application provides a camera extrinsic parameter calibration method. In one embodiment, the method comprises: extracting first lane line detection data and first positioning data from test data of a vehicle in which a target camera is located; the first lane line detection data comprises a plurality of detection results, and the first positioning data comprises positioning results corresponding to the detection results in the first lane line detection data; performing specified detection on the first lane line detection data to obtain second lane line detection data; the second lane line detection data comprises detection results in the first lane line detection data that pass the specified detection; the specified detection comprises one or more of occlusion detection, positioning quality detection and angle distribution detection; obtaining second positioning data, wherein the second positioning data comprises positioning results corresponding to the detection results in the second lane line detection data; and calibrating the extrinsic parameters of the target camera according to the second lane line detection data, the second positioning data and a high-precision map.

[0007] In one embodiment, the test data comprises a plurality of target images captured by the target camera and measurement data of a plurality of specified sensors; the first lane line detection data and the first positioning data are extracted from the test data of the vehicle in which the target camera is located by performing lane line detection on each target image and taking each target image and its detection result as the first lane line detection data; and the first positioning data is obtained by performing pose fusion according to the measurement data of the plurality of specified sensors.

[0008] In one embodiment, the first positioning data is obtained by performing pose fusion according to the measurement data of the plurality of specified sensors, comprising: performing pose fusion according to the measurement data of the plurality of specified sensors to obtain initial positioning data; and optimizing the initial positioning data using an optimization tool to obtain the first positioning data.

[0009] In one embodiment, when the specified detection comprises occlusion detection, positioning quality detection and angle distribution detection, the second lane line detection data is obtained by performing specified detection on the first lane line detection data, comprising: performing occlusion detection on the first lane line detection data to obtain first detection results; the first detection results comprise detection results in the first lane line detection data that pass the occlusion detection; performing positioning quality detection on the first detection results to obtain second detection results; the second detection results comprise detection results in the first detection results that pass the positioning quality detection; performing angle distribution detection on the second detection results to obtain third detection results, wherein the third detection results comprise detection results in the second detection results that pass the angle distribution detection; and taking the third detection results as the second lane line detection data.

[0010] In an embodiment, the extrinsic calibration of the target camera according to the second lane line detection data, the second positioning data and the high-precision map comprises: iteratively calculating the extrinsic parameters of the target camera according to the second lane line detection data, the second positioning data and the high-precision map; wherein each calculation process of the extrinsic parameters of the target camera comprises: obtaining a target detection result from the second lane line detection data, and obtaining a target positioning result corresponding to the target detection result from the second positioning data; calculating a current extrinsic parameter according to a historical extrinsic parameter, the target detection result, the target positioning result and the high-precision map, wherein the historical extrinsic parameter refers to an initial extrinsic parameter or an extrinsic parameter calculated in the last time of the target camera, and the current extrinsic parameter refers to an extrinsic parameter calculated in the present time; and when a preset iteration end condition is met, stopping the iteration and obtaining a latest calculated extrinsic parameter of the target camera.

[0011] In an embodiment, the calculation of the current extrinsic parameter according to the historical extrinsic parameter, the target detection result, the target positioning result and the high-precision map comprises: projecting on a target image corresponding to the target detection result according to the historical extrinsic parameter, the target positioning result and the high-precision map; matching the target detection result and the projection result on the target image to obtain a matching result; and calculating the current extrinsic parameter according to the matching result.

[0012] In an embodiment, the calculation of the current extrinsic parameter according to the matching result comprises: constructing a plurality of target error terms according to the matching result, wherein the target error term refers to an error term related to the extrinsic parameter of the target camera and the distance between a point on an image plane and a line is an error; forming a normal equation with the plurality of target error terms, and iteratively solving the normal equation by using a gradient descent method to obtain the current extrinsic parameter.

[0013] In an embodiment, the above method further comprises: when the current extrinsic parameter is obtained by iteratively solving the normal equation by using the gradient descent method, performing singular value decomposition on the normal equation to obtain a plurality of eigenvectors, comparing eigenvalues of the eigenvectors with a preset threshold value to determine a target eigenvector, and stopping updating a parameter corresponding to the target eigenvector.

[0014] In an embodiment, the test data of the vehicle in which the target camera is located is test data collected by the vehicle in which the target camera is located in a sensor calibration test or a daily road test.

[0015] In an embodiment, the above method further comprises: after successfully calibrating the extrinsic parameter of the target camera, obtaining a current extrinsic parameter of the target camera; comparing the extrinsic parameter with specified threshold data to obtain a difference between the extrinsic parameter and the specified threshold data; and when the difference exceeds a preset adjustment range, sending an alarm information to a specified person.

[0016] The present application provides a camera extrinsic calibration device according to a second aspect. In an embodiment, the device comprises:

[0017] extracting a first lane line detection data and a first positioning data from test data of a vehicle where the target camera is located; the first lane line detection data includes a plurality of detection results, and the first positioning data includes a positioning result corresponding to each detection result in the first lane line detection data;

[0018] detecting the first lane line detection data to obtain a second lane line detection data; the second lane line detection data includes a detection result in the first lane line detection data that passes the specified detection; the specified detection includes one or more of an occlusion detection, a positioning quality detection, and an angle distribution detection;

[0019] obtaining a second positioning data, the second positioning data including a positioning result corresponding to each detection result in the second lane line detection data;

[0020] calibrating the target camera according to the second lane line detection data, the second positioning data, and the high-precision map.

[0021] According to a third aspect, the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the steps of the embodiments of any of the above methods.

[0022] According to a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executable on the processor to implement the steps of the embodiments of any of the above methods.

[0023] In the above embodiments of the present application, the camera extrinsic parameters are calibrated using the high-precision map, and the first lane line detection data extracted from the test data of the vehicle where the target camera is located is detected, such as occlusion detection, positioning quality detection, and / or angle distribution detection, which can quickly complete the calibration of the camera extrinsic parameters in any area where the high-precision map exists without the assistance of a checkerboard, a higher-precision sensor (such as a laser radar), or a high-precision ranging device built indoors, and a high-precision calibration effect can be obtained. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 a flowchart of a camera extrinsic parameter calibration method provided by the present application according to one or more embodiments;

[0025] Figure 2 a flowchart of a specified detection of the first lane line detection data provided by the present application according to one or more embodiments;

[0026] Figure 3An occlusion detection schematic diagram provided by the present application according to one or more embodiments;

[0027] Figure 4-1 An angle distribution schematic diagram provided by the present application according to one or more embodiments;

[0028] Figure 4-2 An angle distribution schematic diagram provided by the present application according to another one or more embodiments;

[0029] Figure 5 A matching schematic diagram of detection result and projection result provided by the present application according to one or more embodiments;

[0030] Figure 6 A schematic diagram of point-to-line distance on an image plane provided by the present application according to one or more embodiments;

[0031] Figure 7 A structural block diagram of a camera extrinsic parameter calibration device provided by the present application according to one or more embodiments;

[0032] Figure 8 An internal structure diagram of a computer device provided by the present application according to one or more embodiments. DETAILED DESCRIPTION

[0033] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application. The "calibration" in the following text means calibration of camera extrinsic parameters, unless otherwise specified.

[0034] The present application provides a camera extrinsic parameter calibration method. In one embodiment, the above-mentioned camera extrinsic parameter calibration method includes the steps as shown in Figure 1 The method can be applied to a vehicle-mounted computing device in a vehicle, or a server at a remote end. In the following various embodiments, the execution subject can be either a vehicle-mounted computing device or a server, unless otherwise specified. The server can be implemented by an independent server or a server cluster composed of multiple servers.

[0035] The above-mentioned method will be described below, which includes the following steps:

[0036] S110: Extracting first lane line detection data and first positioning data from test data of a vehicle where the target camera is located.

[0037] In this embodiment, the target camera refers to a camera whose extrinsic parameters are to be calibrated; the test data of the vehicle in which the target camera is located includes multiple target images captured by the target camera during the test of the vehicle (referring to an autonomous vehicle in which the target camera is installed), and measurement data collected by multiple specified sensors, wherein the target image refers to an image containing a lane line, and the specified sensors can include but are not limited to a speedometer, an IMU (Inertial Measurement Unit, inertial sensor), an RTK (Real-time kinematic, real-time dynamic carrier phase difference positioning technology), and the like. The first lane line detection data includes multiple detection results, and the multiple detection results refer to lane lines (which can be coordinates of lane lines in images) extracted from the multiple target images respectively, and a deep convolutional network can be used for extraction.

[0038] S120: performing specified detection on the first lane line detection data to obtain second lane line detection data.

[0039] In this embodiment, the second lane line detection data includes the detection results in the first lane line detection data that pass the specified detection. The specified detection includes one or more of the following: occlusion detection, positioning quality detection, and angle distribution detection.

[0040] By performing specified detection on the first lane line detection data to filter out the second lane line detection data, subsequent calibration operations based on the second lane line detection data can improve the stability and robustness of parameter solving in calibration, and thus improve the calibration accuracy.

[0041] S130: obtaining second positioning data.

[0042] In this embodiment, the second positioning data is filtered out from the first positioning data after the second lane line detection data is obtained, and the second positioning data includes the positioning results corresponding to the detection results in the second lane line detection data.

[0043] S140: calibrating the target camera based on the second lane line detection data, the second positioning data, and a high-precision map.

[0044] In this embodiment, calibrating the camera refers to calculating the displacement and pose of the camera coordinate system to the center coordinate system of the rear axle of the vehicle, including a translation matrix and a rotation matrix.

[0045] The high-precision map refers to a high-precision electronic map in the field of autonomous driving.

[0046] The embodiment can calibrate the camera extrinsic parameter without the aid of a checkerboard, a higher-precision sensor (such as a laser radar), or a high-precision ranging device built indoors, and can obtain high-precision calibration results.

[0047] In addition, the inventors realize that, since the camera extrinsic parameter calibration method provided in the embodiment can calibrate without the aid of a checkerboard, a higher-precision sensor (such as a laser radar), or a high-precision ranging device built indoors, the data source for calibration can not be limited to the test data of the sensor calibration test commonly used in the prior art, but can use the test data collected in the daily road test to calibrate. Therefore, in the embodiment, the test data of the vehicle where the target camera is located can be test data collected when the vehicle where the target camera is located performs sensor calibration testing, or can be test data selected from test data collected when the daily road test is performed.

[0048] The sensor calibration test and the daily road test are described below.

[0049] At present, many autonomous vehicles are "rear-mounted", that is, the vehicle does not have any autonomous driving sensors (such as cameras and laser radars) when it is first off the assembly line, and needs to be modified and assembled with hardware by hardware engineers. As a result, the following important steps are involved:

[0050] 1. Hardware design, assembly and testing

[0051] The hardware engineer first designs a CAD drawing according to the original appearance of the vehicle and the actual needs, including where the camera is specifically assembled on the vehicle, the distance between the camera and the camera, etc. Then, the hardware engineer assembles the hardware according to the designed CAD drawing within the allowable error range, and tests the hardware after all the hardware is assembled. The above-mentioned CAD refers to a drawing software produced by Autodesk Company.

[0052] 2. Sensor calibration and testing (the above-mentioned sensor calibration test refers to this step)

[0053] After the hardware test is passed, the sensor is calibrated to correct the installation error during the hardware assembly. At this time, the safety officer or hardware engineer manually drives the vehicle to the data collection site (such as a specific site with a two-dimensional code) to collect special calibration data, and then the calibration is performed based on the collected data. Generally, the test time is short, such as a few minutes, ten minutes, etc., the type of collected data is single, the data volume is small, and the data is basically used for calibration.

[0054] 3. Formal road test (the daily road test described above refers to this step)

[0055] After the hardware and software parameters are verified by means such as hardware measurement comparison, test software trial operation, etc., the road test can be performed. During the test, positioning data, lane line detection data, etc. are recorded. After the road test is completed, the data is automatically uploaded to the server. Generally, the test time is long, such as several hours or even longer (such as one day), the type of collected data is rich, the data volume is large, and the use of collected data is more extensive.

[0056] In an embodiment, the execution subject is a vehicle-mounted computing device. Since calibration can be performed anywhere with a high-precision map without the need to specially go to a specific site with a two-dimensional code, when the sensor calibration and test steps are performed, test data for calibration can be collected in real time online, and calibration can be performed immediately based on the collected test data. The calibration result can be displayed in real time on the vehicle-mounted computing device. Since the person responsible for calibration (such as an engineer) can easily see the calibration effect, he or she can timely find out whether there is a problem with the calibration result. If he or she is not satisfied with the calibration result, he or she can immediately re-calibrate.

[0057] In an embodiment, the method further includes: after successfully performing the external parameter calibration on the target camera, obtaining the current external parameter of the target camera; comparing the external parameter with the specified threshold data to obtain the difference between the external parameter and the specified threshold data; and when the difference exceeds a preset adjustment range, sending an alarm message to a specified person. The specified threshold can be set according to experience, such as the initial external parameter (a value obtained after a hardware engineer performs CAD modeling), which is not specifically limited herein. This embodiment can timely find out the external parameter with a calibration problem, and then can warn the specified person to troubleshoot the problem, thereby improving the problem solving efficiency.

[0058] In one embodiment, the step S110 of extracting the first lane line detection data and the first positioning data from the test data of the vehicle where the target camera is located comprises: performing lane line detection on each target image, taking each target image and its detection result as the first lane line detection data; and performing pose fusion according to the measurement data of the plurality of specified sensors to obtain the first positioning data. The pose fusion can be performed based on an optimization-based or filtering-based multi-sensor pose fusion algorithm.

[0059] Optionally, the step of performing pose fusion according to the measurement data of the plurality of specified sensors to obtain the first positioning data comprises: performing pose fusion according to the measurement data of the plurality of specified sensors to obtain initial positioning data; and using an optimization tool to optimize the initial positioning data to obtain the first positioning data. By using the optimization tool to optimize the initial positioning data, a higher-precision positioning result can be obtained. The optimization tool can be a full smoother.

[0060] In one embodiment, as known from the foregoing, the specified detection can include one or more detections. If the specified detection includes more than one detection, the detection order of the various detections included in the specified detection can be unrestricted.

[0061] The following takes the example of the specified detection including the occlusion detection, the positioning quality detection, and the angle distribution detection to illustrate the step S120, i.e., as shown in FIG. 12, the step S120 of performing specified detection on the first lane line detection data to obtain second lane line detection data comprises: Figure 2

[0062] S121: performing occlusion detection on the first lane line detection data to obtain a first detection result; the first detection result includes the detection results in the first lane line detection data that pass the occlusion detection.

[0063] The detection result of the lane line can refer to the upper diagram of FIG. 13, Figure 3 Figure 3 The lower diagram of FIG. 13 shows the detection result after the occlusion detection, Figure 3 Figure 3 The arrow in FIG. 13 indicates the detected lane line (the lane line indicated by the arrow is only part of the lane line in the figure); wherein the detection result of the lane line on the image is specifically composed of a plurality of detection points, some of which can be occluded by an obstacle, such as the detection points boxed in the figure. Using this part of the detection points for subsequent calibration will affect the calibration accuracy. Therefore, the embodiment performs occlusion detection on each detection result in the first lane line detection data to eliminate the detection points occluded by the obstacle in each detection result.

[0064] ​​​S122: Perform positioning quality detection on the first detection result to obtain a second detection result; the second detection result includes detection results in the first detection result that pass the positioning quality detection.

[0065] If the full smoother is used to optimize the positioning data in step S110, the covariance estimated by the full smoother can be used to represent the positioning quality of each positioning result in step S120, and the detection result with poor positioning quality can be removed based on the positioning quality.

[0066] S123: Perform angle distribution detection on the second detection result to obtain a third detection result; the third detection result includes detection results in the second detection result that pass the angle distribution detection.

[0067] In the first lane line detection data, there can be detection results with redundant angles, which can cause the nonlinear optimizer to fall into local optimum. Therefore, the angle distribution detection can effectively cause the nonlinear optimizer to fall into local optimum, thereby improving the optimization speed and the calibration accuracy. The above-mentioned angle refers to the included angle between the vehicle and the lane line, such as Figure 4-1 and Figure 4-2 That is, two angles are shown.

[0068] S124: Take the third detection result as the second lane line detection data.

[0069] The embodiment can improve the stability and robustness of solving the calibration parameters, and further improve the calibration accuracy.

[0070] In one embodiment, the step S140 of calibrating the extrinsic parameters of the target camera according to the second lane line detection data, the second positioning data, and the high-precision map includes: iteratively calculating the extrinsic parameters of the target camera according to the second lane line detection data, the second positioning data, and the high-precision map; and stopping the iterative calculation and obtaining the latest calculated extrinsic parameters of the target camera when a preset iterative calculation end condition is met. The iterative calculation end condition can be parameter convergence and / or the number of iterations being greater than a preset threshold.

[0071] The iterative calculation of the extrinsic parameters of the target camera includes: obtaining a target detection result from the second lane line detection data, and obtaining a target positioning result corresponding to the target detection result from the second positioning data; and calculating a current extrinsic parameter according to a historical extrinsic parameter, the target detection result, the target positioning result, and the high-precision map. The target detection result refers to a detection result obtained from the second lane line detection data in the current calculation. The historical extrinsic parameter refers to the initial extrinsic parameter of the target camera (i.e., the initial extrinsic parameter is used as the historical extrinsic parameter in the first calculation of the iterative calculation) or the extrinsic parameter calculated in the last time, and the current extrinsic parameter refers to the extrinsic parameter of the target camera calculated in the current calculation.

[0072] In one embodiment, the current extrinsic parameter is calculated according to the historical extrinsic parameter, the target detection result, the target positioning result and the high-precision map, including: projecting on a target image corresponding to the target detection result according to the historical extrinsic parameter, the target positioning result and the high-precision map; matching the target detection result and the projection result on the target image to obtain a matching result; and calculating the current extrinsic parameter according to the matching result.

[0073] In the projection on the target image corresponding to the target detection result, the corresponding lane on the high-precision map is obtained according to the intrinsic parameter of the target camera, the historical extrinsic parameter and the target positioning result, and the obtained lane is projected onto the target image corresponding to the target detection result (the target detection result is also on the target image). After the projection is completed, the target detection result and the projection result on the target image can be matched to obtain the matching result. Exemplarily, as shown in FIG. 1, ① in the figure is part of the lane in the target detection result, ② is part of the lane in the projection result, and ③ is the connection between the detection result and the projection result, which is used to represent the matching relationship between the detection result and the projection result (the longer the connection, the weaker the matching relationship, and the shorter the connection, the stronger the matching relationship). Figure 5

[0074] Specifically, first, the lane lines on the high-precision map are uniformly sampled (the points of the lane lines in the high-precision map are referred to as sampling points, and the points of the lane lines in the detection result are referred to as detection points); then, the detection point of each lane line in the target detection result is found and its nearest sampling point on the high-precision map is found; since the detection result and the high-precision map are both organized in units of lane lines, the detection points belonging to the same lane line will correspond to the same sampling point on the high-precision map, and based on the above condition, the out-of-range points in each lane line in the target detection data can be removed. The above out-of-range point refers to a point in the target detection result that does not belong to the lane line, and similarly, the in-range point refers to a point that belongs to the lane line.

[0075] In addition, to further remove the out-of-range points, it can be assumed that the number ratio of the in-range points is much larger than that of the out-of-range points, and the matching distance error of the out-of-range points is much larger than that of the in-range points. Based on the above assumption, a histogram can be established to find the distribution limit of the in-range points and the out-of-range points, and the out-of-range points can be removed based on the found distribution limit.

[0076] In one embodiment, after the above matching operation is completed, the current extrinsic parameter can be calculated according to the matching result, including: constructing a plurality of target error terms according to the matching result, the target error term being an error term related to the extrinsic parameter of the target camera and the distance between the point on the image plane and the line being the error; grouping the plurality of target error terms into a normal equation, and using the gradient descent method to iteratively solve the normal equation to obtain the current extrinsic parameter.

[0077] ​Exemplarily, as shown in Figure 6 "rg0", "rg1" and "rg2" in the figure refer to sampling points, and "f0", "f1" and "f2" refer to detection points, and the "line" refers to a line formed between every two adjacent sampling points, such as Figure 6 "rg0" and "rg1" form a line "L1", and "rg1" and "rg2" form a line "L2". The distance between the point on the image plane and the line refers to the distance between the detection point and the line formed by the sampling points, such as the distances between the detection points "f0", "f1" and "f2" and the line are "d0", "d1" and "d2", respectively.

[0078] Exemplarily, the following takes the initial extrinsic parameter of the target camera as the historical extrinsic parameter as an example to explain the process of calculating the current extrinsic parameter according to the matching result.

[0079] The input information during calculation is as follows:

[0080] Initial value: It can be derived from CAD (engineering three-dimensional modeling)

[0081] Lane line detection points converted to the camera normalized coordinate system: niq

[0082] Corresponding lane line sampling points of the high-precision map in the world coordinate system: wp a , wp b

[0083] High-precision positioning result: T w，ra

[0084] The output information after calculation is as follows:

[0085] Current extrinsic parameter: T ra，c

[0086] The measurement model is as follows:

[0087]

[0088] Where r is the one-dimensional error of the point to the line, and the error term (i.e. the error term above) relative to the Jacobian (derivative) of the extrinsic parameter is:

[0089]

[0090] Where,

[0091]

[0092] All error terms form a normal equation, which is solved by using a gradient descent method, such as Newton-Gauss method or LM (Levenberg-Marquarelt) method:

[0093]

[0094] wherein J and r are Jacobian matrix and error vector of all detection points according to measurement model combination, Δξ is the update increment of T ra，c

[0095]

[0096] Further, in one embodiment, when the current external parameter is obtained by iteratively solving the normal equation using the gradient descent method, singular value decomposition is performed on the normal equation to obtain a plurality of eigenvectors; the eigenvalues of each eigenvector are compared with a preset threshold value to determine a target eigenvector, the target eigenvector being an eigenvector in an underdetermined direction among the plurality of eigenvectors; and the updating of the parameter corresponding to the target eigenvector is stopped. The above-mentioned preset threshold value can be obtained through pre-experiment.

[0097] The external parameter of the target camera includes 6 parameters (i.e. three rotation angles and three displacement amounts), and the plurality of eigenvectors correspond one-to-one to the 6 parameters. The target eigenvector is an eigenvector in an underdetermined direction among the plurality of eigenvectors, and the number of target eigenvectors can be one or more; specifically, if the eigenvalue of a certain eigenvector is less than the corresponding preset threshold value, the direction of the eigenvector is considered to be an underdetermined direction (or called a degenerate direction), and the eigenvector is taken as a target eigenvector.

[0098] The embodiment can further improve the calibration stability in the optimization process.

[0099] It should be noted that, as for each step included in the camera external parameter calibration method provided in any one of the above-mentioned embodiments, unless explicitly stated in this document, the execution of these steps does not have strict order restrictions, and these steps can be executed in other orders. Moreover, at least part of these steps can include multiple sub-steps or multiple stages, and these sub-steps or stages do not necessarily have to be executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages does not necessarily have to be sequential, but can be executed in rotation or alternation with at least part of other steps or sub-steps or stages of other steps.

[0100] Based on the same inventive concept, the present application also provides a camera external parameter calibration device. In the embodiment, as shown in Figure 7 the camera external parameter calibration device includes the following modules:

[0101] ​The extraction module 110 is configured to extract first lane line detection data and first positioning data from test data of a vehicle where the target camera is located. The first lane line detection data includes a plurality of detection results, and the first positioning data includes a positioning result corresponding to each detection result in the first lane line detection data.

[0102] The detection module 120 is configured to perform specified detection on the first lane line detection data to obtain second lane line detection data. The second lane line detection data includes a detection result in the first lane line detection data that passes the specified detection. The specified detection includes one or more of an occlusion detection, a positioning quality detection, and an angle distribution detection.

[0103] The acquisition module 130 is configured to acquire second positioning data. The second positioning data includes a positioning result corresponding to each detection result in the second lane line detection data.

[0104] The calibration module 140 is configured to calibrate the target camera according to the second lane line detection data, the second positioning data, and a high-precision map.

[0105] In an embodiment, the test data includes a plurality of target images captured by the target camera and measurement data of a plurality of specified sensors. The extraction module is configured to perform lane line detection on each target image, and take each target image and its detection result as the first lane line detection data. The first positioning data is obtained by performing pose fusion according to the measurement data of the plurality of specified sensors.

[0106] In an embodiment, the extraction module is further configured to perform pose fusion according to the measurement data of the plurality of specified sensors to obtain initial positioning data. The first positioning data is obtained by optimizing the initial positioning data using an optimization tool.

[0107] In an embodiment, when the specified detection includes the occlusion detection, the positioning quality detection, and the angle distribution detection, the detection module is configured to perform occlusion detection on the first lane line detection data to obtain first detection results. The first detection results include detection results in the first lane line detection data that pass the occlusion detection. The detection module is configured to perform positioning quality detection on the first detection results to obtain second detection results. The second detection results include detection results in the first detection results that pass the positioning quality detection. The detection module is configured to perform angle distribution detection on the second detection results to obtain third detection results. The third detection results include detection results in the second detection results that pass the angle distribution detection. The third detection results are taken as the second lane line detection data.

[0108] In an embodiment, the calibration module is configured to iteratively calculate the extrinsic parameters of the target camera according to the second lane line detection data, the second positioning data, and the high-precision map; wherein each iteration of the calculation of the extrinsic parameters of the target camera comprises: obtaining a target detection result from the second lane line detection data, and obtaining a target positioning result corresponding to the target detection result from the second positioning data; calculating a current extrinsic parameter according to a historical extrinsic parameter, the target detection result, the target positioning result, and the high-precision map, wherein the historical extrinsic parameter refers to an initial extrinsic parameter of the target camera or an extrinsic parameter calculated in a previous iteration, and the current extrinsic parameter refers to an extrinsic parameter calculated in the current iteration; and when a preset iteration end condition is met, stopping the iteration and obtaining a latest calculated extrinsic parameter of the target camera.

[0109] In an embodiment, when calculating the current extrinsic parameter according to the historical extrinsic parameter, the target detection result, the target positioning result, and the high-precision map, the calibration module is specifically configured to: project on a target image corresponding to the target detection result according to the historical extrinsic parameter, the target positioning result, and the high-precision map; match the target detection result and the projection result on the target image to obtain a matching result; and calculate the current extrinsic parameter according to the matching result.

[0110] In an embodiment, when calculating the current extrinsic parameter according to the matching result, the calibration module is specifically configured to: construct a plurality of target error terms according to the matching result, wherein each target error term is related to the extrinsic parameter of the target camera and the distance between a point on an image plane and a line is an error; group the plurality of target error terms into a normal equation, and iteratively solve the normal equation using a gradient descent method to obtain the current extrinsic parameter.

[0111] In an embodiment, the calibration module is further configured to construct a plurality of target error terms according to the matching results of the target images, wherein each target error term is related to the extrinsic parameter of the camera and the distance between a point on an image plane and a line is an error; group the plurality of target error terms into a normal equation, and iteratively solve the normal equation using a gradient descent method to obtain the current extrinsic parameter.

[0112] In an embodiment, when iteratively solving the normal equation using the gradient descent method to obtain the current extrinsic parameter, the calibration module is further configured to perform singular value decomposition on the normal equation to obtain a plurality of eigenvectors; compare the eigenvalues of the eigenvectors with a preset threshold to determine a target eigenvector; and stop updating a parameter corresponding to the target eigenvector.

[0113] In an embodiment, the test data of the vehicle in which the target camera is located is test data collected by the vehicle in which the target camera is located in a sensor calibration test or a daily road test.

[0114] In an embodiment, the device further comprises an alarm module. The alarm module is configured to, after successfully calibrating the extrinsic parameters of the target camera, acquire the current extrinsic parameters of the target camera; compare the extrinsic parameters with the specified threshold data, and acquire a difference between the extrinsic parameters and the specified threshold data; and when the difference exceeds a preset adjustment range, send an alarm message to a specified person.

[0115] The specific limitations of the camera extrinsic parameter calibration device can refer to the limitations of the camera extrinsic parameter calibration method described above, and will not be repeated here. Each module in the camera extrinsic parameter calibration device can be realized by software, hardware, and combinations thereof, in whole or in part. Each module described above can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0116] In an embodiment, a computer device is provided, and an internal structure diagram of the computer device can be as shown in Figure 8 .

[0117] The computer device comprises a processor, a memory, a network interface, and a database connected by a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is configured to store high-precision maps, test data of a vehicle where the target camera is located, and other data. The specific data stored can also refer to the limitations in the method embodiments described above. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor to implement a camera extrinsic parameter calibration method.

[0118] Those skilled in the art can understand that Figure 8 the structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0119] The embodiment also provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the camera extrinsic parameter calibration method provided in any of the embodiments described above.

[0120] In one embodiment, a computer readable storage medium is provided, having stored thereon a computer program, which, when executed by a processor, implements the steps of the camera extrinsic parameter calibration method provided in any of the above embodiments.

[0121] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned method embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus), direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0122] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not contradict, they should be considered within the scope of the present disclosure.

[0123] The above embodiments only express several implementation manners of the present application, and the description is specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for those of ordinary skill in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the patent of the present application should be subject to the appended claims.

Claims

1. A method for calibrating camera extrinsic parameters, characterized in that, The method includes: First lane detection data and first positioning data are extracted from the road test data of the vehicle where the target camera is located; the first lane detection data includes the detection results of multiple lane lines, and the detection results of the lane lines are specifically composed of multiple detection points in the image; the first positioning data includes the positioning results corresponding to each detection result in the first lane detection data; The first lane detection data is subjected to specified detection to obtain second lane detection data. The second lane detection data includes the detection results from the first lane detection data that have passed the specified detection. The specified detection includes obstruction detection, positioning quality detection, and angle distribution detection. Obstruction detection is used to remove detection points in each detection result of the first lane detection data that are obstructed by obstacles. Positioning quality detection is used to remove detection results in the first lane detection data that do not meet the positioning quality requirements. Angle distribution detection is used to remove detection results in the first lane detection data that have redundant angles. The angle refers to the angle formed between the vehicle and the lane line. Acquire second positioning data, which includes positioning results corresponding to each detection result in the second lane line detection data; The target camera is calibrated using the second lane detection data, the second positioning data, and the high-precision map.

2. The method as described in claim 1, characterized in that, The road test data includes multiple target images captured by the target camera, as well as measurement data from multiple designated sensors; The extraction of the first lane detection data and the first positioning data from the road test data of the vehicle where the target camera is located includes: Lane line detection is performed on each target image, and each target image and its detection results are used as the first lane line detection data; Pose fusion is performed based on the measurement data from the multiple designated sensors to obtain the first positioning data.

3. The method as described in claim 2, characterized in that, The step of performing pose fusion based on measurement data from the multiple designated sensors to obtain first positioning data includes: Pose fusion is performed based on the measurement data from the multiple specified sensors to obtain initial positioning data; The initial positioning data is optimized using an optimization tool to obtain the first positioning data.

4. The method as described in claim 1, characterized in that, When the specified detection includes occlusion detection, positioning quality detection, and angle distribution detection, the step of performing specified detection on the first lane line detection data to obtain the second lane line detection data includes: Obstruction detection is performed on the first lane line detection data to obtain a first detection result; the first detection result includes the detection result of passing through the obstruction in the first lane line detection data; The first detection result is subjected to positioning quality detection to obtain a second detection result; the second detection result includes the detection result that passed the positioning quality detection in the first detection result; Angle distribution detection is performed on the second detection result to obtain a third detection result, wherein the third detection result includes the detection result obtained by the angle distribution detection in the second detection result; The third detection result is used as the second lane line detection data.

5. The method as described in claim 1, characterized in that, The step of calibrating the extrinsic parameters of the target camera based on the second lane detection data, the second positioning data, and the high-precision map includes: Based on the second lane detection data, the second positioning data, and the high-precision map, the extrinsic parameters of the target camera are iteratively calculated; wherein, any calculation process of the extrinsic parameters of the target camera includes: obtaining the target detection result from the second lane detection data, obtaining the target positioning result corresponding to the target detection result from the second positioning data; calculating the current extrinsic parameters based on historical extrinsic parameters, the target detection result, the target positioning result, and the high-precision map, wherein the historical extrinsic parameters refer to the initial extrinsic parameters of the target camera or the extrinsic parameters calculated in the last time, and the current extrinsic parameters refer to the extrinsic parameters of the target camera calculated in this instance; When the preset iterative calculation termination condition is met, the iterative calculation is stopped and the latest calculated extrinsic parameters of the target camera are obtained.

6. The method as described in claim 5, characterized in that, The calculation of current extrinsic parameters based on historical extrinsic parameters, target detection results, target localization results, and the high-precision map includes: Based on historical external parameters, the target localization result, and the high-precision map, project onto the target image corresponding to the target detection result; The target detection result and the projection result on the target image are matched to obtain a matching result; The current extrinsic parameters are calculated based on the matching results.

7. The method as described in claim 6, characterized in that, The step of calculating the current extrinsic parameters based on the matching result includes: Multiple target error terms are constructed based on the matching results. Each target error term is an error term that is related to the extrinsic parameters of the target camera and the distance from a point to a line on the image plane is the error. The multiple target error terms are combined into a normal equation, and the gradient descent method is used to iteratively solve the normal equation to obtain the current extrinsic parameters.

8. The method as described in claim 7, characterized in that, The method further includes: when using the gradient descent method to iteratively solve the normal equation to obtain the current extrinsic parameters, performing singular value decomposition on the normal equation to obtain multiple feature vectors, comparing the feature values ​​of each feature vector with a preset threshold to determine the target feature vector, wherein the target feature vector refers to the feature vector in the underdetermined direction among the multiple feature vectors; and stopping the updating of the parameters corresponding to the target feature vector.

9. The method as described in claim 1, characterized in that, The test data of the vehicle where the target camera is located is the test data collected by the vehicle during sensor calibration testing or routine road testing.

10. The method as described in claim 1, characterized in that, The method further includes: After successfully calibrating the extrinsic parameters of the target camera, obtain the current extrinsic parameters of the target camera; The extrinsic data is compared with the specified threshold data to obtain the difference between the extrinsic data and the specified threshold data. When the difference exceeds the preset adjustment range, an alarm message is sent to the designated personnel.

11. A camera extrinsic parameter calibration device, characterized in that, The device includes: The extraction module is used to extract first lane detection data and first positioning data from the road test data of the vehicle where the target camera is located; the first lane detection data includes the detection results of multiple lane lines, and the detection results of the lane lines are specifically composed of multiple detection points on the image; the first positioning data includes the positioning results corresponding to each detection result in the first lane detection data; The detection module is used to perform specified detection on the first lane line detection data and obtain the second lane line detection data; The second lane detection data includes the detection results from the first lane detection data that have passed the specified detection; the specified detection includes obstruction detection, positioning quality detection, and angle distribution detection; the obstruction detection is used to remove detection points obstructed by obstacles in each detection result of the first lane detection data; the positioning quality detection is used to remove detection results in the first lane detection data whose positioning quality does not meet the requirements; the angle distribution detection is used to remove detection results in the first lane detection data where the angle is redundant; the angle refers to the angle formed between the vehicle and the lane line; The acquisition module is used to acquire second positioning data, which includes positioning results corresponding to each detection result in the second lane line detection data. The calibration module is used to calibrate the extrinsic parameters of the target camera based on the second lane detection data, the second positioning data, and the high-precision map.

12. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

Images