Camera calibration method, device, electronic device and readable storage medium

Abstract

The present application relates to the field of image processing technology, and provides a camera calibration method, device, electronic device and readable storage medium. The method comprises: obtaining light source images of at least two moments captured by a target camera, and obtaining the light source pixel position at each moment based on the light source image; wherein the target camera is installed on a target vehicle, and the light source image includes at least one light source; according to the light source pixel position, determining the target light source pixel position at each moment corresponding to the same target light source; obtaining the vehicle posture transformation relationship of the target vehicle at adjacent moments through a vehicle sensor; obtaining the target error according to the target light source pixel position and the vehicle posture transformation relationship; the target error is obtained based on the reprojection error and the vehicle posture transformation error; according to the target error, determining the calibration external parameters of the target camera. The present application solves the problem of low camera calibration accuracy in the prior art.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a camera calibration method, device, electronic device and readable storage medium. Background Art

[0002] Autonomous driving needs to capture images through cameras to perceive the surrounding environment, such as monocular distance measurement, pedestrian estimation, 3D (3D) lane line detection, etc. In order to ensure the accuracy of the perception results, the accuracy of the camera's external parameters is required. However, during the driving process of the vehicle, the camera may have posture deviations due to bumps and other reasons, making the external parameters inaccurate, so the camera needs to be calibrated online to correct the external parameters. Currently, the main methods used for calibration are feature points, vanishing points, and high-precision map matching.

[0003] The problem with the existing technology is that calibration by extracting and matching feature points is easily affected by factors such as noise, mismatched point pairs, and camera calibration errors; calibration by obtaining vanishing points requires multiple pairs of parallel lines of the XYZ axis under the vehicle coordinates, which is harsh and difficult to promote; calibration by matching the camera with a high-precision map requires elevation information, and the current high-precision map has a freshness problem, which greatly affects the calibration accuracy. It can be seen that there is an urgent need for a camera calibration method that is easy to detect, easy to promote, and has high accuracy. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a camera calibration method, device, electronic device and readable storage medium to solve the problem of low camera calibration accuracy in the prior art.

[0005] According to a first aspect of an embodiment of the present application, a camera calibration method is provided, comprising:

[0006] Acquire light source images at at least two moments captured by a target camera, and obtain the light source pixel position at each moment based on the light source images; wherein the target camera is installed on a target vehicle, and the light source images include at least one light source;

[0007] According to the light source pixel position, determine the target light source pixel position corresponding to the same target light source at each moment;

[0008] Obtain the vehicle posture transformation relationship of the target vehicle at adjacent moments through vehicle sensors;

[0009] The target error is obtained based on the target light source pixel position and the vehicle posture transformation relationship; the target error is obtained based on the reprojection error and the vehicle posture transformation error; the reprojection error is used to characterize the error between the projection and the target light source pixel position, and the projection is the estimated three-dimensional position of the target light source, which is projected according to the estimated vehicle posture and the estimated external parameters of the target camera; the vehicle posture transformation error is used to characterize the error between the change of the estimated vehicle posture at adjacent moments and the vehicle posture transformation relationship;

[0010] According to the target error, the calibration extrinsic parameters of the target camera are determined.

[0011] A second aspect of an embodiment of the present application provides a camera calibration device, including:

[0012] A first detection module is used to obtain light source images at at least two moments captured by a target camera, and obtain the light source pixel position at each moment based on the light source images; wherein the target camera is installed on a target vehicle, and the light source image includes at least one light source;

[0013] A tracking module, used to determine the target light source pixel position at each moment corresponding to the same target light source according to the light source pixel position;

[0014] The second detection module is used to obtain the vehicle posture transformation relationship of the target vehicle at adjacent moments through the vehicle sensor;

[0015] The error determination module is used to obtain the target error according to the target light source pixel position and the vehicle posture transformation relationship; the target error is obtained based on the reprojection error and the vehicle posture transformation error; the reprojection error is used to characterize the error between the projection and the target light source pixel position, and the projection is the estimated three-dimensional position of the light source of the target light source, which is projected according to the estimated vehicle posture and the estimated external parameters of the target camera; the vehicle posture transformation error is used to characterize the error between the change of the estimated vehicle posture at adjacent moments and the vehicle posture transformation relationship;

[0016] The calibration module is used to determine the calibration extrinsic parameters of the target camera according to the target error.

[0017] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0018] According to a fourth aspect of an embodiment of the present application, a readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0019] Compared with the prior art, the embodiments of the present application have the following beneficial effects: by acquiring light source images at at least two moments captured by a target camera, and obtaining the light source pixel position at each moment based on the light source image, the system can more easily complete the detection of the light source pixel position, and is conducive to promotion and use; by determining the target light source pixel position at each moment corresponding to the same target light source according to the light source pixel position, the target light source is tracked; through the vehicle sensor, the vehicle posture transformation relationship of the target vehicle at adjacent moments is obtained, providing data support for subsequent target error calculation; by obtaining the target error according to the target light source pixel position and the vehicle posture transformation relationship, the target error is obtained based on the reprojection error and the vehicle posture transformation error, and the calibration external parameters of the target camera are determined according to the target error. Since the target error is obtained based on the reprojection error and the vehicle posture transformation error, the calibration accuracy is improved, thereby realizing easy-to-detect, easy-to-promote and high-precision camera calibration, and solving the problem of low camera calibration accuracy in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 It is a flowchart of a camera calibration method provided in an embodiment of the present application;

[0022] Figure 2A is a schematic diagram of the pixel position of the light source at the first moment;

[0023] Figure 2B is a schematic diagram of the pixel position of the light source at the second moment;

[0024] Figure 2C is a schematic diagram of a light source pixel position at a second moment predicted based on a light source pixel position at a first moment;

[0025] Figure 3 is a schematic diagram of the reprojection error;

[0026] Figure 4 is a structural schematic diagram of a camera calibration device provided in an embodiment of the present application;

[0027] Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0029] The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0030] In addition, it should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "includes..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0031] A camera calibration method and device according to an embodiment of the present application will be described in detail below with reference to the accompanying drawings.

[0032] Figure 1 It is a flowchart of a camera calibration method provided in an embodiment of the present application. Figure 1 The camera method can be executed by the target vehicle. Figure 1 As shown, the camera calibration method includes:

[0033] S101, obtaining light source images captured by a target camera at at least two moments, and obtaining the light source pixel position at each moment based on the light source images.

[0034] The target camera is installed on the target vehicle, and the light source image includes at least one light source.

[0035] Specifically, the light source can be an artificial light source such as an incandescent lamp, a fluorescent lamp, a light-emitting diode (LED), or a natural light source such as the sun. In the actual calibration process, if the light source itself is moving, deforming, flickering, etc., it may affect the accuracy of subsequent tracking of the same target light source. Therefore, in order to reduce the impact of the light source itself, a static light source with uniform and stable brightness can be selected for collection, such as street lamps, ground lamps, etc.

[0036] During the acquisition, the frame rate and duration of the acquisition can be set to obtain light source images at multiple moments. For example, if the target camera is set to acquire images at a frame rate of 30 frames per second for 10 seconds, 300 frames can be acquired, that is, light source images at 300 moments. Generally speaking, the more moments are acquired, the more light source pixel positions can be referenced, and the higher the calibration accuracy, but the corresponding amount of calculation will be greater, so it is necessary to set the number of moments of light source images to be acquired according to the performance of the execution device.

[0037] For a light source image at a certain moment, all light source pixel positions in the light source image can be detected based on the brightness, color, contour and other information of the light source; by detecting the light source images at all moments, the light source pixel position at each moment can be obtained. It should be noted that a light source image may include one or more light sources, and correspondingly, a light source image includes one or more light source pixel positions.

[0038] Since light sources are ubiquitous in life, light source images are easier to obtain and easier to promote and use than the vanishing point method with strict conditions. Also, because the brightness, color, contour and other characteristics of the light source are obvious, the pixel position of the light source in the light source image is easier to detect than markers such as the checkerboard used for feature point calibration method.

[0039] S102, determining the target light source pixel position corresponding to the same target light source at each moment according to the light source pixel position.

[0040] Specifically, the pixel positions of the light source at two moments can be taken, and matching can be performed based on the pixel positions of the light source at the two moments, and the target light source pixel positions of the same target light source at the two moments can be determined based on the matching results. The two moments can be adjacent moments, for example, the pixel positions of the light source at the 5th frame and the 6th frame are matched; if the frame rate collected in S101 is large, the time interval between adjacent moments is very short, and the pixel difference of the light source pixel position will be very small, in order to save computing power, the pixel positions of the light source at non-adjacent moments can also be taken for matching, for example, if it is set to be taken every 10 frames, then the pixel positions of the light source at the 5th frame and the 15th frame can be taken for matching.

[0041] When matching, the Hungarian Algorithm can be used to match the light source pixel positions at two moments, or the light source pixel position at a certain moment can be predicted by Kalman Filter, and then the predicted light source pixel position at that moment is matched with the actual light source pixel position at that moment. Figure 2A , Figure 2B and Figure 2C These two matching methods are explained.

[0042] like Figure 2A and Figure 2B As shown, the light source pixel positions at the first moment are A1, A2, A3, and A4, and the light source pixel positions at the second moment are B1, B2, B3, B4, and B5.

[0043] By matching the light source pixel positions at the first moment and the second moment through the Hungarian algorithm, it can be obtained that: A1 and B1 correspond to the same target light source, and A1 and B1 are the target light source pixel positions corresponding to the target light source at the first moment and the second moment respectively; similarly, A2 and B2 correspond to the same target light source, A3 and B4 correspond to the same target light source, and A4 and B5 correspond to the same target light source, and the target light source pixel positions corresponding to each target light source can be determined.

[0044] Alternatively, based on the light source pixel position at the first moment, the light source pixel position that may appear at the second moment is predicted by Kalman filtering to obtain Figure 2C , that is, the light source corresponding to A1 appears at A1', the light source corresponding to A2 appears at A2', and so on; Figure 2C The light source pixel position in Figure 2B By matching the light source pixel positions in , we can get: A1' and B1 correspond to the same target light source. Since A1' corresponds to A1, A1 and B1 correspond to the same target light source. A1 and B1 are the target light source pixel positions corresponding to the target light source at the first moment and the second moment respectively; similarly, A2 and B2 correspond to the same target light source, A3 and B4 correspond to the same target light source, A4 and B5 correspond to the same target light source, and the target light source pixel position corresponding to each target light source can be determined.

[0045] It should be noted that the reason why B3 failed to match the light source pixel position corresponding to the same target light source at the first moment may be that the light source corresponding to B3 is moving, flickering, etc., or the interval between the first moment and the second moment is too long, resulting in the failure to collect the light source pixel position of the light source at the first moment.

[0046] In addition to the above matching methods, optical flow methods can also be used to track the movement of light source pixel positions at adjacent moments, so as to determine the target light source pixel position at each moment corresponding to the same target light source.

[0047] S103, obtaining the vehicle posture transformation relationship of the target vehicle at adjacent moments through the vehicle sensor.

[0048] Specifically, vehicle sensors may include inertial measurement units (IMU), wheel speed sensors, global navigation satellite systems (GNSS), radars, etc., which measure vehicle speed, wheel speed, acceleration, angular velocity, three-dimensional coordinates and other data. Based on these data, the vehicle posture transformation relationship is calculated to provide data support for subsequent target error calculations.

[0049] S104, obtaining a target error according to the target light source pixel position and the vehicle posture transformation relationship.

[0050] The target error is obtained based on the reprojection error and the vehicle pose transformation error.

[0051] The reprojection error is used to characterize the error between the projection and the pixel position of the target light source. The projection is the estimated three-dimensional position of the target light source according to the estimated vehicle posture and the estimated external parameters of the target camera.

[0052] The vehicle posture transformation error is used to characterize the error between the estimated vehicle posture change at adjacent moments and the vehicle posture transformation relationship.

[0053] Specifically, the reprojection error calculates the error between the projection position of a point in the world coordinate system on the imaging plane and the actual position of the point in the imaging plane. The external parameters are used to convert the world coordinate system to the camera coordinate system, and the internal parameters are used to convert the camera coordinate system to the imaging plane coordinate system. Since the internal parameters are determined when the camera leaves the factory, as long as the reprojection error is determined, the external parameters can be reversed by combining the internal parameters.

[0054] Figure 3 is a schematic diagram of the reprojection error, such as Figure 3 As shown, at a certain moment, point P is the estimated three-dimensional position of the target light source in the three-dimensional space, the two-dimensional plane is the imaging plane, point P1 is the target light source pixel position of the target light source at that moment obtained in S102, and point P2 is the projection of point P onto the imaging plane. It can be seen that the calculated projection P2 does not coincide with the actually collected target light source pixel position P1, and the distance between P1 and P2 is the reprojection error for the target light source at that moment.

[0055] In the process of projecting point P onto the imaging plane, the position and posture of the target camera are needed. Since the target camera is installed on the target vehicle, the position and posture of the target camera are closely related to the position and posture of the target vehicle, so it is necessary to project according to the estimated vehicle position and the estimated external parameters of the target camera.

[0056] In order to ensure that the estimated external parameters are accurate, the estimated vehicle posture needs to be accurate, so the vehicle posture transformation error needs to be calculated. For example, S103 obtains the vehicle posture transformation relationship between the first moment and the second moment as (Δx, Δy, Δz), the estimated vehicle posture at the first moment is (x1, y1, z1), and the estimated vehicle posture at the second moment is (x2, y2, z2), then the difference between (Δx, Δy, Δz) and (x1-x2, y1-y2, z1-z2) can be determined as the vehicle posture transformation error between the first moment and the second moment.

[0057] Based on the reprojection error and the vehicle posture transformation error, the target error is obtained. It should be noted that since there may be multiple target light sources and multiple moments are collected in the above steps, when calculating the target error, it is necessary to comprehensively consider the reprojection error of each target light source at all moments and the vehicle posture transformation error at each adjacent moment.

[0058] S105, determining the calibration extrinsic parameters of the target camera according to the target error.

[0059] Specifically, the parameter values ​​in the target error are changed to optimize the target error until the target error is small enough. At this time, the accuracy of the estimated extrinsic parameter is high and it is determined as the calibration extrinsic parameter.

[0060] According to the technical solution provided in the embodiment of the present application, by acquiring the light source images of at least two moments captured by the target camera and obtaining the light source pixel position at each moment based on the light source image, the system can more easily complete the detection of the light source pixel position and is conducive to promotion and use; by determining the target light source pixel position at each moment corresponding to the same target light source according to the light source pixel position, the target light source is tracked; through the vehicle sensor, the vehicle posture transformation relationship of the target vehicle at adjacent moments is obtained to provide data support for subsequent target error calculation; by obtaining the target error according to the target light source pixel position and the vehicle posture transformation relationship, the calibration external parameters of the target camera are determined according to the target error, thereby improving the calibration accuracy of the camera, thereby realizing easy-to-detect, easy-to-promote and high-precision camera calibration.

[0061] In some embodiments, obtaining the pixel position of the light source at each moment based on the light source image includes:

[0062] Acquire brightness information in the light source image, and determine, based on the brightness information, a position where the brightness is greater than a preset brightness as a light source pixel position; or,

[0063] Acquire color information in the light source image, and determine the position with the preset light source color as the light source pixel position according to the color information; or,

[0064] The light source outline or the light source center in the light source image is obtained, and the light source pixel position is determined according to the light source outline or the light source center.

[0065] Specifically, according to the brightness detection, the brightness information in the light source image can be obtained, and the position where the brightness is greater than the preset brightness is determined as the light source pixel position. The preset brightness can be set to different values ​​according to the color mode of the collected light source image. For example, in the Lab color space, the brightness can be between 0 and 100, 0 is pure black, and 100 is pure white. The preset brightness can be set to 80, and the position where the brightness is greater than 80 in the light source image is determined as the light source pixel position.

[0066] By detecting the color, the color information in the light source image can be obtained, and the position with the preset light source color can be determined as the light source pixel position. Some light sources have their own unique colors. For example, the light of incandescent lamps is usually yellowish, while the light of fluorescent lamps is bluish-green. In this case, the preset light source color can be set to yellow or bluish-green, and the position with the yellow or bluish-green color in the light source image can be determined as the light source pixel position.

[0067] By detecting the light source contour or the light source center, the light source contour or the light source center in the light source image can be obtained, and the point on the light source contour or the light source center can be used to determine the light source pixel position. For example, a rectangular frame can be first obtained through a target detection framework such as the YOLO algorithm (You Only Look Once), and then the edge contour can be extracted in the area; or the type of light source and the edge contour of the light source can be extracted through a semantic segmentation framework such as UNET (U-shaped network); or the edge contour of the light source can be directly detected through a contour point detection model; or the light source center can be detected through a key point detection model.

[0068] Of course, any two or three of the brightness, color, light source profile or light source center may also be combined to jointly determine the light source pixel position.

[0069] According to the technical solution provided in the embodiment of the present application, by detecting the brightness, color, light source outline or light source center, the position that meets the conditions is identified and the light source pixel position is obtained, thereby realizing easy-to-detect and easy-to-generalize light source pixel position detection.

[0070] In some embodiments, the vehicle sensor includes at least one of an inertial measurement unit IMU, a global navigation satellite system GNSS;

[0071] Through the vehicle sensors, the vehicle posture transformation relationship of the target vehicle at adjacent moments is obtained, including:

[0072] The angular velocity and acceleration of the target vehicle at adjacent moments are measured by IMU, and the vehicle posture transformation relationship is determined based on the angular velocity and acceleration; or,

[0073] The three-dimensional coordinates and speed of the target vehicle are measured by GNSS, and the vehicle posture transformation relationship is determined based on the three-dimensional coordinates and speed.

[0074] Specifically, the IMU contains three single-axis accelerometers and three single-axis gyroscopes. The accelerometers can detect the independent three-axis acceleration signals of the target vehicle, and the gyroscopes can detect the angular velocity signals of the target vehicle. By integrating and solving the angular velocity and acceleration at adjacent moments, the vehicle posture transformation relationship can be obtained.

[0075] GNSS can provide the three-dimensional coordinates and speed of the target vehicle. Based on the three-dimensional coordinates and speed at adjacent moments, the vehicle posture transformation relationship can be determined.

[0076] Of course, vehicle sensors can also include wheel speed sensors, radars, etc. When determining the vehicle posture transformation relationship, the data of multiple sensors can be combined. For example, when integrating and solving the angular velocity and acceleration obtained by the IMU, the three-dimensional coordinates provided by the GNSS and the speed provided by the wheel speed sensor can be used as constraints, and combined with the posture changes of other sensors to jointly determine the vehicle posture transformation relationship, thereby further improving the accuracy of the vehicle posture change relationship.

[0077] According to the technical solution provided in the embodiment of the present application, relevant data of the target vehicle is measured by IMU or GNSS, and the vehicle posture transformation relationship is solved to provide data support for subsequent target error calculation.

[0078] In some embodiments, the target error is obtained according to the target light source pixel position and the vehicle posture transformation relationship, including:

[0079] According to the target light source pixel position and vehicle posture transformation relationship, the target error is calculated by the following formula:

[0080]

[0081] Where n is the number of acquisition moments, and m is the number of target light sources;

[0082] is the target light source pixel position of the jth target light source at the i-th moment;

[0083] K is the internal parameter of the target camera;

[0084] To estimate external parameters;

[0085] P j is the estimated three-dimensional position of the j-th target light source;

[0086] Pose i is the estimated vehicle pose of the target vehicle at the i-th moment;

[0087] Pose i-1 is the estimated vehicle pose of the target vehicle at the i-1th moment;

[0088] T i-1，i is the vehicle posture transformation relationship from the i-1th moment to the i-th moment;

[0089] is the reprojection error;

[0090] ||T i-1，i -Pose i *Pose i-1 -1 || is the vehicle posture transformation error.

[0091] Specifically, the intrinsic parameter K is a known parameter of the target camera and is a constant matrix.

[0092] is the projection of the jth target light source on the imaging plane at the i-th moment, that is, the estimated three-dimensional position of the target light source is projected according to the estimated vehicle posture and the estimated external parameters, and the pixel position of the target light source is The difference between this and the projection This is the reprojection error.

[0093] Pose i *Pose i-1 -1 is the change of the estimated vehicle posture from the i-1th moment to the i-th moment, and its relationship with the vehicle posture transformation T i-1，i The difference between ||T i-1，i -Pose i *Pose i-1 -1 || is the vehicle posture transformation error.

[0094] The target error is obtained by summing the reprojection error and vehicle pose transformation error of all target light sources at all times.

[0095] According to the technical solution provided in the embodiment of the present application, by providing a calculation formula for the target error, the calculation accuracy of the target error is improved, which facilitates the subsequent optimization of related parameters to obtain high-precision calibration external parameters.

[0096] In some embodiments, determining a calibration extrinsic parameter of a target camera according to a target error includes:

[0097] Optimize the parameter values ​​of the estimated 3D position of the light source, the estimated vehicle posture, and the estimated external parameters, and calculate the optimized target error corresponding to the parameter values;

[0098] When the optimized target error is less than the preset error, the estimated extrinsic parameter corresponding to the target error is determined as the calibration extrinsic parameter of the target camera.

[0099] Specifically, since the estimated three-dimensional position of the light source cannot completely coincide with the actual light source position, and the estimated vehicle posture cannot completely coincide with the actual vehicle posture, the target error obviously cannot be zero, so the preset error must be greater than zero. The preset error can be set according to the working requirements of the target camera. If the accuracy requirement of the target camera is low, the preset error can be set larger, such as 10, 15, 20, etc.; if the accuracy requirement of the target camera is high, the preset error can be set smaller, such as 1, 2, 3, etc.

[0100] By continuously changing the parameter values ​​of the estimated three-dimensional position of the light source, the estimated vehicle posture and the estimated external parameters, when the optimized target error is less than the preset error, the estimated external parameters corresponding to the target error can be determined as the calibration external parameters of the target camera.

[0101] According to the technical solution provided in the embodiment of the present application, by optimizing the parameter values ​​of the estimated three-dimensional position of the light source, the estimated vehicle posture and the estimated external parameters, the estimated external parameters with a target error less than the preset error are determined as the calibration external parameters, thereby achieving high-precision camera calibration.

[0102] In some embodiments, optimizing the parameter values ​​of the estimated three-dimensional position of the light source, the estimated vehicle posture, and the estimated external parameters, and calculating the optimized target error corresponding to the parameter values, includes:

[0103] Using the General Graphic Optimization (G2O) algorithm, the estimated three-dimensional position of the light source, the estimated vehicle posture and the estimated extrinsic parameters are set as vertices to be optimized, the target light source pixel position and the vehicle posture transformation relationship are set as constraints, and the optimized target error is calculated.

[0104] Specifically, in the use of G2O, there are mainly three types of data: vertices, edges and solvers. Vertices are variables to be optimized, edges are constraints between vertices, and solvers are used to solve linear equations.

[0105] In an embodiment of the present application, the variables to be optimized are the estimated three-dimensional position of the light source, the estimated vehicle posture and the estimated external parameters, so these three parameters are set as vertices, the target light source pixel position and the vehicle posture transformation relationship are set as constraints, and a suitable solver is selected to optimize the target error, continuously reducing the target error until the target error is less than the preset error.

[0106] According to the technical solution provided in the embodiment of the present application, by using G2O to optimize the three-dimensional position of the light source, the estimated vehicle posture and the estimated external parameters, the target error is continuously reduced until the target error is less than the preset error, and finally the target error that meets the user's requirements and the estimated external parameters corresponding to the target error are obtained, so that the system can effectively improve the accuracy of camera calibration through the external parameters.

[0107] In some embodiments, before calculating the optimized target error, the method further includes:

[0108] Check the pixel position of the target light source through the kernel function;

[0109] Remove the incorrect target light pixel position from the constraints.

[0110] Specifically, if there is an error in the target light source pixel position, for example, Figure 2A and Figure 2B When matching the light source pixel position in , A3 and B3, A4 and B4 are mistakenly matched as the same target light source and the target light source pixel position is obtained. If the wrong target light source pixel position is used as a constraint and optimized according to it, the accuracy of the calibration extrinsic parameter will be affected. Therefore, it is necessary to remove the wrong target light source pixel position from the constraint through the kernel function to ensure that the optimized calibration extrinsic parameter is accurate and usable.

[0111] According to the technical solution provided in the embodiment of the present application, by introducing a kernel function, the erroneous pixel position of the target light source is eliminated from the constraint conditions, ensuring that the calibration extrinsic parameters obtained by subsequent optimization are accurate and usable.

[0112] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

[0113] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0114] Figure 4Schematic diagram of a camera calibration device provided in an embodiment of the present application. Figure 4 As shown, the camera calibration device includes:

[0115] The first detection module 401 is used to obtain light source images captured by a target camera at at least two moments, and obtain the light source pixel position at each moment based on the light source image; wherein the target camera is installed on a target vehicle, and the light source image includes at least one light source;

[0116] Tracking module 402, used to determine the target light source pixel position at each moment corresponding to the same target light source according to the light source pixel position;

[0117] The second detection module 403 is used to obtain the vehicle posture transformation relationship of the target vehicle at adjacent moments through the vehicle sensor;

[0118] The error determination module 404 is used to obtain a target error according to the target light source pixel position and the vehicle posture transformation relationship; the target error is obtained based on the reprojection error and the vehicle posture transformation error; the reprojection error is used to characterize the error between the projection and the target light source pixel position, and the projection is the estimated three-dimensional position of the light source of the target light source, which is projected according to the estimated vehicle posture and the estimated external parameters of the target camera; the vehicle posture transformation error is used to characterize the error between the change of the estimated vehicle posture at adjacent moments and the vehicle posture transformation relationship;

[0119] The calibration module 405 is used to determine the calibration extrinsic parameters of the target camera according to the target error.

[0120] According to the technical solution provided in the embodiment of the present application, the first detection module 401 obtains the light source images of at least two moments collected by the target camera, and obtains the light source pixel position at each moment based on the light source image, which makes it easier to complete the detection and is conducive to promotion and use; the tracking module 402 determines the target light source pixel position at each moment corresponding to the same target light source according to the light source pixel position, thereby realizing the tracking of the target light source; the second detection module 403 obtains the vehicle posture transformation relationship of the target vehicle at adjacent moments through the vehicle sensor, thereby providing data support for the subsequent target error calculation; the error determination module 404 obtains the target error according to the target light source pixel position and the vehicle posture transformation relationship; the calibration module 405 determines the calibration external parameters of the target camera according to the target error, thereby realizing easy-to-detect, easy-to-promote and high-precision camera calibration.

[0121] In some embodiments, the first detection module 401 is specifically configured to: obtain the brightness information in the light source image, and determine the light source pixel positions as the positions where the brightness is greater than the preset brightness according to the brightness information; or, obtain the color information in the light source image, and determine the light source pixel positions as the positions where the color is the preset light source color according to the color information; or, obtain the light source contour or the light source center in the light source image, and determine the light source pixel positions according to the light source contour or the light source center.

[0122] In some embodiments, the vehicle sensor includes at least one of an inertial measurement unit (IMU) and a global navigation satellite system (GNSS); the second detection module 403 is specifically configured to: measure the angular velocity and acceleration of the target vehicle at adjacent moments through the IMU, and determine the vehicle pose transformation relationship according to the angular velocity and acceleration; or, measure the three-dimensional coordinates and speed of the target vehicle through the GNSS, and determine the vehicle pose transformation relationship according to the three-dimensional coordinates and speed.

[0123] In some embodiments, the error determination module 404 is specifically configured to: calculate the target error according to the target light source pixel positions and the vehicle pose transformation relationship through the following formula:

[0124]

[0125] where n is the number of moments for acquisition, and m is the number of target light sources;

[0126] is the target light source pixel position of the j-th target light source at the i-th moment;

[0127] K is the internal parameter of the target camera;

[0128] is the estimated external parameter;

[0129] P j is the estimated three-dimensional light source position of the j-th target light source;

[0130] Pose i is the estimated vehicle pose of the target vehicle at the i-th moment;

[0131] Pose i-1 is the estimated vehicle pose of the target vehicle at the (i - 1)-th moment;

[0132] T i-1，i is the vehicle pose transformation relationship from the (i - 1)-th moment to the i-th moment;

[0133] is the reprojection error;

[0134] ||T i-1，i -Pose i *Posei-1 -1 || is the vehicle posture transformation error.

[0135] In some embodiments, the calibration module 405 is specifically used to: optimize the parameter values ​​of the estimated three-dimensional position of the light source, the estimated vehicle posture and the estimated external parameters, and calculate the optimized target error corresponding to the parameter values; when the optimized target error is less than the preset error, the estimated external parameters corresponding to the target error are determined as the calibration external parameters of the target camera.

[0136] In some embodiments, the calibration module 405 is specifically used to: use the general graph optimization algorithm G2O to set the estimated three-dimensional position of the light source, the estimated vehicle posture and the estimated external parameters as vertices to be optimized, set the target light source pixel position and the vehicle posture transformation relationship as constraints, and calculate the optimized target error.

[0137] In some embodiments, the calibration module 405 is further used to: check the target light source pixel position by using a kernel function; and remove the wrong target light source pixel position from the constraint condition.

[0138] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0139] Figure 5 Schematic diagram of an electronic device 5 provided in an embodiment of the present application. Figure 5 As shown, the electronic device 5 of this embodiment includes: a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501. When the processor 501 executes the computer program 503, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 501 executes the computer program 503, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0140] The electronic device 5 may be a desktop computer, a notebook, a PDA, a cloud server, or other electronic device. The electronic device 5 may include, but is not limited to, a processor 501 and a memory 502. Those skilled in the art will appreciate that Figure 5 The electronic device 5 is merely an example and does not limit the electronic device 5 , and may include more or less components than those shown in the figure, or different components.

[0141] The processor 501 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0142] The memory 502 may be an internal storage unit of the electronic device 5, for example, a hard disk or memory of the electronic device 5. The memory 502 may also be an external storage device of the electronic device 5, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 5. The memory 502 may also include both an internal storage unit of the electronic device 5 and an external storage device. The memory 502 is used to store computer programs and other programs and data required by the electronic device.

[0143] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0144] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium (e.g., a computer-readable storage medium). Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. The computer program may include computer program code, which may be in source code form, object code form, executable file or some intermediate form. Computer-readable storage media may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0145] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A camera calibration method, characterized in that: include: Acquire light source images at at least two moments captured by a target camera, and obtain the light source pixel position at each moment based on the light source images; wherein the target camera is mounted on a target vehicle, and the light source image includes at least one light source; According to the light source pixel position, determining the target light source pixel position at each moment corresponding to the same target light source; Obtaining the vehicle posture transformation relationship of the target vehicle at adjacent moments through vehicle sensors; According to the target light source pixel position and the vehicle posture transformation relationship, a target error is obtained; the target error is obtained based on the reprojection error and the vehicle posture transformation error; The reprojection error is used to characterize the error between the projection and the pixel position of the target light source, wherein the projection is obtained by projecting the estimated three-dimensional light source position of the target light source according to the estimated vehicle posture and the estimated external parameters of the target camera; The vehicle posture transformation error is used to characterize the error between the change of the estimated vehicle posture at adjacent moments and the vehicle posture transformation relationship; Determining a calibration extrinsic parameter of the target camera according to the target error; The step of obtaining a target error according to the target light source pixel position and the vehicle posture transformation relationship includes: According to the target light source pixel position and the vehicle posture transformation relationship, the target error is calculated by the following formula: Wherein, n is the number of moments for collection, and m is the number of target light sources; is the target light source pixel position of the jth target light source at the i-th moment; K is the internal parameter of the target camera; is the estimated external parameter; P j is the estimated light source three-dimensional position of the j-th target light source; Pose i is the estimated vehicle pose of the target vehicle at the i-th moment; Pose i-1 is the estimated vehicle posture of the target vehicle at the i-1th moment; T i-1,i is the vehicle posture transformation relationship from the i-1th moment to the i-th moment; is the reprojection error; ||T i-1,i -Pose i *Pose i-1 -1 || is the vehicle posture transformation error.

2. The method according to claim 1, characterized in that The step of obtaining the pixel position of the light source at each moment based on the light source image includes: Acquire brightness information in the light source image, and determine, based on the brightness information, a position where the brightness is greater than a preset brightness as the light source pixel position; or, Acquire color information in the light source image, and determine, based on the color information, a position with a preset light source color as the light source pixel position; or, A light source outline or a light source center in the light source image is acquired, and a pixel position of the light source is determined according to the light source outline or the light source center.

3. The method according to claim 1, characterized in that The vehicle sensor includes at least one of an inertial measurement unit IMU and a global navigation satellite system GNSS; The step of obtaining the vehicle posture transformation relationship of the target vehicle at adjacent moments through the vehicle sensor includes: Measuring the angular velocity and acceleration of the target vehicle at adjacent moments by the IMU, and determining the vehicle posture transformation relationship according to the angular velocity and acceleration; or, The three-dimensional coordinates and speed of the target vehicle are measured by the GNSS, and the vehicle posture transformation relationship is determined according to the three-dimensional coordinates and speed.

4. The method according to claim 1, characterized in that: Determining the calibration extrinsic parameters of the target camera according to the target error includes: Optimizing the parameter values ​​of the estimated three-dimensional position of the light source, the estimated vehicle posture and the estimated external parameter, and calculating the optimized target error corresponding to the parameter value; When the optimized target error is less than the preset error, the estimated extrinsic parameter corresponding to the target error is determined as the calibration extrinsic parameter of the target camera.

5. The method according to claim 4, characterized in that The optimizing the parameter values ​​of the estimated three-dimensional position of the light source, the estimated vehicle posture and the estimated external parameter, and calculating the optimized target error corresponding to the parameter value, includes: Using the general graph optimization algorithm G2O, the estimated three-dimensional position of the light source, the estimated vehicle posture and the estimated external parameters are set as vertices to be optimized, the target light source pixel position and the vehicle posture transformation relationship are set as constraints, and the optimized target error is calculated.

6. The method according to claim 5, characterized in that Before calculating the optimized target error, the method further includes: Checking the pixel position of the target light source by using a kernel function; The erroneous pixel position of the target light source is removed from the constraint condition.

7. A camera calibration device, characterized in that: include: A first detection module is used to obtain light source images captured by a target camera at at least two moments, and obtain the light source pixel position at each moment based on the light source images; wherein the target camera is mounted on a target vehicle, and the light source image includes at least one light source; A tracking module, used to determine the target light source pixel position at each moment corresponding to the same target light source according to the light source pixel position; The second detection module is used to obtain the vehicle posture transformation relationship of the target vehicle at adjacent moments through a vehicle sensor; An error determination module, used to obtain a target error according to the target light source pixel position and the vehicle posture transformation relationship; the target error is obtained based on the reprojection error and the vehicle posture transformation error; The reprojection error is used to characterize the error between the projection and the pixel position of the target light source, wherein the projection is obtained by projecting the estimated three-dimensional light source position of the target light source according to the estimated vehicle posture and the estimated external parameters of the target camera; The vehicle posture transformation error is used to characterize the error between the change of the estimated vehicle posture at adjacent moments and the vehicle posture transformation relationship; A calibration module, used to determine the calibration extrinsic parameters of the target camera according to the target error; The error determination module is specifically used to calculate the target error according to the target light source pixel position and the vehicle posture transformation relationship by the following formula: Wherein, n is the number of moments for collection, and m is the number of target light sources; is the target light source pixel position of the jth target light source at the i-th moment; K is the internal parameter of the target camera; is the estimated external parameter; P j is the estimated light source three-dimensional position of the j-th target light source; Pose i is the estimated vehicle pose of the target vehicle at the i-th moment; Pose i-1 is the estimated vehicle posture of the target vehicle at the i-1th moment; T i-1,i is the vehicle posture transformation relationship from the i-1th moment to the i-th moment; is the reprojection error; ||T i-1,i -Pose i *Pose i-1 -1 || is the vehicle posture transformation error.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images