Off-line calibration method for vehicle-mounted camera, auxiliary driving system and vehicle

Abstract

The invention relates to a method for offline calibration of a vehicle-mounted camera (11), and the method comprises the steps: converting a two-dimensional image, which is collected by the vehicle-mounted camera (11) and is related to a marker on a road surface of a calibration site, into a three-dimensional image related to the marker, based on camera parameters of the vehicle-mounted camera (11), adjusting the pixel number N of each mark point of the marker in the three-dimensional image along with the change of the distance D from the mark point to the vehicle-mounted camera (11) through a collinear theorem; and calibrating the vehicle-mounted camera (11) by using the three-dimensional image. The application also relates to a driving assistance system, a vehicle and a computer program product. According to the method, the influence of the marker with the fuzzy edge on the calibration process of the vehicle-mounted camera is reduced as much as possible, the robustness of the offline calibration method of the vehicle-mounted camera is improved, and a foundation is laid for accurate control and reliable implementation of a vehicle auxiliary driving function.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of vehicle-mounted camera calibration, in particular to a method for off-line calibration of a vehicle-mounted camera, an assisted driving system, a vehicle comprising the assisted driving system according to the present application, and a computer program product. BACKGROUND

[0002] The perception result of the vehicle-mounted camera on the environment plays a key role in the control of the assisted driving function of the vehicle. A special calibration site is usually provided on the production line of the vehicle, and the vehicle is allowed to leave the factory only after the off-line calibration of the vehicle-mounted camera is completed by using the markers on the road surface of the calibration site. However, the edges of the markers on the road surface of some calibration sites are relatively blurred, which makes it difficult for the vehicle-mounted camera to successfully complete the off-line calibration by using the collected marker images.

[0003] Therefore, there is room for improvement in the off-line calibration method of the vehicle-mounted camera. SUMMARY

[0004] The present application aims to provide a method for off-line calibration of a vehicle-mounted camera, an assisted driving system, a vehicle comprising the assisted driving system according to the present application, and a computer program product, to at least partially solve the problems in the prior art.

[0005] According to a first aspect of the present application, a method for off-line calibration of a vehicle-mounted camera is provided, which can comprise: - converting a two-dimensional image about a marker on the road surface of a calibration site collected by a vehicle-mounted camera into a three-dimensional image about the marker, and adjusting the number of pixels N of each marker point of the marker in the three-dimensional image changing with the distance D of the marker point to the vehicle-mounted camera based on the camera parameters of the vehicle-mounted camera by the collinearity theorem; - the vehicle-mounted camera can be calibrated by using the three-dimensional image.

[0006] The core idea of the present application is that: for the three-dimensional image converted from the two-dimensional image about the marker on the road surface of the calibration site, the number of pixels of each marker point in the three-dimensional image changing with the distance of the marker point to the vehicle-mounted camera can be adjusted based on the camera parameters of the vehicle-mounted camera by the collinearity theorem, and optionally the adjusted three-dimensional image can be denoised by a denoising algorithm to reduce the noise of the pixels about the edges of the marker in the three-dimensional image, thereby reducing the influence of the marker with blurred edges on the calibration process of the vehicle-mounted camera as much as possible, improving the robustness of the off-line calibration method of the vehicle-mounted camera, and laying a foundation for the accurate control and reliable implementation of the assisted driving function of the vehicle.

[0007] According to an optional embodiment of the present application, the two-dimensional image about the marker is converted into a three-dimensional image about the marker in the following manner: the X-axis horizontal coordinate and Y-axis vertical coordinate of each marker point of the marker in the three-dimensional image are respectively equal to the X-axis horizontal coordinate and Y-axis vertical coordinate of the corresponding marker point in the two-dimensional image, and the Z-axis depth coordinate of each marker point of the marker in the three-dimensional image is determined as the distance D of the corresponding marker point to the vehicle-mounted camera.

[0008] According to another optional embodiment of the present application, the total number N of pixels of the three-dimensional image can be determined based on the total number N of pixels of the two-dimensional image max and the farthest distance D of the marker point corresponding to a single pixel in the three-dimensional image max The number N of pixels of each marker point of the marker in the three-dimensional image varying with the distance D of the marker point to the vehicle-mounted camera is adjusted by the collinearity theorem.

[0009] According to another optional embodiment of the present application, in the case where the vehicle-mounted camera is configured as a binocular camera, the distance D of each marker point of the marker to the vehicle-mounted camera can be determined by using the stereoscopic effect.

[0010] According to another optional embodiment of the present application, in the case where the vehicle-mounted camera is configured as a monocular camera, the distance D of each marker point of the marker to the vehicle-mounted camera can be determined by using the motion recovery structure algorithm.

[0011] According to another optional embodiment of the present application, the adjusted three-dimensional image can be subjected to a noise reduction processing by using a noise reduction algorithm, such as a Gaussian filtering algorithm and / or a bilateral filtering algorithm, etc., to reduce the noise of the pixels about the edge of the marker in the three-dimensional image, and the vehicle-mounted camera is calibrated by using the three-dimensional image subjected to the noise reduction processing.

[0012] According to another optional embodiment of the present application, the parameters of the calibration of the vehicle-mounted camera can include the internal parameters and external parameters of the vehicle-mounted camera, wherein the internal parameters can include the focal length parameter, principal point coordinate, tilt factor and distortion coefficient of the vehicle-mounted camera, and the external parameters can include the position parameter and attitude parameter of the vehicle-mounted camera 11. Especially in the case where the vehicle-mounted camera is configured as a binocular camera, the parameters of the calibration of the vehicle-mounted camera can further include the binocular baseline, essential matrix, fundamental matrix and / or stereo rectification parameter, etc.

[0013] According to the second aspect of the present application, an assisted driving system is provided, which can include the following components: - a vehicle-mounted camera configured to collect a two-dimensional image about a marker on a road surface of a calibration site; - A control unit for performing the method according to this application.

[0014] According to a third aspect of this application, a vehicle is provided that may include a driver assistance system according to this application.

[0015] According to a fourth aspect of this application, a computer program product, such as a computer-readable program carrier, is provided, comprising or storing computer program instructions that, when executed by a processor, at least assist in implementing the steps of the method described in this application. Attached Figure Description

[0016] The principles, features, and advantages of this application can be better understood by describing it in more detail below with reference to the accompanying drawings. The drawings show: Figure 1 A flowchart illustrating a method for offline calibration of an in-vehicle camera according to an exemplary embodiment of this application is shown. Figure 2 A schematic diagram showing a three-dimensional image of a marker according to an exemplary embodiment of this application; Figure 3 This diagram illustrates the number of pixels N of a marker point in a three-dimensional image as a function of the distance D from the marker point to the vehicle-mounted camera, according to an exemplary embodiment of this application. Figure 4 A schematic diagram showing the pixels of a first marker point in a three-dimensional image according to an exemplary embodiment of this application; and Figure 5 A schematic diagram of a vehicle according to an exemplary embodiment of this application is shown. Detailed Implementation

[0017] To make the technical problems to be solved, the technical solutions, and the beneficial technical effects of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and several exemplary embodiments. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit the scope of protection of this application.

[0018] Figure 1 A flowchart illustrating a method for offline calibration of an in-vehicle camera according to an exemplary embodiment of this application is shown. The following exemplary embodiments describe the method according to this application in more detail.

[0019] like Figure 1As shown, the method may include steps S1 and S2. In step S1, a two-dimensional image of the markers on the calibration site road surface acquired by the vehicle-mounted camera can be converted into a three-dimensional image of the markers. Based on the camera parameters of the vehicle-mounted camera 11, the number of pixels N of each marker point of the marker in the three-dimensional image is adjusted according to the collinearity theorem, as the distance D from the marker point to the vehicle-mounted camera 11 changes. In the current embodiment of this application, a standard calibration site is usually provided on the production line of vehicle 1. Test personnel can drive vehicle 1 on the road surface of the calibration site to perform end-of-line calibration of each vehicle-mounted camera. Markers 2 are provided on the road surface of the calibration site. These markers include, for example, line patterns or patterns with other easily distinguishable shapes (e.g., checkerboard patterns), and the color of the markers is highly distinguishable from the color of the surrounding road surface. In addition, during normal use after the vehicle leaves the factory, factors such as vehicle body vibration may cause the installation position of the vehicle camera 11 to shift. Therefore, it is also necessary to calibrate the vehicle camera at a calibration site such as a 4S store.

[0020] During the driving of vehicle 1 on the road surface of the calibrated site, the vehicle-mounted camera 11 can capture original two-dimensional images of the markers on the road surface. From the two-dimensional images, the X-axis horizontal coordinates and Y-axis vertical coordinates of each marker point of marker 2 can be extracted. The marker 2 is composed of a certain number of marker points with predefined dimensions. Here, the two-dimensional image of the marker 2 can be converted into a three-dimensional image of the marker as follows: the X-axis horizontal coordinates and Y-axis vertical coordinates of each marker point of marker 2 in the three-dimensional image are equal to the X-axis horizontal coordinates and Y-axis vertical coordinates of the corresponding marker point in the two-dimensional image, respectively. The Z-axis depth coordinate of each marker point of marker 2 in the three-dimensional image is determined as the distance D from the corresponding marker point to the vehicle-mounted camera 11.

[0021] Different depth estimation algorithms can be used to calculate the distance D from each marker point of the marker 2 to the vehicle camera 11 for different types of vehicle cameras 11. For example, when the vehicle camera 11 is configured as a binocular camera, the distance D from each marker point of the marker 2 to the vehicle camera 11 can be obtained using the stereo effect. Specifically, the pixel offset—i.e., parallax—of the marker in two two-dimensional images captured by the two lenses of the binocular camera can be calculated. Based on the parallax, the baseline distance (i.e., the mounting distance) and focal length of the two lenses, the distance D from each marker point of the marker 2 to the vehicle camera 11 can be calculated using triangulation formulas. As another example, when the vehicle camera 11 is configured as a monocular camera, the distance D from each marker point of the marker 2 to the vehicle camera 11 can be obtained using the Structure from Motion algorithm. Specifically, multiple frames of images of the marker 2 can be acquired by a monocular camera during vehicle movement, and the disparity of the marker can be calculated by feature point matching. Then, based on the camera parameters of the monocular camera and the disparity, the distance D from each marker point of the marker to the vehicle-mounted camera 11 can be calculated using a triangulation formula.

[0022] exist Figure 2 The schematic diagram of a three-dimensional image of marker 2 shown according to an exemplary embodiment of this application only exemplarily marks the first marker point P1 closest to the vehicle-mounted camera 11 on the left side of the road surface, and the second marker point P2 farthest from the vehicle-mounted camera 11. According to the theorem of intersecting lines, or the rule of three, the number of pixels N of each marker point of marker 2 in the three-dimensional image is negatively correlated, and in particular, negatively proportional, with respect to the distance D from the corresponding marker point to the vehicle-mounted camera 11. For example... Figure 3 The diagram illustrates the number of pixels N of a marker point in a 3D image according to an exemplary embodiment of this application as a function of the distance D from the marker point to the vehicle-mounted camera. The number of pixels N of the marker point in the 3D image, the distance D from the marker point to the vehicle-mounted camera 11, and the total number of pixels N in the 3D image are also shown. max And the farthest distance D of the marker point corresponding to a single pixel in the three-dimensional image. max The following functional relationship exists between them: .

[0023] Therefore, it is possible to base the total number of pixels N of the three-dimensional image on this data. maxAnd the farthest distance D of the marker point corresponding to a single pixel in the three-dimensional image. max The collinearity theorem is used to determine the number of pixels N of each marker point of the marker 2 in the 3D image as the distance D from the marker point to the vehicle-mounted camera 11 changes. The formula is as follows: .

[0024] According to the above formula, the farther the distance D from the marker point to the vehicle-mounted camera 11, the smaller the number of pixels N of the marker point in the three-dimensional image, until the number of pixels N of the marker point in the three-dimensional image decreases to 1 as the distance D increases, and no marker point is shown in the three-dimensional image anymore. Figure 4 A schematic diagram showing the pixels of a first marker point in a three-dimensional image according to an exemplary embodiment of this application is provided. The first marker point P1, which is closest to the vehicle-mounted camera 11 on the left side of the road, is represented as four pixels in the three-dimensional image—its position is... Figure 4 The marker P1 is marked with a bold white pixel box, while other marker points are represented by thin white pixel boxes in the 3D image. The surrounding road surface pixels are represented by gray pixel boxes in the 3D image. In contrast, the second marker P2, the furthest marker from the vehicle-mounted camera 11 on the left side of the road surface, can be represented as a single pixel in the 3D image.

[0025] Considering that the edges of markers 2 on the road surface in some calibration sites are relatively blurry—for example, the contrast between the pixels of the marker and the surrounding road surface is insufficient, or there may be interfering specular reflections or glare on the road surface—the adjusted three-dimensional image can be denoised using a denoising algorithm to reduce the noise of the pixels of the marker edges in the three-dimensional image. The denoising algorithm includes, for example, Gaussian filtering and / or bilateral filtering algorithms.

[0026] In step S2, the vehicle-mounted camera 11 can be calibrated using the three-dimensional image, particularly using a noise-reduced three-dimensional image. The parameters used for calibrating the vehicle-mounted camera 11 can include its internal and external parameters. Specifically, the internal parameters can include the following: focal length, representing the distance from the lens center to the imaging plane; principal point coordinates, representing the intersection of the lens optical axis and the imaging plane; tilt factor, used for non-orthogonality correction of the pixel coordinate axes; and distortion coefficients, including radial and tangential distortion, used to eliminate distortion effects in the acquired image. The external parameters can include the position parameters of the vehicle-mounted camera 11 (including the X, Y, and Z coordinates of the lens center in the vehicle coordinate system) and attitude parameters (including tilt angle, pitch angle, and yaw angle), etc. When the vehicle-mounted camera 11 is configured as a binocular camera, the parameters calibrated for the vehicle-mounted camera 11 may include one or more of the following parameters: binocular baseline, i.e., the distance between the centers of the left and right lenses of the binocular camera; essential matrix, which represents the epipolar geometry between the two lenses of the binocular camera; fundamental matrix, which represents the epipolar geometry of the pixel coordinates; stereo correction parameters, which are used to eliminate vertical parallax of the image, etc.

[0027] According to embodiments of this application, for a three-dimensional image converted from a two-dimensional image of markers on the road surface of the calibration site, the number of pixels of each marker point in the three-dimensional image can be adjusted based on the camera parameters of the vehicle-mounted camera using the collinearity theorem, as the distance from the marker point to the vehicle-mounted camera changes. Optionally, the adjusted three-dimensional image can also be denoised using a noise reduction algorithm to reduce the noise of pixels with respect to the edges of the markers in the three-dimensional image. This minimizes the impact of blurred marker edges on the calibration process of the vehicle-mounted camera, improves the robustness of the vehicle-mounted camera's offline calibration method, and lays the foundation for the precise control and reliable implementation of vehicle assisted driving functions.

[0028] In addition, it should be noted that the step numbers described herein do not necessarily represent the order of steps, but are merely a reference numeral. The order may be changed depending on the specific circumstances, as long as the technical objective of this application can be achieved.

[0029] Figure 5 A schematic diagram of a vehicle 1 according to an exemplary embodiment of this application is shown. Figure 5 As shown, the vehicle 1 is equipped with a driver assistance system 10, which may include the following components: - Vehicle-mounted camera 11, which is configured to acquire two-dimensional images of markers on the calibration site road surface; - Control unit 12, which is used to execute the method according to the present application, wherein the control unit 12 may be integrated into the vehicle camera 11, or may be a vehicle controller such as a domain controller or a vehicle controller.

[0030] It should be understood that the terms “first,” “second,” “third,” etc., used in this document are for descriptive purposes only and should not be construed as indicating or implying relative importance, nor should they be construed as implicitly specifying the number of technical features indicated.

[0031] If an embodiment includes an "and / or" association between a first feature and a second feature, it should be interpreted as follows: according to one implementation, the embodiment has not only the first feature but also the second feature; according to another implementation, the embodiment has either only the first feature or only the second feature.

[0032] Although specific embodiments have been described above, these embodiments are not intended to limit the scope of this application, even when only a single embodiment is described with respect to a particular feature. The feature examples provided in this application are intended for illustrative purposes and not for limitation, unless otherwise stated. In practice, multiple features may be combined with each other as needed and where technically feasible. Various substitutions, modifications, and alterations are also conceived without departing from the spirit and scope of this application.

Claims

1. A method for offline calibration of an in-vehicle camera (11), the method comprising: The two-dimensional image of the markers on the calibration site road surface acquired by the vehicle-mounted camera (11) is converted into a three-dimensional image of the markers. Based on the camera parameters of the vehicle-mounted camera (11), the number of pixels N of each marker point of the marker in the three-dimensional image is adjusted according to the collinearity theorem as the distance D from the marker point to the vehicle-mounted camera (11) changes. The vehicle-mounted camera (11) is calibrated using the three-dimensional image.

2. The method according to claim 1, wherein, The two-dimensional image of the marker is converted into a three-dimensional image of the marker in the following manner: the horizontal X-axis coordinate and vertical Y-axis coordinate of each marker point in the three-dimensional image are equal to the horizontal X-axis coordinate and vertical Y-axis coordinate of the corresponding marker point in the two-dimensional image, and the depth Z-axis coordinate of each marker point in the three-dimensional image is determined as the distance D from the corresponding marker point to the vehicle camera (11).

3. The method according to any one of the preceding claims, wherein, Based on the total number of pixels N of the three-dimensional image max And the farthest distance D of the marker point corresponding to a single pixel in the three-dimensional image. max The collinearity theorem is used to adjust the number of pixels N of each marker point of the marker in the three-dimensional image as the distance D of the marker point to the vehicle camera (11) changes.

4. The method according to any one of the preceding claims, wherein, When the vehicle-mounted camera (11) is configured as a binocular camera, the distance D from each marker point of the marker to the vehicle-mounted camera (11) is obtained by utilizing the stereo effect.

5. The method according to any one of the preceding claims, wherein, When the vehicle-mounted camera (11) is configured as a monocular camera, the distance D from each marker point of the marker to the vehicle-mounted camera (11) is obtained using the motion recovery structure algorithm.

6. The method according to any one of the preceding claims, wherein, The adjusted 3D image is denoised using a denoising algorithm to reduce noise in the pixels of the 3D image with respect to the edges of the marker, and the vehicle camera (11) is calibrated using the denoised 3D image, wherein the denoising algorithm includes, for example, a Gaussian filtering algorithm and / or a bilateral filtering algorithm.

7. The method according to any one of the preceding claims, wherein, The parameters calibrated for the vehicle-mounted camera (11) include the internal parameters and external parameters of the vehicle-mounted camera (11). The internal parameters include the focal length parameter, principal point coordinates, tilt factor and distortion coefficient of the vehicle-mounted camera (11). The external parameters include the position parameter and attitude parameter of the vehicle-mounted camera (11). In particular, when the vehicle-mounted camera (11) is configured as a binocular camera, the parameters calibrated for the vehicle-mounted camera (11) also include the binocular baseline, essential matrix, fundamental matrix and / or stereo correction parameters.

8. A driver assistance system (10), the driver assistance system (10) comprising the following components: A vehicle-mounted camera (11) is configured to capture two-dimensional images of markers on the road surface of the calibration site; Control unit (12) for performing the method according to any one of the preceding claims.

9. A vehicle (1) comprising a driver assistance system (10) according to claim 8.

10. A computer program product, such as a computer-readable program carrier, comprising or storing computer program instructions that, when executed by a processor, at least auxiliaryly implement the steps of the method according to any one of claims 1 to 7.

Images