Camera external parameter calibration method, device and equipment and storage medium

Abstract

The invention provides a camera external parameter calibration method and device, equipment and a storage medium. In one example, the method comprises: for a specified target camera in target cameras, determining coordinates of a vanishing point in a dedistorted image according to a linear equation of a side edge of a lane line in the dedistorted image; determining a roll angle of the specified target camera according to a lane line width constraint; determining a pitch angle and a yaw angle of the specified target camera according to the roll angle of the specified target camera and the coordinates of the vanishing point; determining a z value of the specified target camera in the vehicle body world coordinate system according to the angle extrinsic parameter of the specified target camera, the initial position parameter of the specified target camera and the lane line width; and determining a w value of the second target camera in the vehicle body world coordinate system in a mode of aligning the same lane lines in the first distortion-removed image and the second distortion-removed image. According to the method, camera external parameter calibration in a natural scene can be realized, and accurate h / z / w external parameters of the camera do not need to be provided.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular, to a method, device, equipment and storage medium for calibrating the external parameters of a camera. Background Art

[0002] The in-vehicle 360-degree surround view system can generate a surround view image that facilitates users to quickly understand the surrounding scene of the vehicle body by using four fisheye cameras installed in the front, left, right, and rear of the vehicle, and can provide visual assistance to users under low-speed working conditions. With the continuous development of autonomous driving technology, in-vehicle cameras, as an important tool for perceiving the environment in autonomous driving, users have a particularly strong demand for the in-vehicle 360-degree surround view assistance system, and the surround view system gradually covers more and more vehicle models. In the in-vehicle surround view system, whether the external parameters of the in-vehicle camera are accurate will directly affect the stitching effect. Therefore, the accurate calibration of the external parameters of the in-vehicle camera is crucial. Summary of the Invention

[0003] In view of this, this application provides a method, device, equipment and storage medium for calibrating the external parameters of a camera.

[0004] Specifically, this application is implemented through the following technical solutions: According to the first aspect of the embodiments of this application, a method for calibrating the external parameters of a camera is provided, including: Obtain an image collected by a target camera; wherein, the target camera includes an in-vehicle camera installed on the periphery of the vehicle body; For a specified target camera among the target cameras, determine the straight-line equation of the side of the lane line in the undistorted image based on the undistorted image; wherein, the undistorted image is obtained by undistorting the image collected by the specified target camera according to the internal parameters of the specified target camera; the specified target camera includes a first target camera installed in front of the vehicle body or a second target camera installed behind the vehicle body; Determine the coordinates of the vanishing point in the undistorted image based on the straight-line equation of the side of the lane line in the undistorted image; Determine the roll angle of the specified target camera based on the lane line width constraint; wherein, the lane line width constraint includes that when the two sides of the lane line are projected onto the vehicle body world coordinate system, the distance between the two sides of different lane lines is the same; Determine the pitch angle and yaw angle of the specified target camera based on the roll angle of the specified target camera and the coordinates of the vanishing point; Determine the z value of the specified target camera in the vehicle body world coordinate system based on the external angle parameters of the specified target camera, the initial position parameters of the specified target camera, and the lane line width; wherein, the external angle parameters include roll angle, pitch angle, and yaw angle, and the initial position parameters of the specified target camera include the initial w value, initial h value, and initial z value of the specified target camera in the vehicle body world coordinate system; the initial w value and initial h value are set according to the vehicle body length and vehicle body width, and the initial z value is an empirical value; When the target camera includes the first target camera and the second target camera, determine the w value of the second target camera in the vehicle body world coordinate system by aligning the same lane lines in the first undistorted image and the second undistorted image; wherein, the first undistorted image is obtained by undistorting the image collected by the first target camera according to the internal parameters of the first target camera, and the second undistorted image is obtained by undistorting the image collected by the second target camera according to the internal parameters of the second target camera.

[0005] According to the second aspect of the embodiments of the present application, a device for calibrating external camera parameters is provided, including: An acquisition unit, configured to acquire an image collected by a target camera; wherein, the target camera includes an in-vehicle camera installed on the periphery of the vehicle body; A first determination unit, configured to, for a specified target camera in the target camera, determine the straight line equation of the side of the lane line in the undistorted image according to the undistorted image; wherein, the undistorted image is obtained by undistorting the image collected by the specified target camera according to the internal parameters of the target camera; the specified target camera includes a first target camera installed in front of the vehicle body or a second target camera installed behind the vehicle body; The first determination unit is further configured to determine the coordinates of the vanishing point in the undistorted image according to the straight line equation of the side of the lane line in the undistorted image; A second determination unit, configured to determine the roll angle of the specified target camera according to the lane line width constraint; wherein, the lane line width constraint includes that when the two sides of the lane line are projected onto the vehicle body world coordinate system, the distance between the two sides of different lane lines is the same; The second determination unit is further configured to determine the pitch angle and yaw angle of the specified target camera according to the roll angle of the specified target camera and the coordinates of the vanishing point; A third determination unit, configured to determine the z value of the specified target camera in the vehicle body world coordinate system according to the external angle parameters of the specified target camera, the initial position parameters of the specified target camera, and the lane width; wherein, the external angle parameters include a roll angle, a pitch angle, and a yaw angle, and the initial position parameters of the specified target camera include an initial w value, an initial h value, and an initial z value of the specified target camera in the vehicle body world coordinate system; The third determination unit is further configured to, when the target cameras include the first target camera and the second target camera, determine the w value of the second target camera in the vehicle body world coordinate system by aligning the same lane lines in the first undistorted image and the second undistorted image; wherein, the first undistorted image is obtained by undistorting the image collected by the first target camera according to the internal parameters of the first target camera, and the second undistorted image is obtained by undistorting the image collected by the second target camera according to the internal parameters of the second target camera.

[0006] According to a third aspect of the embodiments of the present application, there is provided an electronic device, including a processor and a memory, where the memory stores machine-executable instructions that can be executed by the processor, and the processor is configured to execute the machine-executable instructions to implement the method provided in the first aspect.

[0007] According to a fourth aspect of the embodiments of the present application, there is provided a machine-readable storage medium, where machine-executable instructions are stored in the machine-readable storage medium, and when the machine-executable instructions are executed by a processor, the method provided in the first aspect is implemented.

[0008] The technical solution provided by the present application can at least bring the following beneficial effects: By acquiring the images captured by the target camera, for a specified target camera installed in front of or behind the vehicle body in the target camera, the straight-line equation of the side of the lane line in the undistorted image can be determined based on the undistorted image, and the coordinates of the vanishing point in the undistorted image can be determined based on the straight-line equation of the side of the lane line in the undistorted image. Moreover, the roll angle of the specified target camera can be determined based on the lane line width constraint. Furthermore, the pitch angle and yaw angle of the specified target camera can be determined based on the roll angle of the specified target camera and the coordinates of the vanishing point, and the z value of the specified target camera in the vehicle body world coordinate system can be determined based on the external angle parameters of the specified target camera, the initial position parameters of the specified target camera, and the lane line width. When the target camera includes a first target camera and a second target camera, the w value of the second target camera in the vehicle body world coordinate system can be determined by aligning the same lane lines in the first undistorted image and the second undistorted image, realizing the calibration of the external camera parameters in a natural scene, without the need to deploy a calibration site, reducing the labor and site costs, and in the calibration process, a rough initial h value and w value can be set according to the vehicle body length and vehicle body width, and the initial z value can be set according to experience, without the need to provide accurate h / w / z external camera parameter values of the camera, improving the practicability. Description of the Drawings

[0009] Figure 1 is a schematic flowchart of a method for calibrating external camera parameters shown in an exemplary embodiment of the present application; Figure 2 is a schematic diagram of mapping a straight line in an undistorted image to the vehicle body world coordinate system shown in an exemplary embodiment of the present application; Figure 3 is a schematic diagram of a calibration environment shown in an exemplary embodiment of the present application; Figure 4 is a schematic structural diagram of a system for calibrating external camera parameters shown in an exemplary embodiment of the present application; Figure 5 is a schematic diagram of a camera capturing an image shown in an exemplary embodiment of the present application; Figure 6 is a schematic flowchart of a data processing process shown in an exemplary embodiment of the present application; Figure 7 is a schematic diagram of an undistorted image shown in an exemplary embodiment of the present application; Figure 8 is a schematic flowchart of a calibration process for external camera parameters of front and rear cameras shown in an exemplary embodiment of the present application; Figure 9 is a schematic diagram of a coordinate system shown in an exemplary embodiment of the present application; Figure 10 is a schematic diagram of the direction of the x vector in the camera coordinate system shown in an exemplary embodiment of the present application; Figure 11 It is a schematic diagram of feature matching of a front left view shown in an exemplary embodiment of the present application; Figure 12 It is a schematic diagram of a calibration effect shown in an exemplary embodiment of the present application; Figure 13 It is a schematic structural diagram of a camera external parameter calibration device shown in an exemplary embodiment of the present application; Figure 14 It is a schematic hardware structure diagram of an electronic device provided in an embodiment of the present application. Detailed implementation manners

[0010] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of the present application, some technical terms involved in the embodiments of the present application will be described below.

[0011] 1. Lane vanishing point (abbreviated as vanishing point): It is the point where the lane lines gradually tend to disappear in the field of view. It represents the intersection point of these lines under perspective projection when the lane lines are parallel to the horizon and extend into the distance. Due to the curvature of the earth and the change of the viewing angle, the lane lines gradually converge to a point from near to far, and this point is usually located in the distance of the field of view.

[0012] To make the above objects, features, and advantages of the embodiments of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0013] It should be noted that the sequence numbers of the steps in the embodiments of the present application do not mean the order of execution. The execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0014] Please refer to Figure 1 , which is a flowchart of a camera external parameter calibration method provided in an embodiment of the present application. As Figure 1 shown, the camera external parameter calibration method may include the following steps: Step S100, obtaining an image collected by a target camera; wherein, the target camera includes an in-vehicle camera installed on the periphery of the vehicle body.

[0015] Exemplarily, the target camera may include a plurality of in-vehicle cameras (which may be referred to as in-vehicle surround-view cameras) installed on the periphery of the vehicle body, and the total field of view of the plurality of in-vehicle cameras covers the entire vehicle body (360-degree surround view).

[0016] Exemplarily, an image of the external scene of the vehicle may be collected by the target camera.

[0017] Exemplarily, when the vehicle is in a natural scenario (such as driving on a road), images of the external scene of the vehicle can be collected by a target camera.

[0018] Step S110: For a specified target camera among the target cameras, determine the straight-line equation of the side of the lane line in the undistorted image according to the undistorted image, where the undistorted image is obtained by undistorting the image collected by the specified target camera according to the internal parameters of the specified target camera.

[0019] Exemplarily, the specified target camera may include a target camera installed in front of the vehicle body (which can be referred to as the first target camera) or a target camera installed behind the vehicle body (which can be referred to as the second target camera).

[0020] In the embodiments of the present application, for a specified target camera among the target cameras, the image collected by the specified target camera can be corrected for distortion according to the internal parameters of the specified target camera to obtain an undistorted image, and the straight-line equation of the side of the lane line in the undistorted image can be determined.

[0021] Taking the specified target camera as the first target camera as an example, for the first target camera installed in front of the vehicle body, the image collected by the first target camera can be corrected for distortion according to the internal parameters of the first target camera to obtain an undistorted image (which can be referred to as the first undistorted image), and the straight-line equation of the side of the lane line in the first undistorted image can be determined.

[0022] In one example, edge point extraction and straight-line fitting can be performed in the undistorted image to obtain the straight-line equation in the undistorted image (the straight-line equation of the side of the lane line).

[0023] Exemplarily, the method for extracting edge points may include, but is not limited to, canny operator, sobel operator, or laplace operator, etc.

[0024] Exemplarily, the straight-line fitting method may include, but is not limited to, least squares method, gradient descent method, etc.

[0025] In another example, a deep learning network can be used to implement straight-line extraction to determine the straight-line equation in the undistorted image.

[0026] Exemplarily, a schematic diagram of mapping the straight line (such as the side of the lane line) in the undistorted image to the vehicle body world coordinate system can be as Figure 2 shown; where l1 and l2 correspond to the same lane line, and l3 and l4 correspond to the same lane line.

[0027] Exemplarily, the undistorted image may include at least two lane lines, that is, at least two pairs of straight-line equations (one lane line corresponds to a pair of straight-line equations).

[0028] Step S120: Determine the coordinates of the vanishing point in the undistorted image based on the straight-line equations of the sides of the lane lines in the undistorted image.

[0029] Exemplarily, based on the straight-line equations of the sides of multiple lane lines in the undistorted image, the intersection points between the straight lines can be obtained, and by averaging multiple intersection points, the coordinates of the vanishing point in the undistorted image can be obtained.

[0030] Taking the first target camera as an example, based on the straight-line equations of the sides of multiple lane lines in the first undistorted image, the intersection points between the straight lines can be obtained, and by averaging multiple intersection points, the coordinates of the vanishing point (which can be called the first vanishing point) in the first undistorted image can be obtained.

[0031] Step S130: Determine the roll angle of the specified target camera according to the lane width constraint; wherein, the lane width constraint includes that when the two sides of the lane line are projected onto the vehicle body world coordinate system, the distance between the two sides of different lane lines is the same.

[0032] In the embodiments of the present application, considering that in the actual scenario, the width of the lane line is fixed. For example, the standard width of the national standard lane line is 15 cm, that is, the widths of different lane lines are the same.

[0033] Therefore, when the lane lines in the undistorted image are projected onto the vehicle body world coordinate system, the widths of different lane lines (which can be characterized by the distance between the two sides of the vehicle line) are the same.

[0034] Exemplarily, the widths of different lane lines being the same may include that the widths of different lane lines are the same, or there are tolerable deviations.

[0035] Correspondingly, when the straight-line equations of the sides of the lane lines in the undistorted image are determined, the roll angle (which can be denoted as roll) of the specified target camera can be determined according to the lane width constraint.

[0036] For example, the value of roll can be optimized based on a numerical iteration algorithm to make Figure 2 the distance between l1 and l2 the same as the distance between l3 and l4.

[0037] Step S140: Determine the pitch angle and yaw angle of the specified target camera according to the roll angle of the target camera and the coordinates of the vanishing point.

[0038] In the embodiments of the present application, when the roll angle of the specified target camera and the coordinates of the vanishing point are determined in the manner described in the above embodiments, the pitch angle (which can be denoted as pitch) and yaw angle (which can be denoted as yaw) of the specified target camera can be determined based on the roll angle of the specified target camera and the coordinates of the vanishing point. The specific implementation process can be described in combination with specific examples below.

[0039] So far, the extrinsic camera parameters of the specified target camera, including roll, pitch, and yaw, have been determined.

[0040] Step S150: Determine the z value of the specified target camera in the vehicle body world coordinate system based on the extrinsic camera parameters of the specified target camera, the initial position parameters of the specified target camera, and the lane line width; wherein, the extrinsic camera parameters include the roll angle, pitch angle, and yaw angle, and the initial position parameters of the specified target camera include the initial w value, initial h value, and initial z value of the specified target camera in the vehicle body world coordinate system.

[0041] In the embodiments of the present application, when the extrinsic camera parameters of the specified target camera are determined, the lane lines can be mapped to the world coordinate system based on the extrinsic camera parameters of the specified target camera and the initial position parameters of the specified target camera. Based on the lane line width (such as the standard width of the national lane line is 15 cm), the z value is optimized based on the numerical iteration algorithm so that the width of the lane lines meets the requirements (such as the width is 15 cm) when mapped to the world coordinate system, thereby determining the z value of the specified target camera in the vehicle body world coordinate system.

[0042] So far, the extrinsic camera parameters and z value of the specified target camera have been determined in the above manner. The w value and h value are included in the initial position parameters, and the extrinsic calibration of the specified target camera is completed.

[0043] It should be noted that in the embodiments of the present application, the w value and h value in the initial position parameters of the target camera can be set manually without precise measurement according to the vehicle body length and vehicle body width of the vehicle where the target camera is located. The z value can be an empirical value. For example, z = 1.

[0044] For example, for a target camera installed in front of the vehicle body, since it is usually installed at the edge of the vehicle body, in the vehicle body world coordinate system established with the vehicle body center as the coordinate origin, the h value (the coordinate value in the vehicle body length direction) of this target camera can be set to -H / 2 (H is the vehicle body length), and the w value can be set to 0.

[0045] Step S160: When the target camera includes a first target camera and a second target camera, determine the w value of the second target camera in the vehicle body world coordinate system by aligning the same lane lines in the first undistorted image and the second undistorted image.

[0046] In the embodiments of the present application, considering that when the vehicle is driving normally on the road, the lane line closest to the left side of the vehicle body in the view collected by the front camera and the lane line closest to the left side of the vehicle body in the view collected by the rear camera generally belong to the same lane line (similarly for the lane line closest to the right side of the vehicle body), therefore, the w value of the rear camera can be determined by aligning the same lane line in the view of the front camera with the view of the rear camera.

[0047] Correspondingly, by aligning the same lane line in the first undistorted image and the second undistorted image, and taking the w value of the first target camera as a reference, the w value of the second target camera (the initial w value is the w value in the initial position parameters of the second target camera) can be optimized based on the numerical iteration algorithm to obtain the optimized w value of the second target camera.

[0048] It can be seen that in Figure 1 the method flow shown, by acquiring the images collected by the target camera, for a specified target camera installed in front of or behind the vehicle body in the target camera, the straight line equation of the side of the lane line in the undistorted image can be determined based on the undistorted image, and the coordinates of the vanishing point in the undistorted image can be determined based on the straight line equation of the side of the lane line in the undistorted image. Moreover, according to the lane line width constraint, the roll angle of the specified target camera can be determined. Furthermore, based on the roll angle of the specified target camera and the coordinates of the vanishing point, the pitch angle and yaw angle of the specified target camera can be determined, and based on the angular extrinsic parameters of the specified target camera, the initial position parameters of the specified target camera, and the lane line width, the z value of the specified target camera in the vehicle body world coordinate system can be determined. When the target camera includes the first target camera and the second target camera, by aligning the same lane line in the first undistorted image and the second undistorted image, the w value of the second target camera in the vehicle body world coordinate system can be determined, realizing the calibration of the camera extrinsic parameters in the natural scene, without the need to deploy a calibration site, reducing the labor and site costs. And during the calibration process, the rough initial h value and w value can be set according to the vehicle body length and vehicle body width, and the initial z value can be set according to experience, without the need to provide the accurate h / w / z extrinsic parameter values of the camera, improving the practicability.

[0049] In some embodiments, the camera extrinsic parameter calibration scheme provided by the embodiments of the present application may further include: When the target camera includes the first target camera installed in front of the vehicle body, using the multi-frame fusion method, the first calibrated extrinsic parameters corresponding to multiple frames of the first undistorted images are fused to obtain the calibrated first calibrated extrinsic parameters; wherein, the first calibrated extrinsic parameters include the angular extrinsic parameters of the first target camera and the z extrinsic parameter of the first target camera; and / or, When the target camera includes a second target camera installed at the rear of the vehicle body, the multi-frame fusion method is used to fuse the second calibration external parameters corresponding to multiple frames of second undistorted images to obtain the calibrated second calibration external parameters; wherein, the second calibration external parameters include the angular external parameters of the second target camera, and the z external parameter and w external parameter of the second target camera.

[0050] Exemplarily, considering that the calibration results obtained based on single-frame images may have some errors due to certain specific reasons. For example, during the driving of the vehicle, the vehicle body may not be parallel to the lane line. However, the above calibration scheme is carried out when the vehicle body is parallel to the lane line ( Figure 2 where l1~l4 in the vehicle body world coordinate system is parallel to the x-axis). Therefore, in order to improve the accuracy of the calibration results, the multi-frame fusion method can be used to calibrate and optimize the calibration results of the first target camera, and in the case of the existence of the second target camera, the multi-frame fusion method can also be used to calibrate and optimize the calibration results of the second target camera.

[0051] It should be noted that since the installation position of the camera is usually at the edge of the vehicle body, therefore, the h of the front camera and the rear camera is generally relatively accurate (half of the vehicle body length H) and can be not optimized.

[0052] In one example, for any one of the first target camera and the second target camera, using the multi-frame fusion method to fuse the calibration external parameters corresponding to multiple frames of undistorted images includes: For each type of calibration external parameters corresponding to the accumulated multiple frames of undistorted images, delete the maximum value of the first ratio and the minimum value of the second ratio respectively; Determine the average value of the remaining values of each type of calibration external parameters respectively; When the difference between the remaining values of each type of calibration external parameters and the average value of the same type of calibration external parameters does not exceed the preset threshold, take the determined average value of each type of calibration external parameters as the calibrated calibration external parameters; Otherwise, accumulate the calibration external parameters corresponding to more frames of undistorted images until the calibrated calibration external parameters are obtained in the above manner.

[0053] Exemplarily, taking the multi-frame fusion for the first target camera as an example.

[0054] The calibration external parameters corresponding to multiple frames of first undistorted images can be accumulated (for the determination method of the calibration external parameters corresponding to a single frame of first undistorted image, refer to the relevant description in the above embodiments), and for each type of calibration external parameters (first calibration external parameters) corresponding to the accumulated multiple frames of first undistorted images, delete the maximum value of the first ratio and the minimum value of the second ratio respectively.

[0055] For example, taking roll as an example, for the roll corresponding to the accumulated multiple frames of the first undistorted images, they can be sorted according to their magnitudes (taking sorting from largest to smallest as an example), and the roll values in the first 10% (taking the first ratio as 10%) and the last 10% (taking the second ratio as 10%) after sorting can be deleted.

[0056] For the remaining various types of calibration extrinsic parameters, the average value of the remaining values of each type of calibration extrinsic parameter can be determined respectively, and it can be determined whether the difference between the remaining value of each type of calibration extrinsic parameter and the average value of the same type of calibration extrinsic parameter exceeds a preset threshold (which can be set according to actual requirements).

[0057] For example, the thresholds for pitch, yaw, and raw can be 1°, and the thresholds corresponding to the z value and w value can be 1 m.

[0058] In the case where the difference between the remaining value of each type of calibration extrinsic parameter and the average value of the same type of calibration extrinsic parameter does not exceed the preset threshold, the average value of each type of calibration extrinsic parameter determined can be used as the calibrated calibration extrinsic parameter.

[0059] Otherwise, that is, it is determined that the difference between the remaining value of at least one type of calibration extrinsic parameter and the average value of the same type of calibration extrinsic parameter exceeds the preset threshold. For example, among the remaining calibration extrinsic parameters, the absolute value of the difference between the pitch corresponding to a certain frame of undistorted image and the average value of the pitch determined in the above manner exceeds 1°. More frames of undistorted image corresponding calibration extrinsic parameters can be continuously accumulated until the calibrated calibration extrinsic parameters are obtained in the above manner.

[0060] It should be noted that, in order to avoid being unable to determine the calibrated calibration extrinsic parameters all the time in practical applications, a calibration stop condition can be preset in advance. For example, a maximum calibration number is set. In the case where the preset calibration stop condition is reached and the calibrated calibration extrinsic parameters are still not obtained, an exception handling process can be entered. For example, an alarm can be given and manual intervention can be carried out. The specific implementation thereof will not be elaborated here.

[0061] In some embodiments, in the case where the target camera includes a third target camera installed on the side of the vehicle body, the camera extrinsic parameter calibration scheme provided by the embodiments of the present application may further include: Performing feature point matching on adjacent frame images collected by the third target camera to obtain matching point pairs; According to the obtained matching point pairs, using the principle of epipolar geometry, determining the rotation matrix R and translation vector T of the movement of the third target camera between adjacent frames; In the case where the vehicle is in a straight-ahead state between adjacent frames, determining the x vector according to the determined translation vector T of the third target camera; When the vehicle is in a turning state between adjacent frames, determine the z vector according to the determined rotation matrix R of the third target camera; Determine the external angle parameters of the third target camera according to the determined x vector and z vector; Determine the z value of the third target camera in the vehicle body world coordinate system according to the external angle parameters of the third target camera, the initial position parameters of the third target camera, and the lane line width; wherein, the initial position parameters of the third target camera include the initial w value, initial h value, and initial z value of the third target camera in the vehicle body world coordinate system.

[0062] Exemplarily, for a target camera (which can be called the third target camera) installed on the side (left or right) of the vehicle body, the calibration of the external angle parameters can be achieved by matching feature points between adjacent frames.

[0063] It should be noted that for the target camera on the left side of the vehicle body and the target camera on the right side of the vehicle body, similar calibration methods can be used for external parameter calibration.

[0064] Exemplarily, feature point matching can be performed on adjacent frame images collected by the third target camera to obtain matching point pairs.

[0065] Among them, the feature point matching method can include but is not limited to: the ORB matching method (a feature detection and description algorithm) or deep learning methods such as superpoint and superglue.

[0066] Exemplarily, according to the obtained matching point pairs, the rotation and translation relationship of the movement of the third target camera between adjacent frames (which can be represented by the rotation matrix R and the translation matrix T) can be determined using the principle of epipolar geometry.

[0067] Exemplarily, when the vehicle is in a straight state between adjacent frames, the x vector can be determined according to the determined translation vector T of the third target camera.

[0068] When the vehicle is in a turning state between adjacent frames, the z vector can be determined according to the determined rotation matrix R.

[0069] Exemplarily, the determination process of the x vector and the z vector can be as follows: The x vector is the unitized representation of the T translation vector: ; The z vector is calculated according to the rotation matrix R:

[0070]

[0071]

[0072] Exemplarily, whether the vehicle is in a straight - line state or a turning state can be determined based on the difference between the slopes of the same lane line in adjacent frames when mapped to the vehicle body world coordinate system; when the difference (which can be the absolute value of the difference) between the slopes of the same lane line between adjacent frames is less than a preset slope difference threshold (which can be set according to the actual scenario), it is determined that the vehicle is in a straight - line state; otherwise, it is determined that the vehicle is in a turning state.

[0073] Exemplarily, the rotation matrix of the third target camera can be determined based on the x - vector and the z - vector, and the external angle parameters of the third target camera can be obtained by decomposing the rotation matrix.

[0074] For example, R cam = [x, (x * z), z] * [X0, (X0 * Z0), Z0] -1 where, R can is the rotation matrix of the third target camera, and X0 and Z0 are the unit vectors [-1, 0, 0] and [0, 0, 1] in the vehicle body world coordinate system respectively.

[0075] Thus, the external angle parameters of the third target camera are determined.

[0076] The z - value of the third target camera in the vehicle body world coordinate system can be determined based on the external angle parameters of the third target camera, the initial position parameters of the third target camera, and the lane line width. The specific implementation can refer to the relevant description of the determination of the z - value of the first target camera, and this embodiment of the present application will not elaborate here.

[0077] In one example, the determination of the external angle parameters of the third target camera based on the x - vector and the z - vector can include: Obtain multiple x - vectors and multiple z - vectors corresponding to multiple pairs of adjacent frames; Filter out abnormal vectors from the multiple x - vectors and multiple z - vectors; where abnormal vectors include vectors with values outside a specified value range, x - vectors and z - vectors that do not satisfy the perpendicular constraint, or x - vectors that do not satisfy the positive - negative constraint; the positive - negative constraint includes that the component of the x - vector on the camera x - axis is positive; Determine the mean value of the filtered x - vectors as the fused x - vector; and determine the mean value of the filtered z - vectors as the fused z - vector; Determine the external angle parameters of the third target camera based on the fused x - vector and the fused z - vector.

[0078] Exemplarily, in order to improve the accuracy of the extrinsic parameters of the angle of the third target camera, during the process of determining the x vector and the z vector, the fused x vector and z vector can be determined based on the multi-frame fusion method, and the extrinsic parameters of the angle of the third target camera can be determined based on the fused x vector and z vector.

[0079] Exemplarily, multiple x vectors and multiple z vectors corresponding to multiple pairs of adjacent frames can be obtained.

[0080] Among them, the implementation method for determining the x vector or z vector of adjacent frames can refer to the relevant description in the above process.

[0081] For the multiple x vectors and multiple z vectors corresponding to the obtained multiple pairs of adjacent frames, the abnormal vectors can be filtered.

[0082] As an example, the abnormal vectors can include vectors whose values are outside the specified value range.

[0083] For example, taking the x vector as an example, for the obtained multiple x vectors, the mean ± standard deviation of the multiple x vectors can be determined as the upper and lower limits of the specified value range to obtain the specified value range, and the x vectors whose values are outside the specified value range among the multiple x vectors can be determined as abnormal vectors.

[0084] As another example, the abnormal vectors can include the x vectors and z vectors that do not satisfy the mutual perpendicular constraint.

[0085] Exemplarily, since the x vector can be understood as the vehicle forward direction and the z vector can be understood as the ground plane normal vector, there is a constraint condition that the x and z vectors are perpendicular to each other.

[0086] As yet another example, the abnormal vectors can include the x vectors that do not satisfy the positive and negative constraints; the positive and negative constraints include that the component of the x vector on the x-axis of the camera is positive.

[0087] Exemplarily, according to the known vehicle in the forward state, the positive and negative of the x and z vectors are constrained. For example, the x vector is the vehicle forward direction. For the left camera, the optical center direction of the camera coordinate system is the positive direction of the z-axis, the x-axis is parallel to the x-axis of the image coordinate system, and the y-axis is parallel to the y-axis of the image coordinate system. Therefore, the component of the x vector on the x-axis of the camera coordinate system must be positive, that is, x[0]>0. Among them, x[0] is the first row of the x vector and is the component of the x vector on the x-axis of the camera coordinate system.

[0088] Similarly, for the right camera, the component of the x vector on the x-axis of the camera coordinate system must be negative, that is, x[0]<0.

[0089] Exemplarily, in the case where the abnormal vectors are filtered, the mean value of the filtered x vector can be determined as the fused x vector; and the mean value of the filtered z vector can be determined as the fused z vector. Furthermore, the external angular parameters of the third target camera can be determined based on the fused x vector and the fused z vector.

[0090] In one example, the camera external parameter calibration scheme provided by the embodiments of the present application may further include: Converting the images captured by the target cameras into BEV views according to the calibrated external parameters of each target camera; Extracting ground feature matching points in the common viewing area between the third target camera and the adjacent target camera of the third target camera based on the BEV view of the third target camera and the BEV view of the adjacent target camera of the third target camera; wherein, the adjacent target camera of the third target camera is a target camera that has a common viewing area with the third target camera; the adjacent target camera of the third target camera includes the first target camera; Mapping the ground feature matching points of the first target camera to the vehicle body world coordinate system according to the calibrated external parameters of the first target camera to obtain the first world coordinate point set; Determining an image coordinate point set based on the ground feature matching points in the common viewing area between the third target camera and the first target camera in the BEV view of the third target camera; Determining the precise calibrated external parameters of the third target camera based on the one-to-one correspondence between the points in the first world coordinate point set and the points in the image coordinate point set.

[0091] Exemplarily, in an actual scenario, in order to achieve 360-degree surround view, the target cameras on the side of the vehicle body usually have a common viewing area with the target cameras in front of the vehicle body, that is, the field of view coverage ranges overlap.

[0092] In order to determine the more precise external parameters of the camera on the side of the vehicle body, the external parameters of the camera on the side of the vehicle body can be updated based on the matching points in the common viewing area between the target camera on the side of the vehicle body and the target camera in front of the vehicle body.

[0093] Correspondingly, converting the images captured by the target cameras (such as the above-mentioned first target camera and third target camera) into BEV (Bird Eye View) views according to the calibrated external parameters of each target camera.

[0094] Ground feature matching points can be extracted in the common viewing area between the third target camera and the adjacent target camera of the third target camera based on the BEV view of the third target camera and the BEV view of the adjacent target camera of the third target camera.

[0095] Exemplarily, the feature matching method may include, but is not limited to: the ORB matching method or deep learning methods such as superpoint and superglue.

[0096] Among them, the adjacent target camera of the third target camera includes the first target camera.

[0097] When the ground feature matching points in the common view area between the third target camera and the first target camera are determined, on the one hand, according to the calibrated extrinsic parameters of the first target camera, the ground feature matching points of the first target camera can be mapped to the vehicle body world coordinate system to obtain the corresponding set of world coordinate points (which can be called the first set of world coordinate points).

[0098] On the other hand, in the BEV view of the third target camera, the set of image coordinate points can be determined based on the ground feature matching points in the common view area between the third target camera and the first target camera.

[0099] Exemplarily, according to the intrinsic parameters of the third target camera and the calibrated extrinsic parameters of the third target camera obtained in the above manner, the above-mentioned ground feature matching points in the BEV view can be mapped to the undistorted image (which can be called the third undistorted image), and the set of image coordinate points can be determined based on the ground feature matching points in the third undistorted image.

[0100] Furthermore, based on the one-to-one correspondence between the points in the first set of world coordinate points and the points in the set of image coordinate points, the precise calibrated extrinsic parameters of the third target camera can be determined.

[0101] For example, based on the points in the first set of world coordinate points and the points in the set of image coordinate points, the pnp (Perspective-n-Point) algorithm can be used to determine the precise calibrated extrinsic parameters of the third target camera.

[0102] In one example, when the target camera includes a second target camera installed behind the vehicle body and the adjacent target camera of the third target camera includes the second target camera, it may further include: According to the calibrated extrinsic parameters of the second target camera, the ground feature matching points of the second target camera are mapped to the vehicle body world coordinate system to obtain the second set of world coordinate points; The ground feature matching points in the common view area between the third target camera and the second target camera in the view of the third target camera are added to the set of image coordinate points; The above-mentioned determination of the precise calibrated extrinsic parameters of the third target camera based on the one-to-one correspondence between the points in the first set of world coordinate points and the points in the set of image coordinate points may include: The first set of world coordinate points and the second set of world coordinate points are merged to obtain the set of world coordinate points; Determine the accurate calibration external parameters of the third target camera according to the one-to-one correspondence between the points in the world coordinate point set and the image coordinate point set.

[0103] Exemplarily, in the case where the target camera includes a target camera (such as the second target camera above) installed at the rear of the vehicle body, the second target camera usually has a common view area with the third target camera.

[0104] In this case, in order to make the calibration external parameters of the third target camera more accurate, the ground feature matching points in the common view area of the second target camera and the third target camera can also be added to the matching.

[0105] Correspondingly, on the one hand, according to the calibration external parameters of the second target camera, map the ground feature matching points of the second target camera to the vehicle body world coordinate system to obtain the second world coordinate point set; On the other hand, in the view of the third target camera, add the ground feature matching points in the common view area of the third target camera and the second target camera to the image coordinate point set; Furthermore, the first world coordinate point set and the second world coordinate point set can be merged to obtain the world coordinate point set, and the accurate calibration external parameters of the third target camera can be determined according to the one-to-one correspondence between the points in the world coordinate point set and the image coordinate point set.

[0106] In some embodiments, the target camera includes a third target camera and a fourth target camera installed on both sides of the vehicle body, and the w values of the third target camera and the fourth target camera in the vehicle body world coordinate system are symmetric with respect to the x-axis of the vehicle body world coordinate system; The camera external parameter calibration solution provided by the embodiments of the present application may further include: When the first w value of the third target camera in the vehicle body world coordinate system and the second w value of the fourth target camera in the vehicle body world coordinate system are determined, determine the average value of the first w value and the second w value as the w adjustment value; Subtract the w adjustment value from the w values of the determined target cameras in the vehicle body world coordinate system respectively to obtain the final w values of the target cameras in the vehicle body world coordinate system.

[0107] Exemplarily, since the camera installation position is usually at the edge of the vehicle body, the w values of the cameras installed on both sides of the vehicle body in the vehicle body world coordinate system are usually symmetric with respect to the x-axis of the vehicle body world coordinate system (that is, the distances from the cameras installed on both sides of the vehicle body to the x-axis - z-axis plane of the vehicle body world coordinate system are usually the same), so the w values of the calibrated target cameras can be optimized and adjusted according to this particularity.

[0108] Exemplarily, when the target cameras include a third camera and a fourth camera installed on both sides of the vehicle body, and the w values of the third target camera and the fourth target camera in the vehicle body world coordinate system are symmetric about the x-axis of the vehicle body world coordinate system, the average value of the determined w value of the third target camera in the vehicle body world coordinate system (which can be referred to as the first w value) and the determined w value of the fourth target camera in the vehicle body world coordinate system (which can be referred to as the second w value) can be determined as the w adjustment value, and the determined w value of each target camera in the vehicle body world coordinate system is subtracted by this w adjustment value to obtain the final w value of each target camera in the vehicle body world coordinate system.

[0109] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of the present application, the technical solutions provided in the embodiments of the present application will be described below with reference to specific examples.

[0110] This embodiment provides an external parameter calibration method for an in-vehicle surround-view camera. During the process of the vehicle driving on the road, the external parameters of the camera are calibrated by using the lane lines and feature points in the natural scene. Calibration based on the natural scene can reduce the deployment cost compared to calibration based on a calibration cloth, improve the practicability of the solution. In addition, in this solution, no initial external parameters are required, reducing the dependence on the initial external parameters and improving the convenience of the actual application of the solution.

[0111] Exemplarily, the schematic diagram of the calibration environment can be as Figure 3 shown.

[0112] In this embodiment, the schematic diagram of the structure of the camera external parameter calibration system can be as Figure 4 shown, including an image acquisition unit, a data transmission unit, a data processing unit, an in-vehicle surround-view panoramic image generation unit, and an image display unit. Among them: The image acquisition unit is composed of vehicle-mounted cameras distributed at four or more locations in the front, rear, left, and right of the vehicle body, and the internal parameters of the vehicle-mounted cameras have been calibrated.

[0113] Among them, taking the image acquisition unit including a front camera, a right camera, a left camera, and a rear camera as an example, the schematic diagrams of the images acquired by each camera can be respectively as Figure 5 shown (from left to right are the schematic diagrams of the images acquired by the front camera, the right camera, the left camera, and the rear camera in sequence).

[0114] The data transmission unit acquires the image data and camera internal parameter information collected by the image acquisition unit and transmits them to the data processing unit.

[0115] The data processing unit is the key processing unit for performing in-vehicle surround-view external parameter calibration. The processing flow chart is as Figure 6 shown.

[0116] The in-vehicle surround-view panoramic image generation unit generates a lookup table between the in-vehicle surround-view panoramic image and the front, left, right, and rear views based on the external parameter results calculated by the data processing unit and the camera internal parameters, and generates a fusion weight table between the front view and the left view, the left view and the rear view, the rear view and the right view, and the right view and the front view. The in-vehicle surround-view panoramic image is obtained according to the original images of the four-view cameras and the lookup table.

[0117] The image display unit presents the seamless panoramic image with calibrated external parameters to the user.

[0118] In this embodiment, as Figure 6 shown, the data processing flow may include: S1. Calibrate the external parameters of the front and rear cameras.

[0119] S2. Coarsely calibrate the angles and heights of the left and right cameras.

[0120] S3. Precisely calibrate the external parameters of the left and right cameras.

[0121] The following describes the specific implementation details of the data processing flow.

[0122] Exemplarily, the initial position parameters of the four cameras can be calculated based on the known vehicle body length H and width W (as shown in Table 1), and then the external parameters of the cameras are calibrated according to the data processing flow.

[0123] Table 1

[0124] S1. Calibrate the front and rear cameras.

[0125] Exemplarily, according to the input front and rear views and the internal parameters of the front and rear cameras, the front and rear views can be corrected for distortion to obtain undistorted images (such as the first undistorted image and the second undistorted image above). Edge points of straight lines are extracted and straight lines are fitted in the undistorted images to obtain the straight line equations in the undistorted images, and the schematic diagram can be as Figure 7 shown.

[0126] Exemplarily, methods for extracting edge points can include the canny operator, sobel operator, Laplace operator, etc., and straight line fitting can use the least squares method, gradient descent method, etc.

[0127] It should be noted that straight line extraction can also be implemented using a deep learning network, and the embodiments of the present application do not limit this.

[0128] The intersection points between the straight lines can be obtained according to the fitted straight lines, and the vanishing point is obtained by averaging multiple intersection points.

[0129] Exemplarily, at least two lane lines, that is, two pairs of straight line equations, are included in a single view (front view or rear view), and the example can be asFigure 2 as shown

[0130] The extrinsic calibration of the front and rear cameras can be performed based on the vanishing point and the lane line straight lines fitted in the undistorted image, and the process is as Figure 8 shown

[0131] Next, Figure 8 the extrinsic calibration process of the front and rear cameras as shown will be described

[0132] First, Figure 9 using the coordinate system as shown, the conversion between the vehicle body world coordinate system Ow-XwYwZw and the camera coordinate system Oc-XcYcZ will be described

[0133] As Figure 9 shown, 0c-XcYcZc is the camera coordinate system (front camera or rear camera), 0w-XwYwZw is the vehicle body world coordinate system, and 0cw-XcwYcwZcw is also a world coordinate system (auxiliary world coordinate system), which is a coordinate system with the intersection point of the camera optical axis and the ground as the origin

[0134] The vehicle body world coordinate system Ow-XwYwZw can be converted to the camera coordinate system Oc-XcYcZ in the following way 1) Rotate around Zw by yaw+ / π2 (a positive angle corresponds to clockwise rotation) to make the direction of Ow-XwYwZw consistent with the coordinate system Ocw-XcwYcwZcw 2) Rotate along the Xw axis by -pitch to make Yw consistent with the direction of Zc 3) Rotate along the Yw axis by -roll to make Xw consistent with the direction of Xc and Zw consistent with the direction of -Yc 4) Rotate along the Xw axis by -90 degrees to make Zw consistent with the direction of Zc and Yw consistent with the direction of Yc 5) Translate the coordinate system by T to make the coordinate system Ow-XwYwZw completely coincide with the camera coordinate system Oc-XcYcZc

[0135] Exemplarily, the conversion is expressed as the following formula (1) (2) where, R 33 represents a 3x3 rotation matrix, T represents a 3x1 translation vector (which can also be called a translation matrix) (which can also be expressed as T 31 ), pitch, yaw, and roll respectively represent the pitch angle, yaw angle, and roll angle of the camera, and (cam_x, cam_y, cam_z) represents the installation position of the camera in the vehicle body world coordinate system

[0136] The conversion relationship between the points on the undistorted image and the world coordinate system is shown in Equation (3). Therefore, according to Equations (1)-(3), and Z w = 0, the homography matrix H between the vehicle body world coordinate system and the camera undistorted image can be derived, as shown in Equation (4).

[0137] (3) (4) where fx, fy, cx, and cy represent the focal length and principal point of the camera internal parameters, r represents the value in the rotation matrix R 33 , t represents the value in the translation vector T 31 , (u, v) represents the pixel coordinates in the image, and (X w , Y w ) represents the coordinates in the vehicle body world coordinate system.

[0138] Exemplarily, when the straight lines l1-l4 are parallel to the X w axis in the vehicle body world coordinate system O w , the angle between the straight line and the Y cw axis is the yaw angle yaw, and the points on the straight line can be expressed as: (Y cw *tan(yaw), Y cw , 0). The coordinates of the vanishing point on the undistorted image are the values of u0 and v0 when Y cw ->∞. By combining Equations (1)-(4) and taking the limit, we get: (5) According to Equation (5), when the roll angle roll, the coordinates of the vanishing point (u0, v0), and the camera internal parameters (fx, fy, cx, cy) are known, the pitch angle and yaw angle of the camera can be obtained as follows: (6) Since the intersection point of the straight lines l1-l4 in the front view is the vanishing point, the vanishing point is known, and the rotation angle of the camera is only related to the roll angle roll. Therefore, only one unknown quantity roll needs to be solved to complete the calibration of the external angle parameters (pitch, yaw, roll). Based on the above formula derivation, the relationship between roll and pitch, yaw can be obtained.

[0139] Exemplarily, the value of roll can be optimized based on the numerical iteration algorithm to make Figure 2 the distance between the parallel lines l1-l2 equal to the distance between the parallel lines l3-l4 (the widths of the national standard lane lines are equal).

[0140] The pitch and yaw are calculated according to Equation (6), and thus the calibration of the external angle parameters of the front and rear cameras is completed.

[0141] Since the known standard width of the national lane line is 15 cm, based on the initial position parameters of the front and rear cameras and the calibrated external angle parameters, the lane line is mapped to the world coordinate system, and the value of z is optimized based on the numerical iteration algorithm to make the lane line width equal to 15 cm to complete the calibration of the height z of the front and rear cameras.

[0142] According to the constraint condition that the lane lines in the front and rear views are the same lane line, the w of the rear camera is optimized based on the numerical iteration algorithm to align the front and rear lane lines and complete the calibration of the external parameter w of the rear camera.

[0143] So far, the single-frame calibration of the pitch, yaw, roll, z of the front camera and the pitch, yaw, roll, z, w of the rear camera has been completed.

[0144] Since the camera installation position is usually at the edge of the vehicle body, it can be considered that the h of the front camera and the rear camera is half of the vehicle length H, which is a more accurate position and does not need to be optimized further.

[0145] The w of the front and rear cameras is calibrated by aligning the lane lines with the w of the front camera as the benchmark. Here, only the relative position relationship between the two can be ensured to be correct. The w of the front and rear cameras can be further optimized in the following text.

[0146] In addition, since it is assumed that the lane line is parallel to the vehicle body when calibrating the external parameters (the straight lines l1 - l4 are parallel to the Xw coordinate axis in the world coordinate system Ow), there will be abnormal samples that are not parallel during the calibration process, which will affect the accuracy. To ensure the accuracy of the calibration results, the single-frame calibration results will be accumulated, and accurate external parameters will be fused through multiple-frame results.

[0147] Exemplarily, by removing the largest 10% and the smallest 10% of the results in the calibrated external parameters and taking the average of the remaining results, if the differences (Δpitch, Δyaw, Δroll) between the remaining results and the average are all less than 1°, and (Δz, Δw) are all less than 0.1 m, then the average can be used as the calibrated external parameters after fusion; otherwise, more single-frame calibration results can be continued to be accumulated.

[0148] S2. Coarse calibration of the angles and heights of the left and right cameras.

[0149] Taking the left camera as an example (the right camera can be obtained in the same way).

[0150] Feature point matching is performed on adjacent frame images collected by the left camera.

[0151] Exemplarily, the matching method can adopt the ORB matching method or deep learning methods such as superpoint and superglue.

[0152] Based on the matching point pairs and the principle of epipolar geometry, the rotation and translation relationship RT (including the rotation matrix R and the translation vector T) of the camera movement between adjacent frames can be calculated.

[0153] According to the external parameters of the front and rear cameras calibrated in S1, the lane lines of adjacent frames are mapped to the world coordinate system, and it is judged whether the difference between the slopes of the lane lines of adjacent frames exceeds the preset slope difference threshold. If it does not exceed, it means that the vehicle is in a straight state between the current adjacent frames; otherwise, it is in a turning state.

[0154] According to the accumulated T in the straight state, the x vector in the camera coordinate system is calculated (one x vector corresponds to a pair of adjacent frames in the straight state); according to the accumulated R in the turning state, the z vector in the camera coordinate system is calculated (one z vector corresponds to a pair of adjacent frames in the turning state), and the calculation process is as follows: The x vector is the unitized representation of the T translation vector: ; The z vector is calculated based on the rotation matrix R:

[0155]

[0156]

[0157] By fusing the accumulated x and z vectors of multiple frames, the fusion method is to remove the outliers outside the range of mean ± standard deviation, and calculate the mean of the remaining vectors to obtain the fused x and z vectors.

[0158] Exemplarily, during the accumulation process of multiple frames, prior information can also be added for filtering abnormal vectors.

[0159] As an example, abnormal vectors can include x vectors and z vectors that do not satisfy the perpendicular constraint.

[0160] As another example, abnormal vectors can include x vectors that do not satisfy the positive and negative constraints; the positive and negative constraints include that the component of the x vector on the camera x-axis is positive, and its schematic diagram can be as Figure 10 shown.

[0161] According to formula (7), the rotation matrix is calculated for the fused x and z vectors, and the angles pitch, yaw, and roll of the camera can be obtained through decomposition.

[0162] R cam = [x, (x * z), z] * [X, (X * Z), Z] -1 (7) Among them, X and Z are the unit vectors [-1, 0, 0] and [0, 0, 1] in the body world coordinate system respectively.

[0163] The rough calibration of the angles of the left and right cameras has been completed through the above process. According to the initial position parameters of the left and right cameras and the calibrated extrinsic parameters of the angles, the lane lines are mapped to the world coordinate system. Based on the numerical iteration algorithm, the value of z is optimized to make the lane width equal to 15 cm, thus completing the rough calibration of the height z of the left and right cameras.

[0164] S3. Precise calibration of the extrinsic parameters of the left and right cameras.

[0165] According to S1 and S2, the extrinsic parameters of the front and rear cameras, as well as the extrinsic parameters of the angles and heights of the left and right cameras, have been obtained. However, there may still be errors in the extrinsic parameters of the left and right cameras, which need to be further corrected.

[0166] Taking the calibration of the extrinsic parameters of the left camera as an example for illustration (the same principle applies to the right camera).

[0167] According to the calibrated extrinsic parameters of the front and rear cameras (pitch, yaw, roll, h, w, z), the extrinsic parameters of the angles (pitch, yaw, roll) and height (z) of the left camera, as well as the initial position parameters (h, w) of the left camera, the three fisheye views of the front, left, and rear are converted into BEV views.

[0168] Extract ground feature matching points in the co - visible area of adjacent cameras (front - left, rear - left), as Figure 11 shown. Among them, Figure 11 from left to right are the BEV view of the left view, the BEV view of the front view in sequence.

[0169] Exemplarily, the feature matching method can include the ORB matching method or deep - learning methods such as superpoint and superglue.

[0170] Since the matching points in the co - visible area are the same feature points in the world coordinate system, therefore, they have the same world coordinates.

[0171] On the one hand, according to the calibrated extrinsic parameters of the front and rear cameras, the matching points of the front view are mapped to the world coordinate system to obtain the three - dimensional world coordinate point set P1 of the feature points of the front view. Similarly, the matching points of the rear view can be mapped to the world coordinate system to obtain the Figure 3 three - dimensional world coordinate point set P2 of the rear view. The two point sets are combined into P.

[0172] On the other hand, the matching point set p1 in the co - visible area between the left view and the front view and the matching point set p2 in the co - visible area between the left view and the rear view can be combined into an image coordinate point set p.

[0173] Since the three-dimensional coordinate point set P and the image coordinate point set p are in one-to-one correspondence, the external parameters (h, w, z, pitch, yaw, roll) of the left camera can be calculated using the PnP algorithm.

[0174] Thus, the external parameter calibration (h, w, z, pitch, yaw, roll) of the four cameras in the front, back, left, and right directions is completed.

[0175] Exemplarily, considering that in the calibration process of S1, the calibration is completed with the front camera w = 0 as the reference, only the relative position relationship of w between the front and rear cameras can be guaranteed to be correct, and the calibration in the subsequent S2 and S3 is also based on this. The w of the four cameras can also be adjusted as a whole.

[0176] Since the camera installation position is usually at the edge of the vehicle body, the w of the cameras can be adjusted as a whole by using the constraint that the w of the left and right cameras is generally symmetric.

[0177] For example, it is known that the external parameters of the calibrated cameras are w 前 , w 后 , w 左 , w 右 (the w values of the front camera, rear camera, left camera, and right camera in sequence), then Δw = (w 左 + w 右 ) / 2, and subtracting Δw from w 前 , w 后 , w 左 , w 右 respectively gives the final calibration result.

[0178] Exemplarily, the calibration effect can be as shown in Figure 12 .

[0179] The method provided by the present application has been described above. Next, the device provided by the present application will be described: Please refer to Figure 13 , which is a schematic structural diagram of a camera external parameter calibration device provided by an embodiment of the present application. As shown in Figure 13 , the camera external parameter calibration device may include: An acquisition unit, configured to acquire an image collected by a target camera; wherein, the target camera includes an in-vehicle camera installed on the periphery of the vehicle body; A first determination unit, configured to, for a specified target camera among the target cameras, determine the straight line equation of the side of the lane line in the undistorted image according to the undistorted image; wherein, the undistorted image is obtained by undistorting the image collected by the specified target camera according to the internal parameters of the specified target camera; the specified target camera includes a first target camera installed in front of the vehicle body or a second target camera installed behind the vehicle body; The first determination unit is further configured to determine the coordinates of the vanishing point in the undistorted image according to the straight line equation of the side of the lane line in the undistorted image; The second determination unit is configured to determine the roll angle of the specified target camera according to the lane line width constraint; wherein, the lane line width constraint includes that when the two sides of the lane line are projected onto the vehicle body world coordinate system, the distance between the two sides of different lane lines is the same; The second determination unit is further configured to determine the pitch angle and yaw angle of the specified target camera according to the roll angle of the specified target camera and the coordinates of the vanishing point; The third determination unit is configured to determine the z value of the specified target camera in the vehicle body world coordinate system according to the external angle parameters of the specified target camera, the initial position parameters of the specified target camera, and the lane line width; wherein, the external angle parameters include the roll angle, pitch angle and yaw angle, and the initial position parameters of the specified target camera include the initial w value, initial h value and initial z value of the specified target camera in the vehicle body world coordinate system; the initial w value and initial h value are set according to the vehicle body length and vehicle body width, and the initial z value is an empirical value; The third determination unit is further configured to, when the target camera includes the first target camera and the second target camera, determine the w value of the second target camera in the vehicle body world coordinate system by aligning the same lane lines in the first undistorted image and the second undistorted image; wherein, the first undistorted image is obtained by undistorting the image collected by the first target camera according to the internal parameters of the first target camera, and the second undistorted image is obtained by undistorting the image collected by the second target camera according to the internal parameters of the second target camera.

[0180] Exemplarily, the specific processing flow for the acquisition unit, the first determination unit, the second determination unit, and the third determination unit to implement the camera external parameter calibration can refer to the relevant descriptions in the above embodiments, and the embodiments of the present application will not be elaborated here.

[0181] An embodiment of the present application provides an electronic device, including a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor is configured to execute the machine-executable instructions to implement the camera external parameter calibration method described above.

[0182] Please refer to Figure 14, which is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present application. The electronic device may include a processor 1401 and a memory 1402 storing machine-executable instructions. The processor 1401 and the memory 1402 may communicate via a system bus 1403. Moreover, by reading and executing the machine-executable instructions corresponding to the camera extrinsic parameter calibration logic in the memory 1402, the processor 1401 may execute the camera extrinsic parameter calibration method described above.

[0183] The memory 1402 mentioned in this article can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, and so on. For example, the machine-readable storage medium can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or a combination thereof.

[0184] In some embodiments, a machine-readable storage medium is also provided, such as Figure 14 the memory 1402 in, which stores machine-executable instructions. When the machine-executable instructions are executed by a processor, the camera extrinsic parameter calibration method described above is implemented. For example, the storage medium can be ROM, RAM, CD-ROM, magnetic tapes, floppy disks, and optical data storage devices, etc.

Claims

1. A method for calibrating the external parameters of a camera, characterized in that, Including: Obtain an image captured by a target camera; wherein, the target camera includes an in-vehicle camera installed on the periphery of the vehicle body; For a specified target camera among the target cameras, determine the straight-line equation of the side of the lane line in the undistorted image according to the undistorted image; wherein, the undistorted image is obtained by undistorting the image captured by the specified target camera according to the internal parameters of the specified target camera; the specified target camera includes a first target camera installed in front of the vehicle body or a second target camera installed behind the vehicle body; Determine the coordinates of the vanishing point in the undistorted image according to the straight-line equation of the side of the lane line in the undistorted image; Determine the roll angle of the specified target camera according to the lane line width constraint; wherein, the lane line width constraint means that when the two sides of the lane line are projected onto the vehicle body world coordinate system, the distance between the two sides of different lane lines is the same; Determine the pitch angle and yaw angle of the specified target camera according to the roll angle of the specified target camera and the coordinates of the vanishing point; Determine the z value of the specified target camera in the vehicle body world coordinate system according to the external angle parameters of the specified target camera, the initial position parameters of the specified target camera, and the lane line width; wherein, the external angle parameters include the roll angle, pitch angle and yaw angle, and the initial position parameters of the specified target camera include the initial w value, initial h value and initial z value of the specified target camera in the vehicle body world coordinate system; the initial w value and initial h value are set according to the vehicle body length and vehicle body width, and the initial z value is an empirical value; When the target camera includes the first target camera and the second target camera, determine the w value of the second target camera in the vehicle body world coordinate system by aligning the same lane lines in the first undistorted image and the second undistorted image; wherein, the first undistorted image is obtained by undistorting the image captured by the first target camera according to the internal parameters of the first target camera, and the second undistorted image is obtained by undistorting the image captured by the second target camera according to the internal parameters of the second target camera.

2. The method according to claim 1, wherein The method further includes: When the target camera includes the first target camera installed in front of the vehicle body, use the multi-frame fusion method to fuse the first calibration external parameters corresponding to multiple frames of the first undistorted image to obtain the calibrated first calibration external parameters; wherein, the first calibration external parameters include the external angle parameters of the first target camera and the z external parameter of the first target camera; And / or When the target camera includes the second target camera installed behind the vehicle body, use the multi-frame fusion method to fuse the second calibration external parameters corresponding to multiple frames of the second undistorted image to obtain the calibrated second calibration external parameters; wherein, the second calibration external parameters include the external angle parameters of the second target camera, the z external parameter and w external parameter of the second target camera; Wherein, for any one of the first target camera and the second target camera, using the multi-frame fusion method to fuse the calibration external parameters corresponding to multiple frames of the undistorted image includes: For each type of calibration extrinsic parameter corresponding to the accumulated multi-frame undistorted images, delete the maximum value of the first ratio and the minimum value of the second ratio respectively; Determine the average value of the remaining values of each type of calibration extrinsic parameter respectively; When the difference between the remaining value of each type of calibration extrinsic parameter and the average value of the same type of calibration extrinsic parameter does not exceed the preset threshold, use the determined average value of each type of calibration extrinsic parameter as the calibrated calibration extrinsic parameter; Otherwise, accumulate the calibration extrinsic parameters corresponding to more frames of undistorted images until the calibrated calibration extrinsic parameters are obtained in the above manner.

3. The method according to claim 1, wherein When the target camera includes a third target camera installed on the side of the vehicle body, the method further includes: Perform feature point matching on adjacent frame images collected by the third target camera to obtain matching point pairs; According to the matching point pairs, use the principle of epipolar geometry to determine the rotation matrix R and translation vector T of the movement of the third target camera between adjacent frames; When the vehicle is in a straight state between adjacent frames, determine the x vector according to the determined translation vector T of the third target camera; When the vehicle is in a turning state between adjacent frames, determine the z vector according to the determined rotation matrix R of the third target camera; Determine the angular extrinsic parameter of the third target camera according to the x vector and the z vector; According to the angular extrinsic parameter of the third target camera, the initial position parameters of the third target camera, and the lane line width, determine the z value of the third target camera in the vehicle body world coordinate system; wherein, the initial position parameters of the third target camera include the initial w value, initial h value, and initial z value of the third target camera in the vehicle body world coordinate system.

4. The method according to claim 3, wherein The determining the angular extrinsic parameter of the third target camera according to the x vector and the z vector includes: Obtain multiple x vectors and multiple z vectors corresponding to multiple pairs of adjacent frames; Filter out abnormal vectors from the multiple x vectors and multiple z vectors; wherein, abnormal vectors include vectors with values outside the specified value range, x vectors and z vectors that do not satisfy the mutual perpendicular constraint, or x vectors that do not satisfy the positive and negative constraint; the positive and negative constraint includes: for a target camera installed on the left side of the vehicle body, the component of the x vector on the camera x-axis is positive, or, for a target camera installed on the right side of the vehicle body, the component of the x vector on the camera x-axis is negative; Determine the mean value of the filtered x vectors as the fused x vector; and determine the mean value of the filtered z vectors as the fused z vector; Determine the angular extrinsic parameter of the third target camera according to the fused x vector and the fused z vector.

5. The method according to claim 3, characterized in that The method further includes: Convert the images collected by the target camera into a bird's-eye view (BEV) according to the calibration extrinsic parameters of each target camera; Extract ground feature matching points in the common view area of the third target camera and the adjacent target camera of the third target camera according to the BEV view of the third target camera and the BEV view of the adjacent target camera of the third target camera; wherein, the adjacent target camera of the third target camera is a target camera having a common view area with the third target camera; the adjacent target camera of the third target camera includes the first target camera; Map the ground feature matching points of the first target camera to the vehicle body world coordinate system according to the calibrated extrinsic parameters of the first target camera to obtain a first set of world coordinate points; Determine an image coordinate point set according to the ground feature matching points in the common view area of the third target camera and the first target camera in the BEV view of the third target camera; Determine the accurate calibrated extrinsic parameters of the third target camera according to the one-to-one correspondence between the points in the first set of world coordinate points and the points in the image coordinate point set.

6. The method according to claim 5, wherein When the target camera includes a second target camera installed at the rear of the vehicle body and the adjacent target camera of the third target camera includes the second target camera, the method further includes: Map the ground feature matching points of the second target camera to the vehicle body world coordinate system according to the calibrated extrinsic parameters of the second target camera to obtain a second set of world coordinate points; Add the ground feature matching points in the common view area of the third target camera and the second target camera in the view of the third target camera to the image coordinate point set; The determining the accurate calibrated extrinsic parameters of the third target camera according to the one-to-one correspondence between the points in the first set of world coordinate points and the points in the image coordinate point set includes: Merge the first set of world coordinate points and the second set of world coordinate points to obtain a set of world coordinate points; Determine the accurate calibrated extrinsic parameters of the third target camera according to the one-to-one correspondence between the set of world coordinate points and the points in the image coordinate point set.

7. The method according to claim 1, wherein The target camera includes a third target camera and a fourth target camera installed on both sides of the vehicle body, and the w values of the third target camera and the fourth target camera in the vehicle body world coordinate system are symmetric about the x-axis of the vehicle body world coordinate system; The method further includes: When the first w value of the third target camera in the vehicle body world coordinate system and the second w value of the fourth target camera in the vehicle body world coordinate system are determined, determine the average value of the first w value and the second w value as the w adjustment value; Subtract the w adjustment value from the w values of the determined target cameras in the vehicle body world coordinate system respectively to obtain the final w values of the target cameras in the vehicle body world coordinate system.

8. An external camera parameter calibration device, characterized in that, Includes: An acquisition unit for acquiring images collected by a target camera; wherein, the target camera includes an in-vehicle camera installed on the periphery of the vehicle body; A first determination unit, configured to determine a straight-line equation of the side of a lane line in the undistorted image for a specified target camera in the target cameras, based on the undistorted image; wherein, the undistorted image is obtained by undistorting an image collected by the specified target camera according to the internal parameters of the target camera; the specified target camera includes a first target camera installed in front of the vehicle body or a second target camera installed behind the vehicle body; The first determination unit is further configured to determine the coordinates of a vanishing point in the undistorted image based on the straight-line equation of the side of the lane line in the undistorted image; A second determination unit, configured to determine the roll angle of the specified target camera according to a lane line width constraint; wherein, the lane line width constraint includes that when the two sides of the lane line are projected onto the vehicle body world coordinate system, the distance between the two sides of different lane lines is the same; The second determination unit is further configured to determine the pitch angle and yaw angle of the specified target camera based on the roll angle of the specified target camera and the coordinates of the vanishing point; A third determination unit, configured to determine the z value of the specified target camera in the vehicle body world coordinate system according to the external angle parameters of the specified target camera, the initial position parameters of the specified target camera, and the lane line width; wherein, the external angle parameters include the roll angle, pitch angle and yaw angle, and the initial position parameters of the specified target camera include the initial w value, initial h value and initial z value of the specified target camera in the vehicle body world coordinate system; The third determination unit is further configured to, when the target cameras include the first target camera and the second target camera, determine the w value of the second target camera in the vehicle body world coordinate system by aligning the same lane lines in a first undistorted image and a second undistorted image; wherein, the first undistorted image is obtained by undistorting an image collected by the first target camera according to the internal parameters of the first target camera, and the second undistorted image is obtained by undistorting an image collected by the second target camera according to the internal parameters of the second target camera.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory stores machine-executable instructions that can be executed by the processor, and the processor is configured to execute the machine-executable instructions to implement the method according to any one of claims 1-7.

10. A machine-readable storage medium, characterized in that, Machine-executable instructions are stored in the machine-readable storage medium, and when the machine-executable instructions are executed by the processor, the method according to any one of claims 1-7 is implemented.

Images