Camera extrinsic parameter calibration method, device, equipment and storage medium

Abstract

The present application provides a camera extrinsic calibration method, apparatus, device and storage medium. In one example, the method includes: for a designated target camera among target cameras, determining the coordinates of the vanishing point in the dedistorted image based on the straight line equation of the side of the lane line in the dedistorted image; determining the roll angle of the designated target camera based on the lane line width constraint; determining the pitch angle and yaw angle of the designated target camera based on the roll angle of the designated target camera and the coordinates of the vanishing point; determining the z value of the designated target camera in the vehicle body world coordinate system based on the angular extrinsic parameters of the designated target camera, the initial position parameters of the designated target camera, and the lane line width; determining the w value of the second target camera in the vehicle body world coordinate system by aligning the same lane line in the first dedistorted image and the second dedistorted image. This method can realize camera extrinsic calibration in natural scenes without providing the precise h / z / w extrinsic parameters of the camera.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a camera extrinsic parameter calibration method, device, equipment and storage medium. Background Art

[0002] The in-vehicle 360-degree surround view system utilizes four fisheye cameras mounted on the front, left, right, and rear of the vehicle to generate surround-view images that allow users to quickly understand the scene around the vehicle, providing visual assistance even at low speeds. With the continuous advancement of autonomous driving technology, in-vehicle cameras, as a crucial tool for environmental perception, have led to a strong demand for in-vehicle 360-degree surround view assistance systems, which are gradually being applied to an increasing number of vehicle models. In an in-vehicle surround view system, the accuracy of the camera's extrinsic parameters directly affects the stitching effect, making accurate calibration of the camera's extrinsic parameters crucial. Summary of the Invention

[0003] In view of this, the present application provides a camera extrinsic parameter calibration method, apparatus, device and storage medium.

[0004] Specifically, this application is implemented through the following technical solutions:

[0005] According to a first aspect of an embodiment of the present application, a camera extrinsic parameter calibration method is provided, comprising:

[0006] Acquire an image captured by a target camera; wherein the target camera includes a vehicle-mounted camera installed on the periphery of the vehicle body;

[0007] For a designated target camera among the target cameras, determining, based on a dedistorted image, a straight line equation of a side edge of a lane line in the dedistorted image; wherein the dedistorted image is obtained by dedistorting an image captured by the designated target camera based on an intrinsic parameter of the designated target camera; the designated target camera includes a first target camera mounted in front of the vehicle body, or a second target camera mounted behind the vehicle body;

[0008] Determining the coordinates of a vanishing point in the dedistorted image based on a straight line equation of a side edge of the lane line in the dedistorted image;

[0009] Determining the roll angle of the designated target camera based on a lane width constraint; wherein the lane width constraint includes that the distance between two sides of different lane lines is consistent when the two sides of the lane lines are projected into the vehicle body world coordinate system;

[0010] Determining the pitch angle and yaw angle of the designated target camera based on the roll angle of the designated target camera and the coordinates of the vanishing point;

[0011] Determining the z value of the designated target camera in the vehicle body world coordinate system based on the angular extrinsic parameters of the designated target camera, the initial position parameters of the designated target camera, and the lane line width; wherein the angular extrinsic parameters include the roll angle, the pitch angle, and the yaw angle; the initial position parameters of the designated target camera include the initial w value, the initial h value, and the initial z value of the designated target camera in the vehicle body world coordinate system; the initial w value and the initial h value are set based on the vehicle body width and the vehicle body length, and the initial z value is an empirical value;

[0012] When the target camera includes the first target camera and the second target camera, the w value of the second target camera in the vehicle body world coordinate system is determined by aligning the same lane line in the first dedistorted image and the second dedistorted image; wherein the first dedistorted image is obtained by dedistorting the image captured by the first target camera based on the intrinsic parameter of the first target camera, and the second dedistorted image is obtained by dedistorting the image captured by the second target camera based on the intrinsic parameter of the second target camera.

[0013] According to a second aspect of an embodiment of the present application, a camera extrinsic parameter calibration device is provided, comprising:

[0014] An acquisition unit, configured to acquire an image captured by a target camera; wherein the target camera comprises a vehicle-mounted camera installed on the periphery of the vehicle body;

[0015] a first determining unit configured to determine, for a designated target camera among the target cameras, a straight line equation of a side edge of a lane line in the dedistorted image based on the dedistorted image; wherein the dedistorted image is obtained by dedistorting an image captured by the designated target camera based on an intrinsic parameter of the target camera; and the designated target camera includes a first target camera mounted in front of the vehicle body, or a second target camera mounted behind the vehicle body;

[0016] The first determining unit is further configured to determine the coordinates of a vanishing point in the dedistorted image based on a straight line equation of a side edge of the lane line in the dedistorted image;

[0017] a second determining unit, configured to determine the roll angle of the designated target camera based on a lane width constraint; wherein the lane width constraint includes that, when two sides of the lane lines are projected onto a vehicle body world coordinate system, the distances between two sides of different lane lines are consistent;

[0018] The second determining unit is further configured to determine the pitch angle and yaw angle of the designated target camera based on the roll angle of the designated target camera and the coordinates of the vanishing point;

[0019] a third determining unit, configured to determine a z value of the designated target camera in a vehicle body world coordinate system based on an angular extrinsic parameter of the designated target camera, an initial position parameter of the designated target camera, and a lane line width; wherein the angular extrinsic parameter includes a roll angle, a pitch angle, and a yaw angle, and the initial position parameter of the designated target camera includes an initial w value, an initial h value, and an initial z value of the designated target camera in the vehicle body world coordinate system;

[0020] The third determination unit is further configured to, when the target camera includes the first target camera and the second target camera, determine the w value of the second target camera in the vehicle body world coordinate system by aligning the same lane lines in the first dedistorted image and the second dedistorted image; wherein the first dedistorted image is obtained by dedistorting the image captured by the first target camera based on the intrinsic parameters of the first target camera, and the second dedistorted image is obtained by dedistorting the image captured by the second target camera based on the intrinsic parameters of the second target camera.

[0021] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor is used to execute the machine-executable instructions to implement the method provided in the first aspect.

[0022] According to a fourth aspect of an embodiment of the present application, a machine-readable storage medium is provided, wherein the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by a processor, the method provided in the first aspect is implemented.

[0023] The technical solution provided by this application can at least bring the following beneficial effects:

[0024] By acquiring the image captured by the target camera, for the designated target camera installed in front of or behind the vehicle body, the straight line equation of the side of the lane line in the dedistorted image can be determined based on the dedistorted image, and the coordinates of the vanishing point in the dedistorted image can be determined based on the straight line equation of the side of the lane line in the dedistorted image, and the roll angle of the designated target camera can be determined based on the lane line width constraint. Furthermore, the pitch angle and yaw angle of the designated target camera can be determined based on the roll angle of the designated target camera and the coordinates of the vanishing point, and the angle extrinsic parameters of the designated target camera, the initial position parameters of the designated target camera, and, Lane line width, determine the z value of the specified target camera in the vehicle body world coordinate system, and when the target cameras include a first target camera and a second target camera, determine the w value of the second target camera in the vehicle body world coordinate system by aligning the same lane lines in the first dedistorted image and the second dedistorted image. This realizes camera extrinsic parameter calibration in natural scenes, without the need to deploy a calibration site, reducing labor and site costs. In addition, during the calibration process, rough initial h and w values ​​can be set according to the vehicle body length and width, and the initial z value can be set based on experience, without the need to provide precise h / w / z extrinsic parameter values ​​of the camera, thereby improving practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 1 is a flow chart of a camera extrinsic calibration method shown in an exemplary embodiment of the present application;

[0026] Figure 2 is a schematic diagram showing an exemplary embodiment of the present application, in which a straight line in a dedistorted image is mapped into a vehicle body world coordinate system;

[0027] Figure 3 is a schematic diagram of a calibration environment shown in an exemplary embodiment of the present application;

[0028] Figure 4 1 is a structural diagram of a camera extrinsic calibration system shown in an exemplary embodiment of the present application;

[0029] Figure 5 This is a schematic diagram of a camera capturing an image, shown in an exemplary embodiment of the present application;

[0030] Figure 6 This is a data processing flow diagram shown in an exemplary embodiment of the present application;

[0031] Figure 7 is a schematic diagram of a dedistorted image shown in an exemplary embodiment of the present application;

[0032] Figure 8 This is a schematic diagram of a front and rear camera extrinsic parameter calibration process shown in an exemplary embodiment of the present application;

[0033] Figure 9 is a schematic diagram of a coordinate system shown in an exemplary embodiment of the present application;

[0034] Figure 10 is a schematic diagram of the direction of an x ​​vector in a camera coordinate system shown in an exemplary embodiment of the present application;

[0035] Figure 11 is a schematic diagram of feature matching of a front left view shown in an exemplary embodiment of the present application;

[0036] Figure 12 This is a schematic diagram of a calibration effect shown in an exemplary embodiment of the present application;

[0037] Figure 13 1 is a schematic structural diagram of a camera extrinsic calibration device shown in an exemplary embodiment of the present application;

[0038] Figure 14 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, some technical terms involved in the embodiments of the present application are explained below.

[0040] 1. Lane Vanishing Point (Vanishing Point): This is the point where lane lines gradually vanish in the field of view. It represents the intersection of lane lines in perspective projection when they are parallel to the horizon and extend into the distance. Due to the curvature of the Earth and changes in perspective, lane lines gradually converge to a single point from near to far, usually at the far end of the field of view.

[0041] In order to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application are further described in detail below with reference to the accompanying drawings.

[0042] It should be noted that the serial numbers of the steps in the embodiments of the present application do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0043] See Figure 1 , is a flow chart of a camera extrinsic parameter calibration method provided in an embodiment of the present application, such as Figure 1 As shown, the camera extrinsic parameter calibration method may include the following steps:

[0044] Step S100: Acquire an image captured by a target camera; wherein the target camera includes a vehicle-mounted camera installed on the periphery of a vehicle body.

[0045] Exemplarily, the target camera may include a plurality of vehicle-mounted cameras (which may be referred to as vehicle-mounted surround view cameras) installed on the periphery of the vehicle body, and the total field of view of the plurality of vehicle-mounted cameras covers all around the vehicle body (360-degree surround view).

[0046] For example, an image of a scene outside the vehicle may be captured by a target camera.

[0047] For example, when the vehicle is in a natural scene (such as driving on a road), an image of a scene outside the vehicle can be captured by a target camera.

[0048] Step S110: For a designated target camera among the target cameras, determine, based on a dedistorted image, a straight line equation of a side edge of the lane line in the dedistorted image; wherein the dedistorted image is obtained by dedistorting an image captured by the designated target camera based on an intrinsic parameter of the designated target camera.

[0049] For example, the designated target camera may include a target camera installed in front of the vehicle body (which may be referred to as a first target camera), or a target camera installed behind the vehicle body (which may be referred to as a second target camera).

[0050] In an embodiment of the present application, for a specified target camera among the target cameras, distortion correction can be performed on the image captured by the specified target camera based on the internal parameters of the specified target camera to obtain a dedistorted image, and the straight line equation of the side of the lane line in the dedistorted image can be determined.

[0051] Taking the designated target camera as the first target camera as an example, for the first target camera installed in front of the vehicle body, distortion correction can be performed on the image captured by the first target camera based on the internal parameters of the first target camera to obtain a dedistorted image (which can be called the first dedistorted image). The straight line equation of the side of the lane line in the first dedistorted image is also determined.

[0052] In one example, edge point extraction and line fitting of a straight line may be performed in the dedistorted image to obtain a straight line equation in the dedistorted image (a straight line equation on the side of the lane line).

[0053] For example, the method of extracting edge points may include but is not limited to a Canny operator, a Sobel operator, or a Laplace operator.

[0054] For example, the straight line fitting method may include but is not limited to the least squares method, the gradient descent method, etc.

[0055] In another example, a deep learning network can be used to implement line extraction and determine the equation of the line in the dedistorted image.

[0056] For example, a schematic diagram of mapping a straight line (such as the side of a lane line) in a dedistorted image to a vehicle body world coordinate system can be as follows: Figure 2As shown; among them, l1 and l2 correspond to the same lane line, and l3 and l4 correspond to the same lane line.

[0057] Exemplarily, the dedistorted image may include at least two lane lines, that is, at least two pairs of straight line equations (one lane line corresponds to a pair of straight line equations).

[0058] Step S120: Determine the coordinates of the vanishing point in the dedistorted image based on the equation of the line on the side of the lane line in the dedistorted image.

[0059] For example, the intersection points between the lines can be obtained based on the equations of the lines on the sides of the multiple lane lines in the dedistorted image, and the coordinates of the vanishing point in the dedistorted image can be obtained by averaging the multiple intersection points.

[0060] Taking the first target camera as an example, the intersection points between the lines can be obtained based on the equations of the lines on the sides of multiple lane lines in the first dedistorted image. By averaging the multiple intersection points, the coordinates of the vanishing point in the first dedistorted image (which can be called the first vanishing point) are obtained.

[0061] Step S130: Determine the roll angle of the designated target camera based on the lane width constraint; wherein the lane width constraint includes that when the two sides of the lane line are projected into the vehicle body world coordinate system, the distance between the two sides of different lane lines is consistent.

[0062] In the embodiment of the present application, it is considered that in actual scenarios, the width of the lane line is fixed. For example, the national standard lane line width is 15 cm, that is, the width of different lane lines is the same.

[0063] Therefore, when the lane lines in the dedistorted image are projected into the vehicle body world coordinate system, the widths of different lane lines (which can be represented by the distance between the two sides of the vehicle line) are consistent.

[0064] Exemplarily, the consistency of widths of different lane lines may include that the widths of different lane lines are the same, or there is a tolerable deviation.

[0065] Accordingly, when the equation of the line on the side of the lane line in the dedistorted image is determined, the roll angle (which can be denoted as roll) of the specified target camera can be determined based on the lane line width constraint.

[0066] For example, the value of roll can be optimized based on a numerical iterative algorithm so that Figure 2 The distance between l1 and l2 is the same as the distance between l3 and l4.

[0067] Step S140: Determine the pitch angle and yaw angle of the designated target camera based on the roll angle of the target camera and the coordinates of the vanishing point.

[0068] In an embodiment of the present application, when the roll angle and the coordinates of the vanishing point of the specified target camera are determined in the manner described in the above embodiment, the pitch angle (which can be recorded as pitch) and yaw angle (which can be recorded as yaw) of the specified target camera can be determined based on the roll angle and the coordinates of the vanishing point of the specified target camera. The specific implementation process can be explained below in conjunction with specific examples.

[0069] At this point, the angular external parameters of the specified target camera are determined, including roll, pitch, and yaw.

[0070] Step S150: Determine the z value of the designated target camera in the vehicle body world coordinate system based on the angular extrinsic parameters of the designated target camera, the initial position parameters of the designated target camera, and the lane line width; wherein the angular extrinsic parameters include the roll angle, the pitch angle, and the yaw angle, and the initial position parameters of the designated target camera include the initial w value, the initial h value, and the initial z value of the designated target camera in the vehicle body world coordinate system.

[0071] In an embodiment of the present application, when the angle extrinsic parameters of the designated target camera are determined, the lane line can be mapped to the world coordinate system based on the angle extrinsic parameters of the designated target camera and the initial position parameters of the designated target camera. Based on the lane line width (such as the national lane line standard width of 15 cm), the value of z is optimized based on a numerical iterative algorithm so that the width of the lane line meets the requirements (such as a width of 15 cm) when mapped to the world coordinate system, thereby determining the z value of the designated target camera in the world coordinate system of the vehicle body.

[0072] At this point, the angle extrinsic parameters and z value of the specified target camera are determined through the above method, the w value and h value are included in the initial position parameters, and the extrinsic parameter calibration of the specified target camera is completed.

[0073] It should be noted that, in the embodiment of the present application, the w value and h value in the initial position parameters of the target camera can be set manually based on the width and length of the vehicle where the target camera is located without the need for precise measurement. The z value can be an empirical value, for example, z=1

[0074] For example, since the target camera installed in front of the vehicle body is usually installed at the edge of the vehicle body, in the vehicle body world coordinate system established with the center of the vehicle body as the coordinate origin, the h value (coordinate value in the length direction of the vehicle body) of the target camera can be set to -H / 2 (H is the length of the vehicle body), and the w value can be set to 0.

[0075] Step S160: When the target camera includes a first target camera and a second target camera, determine the w value of the second target camera in the vehicle body world coordinate system by aligning the same lane lines in the first dedistorted image and the second dedistorted image.

[0076] In the embodiment of the present application, considering that when the vehicle is driving normally on the road, the lane line closest to the left side of the vehicle body in the view captured by the front camera and the lane line closest to the left side of the vehicle body in the view captured by the rear camera generally belong to the same lane line (the same applies to the lane line closest to the right side of the vehicle body). Therefore, the w value of the rear camera can be determined by aligning the view of the front camera with the same lane line in the view of the rear camera.

[0077] Accordingly, by aligning the same lane lines in the first and second dedistorted images, the w value of the second target camera can be optimized based on a numerical iterative algorithm (the initial w value is the w value in the initial position parameters of the second target camera) with the w value of the first target camera as a reference, to obtain the optimized w value of the second target camera.

[0078] It can be seen that in Figure 1 In the method flow shown, by acquiring the image captured by the target camera, for the designated target camera installed in front of the vehicle body or installed at the rear of the vehicle body, the straight line equation of the side of the lane line in the dedistorted image can be determined based on the dedistorted image, and the coordinates of the vanishing point in the dedistorted image can be determined based on the straight line equation of the side of the lane line in the dedistorted image, and the roll angle of the designated target camera can be determined based on the lane line width constraint. Furthermore, the pitch angle and yaw angle of the designated target camera can be determined based on the roll angle of the designated target camera and the coordinates of the vanishing point, and the angle extrinsic parameters of the designated target camera and the initial position parameters of the designated target camera can be determined. , and the lane line width, determine the z value of the specified target camera in the vehicle body world coordinate system, and when the target camera includes a first target camera and a second target camera, determine the w value of the second target camera in the vehicle body world coordinate system by aligning the same lane line in the first dedistorted image and the second dedistorted image, thereby realizing camera extrinsic parameter calibration in natural scenes, without the need to deploy a calibration site, reducing labor and site costs, and during the calibration process, rough initial h and w values ​​can be set according to the vehicle body length and width, and the initial z value can be set according to experience, without the need to provide precise h / w / z extrinsic parameter values ​​of the camera, thereby improving practicality.

[0079] In some embodiments, the camera extrinsic calibration solution provided by the embodiments of the present application may further include:

[0080] In a case where the target camera includes a first target camera installed in front of the vehicle body, first calibration extrinsic parameters corresponding to multiple frames of first dedistorted images are fused using a multi-frame fusion method to obtain calibrated first calibration extrinsic parameters; wherein the first calibration extrinsic parameters include an angle extrinsic parameter of the first target camera and a z extrinsic parameter of the first target camera;

[0081] and / or,

[0082] When the target camera includes a second target camera mounted on the rear of the vehicle body, a multi-frame fusion method is used to fuse the second calibration extrinsic parameters corresponding to the multiple frames of the second dedistorted image to obtain the calibrated second calibration extrinsic parameters; wherein the second calibration extrinsic parameters include the angle extrinsic parameters of the second target camera, as well as the z extrinsic parameters and the w extrinsic parameters of the second target camera.

[0083] For example, considering that the calibration result obtained based on a single frame image may have some errors due to certain specific reasons, for example, when the vehicle is driving, the vehicle body may not be parallel to the lane line, but the above calibration scheme is based on the vehicle body being parallel to the lane line ( Figure 2 Therefore, in order to improve the accuracy of the calibration results, the calibration results of the first target camera can be optimized by multi-frame fusion. If there is a second target camera, the calibration results of the second target camera can also be optimized by multi-frame fusion.

[0084] It should be noted that since the camera is usually installed at the edge of the vehicle body, the h of the front camera and the rear camera is generally more accurate (half of the vehicle body length H) and does not need to be optimized.

[0085] In one example, for any one of the first target camera and the second target camera, fusion of calibration extrinsic parameters corresponding to multiple frames of dedistorted images is performed using a multi-frame fusion method, including:

[0086] For each type of calibration extrinsic parameter corresponding to the accumulated multiple frames of dedistorted images, the maximum value of the first ratio and the minimum value of the second ratio are deleted respectively;

[0087] Determine the average value of the remaining values ​​of each type of calibration external parameter;

[0088] When the difference between the residual value of each type of calibration external parameter and the average value of the same type of calibration external parameters does not exceed a preset threshold, the determined average value of each type of calibration external parameter is used as the calibration external parameter after calibration;

[0089] Otherwise, more frames of calibration extrinsic parameters corresponding to the dedistorted images are accumulated until the calibrated calibration extrinsic parameters are obtained according to the above method.

[0090] For example, take multi-frame fusion for the first target camera as an example.

[0091] Calibration extrinsic parameters corresponding to multiple frames of the first dedistorted image can be accumulated (for the method of determining the calibration extrinsic parameters corresponding to a single frame of the first dedistorted image, refer to the relevant description in the above embodiment), and the maximum value of the first ratio and the minimum value of the second ratio of each type of calibration extrinsic parameter (first calibration extrinsic parameter) corresponding to the accumulated multiple frames of the first dedistorted image are deleted respectively.

[0092] For example, taking roll as an example, the rolls corresponding to the accumulated multiple frames of the first dedistorted images can be sorted by size (taking sorting from large to small as an example), and the roll values ​​of the first 10% (taking the first ratio as 10% as an example) and the roll values ​​of the last 10% (taking the second ratio as 10% as an example) after sorting can be deleted.

[0093] For the remaining calibration external parameters of each type, the average value of the remaining values ​​of each type of calibration external parameter can be determined respectively, and it can be determined whether the difference between the remaining values ​​of each type of calibration external parameter and the average value of the calibration external parameter of the same type exceeds a preset threshold (which can be set according to actual needs).

[0094] For example, the thresholds for pitch, yaw, and raw may be 1°, and the thresholds corresponding to the z value and the w value may be 1 meter.

[0095] When the difference between the residual value of each type of calibration external parameter and the average value of the same type of calibration external parameters does not exceed a preset threshold, the determined average value of each type of calibration external parameter can be used as the calibration external parameter after calibration.

[0096] Otherwise, that is, it is determined that the difference between the remaining value of at least one type of calibration extrinsic parameter and the average value of the calibration extrinsic parameters of the same type exceeds a preset threshold. For example, among the remaining calibration extrinsic parameters, the absolute value of the difference between the pitch corresponding to a frame of dedistorted image and the average value of the pitch determined according to the above method exceeds 1°. The calibration extrinsic parameters corresponding to more frames of dedistorted images can be accumulated until the calibrated calibration extrinsic parameters are obtained according to the above method.

[0097] It should be noted that in order to avoid the situation where the calibrated external parameters cannot be determined in actual applications, the calibration stop conditions can be set in advance, for example, the maximum number of calibration times can be set. When the pre-set calibration stop conditions are reached, if the calibrated external parameters are still not obtained, the exception handling process can be entered, for example, an alarm can be issued and manual intervention can be performed. The specific implementation will not be described here.

[0098] In some embodiments, when the target camera includes a third target camera installed on the side of the vehicle body, the camera extrinsic parameter calibration solution provided in the embodiment of the present application may further include:

[0099] Perform feature point matching on adjacent frame images captured by the third target camera to obtain matching point pairs;

[0100] Based on the obtained matching point pairs, the rotation matrix R and translation vector T of the third target camera movement between adjacent frames are determined using the principle of epipolar geometry;

[0101] When the vehicle is in a straight-moving state between adjacent frames, determining the x vector according to the determined translation vector T of the third target camera;

[0102] When the vehicle is in a turning state between adjacent frames, determining the z vector according to the determined rotation matrix R of the third target camera;

[0103] Determine the angular extrinsic parameters of the third target camera based on the determined x vector and z vector;

[0104] The z value of the third target camera in the vehicle body world coordinate system is determined based on the angular extrinsic parameters of the third target camera, the initial position parameters of the third target camera, and the lane line width; wherein the initial position parameters of the third target camera include the initial w value, initial h value, and initial z value of the third target camera in the vehicle body world coordinate system.

[0105] For example, for a target camera (which may be referred to as a third target camera) installed on the side (left or right) of a vehicle body, the calibration of angle extrinsic parameters may be achieved by matching feature points of adjacent frames.

[0106] It should be noted that, for the target camera on the left side of the vehicle body, and the target camera on the right side of the vehicle body, a similar calibration method can be used to perform extrinsic parameter calibration.

[0107] Exemplarily, feature point matching may be performed on adjacent frame images captured by the third target camera to obtain matching point pairs.

[0108] The feature point matching method may include but is not limited to: ORB matching method (a feature detection and description algorithm) or deep learning methods such as superpoint and superglue.

[0109] For example, based on the obtained matching point pairs, the rotation and translation relationship of the third target camera movement between adjacent frames (which can be represented by the rotation matrix R and the translation matrix T) can be determined using the epipolar geometry principle.

[0110] For example, when the vehicle is in a straight-moving state between adjacent frames, the x vector may be determined according to the determined translation vector T of the third target camera.

[0111] When the vehicle is in a turning state between adjacent frames, the z vector may be determined according to the determined rotation matrix R.

[0112] Exemplarily, the process of determining the x-vector and the z-vector may be as follows:

[0113] The x vector is the normalized representation of the T translation vector: ;

[0114] The z vector is calculated based on the rotation matrix R:

[0115]

[0116]

[0117]

[0118] Exemplarily, whether the vehicle is in a straight-moving state or a turning state can be determined based on the difference between the slopes of the same lane line in adjacent frames when mapped to the vehicle body world coordinate system; when the difference between the slopes of the same lane line between adjacent frames (which can be the absolute value of the difference) is less than a preset slope difference threshold (which can be set according to the actual scenario), it is determined that the vehicle is in a straight-moving state; otherwise, it is determined that the vehicle is in a turning state.

[0119] Exemplarily, the rotation matrix of the third target camera may be determined based on the x vector and the z vector, and the angular extrinsic parameters of the third target camera may be obtained by decomposing the rotation matrix.

[0120] For example, R cam = [x, (x*z), z] * [X0, (X0*Z0), Z0] -1

[0121] Among them, R can is the rotation matrix of the third target camera, X0 and Z0 are the unit vectors [-1, 0, 0] and [0, 0, 1] in the vehicle body world coordinate system respectively.

[0122] At this point, the angle extrinsic parameters of the third target camera are determined.

[0123] The z value of the third target camera in the vehicle body world coordinate system can be determined based on the angular external parameters of the third target camera, the initial position parameters of the third target camera, and the lane line width. For its specific implementation, please refer to the relevant description of the determination of the z value of the first target camera. The embodiments of this application will not be repeated here.

[0124] In one example, determining the angle extrinsic parameter of the third target camera based on the x vector and the z vector may include:

[0125] Obtain multiple x vectors and multiple z vectors corresponding to multiple pairs of adjacent frames;

[0126] Filter outlier vectors for multiple x-vectors and multiple z-vectors. Outlier vectors include vectors with values ​​outside a specified range, x-vectors and z-vectors that do not satisfy the mutual perpendicularity constraint, or x-vectors that do not satisfy the positive and negative constraints. The positive and negative constraints include the x-vector component on the camera x-axis being positive.

[0127] Determine the mean of the filtered x vectors as the fused x vector; and determine the mean of the filtered z vectors as the fused z vector;

[0128] The angular extrinsic parameter of the third target camera is determined based on the fused x vector and the fused z vector.

[0129] For example, in order to improve the accuracy of the angular extrinsic parameters of the third target camera, in the process of determining the x vector and the z vector, the fused x vector and z vector can be determined based on a multi-frame fusion method, and the angular extrinsic parameters of the third target camera can be determined based on the fused x vector and z vector.

[0130] Exemplarily, multiple x vectors and multiple z vectors corresponding to multiple pairs of adjacent frames may be obtained.

[0131] The implementation method of determining the x vector or z vector of adjacent frames may refer to the relevant description in the above process.

[0132] For the multiple x vectors and the multiple z vectors corresponding to the multiple pairs of adjacent frames obtained, abnormal vector filtering can be performed.

[0133] As an example, an exception vector may include a vector whose values ​​are outside a specified value range.

[0134] For example, taking the x vector as an example, for the multiple x vectors obtained, the mean ± standard deviation of the multiple x vectors can be determined as the upper and lower limits of the specified value range to obtain the specified value range, and the x vectors in the multiple x vectors whose values ​​are outside the specified value range are determined as abnormal vectors.

[0135] As another example, outlier vectors may include x-vectors and z-vectors that do not satisfy a mutual perpendicularity constraint.

[0136] For example, since the x vector can be understood as the vehicle's forward direction and the z vector can be understood as the ground plane normal vector, there is a constraint condition that the x and z vectors are perpendicular to each other.

[0137] As yet another example, an abnormal vector may include an x-vector that does not satisfy a positive / negative constraint; the positive / negative constraint includes that the component of the x-vector on the camera x-axis is positive.

[0138] For example, based on the known forward state of the vehicle, constraints are imposed on the positive and negative values ​​of the x and z vectors. For example, the x vector represents the vehicle's forward direction. For the left camera, the optical center of the camera coordinate system is in the positive z-axis direction, the x axis is parallel to the x axis of the image coordinate system, and the y axis is parallel to the y axis of the image coordinate system. Therefore, the component of the x vector on the x axis of the camera coordinate system must be positive, that is, x[0]>0. Here, x[0] represents the first row of the x vector and is the component of the x vector on the x axis of the camera coordinate system.

[0139] Similarly, for the right camera, the component of the x vector on the x-axis of the camera coordinate system must be negative, that is, x[0]<0.

[0140] For example, when the abnormal vectors are filtered, the mean of the filtered x vectors can be determined as the fused x vector; and the mean of the filtered z vectors can be determined as the fused z vector. Then, the angular extrinsic parameters of the third target camera can be determined based on the fused x vector and the fused z vector.

[0141] In one example, the camera extrinsic calibration solution provided by the embodiment of the present application may further include:

[0142] According to the calibration external parameters of each target camera, the target camera captured image is converted into BEV view;

[0143] Extracting ground feature matching points within a common viewing area between the third target camera and its neighboring target cameras based on the BEV view of the third target camera and the BEV views of the third target camera's neighboring target cameras; wherein the neighboring target cameras of the third target camera are target cameras that have a common viewing area with the third target camera; and the neighboring target cameras of the third target camera include the first target camera;

[0144] Based on the calibrated extrinsic parameters of the first target camera, the ground feature matching points of the first target camera are mapped to the vehicle body world coordinate system to obtain a first world coordinate point set;

[0145] Determine an image coordinate point set based on ground feature matching points within a common viewing area of ​​the third target camera and the first target camera in the BEV view of the third target camera;

[0146] According to the one-to-one correspondence between the points in the first world coordinate point set and the points in the image coordinate point set, the precise calibration extrinsic parameters of the third target camera are determined.

[0147] For example, in actual scenarios, in order to achieve 360-degree surround view, the target camera on the side of the vehicle body usually has a common viewing area with the target camera in front of the vehicle body, that is, the field of view coverage overlaps.

[0148] In order to determine the external parameters of the side camera of the vehicle body more accurately, the external parameters of the target camera on the side of the vehicle body can be updated according to the matching points of the common viewing area between the target camera on the side of the vehicle body and the target camera in front of the vehicle body.

[0149] Accordingly, based on the calibration extrinsic parameters of each target camera (such as the first target camera and the third target camera mentioned above), the image captured by the target camera is converted into a BEV (Bird Eye View) view.

[0150] Ground feature matching points may be extracted within a common viewing area between the third target camera and its adjacent target cameras based on the BEV view of the third target camera and the BEV views of the third target camera's adjacent target cameras.

[0151] Exemplarily, feature matching methods may include but are not limited to: ORB matching method or deep learning methods such as superpoint and superglue.

[0152] The adjacent target cameras of the third target camera include the first target camera.

[0153] When the ground feature matching points within the common viewing area of ​​the third target camera and the first target camera are determined, on the one hand, the ground feature matching points of the first target camera can be mapped to the world coordinate system of the vehicle body based on the calibrated external parameters of the first target camera to obtain the corresponding world coordinate point set (which can be called the first world coordinate point set).

[0154] On the other hand, the image coordinate point set may be determined based on the ground feature matching points within the common viewing area of ​​the third target camera and the first target camera in the BEV view of the third target camera.

[0155] Exemplarily, the above-mentioned ground feature matching points in the BEV view can be mapped to a dedistorted image (which can be called a third dedistorted image) based on the intrinsic parameters of the third target camera and the calibrated extrinsic parameters of the third target camera obtained in the above manner, and the image coordinate point set can be determined based on the ground feature matching points in the third dedistorted image.

[0156] Furthermore, the precise calibration extrinsic parameters of the third target camera can be determined based on the one-to-one correspondence between the points in the first world coordinate point set and the points in the image coordinate point set.

[0157] For example, the precise calibration extrinsic parameters of the third target camera can be determined by using a pnp (Perspective-n-Point) algorithm based on points in the first world coordinate point set and points in the image coordinate point set.

[0158] In one example, when the target camera includes a second target camera installed at the rear of the vehicle body, and the target camera adjacent to the third target camera includes the second target camera, the method may further include:

[0159] Based on the calibrated extrinsic parameters of the second target camera, the ground feature matching points of the second target camera are mapped to the vehicle body world coordinate system to obtain a second world coordinate point set;

[0160] Adding ground feature matching points within the common viewing area of ​​the third target camera and the second target camera in the view of the third target camera to the image coordinate point set;

[0161] Determining the precise calibration extrinsic parameters of the third target camera based on the one-to-one correspondence between the points in the first world coordinate point set and the points in the image coordinate point set may include:

[0162] Merging the first world coordinate point set and the second world coordinate point set to obtain a world coordinate point set;

[0163] According to the one-to-one correspondence between the world coordinate point set and the points in the image coordinate point set, the precise calibration extrinsic parameters of the third target camera are determined.

[0164] For example, when the target cameras include a target camera installed at the rear of the vehicle body (such as the second target camera mentioned above), the second target camera usually also has a common viewing area with the third target camera.

[0165] In this case, in order to make the calibration extrinsic parameters of the third target camera more accurate, the ground feature matching points within the common viewing area of ​​the second target camera and the third target camera may also be added to the matching.

[0166] Accordingly, on the one hand, the ground feature matching points of the second target camera can be mapped to the vehicle body world coordinate system based on the calibration extrinsic parameters of the second target camera to obtain a second world coordinate point set;

[0167] On the other hand, ground feature matching points in the common viewing area of ​​the third target camera and the second target camera in the view of the third target camera can be added to the image coordinate point set;

[0168] Furthermore, the first world coordinate point set and the second world coordinate point set can be merged to obtain a world coordinate point set, and the precise calibration extrinsic parameters of the third target camera can be determined based on the one-to-one correspondence between the world coordinate point set and the points in the image coordinate point set.

[0169] In some embodiments, the target cameras include a third target camera and a fourth target camera installed on both sides of the vehicle body, and the w values ​​of the third target camera and the fourth target camera in the vehicle body world coordinate system are symmetrical with respect to the x-axis of the vehicle body world coordinate system;

[0170] The camera extrinsic parameter calibration solution provided in the embodiment of the present application may also include:

[0171] After determining a first w value of the third target camera in the vehicle body world coordinate system and a second w value of the fourth target camera in the vehicle body world coordinate system, determine an average of the first w value and the second w value as a w adjustment value;

[0172] The w value of each target camera in the vehicle body world coordinate system is respectively subtracted from the w adjustment value to obtain a final w value of each target camera in the vehicle body world coordinate system.

[0173] For example, since the camera is usually installed at the edge of the vehicle body, the w values ​​of the cameras installed on both sides of the vehicle body in the vehicle body world coordinate system are usually symmetrical relative to the x-axis of the vehicle body world coordinate system (that is, the distances from the cameras installed on both sides of the vehicle body to the x-axis-z-axis plane of the vehicle body world coordinate system are usually the same). Therefore, the w values ​​of each calibrated target camera can be optimized and adjusted based on this specific pair.

[0174] Exemplarily, when the target cameras include a third camera and a fourth camera installed on both sides of the vehicle body, and the w values ​​of the third target camera and the fourth target camera in the vehicle body world coordinate system are symmetrical with respect to the x-axis of the vehicle body world coordinate system, the average value of the determined w value of the third target camera in the vehicle body world coordinate system (which can be called the first w value) and the determined w value of the fourth target camera in the vehicle body world coordinate system (which can be called the second w value) can be determined as the w adjustment value, and the w adjustment value can be subtracted from the determined w value of each target camera in the vehicle body world coordinate system to obtain the final w value of each target camera in the said vehicle body world coordinate system.

[0175] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described below with reference to specific examples.

[0176] This embodiment provides a method for calibrating the extrinsic parameters of a vehicle-mounted surround-view camera. This method uses lane markings and feature points in natural scenes while the vehicle is driving on the road to complete the camera's extrinsic calibration. Compared to calibration based on a calibration grid, natural scene-based calibration reduces deployment costs and improves the practicality of the solution. Furthermore, this solution does not require initial extrinsic parameters, reducing reliance on them and improving the ease of practical application.

[0177] For example, the calibration environment diagram can be as follows: Figure 3 shown.

[0178] In this embodiment, the structural diagram of the camera extrinsic calibration system can be as follows: Figure 4As shown, it includes an image acquisition unit, a data transmission unit, a data processing unit, a vehicle-mounted panoramic image generation unit and an image display unit.

[0179] The image acquisition unit is composed of four or more on-board cameras distributed on the front, rear, left, and right sides of the vehicle body, and the internal parameters of the on-board cameras have been calibrated.

[0180] Taking the image acquisition unit including the front camera, the right camera, the left camera, and the rear camera as an example, the schematic diagram of image acquisition by each camera can be as follows: Figure 5 As shown (from left to right are schematic diagrams of images captured by the front camera, right camera, left camera, and rear camera).

[0181] The data transmission unit obtains the image data and camera internal parameter information collected by the image acquisition unit and transmits them to the data processing unit.

[0182] The data processing unit is the key processing unit for the vehicle-mounted surround view external parameter calibration. The processing flow chart is as follows Figure 6 shown.

[0183] The vehicle-mounted surround view panoramic image generation unit generates a lookup table between the vehicle-mounted surround view panoramic image and the front, left, right, and rear views based on the external parameters calculated by the data processing unit and the camera's internal parameters. It also generates fusion weight tables between the front view and the left view, the left view and the rear view, the rear view and the right view, and the right view and the front view. The vehicle-mounted surround view panoramic image is obtained based on the original images from the four view cameras and the lookup table.

[0184] The image display unit presents a seamless panoramic image to the user after calibrating the external parameters.

[0185] In this embodiment, Figure 6 As shown, the data processing flow may include:

[0186] S1. Calibration of external parameters of front and rear cameras.

[0187] S2. Rough calibration of left and right camera angles and heights.

[0188] S3. Accurate calibration of the extrinsic parameters of the left and right cameras.

[0189] The specific implementation details of the data processing flow are described below.

[0190] For example, the initial position parameters of the four cameras can be calculated based on the known vehicle body length and width H and W (as shown in Table 1), and then the camera extrinsic parameters can be calibrated according to the data processing flow.

[0191] Table 1

[0192]

[0193] S1. Front and rear camera calibration.

[0194] For example, the front and rear views can be subjected to distortion correction based on the input front and rear views and the front and rear camera internal parameters to obtain dedistorted images (such as the first dedistorted image and the second dedistorted image mentioned above). The edge points of the straight line are extracted and the straight line is fitted in the dedistorted image to obtain the straight line equation in the dedistorted image. The schematic diagram can be shown as follows: Figure 7 shown.

[0195] For example, the method for extracting edge points may adopt the Canny operator, the Sobel operator, the Laplace operator, etc., and the linear fitting may adopt the least square method, the gradient descent method, etc.

[0196] It should be noted that a deep learning network can also be used to implement line extraction, and this embodiment of the present application does not limit this.

[0197] The intersection points between the straight lines can be obtained based on the fitted straight lines, and the vanishing point can be obtained by averaging multiple intersection points.

[0198] For example, a single view (front view or rear view) contains at least two lane lines, that is, two pairs of straight line equations. Examples can be as follows: Figure 2 shown.

[0199] The front and rear camera extrinsic parameters can be calibrated based on the vanishing points and the lane line lines fitted in the dedistorted image. The process is as follows: Figure 8 shown.

[0200] Below Figure 8 The following figure illustrates the calibration process of front and rear camera extrinsic parameters.

[0201] First Figure 9 The coordinate system shown in the figure explains the conversion between the vehicle body world coordinate system Ow-XwYwZw and the camera coordinate system Oc-XcYcZ.

[0202] like Figure 9 As shown, 0c-XcYcZc is the camera coordinate system (front camera or rear camera), 0w-XwYwZw is the vehicle body world coordinate system, and 0cw-XcwYcwZcw is also a world coordinate system (auxiliary world coordinate system), which takes the intersection of the camera optical axis and the ground as the coordinate system origin.

[0203] The body world coordinate system Ow-XwYwZw can be converted to the camera coordinate system Oc-XcYcZ in the following way:

[0204] 1) Rotate yaw+ / π2 around Zw (a positive angle corresponds to clockwise rotation) to make the direction Ow-XwYwZw consistent with the coordinate system Ocw-XcwYcwZcw;

[0205] 2) Rotate -pitch along the Xw axis so that the Yw direction is consistent with the Zc direction;

[0206] 3) Rotate -roll along the Yw axis so that the Xw direction is consistent with the Xc direction and the Zw direction is consistent with the -Yc direction;

[0207] 4) Rotate -90 degrees along the Xw axis, the Zw direction is consistent with the Zc direction, and the Yw direction is consistent with the Yc direction;

[0208] 5) The coordinate system is translated by T so that the coordinate system Ow-XwYwZw completely coincides with the camera coordinate system Oc-XcYcZc.

[0209] For example, the conversion formula is as follows:

[0210] (1)

[0211] (2)

[0212] Among them, R 33 Represents a 3x3 rotation matrix, T represents a 3x1 translation vector (also called a translation matrix) (also represented as T 31 ), pitch, yaw, and roll represent the pitch, yaw, and roll angles of the camera respectively, and (cam_x, cam_y, cam_z) represent the installation position of the camera in the world coordinate system of the vehicle body.

[0213] The transformation relationship between the points on the dedistorted image and the world coordinate system is shown in Equation (3). Therefore, according to Equations (1)-(3), and Z w = 0, the homography matrix H between the vehicle body world coordinate system and the camera distortion correction map can be derived, as shown in Equation (4).

[0214] (3)

[0215] (4)

[0216] Where fx, fy, cx, and cy represent the camera’s intrinsic focal length and principal point, and r represents the rotation matrix R. 33 The value in , t represents the translation vector T 31 The values ​​in (u, v) represent the pixel coordinates in the image, (X w , Y w ) represents the coordinates in the vehicle body world coordinate system.

[0217] For example, the straight line l1-l4 is in the vehicle body world coordinate system O w Down and X w When the coordinate axes are parallel, the straight line is parallel to the Y axis. cwThe angle between the Y axis and the Y axis is the yaw angle, and the points on the straight line can be expressed as: (Y cw *tan(yaw),Y cw , 0). The vanishing point coordinates on the dedistorted image are when Y cw ->∞. The values ​​of u0, v0 when ->∞. Taking the limit of equations (1)-(4) together, we get:

[0218] (5)

[0219] According to formula (5), when the roll angle roll, the vanishing point coordinates (u0, v0) and the camera intrinsic parameters (fx, fy, cx, cy) are known, the pitch angle and yaw angle of the camera can be calculated as follows:

[0220] (6)

[0221] Since the intersection of the lines l1-l4 in the front view is the vanishing point, the vanishing point is known. The camera's rotation angle is only related to the roll angle, so only the unknown value, roll, needs to be solved to complete the calibration of the angle extrinsic parameters (pitch, yaw, and roll). Based on the above formula, we can derive the relationship between roll, pitch, and yaw.

[0222] For example, the value of roll can be optimized based on a numerical iterative algorithm so that Figure 2 The distance between the parallel lines l1-l2 is equal to the distance between the parallel lines l3-l4 (the national standard lane line width is equal).

[0223] The pitch and yaw are calculated according to formula (6), and the extrinsic calibration of the front and rear camera angles is completed.

[0224] Since the national standard lane width is known to be 15 cm, the lane lines are mapped to the world coordinate system based on the initial position parameters of the front and rear cameras and the calibrated angle extrinsic parameters. The z value is optimized based on a numerical iterative algorithm to make the lane line width equal to 15 cm, completing the z height calibration of the front and rear cameras.

[0225] According to the constraint that the lane lines in the front and rear views are the same lane line, the camera's w is optimized based on a numerical iterative algorithm so that the front and rear lane lines are aligned, completing the calibration of the camera's w extrinsic parameter.

[0226] At this point, the single-frame calibration of the front camera's pitch, yaw, roll, z, and the rear camera's pitch, yaw, roll, z, and w has been completed.

[0227] Since the camera is usually installed at the edge of the vehicle body, it can be considered that h for the front and rear cameras is half of the vehicle length H, which is a relatively accurate position and does not require further optimization.

[0228] The w values ​​of the front and rear cameras are calibrated using the lane alignment operation with the front camera's w = 0 as the reference. This only ensures the correct relative position between the two. Further optimization of the w values ​​of the front and rear cameras will be discussed later.

[0229] Furthermore, because the extrinsic parameters are calibrated based on the assumption that the lane lines are parallel to the vehicle body (lines l1-l4 are parallel to the Xw axis in the world coordinate system Ow), non-parallel samples may affect accuracy. To ensure the accuracy of the calibration results, single-frame calibration results are accumulated and multiple frames are fused to generate accurate extrinsic parameters.

[0230] For example, by removing the maximum 10% and minimum 10% of the results in the calibration extrinsic parameters, the remaining results are averaged. If the differences (Δpitch, Δyaw, Δroll) between the remaining results and the mean are all less than 1°, and (Δz, Δw) are all less than 0.1m, the mean can be used as the fused calibration extrinsic parameter; otherwise, more single-frame calibration results can be accumulated.

[0231] S2. Rough calibration of left and right camera angles and heights.

[0232] Take the left camera as an example (the right camera can be implemented similarly).

[0233] Perform feature point matching on adjacent frame images captured by the left camera.

[0234] Exemplarily, the matching method may adopt an ORB matching method or a deep learning method such as superpoint or superglue.

[0235] Based on the matching point pairs and epipolar geometry principles, the rotation and translation relationship RT (including the rotation matrix R and translation vector T) of the camera movement between adjacent frames can be calculated.

[0236] Based on the external parameters of the front and rear cameras calibrated by S1, the lane lines of adjacent frames are mapped to the world coordinate system. It is determined whether the difference in the slope of the lane lines in adjacent frames exceeds the preset slope difference threshold. If it does not exceed, it means that the vehicle is in a straight state between the current adjacent frames; otherwise, it is in a turning state.

[0237] Based on the accumulated T in the straight state, the x vector in the camera coordinate system is calculated (a pair of adjacent frames in the straight state corresponds to an x ​​vector); based on the accumulated R in the turning state, the z vector in the camera coordinate system is calculated (a pair of adjacent frames in the turning state corresponds to a z vector). The calculation process is as follows:

[0238] The x vector is the normalized representation of the T translation vector: ;

[0239] The z vector is calculated based on the rotation matrix R:

[0240]

[0241]

[0242]

[0243] The x and z vectors accumulated in multiple frames are fused by removing outliers outside the range of mean ± standard deviation and calculating the mean of the remaining vectors to obtain the fused x and z vectors.

[0244] For example, during the multi-frame accumulation process, prior information may be added to perform abnormal vector filtering.

[0245] As an example, abnormal vectors may include x-vectors and z-vectors that do not satisfy the mutual perpendicularity constraint.

[0246] As another example, an abnormal vector may include an x-vector that does not satisfy the positive and negative constraints; the positive and negative constraints include that the component of the x-vector on the camera x-axis is positive, and its schematic diagram may be as follows: Figure 10 shown.

[0247] The rotation matrix is ​​calculated for the fused x and z vectors according to formula (7), and the camera angles pitch, yaw, and roll can be obtained by decomposition.

[0248] R cam = [x, (x * z), z] * [X, (X * Z), Z] -1 (7)

[0249] Where X and Z are the unit vectors [-1, 0, 0] and [0, 0, 1] in the vehicle body world coordinate system respectively.

[0250] The above process has roughly calibrated the angles of the left and right cameras. Based on the initial position parameters of the left and right cameras and the calibrated angle extrinsics, the lane lines are mapped to the world coordinate system. Using a numerical iterative algorithm, the z value is optimized to achieve a lane width of 15 cm, completing a rough calibration of the z height of the left and right cameras.

[0251] S3. Accurate calibration of the extrinsic parameters of the left and right cameras.

[0252] According to S1 and S2, the front and rear camera extrinsics and the left and right camera angle extrinsics and height extrinsics have been obtained. However, there may still be errors in the left and right camera extrinsics, which need further correction.

[0253] Take the left camera extrinsic calibration as an example to illustrate (the right camera can be achieved in the same way).

[0254] According to the calibrated front and rear camera extrinsics (pitch, yaw, roll, h, w, z), left camera angle extrinsics (pitch, yaw, roll), height extrinsics (z), and left camera initial position parameters (h, w), the front, rear, and left fisheye views are converted into BEV views.

[0255] Extract ground feature matching points in the common viewing area of ​​adjacent cameras (front left, back left), such as Figure 11 As shown. Among them, Figure 11 From left to right in the figure are the left view BEV view and the front view BEV view.

[0256] Exemplarily, the feature matching method may include an ORB matching method or a deep learning method such as superpoint or superglue.

[0257] Since the matching points in the common view area are the same feature points in the world coordinate system, the two have the same world coordinates.

[0258] On the one hand, the matching points of the front view can be mapped to the world coordinate system according to the calibrated front and rear camera external parameters to obtain the three-dimensional world coordinate point set P1 of the front view feature points. Similarly, the matching points of the rear view can be mapped to the world coordinate system to obtain the rear view feature points. Figure 3 dimensional world coordinate point set P2, the two point sets are merged into P.

[0259] On the other hand, the matching point set p1 in the left view and the front view in the common viewing area and the matching point set p2 in the rear view in the common viewing area can be merged into an image coordinate point set p.

[0260] Since the three-dimensional coordinate point set P and the image coordinate point set p are one-to-one matched, the pnp algorithm can be used to calculate the extrinsic parameters (h, w, z, pitch, yaw, roll) of the left camera.

[0261] At this point, the external parameter calibration (h, w, z, pitch, yaw, roll) of the four cameras in front, behind, left and right has been completed.

[0262] For example, in the calibration process of S1, the calibration is completed based on the front camera w = 0. Therefore, only the relative position relationship of w of the front and rear cameras can be guaranteed to be correct. The subsequent calibrations of S2 and S3 are also completed based on this. It is also possible to adjust the w of all four cameras as a whole.

[0263] Since the camera is usually installed at the edge of the vehicle body, the camera's w can be adjusted as a whole by taking advantage of the constraint that the w of the left and right cameras are generally symmetrical.

[0264] For example, it is known that the extrinsic parameters of the calibrated camera are w前 、w 后 、w 左 、w 右 (w values ​​of the front camera, rear camera, left camera, and right camera, respectively), then Δw = (w 左 +w 右 ) / 2, w 前 、w 后 、w 左 、w 右 Subtracting Δw respectively is the final calibration result.

[0265] For example, the calibration effect can be as follows Figure 12 shown.

[0266] The above describes the method provided by this application. The following describes the device provided by this application:

[0267] See Figure 13 , is a structural diagram of a camera extrinsic parameter calibration device provided in an embodiment of the present application, such as Figure 13 As shown, the camera extrinsic parameter calibration device may include:

[0268] An acquisition unit, configured to acquire an image captured by a target camera; wherein the target camera comprises a vehicle-mounted camera installed on the periphery of the vehicle body;

[0269] a first determining unit configured to determine, for a designated target camera among the target cameras, a straight line equation of a side edge of a lane line in the dedistorted image based on the dedistorted image; wherein the dedistorted image is obtained by dedistorting an image captured by the designated target camera based on an intrinsic parameter of the designated target camera; and the designated target camera includes a first target camera mounted in front of the vehicle body, or a second target camera mounted behind the vehicle body;

[0270] The first determining unit is further configured to determine the coordinates of a vanishing point in the dedistorted image based on a straight line equation of a side edge of the lane line in the dedistorted image;

[0271] a second determining unit, configured to determine the roll angle of the designated target camera based on a lane width constraint; wherein the lane width constraint includes that, when two sides of the lane lines are projected onto a vehicle body world coordinate system, the distances between two sides of different lane lines are consistent;

[0272] The second determining unit is further configured to determine the pitch angle and yaw angle of the designated target camera based on the roll angle of the designated target camera and the coordinates of the vanishing point;

[0273] a third determining unit, configured to determine a z value of the designated target camera in a vehicle body world coordinate system based on an angular extrinsic parameter of the designated target camera, an initial position parameter of the designated target camera, and a lane line width; wherein the angular extrinsic parameter includes a roll angle, a pitch angle, and a yaw angle; and the initial position parameter of the designated target camera includes an initial w value, an initial h value, and an initial z value of the designated target camera in the vehicle body world coordinate system; the initial w value and initial h value are set based on a vehicle body width and a vehicle body length, and the initial z value is an empirical value;

[0274] The third determination unit is further configured to, when the target camera includes the first target camera and the second target camera, determine the w value of the second target camera in the vehicle body world coordinate system by aligning the same lane lines in the first dedistorted image and the second dedistorted image; wherein the first dedistorted image is obtained by dedistorting the image captured by the first target camera based on the intrinsic parameters of the first target camera, and the second dedistorted image is obtained by dedistorting the image captured by the second target camera based on the intrinsic parameters of the second target camera.

[0275] For example, the specific processing flow of the acquisition unit, the first determination unit, the second determination unit and the third determination unit to implement camera extrinsic parameter calibration can be referred to the relevant description in the above embodiment, and the embodiments of the present application will not be repeated here.

[0276] An embodiment of the present application provides an electronic device, including a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor is used to execute the machine-executable instructions to implement the camera extrinsic parameter calibration method described above.

[0277] See Figure 14 , is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device may include a processor 1401 and a memory 1402 storing machine-executable instructions. The processor 1401 and the memory 1402 may communicate via a system bus 1403. Furthermore, by reading and executing the machine-executable instructions corresponding to the camera extrinsic calibration logic in the memory 1402, the processor 1401 may perform the camera extrinsic calibration method described above.

[0278] The memory 1402 mentioned herein may be any electronic, magnetic, optical, or other physical storage device that may contain or store information, such as executable instructions, data, and the like. For example, a machine-readable storage medium may be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, a storage drive (such as a hard disk drive), a solid-state drive, any type of storage disk (such as a CD, DVD, etc.), or similar storage media, or a combination thereof.

[0279] In some embodiments, a machine-readable storage medium is also provided. Figure 14 The memory 1402 in the machine-readable storage medium stores machine-executable instructions. When executed by the processor, the machine-executable instructions implement the camera extrinsic calibration method described above. For example, the storage medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, or optical data storage device.

Claims

1. A camera extrinsic calibration method, characterized in that: include: Acquire an image captured by a target camera; wherein the target camera includes a vehicle-mounted camera installed on the periphery of the vehicle body; For a designated target camera among the target cameras, determining, based on a dedistorted image, a straight line equation of a side edge of a lane line in the dedistorted image; wherein the dedistorted image is obtained by dedistorting an image captured by the designated target camera based on an intrinsic parameter of the designated target camera; the designated target camera includes a first target camera mounted in front of the vehicle body, or a second target camera mounted behind the vehicle body; Determining the coordinates of a vanishing point in the dedistorted image based on a straight line equation of a side edge of the lane line in the dedistorted image; Determining the roll angle of the designated target camera based on a lane width constraint; wherein the lane width constraint includes that the distance between two sides of different lane lines is consistent when the two sides of the lane lines are projected into the vehicle body world coordinate system; Determining the pitch angle and yaw angle of the designated target camera based on the roll angle of the designated target camera and the coordinates of the vanishing point; Determining the z value of the designated target camera in the vehicle body world coordinate system based on the angular extrinsic parameters of the designated target camera, the initial position parameters of the designated target camera, and the lane line width; wherein the angular extrinsic parameters include the roll angle, the pitch angle, and the yaw angle; and the initial position parameters of the designated target camera include the initial w value, the initial h value, and the initial z value of the designated target camera in the vehicle body world coordinate system; the h value corresponds to the x coordinate value in the vehicle body world coordinate system, and the w value corresponds to the y coordinate value in the vehicle body world coordinate system; the initial w value and the initial h value are set based on the vehicle body width and length, and the initial z value is an empirical value; In a case where the target camera includes the first target camera and the second target camera, the w value of the second target camera in the vehicle body world coordinate system is determined by aligning the same lane lines in the first dedistorted image and the second dedistorted image, with the w value of the first target camera as a reference; wherein the first dedistorted image is obtained by dedistorting the image captured by the first target camera based on the intrinsic parameters of the first target camera, and the second dedistorted image is obtained by dedistorting the image captured by the second target camera based on the intrinsic parameters of the second target camera; The target cameras include a third target camera and a fourth target camera installed on both sides of the vehicle body, and the w values ​​of the third target camera and the fourth target camera in the vehicle body world coordinate system are symmetrical with respect to the x-axis of the vehicle body world coordinate system; The method further comprises: After determining a first w value of the third target camera in the vehicle body world coordinate system and a second w value of the fourth target camera in the vehicle body world coordinate system, determine an average of the first w value and the second w value as a w adjustment value; The w value of each target camera in the vehicle body world coordinate system is respectively subtracted from the w adjustment value to obtain a final w value of each target camera in the vehicle body world coordinate system.

2. The method according to claim 1, characterized in that The method further comprises: When the target camera includes a first target camera installed in front of the vehicle body, first calibration extrinsic parameters corresponding to multiple frames of first dedistorted images are fused using a multi-frame fusion method to obtain calibrated first calibration extrinsic parameters; wherein the first calibration extrinsic parameters include an angle extrinsic parameter of the first target camera and a z extrinsic parameter of the first target camera; and / or, When the target camera includes a second target camera installed at the rear of the vehicle body, second calibration extrinsic parameters corresponding to multiple frames of second dedistorted images are fused using a multi-frame fusion method to obtain calibrated second calibration extrinsic parameters; wherein the second calibration extrinsic parameters include an angle extrinsic parameter of the second target camera, and a z extrinsic parameter and a w extrinsic parameter of the second target camera; For any one of the first target camera and the second target camera, the calibration extrinsic parameters corresponding to the multiple frames of dedistorted images are fused using a multi-frame fusion method, including: For each type of calibration extrinsic parameter corresponding to the accumulated multiple frames of dedistorted images, the maximum value of the first ratio and the minimum value of the second ratio are deleted respectively; Determine the average value of the remaining values ​​of each type of calibration external parameter; When the difference between the residual value of each type of calibration external parameter and the average value of the same type of calibration external parameters does not exceed a preset threshold, the determined average value of each type of calibration external parameter is used as the calibration external parameter after calibration; Otherwise, more frames of calibration extrinsic parameters corresponding to the dedistorted images are accumulated until the calibrated calibration extrinsic parameters are obtained according to the above method.

3. The method according to claim 1, characterized in that In the case where the target camera includes a third target camera installed on the side of the vehicle body, the method further includes: Performing feature point matching on adjacent frame images captured by the third target camera to obtain matching point pairs; Based on the matching point pairs, using the principle of epipolar geometry, determine the rotation matrix R and translation vector T of the third target camera movement between adjacent frames; When the vehicle is in a straight-moving state between adjacent frames, determining an x ​​vector according to the determined translation vector T of the third target camera; When the vehicle is in a turning state between adjacent frames, determining a z vector according to the determined rotation matrix R of the third target camera; Determine an angular extrinsic parameter of the third target camera based on the x vector and the z vector; Determine the z value of the third target camera in the vehicle body world coordinate system based on the angular extrinsic parameters of the third target camera, the initial position parameters of the third target camera, and the lane line width; wherein the initial position parameters of the third target camera include the initial w value, initial h value, and initial z value of the third target camera in the vehicle body world coordinate system.

4. The method according to claim 3, characterized in that Determining the angular extrinsic parameter of the third target camera based on the x vector and the z vector includes: Obtain multiple x vectors and multiple z vectors corresponding to multiple pairs of adjacent frames; Performing outlier vector filtering on the multiple x-vectors and the multiple z-vectors; wherein the outlier vectors include vectors whose values ​​are outside a specified value range, x-vectors and z-vectors that do not satisfy a mutual perpendicularity constraint, or x-vectors that do not satisfy a positive or negative constraint; the positive and negative constraints include: for a target camera mounted on the left side of the vehicle body, the component of the x-vector on the camera x-axis is positive, or, for a target camera mounted on the right side of the vehicle body, the component of the x-vector on the camera x-axis is negative; Determine the mean of the filtered x vectors as the fused x vector; and determine the mean of the filtered z vectors as the fused z vector; The angular extrinsic parameter of the third target camera is determined according to the fused x vector and the fused z vector.

5. The method according to claim 3, characterized in that The method further comprises: According to the calibration extrinsic parameters of each target camera, the target camera captured image is converted into a bird's-eye view of BEV; Extracting ground feature matching points within a common viewing area between the third target camera and its neighboring target cameras based on the BEV view of the third target camera and the BEV views of its neighboring target cameras; wherein the neighboring target cameras of the third target camera are target cameras that have a common viewing area with the third target camera; and the neighboring target cameras of the third target camera include the first target camera; Mapping the ground feature matching points of the first target camera to the vehicle body world coordinate system based on the calibrated extrinsic parameters of the first target camera to obtain a first world coordinate point set; determining an image coordinate point set based on ground feature matching points within a common viewing area of ​​the third target camera and the first target camera in the BEV view of the third target camera; According to a one-to-one correspondence between points in the first world coordinate point set and points in the image coordinate point set, accurate calibration extrinsic parameters of the third target camera are determined.

6. The method according to claim 5, characterized in that In a case where the target camera includes a second target camera installed at the rear of the vehicle body, and the target camera adjacent to the third target camera includes the second target camera, the method further includes: Mapping the ground feature matching points of the second target camera to the vehicle body world coordinate system based on the calibrated extrinsic parameters of the second target camera to obtain a second world coordinate point set; adding ground feature matching points within the common viewing area of ​​the third target camera and the second target camera in the view of the third target camera to the image coordinate point set; The determining of the precise calibration extrinsic parameters of the third target camera based on the one-to-one correspondence between the points in the first world coordinate point set and the points in the image coordinate point set includes: Combining the first world coordinate point set and the second world coordinate point set to obtain a world coordinate point set; According to the one-to-one correspondence between the world coordinate point set and the points in the image coordinate point set, accurate calibration extrinsic parameters of the third target camera are determined.

7. A camera extrinsic parameter calibration device, characterized in that: include: An acquisition unit, configured to acquire an image captured by a target camera; wherein the target camera comprises a vehicle-mounted camera installed on the periphery of the vehicle body; a first determining unit configured to determine, for a designated target camera among the target cameras, a straight line equation of a side edge of a lane line in the dedistorted image based on the dedistorted image; wherein the dedistorted image is obtained by dedistorting an image captured by the designated target camera based on an intrinsic parameter of the target camera; and the designated target camera includes a first target camera mounted in front of the vehicle body, or a second target camera mounted behind the vehicle body; The first determining unit is further configured to determine the coordinates of a vanishing point in the dedistorted image based on a straight line equation of a side edge of the lane line in the dedistorted image; a second determining unit, configured to determine the roll angle of the designated target camera based on a lane width constraint; wherein the lane width constraint includes that, when two sides of the lane line are projected onto a vehicle body world coordinate system, the distances between two sides of different lane lines are consistent; The second determining unit is further configured to determine the pitch angle and yaw angle of the designated target camera based on the roll angle of the designated target camera and the coordinates of the vanishing point; a third determining unit, configured to determine a z value of the designated target camera in a vehicle body world coordinate system based on an angular extrinsic parameter of the designated target camera, an initial position parameter of the designated target camera, and a lane line width; wherein the angular extrinsic parameter includes a roll angle, a pitch angle, and a yaw angle; and the initial position parameter of the designated target camera includes an initial w value, an initial h value, and an initial z value of the designated target camera in the vehicle body world coordinate system; the h value corresponds to an x ​​coordinate value in the vehicle body world coordinate system, and the w value corresponds to a y coordinate value in the vehicle body world coordinate system; the initial w value and the initial h value are set based on a vehicle body width and a vehicle body length, and the initial z value is an empirical value; The third determination unit is further configured to, when the target cameras include the first target camera and the second target camera, determine the w value of the second target camera in the vehicle body world coordinate system by aligning the same lane lines in the first dedistorted image and the second dedistorted image, with the w value of the first target camera as a reference; wherein the first dedistorted image is obtained by dedistorting the image captured by the first target camera based on the intrinsic parameters of the first target camera, and the second dedistorted image is obtained by dedistorting the image captured by the second target camera based on the intrinsic parameters of the second target camera; The target cameras include a third target camera and a fourth target camera installed on both sides of the vehicle body, and the w values ​​of the third target camera and the fourth target camera in the vehicle body world coordinate system are symmetrical with respect to the x-axis of the vehicle body world coordinate system; The third determining unit is further configured to, upon determining a first w value of the third target camera in the vehicle body world coordinate system and a second w value of the fourth target camera in the vehicle body world coordinate system, determine an average of the first w value and the second w value as a w adjustment value; The w value of each target camera in the vehicle body world coordinate system is respectively subtracted from the w adjustment value to obtain a final w value of each target camera in the vehicle body world coordinate system.

8. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor is configured to execute the machine-executable instructions to implement the method according to any one of claims 1 to 6.

9. A machine-readable storage medium, characterized in that The machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images