Camera BEV external parameter calibration method, system and device and medium

Through the camera BEV external parameter calibration method based on a single vanishing point, the problems of cumbersome operation and low computing efficiency in the prior art are solved, and efficient and robust camera external parameter calibration is achieved, ensuring the parallelism of lane lines in the BEV image. and calibration accuracy.

CN119068060AActive Publication Date: 2024-12-03YAOYAO

Patent Information

Application Number
CN202411119031.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-14
Publication Date
2024-12-03
Estimated Expiration
2044-08-14

AI Technical Summary

Technical Problem

The existing camera BEV external parameter calibration method requires two sets of orthogonal lane lines or a set of parallel lane lines. It is cumbersome in operation and low calculation efficiency. Especially in crossroad calibration timing, there are safety hazards and nonlinear equations need to be solved.

Method used

Using the camera BEV external parameter calibration method based on a single vanishing point, the vanishing point is determined by obtaining the parallel lane lines in the original image, and the target equation is constructed based on the vanishing point position, the ground normal vector and camera height are calculated to realize the camera and world coordinate system. The external parameter calibration avoids nonlinear equation solving and manual operation.

Benefits of technology

Improve the robustness and calibration efficiency of the calculation results, ensure the parallelism of the lane line in the BEV image, and the longitudinal axis direction is parallel to the lane line, achieving more efficient camera external parameter calibration, reducing operational complexity and safety risks .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068060A_ABST
    Figure CN119068060A_ABST
Patent Text Reader

Abstract

The invention provides a camera BEV external parameter calibration method, system and device and a medium, and relates to the technical field of automatic driving, the method determines a single vanishing point by selecting a group of parallel lane lines, and can select a lane line with a better position based on the position of the vanishing point, thereby improving the robustness of a calculation result; the rotation relation between the camera and the ground can be calculated based on a single vanishing point, and BEV external parameter calibration of the camera is achieved. The method has high calculation efficiency and practicability, in the actually generated BEV, the actual parallel lane lines can be strictly parallel in the BEV image, meanwhile, the longitudinal axis direction of the BEV map coordinate system is parallel to the lane lines, and a better calibration effect is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving technology, and in particular to a camera BEV extrinsic parameter calibration method, system, device and medium. Background Art

[0002] In recent years, vision-based advanced driving assistance systems (ADAS) have gradually entered the user market, and vision-based autonomous driving has attracted more and more attention from researchers. In vision-based autonomous driving, the detection and reconstruction of lane lines and moving objects on the road is undoubtedly the most important task. On the one hand, it provides an instant local map of the vehicle, senses the lane the vehicle is in, and avoids the risks of random lane changes. On the other hand, it provides the relative motion relationship between the vehicle and surrounding objects, providing important reference information for vehicle acceleration, deceleration, and lane changes.

[0003] Due to the lack of depth information in the camera itself and the existence of camera distortion, IPM (Inverse Perspective Mapping) and the resulting BEV (Bird Eye View) are important methods for monocular vision detection and reconstruction. Under the BEV perspective, the relationship between ground target pixels can be restored to the actual physical scale (so the distance relationship between vehicles can be directly calculated), parallel lane lines remain parallel, and the 3D straight line equation of the lane line can even be directly calculated. This greatly overcomes the limitation of the camera's FOV (Field of View) perspective without depth information, and facilitates the detection and reconstruction of lane lines, moving objects, etc.

[0004] The calibration of the camera BEV extrinsic parameters, that is, the calibration of the extrinsic parameters of the camera and the world coordinate system, mainly includes the attitude angle (roll, pitch, yaw) with the world coordinate system and the height of the camera from the ground (h). Among the two common BEV extrinsic parameter calibration methods in the prior art, the first one is based on two sets of orthogonal lane lines and a priori lane line length calibration. The two sets of orthogonal lane lines can obtain two vanishing points (Vanish Point). Assuming that the corresponding image coordinates are v1 and v2 respectively, the camera coordinates on the imaging plane are V1 and V2, and V1 = [v 1x v 1y f] T , V2=[v 2x v 2y f] T , where f is the focal length of the camera.

[0005] Since the line connecting the camera origin and the vanishing point is parallel to the lane line direction, mutually orthogonal, can are the X-axis and Y-axis of the world coordinate system respectively, then, The rotation matrix from the world coordinate system to the camera coordinate system is as follows (1):

[0006]

[0007] To calculate the translation of the camera relative to the world coordinate system, a line segment of known length in the world coordinate system is required. Assume that its endpoints are P1 and P2 respectively. At the same time, its corresponding pixel coordinates are known to be as well as Camera coordinates on the imaging plane but according to Figure 1 As shown, according to the principle of similar triangles, we have the following formula (2):

[0008]

[0009] Without loss of generality, assuming that P1 is the origin of the world coordinate system, then:

[0010]

[0011] In the prior art, another BEV extrinsic parameter calibration method is based on a set of parallel lane lines and a priori lane width calibration.

[0012] In this method, a set of parallel lane lines can obtain a vanishing point (Vanish Point, V p ), the line V between the camera and the vanishing point D Parallel to this set of lane lines, namely V D Perpendicular to the projection plane formed by the camera position and the lane line. Based on this, a set of orthogonal constraints can be constructed, such as formula (4):

[0013] Av=0 (4)

[0014] Where A=[...,n i ...] T , n i is the normal vector of the projection surface formed by the i-th lane line and the camera position, n i =(K -1 p i1 )×(K -1 p i2 ), (p i1 ,p i2 ) is the projection line segment of the i-th lane line on the camera projection surface; v is V D The corresponding unit vector.

[0015] Solving the above equation (4), we can get the vector v, and the camera position points from v to the vanishing point.

[0016] Since v is parallel to the lane line, the z-axis of the world coordinate system can be constructed based on the direction of v. Assuming that the pitch angle and yaw angle of the camera coordinate system relative to the world coordinate system are θ and φ respectively, there are the following transformation relationships (5) and (6):

[0017]

[0018] In the above formula, c θ =cos(θ),s θ = sin(θ), c φ =cos(φ),s φ =sin(φ).

[0019] By solving the above equations (5) and (6), we can obtain θ and φ, namely the pitch angle (pitch) and the yaw angle (yaw).

[0020] To further calibrate the roll angle (roll) and camera height (height), the lane width needs to be known. L ,I R The relationship between the theoretical lane width, the roll angle ψ and the camera height h is (as shown in the attached figure) Figure 2 shown):

[0021]

[0022] In formula (7), α L , α R are the angles between the projections of the left and right lane lines on the image plane and the y-axis of the camera coordinate system. Based on the fact that the theoretical width should be close to the actual width, for multiple lane groups with known widths, The following energy equation (8) can be constructed:

[0023]

[0024] By solving the above energy equation (8), we can obtain ψ,h, that is, the roll angle roll and the camera height height.

[0025] Of the two calibration methods mentioned above, the first method requires two sets of perpendicular lane lines, and needs to measure the coordinates of a lane line in the world coordinate system, and also needs to mark the line segment of this lane line on the image. This places high demands on the scene and is cumbersome to operate. If calibration is performed at an intersection, there will be great safety hazards. Method 2 has lower scene requirements, but the calculation process is more complicated. Both calculation steps require solving nonlinear equations, and the calculation efficiency is not high. Summary of the invention

[0026] In view of this, the first aspect of the present invention provides a camera BEV extrinsic parameter calibration method based on a single vanishing point to solve the problem raised in the background technology section. The method comprises:

[0027] Acquire an original image, and acquire a first lane line and a second lane line according to the original image, wherein the first lane line and the second lane line are parallel to each other, and the original image is captured by a camera device;

[0028] Determine a vanishing point based on the first lane line and the second lane line, the vanishing point being an intersection of the first lane line and the second lane line on a projection surface of the camera device;

[0029] Constructing a target equation, obtaining a first straight line equation of the first lane line on the projection surface of the camera device and a second straight line equation of the second lane line on the projection surface of the camera device according to the target equation, and obtaining a position of the vanishing point according to the first straight line equation and the second straight line equation;

[0030] Selecting a point at the lower left corner of the original image to construct a third lane line, and selecting a point at the lower right corner of the original image to construct a fourth lane line, wherein the third lane line and the fourth lane line intersect at the vanishing point on the projection surface of the camera device;

[0031] According to the position of the vanishing point, a ground normal vector of the target ground is calculated by using the projection surface normal vector of the third lane line and the projection surface normal vector of the fourth lane line;

[0032] Based on the ground normal vector and the position of the vanishing point, the ground equation in the camera coordinate system is determined, and the camera height of the camera device from the target ground is obtained according to the ground equation. The rotation matrix and translation matrix converted from the world coordinate system to the camera coordinate system are used to complete the external parameter calibration of the BEV viewing angle of the camera device.

[0033] Furthermore, the method for constructing the objective equation includes:

[0034] Set the pixel point set to {p ij =(x ij ,y ij )} N , where i is a straight line, j is a pixel point on the straight line, and the pixel point set corresponds to N parallel lines. For the straight line i, the following relationship exists:

[0035]

[0036] Among them, (x ij ,y ij ) is the jth point on the i-th straight line, (dx i,dy i ) is the direction vector of line i, (dx i ,dy i ) is equivalent to (dX i ,1), (x, y) is the coordinate of the vanishing point;

[0037] The target equation is constructed according to the above relationship, and the target equation is shown as follows:

[0038]

[0039] Among them, X is the variable to be optimized, X=[…dX i …xy] T , (x,y) is the pixel coordinate of the vanishing point.

[0040] Furthermore,

[0041] The vanishing point V P The coordinates are (x, y), and the third lane point P1 = (x1, y1) is selected to construct the third lane line V P P1, select the fourth lane point P2 = (x2, y2) to construct the fourth lane line;

[0042] According to V P , the normalized camera coordinates of P1 and P2 and the coordinates of O (0,0,0), calculate OV P The normal vector n of the projection surface corresponding to P1 i =[x i y i z i ] T , and OV P The normal vector n of the projection surface corresponding to P2 j =[x j y j z j ] T , where the camera origin O and the vanishing point V P The third lane point P1 constitutes the projection surface OV of the third lane line on the camera device in the space of the camera device. P P1; camera origin O, vanishing point V P The fourth lane point P2 constitutes the projection surface OV of the fourth lane line on the camera device in the space of the camera device. P P2.

[0043] Furthermore, obtaining the ground normal vector of the target ground includes:

[0044] n i =[x i y i z i] T 、n j =[x j y j z j ] T and the ground normal vector n = [xyz] T The following constraints exist:

[0045] (n i ×n)×(n j ×n)=0

[0046] Among them, n i ×n、n j ×n are the direction vectors of the corresponding lane lines, and the ground normal vector n=[xyz] is obtained according to the constraint relationship. T The solution.

[0047] Further, the rotation matrix for converting from the world coordinate system to the camera coordinate system is obtained according to the ground coordinate system, and the rotation matrix as follows:

[0048]

[0049] Wherein, v is the lane line direction, n is the ground normal vector, v and n are respectively used as the x-axis and z-axis of the world coordinate system, and u is used as the y-axis of the world coordinate system.

[0050] Further, the translation matrix converted from the world coordinate system to the camera coordinate system is obtained according to the ground coordinate system, and the translation matrix as follows:

[0051]

[0052] Wherein, n is the ground normal vector, and c is the constant term in the ground equation.

[0053] Furthermore, the third lane line and the fourth lane line are selected by virtual selection.

[0054] The second aspect of the present invention provides a camera BEV extrinsic calibration system based on a single vanishing point to solve the problem raised by the background technology of the present invention. The system includes: a vanishing point acquisition module, used to acquire an original image, and acquire a first lane line and a second lane line according to the original image, wherein the first lane line and the second lane line are parallel to each other, and the original image is captured by a camera device; a vanishing point is determined based on the first lane line and the second lane line, and the vanishing point is the intersection of the first lane line and the second lane line on the projection surface of the camera device;

[0055] A lane line optimization selection module is used to construct a target equation, obtain a first straight line equation of the first lane line on the projection surface of the camera device and a second straight line equation of the second lane line on the projection surface of the camera device according to the target equation, and obtain the position of the vanishing point according to the first straight line equation and the second straight line equation; select a point in the lower left corner of the original image to construct a third lane line, select a point in the lower right corner of the original image to construct a fourth lane line, and the third lane line and the fourth lane line intersect at the vanishing point on the projection surface of the camera device; according to the position of the vanishing point, calculate and obtain the ground normal vector of the target ground through the projection surface normal vector of the third lane line and the projection surface normal vector of the fourth lane line;

[0056] The camera BEV extrinsic parameter calibration module is used to determine the ground equation in the camera coordinate system based on the ground normal vector and the position of the vanishing point, obtain the camera height of the camera device from the target ground according to the ground equation, and complete the extrinsic parameter calibration of the BEV viewing angle of the camera device with the rotation matrix and translation matrix converted from the world coordinate system to the camera coordinate system.

[0057] The third aspect of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any one of the above-mentioned camera BEV extrinsic parameter calibration methods based on a single vanishing point is implemented.

[0058] A fourth aspect of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for executing any of the above-mentioned camera BEV extrinsic parameter calibration methods based on a single vanishing point.

[0059] Compared with the prior art, the beneficial effects that can be achieved by at least one of the above technical solutions adopted in this specification include at least: providing a camera BEV extrinsic parameter calibration method based on a single vanishing point, determining a single vanishing point by selecting a set of parallel lane lines, and being able to select a lane line with a better position based on the position of the vanishing point, thereby improving the robustness of the calculation result, thereby calculating the rotation relationship between the camera and the ground based on a single vanishing point, and realizing the BEV extrinsic parameter calibration of the camera. The present invention does not involve solving nonlinear equations, nor does it require manual operation, and has high computational efficiency and practicality; in the actually generated BEV, the lane lines that are actually parallel can be kept strictly parallel in the BEV image, and at the same time, the longitudinal axis direction of the BEV map coordinate system is parallel to the lane lines, achieving a better calibration effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0061] Figure 1 It is a schematic diagram of the background technology part of the present invention;

[0062] Figure 2 It is another schematic diagram of the background technology part of the present invention;

[0063] Figure 3 A flow chart of a camera BEV extrinsic calibration method based on a single vanishing point provided in an embodiment of the present invention;

[0064] Figure 4 Schematic diagram of the structure of a camera BEV extrinsic calibration system based on a single vanishing point provided in an embodiment of the present invention;

[0065] Figure 5 is a schematic diagram of a computer device provided by an embodiment of the present invention;

[0066] Reference numerals in the figure: 400, system; 401, vanishing point acquisition module; 402, lane line optimization selection module; 403, camera BEV external parameter calibration module; 501, memory; 502, processor. DETAILED DESCRIPTION

[0067] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0068] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0069] A first aspect of the present embodiment provides a camera BEV extrinsic parameter calibration method based on a single vanishing point, which determines a single vanishing point by selecting a group of parallel lane lines, and can select a lane line with a better position based on the position of the vanishing point, thereby improving the robustness of the calculation result, and calculating the rotation relationship between the camera and the ground based on the single vanishing point, thereby realizing the BEV extrinsic parameter calibration of the camera.

[0070] like Figure 3 As shown, the present invention provides a camera BEV extrinsic parameter calibration method based on a single vanishing point, the method comprising:

[0071] Step S100, acquiring an original image, and acquiring a first lane line and a second lane line according to the original image, wherein the first lane line and the second lane line are parallel to each other, and the original image is captured by a camera device;

[0072] Step S200: determining a vanishing point based on the first lane line and the second lane line, the vanishing point being an intersection of the first lane line and the second lane line on a projection surface of the camera device;

[0073] Specifically, the camera BEV extrinsic parameter calibration method based on a single vanishing point proposed in this embodiment is implemented based on a set of parallel lane lines and a priori lane width. Among them, two parallel lane lines will form two projection planes based on the camera imaging principle. A ground plane can be found to intersect with the two projection planes so that the intersection lines are parallel to each other, and the width between the two parallel lines is equal to the actual width of the corresponding lane. The extrinsic parameters between the ground and the camera coordinate system are the camera BEV extrinsic parameters. At the same time, in order to avoid the angle between the two projection planes being too small, and to use more lane line observations to optimize the parallel lines of the lane lines, the intersection point of the lane lines on the projection plane, that is, the vanishing point, can be fitted. The vanishing point is the intersection point of the projection of a set of parallel lines on the camera imaging plane, which corresponds to a point infinitely far away in the camera space.

[0074] Step S300, constructing a target equation, obtaining a first straight line equation of the first lane line on the projection surface of the camera device and a second straight line equation of the second lane line on the projection surface of the camera device according to the target equation, and obtaining the position of the vanishing point according to the first straight line equation and the second straight line equation;

[0075] Specifically, the position of the vanishing point is determined by constructing the target equation. First, the pixel point set is set as {p ij =(x ij ,y ij )} N , where i is a straight line, j is a pixel point on the straight line, and the pixel point set corresponds to N parallel lines. For the straight line i, there exists the following relationship described in equation (9):

[0076]

[0077] In formula (9), (x ij ,y ij ) is the jth point on the i-th straight line, (dx i ,dy i ) is the direction vector of line i, (dx i ,dy i ) is equivalent to (dX i ,1), (x,y) are the coordinates of the vanishing point.

[0078] The target equation is constructed according to the above formula (9), and the target equation is shown in the following formula (10):

[0079] X * =min x ∑ i ∑ j ||dX i (y ij -y)-(x ij -x)|| 2 (10)

[0080] In formula (10), X is the variable to be optimized, X = [… dX i … xy] T , (x, y) is the pixel coordinate of the vanishing point. By solving the above target equation (10), the first straight line equation of the first lane line on the projection surface of the camera device and the second straight line equation of the second lane line on the projection surface of the camera device are obtained, and the position of the vanishing point is solved by the first straight line equation and the second straight line equation.

[0081] Step S400: Select a point at the lower left corner of the original image to construct a third lane line, select a point at the lower right corner of the original image to construct a fourth lane line, and the third lane line and the fourth lane line intersect at the vanishing point on the projection plane of the camera device;

[0082] Specifically, in this embodiment, based on the position of the vanishing point, by selecting lane lines whose projection surface angles meet the preset values, the robustness of the calculation results is improved. A vanishing point actually constrains a cluster of parallel lane lines. To calculate the posture of the ground relative to the camera coordinate system, only two parallel lane lines need to be selected. In order to reduce the problems caused by lane lines that are not parallel enough and to avoid unstable calculation results caused by too small determinants of the coefficient matrix during the solution process, it is preferred in this embodiment to select a group of third and fourth lane lines that are parallel to each other and have a slightly larger angle based on the position of the vanishing point.

[0083] Preferably, the third lane line and the fourth lane line are selected in a virtual manner. The lower left corner P1 and the lower right corner P2 of the image are selected respectively to construct two new lane lines V P P1 (marked as the third lane line) and V P P2 (marked as the fourth lane line), according to V P , the normalized camera coordinates of P1 and P2 and the coordinates of O (0,0,0), calculate OV P The normal vector n of the projection surface corresponding to P1 i =[x i y i z i ] T , and OV P The normal vector n of the projection surface corresponding to P2 j =[x j y j z j ] T .

[0084] Step S500, calculating a ground normal vector of the target ground through a projection surface normal vector corresponding to the third lane line and a projection surface normal vector of the fourth lane line according to the position of the vanishing point;

[0085] Specifically, suppose the size of the original image in step S100 is (W, H), and the vanishing point V P The coordinates are (x, y), then select two other points P1 = (0, H) and P2 = (W, H), where P1 is any point on the third lane line and P2 is any point on the fourth lane line; the camera origin O and the vanishing point V P The third lane point P1 constitutes the projection surface OV of the third lane line on the camera device in the space of the camera device. P P1; camera origin O, vanishing point V P The fourth lane point P2 constitutes the projection surface OV of the fourth lane line on the camera device in the space of the camera device. P P2.OV P P1 and OV P The two projection planes P2 intersect, and it can be proved that there are infinite planes that intersect with both projection planes and whose intersection lines are parallel, and these infinite planes have the same normal vector, which is the ground normal vector n.

[0086] According to V P (x,y), let OV P The normal vector n of the projection surface corresponding to P1 i =[x i y i z i ] T, OV P The normal vector n of the projection surface corresponding to P2 j =[x j y j z j ] T , then there exists the constraint relationship described in the following formula (11):

[0087] (n i ×n)×(n j ×n)=0 (11)

[0088] In formula (11), n i ×n、n j ×n are the direction vectors of the third lane line and the fourth lane line respectively.

[0089] Simplify the above formula (11) to obtain formula (12):

[0090]

[0091] Then the constraint relationship described in formula (11) is transformed into the following formula (13):

[0092]

[0093] The above determinant (13) corresponds to a three-dimensional vector, and each dimension is 0, that is, the cross product vector of two parallel vectors is 0. Based on this, the linear equation about (x, y, z) is solved according to formula (13), and the solution of the linear equation is the normal vector n of the target ground. Since the third lane line and the fourth lane line are parallel to each other as a constraint condition, the ground normal vector n obtained by the above solution can make the parallel lane lines in the BEV image strictly parallel, so this embodiment can achieve a better BEV effect.

[0094] Step S600, determine the ground equation in the camera coordinate system based on the ground normal vector and the position of the vanishing point, obtain the camera height of the camera device from the target ground according to the ground equation, and convert the rotation matrix and translation matrix from the world coordinate system to the camera coordinate system to complete the external parameter calibration of the BEV viewing angle of the camera device.

[0095] Specifically, after the ground normal vector n is obtained in step S500, the other end point of a lane line on the original image other than the vanishing point is set as p I , let the ground equation in the camera coordinate system be n T x=c; then the straight line from the camera optical center p I The intersection point Q of the ray and the ground satisfies the following equation (14):

[0096]

[0097] According to formula (14), the coordinates of the intersection point Q are:

[0098]

[0099] In the above formula (15), P = K -1 p, p is p I Homogeneous coordinates of , K is the camera projection matrix.

[0100] For any two parallel lane lines l R , l L , let l R , l L The other endpoints of the image besides the vanishing point are p RI 、p LI , then the camera optical center points to p RI 、p LI The distance between the intersection of the ray and the ground is as follows (16):

[0101]

[0102] In the above formula (16), u is the direction vector perpendicular to the target ground and the lane line.

[0103] The direction vector v of the lane line vector is:

[0104]

[0105] u is:

[0106]

[0107] When the lane width d is known, the constant term c in the ground equation can be obtained according to formula (16), thereby determining the ground equation. T x=c, the camera height of the camera device from the target ground is obtained by solving the distance from the point to the plane.

[0108] Furthermore, the external parameter calibration of the camera BEV is to calibrate the external parameters of the camera and the world coordinate system, mainly including the attitude angle (roll, pitch, yaw) and the camera height (h) with the world coordinate system. In step S600, v, u and n are used as the x-axis, y-axis and z-axis of the world coordinate system respectively, and the rotation matrix from the world coordinate system to the camera coordinate system is converted into as follows:

[0109]

[0110] In formula (19), v is the lane line direction, u is the direction vector, n is the ground normal vector, c is the constant term in the ground equation, and v, u, and n are respectively used as the x-axis, y-axis, and z-axis of the world coordinate system.

[0111] The ground coordinate system obtains the translation matrix from the world coordinate system to the camera coordinate system as follows:

[0112]

[0113] In formula (20), n is the ground normal vector, and c is the constant term in the ground equation.

[0114] In the embodiment of the present invention, Figure 4 As shown, a camera BEV extrinsic calibration system 400 based on a single vanishing point is also provided, comprising:

[0115] A vanishing point acquisition module 401 is used to acquire an original image, and acquire a first lane line and a second lane line according to the original image, wherein the first lane line and the second lane line are parallel to each other, and the original image is captured by a camera device; and determine a vanishing point based on the first lane line and the second lane line, wherein the vanishing point is an intersection of the first lane line and the second lane line on a projection surface of the camera device;

[0116] The lane line optimization selection module 402 is used to construct a target equation, obtain a first straight line equation of the first lane line on the projection surface of the camera device and a second straight line equation of the second lane line on the projection surface of the camera device according to the target equation, and obtain the position of the vanishing point according to the first straight line equation and the second straight line equation; select a third lane line and a fourth lane line on the original image, the third lane line and the fourth lane line intersect at the vanishing point on the projection surface of the camera device, and the angle between the third lane line and the fourth lane line on the projection surface of the camera device satisfies a preset value;

[0117] The camera BEV extrinsic parameter calibration module 403 is used to obtain the ground normal vector of the target ground by acquiring the projection surface normal vector corresponding to the third lane line and the projection surface normal vector corresponding to the fourth lane line according to the position of the vanishing point; determine the ground equation in the camera coordinate system based on the ground normal vector and the position of the vanishing point, obtain the camera height of the camera device from the target ground according to the ground equation, and complete the extrinsic parameter calibration of the BEV viewing angle of the camera device with the rotation matrix and the translation matrix converted from the world coordinate system to the camera coordinate system.

[0118] In this embodiment, a computer device is provided, such as Figure 5 As shown, it includes a memory 501, a processor 502, and a computer program stored in the memory 501 and executable on the processor 502. When the processor 502 executes the computer program, any of the above-mentioned camera BEV extrinsic parameter calibration methods based on a single vanishing point is implemented.

[0119] Specifically, the computer device may be a computer terminal, a server or a similar computing device.

[0120] In this embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program for executing any of the above-mentioned camera BEV extrinsic parameter calibration methods based on a single vanishing point.

[0121] Specifically, computer-readable storage media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0122] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0123] The present invention achieves the following technical effects through the above embodiments:

[0124] 1. The present invention determines a single vanishing point by selecting a set of parallel lane lines, and can select a lane line with a better position based on the vanishing point position, thereby improving the robustness of the calculation result, thereby calculating the rotation relationship between the camera and the ground based on a single vanishing point, and realizing the calibration of the camera's BEV extrinsic parameters; and proposes an algorithm for jointly estimating the straight line equation and the vanishing point position on the image, which can calibrate the camera's BEV extrinsic parameters based on a set of parallel lane lines and a priori lane width. The present invention does not involve solving nonlinear equations, nor does it require manual operation, and has high computational efficiency and practicality;

[0125] 2. In the BEV actually generated by the present invention, the lane lines that are actually parallel can be kept strictly parallel in the BEV image. At the same time, the longitudinal axis direction of the BEV map coordinate system is parallel to the lane lines, achieving a better calibration effect. In addition, the method provided in this application can also be used for the calibration of camera and vehicle body external parameters.

[0126] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A camera BEV extrinsic calibration method based on a single vanishing point, characterized in that: The method comprises: Acquire an original image, and acquire a first lane line and a second lane line according to the original image, wherein the first lane line and the second lane line are parallel to each other, and the original image is captured by a camera device; Determine a vanishing point based on the first lane line and the second lane line, the vanishing point being an intersection of the first lane line and the second lane line on a projection surface of the camera device; Constructing a target equation, obtaining a first straight line equation of the first lane line on the projection surface of the camera device and a second straight line equation of the second lane line on the projection surface of the camera device according to the target equation, and obtaining a position of the vanishing point according to the first straight line equation and the second straight line equation; Selecting a point at the lower left corner of the original image to construct a third lane line, and selecting a point at the lower right corner of the original image to construct a fourth lane line, wherein the third lane line and the fourth lane line intersect at the vanishing point on the projection surface of the camera device; According to the position of the vanishing point, a ground normal vector of the target ground is calculated by using the projection surface normal vector of the third lane line and the projection surface normal vector of the fourth lane line; Based on the ground normal vector and the position of the vanishing point, the ground equation in the camera coordinate system is determined, and the camera height of the camera device from the target ground is obtained according to the ground equation. The rotation matrix and translation matrix converted from the world coordinate system to the camera coordinate system are used to complete the external parameter calibration of the BEV viewing angle of the camera device.

2. According to the method for calibrating camera BEV extrinsic parameters based on a single vanishing point according to claim 1, it is characterized in that: The method for constructing the target equation includes: Set the pixel point set to {p ij =(x ij ,y ij )} N , where i is a straight line, j is a pixel point on the straight line, and the pixel point set corresponds to N parallel lines. For the straight line i, the following relationship exists: Among them, (x ij ,y ij ) is the jth point on the i-th straight line, (dx i ,dy i ) is the direction vector of line i, (dx i ,dy i ) is equivalent to (dX i ,1), (x, y) is the coordinate of the vanishing point; The target equation is constructed according to the above relationship, and the target equation is shown as follows: Among them, X is the variable to be optimized, X=[…dX i …xy] T , (x,y) is the pixel coordinate of the vanishing point.

3. The camera BEV extrinsic calibration method based on a single vanishing point according to claim 1, characterized in that: The vanishing point V P The coordinates are (x, y), and the third lane point P1 = (x1, y1) is selected to construct the third lane line V P P1, select the fourth lane point P2 = (x2, y2) to construct the fourth lane line; According to V P , the normalized camera coordinates of P1 and P2 and the coordinates of O (0,0,0), calculate OV P The normal vector n of the projection surface corresponding to P1 i =[x i y i z i ] T , and OV P The normal vector n of the projection surface corresponding to P2 j =[x j y j z j ] T , where the camera origin O and the vanishing point V P The third lane point P1 constitutes the projection surface OV of the third lane line on the camera device in the space of the camera device. P P1; camera origin O, vanishing point V P The fourth lane point P2 constitutes the projection surface OV of the fourth lane line on the camera device in the space of the camera device. P P2.

4. The camera BEV extrinsic calibration method based on a single vanishing point according to claim 3, characterized in that: The obtaining of the ground normal vector of the target ground includes: n i =[x i y i z i ] T 、n j =[x j y j z j ] T and the ground normal vector n = [xyz] T The following constraints exist: (n i ×n)×(n j ×n)=0 Among them, n i ×n、n j ×n are the direction vectors of the corresponding lane lines, and the ground normal vector n=[xyz] is obtained according to the constraint relationship. T The solution.

5. The camera BEV extrinsic calibration method based on a single vanishing point according to claim 1, characterized in that: The rotation matrix converted from the world coordinate system to the camera coordinate system is obtained according to the ground coordinate system. as follows: Wherein, v is the lane line direction, n is the ground normal vector, v and n are respectively used as the x-axis and z-axis of the world coordinate system, and u is used as the y-axis of the world coordinate system.

6. The camera BEV extrinsic calibration method based on a single vanishing point according to claim 1, characterized in that: The translation matrix converted from the world coordinate system to the camera coordinate system is obtained according to the ground coordinate system. as follows: Wherein, n is the ground normal vector, and c is the constant term in the ground equation.

7. The camera BEV extrinsic calibration method based on a single vanishing point according to claim 1, characterized in that: The third lane line and the fourth lane line are selected in a virtual manner.

8. A camera BEV extrinsic calibration system based on a single vanishing point, characterized in that: The system comprises: a vanishing point acquisition module, configured to acquire an original image, acquire a first lane line and a second lane line according to the original image, wherein the first lane line and the second lane line are parallel to each other, and the original image is captured by a camera device; determine a vanishing point based on the first lane line and the second lane line, wherein the vanishing point is an intersection of the first lane line and the second lane line on a projection surface of the camera device; A lane line optimization selection module is used to construct a target equation, obtain a first straight line equation of the first lane line on the projection surface of the camera device and a second straight line equation of the second lane line on the projection surface of the camera device according to the target equation, and obtain the position of the vanishing point according to the first straight line equation and the second straight line equation; select a point in the lower left corner of the original image to construct a third lane line, select a point in the lower right corner of the original image to construct a fourth lane line, and the third lane line and the fourth lane line intersect at the vanishing point on the projection surface of the camera device; according to the position of the vanishing point, calculate and obtain the ground normal vector of the target ground through the projection surface normal vector of the third lane line and the projection surface normal vector of the fourth lane line; The camera BEV extrinsic parameter calibration module is used to determine the ground equation in the camera coordinate system based on the ground normal vector and the position of the vanishing point, obtain the camera height of the camera device from the target ground according to the ground equation, and complete the extrinsic parameter calibration of the BEV viewing angle of the camera device with the rotation matrix and translation matrix converted from the world coordinate system to the camera coordinate system.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, a camera BEV extrinsic parameter calibration method based on a single vanishing point is implemented as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing a camera BEV extrinsic parameter calibration method based on a single vanishing point according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Vehicle-mounted camera on-line automatic calibration method

    CN108898638A

  • Vehicle-mounted camera external parameter calibration simulation analysis method, system, equipment and medium

    CN117372535A

  • Lane line detection method and device and computer program product

    CN118411692A

  • Image processing method, apparatus, and computer readable storage medium

    WO2020133172A1

Cited By

  • BEV conversion method without internal and external parameters based on lane line constraint

    CN120563303A

  • Camera parameter optimization method and device, equipment and medium

    CN120852520A