Calibration method, device and equipment of surround view camera and storage medium

By setting calibration lines on the left and right sides of the vehicle and utilizing the image features of the front and rear cameras and the left and right cameras, the rotation and translation matrix is ​​determined, which solves the problem of low calibration accuracy of the surround-view camera and achieves higher calibration accuracy and robustness.

CN119027506APending Publication Date: 2024-11-26CHANGSHA INTELLIGENT DRIVING INST CORP LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310595219.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-24
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In the existing technology, the calibration process of vehicle-mounted surround view cameras is prone to problems such as inaccurate or undetectable calibration points, especially in non-customized sites, particularly in vehicles where the rear camera is installed at a low position, the target may be unclear or located at the edge of the image, resulting in low calibration accuracy.

Method used

By setting calibration lines on the left and right sides of the vehicle and utilizing image features acquired by the front and rear cameras and the left and right cameras, the rotation and translation matrices are determined. Combined with the relative poses of adjacent cameras, the calibration accuracy is improved.

Benefits of technology

It effectively reduces the probability of inaccurate or unselectable calibration points, improves the robustness and accuracy of surround view camera calibration, and adapts to the calibration requirements of different vehicle models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027506A_ABST
    Figure CN119027506A_ABST
Patent Text Reader

Abstract

The invention relates to the field of all-round view camera calibration, and provides an all-round view camera calibration method and device, equipment and a storage medium. The method comprises the following steps: determining rotation matrixes of the front and rear cameras according to first lane line features in images collected by the front and rear cameras; determining rotation matrixes of the left camera and the right camera according to second lane line features in the images collected by the left camera and the right camera; determining relative poses of the adjacent cameras according to the lane lines and the calibration lines in the images of the adjacent cameras; and determining the center of the around-view camera according to the relative pose, and determining a translation matrix of the around-view camera according to the center and the relative pose. The calibration lines are arranged in the view overlapped areas of the adjacent cameras, so that the probability that the calibration points do not exist or cannot be selected is reduced, and the robustness and accuracy during calibration of the surround-view camera can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of surround view camera calibration, and in particular to a surround view camera calibration method, device, equipment and storage medium. Background Art

[0002] The on-board surround view camera refers to multiple on-board cameras used to collect surround view images or panoramic images of the vehicle. Through the joint work of multiple on-board cameras, the images collected by multiple on-board cameras are fused to effectively reduce the blind spots or blind spots of the driver's field of vision and improve driving safety.

[0003] Due to the wide variety of vehicles produced by commercial vehicle manufacturers, the installation positions and angles of surround-view cameras of different models vary greatly. For example, the greater the difference in the wheelbase of the vehicle, the greater the difference in the installation position and angle of the surround-view camera. When calibrating the surround-view camera, it is usually necessary to rely on the target in the image captured by the camera. However, for non-customized sites, the target in the camera's field of view may be unclear or distorted, especially for flatbed trailers, muck trucks, mixer trucks, etc. where the rear-facing camera is installed at a lower position. The target is often at the edge of the image, and it is easy for the calibration point to be inaccurately detected or undetectable during calibration, which is not conducive to improving the calibration accuracy of the surround-view camera. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a calibration method, device, equipment and storage medium for a surround-view camera to solve the problem in the prior art that when calibrating a surround-view camera, calibration points are easily inaccurately detected or cannot be detected, which is not conducive to improving the calibration accuracy of the surround-view camera.

[0005] A first aspect of an embodiment of the present application provides a method for calibrating a surround view camera, which is used for calibrating a scene of a surround view camera of a vehicle, wherein the left and right sides of the vehicle respectively include at least one lane, and a calibration line is provided in an overlapping area of ​​the field of view of two adjacent cameras, and the surround view camera includes front and rear cameras and left and right cameras, and the method includes:

[0006] Determining a rotation matrix of the front and rear cameras according to a first lane line feature in the images captured by the front and rear cameras;

[0007] Determine the rotation matrix of the left and right cameras according to the second lane line features in the images captured by the left and right cameras;

[0008] Determine the relative positions of adjacent cameras based on the lane lines and calibration lines in the images of adjacent cameras;

[0009] The center of the surround-view camera is determined according to the relative position and posture, and the translation matrix of the surround-view camera is determined according to the center and the relative position and posture.

[0010] In combination with the first aspect, in a first possible implementation manner of the first aspect, the first lane line feature includes a lane width, a vanishing point position, and a lane line parallel feature, and determining a rotation matrix of the front and rear cameras according to the first lane line feature in the images captured by the front and rear cameras includes:

[0011] Determine the pitch angle and yaw angle of the front and rear cameras according to the constraint that the vanishing point positions in the images collected by the front and rear cameras are parallel to the lane lines;

[0012] The roll angles of the front and rear cameras are determined according to the widths of the three lanes in the images captured by the front and rear cameras.

[0013] In combination with the first aspect, in a second possible implementation manner of the first aspect, the second lane line feature includes the same lane width, the vehicle and the lane line are parallel, and a lane width feature, and determining the rotation matrix of the left and right cameras according to the second lane line feature in the images captured by the left and right cameras includes:

[0014] Determining the roll angles of the left and right cameras according to a feature that the widths of the lane lines in the images captured by the left and right cameras are the same;

[0015] Determining the yaw angles of the left and right cameras according to the feature that the vehicle is parallel to the lane line;

[0016] The pitch angles of the left and right cameras are determined according to the predetermined heights of the left and right cameras in combination with the lane width feature.

[0017] In combination with the first aspect, in a third possible implementation manner of the first aspect, when the surround view camera includes four cameras, determining the relative positions of adjacent cameras according to lane lines and demarcation lines in images of adjacent cameras includes:

[0018] Taking the first camera posture as a reference posture, determining the relative postures of a second camera and a third camera adjacent to the first camera with respect to the first camera;

[0019] Taking the posture of the second camera as a reference posture, a relative posture of a fourth camera adjacent to the second camera with respect to the second camera is determined.

[0020] In combination with the third possible implementation manner of the first aspect, in a fourth possible implementation manner of the first aspect, when the first camera is a left camera and the second camera is a front camera, taking the first camera pose as a reference pose, determining a relative pose of a second camera adjacent to the first camera relative to the first camera includes:

[0021] According to the predetermined height of the left camera, and in combination with the optimization goal that the width of the left lane in the images captured by the front camera and the left camera is the same, the height of the front camera relative to the left camera is determined;

[0022] Determine the lateral position of the front camera relative to the left camera based on the overlap of lane lines in the images captured by the front camera and the left camera as an optimization target;

[0023] The longitudinal position of the front camera relative to the left camera is determined based on the coincidence of calibration lines in the images captured by the front camera and the left camera as an optimization target.

[0024] In combination with the first aspect, in a fifth possible implementation manner of the first aspect, when the surround view camera includes six cameras, determining the relative positions of adjacent cameras according to lane lines and demarcation lines in images of adjacent cameras includes:

[0025] Taking the first camera posture as a reference posture, determining the relative postures of a second camera and a third camera adjacent to the first camera with respect to the first camera;

[0026] Taking the posture of the second camera as a reference posture, determining a relative posture of a fourth camera adjacent to the second camera relative to the second camera;

[0027] Taking the posture of the fourth camera as a reference posture, determining a relative posture of a fifth camera adjacent to the fourth camera with respect to the fourth camera;

[0028] Taking the posture of the fifth camera as a reference posture, determining a relative posture of a sixth camera adjacent to the fifth camera with respect to the fifth camera;

[0029] Among the six cameras included in the surround view camera, the left front camera is adjacent to the front camera and the left rear camera, the left rear camera is adjacent to the rear camera, the rear camera is adjacent to the right rear camera, the right rear camera is adjacent to the right front camera, and the right front camera is adjacent to the front camera.

[0030] In combination with the first aspect, in a sixth possible implementation manner of the first aspect, determining the center of the surround view camera according to the relative posture includes:

[0031] Determine the lateral centerline based on the positions of the left and right cameras;

[0032] Determine the longitudinal centerline based on the poses of the front and rear cameras;

[0033] The center of the surround view camera is determined according to the intersection of the transverse center line and the longitudinal center line.

[0034] A second aspect of an embodiment of the present application provides a calibration device for a surround view camera, which is used for calibrating a scene of a surround view camera of a vehicle, wherein the left and right sides of the vehicle respectively include at least one lane, and a calibration line is provided in an overlapping area of ​​the field of view of two adjacent cameras, and the surround view camera includes front and rear cameras and left and right cameras, and the device includes:

[0035] A front and rear camera calibration unit, configured to determine a rotation matrix of the front and rear cameras according to a first lane line feature in an image captured by the front and rear cameras;

[0036] A left and right camera calibration unit, used to determine the rotation matrix of the left and right cameras according to the second lane line features in the images captured by the left and right cameras;

[0037] A relative posture determination unit, used to determine the relative posture of adjacent cameras based on lane lines and demarcation lines in images of adjacent cameras;

[0038] A translation matrix determining unit is used to determine the center of the surround-view camera according to the relative position and posture, and determine the translation matrix of the surround-view camera according to the center and the relative position and posture.

[0039] A third aspect of an embodiment of the present application provides a calibration device for a surround view camera, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in any one of the first aspects when executing the computer program.

[0040] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in any one of the first aspects are implemented.

[0041] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the embodiments of the present application calibrate the rotation matrix of the surround-view camera by calibrating the first lane line features and the second lane line features in the scene, calibrate the translation matrix of the surround-view camera by the calibration lines and lane lines in the adjacent camera images, and set calibration lines in the overlapping areas of the fields of view of adjacent cameras, which helps to reduce the probability that the calibration points do not exist or cannot be selected, thereby improving the robustness and accuracy of the surround-view camera calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 It is a schematic diagram of an implementation flow of a calibration method for a surround view camera provided in an embodiment of the present application;

[0044] Figure 2 It is a schematic diagram of an implementation flow of a calibration method for a surround view camera provided in an embodiment of the present application;

[0045] Figure 3 It is a schematic diagram of an implementation flow of determining the rotation matrix of the front and rear cameras provided in an embodiment of the present application;

[0046] Figure 4 It is a schematic diagram of an implementation flow of determining the rotation matrix of the left and right cameras provided in an embodiment of the present application;

[0047] Figure 5 A schematic diagram of an implementation flow of determining the relative positions of adjacent cameras provided in an embodiment of the present application;

[0048] Figure 6 is a schematic diagram of a calibration device for a surround view camera provided in an embodiment of the present application;

[0049] Figure 7 It is a schematic diagram of a calibration device for a surround view camera provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0051] In order to illustrate the technical solution described in this application, a specific embodiment is provided below for illustration.

[0052] With the development of automobile-related technologies, the safety and convenience of automobile driving have been effectively improved. For example, the surround view image can be generated through the vehicle's surround view camera. Through the joint work of multiple on-board cameras, the images collected by multiple on-board cameras are fused, which can effectively reduce the blind spots or blind spots of the driver's field of vision and improve driving safety.

[0053] Due to the wide variety of vehicles produced by commercial vehicle manufacturers, the installation positions and angles of surround-view cameras of different models vary greatly. For example, the greater the difference in the wheelbase of the vehicle, the greater the difference in the installation position and angle of the surround-view camera. When calibrating the surround-view camera, it is usually necessary to rely on the target in the image captured by the camera. However, for non-customized sites, the target in the camera's field of view may be unclear or distorted, especially for flatbed trailers, muck trucks, mixer trucks, etc. where the rear-facing camera is installed at a lower position. The target is often at the edge of the image, and it is easy for the calibration point to be inaccurately detected or undetectable during calibration, which is not conducive to improving the calibration accuracy of the surround-view camera.

[0054] Based on this, the present application embodiment proposes a calibration method for a surround view camera, such as Figure 1 Shown is a schematic diagram of an implementation scenario of the calibration method for a surround-view camera shown in an embodiment of the present application. In this implementation scenario, four parallel lines of equal width are drawn on a flat ground, and the four parallel lines of equal width constitute three lanes of equal width. The vehicle whose surround-view camera is to be calibrated is the middle vehicle among the three lanes, and the body of the vehicle is parallel to the lane lines. The width of the lane is greater than the width of the vehicle where the surround-view camera to be calibrated is located. For example, the width of a vehicle equipped with a surround-view camera is 2.5 meters, and the width of the lane can be any value between 3 meters and 3.5 meters. Calibration lines are set at the left front, right front, left rear, and right rear of the vehicle. The calibration lines are located in the overlapping area of ​​the field of view of two adjacent cameras. For example, the calibration line in the left front is located in the overlapping area of ​​the field of view of the left camera and the front camera, the calibration line in the right front is located in the overlapping area of ​​the field of view of the right camera and the front camera, the calibration line in the left rear is located in the overlapping area of ​​the field of view of the left camera and the rear camera, and the calibration line in the right rear is located in the overlapping area of ​​the field of view of the right camera and the rear camera. The calibration line can be a paint line drawn with a paintbrush, or it can be a pasted tape. Calibration lines in different positions can be set according to the vehicle where the surround view camera to be calibrated is located. For example, calibration lines with different front and rear distances can be set according to the different wheelbases of the vehicle. The calibration line can be set to appear in the overlapping area of ​​the field of view of adjacent cameras corresponding to the calibration line, and the inclination angle of the calibration line relative to the lane line or the vehicle body can be not specifically limited.

[0055] The surround view camera in the embodiment of the present application may include different numbers of cameras depending on the vehicle in which the surround view camera is located. For example, for a vehicle with a smaller wheelbase, such as an ordinary car, the surround view camera may include four cameras, namely left and right cameras, and front and rear cameras. Among them, the left and right cameras include a left camera and a right camera, and the front and rear cameras include a front camera and a rear camera. For a vehicle with a larger wheelbase, such as a semi-trailer vehicle, the surround view camera may include six cameras, namely left and right cameras, and front and rear cameras. Among them, the left and right cameras include a left camera and a right camera, the left camera includes a left front camera and a left rear camera, the right camera includes a right front camera and a right rear camera, and the front and rear cameras include a front camera and a rear camera.

[0056] Figure 2 A schematic diagram of the implementation flow of a calibration method for a surround view camera provided in an embodiment of the present application is described in detail as follows:

[0057] In S201, the rotation matrices of the front and rear cameras are determined according to the first lane line features in the images captured by the front and rear cameras.

[0058] Among them, the first lane line feature for determining the rotation matrix of the front and rear cameras may include the lane width, the vanishing point position, and the lane line parallel feature. According to the lane line parallel feature and the characteristics of the image being larger near and smaller far away, the lane line will gradually disappear into a point in the image, and this point is the vanishing point. The vanishing point position feature may include the position of the vanishing point of the lane line in the image captured by the front camera in the image, and the position of the vanishing point of the lane line in the image captured by the rear camera in the image.

[0059] The process of determining the rotation matrix of the front and rear cameras can be as follows Figure 3 As shown, including:

[0060] In S301, the pitch angle and yaw angle of the front and rear cameras are determined according to the position of the vanishing points in the images captured by the front and rear cameras and the constraint that the lane lines are parallel.

[0061] Since lane lines are parallel straight lines, in the images captured by the front camera or the rear camera, due to the effect of near being larger and far being smaller, the parallel lane lines will intersect at one point, and the intersection point of the lane lines is the vanishing point of the lane lines. The position of the vanishing point can reflect the pitch angle and yaw angle of the camera. For example, when the yaw angle of the camera changes, when the yaw angle changes to the left, the position of the vanishing point changes to the right in the captured image, and when the yaw angle changes to the right, the position of the vanishing point changes to the left in the captured image. When the pitch angle changes downward, the position of the vanishing point in the captured image changes upward, and when the pitch angle changes upward, the position of the vanishing point in the captured image changes downward. The pitch angle and yaw angle of the front camera, as well as the pitch angle and yaw angle of the rear camera can be determined based on the corresponding relationship between the change amplitude of the position of the vanishing point and the size of the pitch angle and yaw angle.

[0062] In S302, the roll angles of the front and rear cameras are determined according to the widths of the three lanes in the images captured by the front and rear cameras.

[0063] When determining the roll angles of the front and rear cameras, the roll angles of the left and right cameras may be determined according to the width of the lane of the images captured by the front and rear cameras.

[0064] The captured images usually include three lanes, that is, the front image captured by the front camera includes three lanes, and the image captured by the rear camera includes three lanes. When the roll angle of the front and rear cameras is zero, the widths of the three lanes in the front image and the rear image are the same.

[0065] When the roll angles of the front and rear cameras are not zero, for example, when the front camera starts with a roll angle of zero and rolls counterclockwise for a certain angle, in the collected front image, the width of the left lane becomes smaller, the width of the right lane increases, and the width of the left lane is smaller than the width of the right lane. When the front camera starts with a roll angle of zero and rolls clockwise for a certain angle, the width of the left lane increases, the width of the right lane decreases, and the width of the right lane is smaller than the width of the left lane.

[0066] The corresponding roll angle may be determined according to the difference in width between the left and right lanes in the collected front image. For example, the collected image may be optimized according to the optimization goal of the same width of the left and right lanes, and when the optimization goal is met, the roll angle corresponding to the optimization goal is determined.

[0067] In S202, the rotation matrices of the left and right cameras are determined according to the second lane line features in the images captured by the left and right cameras.

[0068] The second lane line feature is used to determine the rotation matrix of the left and right cameras. The second lane line feature may include features such as the same lane width, the vehicle is parallel to the lane line, and the lane width. The rotation matrix of the left and right cameras can be determined based on the second lane line feature. Figure 4 As shown, including:

[0069] In S401, the roll angles of the left and right cameras are determined according to the feature that the widths of the lane lines in the images captured by the left and right cameras are the same.

[0070] When the roll angle of the left and right cameras is zero, the width of the lane lines in the images captured by the left and right cameras can be the same. When the roll angle of the left camera or the right camera is forward roll, the width of the lane lines in the same lane in the captured image changes. The roll angle can be adjusted based on the same width of the lane lines in the same lane as the optimization goal, and the roll angle corresponding to the optimization goal can be determined. Alternatively, the roll angle corresponding to the change amplitude and change direction of the width of the lane can be determined based on the change amplitude and change direction of the width.

[0071] In S402, the yaw angles of the left and right cameras are determined according to the feature that the vehicle is parallel to the lane line.

[0072] When the yaw angles of the left and right cameras are zero, since the vehicle is parallel to the lane line, after distortion correction, the lane line in the images captured by the left and right cameras is a horizontal line. When the yaw angles of the left and right cameras change, the lane line in the captured image will tilt. For example, when the yaw angles of the left and right cameras yaw forward (forward is the direction of the front of the vehicle), the lane line in the image captured by the left camera will tilt to the left, that is, the lane line in the image captured by the left camera is lower on the left and higher on the right; in the image captured by the right camera, the lane line will tilt to the right, that is, the lane line in the image captured by the right camera is higher on the left and lower on the right. When the yaw angles of the left and right cameras yaw backward (toward the rear of the vehicle), in the image captured by the left camera, the lane line will tilt to the right, that is, the lane line in the image captured by the left camera is lower on the right and higher on the left; in the image captured by the right camera, the lane line will tilt to the left, that is, the lane line in the image captured by the right camera is lower on the left and higher on the right.

[0073] The yaw angles of the left and right cameras corresponding to the optimization target can be determined based on the lane lines in the collected image being in the horizontal direction. Alternatively, the yaw angles corresponding to the left and right cameras can be determined based on the inclination angles of the lane lines in the image.

[0074] In S403, the pitch angles of the left and right cameras are determined according to the predetermined heights of the left and right cameras and in combination with the lane width feature.

[0075] The height of the left and right cameras can be determined by pre-measurement. After the height of the left and right cameras relative to the ground is determined, the width of the lane in the collected image can be determined according to the pitch angles of the left and right cameras.

[0076] For example, for the left camera or the right camera at the same height, when the pitch angle changes, the width of the lane line in the captured image will also change. When the pitch angle changes upward, the width of the lane line in the captured image becomes smaller, and when the pitch angle changes downward, the width of the lane line in the captured image becomes larger. The size of the pitch angle of the left and right cameras can be determined based on the change in the width of the lane line in the image relative to the standard width of the lane line, that is, the change in the width of the lane line in the image captured by the left and right cameras when the pitch angle is zero. Alternatively, the width of the lane line corresponding to the pitch angle of zero can be used as the optimization target, and the pitch angles of the left and right cameras corresponding to the optimization target can be determined.

[0077] In S203, the relative positions of the adjacent cameras are determined according to the lane lines and the calibration lines in the images of the adjacent cameras.

[0078] When determining the relative pose of adjacent cameras, you can first set any camera as the reference camera, and determine the relative pose of the reference camera and the adjacent camera based on the coincidence of lane lines or calibration lines in the overlapping field of view of the images collected by the reference camera and the adjacent camera as the optimization target. By calculating the relative poses of multiple adjacent cameras, the relative poses of each camera in the surround view camera relative to the same camera can be obtained.

[0079] In an embodiment of the present application, when the surround view camera includes four cameras, namely, a left and right camera and a front and rear camera, the adjacent cameras may include a front camera and a left camera, a front camera and a right camera, a rear camera and a left camera, and a rear camera and a right camera. When determining the relative posture, the posture of the first camera may be used as a reference posture to determine the relative postures of the second camera and the third camera adjacent thereto. Then, the posture of the second camera is used as a reference posture to determine the relative posture of the fourth camera relative to the second camera. Based on the relative posture of the fourth camera relative to the second camera, and the relative posture of the second camera relative to the first camera, the relative posture of the fourth camera relative to the first camera may be determined.

[0080] When the surround view camera includes six cameras (front camera, rear camera, front left camera, front right camera, rear left camera and rear right camera), the adjacent cameras include the front camera and the front left camera, the front camera and the front right camera, the front left camera and the rear left camera, the front right camera and the rear right camera, the rear camera and the rear left camera, and the rear camera and the rear right camera.

[0081] Similar to the four cameras included in the surround view camera, when determining the relative poses of the six cameras, the first camera pose can be used as the reference pose to determine the relative poses of the second and third cameras. Taking the pose of the second camera as the reference pose, the relative pose of the fourth camera relative to the second camera is determined. Combined with the relative pose of the second camera relative to the first camera, the relative pose of the fourth camera relative to the first camera can be determined. Further, the relative pose of the fifth camera relative to the first camera and the relative pose of the sixth camera relative to the first camera can be determined. It can be understood that when determining the relative pose of the sixth camera relative to the first camera, the relative pose of the sixth camera relative to the first camera can also be determined based on the relative pose of the sixth camera relative to the third camera.

[0082] In a possible implementation, for a surround view camera including four cameras, when the first camera is a left camera, the relative position of the front camera with respect to the left camera can be determined, specifically as follows: Figure 5 As shown, including:

[0083] In S501, according to the predetermined height of the left camera, the height of the front camera relative to the left camera is determined in combination with the optimization goal of making the width of the left lane in the images captured by the front camera and the left camera the same.

[0084] Due to the different camera heights, the width of the lane at the same position in the captured image will also change. Therefore, the width of the lane in the image captured by the front camera can be optimized based on the same width at the same position in the image as the optimization goal. When the width of the lane in the image captured by the front camera is the same as the width of the lane in the image captured by the left camera at the same position, the relative height of the front camera after optimization calculation is determined based on the calibrated height of the left camera.

[0085] In S502, the lateral position of the front camera relative to the left camera is determined according to the overlap of lane lines in the images captured by the front camera and the left camera as an optimization target.

[0086] The images captured by the front camera and the left camera include an overlapping area, and there is a lane line in the overlapping area. The image area belonging to the same lane line in the images captured by the front camera and the left camera can be optimized, and the coincidence of the lane lines in the overlapping area is taken as the optimization target to determine the lateral pose of the front camera relative to the left camera. The lateral pose is the pose in the x-axis direction in the camera coordinate system.

[0087] In S503, the longitudinal position of the front camera relative to the left camera is determined according to the coincidence of calibration lines in the images captured by the front camera and the left camera as an optimization target.

[0088] According to the calibration lines included in the images captured by the front camera and the left camera, the longitudinal pose of the front camera is optimized and calculated with the coincidence of the calibration lines in the image as the optimization goal. When the calibration lines meet the coincidence requirements, the longitudinal pose of the front camera relative to the left camera is calculated. The longitudinal pose is the pose along the y-axis in the camera coordinate system.

[0089] According to the determined relative height, relative lateral position and relative longitudinal position of the front camera, the relative position of the front camera with respect to the left camera can be determined.

[0090] Based on the same method, the relative position and posture of the rear camera relative to the left camera can be determined, and the relative position and posture of the right camera relative to the rear camera, or the relative position and posture of the right camera relative to the front camera can be determined.

[0091] In S204, the center of the surround view camera is determined according to the relative position and posture, and the translation matrix of the surround view camera is determined according to the center and the relative position and posture.

[0092] After determining the relative positions of each camera in the surround-view camera, the lateral center line of the lateral position of the left and right cameras and the longitudinal center line of the longitudinal position of the front and rear cameras can be determined. The intersection of the lateral center line and the longitudinal center line is used as the center of the surround-view camera. The center is used as the origin to determine the transformation matrix of each camera translated to the origin, which is the translation matrix of the surround-view camera.

[0093] The embodiment of the present application sets the calibration line and takes advantage of the line extension to effectively reduce the probability of inaccurate or unselected calibration points for long-axis vehicles, which is conducive to increasing the robustness and accuracy of calibration. In addition, the requirements for the calibration line at the overlapping field of view of adjacent cameras are low, and it can be adjusted according to different vehicle models, which can reduce the calibration workload and meet the calibration requirements of different wheelbase models.

[0094] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0095] Figure 6 A schematic diagram of a calibration device for a surround view camera provided in an embodiment of the present application. In a scenario for calibrating a surround view camera of a vehicle, at least one lane is included on the left and right sides of the vehicle, and a calibration line is provided in the overlapping area of ​​the field of view of two adjacent cameras. The surround view camera includes front and rear cameras and left and right cameras. The device includes:

[0096] A front and rear camera calibration unit 601, configured to determine a rotation matrix of the front and rear cameras according to a first lane line feature in an image captured by the front and rear cameras;

[0097] A left and right camera calibration unit 602, configured to determine a rotation matrix of the left and right cameras according to a second lane line feature in the images captured by the left and right cameras;

[0098] A relative posture determination unit 603, used to determine the relative posture of adjacent cameras based on lane lines and demarcation lines in images of adjacent cameras;

[0099] The translation matrix determining unit 604 is used to determine the center of the surround-view camera according to the relative position and posture, and determine the translation matrix of the surround-view camera according to the center and the relative position and posture.

[0100] Figure 6 The calibration device of the surround view camera shown in FIG. Figure 2 The calibration method of the surround view camera shown corresponds to .

[0101] Figure 7 FIG. 1 is a schematic diagram of a calibration device for a surround view camera provided in an embodiment of the present application. Figure 7 As shown, the calibration device 7 for the surround view camera of this embodiment includes: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70, such as a calibration program for the surround view camera. When the processor 70 executes the computer program 72, the steps in the above-mentioned calibration method embodiments for the surround view camera are implemented. Alternatively, when the processor 70 executes the computer program 72, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0102] Exemplarily, the computer program 72 may be divided into one or more modules / units, which are stored in the memory 71 and executed by the processor 70 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 72 in the calibration device 7 of the surround view camera.

[0103] The calibration device of the surround view camera may include, but is not limited to, a processor 70 and a memory 71. Those skilled in the art will appreciate that Figure 7 It is only an example of the calibration device 7 for the surround-view camera and does not constitute a limitation of the calibration device 7 for the surround-view camera. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the calibration device for the surround-view camera may also include input and output devices, network access devices, buses, etc.

[0104] The processor 70 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0105] The memory 71 may be an internal storage unit of the calibration device 7 of the surround view camera, such as a hard disk or memory of the calibration device 7 of the surround view camera. The memory 71 may also be an external storage device of the calibration device 7 of the surround view camera, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the calibration device 7 of the surround view camera. Further, the memory 71 may also include both an internal storage unit and an external storage device of the calibration device 7 of the surround view camera. The memory 71 is used to store the computer program and other programs and data required by the calibration device of the surround view camera. The memory 71 may also be used to temporarily store data that has been output or is to be output.

[0106] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0107] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0108] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0109] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0110] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0111] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0112] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by hardware related to computer program instructions. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0113] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for calibrating a surround view camera, characterized in that: In a scenario for calibrating a surround view camera of a vehicle, at least one lane is included on the left and right sides of the vehicle, and a calibration line is provided in an overlapping area of ​​the fields of view of two adjacent cameras, and the surround view camera includes front and rear cameras and left and right cameras. The method includes: Determining a rotation matrix of the front and rear cameras according to a first lane line feature in the images captured by the front and rear cameras; Determine the rotation matrix of the left and right cameras according to the second lane line features in the images captured by the left and right cameras; Determine the relative positions of adjacent cameras based on the lane lines and calibration lines in the images of adjacent cameras; The center of the surround-view camera is determined according to the relative position and posture, and the translation matrix of the surround-view camera is determined according to the center and the relative position and posture.

2. The method according to claim 1, characterized in that The first lane line feature includes a lane width, a vanishing point position, and a lane line parallel feature. According to the first lane line feature in the images captured by the front and rear cameras, a rotation matrix of the front and rear cameras is determined, including: Determine the pitch angle and yaw angle of the front and rear cameras according to the constraint that the vanishing point positions in the images collected by the front and rear cameras are parallel to the lane lines; The roll angles of the front and rear cameras are determined according to the widths of the three lanes in the images captured by the front and rear cameras.

3. The method according to claim 1, characterized in that The second lane line features include the same lane width, the vehicle and the lane line are parallel, and the lane width feature. According to the second lane line features in the images captured by the left and right cameras, the rotation matrix of the left and right cameras is determined, including: Determining the roll angles of the left and right cameras according to a feature that the widths of the lane lines in the images captured by the left and right cameras are the same; Determining the yaw angles of the left and right cameras according to the feature that the vehicle is parallel to the lane line; The pitch angles of the left and right cameras are determined according to the predetermined heights of the left and right cameras in combination with the lane width feature.

4. The method according to claim 1, characterized in that: When the surround view camera includes four cameras, determining the relative positions of adjacent cameras according to lane lines and demarcation lines in images of adjacent cameras includes: Taking the first camera posture as a reference posture, determining the relative postures of a second camera and a third camera adjacent to the first camera with respect to the first camera; Taking the posture of the second camera as a reference posture, a relative posture of a fourth camera adjacent to the second camera with respect to the second camera is determined.

5. The method according to claim 4, characterized in that In the case where the first camera is a left camera and the second camera is a front camera, taking the first camera posture as a reference posture, determining the relative posture of the second camera adjacent to the first camera relative to the first camera includes: According to the predetermined height of the left camera, the height of the front camera relative to the left camera is determined in combination with the optimization goal of making the width of the left lane in the images captured by the front camera and the left camera the same; Determine the lateral position of the front camera relative to the left camera based on the overlap of lane lines in the images captured by the front camera and the left camera as an optimization target; The longitudinal position of the front camera relative to the left camera is determined based on the coincidence of calibration lines in the images captured by the front camera and the left camera as an optimization target.

6. The method according to claim 1, characterized in that When the surround view camera includes six cameras, determining the relative positions of adjacent cameras according to lane lines and demarcation lines in images of adjacent cameras includes: Taking the first camera posture as a reference posture, determining the relative postures of a second camera and a third camera adjacent to the first camera with respect to the first camera; Taking the posture of the second camera as a reference posture, determining a relative posture of a fourth camera adjacent to the second camera relative to the second camera; Taking the posture of the fourth camera as a reference posture, determining a relative posture of a fifth camera adjacent to the fourth camera with respect to the fourth camera; Taking the posture of the fifth camera as a reference posture, determining a relative posture of a sixth camera adjacent to the fifth camera with respect to the fifth camera; Among the six cameras included in the surround view camera, the left front camera is adjacent to the front camera and the left rear camera, the left rear camera is adjacent to the rear camera, the rear camera is adjacent to the right rear camera, the right rear camera is adjacent to the right front camera, and the right front camera is adjacent to the front camera.

7. The method according to claim 1, characterized in that Determining the center of the surround view camera according to the relative position and posture includes: Determine the lateral centerline based on the positions of the left and right cameras; Determine the longitudinal centerline based on the poses of the front and rear cameras; The center of the surround view camera is determined according to the intersection of the transverse center line and the longitudinal center line.

8. A calibration device for a surround view camera, characterized in that: In a scenario for calibrating a surround view camera of a vehicle, at least one lane is included on the left and right sides of the vehicle, and a calibration line is provided in the overlapping area of ​​the field of view of two adjacent cameras, the surround view camera includes front and rear cameras and left and right cameras, and the device includes: A front and rear camera calibration unit, configured to determine a rotation matrix of the front and rear cameras according to a first lane line feature in an image captured by the front and rear cameras; A left and right camera calibration unit, used to determine the rotation matrix of the left and right cameras according to the second lane line features in the images captured by the left and right cameras; A relative posture determination unit, used to determine the relative posture of adjacent cameras based on lane lines and demarcation lines in images of adjacent cameras; A translation matrix determining unit is used to determine the center of the surround-view camera according to the relative position and posture, and determine the translation matrix of the surround-view camera according to the center and the relative position and posture.

9. A calibration device for a surround view camera, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.