External parameter calibration method, device and equipment of vehicle-mounted all-round view camera and storage medium
By collecting images by vehicle-mounted surround viewing cameras and automatically computing the rotation matrix and translation matrix using image overlap characteristics, the problem of low degree of automation of external parameter calibration in the prior art is solved, and an efficient and accurate external parameter calibration process is achieved.
Patent Information
- Application Number
- CN202310593396.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-24
- Publication Date
- 2024-11-26
AI Technical Summary
In the prior art, the automatic calibration scheme of the outer parameters of the circumferential camera based on lane lines has high requirements for initial conditions, low degree of automation and easy to cause errors.
The image is collected by the on-board surround view camera, the stop line equation and the lane line equation are determined, and the rotation matrix and translation matrix are automatically calculated based on the image overlap characteristics, and the external parameters are optimized to improve calibration accuracy and automation.
It realizes efficient and automated external parameter calibration of vehicle-mounted surround view cameras without the need for initial external parameter matrix and manual measurement, reducing the error and cost of manual calibration.
Smart Images

Figure CN119027505A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of camera calibration, and in particular to an external parameter calibration method, device, equipment and storage medium for a vehicle-mounted surround view camera. Background Art
[0002] In order to improve driving safety, the vehicle surround camera can be used to collect images around the vehicle, and the collected images can be fused to obtain a scene image that eliminates the driver's visual blind spot. In order to accurately and effectively stitch and fuse the images, the external parameters (i.e., extrinsic parameters) of each camera in the vehicle surround camera need to be calibrated.
[0003] When calibrating the external parameters of the vehicle-mounted surround view camera, it is usually based on a fixed calibration scene. This calibration scene is suitable for batch production installation. For a single set or a small batch of after-installed vehicle-mounted surround view cameras, or after-sales vehicle-mounted surround view cameras, if a fixed site is given for calibration, the site cost and time cost are high. If a surround view camera external parameter automatic calibration solution based on lane lines is adopted, it is usually necessary to provide an initial external parameter matrix, and the translation matrix needs to be manually measured. The initial conditions are required to be high, the degree of automation is not high, and errors are prone to occur. Summary of the invention
[0004] In view of this, the embodiments of the present application provide a method, device, equipment and storage medium for extrinsic calibration of a vehicle-mounted surround view camera to solve the problems in the prior art of using a lane-line-based surround view camera extrinsic automatic calibration scheme, which has high requirements on initial conditions, low degree of automation and is prone to errors.
[0005] A first aspect of an embodiment of the present application provides a method for calibrating extrinsic parameters of a vehicle-mounted surround view camera, wherein the extrinsic parameters include a rotation matrix and a translation matrix, and the method includes:
[0006] Collecting images through a vehicle-mounted surround view camera, determining a stop line equation included in the image, and lane line equations corresponding to left and right lanes;
[0007] Determine the rotation matrices of the left and right cameras according to the stop line equation and the lane line equation, and determine the rotation matrices of the front and rear cameras according to the width consistency of the left and right lanes;
[0008] The translation matrix of the vehicle-mounted surround view camera is determined according to the overlap features of the images captured by the front and rear cameras and the left and right cameras.
[0009] In combination with the first aspect, in a first possible implementation manner of the first aspect, determining a translation matrix of the vehicle-mounted surround view camera according to overlapping features of images captured by the front and rear cameras and the left and right cameras includes:
[0010] Determining a translation matrix of the vehicle-mounted surround view camera according to image consistency of overlapping areas between adjacent images collected by the vehicle-mounted surround view camera;
[0011] After determining the translation matrix of the vehicle-mounted surround view camera according to the overlap features of the images captured by the front and rear cameras and the left and right cameras, the method further includes:
[0012] The external parameters are optimized with the goal of minimizing the error of the overlapping area between adjacent images collected by the on-board surround view camera.
[0013] In combination with the first possible implementation manner of the first aspect, in a second possible implementation manner of the first aspect, determining the translation matrix of the vehicle-mounted surround view camera according to image consistency of overlapping areas between adjacent images captured by the vehicle-mounted surround view camera includes:
[0014] Determine the relative positions of the cameras corresponding to the adjacent images according to the image consistency of the image overlap areas of the adjacent images;
[0015] Determining a camera center according to the relative pose;
[0016] A translation matrix of the camera is determined according to the relative position and the camera center.
[0017] In combination with the second possible implementation manner of the first aspect, in a third possible implementation manner of the first aspect, determining the relative poses of cameras corresponding to adjacent images according to image consistency of image overlap areas of adjacent images includes:
[0018] Calibrate the relative height of the camera according to the consistency of lane widths of the adjacent images;
[0019] Calibrate the relative lateral pose of the camera according to the lane overlap characteristics of the adjacent images;
[0020] The relative longitudinal position of the camera is calibrated according to the overlapping characteristics of the stop lines of the adjacent images.
[0021] In combination with the first aspect, in a fourth possible implementation manner of the first aspect, collecting images through a vehicle-mounted surround view camera includes:
[0022] Control the vehicle to travel along the lane line, collect images including the left and right lanes through the left and right cameras, and collect images including the current driving lane and the left and right lanes through the front and rear cameras;
[0023] When the vehicle moves to a point where the images captured by the front camera, the left camera, and the right camera all include the stop line, the front camera and the left and right cameras capture images that all include the stop line at the same time;
[0024] When the vehicle moves to a point where the images captured by the rear camera, the left camera, and the right camera all include the stop line, the rear camera and the left and right cameras capture images that all include the stop line at the same time.
[0025] In combination with the first aspect, in a fifth possible implementation manner of the first aspect, the rotation matrix includes a yaw angle, a roll angle, and a pitch angle, and the rotation matrices of the left and right cameras are determined according to the stop line equation and the lane line equation, including:
[0026] Determining the roll angles of the left and right cameras according to the consistency of the widths of the lanes in the image;
[0027] Optimizing the slopes of lane line equations in the left and right camera images to determine the yaw angles of the left and right cameras;
[0028] The pitch angles of the left and right cameras are determined by optimizing the stop lines of different frames to be parallel and the slopes of the stop line equations to be the same.
[0029] In combination with the first aspect, in a sixth possible implementation manner of the first aspect, the rotation matrix includes a yaw angle, a roll angle, and a pitch angle, and the rotation matrix of the front and rear cameras is determined according to the width consistency of the left and right lanes, including:
[0030] Determine the real width ratio of the left and right lanes of the vehicle according to the rotation matrix of the left and right cameras and the predetermined height of the left and right cameras;
[0031] Determining the roll angle of the front and rear cameras according to the consistency between the width ratio of the left lane in the front and rear images and the real width ratio;
[0032] The pitch angle and yaw angle of the front and rear cameras are determined according to the vanishing points in the images collected by the front and rear cameras.
[0033] A second aspect of an embodiment of the present application provides an external parameter calibration device for a vehicle-mounted surround view camera, wherein the external parameters include a rotation matrix and a translation matrix, and the device includes:
[0034] An image acquisition unit, used to acquire images through a vehicle-mounted surround view camera, and determine a stop line equation included in the image, and lane line equations corresponding to left and right lanes;
[0035] A rotation matrix determination unit, used to determine the rotation matrices of the left and right cameras according to the stop line equation and the lane line equation, and to determine the rotation matrices of the front and rear cameras according to the width consistency of the left and right lanes;
[0036] The translation matrix determination unit is used to determine the translation matrix of the vehicle-mounted surround view camera according to the overlap features of the images collected by the front and rear cameras and the left and right cameras.
[0037] A third aspect of an embodiment of the present application provides an external parameter calibration device for a vehicle-mounted surround view camera, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the methods described in the first aspect when executing the computer program.
[0038] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in any one of the first aspects are implemented.
[0039] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the embodiments of the present application collect images through a vehicle-mounted surround-view camera, determine the stop line equation and the lane line equations of the left and right lanes included in the image, determine the rotation matrices of the left and right cameras according to the stop line equation and the lane line equation, determine the rotation matrices of the front and rear cameras according to the width consistency of the left and right lanes, and determine the translation matrix of the vehicle-mounted environment camera according to the overlapping features of the images collected by the front and rear cameras and the left and right cameras. Therefore, the external parameters of the vehicle-mounted surround-view camera can be effectively calibrated without the need for an initial external parameter matrix and without the need for manual measurement of the translation matrix. The method has a high degree of automation and is conducive to reducing errors caused by manual calibration. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0041] Figure 1 It is a schematic diagram of the implementation flow of an external parameter calibration method for a vehicle-mounted surround view camera provided in an embodiment of the present application;
[0042] Figure 2 It is a schematic diagram of the implementation process of a camera image acquisition method provided in an embodiment of the present application;
[0043] Figure 3 This is a schematic diagram of a scene when image acquisition starts, provided in an embodiment of the present application;
[0044] Figure 4 This is a schematic diagram of a scene when image acquisition ends, provided in an embodiment of the present application;
[0045] Figure 5 It is a schematic diagram of an implementation flow of determining the rotation matrix of the left and right cameras provided in an embodiment of the present application;
[0046] Figure 6 It is a schematic diagram of an implementation flow of determining the rotation matrix of the front and rear cameras provided in an embodiment of the present application;
[0047] Figure 7 is a schematic diagram of an external parameter calibration device for a vehicle-mounted surround view camera provided in an embodiment of the present application;
[0048] Figure 8 It is a schematic diagram of an external parameter calibration device for a vehicle-mounted surround view camera provided in an embodiment of the present application. DETAILED DESCRIPTION
[0049] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0050] In order to illustrate the technical solution described in this application, a specific embodiment is provided below for illustration.
[0051] A vehicle-mounted surround view camera is a camera that can capture images of the vehicle's surroundings. It usually includes four cameras: a front camera, a rear camera, a left camera, and a right camera. By fusing the images captured by the front and rear cameras and the left and right cameras, a scene image that eliminates the driver's visual blind spots can be obtained. In order to accurately and effectively stitch and fuse the images, it is necessary to calibrate the external parameters of each camera in the vehicle-mounted surround view camera.
[0052] When calibrating the external parameters of a vehicle-mounted surround view camera, it is usually based on a fixed calibration scenario. This calibration scenario is suitable for batch production installation. For a single set or a small batch of after-installed vehicle-mounted surround view cameras, or after-sales vehicle-mounted surround view cameras, if a fixed site is given for calibration, the site cost and time cost are high.
[0053] In order to reduce the dependence on fixed sites during the calibration process, lane lines can be used to detect the external parameters of the onboard surround view camera. However, when the external parameters of the onboard surround view camera are detected based on lane lines, an initial external parameter matrix needs to be provided, and the translation matrix needs to be manually measured. The degree of automation of the calibration process is not high, and manual measurement is not conducive to improving the calibration results.
[0054] In order to solve the above problems, the embodiment of the present application proposes a method for calibrating the external parameters of a vehicle-mounted surround view camera. The method collects data of three lanes and stop lines at intersections, detects edges, and obtains lane line equations and stop line equations. The rotation matrix in the external parameters of the left and right cameras is calibrated by utilizing the straight and parallel characteristics of the lane lines, as well as the parallel characteristics of the stop lines in the images collected by the camera at different times when the vehicle passes the stop lines. The width of the left and right lanes is determined by the installation height of the left and right cameras, and the rotation matrix of the front and rear cameras is calculated based on the principle of parallelism between the width of the left and right lanes and the lane lines. Based on the principle that the same lane line or stop line collected by different cameras at the same time overlaps in the world coordinate system, the translation matrix of the four cameras can be calculated by the stitching error of the images collected by any three adjacent cameras. The external parameters of the four cameras can be evaluated and optimized based on the stitching error of the images collected by the fourth adjacent camera (outside the first three).
[0055] Among them, the implementation process of the external parameter calibration method of the vehicle-mounted surround view camera provided in the embodiment of the present application can be as follows: Figure 1 As shown, including:
[0056] In S101, an image is collected by a vehicle-mounted surround view camera to determine a stop line equation included in the image and lane line equations corresponding to left and right lanes.
[0057] The surround-view camera in the embodiment of the present application may include a total of four cameras, namely, a front camera, a rear camera, a left camera, and a right camera. The front camera and the rear camera can be described as front and rear cameras, and the left camera and the right camera can be described as left and right cameras. The images captured by the vehicle-mounted surround-view camera may include a front image captured by the front camera, a left image captured by the left camera, a right image captured by the right camera, and a rear image captured by the rear camera. The adjacent relationship of the cameras includes that the front camera is adjacent to the left camera, the front camera is adjacent to the right camera, the rear camera is adjacent to the left camera, and the rear camera is adjacent to the right camera. The images captured by adjacent cameras may partially overlap. The front in the embodiment of the present application refers to the forward direction of the vehicle.
[0058] The scenario for extrinsic parameter calibration in the embodiment of the present application may be a road with more than three lanes, and a scenario where the road includes a stop line at an intersection. On a road with more than three lanes, the left and right sides of the lane where the vehicle is located include at least one lane. For example, for a road with three lanes, the vehicle can drive in the middle lane to collect images. For a road with four lanes, the vehicle can drive in the second lane or the third lane and collect images while driving.
[0059] During the driving process of the vehicle, the left and right cameras can be used to capture images including the left and right lanes, and the front and rear cameras can be used to capture images including the current driving lane and the left and right lanes. When the images captured by the front camera, the left camera, and the right camera at the same time all include the stop line, the images captured at that moment are saved. And when the images captured by the rear camera, the left camera, and the right camera at the same time all include the stop line, the images captured at that moment are saved. The specific acquisition process can be as follows: Figure 2 As shown, including:
[0060] In S201, the vehicle is controlled to travel along the lane line, and images including the left and right lanes are collected by the left and right cameras, and images including the current driving lane and the left and right lanes are collected by the front and rear cameras.
[0061] In the embodiment of the present application, the images acquired by the front and rear cameras and the left and right cameras may include images obtained by using the internal parameters of the cameras to correct the distortion of the original images. Figure 3 As shown, when the vehicle starts to travel on a road with more than three lanes, the left camera can capture images including the left lane, the right camera can capture images including the right lane, the front camera can capture images including at least three lanes in front, and the rear camera can capture images including at least three lanes in the rear.
[0062] In S202, when the vehicle moves to a position where the images captured by the front camera, the left camera, and the right camera all include the stop line, when the vehicle moves to a position where the images captured by the front camera, the left camera, and the right camera all include the stop line, the front camera and the left and right cameras capture images that include the stop line at the same moment.
[0063] As the vehicle moves forward, it gradually approaches the intersection. Within a certain distance range between the vehicle and the stop line, the front camera and the left and right cameras can capture images including the stop line. One or more groups of images captured by the front camera and the left and right cameras at the same time can be recorded. A group of images includes the front image, the left image, and the right image, and the shooting time is the same.
[0064] In S203, when the vehicle moves to a point where the images captured by the rear camera, the left camera, and the right camera all include the stop line, when the vehicle moves to a point where the images captured by the rear camera, the left camera, and the right camera all include the stop line, the rear camera and the left and right cameras capture images that include the stop line at the same moment.
[0065] As the vehicle moves forward, the stop line moves backward relative to the vehicle, and the stop line gradually disappears in the front image captured by the front camera. After the stop line in the front image disappears, the stop line can be detected by the rear camera. Figure 4As shown, within a certain time range, the images captured by the rear camera and the left and right cameras of the vehicle at the same time all include the stop line. A group of images captured by the rear camera and the left and right cameras can be recorded, in which the rear image, the left image and the right image in the group of images are captured at the same time and all include the stop line.
[0066] When the rear camera, the left camera, and the right camera capture images including the stop line at the same time, S204 may be executed to stop the vehicle from moving and complete image capture.
[0067] In the embodiment of the present application, for the captured image, the lane line equation and the stop line equation can be determined according to the position of the lane line or the stop line in the image. For example, the straight line equation of the lane line or the stop line can be determined according to two or more points on the lane line or the stop line.
[0068] The lane line equation may include the lane line equation of the left lane in the vehicle's forward direction, the lane line equation of the right lane, and the lane line equation of the vehicle's current driving lane. Since the front camera, the rear camera, the left camera, and the right camera can all capture images including lane lines, the lane line equations corresponding to the lane lines in the image can be determined based on the lane lines included in the image. Accordingly, when the images captured by the front camera, the rear camera, the left camera, and the right camera include stop lines, the stop line equations in the image can be determined.
[0069] In S102, the rotation matrices of the left and right cameras are determined according to the stop line equation and the lane line equation, and the rotation matrices of the front and rear cameras are determined according to the width consistency of the left and right lanes.
[0070] The external parameters in the embodiments of the present application include parameters required to convert the posture in the world coordinate system into the posture in the image coordinate system, including rotation matrices and translation matrices.
[0071] The formula for converting the pose in the world coordinate system to the pose in the image coordinate system may include:
[0072]
[0073] Among them, K is the camera's intrinsic matrix, R is the camera's rotation matrix in the extrinsic parameters, T is the camera's translation matrix in the extrinsic parameters, and z c is the scaling ratio, (X w , Y w , Z w ) is the coordinate position in the world coordinate system, and (u, v) is the coordinate position in the image coordinate system. The rotation matrix can be determined by the camera's yaw angle Yaw, pitch angle Pitch, and roll angle Roll.
[0074] When determining the rotation matrix of the left and right cameras, you can do Figure 5 As shown, including:
[0075] In S501, the roll angles of the left and right cameras are determined according to the consistency of the widths of the lanes in the image.
[0076] The image captured by the left camera or the right camera includes the lane next to the vehicle. For example, the image captured by the left camera includes the left lane, and the image captured by the right camera includes the right lane. Since the widths of the actual lanes are basically the same. Therefore, the roll angles of the left and right cameras can be determined based on the optimization goal that the lane width in the captured left image is equal to the lane width in the right image. That is, by adjusting the roll angle, the lanes in the left and right images are converted to lanes in the world coordinate system. When the widths of the left and right lane lines in the world coordinate system are equal, the current roll angle is the roll angle corresponding to the left and right cameras. The roll angles corresponding to the left and right cameras can be determined by optimization methods such as gradient descent.
[0077] In S502, optimization is performed based on the slopes of the lane line equations in the left and right camera images to determine the yaw angles of the left and right cameras.
[0078] Since the vehicle is parallel to the lane line when driving, the slope of the lane line in the images captured by the left and right cameras tends to infinity. The yaw angles of the left and right cameras of the vehicle can be calculated based on the optimization goal of the slope of the lane line in the image changing to infinity.
[0079] The optimization process may use a gradient descent method to determine the optimized yaw angles of the left and right cameras.
[0080] The optimization formula of gradient descent can be expressed as: Yaw n+1 =Yaw n -lr*Yaw grad Among them, Yaw represents the yaw angle to be optimized, n is the number of optimization calculations, lr is the learning rate, and Yaw grad The yaw angles corresponding to the left and right cameras are obtained through continuous iterative calculations.
[0081] For example, by extracting the lane line equations in the left and right images (the lane line equation or the stop line equation can be an equation determined by the lane line edge or the stop line edge), converting them into the lane line equation in the world coordinate system, taking the maximum value of the slope k of the lane line equation in the converted world coordinate system as the optimization target, initializing lr to 1, iterating the yaw angle in the grad gradient direction, and calculating the absolute value of the lane line equation k in the converted world coordinate system when it starts to become smaller, according to Yaw gradIn the opposite direction, reduce the learning rate, for example, reduce it to 1 / 10 of the initial learning rate, and calculate the absolute value of the lane line equation k in the converted world coordinate system. When the absolute value of the lane line equation k in the converted world coordinate system begins to decrease again, adjust the direction to Yaw. grad The learning rate is reduced again to obtain the maximum value of the yaw angle that gradually increases. The iterative calculation is repeated by adjusting the learning rate and the learning direction until the preset optimization requirements are met, such as the learning rate reaches a predetermined value, or the number of iterations reaches a predetermined number.
[0082] In S503, optimization is performed based on the fact that the stop lines of different frames are parallel and the slopes of the stop line equations are the same, so as to determine the pitch angles of the left and right cameras.
[0083] Since the vehicle is traveling in a direction parallel to the lane line, the stop lines in the images captured by the left and right cameras of the vehicle at different times are parallel. The same slope of the stop lines in different images can be used as an optimization goal, such as using a gradient optimization method to determine the pitch angles of the left and right cameras.
[0084] According to the determined pitch angle, roll angle and yaw angle of the left and right cameras, the rotation matrix of the left and right cameras can be determined.
[0085] When determining the rotation matrix of the front and rear cameras, you can do Figure 6 As shown, including:
[0086] In S601, the actual width ratio of the left and right lanes of the vehicle is determined according to the rotation matrices of the left and right cameras in combination with the predetermined heights of the left and right cameras.
[0087] according to Figure 5 After the rotation matrix of the left and right cameras is determined by the method shown, the actual width of the left and right lanes of the vehicle can be calculated based on the rotation matrix of the left and right cameras, combined with the installation height of the left and right cameras and the internal parameters of the left and right cameras. Alternatively, in a possible implementation, the actual width of the left and right lanes in the scene can also be obtained in advance or actually measured. According to the determined actual widths of the left and right lanes, the ratio of the actual widths of the left and right lanes is obtained.
[0088] In S602, the roll angles of the front and rear cameras are determined according to the consistency between the width ratio of the left lane in the front and rear images and the real width ratio.
[0089] After the real width ratio of the left and right lanes is determined, the ratio of the left and right lanes in the image can be calculated based on the front and rear camera images. The roll angles of the front and rear cameras can be determined based on the consistency of the real width ratio and the width ratio of the left and right lanes in the image as the optimization target.
[0090] In S603, the pitch angle and yaw angle of the front and rear cameras are determined according to the vanishing points in the images captured by the front and rear cameras.
[0091] Since the lane lines are forward-facing and parallel, the pitch angle and yaw angle of the front and rear cameras can be calculated by using the positions of the vanishing points of the lane lines in the front and rear images and the degree of deviation from the preset standard pitch angle and yaw angle.
[0092] In S103, a translation matrix of the vehicle-mounted surround view camera is determined according to the overlap features of the images captured by the front and rear cameras and the left and right cameras.
[0093] The vehicle-mounted surround view camera in the embodiment of the present application includes front and rear cameras and left and right cameras. In the vehicle-mounted surround view camera, adjacent cameras include the front camera and the left camera, the front camera and the right camera, the rear camera and the left camera, and the rear camera and the right camera. The images captured by the adjacent cameras include partially overlapping images.
[0094] In an embodiment of the present application, the first group of adjacent images, the second group of adjacent images, the third group of adjacent images, and the fourth group of adjacent images can be determined based on the four images captured by the vehicle-mounted surround view camera. Any three groups of adjacent images can be selected, and the image consistency of the overlapping area can be used as the optimization target to calculate the translation matrix of the vehicle-mounted surround view camera. The external parameters of the vehicle-mounted surround view camera can be optimized based on the minimum error of the overlapping area of the remaining group of adjacent images. For example, the first group of adjacent images, the second group of adjacent images, and the third group of adjacent images can be used to calculate the translation matrix, and the fourth group of adjacent images can be used for external parameter optimization.
[0095] When calculating the translation matrix of the vehicle-mounted surround view camera, a reference camera can be pre-set, and the relative pose of the adjacent cameras can be determined based on the consistency of the overlapping area of the images collected by the reference camera and the camera to be calculated (the camera adjacent to the reference camera). Based on the same method, the relative pose of any two adjacent cameras can be determined, that is, the displacement matrix of the adjacent cameras. After determining the phase pose of the adjacent cameras, the center of the vehicle-mounted surround view camera can be determined based on the longitudinal midpoint of the front and rear cameras as the longitudinal origin and the transverse midpoint of the left and right cameras as the transverse origin. Based on the center, the translation matrix of each camera in the vehicle-mounted surround view camera relative to the center is determined, that is, the translation matrix of the vehicle-mounted surround view camera.
[0096] When determining the relative pose of the cameras corresponding to adjacent images based on the image consistency of the overlapping areas of adjacent images, the relative height of the camera can be calibrated based on the consistency of the lane width of the adjacent images, the relative lateral pose of the camera can be calibrated based on the characteristics of the lane overlap of the adjacent images, and the relative longitudinal pose of the camera can be calibrated based on the characteristics of the stop line overlap of the adjacent images. The translation matrix of the adjacent cameras can be determined based on the relative height, lateral pose and longitudinal pose. The center of the vehicle-mounted surround view camera can be determined based on the translation matrix of the reference camera and other cameras, and the translation matrix of the vehicle-mounted surround view camera can be determined based on the pose of the camera relative to the center.
[0097] The embodiment of the present application uses dynamic calibration to optimize the calculation using multiple sets of data to obtain iterative external parameters. The closed loop formed by calibration error feedback is conducive to improving the robustness and accuracy of the calibration results and reducing errors caused by manual operations. In addition, the draft scene in the embodiment of the present application is a common intersection with more than three lanes, which can effectively reduce environmental restrictions and help reduce calibration costs.
[0098] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0099] Figure 7 A schematic diagram of an extrinsic calibration device for a vehicle-mounted surround view camera provided in an embodiment of the present application, such as Figure 7 As shown, the device comprises:
[0100] An image acquisition unit 701 is used to acquire images through a vehicle-mounted surround view camera, and determine a stop line equation included in the image, and lane line equations corresponding to left and right lanes;
[0101] A rotation matrix determination unit 702 is used to determine the rotation matrices of the left and right cameras according to the stop line equation and the lane line equation, and determine the rotation matrices of the front and rear cameras according to the width consistency of the left and right lanes;
[0102] The translation matrix determining unit 703 is used to determine the translation matrix of the vehicle-mounted surround view camera according to the overlap features of the images collected by the front and rear cameras and the left and right cameras.
[0103] Among them, the external parameters of the vehicle-mounted surround view camera include the rotation matrix and the translation matrix.
[0104] Figure 7 The external parameter calibration device of the vehicle-mounted surround view camera shown in FIG. Figure 1 The extrinsic calibration method of the vehicle-mounted surround view camera shown corresponds to the extrinsic calibration method of the vehicle-mounted surround view camera shown.
[0105] Figure 8Schematic diagram of an external parameter calibration device for a vehicle-mounted surround view camera provided in one embodiment of the present application. Figure 8 As shown, the extrinsic parameter calibration device 8 of the vehicle-mounted surround view camera of this embodiment includes: a processor 80, a memory 81, and a computer program 82 stored in the memory 81 and executable on the processor 80, such as an extrinsic parameter calibration program of the vehicle-mounted surround view camera. When the processor 80 executes the computer program 82, the steps in the above-mentioned extrinsic parameter calibration method embodiments of the vehicle-mounted surround view camera are implemented. Alternatively, when the processor 80 executes the computer program 82, the functions of each module / unit in the above-mentioned device embodiments are implemented.
[0106] Exemplarily, the computer program 82 may be divided into one or more modules / units, which are stored in the memory 81 and executed by the processor 80 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 82 in the external parameter calibration device 8 of the vehicle-mounted surround view camera.
[0107] The external parameter calibration device of the vehicle-mounted surround view camera may include, but is not limited to, a processor 80 and a memory 81. Those skilled in the art will understand that Figure 8 It is only an example of the external parameter calibration device 8 of the vehicle-mounted surround view camera, and does not constitute a limitation of the external parameter calibration device 8 of the vehicle-mounted surround view camera. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the external parameter calibration device of the vehicle-mounted surround view camera may also include input and output devices, network access devices, buses, etc.
[0108] The processor 80 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0109] The memory 81 may be an internal storage unit of the external parameter calibration device 8 of the vehicle-mounted surround view camera, such as a hard disk or memory of the external parameter calibration device 8 of the vehicle-mounted surround view camera. The memory 81 may also be an external storage device of the external parameter calibration device 8 of the vehicle-mounted surround view camera, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the external parameter calibration device 8 of the vehicle-mounted surround view camera. Further, the memory 81 may also include both the internal storage unit and the external storage device of the external parameter calibration device 8 of the vehicle-mounted surround view camera. The memory 81 is used to store the computer program and other programs and data required by the external parameter calibration device of the vehicle-mounted surround view camera. The memory 81 may also be used to temporarily store data that has been output or is to be output.
[0110] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0111] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0112] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0113] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0114] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0115] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0116] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by hardware related to computer program instructions. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0117] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A method for calibrating external parameters of a vehicle-mounted surround view camera, characterized in that: The external parameters include a rotation matrix and a translation matrix, and the method includes: Collecting images through a vehicle-mounted surround view camera, determining a stop line equation included in the image, and lane line equations corresponding to left and right lanes; Determine the rotation matrices of the left and right cameras according to the stop line equation and the lane line equation, and determine the rotation matrices of the front and rear cameras according to the width consistency of the left and right lanes; The translation matrix of the vehicle-mounted surround view camera is determined according to the overlap features of the images captured by the front and rear cameras and the left and right cameras.
2. The method according to claim 1, characterized in that: Determining the translation matrix of the vehicle-mounted surround view camera according to the overlap features of the images captured by the front and rear cameras and the left and right cameras, including: Determining a translation matrix of the vehicle-mounted surround view camera according to image consistency of overlapping areas between adjacent images collected by the vehicle-mounted surround view camera; After determining the translation matrix of the vehicle-mounted surround view camera according to the overlap features of the images captured by the front and rear cameras and the left and right cameras, the method further includes: The external parameters are optimized with the goal of minimizing the error of the overlapping area between adjacent images collected by the on-board surround view camera.
3. The method according to claim 2, characterized in that Determining a translation matrix of the vehicle-mounted surround view camera according to image consistency of overlapping areas between adjacent images collected by the vehicle-mounted surround view camera includes: Determine the relative positions of the cameras corresponding to the adjacent images according to the image consistency of the image overlap areas of the adjacent images; Determining a camera center according to the relative pose; A translation matrix of the camera is determined according to the relative position and the camera center.
4. The method according to claim 3, characterized in that According to the image consistency of the image overlap area of adjacent images, the relative position and posture of the cameras corresponding to the adjacent images are determined, including: Calibrate the relative height of the camera according to the consistency of lane widths of the adjacent images; Calibrate the relative lateral pose of the camera according to the lane overlap characteristics of the adjacent images; The relative longitudinal position of the camera is calibrated according to the overlapping characteristics of the stop lines of the adjacent images.
5. The method according to claim 1, characterized in that Collect images through the on-board surround view camera, including: Control the vehicle to travel along the lane line, collect images including the left and right lanes through the left and right cameras, and collect images including the current driving lane and the left and right lanes through the front and rear cameras; When the vehicle moves to a point where the images captured by the front camera, the left camera, and the right camera all include the stop line, the front camera and the left and right cameras capture images that all include the stop line at the same time; When the vehicle moves to a point where the images captured by the rear camera, the left camera, and the right camera all include the stop line, the rear camera and the left and right cameras capture images that all include the stop line at the same time.
6. The method according to claim 1, characterized in that The rotation matrix includes a yaw angle, a roll angle, and a pitch angle. According to the stop line equation and the lane line equation, the rotation matrix of the left and right cameras is determined, including: Determining the roll angles of the left and right cameras according to the consistency of the widths of the lanes in the image; Optimizing the slopes of lane line equations in the left and right camera images to determine the yaw angles of the left and right cameras; The pitch angles of the left and right cameras are determined by optimizing the stop lines of different frames to be parallel and the slopes of the stop line equations to be the same.
7. The method according to claim 1, characterized in that The rotation matrix includes yaw angle, roll angle and pitch angle. The rotation matrix of the front and rear cameras is determined according to the width consistency of the left and right lanes, including: Determine the real width ratio of the left and right lanes of the vehicle according to the rotation matrix of the left and right cameras and the predetermined height of the left and right cameras; Determining the roll angle of the front and rear cameras according to the consistency between the width ratio of the left lane in the front and rear images and the real width ratio; The pitch angle and yaw angle of the front and rear cameras are determined according to the vanishing points in the images collected by the front and rear cameras.
8. An external parameter calibration device for a vehicle-mounted surround view camera, characterized in that: The external parameters include a rotation matrix and a translation matrix, and the device includes: An image acquisition unit, used to acquire images through a vehicle-mounted surround view camera, and determine a stop line equation included in the image, and lane line equations corresponding to left and right lanes; A rotation matrix determination unit, used to determine the rotation matrices of the left and right cameras according to the stop line equation and the lane line equation, and to determine the rotation matrices of the front and rear cameras according to the width consistency of the left and right lanes; The translation matrix determination unit is used to determine the translation matrix of the vehicle-mounted surround view camera according to the overlap features of the images collected by the front and rear cameras and the left and right cameras.
9. An external parameter calibration device for a vehicle-mounted surround view camera, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
Citation Information
Cited By
Camera external parameter calibration method, device and equipment and storage medium
CN120747245A
Camera extrinsic parameter calibration method, device, equipment and storage medium
CN120747245B
