A method and device for online optimization of external parameters of surround view camera

Abstract

The present invention relates to an online optimization method and device for extrinsic parameters of a surround-view camera, and belongs to the technical field of camera extrinsic parameter optimization, wherein the method comprises obtaining a vehicle body posture and a surround-view image of a surround-view camera, and obtaining an extrinsic camera posture and an extrinsic camera position according to the surround-view image; when the vehicle body posture is a straight line driving on a straight road, the surround-view image is detected to obtain a lane line point set; the lane line point set is coordinate projected to obtain a vehicle body coordinate system data; the vehicle body coordinate system data is analyzed to obtain an adjustment error; when the adjustment error does not meet a preset error condition, the extrinsic camera posture and the extrinsic camera position of the surround-view camera are adjusted to obtain extrinsic parameter online optimization data. The present invention obtains the extrinsic camera posture and the extrinsic camera position through the surround-view image, and then obtains the adjustment error, so that the extrinsic camera posture and the extrinsic camera position can be adjusted according to the adjustment error, thereby realizing the online optimization of the extrinsic parameters of the surround-view camera.

Description

Technical Field

[0001] The present invention relates to the technical field of camera extrinsic parameter optimization, and in particular to an online optimization method and device for the extrinsic parameters of a surround-view camera. Background Art

[0002] With the widespread application of cameras in intelligent driving technology, especially the deployment and application of detection technology based on image measurement, the accuracy and stability of camera external parameters play an increasingly important role in the reliable operation of related intelligent technologies during their use.

[0003] The external parameter calibration of the camera mainly includes offline calibration and online calibration. Offline calibration refers to the external parameter calibration performed by fixing the calibration board when the module leaves the factory. This calibration method is mature and relatively stable, and is generally used for calibration when the camera equipment is installed for the first time. Online calibration is performed during the use of the vehicle to solve the problem of inaccurate external parameters caused by changes in camera posture and height due to factors such as vibration and tire pressure during use. Existing external parameter online calibration methods are based on vanishing points and lane widths. These features can only perform limited external parameter optimization, so there are limitations.

[0004] Therefore, it is urgent to propose an online optimization method and device for the external parameters of a surround view camera to solve the technical problem that in the prior art, only limited external parameter optimization can be performed based on vanishing points and lane width, resulting in limitations in the optimization of external parameter parameters. Summary of the invention

[0005] In view of this, it is necessary to provide a surround view camera external parameter online optimization method and device to solve the technical problem that only limited external parameter optimization can be performed based on vanishing points and lane widths in the prior art, resulting in limitations in the optimization of external parameter parameters.

[0006] In order to solve the above problems, the present invention provides an online optimization method for the external parameters of a surround view camera, comprising:

[0007] Acquire the vehicle body posture and the surround view image of the surround view camera, and obtain the extrinsic reference camera posture and extrinsic reference camera position according to the surround view image;

[0008] When the vehicle body is in a straight line on a flat road, the surround view image is detected to obtain a lane line point set;

[0009] Performing coordinate projection on the lane line point set to obtain vehicle body coordinate system data;

[0010] Analyzing the vehicle body coordinate system data to obtain an adjustment error;

[0011] When the adjustment error does not meet the preset error condition, the extrinsic camera posture and the extrinsic camera position of the surround view camera are adjusted to obtain extrinsic parameter online optimization data.

[0012] In a possible implementation, the vehicle body posture includes a roll angle, a pitch angle, and a yaw angle;

[0013] After obtaining the vehicle body posture, the method further includes:

[0014] Acquire acceleration information of the vehicle and set a cumulative value; the acceleration information includes linear acceleration and angular acceleration;

[0015] When the roll angle, the pitch angle, the yaw angle, the linear acceleration and the angular acceleration meet a first preset condition, updating the accumulated value;

[0016] When the cumulative value is greater than a preset cumulative value, it is determined that the vehicle body posture of the vehicle is traveling in a straight line on a flat road.

[0017] In a possible implementation, the method further includes:

[0018] When the roll angle, the pitch angle, the yaw angle, the linear acceleration, and the angular acceleration do not meet a first preset condition, the accumulated value is cleared.

[0019] In a possible implementation, detecting the surround view image to obtain a lane line point set includes:

[0020] The surround view image is detected to obtain a plurality of contour points of the lane line, and the plurality of contour points are used as the lane line point set.

[0021] In a possible implementation, performing coordinate projection on the lane line point set to obtain vehicle body coordinate system data includes:

[0022] Based on the initial external parameters and initial internal parameters of the surround view camera, a coordinate correspondence relationship between the vehicle body coordinate system and the camera image coordinate system is established;

[0023] Coordinate projection is performed on the lane line point set according to the coordinate correspondence to obtain vehicle body coordinate system data in the vehicle body coordinate system; the vehicle body coordinate system data includes the coordinates of each point in the lane line point set in the vehicle body coordinate system.

[0024] In a possible implementation, analyzing the vehicle body coordinate system data to obtain an adjustment error includes:

[0025] According to the vehicle body coordinate system data, a first lane line width, a second lane line width, a lane width, a first slope, and a second slope are obtained;

[0026] The first lane line width, the second lane line width, the lane width, the first slope, and the second slope are analyzed to obtain an adjustment error.

[0027] In a possible implementation, when the adjustment error does not satisfy a preset error condition, adjusting the extrinsic reference camera posture and the extrinsic reference camera position of the surround view camera includes:

[0028] Set a fixed optimization angle;

[0029] When the maximum value of the absolute values ​​of the first slope and the second slope is greater than a preset optimized pitch angle and the first slope and the second slope have opposite signs, the adjustment error is determined to be a pitch angle abnormality, the positive or negative value of the fixed optimized angle is determined according to the magnitude of the first slope and the second slope, and the pitch angle is superimposed and adjusted according to the fixed optimized angle;

[0030] When the maximum value of the absolute values ​​of the first slope and the second slope is greater than a preset optimized yaw angle and the first slope and the second slope have the same sign, the adjustment error is determined to be a yaw angle abnormality, the positive or negative value of the fixed optimized angle is determined according to whether the sum of the first slope and the second slope is zero, and the yaw angle is superimposed and adjusted according to the fixed optimized angle;

[0031] When the absolute value of the first difference between the first lane line width and the second lane line width is greater than the preset optimized roll angle, the adjustment error is determined to be a roll angle abnormality, the positive or negative value of the fixed optimized angle is determined based on whether the first difference is greater than zero, and the roll angle is superimposed and adjusted based on the fixed optimized angle.

[0032] In a possible implementation, when the adjustment error does not satisfy a preset error condition, adjusting the extrinsic reference camera posture and the extrinsic reference camera position of the surround view camera further includes:

[0033] Get the actual lane width on the road and set a fixed optimized length;

[0034] When an absolute value of a second difference between the actual lane width and the lane width is greater than a preset optimization length, determining that the adjustment error is a camera position abnormality, and determining whether the second difference is greater than zero;

[0035] If yes, it is determined that the fixed optimization length is a negative value, and the position of the extrinsic reference camera is superimposed and adjusted according to the negative fixed optimization length;

[0036] If not, the position of the extrinsic camera is adjusted in a superimposed manner according to the fixed optimized length.

[0037] In a possible implementation, the adjusting the extrinsic reference camera posture and the extrinsic reference camera position of the surround view camera to obtain extrinsic reference online optimization data includes:

[0038] After the pitch angle abnormality and / or the yaw angle abnormality and / or the roll angle abnormality and / or the camera position abnormality are adjusted, a new pitch angle, a new yaw angle, a new roll angle and a new camera position are obtained;

[0039] The extrinsic camera posture is updated according to the new pitch angle, the new yaw angle and the new roll angle to obtain a new rotation matrix;

[0040] According to the new rotation matrix and the new camera position, a new translation vector is obtained, and a new external parameter is obtained;

[0041] The surround view image is processed according to the new external parameter to obtain a new adjustment error, and when the adjustment error meets the preset error condition, the new external parameter is determined as external parameter online optimization data.

[0042] On the other hand, the present invention also provides an online optimization device for external parameters of a surround view camera, comprising:

[0043] An image acquisition module is used to acquire the vehicle body posture and the surround view image of the surround view camera, and obtain the external reference camera posture and external reference camera position according to the surround view image;

[0044] An image determination module, used to detect the surround image to obtain a lane line point set when the vehicle body is traveling in a straight line on a flat road;

[0045] A coordinate projection module, used to perform coordinate projection on the lane line point set to obtain vehicle body coordinate system data;

[0046] An error determination module, used for analyzing the vehicle body coordinate system data to obtain an adjustment error;

[0047] The external parameter adjustment module is used to adjust the external parameter camera posture and the external parameter camera position of the surround view camera when the adjustment error does not meet the preset error condition, so as to obtain external parameter online optimization data.

[0048] The beneficial effect of the present invention is that the present invention can judge the body posture of the vehicle, and then optimize the external parameter parameters of the surround view camera when it is determined that the vehicle is traveling in a straight line on a straight road, thereby improving the accuracy of the external parameter optimization. Furthermore, the surround view image can be processed to obtain the external parameter camera posture and the external parameter camera position, and then the adjustment error can be obtained according to the detection result of the surround view image, so that when the adjustment error does not meet the preset error condition, the external parameter camera posture and the external parameter camera position are optimized, thereby realizing the optimization of the external parameter camera posture and the external parameter camera position, increasing the optimization parameters, and realizing the online optimization of the surround view camera external parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic flow chart of an embodiment of the method for online optimization of external parameters of a surround view camera provided by the present invention;

[0050] Figure 2 A schematic diagram of the structure of an embodiment of the vehicle body coordinates and Euler angles provided by the present invention;

[0051] Figure 3 A schematic structural diagram of an embodiment of a vehicle equipped with a surround view camera provided by the present invention while driving on a straight road;

[0052] Figure 4 A schematic diagram of the structure of an embodiment of the calibration coordinate system of each surround-view camera and the self-vehicle coordinate system provided by the present invention;

[0053] Figure 5 A schematic diagram of the structure of an embodiment of the lane line contour provided by the present invention;

[0054] Figure 6 A schematic diagram of the structure of an embodiment of two lane lines identified in a forward camera image provided by the present invention;

[0055] Figure 7 A schematic diagram of the structure of an embodiment of the Pitch anomaly provided by the present invention;

[0056] Figure 8 A schematic diagram of the structure of an embodiment of the Roll exception provided by the present invention;

[0057] Fig. 9 A schematic diagram of the structure of an embodiment of the Yaw anomaly provided by the present invention;

[0058] Fig.10 A schematic diagram of the structure of an embodiment of the z anomaly provided by the present invention;

[0059] Fig.11 A schematic diagram of the structure of an embodiment of the device for online optimization of external parameters of a surround view camera provided by the present invention;

[0060] Fig.12 A schematic structural diagram of an embodiment of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0061] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0062] like Figure 1 As shown, a specific embodiment of the present invention discloses an online optimization method for extrinsic parameters of a surround view camera, comprising:

[0063] S101, obtaining a vehicle body posture and a surround view image of a surround view camera, and obtaining an extrinsic reference camera posture and an extrinsic reference camera position according to the surround view image;

[0064] S102, when the vehicle body posture is straight driving on a straight road, detecting the surround view image to obtain a lane line point set;

[0065] S103, performing coordinate projection on the lane line point set to obtain vehicle body coordinate system data;

[0066] S104, analyzing the vehicle body coordinate system data to obtain an adjustment error;

[0067] S105. When the adjustment error does not meet the preset error condition, the extrinsic camera posture and the extrinsic camera position of the surround view camera are adjusted to obtain extrinsic online optimization data.

[0068] In a specific embodiment of the present invention, the vehicle may be equipped with a posture sensor, and the transient posture change of the vehicle in the vehicle body coordinate system may be obtained according to the posture sensor. Figure 2 As shown, the body coordinates are defined as X-axis forward, Y-axis leftward, and Z-axis upward. The origin is located at the center of the rear axle. The rotation angles of the three axes are Roll, Pitch, and Yaw respectively. The attitude sensor can provide the body attitude Roll, Pitch, and Yaw. Roll Angle: The roll angle describes the angle of rotation of the vehicle around its longitudinal axis X. When driving in a straight line on a straight road, the roll angle of the vehicle is usually zero, indicating that the body is parallel to the ground. Pitch Angle: The pitch angle describes the angle of rotation of the vehicle around its horizontal axis Y. When driving in a straight line on a straight road, the pitch angle of the vehicle is usually zero, indicating that the body is perpendicular to the ground. Yaw Angle: The yaw angle describes the angle of rotation of the vehicle around its vertical axis Z. When driving in a straight line on a straight road, the yaw angle of the vehicle may change slightly, depending on the initial orientation of the vehicle and the fine-tuning of the direction.

[0069] The embodiments of the present invention can be applied to adjust external parameters in scenarios with regular ground calibration objects, and are therefore not limited to public road scenarios, the number of cameras, and a certain type of regular calibration objects. The embodiments of the present invention can use 4-way surround view in public road scenarios, with lane lines as calibration calibration objects for example, Figure 3 As shown, the lane line parameters used as calibration objects are known to have the actual lane width W pathReal ( Figure 3 The lane width in the image) and the actual lane line width W LineReal ( Figure 3 The lane line width in the image is calculated), and surround cameras are respectively arranged at the front, back, left and right of the vehicle Egocar. The initial parameters of the surround cameras include internal parameters (K, D) and external parameters (R, T). For the internal parameters, the embodiment of the present invention assumes that the value is correct and remains unchanged, while the external parameters have initial values, but posture and Z-axis errors are generated during the use process. In the external parameters, R represents the rotation matrix. Since the camera orientation is known, the rotation matrix can be transformed to be equivalent to the camera posture, which is expressed by Euler angles, that is, [Roll Pitch Yaw], and the translation vector T can be converted to the camera position [xyz]. Therefore, the external parameters are optimized and adjusted, which is converted into a process of obtaining a more accurate camera posture and position. The external parameter optimization can be completed by importing and correcting more accurate camera posture and position data.

[0070] Compared with the prior art, the embodiment of the present invention can judge the posture of the vehicle body, and then optimize the external parameter parameters of the surround view camera when it is determined that the vehicle is driving in a straight line on a straight road, thereby improving the accuracy of the external parameter optimization. Furthermore, the surround view image can be processed to obtain the external parameter camera posture and the external parameter camera position, and then the adjustment error can be obtained according to the detection result of the surround view image, so that when the adjustment error does not meet the preset error condition, the external parameter camera posture and the external parameter camera position are optimized, thereby realizing the optimization of the external parameter camera posture and the external parameter camera position, increasing the optimization parameters, and realizing the online optimization of the surround view camera external parameters.

[0071] In some embodiments of the present invention, the vehicle body posture includes a roll angle, a pitch angle, and a yaw angle; after obtaining the vehicle body posture, the method further includes:

[0072] Get the vehicle's acceleration information and set the cumulative value; the acceleration information includes linear acceleration and angular acceleration;

[0073] When the roll angle, pitch angle, yaw angle, linear acceleration and angular acceleration meet a first preset condition, updating the accumulated value;

[0074] When the cumulative value is greater than the preset cumulative value, it is determined that the vehicle body posture is traveling in a straight line on a flat road.

[0075] In a specific embodiment of the present invention, a cumulative value can be set, and the attitude sensor can also provide acceleration information of the vehicle, and the acceleration information can include linear acceleration and angular acceleration. When driving in a straight line, the linear acceleration of the vehicle should be close to zero because the speed of the vehicle remains constant. The angular acceleration should also be close to zero, indicating that the vehicle is not turning or rotating. The linear acceleration close to zero and the angular acceleration close to zero can be set to meet the first preset condition. Thus, it can be determined whether the vehicle is driving in a straight line on a straight road according to the first preset condition. When the linear acceleration and angular acceleration meet the first preset condition and it is determined that the vehicle is in a stable state, the cumulative value can be updated, for example, the cumulative value +1. When the cumulative value is greater than the preset cumulative value, it can be determined that the vehicle is driving in a straight line on a straight road.

[0076] In some embodiments of the present invention, the method further comprises:

[0077] When the roll angle, the pitch angle, the yaw angle, the linear acceleration and the angular acceleration do not meet the first preset condition, the accumulated values ​​are cleared.

[0078] In a specific embodiment of the present invention, when the linear acceleration and the angular acceleration do not meet the first preset condition, the accumulated value may be cleared, and the accumulated value is equal to 0, and the above posture detection process is performed again.

[0079] Furthermore, when it is determined that the vehicle is traveling in a straight line on a straight road, the surround view image obtained by the surround view camera can be used to obtain the original coordinate system of the surround view image and other information to obtain the extrinsic camera posture and extrinsic camera position. For example, the forward camera can be taken as an example. According to formula (1), the rotation matrix R of the forward camera extrinsic parameter can be obtained: front With the translation vector T front , formula (1) is as follows:

[0080]

[0081] In the formula, r 11 : The x component of the X axis in the original coordinate system after rotation; r 12 : The y component of the X axis in the original coordinate system after rotation. 13 : The z component of the X axis in the original coordinate system after rotation; r 21 : The x component of the Y axis in the original coordinate system after rotation; r 22 : The y component of the Y axis in the original coordinate system after rotation; r 23 : The z component of the Y axis in the original coordinate system after rotation; r 31 : The x component of the Z axis in the original coordinate system after rotation; r 32 : The y component of the Z axis in the original coordinate system after rotation; r33 : The z component of the Z axis in the original coordinate system after rotation; [t x t y t z ] represent the translation of the three axes XYZ respectively.

[0082] Then the forward camera can be converted into the external parameter formula (1) as the camera pose [Roll front Potch front Yaw front ] and the camera position [x front y front z front ],like Figure 4 As shown, Figure 4 O front_cam is the front camera position coordinate system, O back_cam is the rear camera position coordinate system, O left_cam is the left camera position coordinate system, O right_cam is the right camera position coordinate system, the rotation matrix R in the forward external parameter front_car2cam (i.e. R front In order to distinguish the rotation matrices at different stages of the transformation process, the example R front exist Figure 4 R car2cam Represents) reflects the vehicle coordinate system O car The point below is transformed from the vehicle coordinate system to the camera coordinate system O cam The rotation transformation relationship under this process can be decomposed into the vehicle body coordinate system O car >>Camera position coordinate system O front_cali (calibration coordinate system) >> Camera attitude coordinate system O front_cam , this decomposition process involves two rotation matrices, namely R front_car2cali (Right now Figure 4 R car2cali ), R front_cali2cam ,like Figure 4 From O car Point to O front_cali With from O front_cali Point to O cam The arrows represent the transformation, as shown in formula (2):

[0083] R front_car2cam =R front_car2cali *R front_cali2cam (2)

[0084] Since the installation positions of the four cameras are fixed, the rotation matrix of their calibration coordinate systems is also fixed, such as Figure 4 As shown, the front, back, left, and right coordinates relative to the vehicle coordinate system are as follows:

[0085] Forward camera calibration coordinate system O front_cali Posture and vehicle coordinate system O car The posture is consistent. Through the coordinate relationship, that is, the relationship between the vehicle coordinate system and the forward camera coordinate system, the vehicle coordinate system XZY is consistent with the XYZ direction of the calibrated coordinate system, so the forward R front_car2cali is the unit matrix, that is, no transformation is required; R front_car2cali As shown in formula (3):

[0086]

[0087] Rear camera calibration coordinate system O back_cali The posture is calculated through the vehicle coordinate system O car The Z axis is rotated 180 degrees, and the backward R can be obtained according to the posture rule transformation. back_car2cali :O back_cali The X-axis is composed of O car The negative X-coordinate axis is represented by the transformation matrix, so the first column is the X-axis transformation vector *-1, that is, [1 0 0]*-1; back_cali The Y coordinate axis is O car The negative Y coordinate axis is represented by the transformation matrix, so the first column is the Y axis transformation vector *-1, that is, [0 1 0]*-1; back_cali The Y coordinate axis is O car The Z coordinate axis is represented by

[001] . back_car2cali As shown in formula (4):

[0088]

[0089] Left camera calibration coordinate system O left_cali The posture is calculated through the vehicle coordinate system O car The Z axis is rotated 90 degrees forward, and the R on the left can be obtained according to the posture rule transformation expression left_car2cali ; R left_car2cali As shown in formula (5):

[0090]

[0091] The right camera calibrates the coordinate system O right_cali The posture is calculated through the vehicle coordinate system O car The Z axis is rotated 270 degrees forward, and the R on the right side can be obtained according to the posture rule transformation expression right_car2cali ; R right_car2cali As shown in formula (6):

[0092]

[0093] From the above, we know that R front_car2cam , R front_car2cali , multiply both sides of formula (2) by Rfront_car2cali The inverse As shown in formula (7):

[0094]

[0095] Simplify formula (7), and then we can get the external reference camera posture R according to the surround camera: front_cali2cam ; As shown in formula (8):

[0096]

[0097] This pair R cali2cam Convert to Euler angles to get the external reference camera posture [Roll front Potch front Yaw front ], and can also be based on the translation vector T front Calculate the external reference camera position XYZ front_cam , as shown in formula (9):

[0098]

[0099] In some embodiments of the present invention, step S102 includes:

[0100] The surround view image is detected to obtain a plurality of contour points of the lane line, and the plurality of contour points are used as the lane line point set.

[0101] In a specific embodiment of the present invention, a lane line recognition algorithm based on a classic image processing algorithm and a lane line recognition algorithm based on deep learning can be used. The specific algorithm and process can be set according to the actual situation, and the embodiment of the present invention is not limited here. The numerical representation of the contour line of the lane line can be obtained by the lane line recognition algorithm. A contour line is represented as a series of point sets, and then the points on each contour line can be obtained, that is, multiple contour points of the lane line can be obtained. Multiple contour points can obtain the edge contour of the lane line, such as Figure 5 As shown, Figure 5 ①, ②, ③ and ④ in the figure represent a contour line respectively, so the lane line point set on the edge contour can be obtained based on the point set on ①, ②, ③ and ④ of the edge contour.

[0102] In some embodiments of the present invention, step S103 includes:

[0103] Based on the initial external parameters and initial internal parameters of the surround view camera, the coordinate correspondence between the vehicle body coordinate system and the camera image coordinate system is established;

[0104] The lane line point set is coordinate projected according to the coordinate correspondence to obtain vehicle body coordinate system data in the vehicle body coordinate system; the vehicle body coordinate system data includes the coordinates of each point in the lane line point set in the vehicle body coordinate system.

[0105] In a specific embodiment of the present invention, a coordinate correspondence between a certain point of an object in the vehicle body coordinate system and the point of the object in the camera image coordinate system can be established through the intrinsic parameters and extrinsic parameters of the surround view camera. The coordinate correspondence conversion process can be: the point on the distorted image is converted into a point on the undistorted image through the intrinsic parameter D, and then converted into a point on the camera imaging plane through the intrinsic parameter K, and then converted into a point in the vehicle body coordinate system through the intrinsic parameter RT. The specific conversion process can be set according to actual conditions, and the embodiment of the present invention is not limited here. Through projection, the coordinates of a certain point on the surround view image in the vehicle body coordinate system can be obtained based on the internal and external parameters, so that the coordinates of each point in the lane line point set corresponding to the vehicle body coordinate system can be obtained.

[0106] In some embodiments of the present invention, step S104 includes:

[0107] According to the vehicle body coordinate system data, a first lane line width, a second lane line width, a lane width, a first slope, and a second slope are obtained;

[0108] The first lane line width, the second lane line width, the lane width, the first slope, and the second slope are analyzed to obtain an adjustment error.

[0109] In a specific embodiment of the present invention, the identified lane line contour points are projected to obtain the coordinates of these points in the vehicle body coordinate system, and based on these coordinates, the characteristics of the projected lane line can be determined. The ideal conditions are as follows: Figure 6 As shown in the example, two lane lines are identified in the front camera image. Line 1 is the line on the left, and Line 2 is the line on the right. The widths of the two lines can be expressed as the first lane line width W Lane1 、The width of the second lane line W Lane2 , lane width is W path The slope of the two lane lines with x (vertical axis) as the variable is the first slope S Lane1 , the second slope S Lane2 Thus, the first lane line width, the second lane line width, the lane width, the first slope and the second slope can be analyzed to obtain the adjustment error. For example, the object of the adjustment error can be Pitch, which represents the rotation amount in the Y coordinate system of the camera. The error between Pitch and the true value will cause the lane line to appear in an eight-shaped shape, such as Figure 7 As shown, the lane line features in the left and right figures are expressed as S Lane1 , S Lane2The error adjustment object can be Roll, which indicates the rotation of the camera on the X axis. This value will affect the tilt of the top view to the left or right. Figure 8 As shown in the left and right figures, the lane line characteristics are represented by the width W of the left and right lane lines. Lane1 , W Lane2 The object for adjusting the error can be Yaw, which represents the rotation of the camera on the Z axis, such as Fig. 9 As shown, it will affect the slope of the edge contour of the calibration object that is originally parallel to the X-axis in the top view. The marking lines in the left and right figures are both S Lane1 , S Lane2 The error adjustment object can be z in the external reference camera position, which is the Z-axis coordinate of the camera in the vehicle coordinate system. The tire pressure of the vehicle may change this value, which affects the size of the calibration object after projection, such as Fig.10 As shown, for example, in the left figure, the lane line size W Lane1 , W Lane2 and lane width W paca The difference between x and y is the position of the camera on the XY plane of the vehicle coordinate system. Since the camera is fixedly installed, it can be regarded as a rigid body connection and is not prone to errors. It is omitted here. The error has a small impact, that is, the error caused will not be affected by the distance between the measurement point and the camera. Therefore, it can be based on W Lane1 , W Lane2 , W path , S Lane1 , S Lane2 Optimize and adjust external parameters.

[0110] In some embodiments of the present invention, step S105 includes:

[0111] Set a fixed optimization angle;

[0112] When the maximum value of the absolute values ​​of the first slope and the second slope is greater than the preset optimized pitch angle, and the first slope and the second slope are of opposite signs, the adjustment error is determined to be an abnormal pitch angle, and the positive or negative fixed optimized angle is determined according to the magnitude of the first slope and the second slope, and the pitch angle is superimposed and adjusted according to the fixed optimized angle;

[0113] When the maximum value of the absolute values ​​of the first slope and the second slope is greater than the preset optimized yaw angle and the first slope and the second slope have the same sign, the adjustment error is determined to be a yaw angle abnormality, the positive or negative value of the fixed optimized angle is determined according to whether the sum of the first slope and the second slope is zero, and the yaw angle is superimposed and adjusted according to the fixed optimized angle;

[0114] When the absolute value of the first difference between the width of the first lane line and the width of the second lane line is greater than the preset optimized roll angle, the adjustment error is determined to be a roll angle abnormality, the positive or negative value of the fixed optimized angle is determined based on whether the first difference is greater than zero, and the roll angle is superimposed and adjusted based on the fixed optimized angle.

[0115] In a specific embodiment of the present invention, it is assumed that the actual lane width in the real world is W LineReal , the actual lane width is W pathReal The optimization method uses fixed step length optimization, and the fixed optimization angle and fixed optimization length are d angle , d dist , which is related to the accuracy that the step size needs to achieve. For the optimization objects Pitch, Roll, Yaw, and z, the optimization target thresh needs to be set Pitch , thresh Roll , thresh Yaw , thresh z , so that the optimization stops within the appropriate error range. The optimized parameters are named Pitch′, Roll′, Yaw′, and z′ respectively. The optimization step size and optimization target need to be coordinated according to the actual situation. In order to obtain the best effect, a fixed optimization angle can be set.

[0116] First, we can calculate the slope S of the two lane lines. Lane1 , S Lane2 Make a judgment and determine S Lane1 , S Lane2 Is the maximum absolute value of greater than the preset optimized pitch angle thresh Pitch If it is greater than, it means that the error between the current Pitch and the correct value does not meet the requirement, otherwise it meets the requirement. When it is greater than, there are two cases, one is that the slopes have different signs, and the other is that the slopes have the same sign (positive or negative). The different signs are caused by the Pitch error, and it can be determined that the adjustment error is an abnormal pitch angle. Compare S in the case of different signs Lane1 , S Lane2 The size of S Lane1 <S Lane2 When the correction value is -d angle , otherwise d angle , and further superimpose the correction value on the existing Pitch, that is, Pitch′=Pitch+fix.

[0117] First, the slopes S of the two lane lines Lane1 , S Lane2 Check to determine whether the maximum absolute value is greater than the preset optimized yaw angle thresh YawIf it is greater than, it means that the error between the current Yaw and the correct value does not meet the requirement, and it can be determined that the adjustment error is an abnormal yaw angle. Otherwise, it meets the requirement. In the case of being greater than, if the two values ​​have the same sign, it means that the error comes from Yaw. At this time, further judgment S Lane1 , S Lane2 Is the sum less than zero? If so, the correction value fix is ​​d angle , otherwise -d angle , and further superimpose the correction value on the existing Yaw, that is, Yaw′=Yaw+fix.

[0118] First, the width S of the two lane lines Lane1 , S Lane2 Make a judgment to determine whether the absolute value of the difference is greater than the preset optimized roll angle thresh roll If it is greater than, it means that the error between the current Roll and the correct value does not meet the requirement, and it can be determined that the adjustment error is an abnormal roll angle. Otherwise, it meets the requirement. If it is greater than, if the difference between the two values ​​is greater than zero, then the correction value fix is ​​-d angle , otherwise d angle , and further add a correction value to the existing Roll, that is, Roll′=Roll+fix.

[0119] In some embodiments of the present invention, step S105 further includes:

[0120] Get the actual lane width on the road and set a fixed optimized length;

[0121] When the absolute value of the second difference between the actual lane width and the lane width is greater than the preset optimization length, determining that the adjustment error is a camera position abnormality, and determining whether the second difference is greater than zero;

[0122] If so, the fixed optimization length is determined to be a negative value, and the position of the external reference camera is superimposed and adjusted according to the negative fixed optimization length;

[0123] If not, the external reference camera position is adjusted according to the fixed optimization length.

[0124] In a specific embodiment of the present invention, the lane width W is first determined and calculated. Line and the actual lane width W LineReal Is the absolute value of the difference greater than the preset optimization length thresh z If it is greater than, it means that the error between the current z and the correct value does not meet the requirement, and it can be determined that the adjustment error is an abnormal camera position. Otherwise, it meets the requirement. If it is greater than, if the calculated lane width W Line and the actual lane width W LineReal If the difference is greater than zero, the correction value fix is ​​-d dist , otherwise ddist , and further superimpose the correction value on the existing z, that is, z′=z+fix.

[0125] Further, when the adjustment error is in at least one of the abnormal pitch angle and / or abnormal yaw angle and / or abnormal roll angle and / or abnormal camera position, it is determined that the adjustment error does not meet the preset error condition and needs to be adjusted, and one or more adjustments can be made.

[0126] In some embodiments of the present invention, step S105 includes:

[0127] After the pitch angle abnormality and / or the yaw angle abnormality and / or the roll angle abnormality and / or the camera position abnormality are adjusted, a new pitch angle, a new yaw angle, a new roll angle and a new camera position are obtained;

[0128] Update the external reference camera posture according to the new pitch angle, new yaw angle and new roll angle to obtain a new rotation matrix;

[0129] According to the new rotation matrix and the new camera position, the new translation vector is obtained, and the new external parameters are obtained;

[0130] The surround view image is processed according to the new external parameter to obtain a new adjustment error. When the adjustment error meets the preset error condition, the new external parameter is determined as the external parameter online optimization data.

[0131] In a specific embodiment of the present invention, after the pitch angle abnormality and / or the yaw angle abnormality and / or the roll angle abnormality and / or the camera position abnormality are adjusted, a new pitch angle Pitch′, a new yaw angle Yaw′, a new roll angle Roll′ and a new camera position z′ can be obtained, and R can be updated to R′ through the Euler angle conversion rotation matrix. Based on the new R′ and the new camera position [x, y, z′], T can be updated to T′, thereby completing a round of external parameter updates, and the lane line point set identified by the surround image is mapped again and the lane line feature value is calculated. The adjustment error of the current external parameter can be determined again. When the adjustment error meets the preset error condition, the optimization can be completed, otherwise iterative update continues. When the optimization is completed, the optimized new external parameter, that is, the external parameter online optimization data, can be obtained.

[0132] In order to better implement the surround view camera external parameter online optimization method in the embodiment of the present invention, based on the surround view camera external parameter online optimization method, the embodiment of the present invention also provides a surround view camera external parameter online optimization device, such as Fig.11 As shown, the surround camera external parameter online optimization device 1100 includes:

[0133] The image acquisition module 1101 is used to acquire the vehicle body posture and the surround view image of the surround view camera, and obtain the external reference camera posture and external reference camera position according to the surround view image;

[0134] The image determination module 1102 is used to detect the surround image and obtain a lane line point set when the vehicle body posture is traveling in a straight line on a flat road;

[0135] A coordinate projection module 1103 is used to perform coordinate projection on the lane line point set to obtain vehicle body coordinate system data;

[0136] The error determination module 1104 is used to analyze the vehicle body coordinate system data to obtain an adjustment error;

[0137] The external parameter adjustment module 1105 is used to adjust the external parameter camera posture and the external parameter camera position of the surround camera when the adjustment error does not meet the preset error condition, so as to obtain the external parameter online optimization data.

[0138] The surround-view camera extrinsic parameter online optimization device 1100 provided in the above embodiment can implement the technical solution described in the above embodiment of the surround-view camera extrinsic parameter online optimization method. The specific implementation principles of the above modules or units can refer to the corresponding contents in the above embodiment of the surround-view camera extrinsic parameter online optimization method, which will not be repeated here.

[0139] like Fig.12 As shown, the present invention also provides an electronic device 1200. The electronic device 1200 includes a processor 1201, a memory 1202 and a display 1203. Fig.12 Only some components of the electronic device 1200 are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0140] In some embodiments, the memory 1202 may be an internal storage unit of the electronic device 1200, such as a hard disk or memory of the electronic device 1200. In other embodiments, the memory 1202 may also be an external storage device of the electronic device 1200, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1200.

[0141] Furthermore, the memory 1202 may include both an internal storage unit of the electronic device 1200 and an external storage device. The memory 1202 is used to store application software installed in the electronic device 1200 and various data.

[0142] In some embodiments, the processor 1201 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes or process data stored in the memory 1202, such as the online optimization method of the surround view camera external parameters in the present invention.

[0143] In some embodiments, the display 1203 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 1203 is used to display information of the electronic device 1200 and to display a visual user interface. The components 1201-1203 of the electronic device 1200 communicate with each other via a system bus.

[0144] In some embodiments of the present invention, when the processor 1201 executes the surround camera external parameter online optimization program in the memory 1202, the following steps may be implemented:

[0145] Obtain the vehicle's body posture and the surround view image of the surround view camera, and obtain the extrinsic reference camera posture and extrinsic reference camera position according to the surround view image;

[0146] When the vehicle body is traveling in a straight line on a flat road, the surround view image is detected to obtain a lane line point set;

[0147] Perform coordinate projection on the lane line point set to obtain the vehicle body coordinate system data;

[0148] Analyze the vehicle body coordinate system data to obtain the adjustment error;

[0149] When the adjustment error does not meet the preset error condition, the extrinsic camera posture and extrinsic camera position of the surround camera are adjusted to obtain the extrinsic online optimization data.

[0150] It should be understood that: when the processor 1201 executes the surround camera external parameter online optimization program in the memory 1202, in addition to the above functions, other functions can also be implemented. For details, please refer to the description of the corresponding method embodiment above.

[0151] Furthermore, the embodiment of the present invention does not specifically limit the type of the electronic device 1200 mentioned, and the electronic device 1200 may be a portable electronic device such as a mobile phone, a tablet computer, a personal digital assistant (PDA), a wearable device, a laptop computer, etc. Exemplary embodiments of portable electronic devices include but are not limited to portable electronic devices equipped with IOS, Android, Microsoft or other operating systems. The above-mentioned portable electronic devices may also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, the electronic device 1200 may not be a portable electronic device, but a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0152] Accordingly, an embodiment of the present application also provides a computer-readable storage medium, which is used to store computer-readable programs or instructions. When the program or instructions are executed by a processor, the steps or functions of the online optimization method of the surround view camera external parameters provided in the above-mentioned method embodiments can be implemented.

[0153] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the computer program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0154] The above is a detailed introduction to the online optimization method and device for the external parameters of the surround view camera provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for online optimization of external parameters of a surround camera, characterized in that: include: Acquire the vehicle body posture and the surround view image of the surround view camera, and obtain the extrinsic reference camera posture and extrinsic reference camera position according to the surround view image; When the vehicle body is in a straight line on a flat road, the surround view image is detected to obtain a lane line point set; Performing coordinate projection on the lane line point set to obtain vehicle body coordinate system data; Analyzing the vehicle body coordinate system data to obtain an adjustment error; When the adjustment error does not meet the preset error condition, adjusting the external reference camera posture and the external reference camera position of the surround view camera to obtain external reference online optimization data; The analyzing the vehicle body coordinate system data to obtain the adjustment error includes: According to the vehicle body coordinate system data, a first lane line width, a second lane line width, a lane width, a first slope, and a second slope are obtained; Analyzing the first lane line width, the second lane line width, the lane width, the first slope, and the second slope to obtain an adjustment error; The vehicle body posture includes roll angle, pitch angle and yaw angle; When the adjustment error does not meet the preset error condition, adjusting the external reference camera posture and the external reference camera position of the surround view camera includes: Set a fixed optimization angle; When the maximum value of the absolute values ​​of the first slope and the second slope is greater than a preset optimized pitch angle, and the first slope and the second slope have opposite signs, the adjustment error is determined to be a pitch angle abnormality, the positive or negative value of the fixed optimized angle is determined according to the magnitude of the first slope and the second slope, and the pitch angle is superimposed and adjusted according to the fixed optimized angle; When the maximum value of the absolute values ​​of the first slope and the second slope is greater than a preset optimized yaw angle, and the first slope and the second slope have the same sign, the adjustment error is determined to be a yaw angle abnormality, the positive or negative value of the fixed optimized angle is determined according to whether the sum of the first slope and the second slope is zero, and the yaw angle is superimposed and adjusted according to the fixed optimized angle; When the absolute value of the first difference between the first lane line width and the second lane line width is greater than the preset optimized roll angle, the adjustment error is determined to be a roll angle abnormality, the positive or negative value of the fixed optimized angle is determined based on whether the first difference is greater than zero, and the roll angle is superimposed and adjusted based on the fixed optimized angle.

2. The method for online optimization of surround camera external parameters according to claim 1, characterized in that: After obtaining the vehicle body posture, the method further includes: Acquire acceleration information of the vehicle and set a cumulative value; the acceleration information includes linear acceleration and angular acceleration; When the roll angle, the pitch angle, the yaw angle, the linear acceleration and the angular acceleration meet a first preset condition, updating the accumulated value; When the cumulative value is greater than a preset cumulative value, it is determined that the vehicle body posture of the vehicle is traveling in a straight line on a flat road.

3. The method for online optimization of external parameters of a surround view camera according to claim 2, characterized in that: The method further comprises: When the roll angle, the pitch angle, the yaw angle, the linear acceleration, and the angular acceleration do not meet a first preset condition, the accumulated value is cleared.

4. The method for online optimization of external parameters of a surround view camera according to claim 1, characterized in that: The detecting the surround view image to obtain a lane line point set includes: The surround view image is detected to obtain a plurality of contour points of the lane line, and the plurality of contour points are used as the lane line point set.

5. The method for online optimization of surround camera external parameters according to claim 1, characterized in that: The coordinate projection of the lane line point set to obtain vehicle body coordinate system data includes: Based on the initial external parameters and initial internal parameters of the surround view camera, a coordinate correspondence relationship between the vehicle body coordinate system and the camera image coordinate system is established; Coordinate projection is performed on the lane line point set according to the coordinate correspondence to obtain vehicle body coordinate system data in the vehicle body coordinate system; the vehicle body coordinate system data includes the coordinates of each point in the lane line point set in the vehicle body coordinate system.

6. The method for online optimization of surround view camera external parameters according to claim 1, characterized in that: When the adjustment error does not meet the preset error condition, adjusting the external reference camera posture and the external reference camera position of the surround view camera also includes: Get the actual lane width on the road and set a fixed optimized length; When an absolute value of a second difference between the actual lane width and the lane width is greater than a preset optimization length, determining that the adjustment error is a camera position abnormality, and determining whether the second difference is greater than zero; If yes, it is determined that the fixed optimization length is a negative value, and the position of the extrinsic reference camera is superimposed and adjusted according to the negative fixed optimization length; If not, the position of the extrinsic camera is adjusted in a superimposed manner according to the fixed optimized length.

7. The method for online optimization of surround view camera external parameters according to claim 6, characterized in that: The adjusting the external reference camera posture and the external reference camera position of the surround view camera to obtain external reference online optimization data includes: After the pitch angle abnormality and / or the yaw angle abnormality and / or the roll angle abnormality and / or the camera position abnormality are adjusted, a new pitch angle, a new yaw angle, a new roll angle and a new camera position are obtained; The extrinsic camera posture is updated according to the new pitch angle, the new yaw angle and the new roll angle to obtain a new rotation matrix; According to the new rotation matrix and the new camera position, a new translation vector is obtained, and a new external parameter is obtained; The surround view image is processed according to the new external parameter to obtain a new adjustment error, and when the adjustment error meets the preset error condition, the new external parameter is determined as external parameter online optimization data.

8. An online optimization device for external parameters of a surround camera, characterized in that: include: An image acquisition module is used to acquire the vehicle body posture and the surround view image of the surround view camera, and obtain the external reference camera posture and external reference camera position according to the surround view image; An image determination module, used to detect the surround image to obtain a lane line point set when the vehicle body is traveling in a straight line on a flat road; A coordinate projection module, used to perform coordinate projection on the lane line point set to obtain vehicle body coordinate system data; An error determination module, used for analyzing the vehicle body coordinate system data to obtain an adjustment error; An external parameter adjustment module, used for adjusting the external parameter camera posture and the external parameter camera position of the surround view camera to obtain external parameter online optimization data when the adjustment error does not meet the preset error condition; The error determination module is further used to obtain a first lane line width, a second lane line width, a lane width, a first slope, and a second slope according to the vehicle body coordinate system data; analyze the first lane line width, the second lane line width, the lane width, the first slope, and the second slope to obtain an adjustment error; The vehicle body posture includes roll angle, pitch angle and yaw angle; The external parameter adjustment module is further used to set a fixed optimization angle; when the maximum value of the absolute values ​​of the first slope and the second slope is greater than a preset optimized pitch angle, and the first slope and the second slope are of different signs, the adjustment error is determined to be a pitch angle abnormality, and the positive and negative values ​​of the fixed optimization angle are determined according to the magnitudes of the first slope and the second slope, and the pitch angle is superimposed and adjusted according to the fixed optimization angle; when the maximum value of the absolute values ​​of the first slope and the second slope is greater than a preset optimized yaw angle, and the first slope and the second slope are of the same sign, the adjustment error is determined to be a yaw angle abnormality, and the positive and negative values ​​of the fixed optimization angle are determined according to whether the sum of the first slope and the second slope is zero, and the yaw angle is superimposed and adjusted according to the fixed optimization angle; when the absolute value of the first difference between the first lane line width and the second lane line width is greater than a preset optimized roll angle, the adjustment error is determined to be a roll angle abnormality, and the positive and negative values ​​of the fixed optimization angle are determined according to whether the first difference is greater than zero, and the roll angle is superimposed and adjusted according to the fixed optimization angle.

Images