Camera calibration method, device and computer storage medium

Abstract

Embodiments of the present application provide a camera calibration method, a device with an automatic driving function, and a computer storage medium, wherein the camera calibration method comprises: acquiring three-dimensional point cloud data and two-dimensional image data of an environment in which a device with an automatic driving function is located, which are collected by the device in a straight driving state; determining a first ground normal of the device in an IMU coordinate system according to the three-dimensional point cloud data; determining a second ground normal of the device in a camera coordinate system according to the two-dimensional image data; determining a corresponding roll angle of the device according to the first ground normal and the second ground normal; and performing camera calibration on the device according to at least the roll angle. Through the embodiments of the present application, camera calibration can be effectively performed.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a camera calibration method, a device with autonomous driving function, and a computer storage medium. Background Technology

[0002] Environmental perception is a crucial component of autonomous driving technology. It primarily involves using various types of sensors to collect data on the surrounding environment of autonomous driving devices (such as vehicles, robots, and aircraft), generating quantitative descriptions. Based on this environmental perception data, various backend processes can be performed, including autonomous driving planning, decision-making, localization, and obstacle avoidance.

[0003] However, in the actual process of collecting environmental perception data, on the one hand, after a long period of driving, the camera position of the device with autonomous driving function may become loose or shifted, at which point the calibration parameters of the camera and the actual camera position will no longer match. On the other hand, the device with autonomous driving function may experience vibrations during driving. For example, heavy trucks with autonomous driving function experience high-frequency vibrations during driving, which will also cause the camera position to shift, resulting in a mismatch between the calibration parameters of the camera and the actual camera position. These mismatches will cause inaccuracies in the subsequent autonomous driving data processing results based on environmental perception data, leading to inaccurate autonomous driving behavior and thus posing safety hazards to autonomous driving. Summary of the Invention

[0004] In view of this, embodiments of this application provide a camera calibration scheme to at least partially solve the above-mentioned problems.

[0005] According to a first aspect of the embodiments of this application, a camera calibration method is provided, comprising: acquiring three-dimensional point cloud data and two-dimensional image data of the environment in which the device is located, collected by the device having autonomous driving function in a straight-line driving state; determining a first ground normal of the device in an IMU coordinate system based on the three-dimensional point cloud data; determining a second ground normal of the device in a camera coordinate system based on the two-dimensional image data; determining a roll angle corresponding to the device based on the first ground normal and the second ground normal; and calibrating the device by at least the roll angle.

[0006] According to a second aspect of the present application, a device with autonomous driving function is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction causes the processor to perform an operation corresponding to the method described in the first aspect.

[0007] According to a third aspect of the embodiments of this application, a computer storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0008] According to the camera calibration scheme provided in this application, devices with autonomous driving functions can perform camera calibration, especially roll angle calibration, in straight-line driving scenarios. Compared with traditional schemes that require the device to turn, this expands the applicable scenarios for camera calibration, enabling timely camera calibration in situations such as long-distance or high-speed straight-line driving. This provides accurate environmental perception data for subsequent autonomous driving behavior, preventing inaccuracies in autonomous driving and ensuring autonomous driving safety. Specifically, the scheme obtains the corresponding ground normals based on the 3D point cloud data and 2D image data of the device with autonomous driving functions in straight-line driving conditions. These are the first ground normal in the IMU coordinate system and the second ground normal in the camera coordinate system. Since most perception data uses the IMU coordinate system or the camera coordinate system, the ground normals of these two coordinate systems can be used to determine the roll angle of the device with autonomous driving functions relatively accurately. While using this as a basis for camera calibration, the calibration results can also be widely applied to other perception data of the device. This provides an effective reference for subsequent data fusion and processing based on perception data of the device with autonomous driving functions, ensuring accurate perception and driving safety. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0010] Figure 1 A schematic diagram of an exemplary system for a camera calibration method applicable to embodiments of this application;

[0011] Figure 2A This is a flowchart illustrating the steps of a camera calibration method according to an embodiment of this application.

[0012] Figure 2B for Figure 2A A schematic diagram of the coordinate system corresponding to a standard road lane line in the illustrated embodiment;

[0013] Figure 2C for Figure 2A A schematic diagram of obtaining the corresponding lane rectangle based on two-dimensional image data in the embodiment shown;

[0014] Figure 2D for Figure 2A A schematic diagram of a preset lane shape in the illustrated embodiment;

[0015] Figure 2E for Figure 2A A schematic diagram of a camera calibration process in the illustrated embodiment;

[0016] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0017] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0018] The specific implementation of the embodiments of this application will be further described below with reference to the accompanying drawings.

[0019] Figure 1 An exemplary system for a camera calibration method applicable to embodiments of this application is shown. For example... Figure 1 As shown, the system 100 may include a cloud server 102, a communication network 104, and / or one or more devices with autonomous driving capabilities. Figure 1 The example in the text is 106 autonomous vehicles.

[0020] The cloud server 102 can be any suitable device for storing information, data, programs, and / or any other suitable type of content, including but not limited to distributed storage system devices, server clusters, computing cloud server clusters, etc. In some embodiments, the cloud server 102 can perform any suitable function. For example, in some embodiments, the cloud server 102 can be used to store various data sent by the autonomous vehicle 106, including but not limited to 3D point cloud data, 2D image data, and calibration results and intermediate process data of camera calibration based on 3D point cloud data and 2D image data.

[0021] In some embodiments, communication network 104 may be any suitable combination of one or more wired and / or wireless networks. For example, communication network 104 may include any one or more of the following: the Internet, intranet, wide area network (WAN), local area network (LAN), wireless network, digital subscriber line (DSL) network, frame relay network, asynchronous transfer mode (ATM) network, virtual private network (VPN), and / or any other suitable communication network. Autonomous vehicle 106 may be connected to communication network 104 via one or more communication links (e.g., communication link 112), which may be linked to cloud server 102 via one or more communication links (e.g., communication link 114). Communication links may be any communication link suitable for transmitting data between autonomous vehicle 106 and cloud server 102, such as network links, dial-up links, wireless links, hardwired links, any other suitable communication links, or any suitable combination of such links.

[0022] The autonomous vehicle 106 includes at least sensors (such as LiDAR and cameras) and a processor. The sensors are used to collect environmental data of the environment in which the autonomous vehicle 106 operates, and the processor is used to perform camera calibration based on the environmental data. In some embodiments, the LiDAR is used to collect three-dimensional point cloud data of the autonomous vehicle 106 in a straight-line driving state, and the camera is used to collect two-dimensional image data of the autonomous vehicle 106 in a straight-line driving state. The processor is used to determine a first ground normal of the autonomous vehicle 106 in the IMU coordinate system based on the three-dimensional point cloud data, and a second ground normal of the autonomous vehicle 106 in the camera coordinate system based on the two-dimensional image data. Furthermore, the roll angle of the autonomous vehicle 106 is determined based on these two normals, and combined with the pitch and yaw angles for camera calibration. In some embodiments, the autonomous vehicle 106 can also perform data fusion of the three-dimensional point cloud data and the two-dimensional image data based on the camera calibration results, and perform subsequent application processing such as target detection, driving decision-making, or path planning based on the fusion results.

[0023] The camera calibration scheme of this application will be described below through several embodiments.

[0024] Reference Figure 2A The diagram illustrates a step flowchart of a camera calibration method according to an embodiment of this application.

[0025] The camera calibration method in this embodiment includes the following steps:

[0026] Step S202: Acquire three-dimensional point cloud data and two-dimensional image data of the environment in which the device is located, collected by the device with autonomous driving function in a straight-line driving state.

[0027] In this application embodiment, the device with autonomous driving function may be a device with fully autonomous driving capability, such as a vehicle, aircraft, or robot, or it may be a device with autonomous driving as an auxiliary function, such as a vehicle, aircraft, or robot with assisted autonomous driving function.

[0028] Devices with autonomous driving capabilities are typically equipped with various sensors, such as LiDAR and cameras, to collect environmental data in different modalities, providing a data foundation for the device's perception of its environment. In this embodiment, LiDAR can be used to collect 3D point cloud data of the environment in which the device is located, and cameras can be used to collect 2D image data of the environment in which the device is located.

[0029] Furthermore, unlike traditional solutions, this application's solution is applicable to straight-line driving scenarios, such as highway driving scenarios, in order to achieve camera calibration under conditions with a low probability of turning.

[0030] Step S204: Based on the 3D point cloud data, determine the first ground normal of the device with autonomous driving function in the IMU coordinate system; and based on the 2D image data, determine the second ground normal of the device with autonomous driving function in the camera coordinate system.

[0031] Autonomous driving technology typically involves multiple coordinate systems. In this embodiment, the main coordinate systems involved are the IMU (Inertial Measurement Unit) coordinate system and the camera coordinate system. The origin of the IMU coordinate system is at the origin of the gyroscope and accelerometer coordinates, and the X, Y, and Z axes are parallel to the corresponding axes of the gyroscope and accelerometer, respectively. In the camera coordinate system, assuming point O is the optical center (projection center) of the camera, X... C axis and Y C The axis is parallel to the X and Y axes of the imaging plane coordinate system, and the Z axis is parallel to the X and Y axes. C The axis is the camera's optical axis, perpendicular to the image plane. Therefore, the distance between point O and the X... C Y C Z C The rectangular coordinate system formed by the axes is the camera coordinate system.

[0032] For 3D point cloud data, it inherently possesses 3D coordinates (x, y, z) in the radar coordinate system. Based on this, a subset of feature points can be selected from the 3D point cloud data. Using the conventional transformation relationship between the radar coordinate system and the IMU coordinate system, these selected 3D point cloud data can be projected into the IMU coordinate system. Furthermore, based on a plane fitting algorithm, a corresponding plane in the IMU coordinate system can be fitted from these 3D point cloud data. Then, based on this plane, the normal and normal direction of the device with autonomous driving capabilities in the IMU coordinate system can be determined, which is the first ground normal.

[0033] For two-dimensional image data, this embodiment determines the second ground normal based on the lane line features of the road. This includes: detecting lane lines in the two-dimensional image data to obtain multiple lane lines in the lane currently occupied by the device with autonomous driving capabilities; and determining the second ground normal of the device with autonomous driving capabilities in the camera coordinate system based on the correspondence between some lane lines in these multiple lane lines and pre-stored standard road lane lines in the camera coordinate system. Compared to traditional methods of detection and determination using neural network models, this embodiment eliminates the need for neural network models, significantly reducing the difficulty and cost of implementation and effectively improving the efficiency of determining the ground normal based on two-dimensional images.

[0034] The above method requires pre-storing information on standard road lane lines in the camera coordinate system. In one feasible approach, taking highways as an example, a coordinate system for the rectangle formed by two parallel lane dashed lines in 3D space can be established based on the national standard length of the lane dashed lines. Figure 2B As shown.

[0035] Depend on Figure 2B As can be seen, the national standard stipulates that the length of the dashed lane lines on highways is 6 meters, and the spacing between two parallel dashed lane lines is 3.75 meters, meaning the width between the two dashed lane lines is 3.75 meters. Therefore, a rectangular frame is formed by the two dashed lane lines. One side of this frame corresponding to one dashed lane lies on the Y-axis in the camera coordinate system, and the side connecting the lower endpoints of the two dashed lane lines lies on the X-axis. The axis perpendicular to both the X and Y axes is the Z-axis, which represents the normal direction in the camera coordinate system. For different roads, the national standard for lane dashed lines may vary. By collecting dashed frame data from different roads and establishing the corresponding camera coordinate system, a system can be created that corresponds to the dashed lane lines on different roads. Figure 2B A data sample set of dashed rectangular bounding boxes for lanes on medium and high-speed roads.

[0036] Based on this, optionally, determining the second ground normal of the device with autonomous driving function in the camera coordinate system according to the correspondence between some lane lines in the multiple lane lines and the standard road lane lines in the pre-stored camera coordinate system can be achieved by: determining the two lane edge lines in the multiple lane lines that are closest to the current position of the device with autonomous driving function; constructing a rectangle based on the two lane edge lines; and determining the second ground normal of the device with autonomous driving function in the camera coordinate system according to the correspondence between the constructed rectangle and the rectangle formed by the standard road lane lines in the pre-stored camera coordinate system.

[0037] In one feasible approach, the aforementioned lane line detection of two-dimensional image data to obtain multiple lane lines of the lane currently occupied by the device with autonomous driving function includes: performing feature point detection on the two-dimensional image data to obtain multiple corresponding feature points; determining ground feature points from the multiple feature points according to the contour of the lane currently occupied by the device with autonomous driving function; or, determining ground feature points from the multiple feature points according to a preset lane shape; filtering lane feature points from the ground feature points, and obtaining multiple lane lines of the lane currently occupied by the device with autonomous driving function based on the lane feature points. The filtering of lane feature points from the ground feature points can be implemented as follows: filtering multiple feature points that conform to straight-line characteristics from the ground feature points; determining lane feature points based on the multiple feature points that conform to straight-line characteristics.

[0038] A specific example of the above process is as follows: Figure 2C As shown.

[0039] Figure 2C In the example shown, feature points are first extracted from the two-dimensional image data. In practical applications, those skilled in the art can use any appropriate method to extract feature points from the image, such as using Spatial Pyramid Pooling (SPP). The extracted feature points include both ground features and other environmental features, such as... Figure 2C The top left corner of the image shows the next step: selecting ground feature points from this large number of feature points. To improve the speed and efficiency of this selection process, this embodiment uses lane contours or preset lane shapes for selection, such as... Figure 2C The image is shown in the upper right corner.

[0040] When using lane contours, you can first identify all feature points belonging to the lane from a large number of feature points; then, based on the feature points located at the edges of these feature points, you can determine the lane contour. Feature points within the lane contour can be considered as ground feature points. In this way, the selection of ground feature points is more accurate.

[0041] When using a preset lane shape, for devices with autonomous driving capabilities, the captured two-dimensional images typically exhibit a top-heaven orientation and a bottom-ground orientation. Furthermore, from the device's perspective, the lane will appear wider at the near end and narrower at the very end, resembling a trapezoid. Based on this, a trapezoidal lane shape can be preset, for example, such as... Figure 2D As shown. Furthermore, using the lane shape as the feature filtering range, features within this shape range are selected as ground feature points, while feature points outside this shape range are removed. This method allows for fast and efficient ground feature point filtering.

[0042] However, since not all ground feature points are lane feature points, it is necessary to identify lane feature points from among them. Because multiple dashed line segments on the same lane line are straight and continuous, feature points that conform to lane line characteristics, such as straight line characteristics, can be identified from the ground feature points. Furthermore, since different lanes correspond to different straight lines, at least two lane lines can be selected from the ground feature points. For example, the RANSAC (Random Sample Consensus) algorithm can be used to select lane line feature points. After determining the lane feature points, two consecutive lane lines consisting of dashed lines, i.e., two lane edge lines, can be identified for the lane currently occupied by the device with autonomous driving capabilities. Figure 2C The image is shown in the lower left corner.

[0043] Then, select the feature points of the two nearest dashed lane edges in the current field of view. Typically, each dashed line segment corresponds to two feature points, forming the four corner points for constructing a rectangular bounding box, such as... Figure 2C The image is shown in the bottom right corner.

[0044] Furthermore, a dashed rectangle is constructed based on these four corner points. This constructed dashed rectangle is compared with a pre-stored sample of dashed rectangles corresponding to standard road lane lines in the camera coordinate system to determine the matching dashed rectangle sample. Then, based on the normal corresponding to this dashed rectangle sample, the second ground normal in the camera coordinate system corresponding to the 2D image data can be determined. That is, based on the normal corresponding to the matrix frame formed by the standard road lane lines, the normal of the constructed rectangle is determined, and this determined normal is used as the second ground normal of the device with autonomous driving function in the camera coordinate system.

[0045] Through the above process, the first ground normal corresponding to the 3D point cloud data and the second ground normal corresponding to the 2D image data can be determined.

[0046] Step S206: Determine the roll angle corresponding to the device with autonomous driving function based on the first ground normal and the second ground normal.

[0047] For extrinsic parameter calibration between a camera and an IMU, the extrinsic parameter refers to the transformation relationship between the two sensors (i.e., the camera and the IMU), specifically including translation and rotation. Translation includes three degrees of freedom (x, y, z), and rotation includes three degrees of freedom (roll, pitch, yaw). Extrinsic parameter calibration calculates the transformation relationship between these two sensors. Specifically, in this embodiment, the calibration mainly focuses on the rotation angle calibration between the camera and the IMU, i.e., the calibration of (roll, pitch, yaw). Here, roll represents the roll angle in the extrinsic parameter rotation, pitch represents the pitch angle, and yaw represents the yaw angle. The solution in this embodiment can achieve the calibration of rotation angles, especially the roll angle, without relying on complex neural network models or requiring the device to perform turning operations.

[0048] Specifically, based on the first ground normal of the device in the IMU coordinate system obtained from 3D point cloud data, and the second ground normal of the device in the camera coordinate system obtained from 2D image data, the roll angle between the camera and the IMU can be calculated. However, to avoid deviations caused by a single frame, in this embodiment, multiple first ground normals and multiple second ground normals corresponding to multiple frames of 3D point cloud data and multiple frames of 2D image data are obtained respectively; based on the multiple first ground normals and the corresponding multiple second ground normals, multiple roll angles corresponding to the device with autonomous driving function are determined; the multiple roll angles are smoothed, and the roll angle corresponding to the device with autonomous driving function is determined based on the smoothing result. That is, the normals of multiple frames of camera coordinate system and IMU coordinate system are accumulated, and multiple frames of data are used for verification and roll angle estimation. Then, Kalman filtering is used for smoothing to ensure the accuracy and objectivity of the final result.

[0049] Step S208: Perform camera calibration on the device with autonomous driving function, at least based on the roll angle.

[0050] Once the roll angle is determined, the pitch angle and yaw angle, obtained by any other means, can be combined to calibrate the rotation angle between the camera and the IMU.

[0051] However, to ensure calibration accuracy, this embodiment employs hand-eye calibration for the pitch and yaw angles. This includes: performing hand-eye calibration of the IMU and camera of the device with autonomous driving capabilities based on 3D point cloud data and 2D image data; determining the pitch and yaw angles corresponding to the device with autonomous driving capabilities based on the hand-eye calibration results; and calibrating the camera of the device with autonomous driving capabilities based on the roll, pitch, and yaw angles. Hand-eye calibration is a method for obtaining coordinate transformation relationships between the camera and other robotic arms. In this embodiment, it is applied to the camera and IMU to obtain the coordinate transformation relationships between them, particularly the transformation relationships between rotation and translation parameters.

[0052] Specifically, the essential matrix between two two-dimensional image data can be obtained; the essential matrix can be decomposed to obtain the rotation and translation parameters in the camera coordinate system; based on the rotation and translation parameters in the camera coordinate system and the rotation and translation parameters in the IMU coordinate system, hand-eye calibration of the IMU and camera of the device with autonomous driving function can be performed.

[0053] The essential matrix is ​​the transformation between the coordinate systems of two camera views. It consists of two parts: the translation vector t and the rotation matrix R, which can be expressed as E = [t] × R. The essential matrix has 5 degrees of freedom: 3 degrees of freedom for translation and 3 degrees of freedom for rotation, minus 1 degree of freedom for scale uncertainty. Because the viewpoints of two 2D image data are different, the essential matrix between them can be determined using the two 2D image data. In one feasible approach, two adjacent keyframe 2D image data can be selected to obtain the essential matrix between them. The decomposition of the essential matrix aims to obtain R and t, the relative transformations of the two images, through the essential matrix. This can be obtained by performing singular value decomposition on the essential matrix, where R corresponds to the rotation parameter and t corresponds to the translation parameter.

[0054] For the rotation and translation parameters in the IMU coordinate system, the position (R) and translation (t) between the two keyframe 2D image data can be interpolated based on the fused localization result. In this embodiment, there are no restrictions on the specific method of obtaining the fused localization result or the specific implementation of the fused localization; it is sufficient that the result can be obtained.

[0055] However, to ensure the accuracy of hand-eye calibration and avoid the adverse effects of invalid data on hand-eye calibration, constraints are set for the essential matrix in this embodiment to filter out invalid feature point data of the essential matrix. In this case, hand-eye calibration of the IMU and camera of the device with autonomous driving function can be achieved by determining whether the essential matrix satisfies the constraints, which include: feature point distribution constraints indicating whether the feature points corresponding to the essential matrix are uniformly distributed; camera translation constraints indicating whether the translation parameters in the camera coordinate system decomposed from the essential matrix satisfy a preset translation relationship; and rotation consistency constraints indicating whether the rotation parameters decomposed from the essential matrix satisfy a preset rotation relationship. If all are satisfied, then hand-eye calibration of the IMU and camera of the device with autonomous driving function is performed based on the camera rotation and translation parameters, as well as the rotation and translation parameters in the IMU coordinate system.

[0056] The feature point distribution constraints used to indicate whether the feature points corresponding to the essential matrix are uniformly distributed include: determining whether the feature points corresponding to the essential matrix are uniformly distributed in the two-dimensional image represented by the two-dimensional image data; the camera translation constraints used to indicate whether the translation parameters in the camera coordinate system decomposed by the essential matrix satisfy the preset translation relationship include: determining whether the absolute value of the translation parameters in the camera coordinate system decomposed by the essential matrix in the vertical direction is much greater than the absolute value in the horizontal direction and the absolute value in the vertical direction; the rotation angle consistency constraints used to indicate whether the rotation parameters decomposed by the essential matrix satisfy the preset rotation relationship include: determining whether the magnitude corresponding to the rotation parameters decomposed by the essential matrix and the magnitude corresponding to the rotation parameters in the IMU coordinate system are approximately equal.

[0057] For example, feature point distribution constraints can be achieved by plotting a grid of a preset size on the image represented by the two-dimensional image data. A certain proportion of the grid must contain feature points, and essential matrices with more than a certain number of feature points within the grid are filtered out. The preset grid size can be appropriately set by those skilled in the art according to actual needs, such as 100 pixels * 100 pixels; this embodiment does not impose such limitations. Similarly, the number threshold can be appropriately set by those skilled in the art according to actual needs. When the number of feature points within the grid exceeds this threshold, the essential matrix and its corresponding two-dimensional image data are discarded and no longer used for subsequent decomposition and hand-eye calibration.

[0058] Camera translation constraints can be determined based on the translation parameter t obtained from the essential matrix decomposition. The absolute value of t in the Z-axis direction is considered significantly greater than its absolute values ​​in the X-axis and Y-axis directions. If it is significantly greater, the camera translation constraint is considered satisfied.

[0059] The rotation consistency constraint can be determined by judging whether the magnitudes of the rotation vectors corresponding to the rotation parameters in the IMU coordinate system and the rotation parameters decomposed from the essential matrix are approximately equal. If they are approximately equal, the rotation consistency constraint is considered to be satisfied; otherwise, the essential matrix and its two corresponding two-dimensional image data are discarded.

[0060] The above constraints can effectively ensure the validity and accuracy of the data used for hand-eye calibration.

[0061] However, to make the hand-eye calibration results more accurate and objective, one feasible approach is to perform hand-eye calibration of the device's IMU and camera based on rotation and translation parameters in the camera coordinate system and rotation and translation parameters in the IMU coordinate system. This can be achieved by: acquiring multiple sets of rotation and translation parameters in the camera coordinate system corresponding to two adjacent sets of 2D image data, and multiple sets of rotation and translation parameters in the IMU coordinate system corresponding to multiple frames of 3D point cloud data; performing hand-eye calibration based on the multiple sets of rotation and translation parameters in the camera coordinate system and the multiple sets of rotation and translation parameters in the IMU coordinate system to obtain multiple hand-eye calibration results; smoothing the multiple hand-eye calibration results; and determining the hand-eye calibration results of the device's IMU and camera with autonomous driving function based on the smoothing result.

[0062] The following, combined with Figure 2E The complete process of the above camera calibration is illustrated by example.

[0063] Depend on Figure 2E As can be seen, for the online roll angle calibration: First, the ground normal in the IMU coordinate system, i.e., the first ground normal, is obtained from the 3D point cloud data in the IMU coordinate system. Then, for the 2D image data, the ground normal in the camera coordinate system, i.e., the second ground normal, is calculated using the PnP method through methods such as ground feature point filtering, lane line feature point filtering, and dashed matrix box endpoint filtering, as described above. Then, the roll angle between the camera and the IMU is calculated using the first ground normal in the IMU coordinate system and the second ground normal in the camera coordinate system. Specifically, this includes: accumulating the ground normals in the camera coordinate system and the IMU coordinate system for multiple frames, verifying them by combining multiple frames of 3D point cloud data and 2D image data, and estimating the roll angle. Finally, Kalman filtering is used for smoothing, and the final roll angle between the camera and the IMU is obtained based on the smoothing result.

[0064] For the online calibration of pitch and yaw angles: First, based on two adjacent image keyframes i and i+1, superpoint+superglue is used to complete feature extraction and matching between the two image keyframes. Then, the essential matrix between the two image keyframes is calculated, and the essential matrix is ​​filtered based on constraints, including: i) Feature point distribution constraint: whether the feature points of the essential matrix are uniformly distributed on the image. For example, a grid can be drawn on the image, and a certain proportion (e.g., 80%) of the grid needs to have feature points distributed. The essential matrix and its corresponding two image keyframes with a large number of feature points in the grid are filtered; ii) Camera translation constraint: for the translation parameter t decomposed from the essential matrix, its absolute value in the Z-axis direction should be much greater than its absolute values ​​in the X-axis and Y-axis. If this is not met, the essential matrix and its corresponding two key image frames are discarded; iii) Rotation consistency constraint: whether the magnitude of the rotation vector corresponding to the rotation parameter in the IMU coordinate system and the rotation parameter decomposed from the essential matrix are approximately equal. If the difference is too large, the essential matrix and its corresponding two key image frames are discarded. Based on the filtered essential matrix, the essential matrix is ​​decomposed to obtain R and t in the camera coordinate system, denoted as Ri. c and t c Furthermore, for 3D point cloud data, based on the fused localization results, the localization pose R and t between two image keyframes are interpolated and calculated, denoted as R. i and t i Then, accumulate multiple sets of R... c and t c , and R i and t i The results are then processed using hand-eye calibration to calculate the pitch and yaw angles; after smoothing using Kalman filtering, the final pitch and yaw angles are output. Combined with the previously calculated roll angle, the roll, pitch, and yaw angles can be output.

[0065] This embodiment enables camera calibration, particularly roll angle calibration, for devices with autonomous driving capabilities during straight-line driving scenarios. Compared to traditional methods that require camera calibration during turning, this expands the applicable scenarios for camera calibration, allowing for timely calibration in situations such as long-distance or high-speed straight-line driving. This provides accurate environmental perception data for subsequent autonomous driving behavior, preventing inaccuracies and ensuring safety. Specifically, the solution uses 3D point cloud data and 2D image data from the device's straight-line driving state to obtain the corresponding ground normals: a first ground normal in the IMU coordinate system and a second ground normal in the camera coordinate system. Since most perception data uses the IMU or camera coordinate system, using the ground normals of these two coordinate systems allows for relatively accurate determination of the roll angle of the device. This calibration result can also be widely applied to other perception data of the device, providing an effective reference for subsequent data fusion and processing based on perception data, ensuring accurate perception and driving safety.

[0066] Reference Figure 3 This document illustrates a structural schematic diagram of a device with autonomous driving functionality according to an embodiment of this application. The specific embodiments of this application do not limit the specific implementation of the device with autonomous driving functionality.

[0067] like Figure 3 As shown, the device with autonomous driving function may include: processor 302, communication interface 304, memory 306, and communication bus 308.

[0068] in:

[0069] The processor 302, communication interface 304, and memory 306 communicate with each other via communication bus 308.

[0070] Communication interface 304 is used to communicate with other electronic devices or servers.

[0071] The processor 302 is used to execute program 310, specifically the relevant steps in the above method embodiments.

[0072] Specifically, program 310 may include program code that includes computer operation instructions.

[0073] Processor 302 may be a CPU, an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The smart device may include one or more processors of the same type, such as one or more CPUs; or it may include processors of different types, such as one or more CPUs and one or more ASICs.

[0074] Memory 36 is used to store program 310. Memory 306 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0075] Specifically, program 310 can be used to cause processor 302 to perform the operation corresponding to the camera calibration method described in any of the foregoing multiple method embodiments.

[0076] The specific implementation of each step in program 10 can be found in the corresponding descriptions of the steps and units in the above method embodiments, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0077] This application also provides a computer program product, including computer instructions that instruct a computing device to perform operations corresponding to the camera calibration methods in the above-described multiple method embodiments.

[0078] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0079] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code that, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0080] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0081] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. A camera calibration method, comprising: Acquire three-dimensional point cloud data and two-dimensional image data of the environment in which the device is located, collected by the device with autonomous driving function in a straight-line driving state; Based on the three-dimensional point cloud data, determine the first ground normal of the device in the IMU coordinate system; And, based on the two-dimensional image data, determining the second ground normal of the device in the camera coordinate system includes: performing lane line detection on the two-dimensional image data to obtain multiple lane lines of the lane where the device is currently located; determining the two lane edge lines among the multiple lane lines that are closest to the current position of the device; constructing a rectangular frame based on the two lane edge lines; and determining the second ground normal of the device in the camera coordinate system based on the correspondence between the constructed rectangular frame and the rectangular frame formed by the standard road lane lines in the pre-stored camera coordinate system. The roll angle corresponding to the equipment is determined based on the first ground normal and the second ground normal; The device is calibrated at least based on the roll angle.

2. The method according to claim 1, wherein, The step of calibrating the camera on the device at least based on the roll angle includes: Based on the three-dimensional point cloud data and the two-dimensional image data, perform hand-eye calibration of the device's IMU and camera; Based on the hand-eye calibration results, determine the pitch angle and yaw angle corresponding to the device; The device is calibrated based on the roll angle, pitch angle, and yaw angle.

3. The method according to claim 1 or 2, wherein, Determining the second ground normal of the device in the camera coordinate system includes: Based on the normal corresponding to the matrix frame formed by the standard road lane lines, the normal of the constructed rectangular frame is determined, and the determined normal of the rectangular frame is used as the second ground normal of the device in the camera coordinate system.

4. The method according to claim 1 or 2, wherein, The step of performing lane line detection on the two-dimensional image data to obtain multiple lane lines in the lane where the device is currently located includes: Feature point detection is performed on the two-dimensional image data to obtain multiple corresponding feature points; Ground feature points are determined from the plurality of feature points according to the outline of the lane where the device is currently located; or, ground feature points are determined from the plurality of feature points according to a preset lane shape. Lane feature points are selected from the ground feature points, and multiple lane lines of the lane where the device is currently located are obtained based on the lane feature points.

5. The method according to claim 4, wherein, The step of filtering lane feature points from the ground feature points includes: Multiple feature points that conform to the straight line feature are selected from the ground feature points; Lane feature points are determined based on multiple feature points that conform to the characteristics of a straight line.

6. The method according to claim 1 or 2, wherein, Determining the roll angle corresponding to the equipment based on the first ground normal and the second ground normal includes: Acquire multiple first ground normals and multiple second ground normals corresponding to multiple frames of 3D point cloud data and multiple frames of 2D image data, respectively; Based on multiple first ground normals and corresponding multiple second ground normals, multiple roll angles corresponding to the equipment are determined; The multiple roll angles are smoothed, and the roll angle corresponding to the device is determined based on the result of the smoothing process.

7. The method according to claim 2, wherein, The step of performing hand-eye calibration of the device's IMU and camera based on the 3D point cloud data and the 2D image data includes: Obtain the essential matrix between two two-dimensional image data; The essential matrix is ​​decomposed to obtain the rotation and translation parameters in the camera coordinate system; Based on the rotation and translation parameters in the camera coordinate system and the rotation and translation parameters in the IMU coordinate system, the hand-eye calibration of the device's IMU and camera is performed.

8. The method according to claim 7, wherein, The step of performing hand-eye calibration of the device's IMU and camera based on the rotation and translation parameters in the camera coordinate system and the IMU coordinate system includes: Determine whether the essential matrix satisfies the constraints, which include: feature point distribution constraints indicating whether the feature points corresponding to the essential matrix are uniformly distributed; camera translation constraints indicating whether the translation parameters in the camera coordinate system decomposed by the essential matrix satisfy a preset translation relationship; and rotation consistency constraints indicating whether the rotation parameters decomposed by the essential matrix satisfy a preset rotation relationship. If all conditions are met, then the hand-eye calibration of the device's IMU and camera is performed based on the rotation and translation parameters in the camera coordinate system and the rotation and translation parameters in the IMU coordinate system.

9. The method according to claim 8, wherein, The feature point distribution constraint used to indicate whether the feature points corresponding to the essential matrix are uniformly distributed includes: determining whether the feature points corresponding to the essential matrix are uniformly distributed in the two-dimensional image represented by the two-dimensional image data; The camera translation constraint used to indicate whether the translation parameters in the camera coordinate system decomposed by the essential matrix satisfy the preset translation relationship includes: determining whether the absolute value of the translation parameters in the camera coordinate system decomposed by the essential matrix in the vertical direction is much greater than the absolute value in the horizontal direction and the absolute value in the vertical direction. The rotational consistency constraint used to indicate whether the rotational parameters decomposed from the essential matrix satisfy the preset rotational relationship includes: determining whether the modulus corresponding to the rotational parameters decomposed from the essential matrix and the modulus corresponding to the rotational parameters in the IMU coordinate system are approximately equal.

10. The method according to any one of claims 7-9, wherein, The step of performing hand-eye calibration of the device's IMU and camera based on the rotation and translation parameters in the camera coordinate system and the IMU coordinate system includes: Obtain multiple sets of rotation and translation parameters in camera coordinate systems corresponding to two adjacent sets of 2D image data, and multiple sets of rotation and translation parameters in IMU coordinate systems corresponding to multiple frames of 3D point cloud data; Hand-eye calibration is performed based on rotation and translation parameters in multiple camera coordinate systems and multiple IMU coordinate systems to obtain multiple hand-eye calibration results. Multiple hand-eye calibration results are smoothed, and the hand-eye calibration results of the device's IMU and camera are determined based on the smoothing results.

11. A device with autonomous driving capability, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform an operation corresponding to the method as described in any one of claims 1-10.

12. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1-10.

Images