System and method for calibrating a camera on an autonomous driving system

Abstract

Various aspects for online calibration of cameras on an autonomous driving system are disclosed, providing a system and method for calibrating cameras on an autonomous driving system. The aspects can include a wheel odometer configured to measure a current speed of a traveling vehicle; a speed monitor configured to determine whether the current speed of the traveling vehicle is high or low, and a camera calibrator configured to calibrate at least one front camera of a moving vehicle according to a position of vanishing points of two lane lines when the current speed is high. When the current speed is low and the vehicle is still moving, the camera calibrator can be further configured to calibrate a plurality of side cameras and a rear camera of the moving vehicle according to a pose graph including relative positions of the plurality of side cameras and the rear camera.

Description

Technical Field

[0001] This invention generally relates to the field of autonomous driving technology, and in particular to a system and method for calibrating cameras for autonomous driving systems. Background Technology

[0002] The development of vision-based autonomous driving systems is crucial for the safety of driving vehicles. These systems may require information about the position and orientation of cameras relative to the vehicle's coordinates. Typical applications include mapping lane lines onto a top-down view to keep the vehicle within its lane, estimating the distance to detected moving objects, and stitching together surround-view images from front, side, and rear cameras. Since the positions of the cameras and the vehicle are typically fixed initially but may undergo slight displacements during subsequent operations, camera orientation calibration is critical for driver assistance systems.

[0003] Traditional vehicle camera calibration methods can be performed in a controlled environment (i.e., offline calibration) with the aid of controlled visual targets and tools. This is typically used in factory production lines or service stations, where a well-controlled environment can be established with specialized equipment such as camera modes, vehicle centering platforms, and lighting control devices.

[0004] Offline calibration has several limitations. First, it is inconvenient because the placement and size of the equipment are limited by the camera's position, resolution, and field of view. Second, the cost of setting up a calibration station is high, and it often requires a significant amount of manpower to assist with the process.

[0005] In some cases, vehicle cameras deviate from their calibration orientation, requiring immediate recalibration. One possibility is that the camera has shifted from its original position after servicing, repair, or replacement at a service station. Another possibility is that the camera has shifted over time due to vibration, installation issues, or vehicle deformation after a period of driving. The first scenario can be addressed with offline calibration in a dedicated camera calibration room, but this is costly in terms of equipment and manpower. However, there is no solution for the second scenario, as using an inaccurate camera orientation to monitor the surrounding environment poses a risk to driver assistance perception. In contrast, online calibration offers a solution that completes the entire calibration process in an uncontrolled environment without the need for any targets or tools. It not only saves significant costs for the first scenario but also automatically monitors and completes the calibration for the second scenario by driving on the road for a certain period. The online solution is cost-free, requires only driving, and involves no human intervention; each vehicle owner doesn't even notice it.

[0006] Existing online calibration methods have limitations when applied to a complete vehicle camera system consisting of front, side, and rear cameras to cover the vehicle's perceived surrounding field of view. Odometer-based methods require high accuracy in the location and measurements of other sensors and often include cumulative errors due to bias or measurement inaccuracies. From a vehicle setup perspective, using LiDAR as an anchor for camera calibration limits its application in vehicles without such sensors. Vision-based methods are often not a general solution for complete camera systems; they either calibrate a single front camera or calibrate one or two surround cameras using prior camera setups.

[0007] To overcome these limitations in previously published patents and literature, the integration of vehicle sensor measurements and camera perception systems is crucial. Instead of relying primarily on odometer sensor measurements, these measures can be utilized as supplementary signals to a vision solution to avoid accuracy issues. Vision measurements, including lane markings and object detection, can be readily obtained from the camera perception system of a vehicle equipped with a driver assistance package. Therefore, the online calibration system can be tightly integrated with the overall driving system, rather than becoming a resource-intensive standalone system. Summary of the Invention

[0008] The following is a brief summary of one or more aspects to provide a basic understanding of these aspects. This invention is not a broad overview of all conceived aspects, nor is it intended to identify key or important factors in all aspects, nor to define the scope of any or all aspects. Its sole purpose is to introduce some concepts of one or more aspects in a simplified form as a prelude to a more detailed description later.

[0009] One aspect of the present invention provides a method for calibrating cameras on an autonomous driving system. The method includes: receiving the current speed of a vehicle from a wheel odometer; determining that the current speed of the vehicle is greater than a first preset speed threshold; calibrating at least one front camera of the vehicle based on the determination that the current speed is greater than the first preset speed threshold and according to the positions of the vanishing points of two lane lines; determining that the current speed of the vehicle is less than a second preset speed threshold and greater than zero; and calibrating a plurality of side cameras and a rear camera of the vehicle based on the determination that the current speed is less than the second preset speed threshold and greater than zero and according to a pose diagram including the relative poses of a plurality of side cameras and a rear camera.

[0010] Another aspect of the present invention provides a system for calibrating cameras on an autonomous driving system. The system includes: a wheel odometer configured to measure the current speed of a traveling vehicle; a speed monitor configured to determine that the current speed of the traveling vehicle is greater than a first preset speed threshold; and a camera calibrator configured to calibrate at least one front camera of the traveling vehicle based on the determination that the current speed is greater than the first preset speed threshold, according to the positions of the vanishing points of two lane lines. The speed monitor is further configured to determine that the current speed of the traveling vehicle is less than a second preset speed threshold and greater than zero. The camera calibrator is further configured to calibrate a plurality of side cameras and a rear camera of the traveling vehicle based on a pose diagram including the relative poses of a plurality of side cameras and a rear camera, based on the determination that the current speed is less than the second preset speed threshold and greater than zero.

[0011] To achieve the foregoing and related objectives, one or more aspects include the features fully described herein and particularly pointed out in the claims. The following description and the accompanying drawings set forth certain illustrative features of one or more aspects in detail. However, these features merely illustrate several ways in which the principles of each aspect can be employed, and this description is intended to include all such aspects and their equivalents. Attached Figure Description

[0012] The disclosed aspects will be described below in conjunction with the accompanying drawings, which are provided for illustrative purposes and not for limiting the scope of the disclosure, wherein similar names denote similar elements, and wherein:

[0013] Figure 1 This is a schematic diagram of an autonomous driving system equipped with multiple cameras according to the present invention.

[0014] Figure 2 This is a schematic diagram of a camera calibrator for calibrating multiple cameras in an autonomous driving system according to the present invention.

[0015] Figure 3 This is a flowchart of a method for calibrating a camera on an autonomous driving system according to the present invention.

[0016] Figure 4 This is a schematic diagram of the vanishing point according to the present invention.

[0017] Figure 5 This is a flowchart of a method for determining the relative pose of a plurality of side cameras and a rear camera according to the present invention.

[0018] Figure 6 This is a schematic diagram of the process for determining the optimized loss of multiple side cameras and a rear camera according to the present invention.

[0019] Figure 7This is a schematic diagram of another process for calibrating multiple cameras according to the present invention. Detailed Implementation

[0020] The various aspects will now be described with reference to the accompanying drawings. In the following description, numerous specific details are set forth for illustrative purposes to provide a comprehensive understanding of one or more aspects. However, it will be apparent that these aspects can be practiced without these specific details.

[0021] In this specification, the terms “comprising” and “including”, and their derivatives, mean inclusion rather than limitation; the term “or”, also inclusive, means and / or.

[0022] In this specification, the various embodiments described below to illustrate the principles of the invention are for illustrative purposes only and should not be construed as limiting the scope of the invention in any way. The following description, taken in conjunction with the accompanying drawings, is intended to facilitate a thorough understanding of the illustrative embodiments of the invention as defined by the claims and their equivalents. In the following description, some specific details are provided for ease of understanding. However, these details are merely for illustrative purposes. Therefore, those skilled in the art should understand that various substitutions and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Furthermore, for the purpose of clarity and conciseness, some known functions and structures have not been described. Moreover, throughout the drawings, the same reference numerals refer to the same functions and operations.

[0023] This paper describes an online calibration system for a moving vehicle to calibrate the yaw and pitch directions of multiple (e.g., seven) cameras with overlapping fields of view, providing 360-degree surveillance. In some examples, two of the multiple cameras are mounted at the front of the moving vehicle, four at the sides, and one at the rear.

[0024] The vanishing point can be used to calibrate the two front cameras. The vanishing point is estimated by calculating the intersection of lane lines detected by the deep learning model of the autonomous driving system's built-in camera perception system.

[0025] The side and rear cameras can be calibrated using a pose graph optimization algorithm. The pose graph is constructed by using the absolute orientation of the cameras as vertices and the relative orientation between the two cameras as edges. The relative orientation is estimated through feature detection and matching. The pose graph is then optimized accordingly with a fixed front camera to balance errors across all edges.

[0026] Figure 1This is a schematic diagram of an autonomous driving system 100 equipped with multiple cameras according to the present invention. As shown, the autonomous driving system 100 can be implemented on a land vehicle 101.

[0027] The autonomous driving system 100 may include multiple cameras mounted at different locations on the vehicle 101. As mentioned above, the orientation of the cameras may shift due to vibration, mounting, and vehicle body deformation. This shift may occur in three different principal axes or directions: yaw, pitch, and roll. In contrast to conventional "offline" camera calibration, this invention provides a system and method for calibrating cameras while the vehicle is in motion. It should be noted that the term "online" refers to the inclusion of working components communicating with each other on a moving vehicle and a system for updating configurations including camera orientation, rather than requiring internet support. In some instances, the term "online" may be used interchangeably with "real-time."

[0028] In some examples, multiple cameras are mounted from the vehicle 101 facing different directions and can cover different fields of view (FOV). For example, the front main camera 102 and the front narrow camera 104 (collectively referred to as the "front cameras") can be mounted simultaneously on the top of the vehicle 101, facing forward. The FOV of the front main camera 102 is typically wider than that of the front narrow camera 104.

[0029] Other cameras may be mounted on the sides and rear of vehicle 101 (collectively referred to as "side cameras and rear cameras"). For example, a front right camera 106 facing forward right and a right rear camera 108 facing right rear may be mounted on the right side of vehicle 101. A front left camera 110 facing forward left and a left rear camera 112 facing left rear may be mounted on the left side of vehicle 101. A rear main camera 114 facing rearward may be mounted at the rear of vehicle 101.

[0030] The overlap FOV between two adjacent cameras can be manually adjusted and set to a preset degree. For example, the overlap FOV between the front main camera 102 and the front left camera 110 or the front right camera 106 can be approximately 28 degrees. The overlap FOV between the front right camera 106 and the right rear camera 108 can be the same as the overlap FOV between the front left camera 110 and the left rear camera 112. Both can be set to approximately 28 degrees. The overlap FOV between the rear main camera 114 and the left rear camera 112 or the right rear camera 108 can be approximately 36 degrees.

[0031] The example autonomous driving system 100 may further include a wheel odometer 116 configured to measure the current speed of the vehicle 101. As described in more detail below, online calibration according to the invention may include two-part calibration: calibrating the front camera when the vehicle 101 is traveling at a high speed, and calibrating the side and rear cameras when the vehicle 101 is traveling at a low speed. Therefore, the example autonomous driving system 100 may further include a speed monitor 118 configured to determine whether the measured current speed is “high” or “low” based on a preset speed threshold. For example, when the current speed is greater than 50 km / h, the current speed may be determined to be “high,” while when the current speed is less than 30 km / h, the speed may be determined to be “low.” The result of the speed determination may be sent to a camera calibrator 120 to trigger the calibration process.

[0032] Figure 2 This is a schematic diagram of a camera calibrator 120 for calibrating multiple cameras of an autonomous driving system according to the present invention.

[0033] If the speed monitor 118 determines that the current speed of the vehicle 101 is high, the camera calibrator 120 can be configured to initiate the calibration process of the pre-camera first.

[0034] In some examples, the camera calibrator 120 may first check whether one or more additional conditions are met before the calibration process to ensure that the environment around the vehicle 101 is suitable for the calibration process. For example, the camera calibrator 120 may check signals transmitted from other modules of the vehicle 101 via the Controller Area Network (CAN) service. The camera calibrator 120 may check signals from the wiper indicator to determine whether the wipers are working, indicating whether it is raining. The camera calibrator 120 may check signals from the angular rate display to determine whether the angular rate is less than one degree per second, indicating whether the vehicle 101 is turning.

[0035] Furthermore, the example autonomous driving system 100 may also include a visual perception system configured to detect lane lines and surrounding objects based on deep learning or artificial intelligence algorithms. The detection results can be sent to a camera calibrator 120. The camera calibrator 120 can be configured to determine whether the lane lines are straight and whether the vehicle 101 is moving forward in the middle of the lane lines, for example, by using a polynomial fit with second-order coefficients < α.

[0036] In some instances where additional conditions are met—namely, no rain, forward movement, and straight lane lines—the vanishing point determiner 202 of the camera calibrator 120 can be configured to calculate the intersection of two lane lines and determine that intersection as the vanishing point of the two lane lines. As the vehicle 101 moves, the vanishing point determiner 202 can be configured to periodically calculate multiple vanishing points. Once the count of multiple vanishing points reaches a preset threshold, the multiple vanishing points can be sent to the direction calculator 204.

[0037] The orientation calculator 204 can be configured to determine the offset between a preset principal point and a vanishing point, and further determine the yaw and pitch directions based on this offset. The yaw and pitch directions can indicate the displacement of the corresponding cameras. The example autonomous driving system 100 can be configured to accordingly utilize the new camera orientation parameters to compensate for the displacement in the autonomous driving algorithm.

[0038] according to Figure 4 The vanishing point, as well as the calculations for yaw and pitch, are described in more detail. For example... Figure 3 As shown, the yaw and pitch directions can be sent to the attitude diagram generator 214 for further processing.

[0039] If the speed monitor 118 determines that the current speed of vehicle 101 is low, the camera calibrator 120 can be configured to initiate the calibration process for the side and rear cameras after calibrating the pre-calibrated camera, provided a set of additional conditions are met. The camera calibrator 120 can be configured to check the signal from the wiper indicator to determine if the wipers are working to indicate whether it is raining. The camera calibrator 120 can check the signal from the angular rate display to determine if the angular rate is less than one degree per second. The camera calibrator 120 can further check the status of the low / high / fog beams and the ambient light to determine if the ambient light is sufficient.

[0040] For example, the pose determiner 206 of the camera calibrator 120 can be configured to determine the relative poses of a plurality of side cameras and a rear camera. The relative poses of the plurality of side cameras and the rear camera can refer to the position of each camera relative to a reference camera. The reference camera can be one of two adjacent cameras that have a connecting edge in the pose diagram.

[0041] More specifically, the pose determiner 206 may further include a feature detector 208 configured to detect features of surrounding static objects in images captured by the side and rear cameras. The feature point matcher 210 of the pose determiner 206 may be configured to match feature points of the same static object in different images based on ordered similarity. The pose calculator 212 of the pose determiner 206 may then be configured to determine relative poses.

[0042] Based on the relative poses of multiple side and rear cameras, and the yaw and pitch directions of the front camera, the pose map generator 214 can be configured to generate a pose map with one of the front cameras set as a reference point or anchor point. The loss calculator 216 can be configured to calculate the optimized loss for each of the side and rear cameras based on the pose map. The optimized loss can refer to the direction representing the displacement of the side and rear cameras. Similarly, the example autonomous driving system 100 can be configured to adjust the parameters in the autonomous driving algorithm accordingly to compensate for displacement.

[0043] Figure 3 This is a flowchart of a method 300 for calibrating a camera on an autonomous driving system according to the present invention. The process of performing the example method 300 may begin at block 302.

[0044] In block 302, the operation of method 300 may include receiving the current speed of the traveling vehicle from the wheel odometer. For example, speed monitor 118 may be configured to receive the current speed of vehicle 101 from wheel odometer 116.

[0045] In block 304, the operation of method 300 may include determining whether the current speed of the vehicle is greater than a first preset speed threshold or less than a second preset speed threshold. If the current speed is determined to be greater than the first preset speed threshold, the process of method 300 may continue to block 308. If the current speed is determined to be less than the second preset speed threshold but still greater than zero, the process of method 300 may continue to block 314.

[0046] At block 308, the operation of method 300 may include checking whether a first set of additional conditions are met based on signals transmitted via CAN service from wiper indicator 301, angular rate monitor 303, and lane detector 306. In some examples, camera calibrator 120 may be configured to determine whether it is raining based on signals from wiper indicator 301, whether vehicle 101 is turning based on signals from angular rate monitor 303, and whether lane lines are straight based on outputs from lane detector 306 in the visual perception system.

[0047] In block 310, the operation of method 300 may include determining the vanishing point by calculating the intersection of the two lane lines on the image plane. For example, the vanishing point determiner 202 of the camera calibrator 120 may be configured to calculate the intersection of the two lane lines and determine the intersection as the vanishing point of the two lane lines. As the vehicle 101 moves, the vanishing point determiner 202 may be configured to periodically calculate multiple vanishing points. Once the count of multiple vanishing points reaches a preset threshold, the multiple vanishing points may be sent to the direction calculator 204.

[0048] In block 312, the operation of method 300 may include calculating the yaw and pitch directions of at least one front camera based on the difference between the vanishing point and the principal point, and calibrating the front camera. For example, the orientation calculator 204 may be configured to determine the offset between a preset principal point and the vanishing point, and further determine the yaw and pitch directions based on that offset. The process of method 300 may continue to block 314.

[0049] In block 314, the operation of method 300 may include determining that a second set of additional conditions are met. For example, camera calibrator 120 may be configured to check signals from a wiper display to determine whether the wipers are working and whether it is raining. Camera calibrator 120 may check signals from an angular rate display to determine whether the angular rate is less than one degree per second. Camera calibrator 120 may further check the status of the low / high / fog beams and the ambient brightness to determine whether the ambient light is sufficient.

[0050] In block 316, the operation of method 300 may include determining the relative poses of a plurality of side cameras and a rear camera. For example, the pose determiner 206 of camera calibrator 120 may be configured to determine the relative poses of a plurality of side cameras and a rear camera.

[0051] In block 318, the operation of method 300 may include generating a pose diagram that includes the relative poses of multiple side cameras and a rear camera. For example, based on the relative poses of the multiple side cameras and the rear camera, as well as the yaw and pitch directions of the front camera, the pose diagram generator 214 may be configured to generate a pose diagram with one of the front cameras set as a reference or anchor point.

[0052] In block 320, the operation of method 300 may include calculating the optimization loss for each of the plurality of side cameras and rear cameras based on the pose map. For example, loss calculator 216 may be configured to calculate the optimization loss for each of the side cameras and rear cameras based on the pose map.

[0053] Figure 4 This is a schematic diagram of an example vanishing point according to the present invention.

[0054] Generally, given a three-dimensional (3D) point in the real world, the corresponding two-dimensional (2D) image point can be represented as x. 2d =K[R|T]X 3d Where K is the intrinsic matrix of the camera capturing the 2D image, and R and T represent 3D rotation and translation, respectively. In this invention, the vanishing point can be defined as a point on the image plane, representing the intersection of the 2D perspective projections of mutually parallel lines in 3D space. Specifically, the vanishing point is an infinite point independent of the translation T in 3D coordinates. Therefore, the vanishing point on a 2D image can be represented as VP.2d =KR·VP 3d .

[0055] Among them VP 2d This refers to the location of the vanishing point in a 2D image, and VP. 3d This refers to the 3D coordinates of the vanishing point. The vanishing point may correspond to the yaw and pitch directions of the front camera. To estimate the vanishing point, it is assumed that when the vehicle is traveling straight in the center of the two lanes, the lane lines can be considered parallel to its direction of motion.

[0056] Consider an image plane with principal points (cx, cy) and estimated vanishing points (x, y). Principal points are typically provided in the camera's inherent information and determined during camera manufacturing. For example... Figure 4 As shown, the vanishing point offset from the principal point in the plane can be represented as (u vp′ v vp )=(x vp′ y vp )-(cx′cy),

[0057] The yaw and pitch directions can be calculated separately as: u vp =f x tan(yaw) and v vp =-f y tan (pitch).

[0058] Figure 5 This is a flowchart of an example method 500 for determining the relative pose of a plurality of side cameras and a rear camera according to the present invention.

[0059] As part of calibrating the side and rear cameras, the process of determining or estimating the relative pose of the side and rear cameras can be performed based on the overlapping field of view (FOV) between two adjacent cameras, such as the overlapping FOV between the front main camera 102 and the front right camera 106. Images captured by the two adjacent cameras can be provided at the beginning of the process of method 500 to determine the relative pose. Dashed blocks may represent optional operations of method 500.

[0060] In block 502, the operation of method 500 may include detecting features of objects in images captured by multiple side cameras and a rear camera. For example, feature detector 208 may be configured to detect features of surrounding static objects according to some existing algorithm, such as the Scale Invariant Feature Transform (SIFT) algorithm. Since feature points of moving objects may cause errors in subsequent feature matching processes, feature points of moving objects may be removed. The moving objects in the image can then be labeled with bounding boxes. In other words, feature points in the bounding boxes may not be considered in later processes.

[0061] In block 504, the operation of method 500 may include matching feature points in an image based on ranked similarity. For example, feature point matcher 210 may be configured to match feature points in different images captured by two adjacent cameras and rank the feature points according to their similarity. The ranking may be performed by feature point matcher 210 according to the k-nearest neighbor (KNN) algorithm.

[0062] In some examples, the feature point matcher 210 can be further configured to filter and discard unqualified feature points. Filtering may include one or more stages, such as distance filtering, symmetry filtering, and nominal extrinsic filtering. In distance filtering, the feature point matcher 210 can be configured to discard two similar points that are too close together. In symmetry filtering, the feature point matcher 210 can be configured to discard feature points that can only be projected from one image to another, rather than the other way around. In nominal extrinsic filtering, the feature point matcher 210 can be configured to discard feature points based on the point correspondence between two cameras, which are constrained by the initial camera settings, including the camera translation relative to the vehicle and the ideal camera mounting angle. Feature points that conform to a criterion are retained. This criterion can be expressed as… Where x1 and x2 are the normalized corresponding points of two images taken by two adjacent cameras, t and R are the ideal mount translation and rotation, and δ is a preset threshold.

[0063] In block 508, the operation of method 500 may include determining the relative pose based on the matched feature points. For example, pose calculator 212 may be configured to estimate or determine the relative pose of each camera using adjacent cameras as reference points. More specifically, pose calculator 212 may be configured to compute the fundamental matrix between two adjacent cameras and decompose the fundamental matrix into rotation and translation matrices as the relative pose.

[0064] In block 510, the operation of method 500 may include rejecting unqualified relative poses. For example, when the relative pose appears to be a result of reprojection error and depth distribution, the relative pose may be discarded and rejected as invalid by pose calculator 212. Reprojection error may refer to the geometric error when projecting a point from one image to another using the estimated relative pose. The average error of all matching points should be less than a preset threshold. Otherwise, the estimated relative pose may be rejected. Furthermore, the depth (distance) of each point can be calculated based on the estimated relative pose. The standard deviation of the depth distribution formed by all matching points should be greater than a preset threshold. Otherwise, the estimated relative pose will be rejected.

[0065] In block 512, the operation of method 500 may include outputting a valid relative pose. For example, pose calculator 212 may be configured to output a valid relative pose to pose graph generator 214.

[0066] Figure 6 This is a schematic diagram of an example process for determining the optimized loss of multiple side cameras and a rear camera according to the present invention.

[0067] In some examples, the loss calculator 216 can be configured to generate a pose graph where each of the six cameras is a vertex, and the relative poses between any two adjacent cameras (e.g., R1 to R6, as shown) are edges. Adjacent cameras can refer to a pair of cameras connected by edges on the pose graph (e.g., front right camera 106 and right rear camera 108). Since the front main camera 102 has been calibrated before the other side cameras and the rear camera, the front main camera 102 can be set as the anchor point. The graph optimization loss can be formulated to minimize the difference between each side camera and the rear camera, as follows:

[0068] loss 后_主 =R 前_主 R1R2R3-R 前_主 R6R5R4

[0069]

[0070]

[0071]

[0072]

[0073] If other cameras are well calibrated, the pose map can also be applied to calibrate one or two cameras. For example, if only the left rear and front right cameras need calibration, the optimization loss can be expressed as:

[0074]

[0075]

[0076] Figure 7 This is a schematic diagram of another example process for calibrating multiple cameras according to the present invention.

[0077] As depicted, camera image streams 702 can be fed into one or more storage ring buffers 704 for each camera. Storage ring buffers 704 can temporarily store camera image streams 702 and feed them to vehicle perception system 720 (instead of the "visual perception system" described above). Vehicle perception system 720 can be configured to detect or identify static objects, including lane lines, in the camera image streams 702. Detected static objects can be passed to pose determiner 206, e.g., block 710. To reduce the possibility of later errors, moving objects can be detected but marked with bounding boxes, so that feature points within the bounding boxes are not considered input to feature point matcher 210.

[0078] When lane lines are detected in the camera image, the camera calibrator 120 can be configured to check the straightness of the lane lines (block 722) and other signals via CAN service 708. For example, the camera calibrator 120 can be configured to determine whether it is raining based on a signal from the wiper indicator 301, whether the vehicle 101 is turning based on a signal from the angular rate monitor 303, and whether the lane lines are straight based on the output from the lane detector 306 in the visual perception system.

[0079] In block 726, the vanishing point determiner 202 of the camera calibrator 120 can be configured to calculate the intersection of two lane lines and determine the intersection as the vanishing point of the two lane lines. Further, in block 728, when the collected vanishing point count, for example, the vanishing point count stored by the vanishing point counter 724, is greater than a preset threshold, for example, NH, a histogram can be created, and the peak value of the histogram can be selected as the rotation angle of the final calibration result shown in block 730. The calibration result can be sent to the calibration file management system 732, which manages the calibration profiles of the cameras in the autonomous driving system 100. In some examples, when the camera calibrator 120 sends the latest calibrated camera orientation to the file management system 732, the file management system 732 can update the camera profile and notify other modules of the autonomous driving system 100.

[0080] In block 706, camera calibrator 120 can similarly check whether additional conditions are met based on signals transmitted via CAN service 708. For example, camera calibrator 120 can be configured to check signals from a wiper display to determine whether the wipers are working and whether it is raining. Camera calibrator 120 can check signals from an angular rate display to determine whether the angular rate is below one degree per second. Camera calibrator 120 can further check the status of the low / high / fog beams and the ambient brightness to determine whether the ambient light is sufficient.

[0081] When the added conditions are met, the camera calibrator 120 can be configured to identify image pairs captured by two adjacent cameras from the storage ring buffer 704.

[0082] In block 710, feature detector 208 can be configured to detect features of surrounding static objects according to some existing algorithm, such as the scale-invariant feature transform (SIFT) algorithm. Feature point matcher 210 can be configured to match feature points in different images captured by two adjacent cameras and sort the feature points according to their similarity.

[0083] Similarly, in block 714, the pose calculator 212 can be configured to check the edge counter 712 storing the counts of estimated relative poses. When the count of estimated relative poses exceeds a preset threshold, e.g., NL, the pose map generator 214 can be configured to generate a pose map. In block 718, the loss calculator 216 can be configured to calculate the optimized loss for each of the side and rear cameras based on the pose map. The calibration results can also be sent to the calibration file management system 732.

[0084] The processes and methods described in the foregoing figures can be executed by processing logic, including hardware (e.g., circuits, special logic, etc.), firmware, software (e.g., software embodied in a non-transitory computer-readable medium), or a combination of both. Although the foregoing describes the processes or methods according to a certain sequence of operations, it should be understood that some of the operations described herein can be performed in a different order. Furthermore, some operations can be performed simultaneously rather than sequentially.

[0085] In the foregoing description, each embodiment of the invention has been described with reference to certain illustrative embodiments. Any of the components or devices described above can be implemented using hardware circuitry (e.g., application-specific integrated circuits (ASICs)). It is apparent that various modifications can be made to each embodiment without departing from the broader spirit and scope of the invention as set forth in the appended claims. Accordingly, the specification and drawings should be understood as illustrative rather than limiting. It is understood that the specific order or hierarchy of steps in the disclosed process is illustrative of the exemplary method. Based on design preferences, it is understood that the specific order or hierarchy of steps in the process can be rearranged. Furthermore, some steps can be combined or omitted. The appended method claims present elements of various steps in a sample order and are not intended to limit the specific order or hierarchy presented.

[0086] The foregoing description is intended to enable those skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein, but rather to be given the full scope consistent with the language of the claims, wherein the singular element referred to herein does not mean "one and only one," but rather "one or more," unless otherwise stated. Unless otherwise stated, the term "one" means one or more, and all structural and functional equivalents of the elements of the aspects described herein that are known to or subsequently known to those skilled in the art are expressly incorporated herein by reference and are covered by the claims. Furthermore, nothing disclosed herein is intended to be offered to the public, whether or not such disclosure is expressly mentioned in the claims. Unless explicitly stated using the phrase "means for," no claim element should be interpreted as a means plus a function.

[0087] Furthermore, the term "or" is meant to include, not exclude, "or." That is, unless otherwise specified or apparent from the context, the phrase "X employs A or B" refers to any natural inclusive permutation or combination. Specifically, the phrase "X employs A or B" is satisfied by any of the following: X employs A; X employs B; or X employs both A and B. Additionally, the terms "a" and "an" as used in this application and the appended claims should generally be understood as "one or more," unless otherwise specified or clearly indicated from the context as referring to the singular form.

Claims

1. A method for calibrating a camera on an autonomous driving system, comprising: Receive the current speed of the vehicle from the wheel odometer; Determine that the current speed of the vehicle is greater than a first preset speed threshold; Based on the determination that the current speed is greater than the first preset speed threshold, at least one front camera of the driving vehicle is calibrated according to the position of the vanishing point of the two lane lines. The current speed of the vehicle is determined to be less than a second preset speed threshold and greater than zero; as well as Based on the determination that the current speed is less than the second preset speed threshold and greater than zero, the multiple side cameras and rear cameras of the driving vehicle are calibrated according to the posture diagram including the relative postures of multiple side cameras and rear cameras. The calibration of the multiple side cameras and rear camera of the driving vehicle further includes: Determine the relative poses of the plurality of side cameras and rear cameras; Generate a pose map, the pose map including the relative poses of the plurality of side cameras and the rear camera; and The optimization loss for each of the plurality of side cameras and rear cameras is calculated based on the pose diagram; The method for calibrating cameras on an autonomous driving system further includes: prior to calibrating at least one front camera, determining that a first set of additional conditions is met, wherein the first set of additional conditions includes a wiper indicator showing no rain, an angular rate monitor showing a current angular rate of less than one degree per second, and the lane line being determined to be a straight line.

2. The method for calibrating a camera on an autonomous driving system according to claim 1, further comprising: Before calibrating the multiple side cameras and the rear camera, a second set of additional conditions is determined to be met, wherein the second set of additional conditions includes the wiper indicator showing no rain, the light sensor and beam indicator showing sufficient ambient light, and the angular rate monitor showing that the current angular rate is less than one degree per second.

3. The method for calibrating a camera on an autonomous driving system according to claim 1, characterized in that, The at least one front camera includes a front main camera and a front narrow camera mounted on the top of the vehicle, wherein the field of view of the front main camera is wider than that of the front narrow camera.

4. The method for calibrating a camera on an autonomous driving system according to claim 1, characterized in that, The calibration of at least one front camera further includes: The vanishing point is determined by calculating the intersection of the two lane lines on the image plane; and Based on the difference between the vanishing point and the principal point, the yaw direction and pitch direction of the at least one front camera are calculated respectively.

5. The method for calibrating a camera on an autonomous driving system according to claim 1, characterized in that, Determining the relative poses of the multiple side cameras and the rear camera further includes: Detect features of objects in images captured by the plurality of side cameras and rear cameras; Based on the similarity of the sorted data, feature points in the image are matched; and The relative pose is determined based on the matched feature points.

6. The method for calibrating a camera on an autonomous driving system according to claim 5, characterized in that, The features of the object were detected using the Scale Invariant Feature Transform (SIFT) algorithm.

7. The method for calibrating a camera on an autonomous driving system according to claim 5, characterized in that, The similarity of feature points in the image is sorted using the k-nearest neighbor algorithm (KNN).

8. The method for calibrating a camera on an autonomous driving system according to claim 5, characterized in that, The determination of relative pose further includes: calculating the basic matrix of two adjacent cameras, and decomposing the basic matrix into rotation matrix and translation matrix to form one of the relative poses.

9. A system for calibrating cameras on an autonomous driving system, comprising: Wheel odometer, configured to measure the current speed of the moving vehicle; A speed monitor is configured to determine that the current speed of the vehicle is greater than a first preset speed threshold. as well as A camera calibrator is configured to calibrate at least one front camera of the vehicle based on the position of the vanishing point of the two lane lines, after determining that the current speed is greater than the first preset speed threshold. The speed monitor is further configured to determine that the current speed of the vehicle is less than a second preset speed threshold and greater than zero; and The camera calibrator is further configured to calibrate multiple side cameras and a rear camera of the driving vehicle based on a pose diagram including the relative poses of multiple side cameras and a rear camera, based on determining that the current speed is less than the second preset speed threshold and greater than zero. The camera calibrator also includes: A pose determiner is configured to determine the relative poses of multiple side cameras and a rear camera; A pose graph generator is configured to generate a pose graph including the relative poses of the plurality of side cameras and a rear camera; and A loss calculator is configured to calculate the optimized loss of each of the plurality of side cameras and rear cameras based on the pose graph; Prior to calibrating at least one front camera, the camera calibrator is further configured to: The system determines that there is no rain based on signals from the wiper indicator. Based on the signal from the angular rate monitor, it is determined that the current angular rate is less than one degree per second; and Based on images captured by at least one front camera, it is determined that the lane lines are straight.

10. The system for calibrating a camera on an autonomous driving system according to claim 9, characterized in that, Prior to calibrating the multiple side cameras and the rear camera, the camera calibrator is further configured to: The system determines that there is no rain based on signals from the wiper indicator. The ambient light level is determined based on a light sensor and a beam indicator; and Based on the signal from the angular rate monitor, it is determined that the current angular rate is less than one degree per second.

11. The system for calibrating a camera on an autonomous driving system according to claim 9, characterized in that, The at least one front camera includes a front main camera and a front narrow camera mounted on the top of the vehicle, wherein the field of view of the front main camera is wider than that of the front narrow camera.

12. The system for calibrating a camera on an autonomous driving system according to claim 9, characterized in that, The camera calibrator includes: A vanishing point determiner is configured to calculate the intersection of the two lane lines on the image plane to determine the vanishing point; and A direction calculator is configured to calculate the yaw and pitch directions of the at least one front camera based on the difference between the vanishing point and the principal point.

13. The system for calibrating a camera on an autonomous driving system according to claim 9, characterized in that, The posture determiner further includes: A feature detector is configured to detect features of objects in images captured by the plurality of side cameras and the rear camera; A feature point matcher is configured to match feature points in the image based on ranked similarity; and A pose calculator is configured to determine the relative pose based on matched feature points.

14. The system for calibrating a camera on an autonomous driving system according to claim 13, characterized in that, The feature detector is configured to detect features of an object using the scale-invariant feature transformation algorithm SIFT.

15. The system for calibrating a camera on an autonomous driving system according to claim 13, characterized in that, The feature point matcher is further configured to rank the similarity of feature points in the image using the k-nearest neighbor algorithm (KNN).

16. The system for calibrating a camera on an autonomous driving system according to claim 13, characterized in that, The pose calculator is further configured to calculate the fundamental matrix of two adjacent cameras, and to decompose the fundamental matrix into rotation and translation matrices to form one of the relative poses.

Images