Lane-based automatic calibration of vehicle-mounted LIDAR

Abstract

A method of calibration for a vehicle includes: acquiring speed data from at least one vehicle controller of the vehicle while traveling; determining that the speed data meets a speed threshold; collecting a first plurality of LIDAR data frames and a second plurality of LIDAR data frames from a light detection and ranging (LiDAR) of the vehicle while the speed threshold is met; calculating a first yaw angle using the first plurality of LIDAR data frames; calculating a second yaw angle using the second plurality of LIDAR data frames; determining whether the first yaw angle and the second yaw angle meet a consistency criterion; determining a third yaw angle for the LiDAR using the first yaw angle and the second yaw angle in response to the consistency criterion being met; and calibrating the LiDAR using the third yaw angle.

Description

[Technical Field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application is a continuation of and claims priority to U.S. Non-provisional Patent Application No. 18 / 329,328, entitled "LANE-BASED AUTOMATIC CALIBRATION OF LIDAR ON A VEHICLE," filed June 5, 2023, which claims priority to U.S. Provisional Patent Application No. 63 / 480,755, entitled "LANE-BASED AUTOMATIC CALIBRATION OF LIDAR ON A VEHICLE," filed January 20, 2023, the disclosures of which are incorporated herein by reference in their entireties.

[0002] This application also claims priority to U.S. Provisional Patent Application No. 63 / 480,755, filed January 20, 2023, the disclosure of which is incorporated herein by reference in its entirety.

[0003] This paper relates to lane-based automatic calibration of vehicle-mounted light detection and ranging (LiDAR). [Background technology]

[0004] LiDAR sensors are currently used in many Advanced Driver Assisted Systems (ADAS) and Autonomous Driving (AD) technology features. As this technology advances, more and more vehicles are being equipped with these sensors, which use laser scanning technology to reconstruct the three-dimensional environment around the vehicle. However, LiDAR can become misaligned due to impacts, installation, and / or repair work. Summary of the Invention

[0005] In one aspect, a method of calibration for a vehicle includes: acquiring speed data from at least one vehicle controller of the vehicle while traveling; determining that the speed data meets a speed threshold; collecting a first plurality of LIDAR data frames and a second plurality of LIDAR data frames from a Light Detection and Ranging (LiDAR) of the vehicle while the speed threshold is met; calculating a first yaw angle using the first plurality of LIDAR data frames; calculating a second yaw angle using the second plurality of LIDAR data frames; determining whether the first yaw angle and the second yaw angle meet a consistency criterion; in response to the consistency criterion being met, determining a third yaw angle for the LiDAR using the first yaw angle and the second yaw angle; and calibrating the LiDAR using the third yaw angle.

[0006] Implementations may include any or all of the following features. The method further comprises acquiring vehicle yaw rate data from the at least one vehicle controller during the journey and determining that the vehicle yaw rate data meets a yaw rate threshold, wherein the first plurality of LIDAR data frames and the second plurality of LIDAR data frames are collected while the yaw rate threshold is also met. In response to at least one of the velocity threshold or the yaw rate threshold not being met, the method further comprises discarding LiDAR data. Collecting the first plurality of LIDAR data frames and the second plurality of LIDAR data frames comprises: extracting first ground points from each LiDAR data frame of the first plurality of LIDAR data frames; and extracting second ground points from each LiDAR data frame of the second plurality of LIDAR data frames; wherein the first ground points and the second ground points are used in calculating the first yaw angle and the second yaw angle. The step of extracting the first ground point from each LiDAR data frame of the first plurality of LiDAR data frames is performed for each frame of the first plurality of LiDAR data frames, and the step of extracting the second ground point from each LiDAR data frame of the second plurality of LiDAR data frames is performed for each frame of the second plurality of LiDAR data frames. The first ground point is extracted based on being within a region relative to the vehicle, and the second ground point is extracted based on being within the region relative to the vehicle. The first plurality of LiDAR data frames are collected during a first session, and the second plurality of LiDAR data frames are collected during a second session. Each of the first session and the second session includes the steps of: extracting lane-marking points based on light intensity; fitting a line to the lane-marking points; and evaluating the fit of the line to the lane-marking points. In response to the fit of the line to the lane marking points not meeting a criterion, the method further comprises discarding the current frame.The determination of whether the first yaw angle and the second yaw angle satisfy the consistency criterion is performed in response to having at least the first session and the second session of the first plurality of LIDAR data frames and the second plurality of LIDAR data frames, respectively. The evaluating the fit includes evaluating a standard deviation of point-to-line distances. The method further comprises determining whether a threshold number of frames have been accumulated in each session. In response to the consistency criterion not being satisfied, the method further comprises saving the session with a better fit of the line to the lane-marking points as a previous session. After saving the session with the better fit as the previous session, the method further comprises determining whether a threshold number of frames have been processed. In response to the threshold number of frames being processed, the method further comprises using the yaw angle from the previous session. The lane-marking points correspond to a curved road, and the line fitted to the lane-marking points is a curve. Determining the third yaw angle includes calculating an average of the first yaw angle and the second yaw angle. The third yaw angle is calculated repeatedly over time and used to calibrate the LiDAR according to a calibration schedule. The third yaw angle is calculated each time an event is detected by the vehicle and used to calibrate the LiDAR. The event includes an inertial measurement unit output meeting a criterion. [Brief explanation of the drawings]

[0007] [Figure 1] 1 illustrates an example method for lane-based auto-calibration of LiDAR, where the method is performed frame by frame.

[0008] [Figure 2] 1 illustrates an example method for lane-based auto-calibration of LiDAR, which is performed on multiple frames as part of a session.

[0009] [Figure 3] 1 illustrates an example method for lane-based auto-calibration of LiDAR, the method being performed over multiple sessions.

[0010] [Figure 4] 1 shows an example of LiDAR point cloud data before LiDAR calibration according to the present subject matter.

[0011] [Figure 5] 1 shows an example of LiDAR point cloud data after LiDAR calibration according to the present subject matter.

[0012] [Figure 6] An example of calculating LiDAR deviation angle for lane-based auto-calibration of LiDAR on curved roads is given.

[0013] [Figure 7] An example of a vehicle is shown.

[0014] [Figure 8] 1 illustrates an exemplary architecture of a computing device that may be used to implement aspects of the present disclosure.

[0015] Like reference symbols in the various drawings indicate like elements. DETAILED DESCRIPTION OF THE INVENTION

[0016] This document describes example systems and techniques for providing lane-based automatic calibration of vehicle-mounted LiDARs. Some implementations provide a solution for automatically calibrating vehicle-mounted LiDARs in real time by detecting lane-marking lines and accurately determining the LiDAR's misalignment. Automatic calibration can correct for misalignment errors. For example, if a target (e.g., another vehicle or another object on the road) is located approximately 100 meters from the vehicle, a misalignment of approximately 1 degree in the LiDAR's yaw angle can result in the ADAS / AD unintentionally placing the target in the wrong lane. The present subject matter can use a LiDAR point cloud accumulated from road lane markings over a relatively short period of time to fit a line to a left marking and / or a right marking. The yaw angle (defined by the LiDAR coordinate system) between the fitted line and the longitudinal axis of the LiDAR can be determined and used in automatically calibrating the LiDAR. Calibration can be performed based on data from the LiDAR alone (e.g., a camera signal may not be required) and does not require the use of physical targets such as checkerboards or road landmarks. Furthermore, unlike traditional approaches such as integrated closest point (ICP), processing may require relatively few computing resources. For example, in ICP, a LiDAR frame is selected and compared to the host vehicle's odometry to calculate a difference, which can be computationally intensive and / or can suffer from a lack of convergence if there are not many fixed vertical features available. Thus, the present subject matter can improve the safety of vehicles equipped with ADAS or AD technology.

[0017] Examples herein refer to vehicles. A vehicle is a machine that transports passengers, cargo, or both. A vehicle may have one or more motors that use at least one type of fuel or other energy source (e.g., electricity). Examples of vehicles include, but are not limited to, cars, trucks, and buses. The number of wheels may vary between vehicle types, and one or more (e.g., all) of the wheels may be used to propel the vehicle, or the vehicle may be unpowered (e.g., when a trailer is attached to another vehicle). A vehicle may include a passenger compartment that accommodates one or more people. At least one vehicle occupant may be considered the driver; in this case, various tools, implements, or other devices may be provided to the driver. In examples herein, any person carried by a vehicle may be referred to as the "driver" or "passenger" of the vehicle, regardless of whether that person is driving the vehicle, whether that person has access to the controls to drive the vehicle, or whether that person lacks the controls to drive the vehicle. The vehicles in this example are shown as being similar or identical to one another for illustrative purposes only.

[0018] Examples herein refer to ADAS. In some implementations, the ADAS may perform driver assistance and / or autonomous driving. The ADAS may at least partially automate one or more dynamic driving tasks. The ADAS may operate, in part, based on the output of one or more sensors typically positioned on, under, or within the vehicle. The ADAS may plan one or more trajectories for the vehicle before and / or while controlling the vehicle's motion. The planned trajectories may define a path for the vehicle to travel. Thus, propelling the vehicle according to the planned trajectories may correspond to controlling one or more aspects of the vehicle's operating behavior, such as, but not limited to, the vehicle's steering angle, gear (e.g., forward or reverse), speed, acceleration, and / or braking.

[0019] Although an autonomous vehicle is an example of an ADAS, not all ADAS are designed to provide fully autonomous vehicles. SAE International has defined multiple levels of driving automation, commonly referred to as Levels 0, 1, 2, 3, 4, and 5. For example, a Level 0 system or driving mode may not involve persistent vehicle control by the system. For example, a Level 1 system or driving mode may include adaptive cruise control, emergency brake assist, automatic emergency brake assist, lane keeping, and / or lane centering. For example, a Level 2 system or driving mode may include highway assist, autonomous obstacle avoidance, and / or autonomous parking. For example, a Level 3 or 4 system or driving mode may include incremental control of the vehicle by the driver assistance system. For example, a Level 5 system or driving mode may not require human intervention in the driver assistance system.

[0020] Examples herein refer to sensors. A sensor is configured to detect one or more aspects of its environment and output a signal reflective of the detection. The detected aspect may be static or dynamic at the time of detection. By way of illustrative example only, a sensor may indicate one or more of the following: a distance between the sensor and an object, a speed of a vehicle carrying the sensor, a trajectory of the vehicle, or an acceleration of the vehicle. A sensor may generate an output without probing its surroundings with anything (e.g., passive detection such as an image sensor capturing electromagnetic radiation), or the sensor may probe its surroundings (e.g., active detection by sending out electromagnetic radiation and / or sound waves) and detect a response to the probing. Examples of sensors that may be used in one or more embodiments include, but are not limited to, optical sensors (e.g., cameras); light-based detection systems (e.g., LiDAR devices); radio-based sensors (e.g., radar); acoustic sensors (e.g., ultrasonic devices and / or microphones); inertial measurement units (e.g., gyroscopes and / or accelerometers); speed sensors (e.g., for a vehicle or its components); position sensors (e.g., for a vehicle or its components); orientation sensors (e.g., for a vehicle or its components); torque sensors; thermal sensors; temperature sensors (e.g., primary or secondary thermometers); pressure sensors (e.g., for the ambient air or vehicle components); humidity sensors (e.g., rain detectors); or occupancy sensors.

[0021] Examples herein refer to lane markings. As used herein, lane markings include any feature that enables an ADAS to detect and recognize where a lane ends or begins in any direction. Lane markings include, but are not limited to, areas of a surface that visually contrast with another area of ​​the surface to indicate lane boundaries (e.g., by paint or other coloring material and / or different surface materials), Bott's dots, so-called turtles, so-called buttons, pavement markings, rumble strips, reflective markings, non-reflective markings, raised markings above the surface, recessed markings below the surface, and combinations thereof.

[0022] 1 illustrates an example method 100 for lane-based automatic calibration of a LiDAR, where the method is performed on a frame. Method 100 may be used with one or more other examples described elsewhere herein. More or fewer operations than those shown may be performed. Unless otherwise indicated, two or more operations may be performed in a different order.

[0023] Method 100 may involve processing a single frame of data from a LiDAR on a moving vehicle. The LiDAR may continuously generate an output, sometimes referred to as a point cloud, based on recording reflections of light emitted by the LiDAR. The point cloud represents, at each point in time, the surroundings near the vehicle as detected by the LiDAR, including the roadway, vehicles or other objects on the roadway, and roadside structures. Thus, method 100 may be performed repeatedly and substantially continuously while calibration is performed to process multiple frames of point cloud data from the LiDAR.

[0024] Method 100 may begin at operation 102. LiDAR data frames may be received at operation 104. For example, one or more processors of the vehicle (e.g., in one of the vehicle's controllers) may receive LiDAR data frames generated by the LiDAR.

[0025] In operation 106, one or more thresholds may be evaluated. In some implementations, speed data and / or vehicle yaw rate data may be obtained from at least one vehicle controller of the vehicle while traveling. The vehicle yaw rate is the change in vehicle yaw angle per time unit. The vehicle yaw angle is the heading angle of the vehicle relative to a world coordinate system. The speed data indicates the speed of the vehicle (e.g., based on sensor output indicating wheel speed). In some implementations, the speed data may be determined to satisfy a speed threshold. For example, method 100 is performed only if the vehicle is traveling at or faster than a threshold speed. In some implementations, the vehicle yaw rate data may be determined to satisfy a yaw angle threshold. For example, method 100 is performed only if the vehicle yaw rate is zero or substantially zero. If the evaluation in operation 106 results in a negative result, method 100 may perform operation 108. For example, the LiDAR point cloud data of the current frame acquired in operation 104 may be discarded. In some implementations, the condition regarding the vehicle yaw rate need not be satisfied. Thus, the expansion need not be limited to only straight sections of road, but can also work when the vehicle is traveling on curved roads.

[0026] If the evaluation at operation 106 results in a positive result, method 100 may perform extraction at operation 110. In some implementations, ground points may be extracted for accumulation. Points within a region relative to the vehicle may be extracted (e.g., as illustrated in FIGS. 4-5 described below). For example, the region may be a predefined, relatively small area ahead of the vehicle in the direction of travel. Extracting ground points may include obtaining LiDAR image information associated with spatial coordinates. Thus, method 100 may result in ground points being extracted from frames for one or more purposes in calibration.

[0027] Method 100 may end at operation 112. For example, method 100 may be subsequently performed again on a new frame of point cloud data from a LiDAR.

[0028] 2 illustrates an example method 200 for lane-based auto-calibration of a LiDAR, where the method 200 is performed for multiple frames as part of a session. Processing may be performed frame-by-frame. The method 200 may be used with one or more other examples described elsewhere herein. More or fewer operations than those shown may be performed. Unless otherwise indicated, two or more operations may be performed in a different order.

[0029] Method 200 may process LiDAR point cloud data that is part of a single calibration session, where the session is based on multiple frames of LiDAR point cloud data. Using multiple frames in a session may improve line fit performance. For example, because road markings detected by the LiDAR occur more sparsely at greater distances from the vehicle, accumulating frames may increase the number of valid LiDAR points. As another example, if the vehicle is maneuvered (e.g., by changing lanes) during a calibration session, or if lanes merge or split on the road, line fit quality may temporarily degrade, and the affected frames of LiDAR data may then be rejected without terminating the calibration process.

[0030] At operation 202, method 200 may begin. At operation 204, a LiDAR data frame may be processed. For example, a LiDAR data frame may be processed according to method 100 in FIG. 1 . At operation 206, point extraction from a subset of ground points (e.g., those within a predefined region near the vehicle) may be performed. In some implementations, the extraction is performed based on light intensity in the LiDAR point cloud. Because lane markings may have higher reflectivity than the roadway surface, lane marking points may be obtained from the LiDAR point cloud data by extracting points having at least a threshold intensity.

[0031] In operation 208, a line fit may be performed on the points extracted in operation 206. Any of several techniques for fitting a line to the points may be used. The resulting line may be mathematically defined using coordinates and represent an approximately linear configuration of lane markings. In some implementations, the line fit may be performed on the lane markings on the left side of the vehicle (sometimes referred to as the ego-left lane marking) and the lane markings on the right side of the vehicle (sometimes referred to as the ego-right lane marking). The quality of the lane fit indicates how well the line fits the extracted lane marking points.

[0032] In operation 210, lane fit quality may be considered. Any of several methods for characterizing the quality of the lane fit may be used. In some implementations, the average distance from each of the extracted lane marking points to the fitted line may be considered. For example, the standard deviation of the point-to-line distances may be used. If the result of the evaluation in operation 210 is negative, method 200 may perform operation 212 before returning to operation 204. For example, the LiDAR point cloud data of the current frame acquired in operation 204 may be discarded in operation 212.

[0033] If the evaluation at operation 210 results in a positive outcome, method 200 may determine, at operation 214, whether at least a threshold number of frames have been accumulated in method 200. The threshold may be 10 frames, or a number less than or greater than 10. In some implementations, yaw angle determination for the session should only be performed if a representative number of frames have been successfully accumulated. If the evaluation at operation 214 results in a negative outcome, method 200 may increment a frame counter at operation 216, wait for the next frame, and return to operation 204.

[0034] If the evaluation at operation 214 results in a positive outcome, method 200 may determine a lane-marking line yaw angle at operation 218. The yaw angle may be calculated for the current session using the accumulated LiDAR frames. The lane-marking line yaw angle may represent the tilt of the line fitted to the lane-marking points at operation 208 relative to the LiDAR coordinate system.

[0035] If the vehicle has a fitted curve because it is traveling on a curved road, the lane-marking line yaw angle can be calculated as the yaw angle of the tangent to the fitted curve at a starting point in the LiDAR coordinate system. An example is provided below in FIG. 6. The LiDAR misalignment angle can be calculated in operation 220, which can be performed before operation 218. The starting point is the point where a perpendicular to the vehicle longitudinal axis at the location of the LiDAR coordinate system intersects with the lane-marking line. If the lane-marking line yaw angle is non-zero, the LiDAR is misaligned. Such misalignment can be zeroed out in software by rotating everything from the LiDAR in the opposite direction of that angle. An assumption in such calculations can be that the vehicle is not skidding; i.e., the vehicle longitudinal axis is parallel to the tangent to the road.

[0036] Method 200 may end at operation 222. For example, method 200 may be subsequently performed again for a new session with new frames of point cloud data from the LiDAR.

[0037] 3 illustrates an example method 300 for lane-based automatic calibration of LiDAR, performed over multiple sessions. Method 300 may be used with one or more other examples described elsewhere herein. More or fewer operations than those shown may be performed. Unless otherwise indicated, two or more operations may be performed in a different order.

[0038] At operation 302, method 300 may be initiated. Execution of method 300 may be triggered in any of several ways. In some implementations, a calibration schedule may be established for the vehicle. For example, the calibration schedule may specify that calibration should be performed repeatedly at regular or irregular intervals (e.g., by performing methods 100, 200, and / or 300 one or more times). In some implementations, calibration may be triggered by event detection. For example, an output from an inertial measurement unit may indicate an event in response to the vehicle experiencing an impact (e.g., a minor collision such as a collision accident), and calibration may be performed in response to the event.

[0039] At operation 304, a session based on multiple LiDAR data frames may be processed. For example, each session may be processed according to method 200 in FIG.

[0040] At operation 306, it may be determined whether the system has already stored a previous session in addition to the session processed at operation 304. If the result of the evaluation at operation 306 is negative, method 300 may, at operation 308, save the current session (of operation 304) as the previous session and return to operation 304.

[0041] If the evaluation result at operation 306 is positive, then method 300 may determine, at operation 310, whether the results of the multiple sessions meet a consistency criterion. The number of sessions may be two or more. In some implementations, the evaluation involves the respective yaw angles calculated for the sessions and determines whether these values ​​match or disagree. For example, yaw angle values ​​may match if they are substantially the same or do not differ from each other by at least more than a threshold. If the evaluation result at operation 310 is negative, then method 300 may save at least one of the sessions as a previous session at operation 312. In some implementations, one of the sessions whose line fit is better (e.g., has higher quality) may be retained. For example, the session with a smaller standard deviation may be used. After operation 312, method 300 may determine, at operation 314, whether a threshold number of frames have been processed. If the evaluation at operation 314 results in a negative outcome, method 300 may return to operation 304 and process additional sessions. In some implementations, operation 314 may be used as a termination criterion for insufficient convergence of a session. For example, if method 300 has not reached convergence (as determined at operation 310) despite processing a threshold number of frames, method 300 may terminate processing of operations 304-314 and continue with operation 316, where the yaw angle for the previous session is applied. For example, this may be the session with the best line fit quality among multiple sessions.

[0042] If the evaluation in operation 310 results in a positive result, method 300 may determine a yaw angle for the LiDAR using the yaw angle of the session being considered in operation 318. In some implementations, two sessions (the current session and the previous session) are evaluated for consistency in operation 310, and if they match each other, they may be used to determine the yaw angle to be applied in the calibration in operation 318. For example, an average of the respective yaw angles may be calculated.

[0043] Method 300 may end at operation 320. The yaw angle determined in operation 318 or the resulting yaw angle from operation 316 may then be applied to calibrate the LiDAR. For example, the LiDAR point cloud data may be rotated by the determined yaw angle to correct for misalignment.

[0044] 4 illustrates an example of LiDAR point cloud data 400 before LiDAR calibration in accordance with the present subject matter. The LiDAR point cloud data 400 may be used with one or more other examples described elsewhere herein.

[0045] The LiDAR point cloud data 400, shown here in a two-dimensional coordinate system, represents the surroundings as seen from above, as detected by the LiDAR, of a vehicle positioned at location 402 and traveling north through the image. The region 404 may be defined relative to the vehicle coordinate system. In some implementations, the region 404 represents an area in the nearest vicinity of the vehicle of interest for performing LiDAR calibration. For example, the region 404 may extend a specified distance forward from the vehicle and another specified distance to either side of the vehicle (e.g., far enough to capture road lane markings). The region 404 may have a rectangular shape. The point cloud data 406 are examples of points included in the LiDAR point cloud data 400 and are visible in this example. For example, the point cloud data 406 originates from one frame generated by the LiDAR.

[0046] Here, point cloud data 408A-408B are seen to reside near location 402 and form the beginning of two substantially straight lines at the bottom of region 404. Point cloud data 408A-408B have higher intensity than other aspects of LiDAR point cloud data 400 within region 404, indicating the presence of lane markings on both sides of the vehicle. LiDAR can only detect lane markings within a relatively short distance in front of the vehicle. While point cloud data 406 may be from a single frame, point cloud data 408A-408B in this example is an accumulation from multiple frames (e.g., because only a relatively small number of lane markings are detected in any individual frame).

[0047] A line may be fitted to the point cloud data 408A-408B such that the line extends farther from the vehicle than the point cloud data 408A-408B. Here, line 410A is fitted to the lane-marking points of point cloud data 408A, and line 410B is fitted to the lane-marking points of point cloud data 408B. For example, this line fitting may be performed in operation 208 of method 200 in FIG. 2. Here, lines 410A-410B and point cloud data 408A-408B are misaligned with respect to the longitudinal axis of the vehicle. This may occur because the LiDAR was not properly aligned during installation, or because the LiDAR later moves out of its previous alignment. Therefore, the vehicle's LiDAR needs to be calibrated.

[0048] FIG. 5 shows an example of LiDAR point cloud data 400 after LiDAR calibration in accordance with the present subject matter. During calibration, a determined yaw angle (e.g., from operations 318 or 316 of method 300 in FIG. 3 ) may be applied to rotate LiDAR point cloud data 400. Thus, LiDAR point cloud data 400 (which looks different from that in FIG. 4 because another LiDAR frame is currently being processed) has now been rotated by the yaw angle, resulting in lines 410A-410B and point cloud data 408A-408B being aligned with the vehicle's longitudinal axis. Meanwhile, region 404, defined relative to the vehicle coordinate system, remains unchanged in this illustration. Thus, the present subject matter may provide automatic LiDAR calibration based solely on LiDAR data.

[0049] 6 shows an example of calculating LiDAR deviation angles for lane-based automatic calibration of a LiDAR 600 on a curved road 602. This example may be used with one or more other examples described elsewhere herein. The LiDAR 600 is mounted to the front of a vehicle 604 traveling on a curved road 602 having lane-marking lines 602A and 602B on each side of the vehicle 604. The lane-marking lines 602A-602B are curved lines in this example. Next, in operation 208, the curves may be fitted to the lane-marking lines 602A-602B (FIG. 2). Thus, the lane-marking points of the lane-marking lines 602A-602B may correspond to the curved road, and the curves may be fitted to the lane-marking points.

[0050] The LiDAR 600 in this example is assumed to be misaligned with respect to the vehicle 604. For example, the LiDAR coordinate system 606 of the LiDAR 600 has its axes misaligned with the respective longitudinal and lateral axes of the vehicle 604. Based on the LiDAR coordinate system 606, a line 608 may be defined. A starting point 610A may be defined based on the point where the line 608 intersects with the lane-marking line 602A; similarly, a starting point 610B may be defined based on the point where the line 608 intersects with the lane-marking line 602B. A tangent line 612A to the lane-marking line 602A may be defined to start at the starting point 610A; similarly, a tangent line 612B to the lane-marking line 602B may be defined to start at the starting point 610B. The angle of any of the tangent lines 612A-612B with respect to the LiDAR coordinate system 606 may be a LiDAR misalignment angle.

[0051] 7 illustrates an example of a vehicle 700. The vehicle 700 may be used with one or more other examples described elsewhere herein. The vehicle 700 includes an ADAS 702 and a vehicle control 704. The ADAS 702 may be implemented using some or all of the components described with reference to FIG. 8 below. The ADAS 702 includes sensors 706 and a planning algorithm 708. Other aspects that the vehicle 700 may include, including, but not limited to, other components of the vehicle 700 in which the ADAS 702 may be implemented, are omitted here for simplicity.

[0052] The sensor 706 is described herein as also including appropriate circuitry and / or executable programming for processing the sensor output and performing detection based on the processing. The sensor 706 may include a radar 710. In some implementations, the radar 710 may include any object detection system based at least in part on radio waves. For example, the radar 710 may be oriented in a forward direction relative to the vehicle and may be used to detect at least the distance to one or more other objects (e.g., another vehicle). The radar 710 may detect the presence of objects relative to the vehicle 700, thereby detecting the surroundings of the vehicle 700.

[0053] The sensors 706 may include an active optical sensor 712. In some implementations, the active optical sensor 712 may include any object detection system based at least in part on laser light. For example, the active optical sensor 712 may be oriented in any direction relative to the vehicle and may be used to detect at least the distance to one or more other objects (e.g., lane boundaries). The active optical sensor 712 may detect the presence of objects relative to the vehicle 700, thereby detecting the surrounding conditions of the vehicle 700. The active optical sensor 712 may be a scanning LiDAR or a non-scanning LiDAR (e.g., a flash LiDAR), to name just two examples.

[0054] Sensors 706 may include a camera 714. In some implementations, camera 714 may include any image sensor whose signals are considered by vehicle 700. For example, camera 714 may be oriented in any direction relative to the vehicle and may be used to detect vehicles, lanes, lane markings, curbs, and / or road signs. Camera 714 may detect the surroundings of vehicle 700 by visually recording the situation in relation to vehicle 700.

[0055] The sensors 706 may include an ultrasonic sensor 716. In some implementations, the ultrasonic sensor 716 may include any transmitter, receiver, and / or transceiver used in detecting at least the proximity of an object based on ultrasonic waves. For example, the ultrasonic sensor 716 may be positioned near the vehicle or on the exterior of the vehicle. The ultrasonic sensor 716 may detect the presence of an object in relation to the vehicle 700, thereby detecting the surroundings of the vehicle 700.

[0056] Regardless of whether ADAS 702 is controlling the movement of vehicle 700, any of sensors 706 alone, or two or more of sensors 706 collectively, may detect the surrounding conditions of vehicle 700. In some implementations, at least one of sensors 706 may generate an output that is considered in providing an alert or other prompt to the driver and / or in controlling the movement of vehicle 700. For example, the outputs of two or more sensors (e.g., the outputs of radar 710, active light sensor 712, and camera 714) may be combined. In some implementations, one or more other types of sensors may additionally or instead be included in sensors 706.

[0057] The planning algorithm 708 may plan for the ADAS 702 to perform one or more actions, or no action, in response to monitoring the vehicle's 700 surroundings and / or driver inputs. The output of one or more of the sensors 706 may be taken into account. In some implementations, the planning algorithm 708 may perform motion planning for the vehicle 700 and / or plan its trajectory.

[0058] Vehicle control 704 may include steering control 718. In some implementations, ADAS 702 and / or another driver of vehicle 700 controls the trajectory of vehicle 700 by adjusting the steering angle of at least one wheel by manipulating steering control 718. Steering control 718 may be configured to control the steering angle through a mechanical connection between steering control 718 and an adjustable wheel, or may be part of a steer-by-wire system.

[0059] Vehicle control 704 may include gear control 720. In some implementations, ADAS 702 and / or another operator of vehicle 700 uses gear control 720 to select among multiple operating modes of the vehicle (e.g., drive mode, neutral mode, or park mode). For example, gear control 720 may be used to control an automatic transmission in vehicle 700.

[0060] Vehicle control 704 may include traffic light control 722. In some implementations, traffic light control 722 may control one or more traffic lights that may be generated by vehicle 700. For example, traffic light control 722 may control the headlights, turn signals, and / or horn of vehicle 700.

[0061] Vehicle control 704 may include brake control 724. In some implementations, brake control 724 may control one or more types of braking systems designed to slow the vehicle, stop the vehicle, and / or keep the vehicle stationary when stopped. For example, brake control 724 may be actuated by ADAS 702. As another example, brake control 724 may be actuated by the driver using a brake pedal.

[0062] Vehicle control 704 may include a vehicle dynamics system 726. In some implementations, vehicle dynamics system 726 may control one or more functions of vehicle 700 in addition to, in the absence of, or instead of driver control. For example, when the vehicle is stopped on a hill, if the driver does not activate brake control 724 (e.g., by pressing the brake pedal), vehicle dynamics system 726 may hold the vehicle stationary.

[0063] Vehicle control 704 may include acceleration control 728. In some implementations, acceleration control 728 may control one or more types of propulsion motors of the vehicle. For example, acceleration control 728 may control an electric motor and / or an internal combustion motor of vehicle 700.

[0064] Vehicle controls may further include one or more additional controls, collectively shown herein as controls 730. Controls 730 may provide vehicle control of one or more functions or components. In some implementations, controls 730 may adjust one or more sensors of vehicle 700. For example, vehicle 700 may adjust sensor settings (e.g., frame rate and / or resolution) based on ambient condition data measured by the sensors of vehicle 700 and / or any other sensors.

[0065] Vehicle 700 may include a user interface 732. User interface 732 may include an audio interface 734 that may be used to generate alerts regarding detection. In some implementations, audio interface 734 may include one or more speakers positioned within the passenger compartment. For example, audio interface 734 may operate, at least in part, with an infotainment system within the vehicle.

[0066] User interface 732 may include a visual interface 736 that may be used to generate an alert regarding the detection. In some implementations, visual interface 736 may include at least one display device in the passenger compartment of vehicle 700. For example, visual interface 736 may include a touchscreen device and / or an instrument cluster display.

[0067] FIG. 8 illustrates an example architecture of a computing device 800 that may be used to implement aspects of the present disclosure, including any of the systems, devices, and / or techniques described herein or any other systems, devices, and / or techniques that may be utilized in various possible embodiments.

[0068] The computing device illustrated in FIG. 8 may be used to execute the operating system, application programs, and / or software modules (including software engines) described herein.

[0069] In some embodiments, computing device 800 includes at least one processing device 802 (e.g., processor), such as a central processing unit (CPU). Various processing devices are available from various manufacturers, such as Intel or Advanced Micro Devices. In this example, computing device 800 also includes a system memory 804 and a system bus 806 that couples various system components including the system memory 804 to the processing device 802. The system bus 806 is one of any number of types of bus structures that can be used, including, but not limited to, a memory bus or memory controller; a peripheral bus; and a local bus, using any of a variety of bus architectures.

[0070] Examples of computing devices that may be implemented using computing device 800 include a desktop computer, a laptop computer, a tablet computer, a mobile computing device (such as a smartphone, touchpad mobile digital device, or other mobile device), or other device configured to process digital instructions.

[0071] The system memory 804 includes a read-only memory 808 and a random access memory 810. A basic input / output system 812, containing the basic routines that act to transfer information within the computing device 800, such as during start-up, may be stored in the read-only memory 808.

[0072] In some embodiments, computing device 800 also includes a secondary storage device 814, such as a hard disk drive for storing digital data. The secondary storage device 814 is connected to system bus 806 by a secondary storage interface 816. The secondary storage device 814 and its associated computer-readable media provide non-volatile, non-transitory storage of computer-readable instructions (including application programs and program modules), data structures, and other data for computing device 800.

[0073] Although the exemplary environment described herein employs a hard disk drive as the secondary storage device, other types of computer-readable storage media are used in other embodiments. Examples of these other types of computer-readable storage media include a magnetic cassette, a flash memory card, a solid-state drive (SSD), a digital video disk, a Bernoulli cartridge, a compact disk read-only memory, a digital versatile disk read-only memory, a random access memory, or a read-only memory. Some embodiments include non-transitory media. For example, a computer program product may be tangibly embodied in a non-transitory storage medium. Additionally, such computer-readable storage media may include local storage or cloud-based storage.

[0074] A number of program modules may be stored in secondary storage device 814 and / or system memory 804, including an operating system 818, one or more application programs 820, other program modules 822 (such as the software engines described herein), and program data 824. Computing device 800 may utilize any suitable operating system.

[0075] In some embodiments, a user provides input to the computing device 800 through one or more input devices 826. Examples of input devices 826 include a keyboard 828, a mouse 830, a microphone 832 (e.g., for voice and / or other audio input), a touch sensor 834 (e.g., a touchpad or touch-sensitive display), and a gesture sensor 835 (e.g., for gesture input). In some implementations, the input devices 826 provide presence, proximity, and / or motion-based detection. Other embodiments include other input devices 826. The input devices may be connected to the processing device 802 through an input / output interface 836 that is coupled to the system bus 806. These input devices 826 may be connected by any number of input / output interfaces, such as a parallel port, a serial port, a game port, or a universal serial bus. In some possible embodiments, wireless communication between the input device 826 and the input / output interface 836 is also possible, including infrared, BLUETOOTH® wireless technology, 802.11a / b / g / n, cellular, ultra-wideband (UWB), ZigBee®, or other radio frequency communication systems, to name just a few.

[0076] In this exemplary embodiment, a display device 838, such as a monitor, liquid crystal display device, light emitting diode display device, projector, or touch-sensitive display device, is also connected to system bus 806 via an interface, such as a video adapter 840. In addition to the display device 838, computing device 800 may also include various other peripheral devices (not shown), such as speakers or a printer.

[0077] Computing device 800 may be connected to one or more networks through network interface 842. Network interface 842 may provide wired and / or wireless communication. In some implementations, network interface 842 may include one or more antennas for transmitting and / or receiving wireless signals. When used in a local area networking environment or a wide area networking environment (such as the Internet), network interface 842 may include an Ethernet interface. Other possible embodiments use other communication devices. For example, some embodiments of computing device 800 include a modem for communicating over a network.

[0078] Computing device 800 may include at least some form of computer-readable media. Computer-readable media includes any available media that can be accessed by computing device 800. By way of example, computer-readable media include computer-readable storage media and computer-readable communication media.

[0079] Computer-readable storage media include volatile and nonvolatile, removable and non-removable media implemented in any device configured to store information such as computer-readable instructions, data structures, program modules, or other data, including, but not limited to, random access memory, read-only memory, electrically erasable programmable read-only memory, flash memory or other memory technology, compact disc read-only memory, digital versatile disk or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by computing device 800.

[0080] Computer-readable communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term "modulated data signal" refers to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, computer-readable communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency, infrared and other wireless media. Combinations of any of the above are also included within the scope of computer-readable media.

[0081] The computing device illustrated in FIG. 8 is also an example of a programmable electronic apparatus that may include one or more such computing devices, and when multiple computing devices are included, such computing devices may be coupled together via a suitable data communications network to collectively perform various functions, methods, or operations disclosed herein.

[0082] In some implementations, computing device 800 may be characterized as an ADAS computer. For example, computing device 800 may include one or more components potentially used to process tasks arising in the field of artificial intelligence (AI). Computing device 800, in turn, includes sufficient processing power and the necessary support architecture for the demands of ADAS or AI in general. For example, processing device 802 may include a multi-core architecture. As another example, computing device 800 may include one or more coprocessors in addition to or as part of processing device 802. In some implementations, at least one hardware accelerator may be coupled to system bus 806. For example, a graphics processing unit may be used. In some implementations, computing device 800 may implement neural network-specific hardware to handle one or more ADAS tasks.

[0083] As used throughout this specification, the terms "substantially" and "about" are used to describe and take into account small variations, such as those due to processing variations. For example, they can refer to less than or equal to ±5%, such as less than or equal to ±2%, such as less than or equal to ±1%, such as less than or equal to ±0.5%, such as less than or equal to ±0.2%, such as less than or equal to ±0.1%, such as less than or equal to ±0.05%. Also, as used herein, indefinite articles such as "a" or "an" mean "at least one."

[0084] It should be understood that all combinations of the above concepts, and additional concepts discussed in more detail below, (provided that such concepts are not mutually inconsistent) are contemplated as being part of the inventive subject matter disclosed herein. In particular, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the inventive subject matter disclosed herein.

[0085] Although several implementations have been described, it will nevertheless be understood that various modifications may be made without departing from the spirit and scope of the specification.

[0086] Additionally, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. Additionally, other processes may be provided or processes may be eliminated from the described flows, and other components may be added to or removed from the described systems. Accordingly, other implementations are within the scope of the following claims.

[0087] While certain features of the described implementations have been shown and described herein, many modifications, substitutions, changes, and equivalents will now occur to those skilled in the art. It is therefore to be understood that the appended claims are intended to cover all such modifications and variations that fall within the scope of these implementations. They have been presented by way of example only, and not limitation, and it should be understood that various changes in form and detail may be made. Except for mutually exclusive combinations, any portion of the apparatus and / or methods described herein may be combined in any combination. The implementations described herein may include various combinations and / or subcombinations of the functions, components, and / or features of the different implementations described.

Claims

1. 1. A method of calibration for a vehicle, comprising: acquiring speed data from at least one vehicle controller of said vehicle while traveling; determining that the velocity data meets a velocity threshold; collecting a first plurality of frames of LIDAR data and a second plurality of frames of LIDAR data from a Light Detection and Ranging (LiDAR) of the vehicle while the speed threshold is met; calculating a first yaw angle using the first plurality of LIDAR data frames; calculating a second yaw angle using the second plurality of LIDAR data frames; determining whether the first yaw angle and the second yaw angle meet a consistency criterion; determining a third yaw angle for the LiDAR using the first yaw angle and the second yaw angle in response to the consistency criterion being satisfied; and calibrating the LiDAR using the third yaw angle. A method for providing the above.

2. 10. The method of claim 1, further comprising: acquiring vehicle yaw rate data from the at least one vehicle controller during the journey; and determining that the vehicle yaw rate data meets a yaw rate threshold, wherein the first plurality of LIDAR data frames and the second plurality of LIDAR data frames are collected while the yaw rate threshold is also met.

3. 3. The method of claim 2, wherein in response to at least one of the velocity threshold or the yaw rate threshold not being met, the method further comprises discarding LiDAR data.

4. Collecting the first plurality of LIDAR data frames and the second plurality of LIDAR data frames includes: extracting a first ground point from each LiDAR data frame of the first plurality of LiDAR data frames; and extracting second ground points from each LiDAR data frame of the second plurality of LiDAR data frames; wherein the first ground point and the second ground point are used when calculating the first yaw angle and the second yaw angle.

4. The method of claim 3, comprising:

5. 5. The method of claim 4, wherein extracting the first ground points from each LiDAR data frame of the first plurality of LIDAR data frames is performed for each frame of the first plurality of LIDAR data frames, and extracting the second ground points from each LiDAR data frame of the second plurality of LIDAR data frames is performed for each frame of the second plurality of LIDAR data frames.

6. The method of claim 4 , wherein the first ground point is extracted based on being within a region relative to the vehicle, and the second ground point is extracted based on being within the region relative to the vehicle.

7. 7. The method of claim 1, wherein the first plurality of LIDAR data frames is collected during a first session and the second plurality of LIDAR data frames is collected during a second session.

8. Each of the first session and the second session: extracting lane marking points based on light intensity; fitting a line to the lane marking points; and evaluating the fit of said line to said lane marking points.

8. The method of claim 7, comprising:

9. The method of claim 8 , wherein in response to the fit of the line to the lane marking points not meeting a criterion, the method further comprises discarding the current frame.

10. 9. The method of claim 8, wherein the determining whether the first yaw angle and the second yaw angle meet the consistency criterion is performed in response to having at least the first session and the second session of the first plurality of LIDAR data frames and the second plurality of LIDAR data frames, respectively.

11. The method of claim 8 , wherein assessing the goodness of fit comprises assessing a standard deviation of the distances from the points to the lines.

12. The method of claim 8 , further comprising determining whether a threshold number of frames have been accumulated in each session.

13. 9. The method of claim 8, wherein in response to the consistency criterion not being met, the method further comprises saving the session having a better fit of the line to the lane marking points as a previous session.

14. The method of claim 13 , after saving the session with the better fit as the previous session, the method further comprises determining whether a threshold number of frames have been processed.

15. The method of claim 14 , responsive to the threshold number of frames being processed, the method further comprising using a yaw angle from the previous session.

16. The method of claim 8 , wherein the lane marking points correspond to a curved road and the line fitted to the lane marking points is a curve.

17. The method of claim 1 , wherein determining the third yaw angle comprises calculating an average of the first yaw angle and the second yaw angle.

18. 7. The method of claim 1, wherein the third yaw angle is calculated repeatedly over time and used in calibrating the LiDAR according to a calibration schedule.

19. The method of claim 1 , wherein the third yaw angle is calculated each time an event is detected by the vehicle and is used in calibrating the LiDAR.

20. The method of claim 19 , wherein the event comprises an output of an inertial measurement unit meeting a criterion.

Images