Calibration of LiDAR Sensors

Abstract

A method for calibrating a LiDAR sensor mounted on a vehicle includes positioning the vehicle at a distance from a target, the target including a planar reflector and features surrounding the reflector. The vehicle is positioned and oriented relative to the reflector such that an optical axis of the LiDAR sensor is nominally parallel to an optical axis of the reflector and the target is nominally located at a center of a field of view of the LiDAR sensor. The method also includes acquiring a three-dimensional image of the target using the LiDAR sensor, the three-dimensional image of the target including an image of the features of the target and a mirror image of the vehicle formed by the reflector. The method also includes determining a deviation from an expected alignment of the LiDAR sensor relative to the vehicle by analyzing the image of the features in the three-dimensional image of the target and the mirror image of the vehicle.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 915,563, filed on October 15, 2019, entitled “DYNAMIC CALIBRATION OF LASER IMAGING SENSOR,” U.S. Patent Application No. 17 / 069,727, filed on October 13, 2020, entitled “CALIBRATION OF LASER SENSOR,” and U.S. Patent Application No. 17 / 069,733, filed on October 13, 2020, entitled “DYNAMIC CALIBRATION OF LASER SENSOR,” the contents of which are incorporated by reference in their entirety. Background Art

[0003] Three-dimensional sensors can be applied to autonomous vehicles, drones, robots, security applications, etc. A LiDAR sensor is a three-dimensional sensor that can achieve high angular resolution suitable for such applications. A LiDAR sensor may include one or more laser sources for emitting laser pulses and one or more detectors for detecting reflected laser pulses. The LiDAR sensor measures the time it takes for each laser pulse to travel from the LiDAR sensor to an object within the sensor's field of view, then bounce off the object and return to the LiDAR sensor. The LiDAR sensor determines how far away an object is from the LiDAR sensor based on the time of flight of the laser pulse. By scanning across a scene, a three-dimensional image of the scene can be obtained.

[0004] For accurate measurements, the orientation of the optical axis of the LiDAR sensor may need to be calibrated relative to some mechanical reference points (such as mounting holes on the housing of the LiDAR sensor). In addition, when installed in a vehicle, the position and orientation of the LiDAR sensor may need to be calibrated relative to the vehicle. Such calibration can be performed, for example, at the manufacturer's factory. In the event of a collision or other mechanical disturbance to the LiDAR sensor, its calibration relative to the housing or vehicle may change. Therefore, it may be desirable to be able to detect a loss of calibration accuracy and correct the calibration, thereby ensuring safe and accurate long-term operation of the LiDAR sensor. Summary of the invention

[0005] According to some embodiments, a method for calibrating a LiDAR sensor mounted on a vehicle includes positioning the vehicle at a distance from a target. The target includes a planar reflector and features surrounding the reflector. The optical axis of the reflector is substantially horizontal. The vehicle is positioned and oriented relative to the reflector so that the optical axis of the LiDAR sensor is nominally parallel to the optical axis of the reflector and the target is nominally located at the center of the field of view of the LiDAR sensor. The method also includes acquiring a three-dimensional image of the target using the LiDAR sensor. The three-dimensional image of the target includes an image of the features of the target and a mirror image of the vehicle formed by the reflector. The method also includes determining a deviation from an expected alignment of the LiDAR sensor relative to the vehicle by analyzing the image of the features in the three-dimensional image of the target and the mirror image of the vehicle.

[0006] According to some embodiments, a method of calibrating a LiDAR sensor mounted on a vehicle includes storing a reference three-dimensional image acquired by the LiDAR sensor when the LiDAR sensor is in an expected alignment relative to the vehicle. The reference three-dimensional image includes a first image of a fixed feature on the vehicle. The method also includes acquiring, using the LiDAR sensor, a three-dimensional image including a second image of the fixed feature, and determining a deviation from the expected alignment of the LiDAR sensor relative to the vehicle by comparing the second image of the fixed feature in the three-dimensional image with the first image of the fixed feature in the reference three-dimensional image.

[0007] According to some embodiments, a method of calibrating a LiDAR sensor mounted on a vehicle includes: acquiring one or more three-dimensional images using the LiDAR sensor while the vehicle is traveling on a road having fixed road features. Each of the one or more three-dimensional images includes an image of the road feature. The method also includes analyzing a spatial relationship between an image of the road feature in the one or more three-dimensional images and an orientation of a field of view of the LiDAR sensor, and determining a deviation from an expected alignment of the LiDAR sensor relative to the vehicle based on the spatial relationship between the image of the road feature and the field of view of the LiDAR sensor. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 An exemplary LiDAR sensor for three-dimensional imaging is shown in accordance with some embodiments.

[0009] Figure 2A A LiDAR sensor mounted on a vehicle is schematically shown in accordance with some embodiments.

[0010] Figure 2B A vehicle coordinate system is shown in accordance with some embodiments.

[0011] Figure 2CA LiDAR coordinate system is shown in accordance with some embodiments.

[0012] Figure 3A and 3B A mounting mechanism that may be used to mount a LiDAR sensor on a vehicle is schematically shown according to some embodiments.

[0013] FIG. 4A to FIG. 4C An exemplary calibration setup for a LiDAR sensor mounted on a vehicle is shown in accordance with some embodiments.

[0014] FIG. 5A to FIG. 5C and FIG. 6A to FIG. 6D Schematic illustration of what a LiDAR sensor may see under various alignment conditions according to some embodiments.

[0015] Figure 7 A simplified flow chart is shown illustrating a method of calibrating a LiDAR sensor mounted on a vehicle using a target with an embedded reflector, according to some embodiments.

[0016] Figure 8 A LiDAR sensor is shown mounted behind a vehicle's front windshield including a visor, in accordance with some embodiments.

[0017] Fig.9A and 9B Schematically shows Figure 8 Side view of a windshield with a sun visor shown in FIG.

[0018] FIG. 10A to FIG. 10C Schematically illustrates a method according to some embodiments Figure 8 Some examples of the effect of the light shield on the effective field of view of the LiDAR sensor under various alignment conditions shown in FIG.

[0019] FIG. 10D to FIG. 10F Some exemplary images of certain features of a vehicle acquired by a LiDAR sensor according to some embodiments are schematically shown.

[0020] Fig.11 A simplified flow chart is shown illustrating a method of calibrating a LiDAR sensor mounted on a vehicle using features on the vehicle, according to some embodiments.

[0021] FIG. 12A to FIG. 12B and FIG. 13A to FIG. 13C A method of dynamically calibrating a LiDAR sensor mounted on a vehicle using lane markings on a road is shown in accordance with some embodiments.

[0022] Fig.14A simplified flow chart is shown illustrating a method of calibrating a LiDAR sensor mounted on a vehicle using road features according to some embodiments. DETAILED DESCRIPTION

[0023] According to some embodiments, a method of calibrating a LiDAR sensor mounted on a vehicle is provided. The calibration may not require the vehicle with the LiDAR sensor mounted thereon to be returned to the manufacturer's factory or repair shop. The calibration may be performed periodically or continuously while the vehicle is parked or even while driving.

[0024] Figure 1 An exemplary LiDAR sensor 100 for three-dimensional imaging according to some embodiments is shown. The LiDAR sensor 100 includes a transmitting lens 130 and a receiving lens 140. The LiDAR sensor 100 includes a light source 110a disposed substantially in a back focal plane of the transmitting lens 130. The light source 110a is operable to transmit light pulses 120 from corresponding emission positions in the back focal plane of the transmitting lens 130. The transmitting lens 130 is configured to collimate the light pulses 120 and direct them toward an object 150 located in front of the LiDAR sensor 100. For a given emission position of the light source 110a, the collimated light pulses 120′ are directed toward the object 150 at a corresponding angle.

[0025] A portion 122 of the collimated light pulse 120' is reflected from the object 150 toward the receiving lens 140. The receiving lens 140 is configured to focus the portion 122' of the light pulse reflected from the object 150 onto a corresponding detection position in a focal plane of the receiving lens 140. The LiDAR sensor 100 also includes a detector 160a disposed substantially at the focal plane of the receiving lens 140. The detector 160a is configured to receive and detect the portion 122' of the light pulse 120 reflected from the object at the corresponding detection position. The corresponding detection position of the detector 160a is optically conjugate with the corresponding emission position of the light source 110a.

[0026] The light pulse 120 may be short in duration, for example, 10 ns pulse width. The LiDAR sensor 100 also includes a processor 190 coupled to the light source 110a and the detector 160a. The processor 190 is configured to determine the time of flight (TOF) of the light pulse 120 from emission to detection. Since the light pulse 120 travels at the speed of light, the distance between the LiDAR sensor 100 and the object 150 can be determined based on the determined time of flight.

[0027] One way to scan the laser beam 120′ across the FOV is to move the light source 110a laterally relative to the emission lens 130 in the back focal plane of the emission lens 130. For example, the light source 110a can be raster scanned to multiple emission positions in the back focal plane of the emission lens 130, such as Figure 1 . The light source 110a may emit a plurality of light pulses at a plurality of emission positions. Each light pulse emitted at a corresponding emission position is collimated by the emission lens 130 and directed toward the object 150 at a corresponding angle, and impinges at a corresponding point on the surface of the object 150. Thus, when the light source 110a is raster scanned within a particular region in the rear focal plane of the emission lens 130, a corresponding object region on the object 150 is scanned. The detector 160a may be raster scanned to be positioned at a plurality of corresponding detection positions in the focal plane of the receiving lens 140, as shown in FIG. Figure 1 The scanning of the detector 160a is typically performed synchronously with the scanning of the light source 110a, so that the detector 160a and the light source 110a are always optically conjugate to each other at any given time.

[0028] By determining the flight time of each light pulse emitted at the corresponding emission position, the distance from the LiDAR sensor 100 to each corresponding point on the surface of the object 150 can be determined. In some embodiments, the processor 190 is coupled to a position encoder that detects the position of the light source 110a at each emission position. Based on the emission position, the angle of the collimated light pulse 120′ can be determined. The XY coordinates of the corresponding point on the surface of the object 150 can be determined based on the angle and distance to the LiDAR sensor 100. Therefore, a three-dimensional image of the object 150 can be constructed based on the measured distances from the LiDAR sensor 100 to each point on the surface of the object 150. In some embodiments, the three-dimensional image can be represented as a point cloud, that is, a collection of X, Y, and Z coordinates of points on the surface of the object 150.

[0029] In some embodiments, the intensity of the returning light pulse 122' is measured and used to adjust the power of subsequent light pulses from the same emission point to prevent detector saturation, improve eye safety, or reduce overall power consumption. The power of the light pulse can be changed by changing the duration of the light pulse, the voltage or current applied to the laser, or the charge stored in the capacitor used to power the laser. In the latter case, the charge stored in the capacitor can be changed by changing the charging time, charging voltage, or charging current of the capacitor. In some embodiments, the reflectivity determined by the intensity of the detected pulse can also be used to add another dimension to the image. For example, an image can contain X, Y, and Z coordinates as well as reflectivity (or brightness).

[0030] The angular field of view (AFOV) of the LiDAR sensor 100 can be estimated based on the scanning range of the light source 110a and the focal length of the transmitting lens 130 as:

[0031]

[0032] Wherein, h is the scanning range of the light source 110a along a particular direction, and f is the focal length of the transmitting lens 130. For a given scanning range h, a shorter focal length will produce a wider AFOV. For a given focal length f, a larger scanning range will produce a wider AFOV. In some embodiments, the LiDAR sensor 100 may include a plurality of light sources arranged in an array at a rear focal plane of the transmitting lens 130, so that a larger total AFOV can be achieved while keeping the scanning range of each individual light source relatively small. Therefore, the LiDAR sensor 100 may include a plurality of detectors arranged in an array at the focal plane of the receiving lens 140, each detector being conjugated to a corresponding light source. For example, as Figure 1 As shown in , the LiDAR sensor 100 may include a second light source 110b and a second detector 160b. In other embodiments, the LiDAR sensor 100 may include four light sources and four detectors, or eight light sources and eight detectors. In one embodiment, the LiDAR sensor 100 may include eight light sources arranged in a 4×2 array and eight detectors arranged in a 4×2 array, so that the AFOV of the LiDAR sensor 100 in the horizontal direction may be wider than its AFOV in the vertical direction. According to various embodiments, depending on the focal length of the transmitting lens, the scanning range of each light source, and the number of light sources, the total AFOV of the LiDAR sensor 100 may be in the range of about 5° to about 15°, or about 15° to about 45°, or about 45° to about 120°.

[0033] The light source 110a can be configured to emit light pulses in the ultraviolet, visible, or near infrared wavelength ranges. The energy of each light pulse can be on the order of microjoules, which is generally considered eye-safe for repetition rates in the KHz range. For light sources operating in wavelengths greater than about 1500 nanometers, the energy levels can be higher because the eye does not focus on these wavelengths. The detector 160a can include a silicon avalanche photodiode, a photomultiplier tube, a PIN diode, or other semiconductor sensor.

[0034] When a LiDAR sensor (such as Figure 1 When the LiDAR sensor 100 shown in FIG. 1 is used for obstacle detection in an autonomous vehicle, it may be necessary to accurately know the position and orientation of the LiDAR sensor relative to the vehicle in order to accurately determine the position of the obstacle relative to the vehicle. Figure 2AA LiDAR sensor 210 is schematically shown mounted on a vehicle 220. For example, the LiDAR sensor 210 may be mounted at the top center behind the windshield of the vehicle 220. The LiDAR sensor 210 may be characterized by an optical axis 250 (e.g., an optical axis of a transmitting lens or a receiving lens). The vehicle 220 may have a longitudinal axis 240. It may be advantageous to align the optical axis 250 of the LiDAR sensor 210 with the longitudinal axis 240 of the vehicle 220 so that the LiDAR sensor 210 is looking straight ahead in the direction the vehicle 220 is heading.

[0035] Assume that the LiDAR sensor 210 is initially calibrated according to this alignment condition. If the orientation of the LiDAR sensor is shifted (e.g., turned to the left) due to some mechanical disturbance, the optical axis 250 of the LiDAR sensor 210 is no longer aligned with the longitudinal axis 240 of the vehicle 220, as shown in FIG. Figure 2A . Misalignment may result in inaccurate measurement of the position of an obstacle (eg, person 260) relative to vehicle 220. Therefore, recalibration of LiDAR sensor 210 may be required.

[0036] Figure 2B A vehicle coordinate system is shown. The vehicle coordinate system can have three translational degrees of freedom, which can be represented, for example, by x, y, and z coordinates. The vehicle coordinate system can have three rotational degrees of freedom, which can be represented, for example, by roll, pitch, and yaw angles about the x-axis, y-axis, and z-axis, respectively. For example, the x-axis can be along the longitudinal axis of the vehicle 220; the z-axis can be along the vertical direction; and the y-axis can be along the lateral direction. The position and orientation of the LiDAR sensor 210 relative to the vehicle 220 can be characterized by (x, y, z, roll, pitch, yaw) coordinates in the vehicle coordinate system.

[0037] Figure 2C The LiDAR coordinate system is shown. The LiDAR coordinate system may also have three translational degrees of freedom (e.g., represented by X, Y, and Z coordinates) and three rotational degrees of freedom (e.g., represented by roll, pitch, and yaw angles around the X, Y, and Z axes, respectively). For example, the X-axis may be along the optical axis of the LiDAR sensor 210; the Z-axis may be along the nominal vertical direction; and the Y-axis may be along a direction orthogonal to the X-axis and the Z-axis.

[0038] The raw data of the point cloud acquired by the LiDAR sensor 210 may be in the LiDAR coordinate system. In order to determine the position of the obstacle relative to the vehicle 220, if the position and orientation of the LiDAR sensor 210 in the vehicle coordinate system are known, the point cloud data may be converted to the vehicle coordinate system. The conversion from the LiDAR coordinate system to the vehicle coordinate system may be referred to herein as calibration of the LiDAR sensor 210 relative to the vehicle 220.

[0039] According to some embodiments, the LiDAR sensor 220 may be mounted on the vehicle 220 such that the LiDAR sensor 210 is nominally aligned relative to the vehicle 220. For example, the LiDAR sensor 210 may be mounted on the vehicle 220 such that the X-axis of the LiDAR coordinate system (e.g., along the optical axis of the LiDAR sensor 210) is nominally aligned with the X-axis of the vehicle coordinate system (e.g., along the longitudinal axis of the vehicle); the Y-axis of the LiDAR coordinate system is nominally aligned with the Y-axis of the vehicle coordinate system; and the Z-axis of the LiDAR coordinate system is nominally aligned with the Z-axis of the vehicle coordinate system. Thus, the roll, pitch, and yaw angles in the vehicle coordinate system are all approximately zero. Calibration may be required to compensate for any residual deviation from the nominal alignment to achieve sufficient accuracy. For example, it may be desirable to calibrate the LiDAR sensor 210 to a translational accuracy of 2 cm along each of the X-axis, Y-axis, and Z-axis, and a rotational accuracy of 0.1 degrees for each of the roll, pitch, and yaw angles.

[0040] Figure 3A and Figure 3B Schematically illustrates a mounting mechanism that can be used to mount a LiDAR sensor on a vehicle according to some embodiments. Figure 3A As shown in , the LiDAR sensor can have an outer housing 310 with three mounting holes 320 (e.g., on the top surface of the outer housing 310). The bracket 330 can have three holes 340 that match the mounting holes 320 on the outer housing 310 so that the LiDAR sensor can be attached to the bracket 330 in a fixed orientation, such as Figure 3B Bracket 330 may be attached to the vehicle (eg, to the ceiling of the interior cabin) using similar mounting holes (not shown) for proper alignment.

[0041] The LiDAR sensor may be pre-calibrated at the manufacturer's factory to correct for any residual misalignment of the LiDAR sensor relative to the vehicle. During operation of the vehicle over time, the optical components of the LiDAR sensor may shift relative to the housing 310, or the housing 310 of the LiDAR sensor may shift relative to the mounting bracket 330 and / or the vehicle. This may occur, for example, due to tear and wear of the internal mechanisms of the LiDAR sensor, collisions or vibrations of the vehicle, misalignment of the tires, aging of the vehicle suspension, etc. Therefore, over time, the calibration may become inaccurate and a new calibration may be required. According to various embodiments, the LiDAR calibration may be performed periodically using a target with an embedded reflector, or the LiDAR calibration may be performed periodically or continuously using fixed features on the vehicle or fixed road features, as described in more detail below.

[0042] A. Use a target with an embedded reflector to calibrate the LiDAR sensor

[0043] FIG. 4A to FIG. 4C An exemplary calibration setup for a LiDAR sensor mounted on a vehicle using a target with an embedded reflector is shown in accordance with some embodiments. In this example, the LiDAR sensor 410 is shown mounted behind the front windshield of the vehicle 420. However, this is not required. For example, the LiDAR sensor 410 can be mounted at other locations on the vehicle 420, such as on the front bumper, on the rear windshield, on the rear bumper, etc. The target 440 is placed at a certain distance D in front of the LiDAR sensor 410. The target 440 includes a planar reflector 430 and an identifiable feature 490 surrounding the reflector 430.

[0044] Reference Figure 4A , target 440 may be mounted on a bracket (not shown) such that reflector 430 is nominally vertical. Thus, optical axis 432 of reflector 430, which is perpendicular to the surface of reflector 430, may be nominally horizontal. Vehicle 420 is positioned and oriented relative to reflector 430 such that optical axis 412 of the LiDAR sensor is nominally parallel to optical axis 432 of reflector 430, and target 440 is nominally located at the center of the field of view of LiDAR sensor 410. In the case where LiDAR sensor 410 is mounted behind the front windshield of vehicle 420 (e.g., as Figure 4A ), the vehicle 420 may be positioned such that the longitudinal axis 422 of the vehicle 420 is nominally parallel to the optical axis 432 of the reflector 430 .

[0045] refer to Figure 4B , the LiDAR sensor 410 may acquire a three-dimensional image of the target 440 by scanning across the target 440. The three-dimensional image of the target may include an image of features 490 surrounding the reflector 430 and a mirror image of the vehicle 420 formed by the reflector 430. For example, the LiDAR sensor 410 may emit a laser pulse 450 (or 450a) directed toward the reflector 430. The reflected laser pulse 450b may be directed toward certain parts of the vehicle (e.g., the license plate at the front bumper), which may then be reflected back toward the reflector 430 (at Figure 4B 450c in FIG. 4 and returns to the LiDAR sensor 410 (in FIG. Figure 4B Therefore, if Figure 4C As shown in , the LiDAR sensor 410 can effectively obtain a three-dimensional mirror image of the vehicle 420.

[0046] According to some embodiments, the distance D between target 440 and vehicle 420 can be made large enough to meet the desired accuracy. For example, distance D can be about 3m. The size of reflector 430 can be made large enough so that the entire vehicle 420 is visible, but this is not required. For example, the size of reflector 430 can be about 1m high and about 2m wide. The size of target 440 can be about 2m high and about 3m wide (for example, the edge of target 440 with feature 490 can be 0.5m wide).

[0047] Figure 5A Schematically illustrates what the LiDAR sensor 410 may see when the vehicle 420 is properly aligned relative to the target 440 (e.g., the longitudinal axis of the vehicle 420 is perpendicular to the surface of the reflector 430) and the LiDAR sensor 410 is properly aligned relative to the vehicle 420 (e.g., the optical axis of the LiDAR sensor 410 is parallel to the longitudinal axis of the vehicle 420). It is also assumed that the vehicle 420 is laterally centered relative to the width of the reflector 430 and the LiDAR sensor 410 is laterally centered relative to the vehicle 420. In this case, the mirror image 420′ of the vehicle 420 as seen by the LiDAR sensor 410 may appear symmetrical and may not have any roll, pitch, or yaw angles. Target 440 may also be symmetrical relative to the field of view 414 of the LiDAR sensor (e.g., the lateral margin d from the left edge of target 440 to the left boundary of field of view 414 is approximately the same as the lateral margin d from the right edge of target 440 to the right boundary of field of view 414) and may not have any roll, pitch, or yaw angles.

[0048] Figure 5B Schematically illustrates what the LiDAR sensor 410 may see when the LiDAR sensor 410 is properly aligned relative to the vehicle 420, but the vehicle 420 is misaligned relative to the target 440 (e.g., the longitudinal axis of the vehicle 420 has a finite yaw angle relative to the optical axis of the reflector 430). In this case, the target 440 is no longer centered in the LiDAR's field of view 414 (e.g., laterally offset to the right). The amount of the offset (e.g., the difference between the new margin D and the nominal margin d) can be related to the amount of misalignment. Therefore, the amount of misorientation (e.g., yaw error) of the vehicle 420 relative to the target 440 can be determined based on the offset. In addition, the mirror image 420′ of the vehicle 420 seen by the LiDAR sensor 410 also appears to have a finite yaw angle (note that in Figure 5B (The amount of rotation is slightly exaggerated in FIG. 4 ). Thus, LiDAR sensor 410 can also determine the amount of misorientation of vehicle 420 relative to target 440 by measuring the distance of certain features of vehicle 420. For example, Figure 5BAs shown in , the LiDAR sensor 410 can measure that one headlight (e.g., the left headlight) is farther away from the LiDAR sensor 410 than another headlight (e.g., the right headlight), thereby calculating the yaw angle of the vehicle 420.

[0049] Figure 5C Schematically illustrates what the LIDAR sensor 410 may see when the vehicle 420 is properly aligned relative to the target 440 (e.g., the longitudinal axis of the vehicle 420 is perpendicular to the surface of the reflector 430), but the LiDAR sensor 410 is not properly aligned relative to the vehicle 420 (e.g., the LiDAR sensor 410 is looking to the left instead of looking straight ahead). In this case, the mirror image 420′ of the vehicle 420 may appear symmetrical, but the target 440 may be offset laterally (e.g., to the right) relative to the LiDAR's field of view 414. The LiDAR sensor 410 may determine the amount of misorientation (e.g., yaw error) of the LiDAR sensor 410 based on the amount of offset (e.g., the difference between the new margin D and the nominal margin d).

[0050] Fig. 6A Schematically illustrates what a LiDAR sensor 410 may see when the LiDAR sensor 410 is properly aligned relative to the vehicle 420, but the vehicle 420 has a pitch error relative to the target 440. In this case, the target 440 is no longer centered in the LiDAR's field of view 414 (e.g., offset vertically downward). The amount of vertical offset may be related to the amount of pitch error. Thus, the amount of pitch error of the vehicle 420 relative to the target 440 may be determined based on the amount of vertical offset. Additionally, the mirror image 420′ of the vehicle 420 as seen by the LiDAR sensor 410 also appears to have a finite pitch angle (note that the vertical offset is not in the image above). Fig. 6A The amount of rotation is slightly exaggerated). Therefore, LiDAR sensor 410 can also determine the amount of pitch error of vehicle 420 relative to target 440 by measuring the distance of certain features of vehicle 420. For example, Fig. 6A As shown in , the LiDAR sensor 410 can measure the roof tilt of the vehicle 420, and the pitch error can be calculated based on the tilt amount.

[0051] Figure 6BSchematically illustrates what the LiDAR sensor 410 may see when the vehicle 420 is properly aligned relative to the target 440, but the LiDAR sensor 410 has a pitch error relative to the vehicle 420 (e.g., the LiDAR sensor 410 is looking upward instead of straight ahead). In this case, the mirror image 420′ of the vehicle 420 may appear to have no pitch angle, but the target 440 may be offset vertically (e.g., downward) relative to the LiDAR's field of view 414. The LiDAR sensor 410 may determine the amount of pitch error of the LiDAR sensor 410 based on the amount of vertical offset.

[0052] Figure 6C Schematically illustrates what a LiDAR sensor 410 may see when the LiDAR sensor 410 is properly aligned relative to a vehicle 420, but the vehicle 420 has a roll error relative to a target 440. In this case, a mirror image 420′ of the vehicle 420 as seen by the LiDAR sensor 410 appears to have no roll angle, but the target 440 appears to have a limited roll angle relative to the LiDAR's field of view 414. The amount of roll error of the vehicle 420 relative to the target 440 may be determined based on the amount of the roll angle of the target 440 relative to the LiDAR sensor's field of view 414.

[0053] Fig.6D Schematically illustrates what the LiDAR sensor 410 may see when the vehicle 420 is properly aligned relative to the target 440, but the LiDAR sensor 410 has a roll error relative to the vehicle 420. In this case, the mirror image 420′ of the vehicle 420 and the target 440 both appear to have a finite roll angle relative to the LiDAR's field of view 414. The LiDAR sensor 410 may determine the amount of the roll error of the LiDAR sensor 410 based on the amount of roll rotation.

[0054] Thus, the LiDAR sensor 410 (or a computing unit of the vehicle 420) can determine deviations from an expected alignment of the LiDAR sensor 410 relative to the vehicle 420 (e.g., an initial alignment performed at a manufacturer's factory) by analyzing a three-dimensional image acquired by the LiDAR sensor 410, the acquired three-dimensional image including an image of a feature 490 on a target and a mirror image 420′ of the vehicle 420. Deviations from the expected alignment may include yaw errors, roll errors, pitch errors, and translation errors (e.g., δx, δy, δz).

[0055] According to some embodiments, determining a deviation from an expected alignment of the LiDAR sensor 410 relative to the vehicle 420 may include the following steps. The position and orientation of the LiDAR sensor 410 relative to the target 440 may be determined based on an image of a feature 490 on the target 440. The position and orientation of the LiDAR sensor 410 relative to the mirror image 420′ of the vehicle 420 may be determined based on the mirror image 420′ of the vehicle. Then, a transformation from the LiDAR coordinate system to the vehicle coordinate system may be determined based on (i) the position and orientation of the LiDAR sensor 410 relative to the target 440, and (ii) the position and orientation of the LiDAR sensor 410 relative to the mirror image 420′ of the vehicle 420.

[0056] In some embodiments, the computing unit may store a reference image. For example, the reference image may be an image acquired by the LiDAR sensor 410 just after an initial alignment is performed at the manufacturer's factory, or may be a simulated image for an intended alignment. During recalibration, the computing unit may compare the three-dimensional image acquired by the LiDAR sensor 410 with the reference image and perform multivariate minimization (e.g., using gradient descent or other algorithms) to determine a transformation matrix so that the acquired three-dimensional image most closely matches the reference image. Deviations from the intended alignment (e.g., yaw error, roll error, pitch error, δx, δy, δz) may then be derived from the transformation matrix.

[0057] According to various embodiments, the following exemplary method may be used to determine the relationship between the LiDAR sensor 410 and the vehicle 420 in up to six degrees of freedom. Other methods and techniques may also be used by those skilled in the art. In the following description of the exemplary embodiments, the following symbols and terms will be used. t represents the matrix describing the relationship between the LiDAR sensor 410 and the target 440, C t represents a matrix describing the relationship between the vehicle 420 and the target 440, L C represents a matrix describing the relationship between the LiDAR sensor 410 and the vehicle 420 (used to correct any miscalibration of the LiDAR sensor 410), M represents a mirror transformation matrix, and L mC 4 represents a matrix describing the relationship of the LiDAR sensor 410 to the vehicle 420 , as seen by the LiDAR sensor 410 in the reflector 430 .

[0058] In some embodiments, the LiDAR sensor 410 can establish its positional relationship with the target features 490 around the reflector 430 by triangulating the distances to at least three target features 490 (e.g., similar to how a GPS receiver triangulates its position relative to GPS satellites). The rotational relationship of pitch, roll, and yaw can be determined by measuring the positions of at least three target features 490 (which can be the same target features for x, y, and z). Thus, a matrix L describing the relationship of the LiDAR sensor 410 to the target 440 can be established. t The matrix L t is a 4×4 matrix defining the x, y, and z positions as well as pitch, roll, and yaw.

[0059] Next, the relationship between vehicle 420 and target 440 can be established through a similar process: triangulate its position relative to target 440 by measuring the distance to certain features of vehicle 420 (as seen in reflector 430), and determine the pitch, roll, and yaw of vehicle 420 by measuring the position of features on vehicle 420 relative to reflector 430. Therefore, a matrix C describing the relationship between vehicle 420 and target 440 can be established. t Matrix C t is a 4×4 matrix that defines the x, y, and z positions as well as pitch, roll, and yaw. The relationship between the LiDAR sensor 410 and the vehicle 420 can then be given by to define.

[0060] According to some embodiments, LiDAR sensor 410 may determine its position relative to target 440 as described above to establish matrix L t The LiDAR sensor 410 may also determine the relationship of the mirror image of the vehicle 420′ to the LiDAR sensor 410, as seen through the reflector 430, to establish the matrix L mC The relationship between the LiDAR sensor 410 and the vehicle 420 can be determined by the following matrix multiplication:

[0061] L C =L t ·M(L t -1 )·L mC ·M.

[0062] According to some embodiments, the image of the vehicle 420′ seen in the reflector 430 and the image of the target 440 acquired by the LiDAR sensor 410 are compared to a stored reference image. The reference image can be a simulated image or an image taken during factory calibration. Then, a minimization technique (e.g., using a gradient descent algorithm) can be used to determine the transformation parameters (δx, δy, δz, pitch error, roll error, and yaw error) that minimize the difference between the currently acquired image and the stored reference image. In this process, the currently acquired image can be transformed to match the stored reference image, or the stored reference image can be transformed to match the currently acquired image. The transformation parameters can represent the difference between the current LIDAR position and the ideal (or factory calibrated) LiDAR position relative to the vehicle 420.

[0063] Once the relationship of the LiDAR 410 to the vehicle 420 is determined, any differences from that relationship and the current LiDAR calibration may be used to correct the LiDAR calibration.

[0064] In some embodiments, one or more distance sensors may be used to determine the position and orientation of vehicle 420 relative to target 440 (e.g., in a world coordinate system). For example, by placing two or three distance sensors in the road surface below vehicle 420, the pitch, roll, and z coordinates (height) of the vehicle may be determined. By using six distance sensors, all degrees of freedom (x, y, z, pitch, roll, and yaw) may be determined. The distance sensors may be ultrasonic sensors or laser sensors.

[0065] Figure 5A An example of using distance sensors to determine the position and orientation of vehicle 420 is shown. In this example, four distance sensors 590 can be positioned at appropriate locations around vehicle 420. For example, each distance sensor 590 can be located near a corresponding wheel of vehicle 420. The distance sensor 590 can send ultrasonic pulses or laser pulses to the wheel and measure the return pulses, thereby estimating the distance to the wheel. Therefore, the y coordinate and yaw angle of vehicle 420 can be determined. In some embodiments, only two distance sensors 590 on one side of the vehicle (driver's side or passenger side) may be required. Another distance sensor 592 can be positioned in front of vehicle 420 to determine the x coordinate of vehicle 420.

[0066] According to some embodiments, additional corrections or calibrations may be performed. For example, windshield deformation effects may be measured and corrected. Whenever a windshield is replaced, corrections for windshield deformation may be required.

[0067] Figure 7A simplified flow chart is shown illustrating a method 700 of calibrating a LIDAR sensor mounted on a vehicle using a target with an embedded reflector, according to some embodiments.

[0068] The method 700 includes, at 702, positioning a vehicle at a distance from a target. The target includes a planar reflector and features surrounding the reflector. The optical axis of the reflector is substantially horizontal. The vehicle is positioned and oriented relative to the reflector so that the optical axis of a LiDAR sensor is nominally parallel to the optical axis of the reflector and the target is nominally located at a center of a field of view of the LiDAR sensor.

[0069] The method 700 also includes acquiring a three-dimensional image of the target using the LiDAR sensor at 704. The three-dimensional image of the target includes an image of features of the target and a mirror image of the vehicle formed by the reflector.

[0070] The method 700 also includes, at 706 , determining a deviation from an expected alignment of the LiDAR sensor relative to the vehicle by analyzing an image of a feature in the three-dimensional image of the target and a mirror image of the vehicle.

[0071] In some embodiments, method 700 also includes, at 708 , recalibrating the LiDAR sensor relative to the vehicle based on the deviation from the expected alignment of the LiDAR sensor relative to the vehicle.

[0072] In some embodiments, method 700 also includes determining that the deviation from the expected alignment of the LiDAR sensor exceeds a threshold, and providing an alert in response to determining that the deviation from the expected alignment of the LiDAR sensor exceeds the threshold.

[0073] In some embodiments, the field of view of the LiDAR sensor is less than 180 degrees in the horizontal direction.

[0074] In some embodiments, determining a deviation from an expected alignment of a LiDAR sensor relative to a vehicle may include: determining a position and orientation of the LiDAR sensor relative to a target based on an image of a feature; determining a position and orientation of the LiDAR sensor relative to a mirror image of the vehicle based on a mirror image of the vehicle; and determining a transformation from a LiDAR coordinate system to a vehicle coordinate system based on (i) the position and orientation of the LiDAR sensor relative to the target, and (ii) the position and orientation of the LiDAR sensor relative to the mirror image of the vehicle.

[0075] In some embodiments, determining the deviation from an expected alignment of the LiDAR sensor relative to the vehicle may include: storing a reference matrix associated with an expected relationship between the LiDAR sensor and the vehicle; determining a matrix associated with a current relationship between the LiDAR sensor and the vehicle; and determining the deviation from the expected alignment of the LiDAR sensor relative to the vehicle by comparing the matrix to the reference matrix. In some embodiments, method 700 also includes recalibrating the LiDAR sensor relative to the vehicle based on the matrix associated with the current relationship between the LiDAR sensor and the vehicle.

[0076] It should be understood that Figure 7 The specific steps shown in provide a specific method for calibrating a LiDAR sensor according to some embodiments. According to alternative embodiments, other step sequences may also be performed. For example, alternative embodiments of the present invention may perform the above steps in a different order. In addition, Figure 7 The various steps shown in can include multiple sub-steps, and these sub-steps can be performed in various orders suitable for each step. In addition, according to specific applications, additional steps can be added and some steps can be removed. Those of ordinary skill in the art will recognize many variations, modifications and substitutions.

[0077] B. Using features on the vehicle to dynamically calibrate the LiDAR sensor

[0078] It may be desirable to have a method to periodically or continuously check the calibration of the LiDAR sensor relative to the vehicle while the vehicle is parked or even while driving. According to some embodiments, calibration of the LiDAR sensor may be performed using features on the vehicle so that calibration may be performed during normal operation of the vehicle. This method is referred to herein as dynamic calibration.

[0079] Figure 8 An example of dynamic calibration of a LiDAR sensor using a sun visor attached to a windshield is shown in accordance with some embodiments. As shown in the figure, the LiDAR sensor 810 ( Figure 8 The visor 820 is mounted behind the front windshield 840 of the vehicle 850 (not visible in the image). The visor 820 is attached to the windshield 840 in the area directly in front of the LiDAR sensor 810. Assuming that the operating wavelength of the LiDAR sensor 810 is in the infrared wavelength range, the visor 820 can be configured to block infrared light (e.g., opaque to infrared light). In some embodiments, the visor 820 can be annular in shape around the IR transparent portion 830 of the windshield 840. The geometry and size of the IR-blocking visor 820 can be made so that the perimeter around the edge of the field of view of the LiDAR sensor 810 is blocked.

[0080] Fig.9A and 9B Schematically illustrates a side view of a windshield 840 with an infrared (IR) blocking visor 820 in accordance with some embodiments. An infrared (IR) transparent portion 830 of the windshield 840 is surrounded by the IR blocking visor 820. The IR transparent portion 830 is slightly smaller than the field of view 910 of the LiDAR sensor 810. Fig.9A , in the correct alignment (e.g., the intended alignment), the LiDAR sensor 810 can be positioned behind the windshield 840 so that its field of view 910 is centered on the IR transparent portion 830. Thus, the edges of the field of view 910 are blocked on all four sides ( Fig.9A Side view showing the upper and lower sides).

[0081] Fig. 9B An example is shown when the LiDAR sensor 810 is offset from the correct alignment position (e.g., tilted upwards). As a result, the IR transparent portion 830 is no longer located in the center of the field of view 830 of the LiDAR sensor 810. Therefore, a larger portion of its field of view 910 is blocked on the upper side than on the lower side.

[0082] Figure 10A-10C Schematically illustrates a method according to some embodiments Figure 8 Some examples of the effect of the IR blocking mask 820 under various alignment conditions are shown in FIG. Fig. 10A An example of a properly aligned LiDAR sensor 810 is shown. The IR transparent portion 830 of the windshield 840 is located in the center of the field of view 910 of the LiDAR sensor 810. Therefore, the edge of the field of view 910 that is blocked by the IR blocking visor 820 (the gray area) has approximately the same width on all four sides.

[0083] Fig. 10B An example is shown in which the LiDAR sensor 810 is offset to the right from the correct alignment position (e.g., the LiDAR sensor 810 has a yaw error). As a result, the IR transparent portion 830 of the windshield 840 is offset to the left relative to the field of view 910 of the LiDAR sensor 810. Therefore, a larger portion of the field of view 910 is blocked on the right by the IR blocking visor 820 than on the left.

[0084] Fig. 10C An example is shown where the LiDAR sensor 810 is rotated from a correct alignment position (eg, the LiDAR sensor 810 has a roll error). As a result, the IR transparent portion 830 of the windshield 840 is rotated relative to the field of view 910 of the LiDAR sensor 810.

[0085] According to some embodiments, the relative position and orientation of the IR transparent portion 830 of the windshield 840 relative to the field of view 910 of the LiDAR sensor 810 can be used to calibrate the LiDAR sensor 810. For example, the computing unit can store a reference image. The reference image can be acquired by the LiDAR sensor 810 when the LiDAR sensor is in the correct alignment position, or can be obtained by simulation. For example, the reference image can be acquired by the LiDAR sensor just after the LiDAR sensor has been pre-calibrated in a manufacturing facility. When the LiDAR sensor 810 is in normal operation (when the vehicle is parked or driving), the computing unit can periodically or continuously compare the current LiDAR image with the reference image. Based on the comparison, a deviation from the correct alignment position of the LiDAR sensor 810 can be obtained.

[0086] In some embodiments, a multivariate minimization may be performed to determine a transformation matrix such that the IR transparent portion 830 of the windshield 840 in the current LiDAR image most closely matches that in the reference image. Deviations from the correct alignment position (e.g., yaw error, roll error, pitch error, δx, δy, and δz) may then be derived from the transformation matrix. According to various embodiments, the LiDAR sensor 810 may automatically recalibrate itself or provide an alert to the vehicle in response to determining that the deviation from the correct alignment position exceeds a threshold.

[0087] Additionally or alternatively, images of some fixed features on the vehicle acquired by the LiDAR sensor 810 may also be used to calibrate the LiDAR sensor 810 . Figure 10D-10F 100 and 101. Some exemplary images that may be acquired by the LiDAR sensor 810 are shown. These images show a portion of a hood 1020 of a vehicle and a decorative feature 1030 (e.g., a manufacturer's logo) attached to the hood 1020. Fig. 10D , decorative feature 1030 is approximately laterally centered in field of view 910 of LiDAR sensor 810. Fig.10E , the decorative feature 1030 is offset to the left of the field of view 910, indicating that the LiDAR sensor 810 may be turned to the right (e.g., has a yaw error). Fig.10F , the decorative feature 1030 is rotated, indicating that the LiDAR sensor 810 can be rotated (e.g., with a roll error). Note that Fig.10E and 10F The image of the transmitter hood 1020 in the image is also offset and / or rotated accordingly. Other exemplary features of the vehicle that can be used for calibration include a cover above a LiDAR sensor mounted in a grille, features in a headlight or taillight (e.g., if the LiDAR sensor is mounted inside a headlight or taillight), etc.

[0088] According to some embodiments, the computing unit may store a reference image acquired by the LiDAR sensor when the LiDAR sensor is in correct alignment (e.g., expected alignment) relative to the vehicle. The reference image may include a first image of a fixed feature on the vehicle. When the LiDAR sensor 810 is in normal operation (when the vehicle is parked or driving), the computing unit may periodically or continuously compare the current LiDAR image with the reference image. The current LiDAR image includes a second image of a fixed feature of the vehicle. Deviations from the correct alignment may be determined by comparing the position and orientation of the fixed feature in the second image with the position and orientation of the fixed feature in the first image.

[0089] Fig.11 A simplified flow chart is shown illustrating a method 1100 of calibrating a LiDAR sensor mounted on a vehicle using features on the vehicle, according to some embodiments.

[0090] The method 1100 includes, at 1102, storing a reference three-dimensional image acquired by a LiDAR sensor when the LiDAR sensor is in an expected alignment relative to a vehicle. The reference three-dimensional image includes a first image of a fixed feature on the vehicle.

[0091] The method 1100 also includes, at 1104 , acquiring a three-dimensional image including a second image of the fixed feature using the LiDAR sensor.

[0092] The method 1100 also includes, at 1106 , determining a deviation from an expected alignment of the LiDAR sensor relative to the vehicle by comparing the second image of the fixed feature in the three-dimensional image and the first image of the fixed feature in the reference three-dimensional image.

[0093] In some embodiments, method 1100 also includes, at 1108 , recalibrating the LiDAR sensor based on the deviation from an expected alignment of the LiDAR sensor relative to the vehicle.

[0094] In some embodiments, method 1100 further includes determining a transformation to be applied to the second image of the fixed feature in the three-dimensional image to match the first image of the fixed feature in the reference three-dimensional image, and recalibrating the LiDAR sensor based on the transformation.

[0095] In some embodiments, method 1100 further includes determining that the deviation from the expected alignment of the LiDAR sensor exceeds a threshold, and providing an alert in response to determining that the deviation from the expected alignment of the LiDAR sensor exceeds the threshold.

[0096] In some embodiments, the reference three-dimensional image is acquired by the LIDAR sensor after the LiDAR sensor has been pre-calibrated in a manufacturing facility.

[0097] In some embodiments, the fixed feature comprises a portion of a hood of the vehicle or an object attached to the hood.

[0098] In some embodiments, the LiDAR sensor is positioned behind a windshield of the vehicle, and the fixed feature includes a light shield attached to the windshield in an area directly in front of the LiDAR sensor. The light shield is configured to block light at an operating wavelength of the LiDAR sensor and is shaped to block a portion of the field of view of the LiDAR sensor. The light shield may have an outer boundary and an inner boundary, and the size of the inner boundary is such that the light shield encroaches on the perimeter of the field of view of the LiDAR sensor.

[0099] It should be understood that Fig.11 The specific steps shown in provide a specific method for calibrating a LiDAR sensor according to some embodiments. According to alternative embodiments, other step sequences may also be performed. For example, alternative embodiments of the present invention may perform the above steps in a different order. In addition, Fig.11 The various steps shown in can include multiple sub-steps, and these sub-steps can be performed in various orders suitable for each step. In addition, according to specific applications, additional steps can be added and some steps can be removed. Those of ordinary skill in the art will recognize many variations, modifications and substitutions.

[0100] C. Dynamic Calibration of LiDAR Sensors Using Road Features

[0101] According to some embodiments, a method of dynamic calibration of a LiDAR sensor mounted on a vehicle may use road features while the vehicle is traveling in a relatively straight section of the road. Fig. 12A and 12B An example is shown. The LiDAR sensor 1210 is shown as being mounted behind the front windshield of a vehicle 1220 (the LiDAR sensor 1210 may be mounted in other locations, such as on the front bumper). The vehicle 1220 is traveling on a straight section of a road 1230. Painted lane markings 1240 may be used to dynamically calibrate the position and orientation of the LiDAR sensor 1210 relative to the vehicle 1220 based on LiDAR images acquired while the vehicle is traveling along the road 1230. Calibration using painted lane markings may be advantageous because they are present on almost all roads. Additionally, the distance between a pair of lane markings 1240 is typically a standard distance. Lane markings with reflective paint may be clearly displayed in the LiDAR images.

[0102] like Fig. 12AAs shown in , if the LiDAR sensor 1210 is properly aligned relative to the vehicle 1220 (e.g., the LiDAR sensor 1210 is facing the same direction as the vehicle 1220's forward direction), a pair of lane markers 1240 may appear to pass equally on each side of the vehicle path 1250. Fig. 12B , if the LiDAR sensor 1210 is misaligned relative to the vehicle 1220 (e.g., the LiDAR sensor 1210 is looking to the left relative to the longitudinal axis of the vehicle 1220), then a pair of lane markers 1240 may appear to pass through the vehicle 1220 asymmetrically relative to the vehicle path 1250. For example, the vehicle path 1250 may appear to move closer to the lane marker 1240a on the driver's side than the lane marker 1240b on the passenger side. Therefore, by analyzing the LiDAR image of the lane marker 1240 relative to the vehicle path 1250, the amount of misalignment (e.g., yaw error) of the LiDAR sensor 1210 relative to the vehicle 1220 may be estimated.

[0103] Figure 13A-13C Some more examples of using lane markings for dynamic calibration of a LiDAR sensor mounted on a vehicle according to some embodiments are shown. Fig.13A As shown in , if the alignment of the LiDAR sensor 1210 has a pitch error, the lane marking 1240 may appear tilted relative to the vehicle path (assuming the road is relatively flat).

[0104] like Fig. 13B As shown in , if the alignment of the LiDAR sensor 1210 has a roll error, the lane marking 1240a on one side of the vehicle 1220 (e.g., the driver's side) may appear to be higher than the lane marking 1240b on the other side of the vehicle 1220 (e.g., the passenger side).

[0105] like Fig. 13C As shown in , if the alignment of LiDAR sensor 1210 has a Z error (in the vertical direction), the height of lane marker 1240 may deviate from the vehicle path. Translation errors in the other two orthogonal directions (e.g., X error and Y error) can also be detected by monitoring the motion of objects (e.g., lane markers) (relative to the expected motion) as vehicle 1220 moves forward.

[0106] Thus, by analyzing the spatial relationship between the image of the lane markings 1240 (or other road features) and the vehicle path, various rotational and translational misalignments of the LiDAR sensor 1210 relative to the vehicle 1220 can be detected and estimated. If the LiDAR sensor is mounted on the side or rear of the vehicle, a similar calibration process can be used, with appropriate modifications and mathematical transformations to account for the different perspectives.

[0107] According to some embodiments, the measurement may be repeated multiple times and the results may be averaged to account for, for example, road irregularities, poorly painted lane markings, curves, inclines, potholes, etc. Outliers may be discarded. Outliers may be caused by, for example, lane changes and other driving violations, obstructed view of lane markings by other vehicles, etc.

[0108] According to some embodiments, other information may be used to enhance the quality and reliability of the calibration data. For example, data from vehicle steering sensors, global navigation satellite system (e.g., GPS) data, and inertial measurement unit (IMU) data, etc., may be used to ensure that the vehicle is not turning. Map data may be used to select good road sections for calibration, where the road is straight, level, and lane markings are fresh and properly spaced. Other road features such as curbs, guardrails, and road signs may also be used as input to the calibration algorithm.

[0109] Fig.14 A simplified flow chart illustrating a method 1400 of calibrating a LiDAR sensor mounted on a vehicle using road features in accordance with some embodiments is shown.

[0110] The method 1400 includes, at 1402, acquiring one or more three-dimensional images using a LiDAR sensor while a vehicle is traveling on a road having fixed road features. Each of the one or more three-dimensional images includes an image of a road feature.

[0111] The method 1400 also includes, at 1404 , analyzing a spatial relationship between images of road features in the one or more three-dimensional images and an orientation of a field of view of the LiDAR sensor.

[0112] The method 1400 also includes, at 1406 , determining a deviation from an expected alignment of the LiDAR sensor relative to the vehicle based on a spatial relationship between the image of the road feature and the field of view of the LiDAR sensor.

[0113] In some embodiments, method 1400 also includes, at 1408 , recalibrating the LiDAR sensor based on the deviation from an expected alignment of the LiDAR sensor relative to the vehicle.

[0114] In some embodiments, method 1400 also includes determining that the deviation from the expected alignment of the LiDAR sensor exceeds a threshold, and providing an alert in response to determining that the deviation from the expected alignment of the LiDAR sensor exceeds the threshold.

[0115] In some embodiments, the road features include one or more pairs of lane markings on both sides of the vehicle. Analyzing the spatial relationship may include: determining a pitch angle between a pair of lane markings in the one or more pairs of lane markings and a field of view of the LiDAR sensor; and determining the deviation from the expected alignment of the LiDAR sensor may include: determining a pitch error of the LiDAR sensor based on the pitch angle. In some embodiments, the one or more pairs of lane markings may include a first lane marking on a driver's side of the vehicle and a second lane marking on a passenger side of the vehicle; analyzing the spatial relationship may include determining a height difference between the first lane marking and the second lane marking; and determining the deviation from the expected alignment of the LiDAR sensor may include determining a roll error of the LiDAR sensor based on the height difference.

[0116] In some embodiments, the one or more three-dimensional images may include: a plurality of three-dimensional images acquired by the LiDAR sensor over a time interval as the vehicle travels a distance interval on the road. The road may be substantially straight and level over a distance interval. In some embodiments, method 1400 further includes: determining a path of the vehicle over a distance interval, and comparing the path of the vehicle with the path of the road features from the plurality of three-dimensional images. In some embodiments, one or more pairs of lane markings include a first lane marking on the driver's side of the vehicle and a second lane marking on the passenger side of the vehicle; analyzing the spatial relationship may include determining an amount of lateral asymmetry between the first lane marking from the vehicle's path and the second lane marking from the vehicle's path; and determining the deviation from the expected alignment of the LiDAR sensor may include determining a yaw error of the LiDAR sensor based on the amount of lateral asymmetry. In some embodiments, analyzing the spatial relationship may include determining a height difference between a pair of lane markings and the path of the vehicle; and determining the deviation from the expected alignment of the LiDAR sensor may include determining a vertical error of the LiDAR sensor based on the height difference.

[0117] It should be understood that Fig.14 The specific steps shown in provide a specific method for calibrating a LiDAR sensor according to some embodiments. According to alternative embodiments, other step sequences may also be performed. For example, alternative embodiments of the present invention may perform the above steps in a different order. In addition, Fig.14 The various steps shown in can include multiple sub-steps, and these sub-steps can be performed in various orders suitable for each step. In addition, according to specific applications, additional steps can be added and some steps can be removed. Those of ordinary skill in the art will recognize many variations, modifications and substitutions.

[0118] It should also be understood that the examples and embodiments described herein are for illustrative purposes only, and that various modifications or changes to the examples and embodiments described herein will be suggested to those skilled in the art, and that various modifications or changes will be included within the spirit and scope of the present application and the scope of the appended claims.

Claims

1. A method for calibrating a LiDAR sensor mounted on a vehicle, the method comprising: Positioning the vehicle at a distance from a target, wherein: The target includes a plane reflector and features surrounding the plane reflector, the optical axis of the plane reflector being substantially horizontal; The vehicle is positioned and oriented relative to the plane mirror so that the optical axis of the LiDAR sensor is nominally parallel to the optical axis of the plane mirror and the target is nominally located at the center of the field of view of the LiDAR sensor; acquiring a three-dimensional image of the target using the LiDAR sensor, wherein the three-dimensional image of the target includes an image of the feature of the target and a mirror image of the vehicle formed by the plane mirror; determining a deviation from an expected alignment of the LiDAR sensor relative to the vehicle by analyzing an image of the feature in the three-dimensional image of the target and a mirror image of the vehicle; determining a position and orientation of the LiDAR sensor relative to the target based on the image of the feature; determining a position and orientation of the LiDAR sensor relative to the mirror image of the vehicle based on the mirror image of the vehicle; and A transformation from a LiDAR coordinate system to a vehicle coordinate system is determined based on (i) a position and orientation of the LiDAR sensor relative to the target and (ii) a position and orientation of the LiDAR sensor relative to the mirror image of the vehicle.

2. The method according to claim 1, further comprising: The LiDAR sensor is recalibrated relative to the vehicle based on the deviation from the expected alignment of the LiDAR sensor relative to the vehicle.

3. The method according to claim 1, further comprising: determining that the deviation from the expected alignment of the LiDAR sensor exceeds a threshold; as well as An alert is provided in response to determining that the deviation from the expected alignment of the LiDAR sensor exceeds the threshold.

4. The method according to claim 1, wherein: The field of view of the LiDAR sensor is less than 180 degrees in the horizontal direction.

5. The method according to claim 1, wherein: The LiDAR coordinate system has three translational degrees of freedom and three rotational degrees of freedom, and the vehicle coordinate system has three translational degrees of freedom and three rotational degrees of freedom.

6. The method according to claim 5, wherein: The three translational degrees of freedom of the LiDAR coordinate system are along three orthogonal axes, the three orthogonal axes include an x-axis, a y-axis, and a z-axis, and the three rotational degrees of freedom of the LiDAR coordinate system include a roll rotation about the x-axis, a pitch rotation about the y-axis, and a yaw rotation about the z-axis; and The three translational degrees of freedom of the vehicle coordinate system are along three orthogonal axes, including an X-axis, a Y-axis, and a Z-axis, and the three rotational degrees of freedom of the vehicle coordinate system include a roll rotation about the X-axis, a pitch rotation about the Y-axis, and a yaw rotation about the Z-axis.

7. The method according to claim 6, further comprising: A yaw angle of the vehicle relative to an optical axis of the planar reflector is determined using two or more range sensors positioned proximate the vehicle.

8. The method according to claim 7, further comprising: The two or more range sensors are used to determine a lateral position of the vehicle along an axis orthogonal to the optical axis and the vertical axis of the plane reflector.

9. The method according to claim 7, wherein: Each of the two or more distance sensors includes an ultrasonic sensor or a laser sensor.

10. The method according to claim 6, further comprising: Six range sensors are used to determine the yaw, roll, pitch and translational position of the vehicle along three orthogonal axes in the coordinate system of the plane mirror.

11. The method according to claim 10, wherein: Each of the six distance sensors includes an ultrasonic sensor or a laser sensor.

12. The method according to claim 1, wherein: Determining a deviation from the expected alignment of the LiDAR sensor relative to the vehicle includes: storing a reference matrix associated with an expected relationship between the LiDAR sensor and the vehicle; determining a matrix associated with a current relationship between the LiDAR sensor and the vehicle; and Deviations from the expected alignment of the LiDAR sensor relative to the vehicle are determined by comparing the matrix to the reference matrix.

13. The method according to claim 12, further comprising: The LiDAR sensor is recalibrated based on a matrix related to a current relationship between the LiDAR sensor and the vehicle.

Images