Calibration of lidar sensors

Abstract

To provide a method of calibrating a LiDAR sensor mounted on a vehicle, the method including positioning the vehicle at a distance from a target including a planar mirror and features surrounding the mirror.SOLUTION: A vehicle is positioned and oriented relative to a mirror so that an optical axis of the LiDAR sensor is nominally parallel to an optical axis of the mirror and so that the target is nominally centered at a field of view of the LiDAR sensor. The method further includes acquiring, using the LiDAR sensor, a three-dimensional image of the target including images of the features of the target and a mirror image of the vehicle formed by the mirror. The method further includes determining a deviation from an expected alignment of the LiDAR sensor with respect to the vehicle by analyzing the images of the features and the mirror image of the vehicle in the three-dimensional image of the target.SELECTED DRAWING: Figure 4A

Description

[Technical field]

[0001]

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 915,563, entitled "DYNAMIC CALIBRATION OF LIDAR IMAGING SENSORS," filed on October 15, 2019, U.S. Patent Application No. 17 / 069,727, entitled "CALIBRATION OF LIDAR SENSORS," filed on October 13, 2020, and U.S. Patent Application No. 17 / 069,733, entitled "DYNAMIC CALIBRATION OF LIDAR SENSORS," filed on October 13, 2020, the contents of which are incorporated by reference in their entireties. [Background technology]

[0002]

[0002] Three-dimensional sensors can be applied in autonomous vehicles, drones, robotics, security applications, and the like. Lidar (LiDAR) sensors are a type of three-dimensional sensor that can achieve high angular resolution suitable for such applications. Lidar sensors can include one or more laser sources for emitting laser pulses and one or more detectors for detecting the reflected laser pulses. Lidar sensors measure the time it takes each laser pulse to travel from the Lidar sensor to an object in the sensor's field of view and then bounce off the object and return to the Lidar sensor. Based on the time of flight of the laser pulses, the Lidar sensor determines how far the object is from the Lidar sensor. By scanning across the scene, a three-dimensional image of the scene can be acquired.

[0003]

[0003] For accurate measurements, the orientation of the optical axis of the lidar sensor may need to be calibrated relative to some mechanical reference point, such as a mounting hole on the lidar sensor's case. In addition, when mounted on a vehicle, the position and orientation of the lidar sensor may need to be calibrated relative to the vehicle. Such calibration may be performed, for example, at the manufacturer's plant. In the event of a crash or other mechanical disturbance to the lidar sensor, the calibration relative to either the case or the vehicle may change. Thus, to ensure safe and accurate long-term operation of the lidar sensor, it may be desirable to be able to detect loss of calibration accuracy and correct the calibration. Summary of the Invention

[0004] According to some embodiments, a method for calibrating a lidar sensor mounted on a vehicle includes positioning the vehicle away from a target. The target includes a planar mirror and features surrounding the mirror. An optical axis of the mirror is substantially horizontal. The vehicle is positioned and oriented relative to the mirror such that the optical axis of the lidar sensor is nominally parallel to the optical axis of the mirror and the target is nominally centered in the field of view of the lidar sensor. The method further includes acquiring a three-dimensional image of the target using the lidar sensor. The three-dimensional image of the target includes images of the features of the target and a mirror image of the vehicle formed by the mirror. The method further includes determining a deviation from an expected alignment of the lidar sensor with respect to the vehicle by analyzing the images of the features in the three-dimensional image of the target and the mirror image of the vehicle.

[0005] According to some embodiments, a method for calibrating a lidar sensor mounted on a vehicle includes storing a reference three-dimensional image acquired by the lidar sensor while the lidar sensor is in an expected alignment with respect to the vehicle. The reference three-dimensional image includes a first image of a fixed feature on the vehicle. The method further includes acquiring a three-dimensional image using the lidar sensor, the three-dimensional image including a second image of the fixed feature, and determining a deviation from the expected alignment of the lidar sensor with respect to the vehicle by comparing the second image of the fixed feature in the three-dimensional image to the first image of the fixed feature in the reference three-dimensional image.

[0006] According to some embodiments, a method for calibrating a lidar sensor mounted on a vehicle includes acquiring one or more three-dimensional images using the lidar sensor while the vehicle is traveling along a road having fixed road features, each of the one or more three-dimensional images including an image of the road feature, and further including analyzing a spatial relationship between an image of the road feature in the one or more three-dimensional images and an orientation of a field of view of the lidar sensor, and determining a deviation from an expected positional alignment of the lidar sensor with respect to the vehicle based on the spatial relationship between the image of the road feature and the field of view of the lidar sensor. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 illustrates an exemplary lidar sensor for three-dimensional imaging in accordance with some embodiments. [Figure 2A] FIG. 1 illustrates a schematic diagram of a lidar sensor mounted on a vehicle, in accordance with some embodiments. [Figure 2B] FIG. 2 illustrates a vehicle coordinate system, according to some embodiments. [Figure 2C] FIG. 2 illustrates a lidar coordinate system, according to some embodiments. [Figure 3A] 1A-1C are schematic diagrams illustrating mounting mechanisms that can be used to mount a lidar sensor on a vehicle, in accordance with some embodiments. [Figure 3B] 1A-1C are schematic diagrams illustrating mounting mechanisms that can be used to mount a lidar sensor on a vehicle, in accordance with some embodiments. [Figure 4A] FIG. 1 illustrates an example calibration setup for a vehicle-mounted lidar sensor in accordance with some embodiments. [Figure 4B] FIG. 1 illustrates an example calibration setup for a vehicle-mounted lidar sensor in accordance with some embodiments. [Figure 4C] FIG. 1 illustrates an example calibration setup for a vehicle-mounted lidar sensor in accordance with some embodiments. [Figure 5A] 1A-1C are schematic diagrams illustrating what a lidar sensor sees under certain alignment conditions, in accordance with some embodiments. [Figure 5B] 1A-1C are schematic diagrams illustrating what a lidar sensor sees under different alignment conditions, in accordance with some embodiments. [Figure 5C] 1A-1C are schematic diagrams illustrating what a lidar sensor sees under different alignment conditions, in accordance with some embodiments. [Figure 6A] 1A-1C are schematic diagrams illustrating what a lidar sensor sees under certain alignment conditions, in accordance with some embodiments. [Figure 6B] 1A-1C are schematic diagrams illustrating what a lidar sensor sees under different alignment conditions, in accordance with some embodiments. [Figure 6C] 1A-1C are schematic diagrams illustrating what a lidar sensor sees under different alignment conditions, in accordance with some embodiments. [Figure 6D] 1A-1C are schematic diagrams illustrating what a lidar sensor sees under different alignment conditions, in accordance with some embodiments. [Figure 7] 1 is a simplified flowchart illustrating a method for calibrating a lidar sensor mounted on a vehicle using a target having an embedded mirror, in accordance with some embodiments. [Figure 8]FIG. 1 illustrates a lidar sensor mounted behind a vehicle windshield, including a mask, in accordance with some embodiments. [Figure 9A] FIG. 9 is a schematic side view of a windshield having the mask shown in FIG. 8. [Figure 9B] FIG. 9 is a schematic side view of a windshield having the mask shown in FIG. 8. [Figure 10A] 9A-9C are schematic diagrams illustrating examples of the effect of the mask shown in FIG. 8 on the effective field of view of a lidar sensor under certain alignment conditions, in accordance with some embodiments. [Figure 10B] 9A-9C are schematic diagrams illustrating another example of the effect of the mask shown in FIG. 8 on the effective field of view of the lidar sensor under different alignment conditions, in accordance with some embodiments. [Figure 10C] 9A-9C are schematic diagrams illustrating another example of the effect of the mask shown in FIG. 8 on the effective field of view of the lidar sensor under different alignment conditions, in accordance with some embodiments. [Figure 10D] 1A-1C are schematic diagrams illustrating example images of certain features of a vehicle captured by a lidar sensor, in accordance with some embodiments. [Figure 10E] FIG. 2 is another diagram that illustrates a schematic of an example image of a particular feature of a vehicle acquired by a lidar sensor, in accordance with some embodiments. [Figure 10F] FIG. 2 is another diagram that illustrates a schematic of an example image of a particular feature of a vehicle acquired by a lidar sensor, in accordance with some embodiments. [Figure 11] 1 is a simplified flowchart illustrating a method for calibrating a lidar sensor mounted on a vehicle using on-vehicle features, in accordance with some embodiments. [Figure 12A] FIG. 1 illustrates a method for dynamic calibration of a vehicle-mounted lidar sensor using lane markings on a road, according to some embodiments. [Figure 12B] FIG. 1 illustrates a method for dynamic calibration of a vehicle-mounted lidar sensor using lane markings on a road, according to some embodiments. [Figure 13A]FIG. 1 illustrates a method for dynamic calibration of a vehicle-mounted lidar sensor using lane markings on a road, according to some embodiments. [Figure 13B] FIG. 1 illustrates a method for dynamic calibration of a vehicle-mounted lidar sensor using lane markings on a road, according to some embodiments. [Figure 13C] FIG. 1 illustrates a method for dynamic calibration of a vehicle-mounted lidar sensor using lane markings on a road, according to some embodiments. [Figure 14] 1 is a simplified flowchart illustrating a method for calibrating a lidar sensor mounted on a vehicle using road features, in accordance with some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008]

[0022] According to some embodiments, a method is provided for calibrating a lidar sensor mounted on a vehicle. The calibration may not require returning the vehicle on which the lidar sensor is mounted to a manufacturer's plant or repair shop. The calibration may be performed periodically or continuously, even while the vehicle is parked or in operation.

[0009]

[0023] FIG. 1 illustrates an exemplary lidar sensor 100 for three-dimensional imaging according to some embodiments. The lidar sensor 100 includes an emission lens 130 and a receiving lens 140. The lidar sensor 100 includes a light source 110a disposed substantially at a back focal plane of the emission lens 130. The light source 110a operates to emit light pulses 120 from respective emission positions within the back focal plane of the emission lens 130. The emission lens 130 is configured to collimate and direct the light pulses 120 toward an object 150 located in front of the lidar sensor 100. For a given emission position of the light source 110a, the collimated light pulses 120′ are directed at a corresponding angle toward the object 150.

[0010]

[0024] A portion 122 of the collimated light pulse 120' is reflected from the object 150 towards the receiving lens 140. The receiving lens 140 is configured to focus the portion 122' of the light pulse reflected from the object 150 at a corresponding detection location in a focal plane of the receiving lens 140. The lidar sensor 100 further includes a detector 160a disposed substantially in the focal plane of the receiving lens 140. The detector 160a is configured to receive and detect the portion 122' of the light pulse 120 reflected from the object at a corresponding detection location. The corresponding detection locations of the detector 160a are optically conjugate with the respective emission locations of the light source 110a.

[0011]

[0025] The light pulse 120 may be of short duration, for example a pulse width of 10 ns. The lidar sensor 100 further includes a processor 190 coupled to the light source 110a and the detector 160a. The processor 190 is configured to determine a time of flight (TOF) of the light pulse 120 from emission to detection. Because the light pulse 120 travels at the speed of light, a distance between the lidar sensor 100 and the object 150 may be determined based on the determined time of flight.

[0012]

[0026] One way to scan the laser beam 120' across the FOV is to move the light source 110a laterally relative to the output lens 130 within the back focal plane of the output lens 130. For example, the light source 110a may be raster scanned to multiple output positions in the back focal plane of the output lens 130, as shown in FIG. 1. The light source 110a may emit multiple light pulses at multiple output positions. Each light pulse emitted at each output position is collimated by the output lens 130 and directed at the object 150 at a respective angle to impinge on a corresponding point on the surface of the object 150. Thus, as the light source 110a is raster scanned within a particular area within the back focal plane of the output lens 130, a corresponding object area on the object 150 is scanned. The detector 160a may be raster scanned to be positioned at multiple corresponding detection positions within the focal plane of the receiving lens 140, as shown in FIG. 1. The scanning of detector 160a is typically synchronized with the scanning of light source 110a, so that detector 160a and light source 110a are always optically conjugate to each other at any given time.

[0013]

[0027] By determining the time of flight of each emitted light pulse at each emission position, the distance from the lidar sensor 100 to each corresponding point on the surface of the object 150 can be determined. In some embodiments, the processor 190 is coupled to a position encoder that detects the position of the light source 110a at each emission position. Based on the emission position, the angle of the collimated light pulse 120' can be determined. The XY coordinates of the corresponding point on the surface of the object 150 can be determined based on the angle and the distance to the lidar sensor 100. Thus, a three-dimensional image of the object 150 can be constructed based on the measured distances from the lidar sensor 100 to various points on the surface of the object 150. In some embodiments, the three-dimensional image can be represented as a point cloud, i.e., a set of X, Y, and Z coordinates of points on the surface of the object 150.

[0014]

[0028] In some embodiments, the intensity of the return light pulse 122' is measured and used to adjust the power of subsequent light pulses from the same emission point to prevent detector saturation, improve eye safety, or reduce overall power consumption. The power of the light pulse may be varied by changing the duration of the light pulse, the voltage or current applied to the laser, or the charge stored on a capacitor used to power the laser. In the latter case, the charge stored on the capacitor may be varied by changing the charging time, charging voltage, or charging current to the capacitor. In some embodiments, the reflectance, determined by the intensity of the detected pulse, may also be used to add another dimension to the image. For example, the image may include X, Y, and Z coordinates, as well as reflectance (or brightness).

[0015]

[0029] The angle of view (AFOV) of the lidar sensor 100 may be estimated based on the scanning range of the light source 110a and the focal length of the output lens 130 as follows:

number

[0016]

[0030] The light source 110a may be configured to emit light pulses in the ultraviolet, visible, or near-infrared wavelength range. The energy of each light pulse may be on the order of microjoules, which is typically considered eye-safe for repetition rates in the kHz range. For light sources operating at wavelengths above about 1500 nm, the energy levels can be higher since the eye does not focus on those wavelengths. The detector 160a may include a silicon avalanche photodiode, a photomultiplier tube, a PIN diode, or other semiconductor sensor.

[0017]

[0031] When a lidar sensor, such as the lidar sensor 100 shown in FIG. 1, is used for obstacle detection in an autonomous vehicle, the position and orientation of the lidar sensor relative to the vehicle may need to be precisely known in order to precisely determine the location of an obstacle relative to the vehicle. FIG. 2A shows a lidar sensor 210 mounted on a vehicle 220. For example, the lidar sensor 210 may be mounted at the top center behind the windshield of the vehicle 220. The lidar sensor 210 may be characterized by an optical axis 250 (e.g., the optical axis of the exit lens or the receiving lens). The vehicle 220 may have a longitudinal axis 240. It may be advantageous to align the optical axis 250 of the lidar sensor 210 with the longitudinal axis 240 of the vehicle 220 so that the lidar sensor 210 looks straight ahead in the direction the vehicle 220 is traveling.

[0018]

[0032] Assume that the lidar sensor 210 is initially calibrated according to this alignment condition. If the lidar sensor becomes misaligned (e.g., turn left) due to some mechanical disturbance, the optical axis 250 of the lidar sensor 210 will no longer be aligned with the longitudinal axis 240 of the vehicle 220, as shown in FIG. 2A. The misalignment may result in an inaccurate measurement of the position of an obstacle (e.g., person 260) relative to the vehicle 220. Thus, recalibration of the lidar sensor 210 may be required.

[0019]

[0033] 2B illustrates a vehicle coordinate system. The vehicle coordinate system may have three translational degrees of freedom that may be represented, for example, by X, Y, and Z coordinates. The vehicle coordinate system may have three rotational degrees of freedom that may be represented, for example, by roll, pitch, and yaw angles about the x, y, and z axes, respectively. For example, the x-axis may be along the longitudinal axis of the vehicle 220, the z-axis may be along the vertical direction, and the y-axis may be along the lateral direction. The position and orientation of the lidar sensor 210 relative to the vehicle 220 may be characterized by (x, y, z, roll, pitch, yaw) coordinates in the vehicle coordinate system.

[0020]

[0034] 2C illustrates a lidar coordinate system. The lidar coordinate system may also have three translational degrees of freedom (e.g., represented by X, Y, and Z coordinates) and three rotational degrees of freedom (e.g., represented by roll, pitch, and yaw angles about the X, Y, and Z axes, respectively). For example, the X axis may be along the optical axis of the lidar sensor 210, the Z axis may be along a nominal vertical direction, and the Y axis may be along a direction orthogonal to the X and Z axes.

[0021]

[0035] The raw point cloud data acquired by the lidar sensor 210 may be in a lidar coordinate system. To determine the position of an obstacle relative to the vehicle 220, the point cloud data may be transformed to the vehicle coordinate system if the position and orientation of the lidar sensor 210 in the vehicle coordinate system is known. The transformation from the lidar coordinate system to the vehicle coordinate system may be referred to herein as calibrating the lidar sensor 210 relative to the vehicle 220.

[0022]

[0036] According to some embodiments, the lidar sensor 210 can be mounted on the vehicle 220 such that the lidar sensor 210 is nominally aligned with respect to the vehicle 220. For example, the lidar sensor 210 can be mounted on the vehicle 220 such that the x-axis of the lidar coordinate system (e.g., along the optical axis of the lidar sensor 210) is nominally aligned with the x-axis of the vehicle coordinate system (e.g., along the longitudinal axis of the vehicle), the y-axis of the lidar coordinate system is nominally aligned with the y-axis of the vehicle coordinate system, and the z-axis of the lidar coordinate system is nominally aligned with the z-axis of the vehicle coordinate system. Thus, the roll, pitch, and yaw angles in the vehicle coordinate system are all approximately zero. Calibration may be required to correct any remaining deviations from the nominal alignment to a sufficient accuracy. For example, it may be desirable to calibrate the lidar sensor 210 to a translational accuracy of 2 cm along each of the x-, y-, and z-axes, and a rotational accuracy of 0.1 degrees for each of the roll, pitch, and yaw angles.

[0023]

[0037] 3A and 3B show schematics of mounting mechanisms that can be used to mount a lidar sensor to a vehicle, according to some embodiments. The lidar sensor may have an outer housing 310 with three mounting holes 320 (e.g., on a top surface of the outer housing 310), as shown in FIG. 3A. A bracket 330 may have three holes 340 that match the mounting holes 320 of the outer housing 310, thereby allowing the lidar sensor to be mounted to the bracket 330 in a fixed orientation, as shown in FIG. 3B. The bracket 330 may be mounted to the vehicle (e.g., on the roof of the passenger compartment) using similar mounting holes (not shown) for proper alignment.

[0024]

[0038] The lidar sensor can be pre-calibrated at the manufacturer's plant to correct for residual misalignment of the lidar sensor relative to the vehicle. During operation of the vehicle over time, the lidar sensor's optics can shift relative to the housing 310, or the lidar sensor's housing 310 can shift relative to the mounting bracket 330 and / or the vehicle. This can occur, for example, due to tearing and wear of the internal mechanisms of the lidar sensor, collisions or vibrations of the vehicle, misalignment of the tires, aging of the vehicle's suspension, etc. Thus, over time, the calibration can become inaccurate and a new calibration may be required. According to various embodiments, the lidar calibration can be performed periodically using a target with an embedded mirror, or periodically or continuously using fixed features on the vehicle or fixed road features, as described in more detail below.

[0025]

[0039] A. Calibrating a Lidar Sensor Using a Target with an Embedded Mirror 4A-4C illustrate an example calibration setup for a lidar sensor mounted on a vehicle using a target with an embedded mirror, according to some embodiments. The lidar sensor 410 is shown mounted behind the windshield of the vehicle 420 in this example. However, this is not required. For example, the lidar sensor 410 can be mounted in other locations on the vehicle 420, such as the front bumper, rear window, rear bumper, etc. The target 440 is positioned a distance D in front of the lidar sensor 410. The target 440 includes a planar mirror 430 and an identifiable feature 490 surrounding the mirror 430.

[0026]

[0040] 4A, the target 440 may be mounted on a stand (not shown) such that the mirror 430 is nominally vertical. Thus, the optical axis 432 of the mirror 430, which is perpendicular to the surface of the mirror 430, may be nominally horizontal. The vehicle 420 is positioned and oriented relative to the mirror 430 such that the optical axis 412 of the lidar sensor is nominally parallel to the optical axis 432 of the mirror 430, and the target 440 is nominally centered in the field of view of the lidar sensor 410. If the lidar sensor 410 is mounted behind the windshield of the vehicle 420 (e.g., as shown in FIG. 4A), the vehicle 420 may be positioned such that the longitudinal axis 422 of the vehicle 420 is nominally parallel to the optical axis 432 of the mirror 430.

[0027]

[0041] With reference to FIG. 4B, the lidar sensor 410 can acquire a three-dimensional image of the target 440 by scanning across the target 440. The three-dimensional image of the target can include images of features 490 surrounding the mirror 430, and a mirror image of the vehicle 420 formed by the mirror 430. For example, the lidar sensor 410 can emit a laser pulse 450 (or 450a) that is directed toward the mirror 430. The reflected laser pulse 450b can be directed toward a particular portion of the vehicle (e.g., the license plate on the front bumper) and then reflected toward the mirror 430 (shown as 450c in FIG. 4B) and back to the lidar sensor 410 (shown as 450d in FIG. 4B). Thus, as shown in FIG. 4C, the lidar sensor 410 can effectively acquire a three-dimensional mirror image of the vehicle 420.

[0028]

[0042] According to some embodiments, the distance D between the target 440 and the vehicle 420 can be large enough to meet the desired accuracy. For example, the distance D can be about 3 m. The size of the mirror 430 can be large enough to view the entire vehicle 420, but this is not required. For example, the size of the mirror 430 can be about 1 m high and about 2 m wide. The size of the target 440 can be about 2 m high and about 3 m wide (e.g., the edge of the target 440 with the features 490 can have a width of 0.5 m).

[0029]

[0043] 5A shows a schematic of what the lidar sensor 410 sees when the vehicle 420 is properly aligned with the target 440 (e.g., the longitudinal axis of the vehicle 420 is perpendicular to the surface of the mirror 430) and the lidar sensor 410 is properly aligned with the vehicle 420 (e.g., the optical axis of the lidar sensor 410 is parallel to the longitudinal axis of the vehicle 420). We also assume that the vehicle 420 is laterally centered with respect to the width of the mirror 430 and that the lidar sensor 410 is laterally centered with respect to the vehicle 420. In such a case, the mirror image 420' of the vehicle 420 as seen by the lidar sensor 410 may appear symmetrical and have no roll, pitch, or yaw angles. The target 440 also appears symmetric with respect to the lidar sensor's field of view 414 (e.g., the lateral margin d from the left edge of the target 440 to the left boundary of the field of view 414 is approximately the same as the lateral margin d from the right edge of the target 440 to the right boundary of the field of view 414) and may have no roll, pitch, or yaw angles.

[0030]

[0044] FIG. 5B shows a schematic of what the lidar sensor 410 sees when the lidar sensor 410 is correctly aligned with the vehicle 420, but the vehicle 420 is misaligned with the target 440 (e.g., the longitudinal axis of the vehicle 420 has a finite yaw angle with respect to the optical axis of the mirror 430). In such a case, the target 440 is no longer in the center of the lidar field of view 414 (e.g., shifted laterally to the right). The amount of shift (e.g., the difference between the new margin D and the nominal margin d) may be related to the amount of misalignment. Thus, it may be possible to determine the amount of misalignment of the vehicle 420's orientation with respect to the target 440 based on the amount of shift (e.g., yaw error). In addition, the mirror image 420' of the vehicle 420 seen by the lidar sensor 410 also appears to have a finite yaw angle (note that the amount of rotation is somewhat exaggerated in FIG. 5B). Thus, the lidar sensor 410 can also determine the amount of misalignment of the vehicle 420 relative to the target 440 by measuring the distance of certain features of the vehicle 420. For example, as shown in FIG 5B, the lidar sensor 410 can measure that one headlamp (e.g., the left headlamp) is farther from the lidar sensor 410 than the other headlamp (e.g., the right headlamp), and thus the yaw angle of the vehicle 420 can be calculated.

[0031]

[0045] FIG. 5C illustrates generally what the lidar sensor 410 sees when the vehicle 420 is correctly aligned with the target 440 (e.g., the longitudinal axis of the vehicle 420 is perpendicular to the surface of the mirror 430), but the lidar sensor 410 is incorrectly aligned with the vehicle 420 (e.g., the lidar sensor 410 is looking left instead of looking straight ahead). In such a case, the mirror image 420′ of the vehicle 420 may appear symmetrical, but the target 440 may be shifted laterally (e.g., to the right) relative to the lidar field of view 414. Based on the amount of shift (e.g., the difference between the new margin D and the nominal margin d), the lidar sensor 410 can determine the amount of misalignment of the lidar sensor 410's orientation (e.g., yaw error).

[0032]

[0046] FIG. 6A shows a schematic of what the lidar sensor 410 sees when the lidar sensor 410 is correctly aligned with the vehicle 420, but the vehicle 420 has a pitch error with respect to the target 440. In such a case, the target 440 is no longer in the center of the lidar field of view 414 (e.g., shifted vertically downward). The amount of vertical shift may be related to the amount of pitch error. Thus, it may be possible to determine the amount of pitch error of the vehicle 420 with respect to the target 440 based on the amount of vertical shift. In addition, the mirror image 420' of the vehicle 420 as seen by the lidar sensor 410 also appears to have a finite pitch angle (note that the amount of rotation is somewhat exaggerated in FIG. 6A). Thus, the lidar sensor 410 can also determine the amount of pitch error of the vehicle 420 with respect to the target 440 by measuring the distance of certain features of the vehicle 420. For example, as shown in FIG. 6A, the lidar sensor 410 can measure that the roof of the vehicle 420 is tilted, and can calculate the pitch error based on the amount of tilt.

[0033]

[0047] 6B illustrates generally what the lidar sensor 410 sees when the vehicle 420 is correctly aligned with respect to the target 440, but the lidar sensor 410 has a pitch error relative to the vehicle 420 (e.g., the lidar sensor 410 looks upward instead of looking straight ahead). In such a case, the mirror image 420' of the vehicle 420 may appear to have no pitch angle, but the target 440 may be shifted vertically (e.g., downward) relative to the lidar field of view 414. Based on the amount of vertical shift, the lidar sensor 410 can determine the amount of pitch error of the lidar sensor 410.

[0034]

[0048] 6C illustrates generally what the lidar sensor 410 sees when the lidar sensor 410 is correctly aligned with respect to the vehicle 420, but the vehicle 420 has a roll error relative to the target 440. In such a case, the mirror image 420' of the vehicle 420 seen by the lidar sensor 410 may appear to have no roll angle, while the target 440 may appear to have a finite roll angle relative to the lidar field of view 414. Based on the amount of roll angle of the target 440 relative to the lidar sensor field of view 414, it may be possible to determine the amount of roll error of the vehicle 420 relative to the target 440.

[0035]

[0049] 6D illustrates generally what the lidar sensor 410 sees when the vehicle 420 is correctly aligned with respect to the target 440, but the lidar sensor 410 has a roll error with respect to the vehicle 420. In such a case, both the mirror image 420' of the vehicle 420 and the target 440 may appear to have a finite roll angle with respect to the lidar field of view 414. Based on the amount of roll rotation, the lidar sensor 410 can determine the amount of roll error of the lidar sensor 410.

[0036]

[0050] Thus, the lidar sensor 410 (or a computing unit of the vehicle 420) can determine deviations from an expected alignment of the lidar sensor 410 with respect to the vehicle 420 (e.g., an initial alignment performed at the manufacturer's plant) by analyzing three-dimensional images acquired by the lidar sensor 410, which include images of the features 490 on the target and a mirror image 420' of the vehicle 420. The deviations from the expected alignment can include yaw error, roll error, pitch error, and translation error (e.g., δx, δy, δz).

[0037]

[0051] According to some embodiments, determining a deviation from an expected alignment of the lidar sensor 410 with respect to the vehicle 420 may include the following steps: A position and orientation of the lidar sensor 410 with respect to the target 440 may be determined based on an image of the features 490 on the target 440. A position and orientation of the lidar sensor 410 with respect to a mirror image 420' of the vehicle 420 may be determined based on the mirror image 420' of the vehicle. A transformation from the lidar coordinate system to the vehicle coordinate system may then be determined based on (i) the position and orientation of the lidar sensor 410 with respect to the target 440 and (ii) the position and orientation of the lidar sensor 410 with respect to the mirror image 420' of the vehicle 420.

[0038]

[0052] In some embodiments, the computing unit can store a reference image. For example, the reference image can be an image acquired by the lidar sensor 410 immediately after an initial alignment is performed at the manufacturer's plant, or it can be a simulated image of the expected alignment. During recalibration, the computing unit can compare the three-dimensional image acquired by the lidar sensor 410 to the reference image and perform a multivariate minimization (e.g., using gradient descent or other algorithms) to determine a transformation matrix such that the acquired three-dimensional image most closely matches the reference image. Deviations from the expected alignment (e.g., yaw error, roll error, pitch error, δx, δy, and δz) can then be derived from the transformation matrix.

[0039]

[0053] The following example method may be used to determine the relationship of the lidar sensor 410 to the vehicle 420 in six degrees of freedom according to various embodiments. Other methods and techniques may also be used by those skilled in the art. In the description of the example embodiments below, the following notation and terminology are used: t represents a matrix describing the relationship of the lidar sensor 410 to the target 440, and C t represents a matrix describing the relationship of the vehicle 420 to the target 440, and L Crepresents a matrix describing the relationship of the lidar sensor 410 to the vehicle 420 (to correct for miscalibration of the lidar sensor 410), M represents the mirror transformation matrix, and L mC represents a matrix that describes the relationship of the lidar sensor 410 to the vehicle 420 as the lidar sensor 410 looks in the mirror 430.

[0040]

[0054] In some embodiments, the lidar sensor 410 can establish its position (x, y, z) relationship to the target features 490 around the mirror 430 by triangulating the distance from at least three target features 490 (e.g., similar to how a GPS receiver triangulates its position relative to GPS satellites). The pitch, roll, and yaw rotational relationships can be determined by measuring the positions of at least three target features 490 (which can be the same target features used for x, y, and z). Thus, the lidar sensor 410 can be aligned with the target features 490 by measuring the position of the target features 490. t A matrix can be established that describes the relationship of the lidar sensor 410 to the matrix L. t is a 4x4 matrix that defines the x, y, and z position, as well as the pitch, roll, and yaw.

[0041]

[0055] The relationship of the vehicle 420 to the target 440 can then be established by a similar procedure, namely, by measuring the distance to a particular feature of the vehicle 420 seen in the mirror 430 to triangulate its position relative to the target 440, and by measuring the position of the feature on the vehicle 420 relative to the mirror 430 to determine the pitch, roll, and yaw of the vehicle 420. Thus, the relationship of the target 440C to the target 440C can be established by a similar procedure, namely, by measuring the distance to a particular feature of the vehicle 420 seen in the mirror 430 to triangulate its position relative to the target 440, and by measuring the position of the feature on the vehicle 420 relative to the mirror 430 to determine the pitch, roll, and yaw of the vehicle 420. t A matrix can be established that describes the relationship of the vehicle 420 to the t is a 4×4 matrix that defines the x, y, and z positions, as well as the pitch, roll, and yaw. The relationship of the lidar sensor 410 to the vehicle 420 is then expressed as L C =(C t -1 )·L t It can be defined as follows:

[0042]

[0056] According to some embodiments, the lidar sensor 410 calculates the matrix L t The lidar sensor 410 can determine its position relative to the target 440 as described above to establish the matrix L mC The relationship of the mirror image 420′ of the vehicle to the lidar sensor 410 as seen through the mirror 430 can also be determined to establish θ. The relationship of the lidar sensor 410 to the vehicle 420 can then be determined by matrix multiplication as follows: L C =L t M(L t -1 )L mC M

[0043]

[0057] According to some embodiments, the image 420' of the vehicle as seen in the mirror 430 and the image of the target 440 acquired by the lidar sensor 410 are compared to a stored reference image. The reference image can be either a simulated image or an image taken during factory calibration. A minimization technique (e.g., using a gradient descent algorithm) can then be used to determine the transformation parameters (δx, δy, δz, pitch error, roll error, and yaw error) that minimize the difference between the currently acquired image and the stored reference image. In this process, the currently acquired image can be transformed to match the stored reference image, or the stored reference image can be transformed to match the currently acquired image. The transformation parameters can represent the difference between the current lidar position relative to the vehicle 420 and an ideal (or factory calibrated) lidar position.

[0044]

[0058] Once the relationship of the lidar 410 to the vehicle 420 is determined, any discrepancy in this relationship with the current lidar calibration can be used to correct the lidar calibration.

[0045]

[0059] In some embodiments, one or more distance sensors can be used to determine the position and orientation of the vehicle 420 relative to the target 440 (e.g., in a world coordinate system). For example, by placing two or three distance sensors in the pavement under the vehicle 420, the pitch, roll, and z coordinate (height) of the vehicle can be determined. By using six distance sensors, all degrees of freedom (x, y, z, pitch, roll, yaw) can be determined. The distance sensors can be ultrasonic or laser sensors.

[0046]

[0060] An example of using distance sensors to determine the position and orientation of the vehicle 420 is shown in FIG. 5A. In this example, four distance sensors 590 can be placed at suitable locations around the vehicle 420. For example, each distance sensor 590 can be placed near a respective wheel of the vehicle 420. The distance sensors 590 can estimate the distance from the wheel by sending an ultrasonic or laser pulse toward the wheel and measuring the return pulse. Thus, the y coordinate and yaw angle of the vehicle 420 can be determined. In some embodiments, only two distance sensors 590 on one side of the vehicle (either the driver's side or the passenger's side) may be needed. Another distance sensor 592 can be placed in front of the vehicle 420 to determine the x coordinate of the vehicle 420.

[0047]

[0061] According to some embodiments, additional corrections or calibrations can be performed. For example, the effects of windshield distortion can be measured and corrected. Windshield distortion correction may be required whenever the windshield is replaced.

[0048]

[0062] FIG. 7 shows a simplified flowchart illustrating a method 700 for calibrating a lidar sensor mounted on a vehicle using a target with an embedded mirror, according to some embodiments.

[0049]

[0063] The method 700 includes, at 702, positioning a vehicle away from a target. The target includes a planar mirror and features surrounding the mirror. The optical axis of the mirror is approximately horizontal. The vehicle is positioned and oriented relative to the mirror such that the optical axis of the lidar sensor is nominally parallel to the optical axis of the mirror and the target is nominally centered in the field of view of the lidar sensor.

[0050]

[0064] The method 700 further includes acquiring a three-dimensional image of the target using the lidar sensor, at 704. The three-dimensional image of the target includes images of the features of the target and a mirror image of the vehicle formed by the mirror.

[0051]

[0065] The method 700 further includes, at 706, determining a deviation from an expected alignment of the lidar sensor with respect to the vehicle by analyzing the image of the feature in the three-dimensional image of the target and a mirror image of the vehicle.

[0052]

[0066] In some embodiments, the method 700 further includes, at 708, recalibrating the lidar sensor relative to the vehicle based on a deviation from an expected positional alignment of the lidar sensor relative to the vehicle.

[0053]

[0067] In some embodiments, method 700 further includes determining that a deviation from an expected alignment of the lidar sensor exceeds a threshold, and issuing an alert in response to determining that the deviation from the expected alignment of the lidar sensor exceeds the threshold.

[0054]

[0068] In some embodiments, the field of view of the lidar sensor is less than 180 degrees horizontally.

[0055]

[0069] In some embodiments, determining a deviation from an expected positional alignment of the lidar sensor relative to the vehicle may include determining a position and orientation of the lidar sensor relative to the target based on the image of the feature; determining a position and orientation of the lidar sensor relative to the mirror image of the vehicle based on a mirror image of the vehicle; and determining a transformation from the lidar coordinate system to the vehicle coordinate system based on (i) the position and orientation of the lidar sensor relative to the target and (ii) the position and orientation of the lidar sensor relative to the mirror image of the vehicle.

[0056]

[0070] In some embodiments, determining a deviation from an expected alignment of the lidar sensor with respect to the vehicle may include storing a reference matrix for an expected relationship between the lidar sensor and the vehicle, determining a matrix for a current relationship between the lidar sensor and the vehicle, and determining a deviation from an expected alignment of the lidar sensor with respect to the vehicle by comparing the matrix to the reference matrix. In some embodiments, method 700 further includes recalibrating the lidar sensor with respect to the vehicle based on the matrix for the current relationship between the lidar sensor and the vehicle.

[0057]

[0071] It should be understood that the specific steps illustrated in FIG. 7 provide a particular method of calibrating a lidar sensor according to some embodiments. Other orders of steps may be performed according to alternative embodiments. For example, alternative embodiments of the present invention may perform the steps outlined above in a different order. Additionally, individual steps illustrated in FIG. 7 may include multiple sub-steps that may be performed in various orders as appropriate for the individual step. Additionally, additional steps may be added or some steps may be removed depending on the particular application. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0058]

[0072] B. Dynamic Calibration of LiDAR Sensors Using On-Vehicle Features It may be desirable to have a method for periodically or continuously checking the calibration of a lidar sensor for a vehicle, even while the vehicle is parked or in operation. According to some embodiments, the calibration of the lidar sensor can be performed using characteristics of the vehicle such that the calibration can be performed during normal operation of the vehicle. Such a method is referred to herein as dynamic calibration.

[0059]

[0073] FIG. 8 illustrates an example of dynamic calibration of a lidar sensor using a windshield-mounted mask, according to some embodiments. As shown, a lidar sensor 810 (not visible in FIG. 8 ) is mounted behind a windshield 840 of a vehicle 850. A mask 820 is mounted to the windshield 840 in an area directly in front of the lidar sensor 810. Assuming that the operating wavelength of the lidar sensor 810 is in the IR wavelength range, the mask 820 can be configured to block infrared (IR) light (e.g., be optically opaque to infrared light). In some embodiments, the mask 820 can be annular in shape that surrounds the IR-transmitting portion 830 of the windshield 840. The geometry and size of the IR-blocking mask 820 can be made to block around the edges of the field of view of the lidar sensor 810.

[0060]

[0074] 9A and 9B show schematic side views of a windshield 840 with an IR blocking mask 820, according to some embodiments. An IR transparent portion 830 of the windshield 840 is surrounded by the IR blocking mask 820. The IR transparent portion 830 is slightly smaller than the field of view 910 of the lidar sensor 810. Referring to FIG. 9A, under correct alignment (e.g., expected alignment), the lidar sensor 810 can be placed behind the windshield 840 such that its field of view 910 is centered on the IR transparent portion 830. Thus, the edges of the field of view 910 are blocked on all four sides (the side view in FIG. 9A shows the top and bottom).

[0061]

[0075] 9B illustrates an example where the lidar sensor 810 is misaligned (e.g., tilted upward) from the correct alignment position. As a result, the IR transparent portion 830 is no longer centered on the field of view 910 of the lidar sensor 810. Thus, a larger portion of its field of view 910 may be blocked above than below.

[0062]

[0076] 10A-10C are schematic diagrams illustrating several examples of the effect of the IR blocking mask 820 shown in FIG. 8 under various alignment conditions, according to some embodiments. FIG. 10A illustrates an example where the lidar sensor 810 is correctly aligned. The IR transparent portion 830 of the windshield 840 is centered on the field of view 910 of the lidar sensor 810. Thus, the edges of the field of view 910 that are blocked by the IR blocking mask 820 (the gray areas) have approximately the same width on all four sides.

[0063]

[0077] 10B illustrates an example in which the lidar sensor 810 is displaced to the right from the correct alignment position (e.g., the lidar sensor 810 has a yaw error). As a result, the IR transparent portion 830 of the windshield 840 is displaced to the left relative to the field of view 910 of the lidar sensor 810. Thus, a larger portion of the field of view 910 is blocked by the IR blocking mask 820 on the right side than on the left side.

[0064]

[0078] 10C illustrates an example in which the lidar sensor 810 has been rotated from a correct alignment position (e.g., the lidar sensor 810 has roll error). As a result, the IR transparent portion 830 of the windshield 840 has been rotated relative to the field of view 910 of the lidar sensor 810.

[0065]

[0079] According to some embodiments, the relative position and orientation of the IR-transmissive portion 830 of the windshield 840 with respect to the field of view 910 of the lidar sensor 810 can be used to calibrate the lidar sensor 810. For example, the computing unit can store a reference image. The reference can be acquired by the lidar sensor 810 while the lidar sensor is in a correct alignment position or can be acquired by simulation. For example, the reference image can be acquired by the lidar sensor immediately after the lidar sensor is pre-calibrated at a manufacturing facility. When the lidar sensor 810 is in normal operation (when the vehicle is parked or running), the computing unit can periodically or continuously compare the current lidar image with the reference image. Based on the comparison, a deviation from the correct alignment position of the lidar sensor 810 can be generated.

[0066]

[0080] In some embodiments, a multivariate minimization can be performed to determine a transformation matrix such that the IR transparent portion of the windshield 840 in the current lidar image best matches the IR transparent portion 830 in the reference image. The deviations from the correct alignment position (e.g., yaw error, roll error, pitch error, δx, δy, and δz) can then be derived from the transformation matrix. According to various embodiments, the lidar sensor 810 can also automatically recalibrate itself or issue an alert to the vehicle in response to determining that the deviation from the correct alignment position exceeds a threshold.

[0067]

[0081] Additionally or alternatively, images of some fixed features on the vehicle acquired by the lidar sensor 810 may also be used to calibrate the lidar sensor 810. FIGS. 10D-10F show some example images that may be acquired by the lidar sensor 810. The images show a portion of the vehicle's hood 1020 and a decorative feature 1030 (e.g., a manufacturer's logo) attached to the hood 1020. In FIG. 10D, the decorative feature 1030 is approximately centered laterally within the field of view 910 of the lidar sensor 810. In FIG. 10E, the decorative feature 1030 is shifted to the left of the field of view 910, indicating that the lidar sensor 810 may be pointing toward the right (e.g., having a yaw error). In FIG. 10F, the decorative feature 1030 is rotated, indicating that the lidar sensor 810 may be rotated (e.g., having a roll error). Note that the images of the hood 1020 in FIGS. 10E and 10F are also shifted and / or rotated accordingly. Other exemplary vehicle features that can be used for calibration include a cover over a grill-mounted lidar sensor, headlamp or taillamp features (e.g., if the lidar sensor is installed inside a headlamp or taillamp), etc.

[0068]

[0082] According to some embodiments, the computing unit can store a reference image acquired by the lidar sensor while the lidar sensor is in correct alignment (e.g., expected alignment) with the vehicle. The reference image can include a first image of fixed features on the vehicle. When the lidar sensor 810 is in normal operation (when the vehicle is parked or traveling), the computing unit can periodically or continuously compare a current lidar image to the reference image. The current lidar image includes a second image of fixed features of the vehicle. Deviations from correct alignment can be determined by comparing the position and orientation of the fixed features in the second image to the position and orientation in the first image.

[0069]

[0083] FIG. 11 illustrates a simplified flowchart illustrating a method 1100 for calibrating a lidar sensor mounted on a vehicle using on-vehicle features, according to some embodiments.

[0070]

[0084] The method 1100 includes storing a reference three-dimensional image acquired by the lidar sensor while the lidar sensor is in an expected alignment with the vehicle, at 1102. The reference three-dimensional image includes a first image of a fixed feature on the vehicle.

[0071]

[0085] The method 1100 further includes, at 1104, acquiring a three-dimensional image using the lidar sensor, the three-dimensional image including a second image of the fixed feature.

[0072]

[0086] The method 1100 further includes, at 1106, determining a deviation from an expected positional alignment of the lidar sensor with respect to the vehicle by comparing a second image of the fixed feature in the three-dimensional image to the first image of the fixed feature in the reference three-dimensional image.

[0073]

[0087] In some embodiments, the method 1100 further includes, at 1108, recalibrating the lidar sensor based on a deviation from an expected positional alignment of the lidar sensor relative to the vehicle.

[0074]

[0088] In some embodiments, the method 1100 further includes determining a transformation to be applied to a second image of a fixed feature in the three-dimensional image to match the first image of the fixed feature in the reference three-dimensional image, and recalibrating the lidar sensor based on the transformation.

[0075]

[0089] In some embodiments, the method 1100 further includes determining that a deviation from an expected position alignment of the lidar sensor exceeds a threshold, and issuing an alert in response to determining that the deviation from the expected position alignment of the lidar sensor exceeds the threshold.

[0076]

[0090] In some embodiments, the reference three-dimensional image is acquired by a lidar sensor after the lidar sensor has been pre-calibrated at a manufacturing facility.

[0077]

[0091] In some embodiments, the fixed feature comprises a portion of the hood of the vehicle or an object attached to the hood.

[0078]

[0092] In some embodiments, the lidar sensor is disposed behind a windshield of the vehicle, and the fixed feature includes a mask attached to an area of ​​the windshield directly in front of the lidar sensor. The mask is configured to block light at an operating wavelength of the lidar sensor and is shaped to block a portion of the field of view of the lidar sensor. The mask can have an outer boundary and an inner boundary, the inner boundary being sized such that the mask encroaches on the periphery of the field of view of the lidar sensor.

[0079]

[0093] It should be understood that the specific steps illustrated in FIG. 11 provide a particular method of calibrating a lidar sensor according to some embodiments. Other orders of steps may be performed in accordance with alternative embodiments. For example, alternative embodiments of the present invention may perform the steps outlined above in a different order. Additionally, individual steps illustrated in FIG. 11 may include multiple sub-steps that may be performed in various orders as appropriate for the individual step. Additionally, additional steps may be added or some steps may be removed depending on the particular application. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0080]

[0094] C. Dynamic Calibration of Lidar Sensors Using Road Features According to some embodiments, a method for dynamic calibration of a lidar sensor mounted on a vehicle can use road features as the vehicle travels along a relatively straight section of road. An example is shown in FIGS. 12A and 12B. A lidar sensor 1210 is shown mounted behind the windshield of a vehicle 1220 (the lidar sensor 1210 can be mounted in other locations, e.g., on the front bumper). The vehicle 1220 is traveling along a straight section of road 1230. Painted lane markings 1240 can be used to dynamically calibrate the position and orientation of the lidar sensor 1210 relative to the vehicle 1220 based on lidar images acquired while the vehicle is traveling along the road 1230. Using painted lane markings for calibration can be advantageous since lane markings are present on nearly all roads. Also, the distance between pairs of lane markings 1240 is usually a standard distance. Lane markings using retroreflective paint can appear clearly in the lidar images.

[0081]

[0095] 12A, if the lidar sensor 1210 is properly aligned with the vehicle 1220 (e.g., the lidar sensor 1210 is looking in the same direction that the vehicle 1220 is traveling), the pair of lane markings 1240 may appear to pass equally on either side of the vehicle path 1250. As shown in FIG. 12B, if the lidar sensor 1210 is misaligned with the vehicle 1220 (e.g., the lidar sensor 1210 is looking left with respect to the longitudinal axis of the vehicle 1220), the pair of lane markings 1240 may appear to pass the vehicle 1220 asymmetrically with respect to the vehicle path 1250. For example, the vehicle path 1250 may appear to move closer to the driver's side lane marking 1240a than the passenger side lane marking 1240b. Thus, by analyzing lidar images of the lane markings 1240 relative to the vehicle path 1250, the amount of positional misalignment (e.g., yaw error) of the lidar sensor 1210 relative to the vehicle 1220 can be estimated.

[0082]

[0096] 13A-13C show some further examples of using lane configurations for dynamic calibration of a vehicle-mounted lidar sensor, according to some embodiments. As shown in FIG. 13A, if the positional alignment of the lidar sensor 1210 has a pitch error, the lane markings 1240 may appear to be tilted relative to the vehicle path (assuming the road is relatively level).

[0083]

[0097] As shown in FIG. 13B, if the positional alignment of the lidar sensor 1210 has roll error, the lane marking 1240a on one side of the vehicle 1220 (e.g., the driver's side) may appear higher than the lane marking 1240b on the other side of the vehicle 1220 (e.g., the passenger side).

[0084]

[0098] 13C, if the lidar sensor 1210 alignment has a Z error (vertically), the height of the lane markings 1240 may be offset from the vehicle path. Translation errors in two other orthogonal directions (e.g., X and Y errors) can also be detected by monitoring the movement of objects such as lane markings (relative to expected movement) as the vehicle 1220 moves forward.

[0085]

[0099] Thus, by analyzing the spatial relationship between images of the lane markings 1240 (or other road features) and the vehicle path, various rotational and translational misalignments of the lidar sensor 1210 relative to the vehicle 1220 can be detected and estimated. If the lidar sensor is mounted on the side or rear of the vehicle, a similar calibration procedure can be used, with appropriate modifications and mathematical transformations to take into account the different viewing angles.

[0086]

[0100] According to some embodiments, the measurements can be repeated many times and the results can be averaged to account for, for example, road irregularities, poorly painted lane markings, curves, potholes, etc. Outliers can be discarded. Outliers can be due, for example, to lane changing and other driving irregularities, obstruction of the view of the lane markings by other vehicles, etc.

[0087]

[0101] According to some embodiments, other information can be used to improve the quality and reliability of the calibration data. For example, data from vehicle steering sensors, Global Navigation Satellite System (e.g., GPS) data, and Inertial Measurement Unit (IMU) data can be used to ensure that the vehicle is not turning. Map data can be used to select good road sections for calibration, where the road is straight, level, and the lane markings are new and properly spaced. Other road features, such as curbs, guard rails, and road signs, can also be used as inputs to the calibration algorithm.

[0088]

[0102] FIG. 14 illustrates a simplified flowchart illustrating a method 1400 for calibrating a lidar sensor mounted on a vehicle using road features, according to some embodiments.

[0089]

[0103] The method 1400 includes, at 1402, acquiring one or more three-dimensional images using a lidar sensor while the vehicle is traveling along a road having fixed road features, each of the one or more three-dimensional images including an image of the road features.

[0090]

[0104] The method 1400 further includes, at 1404, analyzing a spatial relationship between the images of road features in the one or more three-dimensional images and an orientation of the field of view of the lidar sensor.

[0091]

[0105] The method 1400 further includes, at 1406, determining a deviation from an expected alignment of the lidar sensor relative to the vehicle based on the spatial relationship between the image of the road features and the field of view of the lidar sensor.

[0092]

[0106] In some embodiments, the method 1400 further includes, at 1408, recalibrating the lidar sensor based on a deviation from an expected positional alignment of the lidar sensor relative to the vehicle.

[0093]

[0107] In some embodiments, the method 1400 further includes determining that a deviation from an expected positional alignment of the lidar sensor exceeds a threshold, and issuing an alert in response to determining that the deviation from the expected positional alignment of the lidar sensor exceeds the threshold.

[0094]

[0108] In some embodiments, the road features include one or more pairs of lane markings on either side of the vehicle. Analyzing the spatial relationship may include determining a pitch angle between a pair of lane markings of the one or more pairs of lane markings and a field of view of the lidar sensor, and determining a deviation from expected alignment of the lidar sensor may include determining a pitch error of the lidar sensor based on the pitch angle. In some embodiments, the one or more pairs of lane markings may include a first lane marking on a driver's side of the vehicle and a second lane marking on a passenger's side of the vehicle, and analyzing the spatial relationship may include determining a height difference between the first lane marking and the second lane marking, and determining a deviation from expected alignment of the lidar sensor may include determining a roll error of the lidar sensor based on the height difference.

[0095]

[0109] In some embodiments, the one or more three-dimensional images may include a plurality of three-dimensional images acquired by the lidar sensor over a time interval as the vehicle travels on a road over a distance interval. The road may be substantially straight and level over the distance interval. In some embodiments, the method 1400 further includes determining a path of the vehicle over the distance interval and comparing the path of the vehicle to a path of the road features from the plurality of three-dimensional images. In some embodiments, the one or more pairs of lane markings include a first lane marking on a driver's side of the vehicle and a second lane marking on a passenger's side of the vehicle, and analyzing the spatial relationship may include determining a lateral asymmetry between the first lane marking from the path of the vehicle and the second lane marking from the path of the vehicle, and determining a deviation from an expected alignment of the lidar sensor may include determining a yaw error of the lidar sensor based on the lateral asymmetry. In some embodiments, analyzing the spatial relationship may include determining an elevation difference between a pair of lane markings and a path of the vehicle, and determining a deviation from an expected alignment of the lidar sensor may include determining a vertical error of the lidar sensor based on the elevation difference.

[0096]

[0110] It should be understood that the specific steps illustrated in FIG. 14 provide a particular method of calibrating a lidar sensor according to some embodiments. Other orders of steps may be performed according to alternative embodiments. For example, alternative embodiments of the present invention may perform the steps outlined above in a different order. Additionally, individual steps illustrated in FIG. 14 may include multiple sub-steps that may be performed in various orders as appropriate for the individual step. Additionally, additional steps may be added or some steps may be removed depending on the particular application. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0097]

[0111] It should also be understood that the examples and embodiments described herein are for illustrative purposes only, and that various modifications or changes therein may be suggested to those skilled in the art and are to be included within the spirit and scope of this application and the scope of the appended claims.

Claims

1. 1. A method for calibrating a lidar sensor mounted on a vehicle, the method comprising: positioning the vehicle away from a target, the target comprises a flat mirror and features surrounding the flat mirror, the optical axis of the flat mirror being substantially horizontal; the vehicle is positioned and oriented relative to the planar mirror such that an optical axis of the LIDAR sensor is nominally parallel to the optical axis of the planar mirror and the target is nominally centered in the field of view of the LIDAR sensor; acquiring a three-dimensional image of the target using the lidar sensor, the three-dimensional image of the target including an image of the features of the target and a mirror image of the vehicle formed by the plane mirror; determining deviations from an expected alignment of the lidar sensor with respect to the vehicle by analyzing the images of the features in the three-dimensional image of the target and the mirror image of the vehicle; A method comprising:

2. recalibrating the lidar sensor relative to the vehicle based on the deviation from the expected alignment of the lidar sensor relative to the vehicle. The method of claim 1 further comprising:

3. determining that the deviation from the expected alignment of the lidar sensor exceeds a threshold; issuing an alert in response to determining that the deviation from the expected alignment of the lidar sensor exceeds the threshold. The method of claim 1 further comprising:

4. The method of claim 1 , wherein the field of view of the lidar sensor is less than 180 degrees horizontally.

5. determining the deviation from the expected alignment of the lidar sensor with respect to the vehicle, determining a position and orientation of the LIDAR sensor relative to the target based on the image of the feature; determining a position and orientation of the lidar sensor relative to the mirror image of the vehicle based on the mirror image of the vehicle; determining a transformation from a lidar coordinate system to a vehicle coordinate system based on (i) the position and orientation of the lidar sensor relative to the target, and (ii) the position and orientation of the lidar sensor relative to the mirror image of the vehicle; The method of claim 1 , comprising:

6. 6. The method of claim 5, wherein the rider coordinate system has three translational and three rotational degrees of freedom and the vehicle coordinate system has three translational and three rotational degrees of freedom.

7. the three translational degrees of freedom of the rider coordinate system are along three orthogonal axes including an x-axis, a y-axis, and a z-axis, and the three rotational degrees of freedom of the rider coordinate system include a roll rotation about the x-axis, a pitch rotation about the y-axis, and a yaw rotation about the z-axis; the three translational degrees of freedom of the vehicle coordinate system are along three orthogonal axes including an X-axis, a Y-axis, and a Z-axis, and the three rotational degrees of freedom of the vehicle coordinate system include a roll rotation about the X-axis, a pitch rotation about the Y-axis, and a yaw rotation about the Z-axis; The method according to claim 6.

8. determining a yaw angle of the vehicle relative to the optical axis of the plane mirror using two or more distance sensors positioned adjacent to the vehicle; The method of claim 7 further comprising:

9. determining a lateral position of the vehicle along an axis orthogonal to the optical axis and normal axis of the plane mirror using the two or more distance sensors; The method of claim 8 , further comprising:

10. The method of claim 8 , wherein each of the two or more distance sensors comprises an ultrasonic sensor or a laser sensor.

11. determining the yaw, roll, pitch and translational position of the vehicle along three orthogonal axes in a coordinate system of the plane mirror using six range sensors; The method of claim 7 further comprising:

12. The method of claim 11 , wherein each of the six distance sensors comprises an ultrasonic sensor or a laser sensor.

13. determining the deviation from the expected alignment of the lidar sensor with respect to the vehicle, storing a reference matrix regarding an expected relationship between the lidar sensor and the vehicle; determining a matrix relating to a current relationship between the lidar sensor and the vehicle; determining the deviation from the expected alignment of the lidar sensor with respect to the vehicle by comparing the matrix to a reference matrix; The method of claim 1 , comprising:

14. recalibrating the lidar sensor with respect to the vehicle based on the matrix regarding the current relationship between the lidar sensor and the vehicle. The method of claim 13 further comprising:

15. 1. A method for calibrating a lidar sensor mounted on a vehicle, the method comprising: storing a reference three-dimensional image acquired by the LIDAR sensor while the LIDAR sensor is in an expected alignment with the vehicle, the reference three-dimensional image including a first image of a fixed feature on the vehicle; acquiring a three-dimensional image using the lidar sensor, the three-dimensional image including a second image of the fixed feature; determining a deviation from the expected alignment of the lidar sensor with respect to the vehicle by comparing the second image of the fixed feature in the three-dimensional image to the first image of the fixed feature in the reference three-dimensional image; A method comprising:

16. recalibrating the lidar sensor based on the deviation from the expected alignment of the lidar sensor with respect to the vehicle. The method of claim 15 further comprising:

17. determining a transformation to be applied to the second image of the fixed feature in the three-dimensional image to match the first image of the fixed feature in the reference three-dimensional image; recalibrating the lidar sensor based on the transformation; and The method of claim 15 further comprising:

18. determining that the deviation from the expected alignment of the lidar sensor exceeds a threshold; issuing an alert in response to determining that the deviation from the expected alignment of the lidar sensor exceeds the threshold. The method of claim 15 further comprising:

19. The method of claim 15 , wherein the reference three-dimensional image is acquired by a lidar sensor after the lidar sensor is pre-calibrated at a manufacturing facility.

20. 16. The method of claim 15, wherein the deviations from the expected positional alignment of the lidar sensor include one or more of yaw deviations, roll deviations, pitch deviations, and translation deviations along three orthogonal axes.

21. The method of claim 15 , wherein the fixed feature comprises a portion of a hood of the vehicle or an object attached to the hood.

22. the lidar sensor is disposed behind a windshield of the vehicle; the fixed feature includes a mask attached to an area of ​​the windshield directly in front of the lidar sensor, the mask configured to block light at an operating wavelength of the lidar sensor and shaped to block a portion of the field of view of the lidar sensor; The method of claim 15.

23. 23. The method of claim 22, wherein the mask has an outer boundary and an inner boundary, the inner boundary being sized such that the mask encroaches on a perimeter of the field of view of the lidar sensor.

24. 1. A method for calibrating a lidar sensor mounted on a vehicle, the method comprising: acquiring one or more three-dimensional images using the lidar sensor while the vehicle is traveling along a road having fixed road features, each of the one or more three-dimensional images including an image of the road features; analyzing a spatial relationship between the images of the road features in the one or more three-dimensional images and a field of view orientation of the lidar sensor; determining a deviation from an expected alignment of the lidar sensor relative to the vehicle based on the spatial relationship between the image of the road features and the field of view of the lidar sensor; A method comprising:

25. recalibrating the lidar sensor based on the deviation from the expected alignment of the lidar sensor with respect to the vehicle.

25. The method of claim 24, further comprising:

26. determining that the deviation from the expected alignment of the lidar sensor exceeds a threshold; issuing an alert in response to determining that the deviation from the expected alignment of the lidar sensor exceeds the threshold.

25. The method of claim 24, further comprising:

27. The method of claim 24 , wherein the road features include one or more pairs of lane markings on either side of the vehicle.

28. analyzing the spatial relationship includes determining a pitch angle between a pair of lane markings of the one or more pairs of lane markings and the field of view of the lidar sensor; determining the deviation from the expected alignment of the lidar sensor includes determining a pitch error of the lidar sensor based on the pitch angle.

28. The method of claim 27.

29. the one or more pairs of lane markings include a first lane marking on a driver's side of the vehicle and a second lane marking on a passenger's side of the vehicle; the step of analyzing the spatial relationship includes determining an elevation difference between the first lane marking and the second lane marking; determining the deviation from the expected alignment of the lidar sensor includes determining a roll error of the lidar sensor based on the elevation difference.

28. The method of claim 27.

30. 28. The method of claim 27, wherein the one or more three-dimensional images comprise a plurality of three-dimensional images acquired by the lidar sensor over a time interval as the vehicle travels on the road over a distance.

31. 31. The method of claim 30, wherein the road is substantially straight and level over the distance.

32. determining a route for the vehicle over the distance; comparing the path of the vehicle to paths of the road features from the plurality of three-dimensional images; 31. The method of claim 30, further comprising:

33. the one or more pairs of lane markings include a first lane marking on a driver's side of the vehicle and a second lane marking on a passenger's side of the vehicle; analyzing the spatial relationship includes determining a lateral asymmetry between the first lane marking from the path of the vehicle and the second lane marking from the path of the vehicle; determining the deviation from the expected alignment of the lidar sensor includes determining a yaw error of the lidar sensor based on the lateral asymmetry.

33. The method of claim 32.

34. the step of analyzing the spatial relationship includes determining an elevation difference between a pair of lane markings and the path of the vehicle; determining the deviation from the expected alignment of the LIDAR sensor includes determining a vertical error of the LIDAR sensor based on the elevation difference.

33. The method of claim 32.

Images