Lidar chip with multiple detector arrays

Abstract

To provide lidar and overcome saturation of lidar photodetectors.SOLUTION: A lidar system uses a detector chip having a first column of light sensors and a second column of light sensors, arranged to detect light from pulses of laser light reflected by one or more objects in the environment. The second column of light sensors is arranged to scan a field of view before the first column of light sensors. A processor can adjust power of an emitter, or adjust detector gain of the first column of light sensors, in response to data from the second column of light sensors scanning the field of view, so that light in a dynamic range of the first column of sensors can be used to generate data for a three-dimensional point cloud.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 621,922, filed January 17, 2024, which is incorporated by reference in its entirety for all purposes. [Background technology]

[0002] Three-dimensional sensors can be applied to autonomous vehicles, drones, robots, security applications, and the like. For example, lidar projects a light beam and detects light from the light beam reflected by one or more objects in an environment. Lidar can be used to create a three-dimensional map of an environment or portion thereof based on detecting the reflected light from the light beam. Scanning lidar sensors can provide high angular resolution suitable for such applications at an affordable cost. An example of a scanning lidar system is described in U.S. Patent Application No. 10,690,754, which was granted on June 23, 2020, and is incorporated by reference for all purposes. However, improved scanning systems, apparatus, and / or methods are desired. Summary of the Invention [Means for solving the problem]

[0003]

[0003] The present disclosure relates to, but is not limited to, overcoming saturation in lidar and lidar photodetectors.

[0004]

[0004] In a particular configuration, the LIDAR system comprises an emitter arranged to emit light pulses; a mirror arranged to reflect the light pulses into an environment; a detector arranged to detect light from the light pulses reflected by one or more objects in the environment, wherein the detector comprises a first array of light sensors and a second array of light sensors, the second array of light sensors arranged to scan a field of view in front of the first array of light sensors; and / or a memory device comprising instructions that, when executed by one or more processors, cause the one or more processors to adjust an output of the emitter in response to data from the second array of light sensors scanning the field of view, or adjust a detector gain of the first array of light sensors, and / or generate three-dimensional point cloud data, the point cloud including at least one data point of one or more objects in the environment based on light detected by one or more light sensors in the first array of light sensors.

[0005]

[0005] In a particular configuration, the lidar system comprises an emitter arranged to emit light pulses; a mirror arranged to reflect the light pulses into an environment; a detector arranged to detect light from the light pulses reflected by one or more objects in the environment, wherein the detector comprises a first array of light sensors and a second array of light sensors, the first array of light sensors arranged to detect a different intensity of light than the second array of light sensors; and a memory device comprising instructions that, when executed by one or more processors, cause the one or more processors to generate three-dimensional point cloud data, the point cloud including at least one data point of one or more objects in the environment based on the light detected by the detectors.

[0006]

[0006] In some configurations, the different intensities are caused by the emitter reducing the output of the light pulse after the second string of light sensors detects light above a threshold, the different intensities are caused by the emitter increasing the output of the light pulse after the second string of light sensors detects light below a threshold, the different intensities are caused by one or more filters in front of the first string of light sensors and / or in front of the second string of light sensors, the different intensities are caused by the first string of light sensors being less sensitive to light or by a change in gain of an amplifier receiving signals from the first string of light sensors, the detector comprises a third string of light sensors, the first string of light sensors is between the second string of light sensors and the third string of light sensors, the first string of light sensors is positioned to detect a different intensity of light than the third string of light sensors, the mirror is positioned to rotate or oscillate, the instructions cause the one or more processors to use pixel data from one or more strings of light sensors having a desired data range, and the different intensities are caused by decreasing or increasing the gain of the first string of light sensors compared to the gain of the second string of light sensors. the emitter comprises a first row of lasers and a second row of lasers, the different intensities being caused by the first row of lasers emitting light at a different power than the second row of lasers; the detector further comprises four or more rows of light sensors, each row of light sensors detecting a different intensity of light; the detector further comprises an image sensor arranged to create a two-dimensional color image, the image sensor being on the same chip as the first row of light sensors and the second row of light sensors, and the image sensor, the first row of light sensors, and the second row of light sensors sharing a common lens. the image sensor comprises three rows of light sensors with three different color filters, the system comprises the image sensor on a chip separate from the detector, the system comprises a beam splitter that sends a first portion of the light to the image sensor and a second portion of the light to the detector, the mirror is arranged to scan the light pulses in a direction orthogonal to a direction of the first row of light sensors, the mirror is a first mirror, the system comprises a second mirror, the second mirror is arranged to scan the light pulses in a direction parallel to the direction of the first row of light sensors, and generating the three-dimensional point cloud dataadding data from a first array of light sensors to data from a second array of light sensors, the second array of light sensors being positioned to detect stray light emitted toward and received by the first array of light sensors, and / or the detector comprising a third array of light sensors for detecting thermal light from the environment;

[0007]

[0007] In a particular configuration, a LIDAR method comprises the steps of emitting light pulses using one or more lasers; reflecting the light pulses into an environment using a mirror; detecting light from the light pulses reflected by one or more objects in the environment using a detector, wherein the detector comprises a first array of light sensors and a second array of light sensors, and the first array of light sensors detects light of a different intensity than the second array of light sensors; and generating three-dimensional point cloud data based on the light detected by the detector, wherein the three-dimensional point cloud includes at least one data point of one or more objects in the environment.

[0008]

[0008] In certain embodiments, the detector chip includes a single row of photodetectors (e.g., IR filters) for detecting reflected laser light for the lidar and one, two, or three rows of photodetectors (e.g., red, green, and blue filters) for imaging ambient light on the same chip.

[0009]

[0009] Further scope of applicability of the present disclosure will become apparent from the following detailed description. It should be understood that the detailed description and specific examples, while indicating various embodiments, are intended for purposes of illustration only and are not intended to necessarily limit the scope of the present disclosure. [Brief explanation of the drawings]

[0010]

[0010] The present disclosure is described in conjunction with the accompanying drawings.

[0011] [Figure 1] 1 illustrates an embodiment of a lidar sensor for three-dimensional imaging. [Figure 2]1 illustrates an embodiment of a lidar sensor with a turning mirror and a routing mirror. [Figure 3] 1 illustrates an embodiment of a chip for lidar sensing. [Figure 4] 1 illustrates an embodiment of a lidar system with an emitter and detector that uses arrays. [Figure 5] 1 illustrates an embodiment of a chip with multiple rows of photodetectors. [Figure 6] 6 illustrates the chip of FIG. 5 incorporated into one embodiment of a lidar system. [Figure 7] 1 illustrates an embodiment of a chip including a lidar array and an image sensor array. [Figure 8] 1 illustrates an embodiment of a system having a lidar chip that is separate from the camera chip. [Figure 9] 1 shows a flowchart of one embodiment of a process for a rider.

[0020] In the accompanying figures, similar components and / or functions may have similar reference labels. Furthermore, various components of the same type may be distinguished by following the reference label with a dash and a second label that distinguishes the similar components. When only a first reference label is used in the specification, the description applies to any similar component having the same first reference label, regardless of the second reference label. DETAILED DESCRIPTION OF THE INVENTION

[0012]

[0021] The following description provides only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of preferred exemplary embodiments will provide those skilled in the art with an effective description for implementing the preferred exemplary embodiments. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope as set forth in the appended claims.

[0013]

[0022] Lidar sensors are used to create a three-dimensional image of the object space within the sensor's field of view (FOV). One common challenge in generating accurate images is that some common objects found in driving scenes, especially retro-reflective objects such as road signs and license plates, can appear 1,000 times brighter than normal objects when actively illuminated by the sensor. This can saturate the photodetectors and associated electronics. Also, small amounts of stray light from imperfections (such as lens imperfections, windshield dust, or reflections from internal components such as the lens barrel) can cause false positives around the reflector, resulting in "blooming," which makes the reflector appear much larger than it actually is. Such blooming can be particularly problematic, for example, when an image of a speed limit sign blooms into the driving lane, causing the vehicle to brake because it believes there is an obstacle on the road when in fact there is none.

[0014]

[0023] A possible architecture for an imaging lidar system includes a 1D array of sensors integrated on a silicon integrated circuit ("IC" or "chip"). Illumination can be a vertical line of laser light (e.g., a 1D array of lasers). The laser line and returning photons are scanned synchronously (e.g., by separate scanning optics or together by the same scanning optics), which causes reflected laser photons to return from one or more objects within the lidar system's field of view and focus onto corresponding detectors. Multiple rows of photodetectors may be used to expand the dynamic range of detection, allowing imaging of bright as well as dark objects. Multiple rows may also be used to provide additional imaging modes, such as color camera images or thermal infrared images.

[0015]

[0024] FIG. 1 illustrates an embodiment of a lidar sensor 100 for three-dimensional imaging. The lidar sensor 100 comprises an emitting lens 130 and a receiving lens 140. The lidar sensor 100 includes a light source 110-a disposed substantially at the back focal plane of the emitting lens 130. The light source 110-a operates to emit light pulses 120 from respective emission positions within the back focal plane of the emitting lens 130. The emission lens 130 is configured to collimate and direct the light pulses 120 toward an object 150 located in front of the lidar sensor 100. For a given emission position of the light source 110-a, the collimated light pulses 120' are directed toward the object 150 at a corresponding angle.

[0016]

[0025] Portions 122 of the collimated light pulses 120' are reflected from the object 150 and travel toward the receiving lens 140. The receiving lens 140 is configured to focus the portions 122' of the light pulses reflected from the object 150 onto corresponding detection locations within the focal plane of the receiving lens 140. The lidar sensor 100 further includes a detector 160-a positioned substantially in the focal plane of the receiving lens 140. The detector 160-a is configured to receive and detect the portions 122' of the light pulses 120 reflected from the object at the corresponding detection locations. The corresponding detection locations of the detector 160-a are optically conjugate with the respective light emission locations of the light sources 110-a.

[0017]

[0026] The light pulses 120 may be short in duration, e.g., have a pulse width of 10 ns. The lidar sensor 100 further includes a processor 190 coupled to the light source 110-a and the detector 160-a. The processor 190 is configured to determine a time of flight (TOF) from emission to detection of the light pulses 120. Because the light pulses 120 travel at the speed of light, the distance between the lidar sensor 100 and the object 150 may be determined based on the determined time of flight.

[0018]

[0027] One way to scan a laser beam (e.g., light pulses 120′) across the field of view is to move light source 110-a laterally relative to light-emitting lens 130 within the back focal plane of light-emitting lens 130. For example, light source 110-a may be raster-scanned to multiple light-emitting positions within the back focal plane of light-emitting lens 130 as shown in FIG. 1 . Light source 110-a may emit multiple light pulses at the multiple light-emitting positions. Each light pulse emitted at each emission position is collimated by light-emitting lens 130, directed toward object 150 at a respective angle, and directed to and impinges on a corresponding point on the surface of object 150. Thus, when light source 110-a is raster-scanned within a certain region within the back focal plane of light-emitting lens 130, a corresponding object region on object 150 is scanned. As shown in FIG. 1 , detector 160-a may be raster-scanned to be positioned at multiple corresponding detection positions within the focal plane of receiving lens 140. Typically, the scanning of detector 160-a is performed synchronously with the scanning of light source 110-a, so that detector 160-a and light source 110-a are always optically conjugate to each other at any given time.

[0019]

[0028] By determining the time of flight of each light pulse emitted at each emission position, the distance from the lidar sensor 100 to each corresponding point on the surface of the object 150 can be determined. In some embodiments, the processor 190 is coupled to a position encoder that detects the position of the light source 110-a at each emission position. The angle of the collimated light pulse 120' can be determined based on the emission position. The X and Y coordinates of the corresponding point on the surface of the object 150 can be determined based on the angle and the distance to the lidar sensor 100. Thus, a three-dimensional image of the object 150 can be constructed based on the measured distances from the lidar sensor 100 to various points on the surface of the object 150. In some embodiments, the three-dimensional image can be represented as a point cloud, i.e., a collection of X, Y, and Z coordinates of points on the surface of the object 150.

[0020]

[0029] In some embodiments, the intensity of the returned light pulse 122' is measured and used to adjust the power of subsequent light pulses from the same emission point to prevent detector saturation, improve eye safety, or reduce overall power consumption. The power of the light pulse can be changed by varying the duration of the light pulse, the voltage or current applied to the laser, or the charge accumulated on a capacitor used to power the laser. In the latter case, the charge stored on the capacitor can be changed by changing the charging time, charging voltage, or charging current to the capacitor. In some embodiments, the reflectance, determined by the intensity of the detected pulse, can also be used to add another dimension to the image. For example, the image can include X, Y, and Z coordinates as well as reflectance (or brightness).

[0021]

[0030] The angle of view (AFOV) of the lidar sensor 100 may be estimated based on the scanning range of the light source 110-a and the focal length of the light-emitting lens 130,

number

[0022]

[0031] The light source 110-a may be configured to emit light pulses in the near-infrared wavelength range. The energy of each light pulse may be on the order of microjoules and is typically considered eye-safe at repetition rates in the kHz range. For light sources operating at wavelengths above approximately 1500 nm (the near-infrared wavelength range), the energy levels may be higher because the eye does not focus on those wavelengths. The detector 160-a may be composed of a silicon avalanche photodiode, a photomultiplier tube, a PIN diode, or other semiconductor sensor.

[0023]

[0032] FIG. 2 illustrates one embodiment of a lidar system 200 having a rotating mirror 204 and a routing mirror 208. The rotating mirror 204 may be a polygonal mirror with a reflective surface 212. The rotating mirror 204 rotates about a vertical axis 216 (e.g., rotates in a complete circle). Light is reflected by the rotating mirror 204 into a field of view (FOV) 220. The FOV 220 has a horizontal 222 component and a vertical 224 component. For example, the lidar system can be positioned on a vehicle so that the vertical 224 component of the FOV 220 is in the direction of gravity and the horizontal 222 component of the FOV 220 is perpendicular to the direction of gravity. While the rotating mirror 204 is shown as a spinning mirror, in some embodiments, the rotating mirror rotates back and forth about an axis in an oscillatory motion.

[0024]

[0033] The rotating mirror 204 is used to horizontally scan one or more laser beams (e.g., pulsed laser beams) within the FOV 220 of the lidar system 200. The rotating mirror 204 has a large number of mirror facets of equal planar size fabricated on a rotating rotor. While the number of facets 212 shown in FIG. 2 is six, the number of facets 212 can be two, three, or four or more, and / or four, five, six, seven, eight, ten, or twelve or fewer. Spinning mirrors, such as the rotating mirror 204 shown in FIG. 2, have low vibration, low power requirements, and linear scanning characteristics. In imaging lidar applications, the rotating mirror 204 scans horizontally. To achieve high resolution in other (e.g., vertical) directions, a vertical array of lasers (e.g., a large vertical array) and / or a galvo mirror may be used. The lidar system 200 shown in FIG. 2 includes a laser 228 and a detector 232. Although FIG. 2 shows only one laser 228 and one detector 232, it should be understood that multiple lasers 228 and / or detectors 232 may be used (e.g., a laser array and / or detector array may be used).

[0025]

[0034] In some embodiments, a rotating polygon mirror is used to scan one or more lasers 228 horizontally, and a routing mirror 208 is used to fold the beam path to create a more compact layout. The routing mirror 208 in FIG. 2 is also used to vertically position the light pulses from the lasers 228 by dynamically tilting them back and forth in the vertical direction (e.g., about a horizontal axis). This positioning can achieve high vertical resolution while reducing and / or minimizing the number of lasers and / or can also make the lidar system 200 more compact. As shown in FIG. 2, the routing mirror 208 can effectively increase the number of scan lines 236 in the vertical 224 component of the FOV 220. This can be done using a single laser and / or a vertical laser array. Each dot on the scan line 236 represents a laser pulse.

[0026]

[0035] In some embodiments, the scan lines 236 are in discrete vertical steps. For example, the routing mirror 208 rotates at discrete positions for each scan line 236 (e.g., rather than the routing mirror 208 rotating continuously). Continuous rotation of the routing mirror 208 would cause the scan lines to bend, preventing the laser pulses from emitting in a straight horizontal line. When using only one laser 228, curved scan lines for the laser pulses may not pose much of a problem. However, when using multiple lasers, using straight scan lines can help create a scan pattern with a desired data point density within the FOV 220 (e.g., spacing the light pulses from the lasers in a desired density pattern).

[0027]

[0036] In some embodiments, a lidar system (e.g., lidar system 200) includes an illumination source including multiple lasers (e.g., an array of lasers including laser 228) and a mirror system (e.g., including a rotating mirror 204 and a routing mirror 208). The mirror system is positioned to reflect light from the illumination source into the environment in a field of view (e.g., FOV 220). The mirror system includes a mirror (e.g., rotating mirror 204) that is positioned to rotate to reflect light from the illumination source and horizontally scan the light from the multiple lasers within the field of view of the system. The mirror system is configured to reflect light from the illumination source (e.g., using routing mirror 208) and vertically scan the light from the multiple lasers in discrete vertical steps within the field of view (e.g., scan lines 236 separated by discrete vertical steps). The scan lines 236 are straight (e.g., horizontal).

[0028]

[0037] Additional lidar sensors are described in U.S. patent application Ser. Nos. 15 / 267,558, filed September 15, 2016, 15 / 971,548, filed May 4, 2018, 16 / 504,989, filed July 8, 2019, 16 / 775,166, filed January 28, 2020, 17 / 032,526, filed September 25, 2020, 17 / 133,355, filed December 23, 2020, 17 / 205,792, filed March 18, 2021, 17 / 380,872, filed July 20, 2021, and 18 / 531,507, filed December 6, 2023, the disclosures of which are incorporated by reference for all purposes.

[0029]

[0038] FIG. 3 illustrates a simplified embodiment of a lidar sensing chip 300. The chip 300 is comprised of an array of photodetectors 304. The array of photodetectors 304 is sometimes referred to as an array, and in this case is a one-dimensional array. The photodetectors 304 are optical sensors. Two-dimensional scanning of the field of view can be achieved by horizontally (and / or vertically) scanning the field of view (FOV) of the photodetectors 104 using, for example, a scanning mirror, such as a galvanometer mirror (“galvo”) and / or a rotating polygon mirror (e.g., as shown in FIG. 2 ). One or more lasers are positioned to provide light pulses that are scanned across the FOV (e.g., by the same scanning mirror), thereby illuminating objects in the environment to be imaged by the lidar detector (e.g., objects within the FOV of the lidar system).

[0030]

[0039] The photodetectors 304 for lidar are typically infrared detectors, although other wavelengths may be used. The photodetectors 304 may be simple photodiodes, avalanche photodiodes (APDs), single-photon detectors (SPADs), or arrays of SPADs called silicon photomultipliers (SiPMs). Each photodetector 304 is coupled to a time-of-flight (ToF) detector via an optional amplifier (typically a transimpedance amplifier). The ToF detector determines the time between the laser pulse and the detection event to calculate the distance from the lidar sensor to the detected object. Lidar systems use an active laser illumination system that emits short (e.g., nanosecond-scale) laser light pulses. Each photodetector may be illuminated by its own laser, a group of lasers, or the entire array of detectors may share a single laser. Examples of lasers include fiber-coupled lasers, edge-emitting lasers (EELs), and vertical-cavity surface-emitting lasers (VCSELs).

[0031]

[0040] A transimpedance amplifier (TIA) 308 can amplify the signal from the photodetector 304. The amplified signal is then sent to a time-of-flight (ToF) detector 312, which calculates the distance to the reflection. An image processing circuit 316 is used to calculate three-dimensional point cloud data of the environment within the FOV of the lidar sensor, which can be used to generate lidar image data. In some embodiments, the memory device includes instructions that, when executed by one or more processors (e.g., the image processing circuit 316 and / or the ToF detector 312), direct the one or more processors to generate three-dimensional point cloud data including at least one data point for one or more objects in the environment based on the light detected by the detector. The chip 100 can be used as a detector in a lidar system (e.g., detector 232 in FIG. 2).

[0032]

[0041] 4 shows an embodiment of a lidar system 400 having an emitter 404 and a detector 408 that use arrays. The lidar system 400 is comprised of a turning mirror 204 and a routing mirror 208. The emitter 404 is comprised of a one-dimensional laser array arranged to emit light pulses. The routing mirror 208 and the turning mirror 204 are positioned to reflect the light pulses into an environment within the FOV 220 of the lidar system 400. The detector 408 (e.g., including the chip 300) is positioned to detect light from the light pulses reflected by one or more objects in the environment.

[0033]

[0042] The rotating mirror 204 (e.g., the first mirror) is positioned to scan the light pulses horizontally, or in a direction perpendicular to the direction of the sensor columns. For example, the columns of photodetectors 304 in chip 300 of FIG. 3 are vertically arranged within detector 408 of FIG. 4, and the rotating mirror 204 scans the light pulses from the photodetectors horizontally along scan line 426. The vertical resolution 430 can be determined by the size and / or pitch of the vertical array of photodetectors 304 on chip 300. The horizontal resolution 432 can be determined by the scanning speed of the rotating mirror 204 and / or the firing rate of the emitters 404.

[0034]

[0043] In some embodiments, the routing mirror 208 (e.g., the second mirror) is positioned to scan the light pulses vertically (e.g., in a direction parallel to the direction of the first sensor column). For example, the light pulses are vertically scanned up, down, and / or alternating scan lines 426 shown in FIG. 4. While FIG. 4 shows six laser arrays and six photodetector arrays, other numbers of lasers and photodetectors per column in each array can be used. For example, there can be 2, 5, 10, 20, 30, 40, or 50 or more components per column, and / or there can be 40, 70, 100, 256, or 512 or fewer components per column. The chip size, component size, and / or on-chip electronics can limit the number of components in a column or array.

[0035]

[0044] Some lidars have the ability to reduce the intensity of the laser pulse when a bright object, such as a reflector, enters the field of view to prevent the detector from saturating or blinding. However, in systems such as those shown in Figures 3 and 4, if a retroreflective object is not known in advance to be introduced into the FOV (e.g., within one scan line 426), reducing the laser intensity is a reactive, not proactive, action. By the time a bright object is detected and the laser power is reduced, one or more frames of the image may already have compromised data.

[0036]

[0045] FIG. 5 illustrates one embodiment of a chip 500 including multiple columns 504 of photodetectors 304. The chip 500 is shown including a first column 504-1 of photodetectors 304, a second column 504-2 of photodetectors 304, and a third column 504-3 of photodetectors 304. While three columns 504 are shown in FIG. 5, there may be fewer (e.g., two) or more (e.g., four, five, six, or more) columns. The chip 500 includes a TIA 308, a ToF detector 312, and image processing circuitry 316.

[0037]

[0046] 6 shows an embodiment of a chip 500 incorporated into an embodiment of a lidar system 600. The lidar system 600 is comprised of an emitter 404, a detector 608, a turning mirror 204, and a routing mirror 208. The detector 608 is comprised of a chip 500. The chip 500 includes a first row 504-1 of photodetectors 304 and a second row 504-2 of photodetectors 304.

[0038]

[0047] The second row 504-2 of the photodetectors 304 scans the FOV 220 before the first row 504-1 of the photodetectors 304. The first row 504-1 and second row 504-2 of the photodetectors scan over the same or approximately the same FOV 220. For example, the FOV scanned by the first row 504-1 overlaps with the FOV scanned by the second row by 90%, 95%, 97%, 98%, 99%, or 100% or more. In some embodiments, the difference is 5, 3, 2, or 1 pixel width or less.

[0039]

[0048] 6 illustrates pixels in a first region 611 in the FOV 220 imaged by a first row of sensors 504-1 and pixels in a second region 612 in the FOV 220 imaged by a second row of sensors 504-2 at the same time the pixels in the first region 611 are imaged by the first row of sensors 504-1. The first row of photodetectors 304 504-1 may detect a different intensity of light than the second row of photodetectors 304 504-2. For example, when initially scanning the second row of photodetectors 304 504-2 (e.g., region 612), if the lidar system 600 detects an overly bright object by the second row of photodetectors 304 504-2, this allows the processor time to adjust the laser illumination pulse power to prevent saturation of the first row of photodetectors 304 504-1. In some circumstances, the second row of photodetectors 304 may detect light below a threshold power, causing the system to increase laser power so that more intense light is detected by the first row 504-1 of photodetectors 304. Thus, the second row 504-2 may be used first to scan objects in a scene, after which the first row 504-1 is scanned, and the processor may use information from one or more photodetectors 504 in the second row 504-2 to adjust (e.g., increase or decrease) the laser power and / or detector gain of the first row 504-1. In some configurations, the different rows of detectors 504 have different filters (e.g., different transmittance values for neutral density filters) and / or different sensitivity values for the photodetectors 304.

[0040]

[0049] In some systems that use an oscillating mirror rather than a rotating mirror, the scan direction may be reversed periodically. In this case, the processor can determine the laser power based on which row scanned the FOV 220 first, so that rows scanning later objects use laser power optimized for illumination. A third row of detectors (e.g., 504-3 in FIG. 5) may be added opposite the second row of detectors 504-2 so that either the second row 504-2 or the third row 504-3 can be used to detect overly bright objects, depending on the scan direction.

[0041]

[0050] In some configurations, a less sensitive detector or a lower-gain amplifier may be used for the second column of detector 504-2. This allows for non-saturating image data even when the laser power remains high. The processor can select pixel data from photosensors in one or more columns 504 having the desired data range (e.g., below a high threshold and / or above a low threshold). The third column 504-3 may have a less sensitive detector or a lower-gain amplifier, and / or the second column 504-3 may be located between the first column 504-1 and the third column 504-3. Furthermore, rather than (or in addition to) reducing the laser power, the processor may electronically reduce the gain of the front-end detection system through techniques such as variable gain amplifiers or photodetector bias reduction. In some configurations, different laser powers are emitted for detection by different columns of detectors, the laser intensity is cycled, and the system selects data from one or more columns 504 (e.g., the columns that are not saturated).

[0042]

[0051] In some configurations, the laser illumination may be arranged to primarily illuminate the first row 504-1 and the second row 504-2 at a much lower intensity. Thus, bright objects are less likely to saturate the photosensors in the second row 504-2. In some configurations, a second set of lasers may be used to individually illuminate the second row 504-2 with a higher or lower laser power than the laser power used to illuminate the first row 504-1.

[0043]

[0052] In some embodiments, the array of detectors may be a two-dimensional array. One or more columns 504 may be selected to provide prior data for setting the laser power. The selected columns may be configured according to the brightness of the object being imaged. In some configurations, the laser intensity may be configured to decrease (e.g., gradually or stepwise) across multiple columns of photodetectors. The processor may select a column or columns having an appropriate illumination level for the object within the FOV 220. Also, note that "row" may be substituted for "column" and "horizontal" for "vertical" without changing the nature of the invention. For example, scanning may be vertical rather than horizontal, and photodetectors may be arranged in rows rather than columns.

[0044]

[0053] In some configurations, the different intensities at the light sensors are caused by the emitter reducing the output of the light pulse after the second row of light sensors detects light above a threshold; the different intensities can be caused by one or more filters before the first row of light sensors or the second row of light sensors; the different intensities can be caused by the second row of light sensors having a lower sensitivity or a lower gain amplifier than the first row of light sensors; the detector includes a third row of light sensors, the first row of light sensors is between the second row of light sensors and the third row of light sensors, the first row of light sensors detects a different intensity of light than the third row of light sensors; the mirror is arranged to oscillate or rotate (e.g., rotate 360 degrees); the instructions cause the one or more processors to use pixel data from one or more rows of light sensors having a desired data range (e.g., not saturated); and the different intensities can be caused by comparing the gain of the first row of light sensors with the gain of the second row of light sensors. the emitter comprises a first row of lasers and a second row of lasers, and the different intensities are caused by the first row of lasers emitting light at a different power than the second row of lasers; the detector comprises four or more rows of light sensors, each row of light sensors detecting light at a different intensity than the other rows of light sensors; the detector comprises three, four, or more rows of light sensors, each row of light sensors detecting light at a different wavelength than the other rows of light sensors (e.g., a light sensor having four rows (R, G, B, and IR) for detecting ambient light on the same chip as one, two, three, or more rows used to detect reflected laser light, and also a light sensor with an IR filter for IR ambient light that does not overlap with the bandwidth of light used by the emitter); and generating the 3D point cloud data includes adding data from an adjacent row of light sensors, and / or a second row of light sensors positioned to detect stray light emitted toward and received by the first row of light sensors.

[0045]

[0054] To improve object detection and / or provide redundancy, lidar sensor data can be combined with data from other sensors, such as cameras. Color data can also be important for detecting traffic lights and brake lights. One challenge with such sensor fusion is that perfectly overlaying the lidar and camera images can be difficult due to different lens distortion characteristics. To further complicate the issue, protective covers (in some cases, a car's windshield) can further introduce different image distortion between the lidar sensor and the camera, and this distortion can change when the cover (or windshield) is replaced during a service event.

[0046]

[0055] FIG. 7 illustrates one embodiment of a chip 700 including a lidar array 704 and an image sensor array 708. Integrating the camera sensor and lidar sensor into a single sensing device (e.g., a silicon integrated circuit) and using a common lens for imaging can reduce or completely avoid the difficulties of overlaying the lidar and camera images. FIG. 7 illustrates an example silicon sensor incorporating a 1D array of photodetectors for lidar and a second array of photodetectors for camera imaging. The image sensor array 708 can be a single row of detectors for black-and-white imaging, a set of three or more rows of detectors for color imaging, or a combination of color sensors on a single row. The lidar sensor array 704 can be a single row (e.g., as described in connection with FIG. 3) or multiple rows (e.g., as described in connection with FIG. 5). In FIG. 7, the image sensor array 708 is used to image light from the FOV using red (R), green (G), and blue (B) sensitive photodetectors, although other schemes can be used.

[0047]

[0056] 7, the lidar sensor array 704 and image sensor array 708 are physically close enough to share a common lens and scanning mechanism, and the offset between the images is small enough to be removed by calibration and / or image processing. In some configurations, multiple lidar photodetector arrays are used in combination with one or more camera photodetector arrays.

[0048]

[0057] In some configurations, the detector further comprises an image sensor (e.g., image sensor array 708) arranged to create a two-dimensional color image, where the image sensor is on the same chip as the first and second rows of light sensors. For example, chip 700 can be used as part of detector 608 in FIG. 6. The image sensor, the first row of light sensors, and the second row of light sensors can share a common lens (e.g., a receiving lens). The image sensor can consist of three rows of light sensors with three different color filters (labeled R, G, and B in FIG. 7).

[0049]

[0058] 8 shows an embodiment of a lidar system with a lidar chip 804 and a camera chip 808 on two separate chips. A beam splitter 812 (e.g., high pass IR, highly reflective to visible light) is used to direct the infrared light to the lidar chip 804 and the visible light to the camera chip 808.

[0050]

[0059] The camera chip 808 has a corresponding array of “ambient light” or visible light detectors. Ambient light detectors are not positioned to image light from laser reflections off objects, but are used to image objects in a scene using light that occurs naturally in the scene (such as the sun) or light already occurring in the scene (such as car headlights and / or street lamps). Typically, the array of sensors on the camera chip 808 (e.g., multiple sensors in a row) may match the array of sensors on the lidar chip 804 one-to-one, and there may be more or fewer sensors on the camera chip 808, resulting in higher or lower resolution camera images relative to the lidar image. For full-color images, there may be three or more rows of sensors, with color filters placed over each row. Figure 7 shows three rows of sensors: one for red, one for green, and one for blue. The sensors may be simple photodiodes, avalanche photodiodes (APDs), single-photon detectors (SPADs), and / or SPAD arrays called silicon photomultipliers (SiPMs). The output of the photodetector is then amplified, digitized, and / or processed to form a camera image. In some configurations, the system includes an image sensor (e.g., camera chip 808 in FIG. 8) on a chip separate from the lidar detector (e.g., lidar chip 804 in FIG. 8) and / or a beam splitter (e.g., beam splitter 812) positioned to send a first portion of the light to the image sensor (e.g., camera chip 808) and a second portion of the light to the detector (e.g., lidar chip 804). The pixels on the lidar chip 804 and camera chip 808 can be synchronized in both angular and timing space.

[0051]

[0060] 8, a narrowband IR filter 820 is positioned between the beam splitter 812 and the lidar chip 804, and / or a color control mask 824 is positioned between the beam splitter 812 and the camera chip 808. The beam splitter 812 is positioned (optically) between the receive optics 828 and the lidar chip 804, and the beam splitter is positioned (optically) between the receive optics 828 and the camera chip 808.

[0052]

[0061] Thermal infrared detectors may be used in addition to or instead of visible light sensors and can be advantageous for detecting people, animals, and vehicles at night. In some configurations, the chip is configured with a row of light sensors (e.g., a third row of light detectors 504-3 for detecting thermal light from the environment).

[0053]

[0062] In some embodiments, the number of vertical pixels, or the vertical FOV, may be less than desired due to the size and / or cost of the silicon chip used to support so many pixels. To improve vertical resolution and / or FOV, it may be desirable to add vertical and / or horizontal scanning. In some implementations, the chip is positioned to cover half of the vertical FOV (e.g., a linear array of lasers and / or a linear array of sensors on the chip is positioned to cover half of the vertical FOV). After a scan that images the lower part (e.g., the lower half) of the FOV, a mirror or other vertical scanning device deflects the image so that a subsequent scan covers the upper part (e.g., the upper half) of the FOV. For example, the routing mirrors shown in FIGS. 2, 4, and 6 may be configured to rotate or scan vertically (e.g., as disclosed in commonly owned U.S. patent application Ser. No. 18 / 531,507, filed December 6, 2023, which is incorporated by reference for all purposes). The rotating polygon mirror (spinning mirror, oscillating mirror, or galvo mirror) in FIG. 2 may incorporate vertical pivoting as well as horizontal scanning (e.g., as disclosed in commonly owned U.S. Patent Application No. 18 / 200,457, filed May 22, 2023, which is incorporated by reference for all purposes). In some implementations, the upper and lower scans of the FOV may abut each other (e.g., with pixel spacing similar to the spacing between sensors). In some implementations, they may overlap to provide a central region with higher resolution than the upper and lower regions. In some configurations, different mirror facets of the rotating polygon may have different angles in the vertical direction, each angle directing light from a different vertical portion of the FOV to the detector array. In some implementations, the detectors are arranged in a sparse matrix with gaps between each detector. After each horizontal scan pass, a vertical mirror or other vertical scanning device slightly changes the imaging angle, allowing subsequent horizontal scan passes to fill in the gaps between pixels in the first horizontal scan.

[0054]

[0063] To increase detection sensitivity and share the same laser pulse, signals from two or more adjacent detector rows may be summed. This summing may be done in either the analog or digital domain. Detectors may share a single laser, or each detector may use a different detector. In some implementations, detectors may use two different laser pulses from the same laser or separate lasers separated in time by a time delay depending on the horizontal scan rate, so that each detector sees the same location in the FOV when the corresponding laser is fired. For SPAD detectors, a histogram approach can be used, in which multiple laser pulses are sent to each pixel, and an object is deemed detected if a number of pulses above a threshold are detected in a particular time slot. Combining multiple rows can reduce the number of laser pulses or increase the probability of reaching the detection threshold, effectively increasing sensitivity.

[0055]

[0064] In some embodiments, the power of the laser is modulated from a first row of detectors to a second row of detectors, the second row of detectors is less sensitive to light than the first row of detectors, the gain is variable and / or different gains can be used for detectors in different rows, the second row of detectors can be used to detect stray light from the first row, detectors can be used to image ambient light, one or more rows of detectors can be used for thermal imaging, and / or scanning can be performed vertically as well as horizontally.

[0056]

[0065] 9, a flowchart of one embodiment of a process 900 for LIDAR is shown. Process 900 begins in step 904 with emitting pulses of light using one or more lasers, such as light from laser 228 in FIG. 2 or light from emitter 404 in FIG. 4 or 6.

[0057]

[0066] In step 908, the light pulse is reflected back into the environment using one or more mirrors. For example, mirrors 204 and 208 in FIG. 2 reflect the light into FOV 220.

[0058]

[0067] In step 912, light from the light pulses reflected by one or more objects in the environment is detected using a detector. The detector may comprise a multiple detector array. For example, the detector may include a first row of light sensors and a second row of light sensors, where the first row of light sensors detects light at a different intensity (e.g., sensitivity) than the second row of light sensors. For example, chip 500 of FIG. 5 , chip 700 of FIG. 7 , or lidar chip 804 of FIG. 8 may be used to detect IR light reflected in the environment from one or more objects in the environment. In some embodiments, the second row of light sensors scans a field of view in front of the first row of light sensors, and one or more processors adjust (e.g., increase or decrease) the output of the emitter or adjust the detector gain of the first row of light sensors in response to data from the second row of light sensors scanning the field of view (e.g., light above or below a threshold detected by one or more sensors in the second row).

[0059]

[0068] In step 916, a three-dimensional point cloud is generated based on the light detected by the detector. The point cloud includes at least one data point for one or more objects in the environment. In some configurations, a two-dimensional image is generated based on the three-dimensional point cloud data. For example, the lidar image data of FIG. 5 is generated from the image processing circuitry 316. In some configurations, the two-dimensional image is displayed to a user.

[0060]

[0069] Various features described herein, e.g., methods, apparatus, computer-readable media, etc., can be implemented using a combination of dedicated components, programmable processors, and / or other programmable devices. Some processes described herein can be implemented on the same processor or different processors. Where components are described as being configured to perform certain operations, such configuration can be achieved, for example, by designing electronic circuitry to perform the operations, by programming programmable electronic circuitry (e.g., a microprocessor) to perform the operations, or a combination thereof. Furthermore, while the above-described embodiments refer to specific hardware and software components, those skilled in the art will understand that different combinations of hardware and / or software components may also be used, and that certain operations described as being implemented in hardware may also be implemented in software, or vice versa.

[0061]

[0070] Although details are given in the above description to provide an understanding of the embodiments, it will be understood that the embodiments may be practiced without some of the specific details. In some instances, well-known circuits, processes, algorithms, structures, and techniques may not be shown in the figures.

[0062]

[0071] While the principles of the present disclosure have been described above in connection with specific apparatus and methods, it should be understood that this description is made by way of example only and is not intended to limit the scope of the disclosure. The embodiments have been chosen and described in order to explain the principles and practical applications of the invention and to enable those skilled in the art to utilize the invention in various embodiments and with various modifications to suit the particular uses envisioned. It should be understood that this description is intended to cover all modifications and equivalents.

[0063]

[0072] It should also be noted that the embodiments may be described as a process that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. While a flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. The order of operations may also be changed. A process terminates when an operation is completed, but there may be additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0064]

[0073] The use of "a," "an," or "the" is intended to mean "one or more," unless specifically indicated to the contrary. All patents, patent applications, publications, and descriptions referred to herein are incorporated by reference in their entirety for all purposes. None are admitted as prior art.

[0065]

[0074] The specific details of particular embodiments may be combined in any suitable manner without departing from the spirit and scope of embodiments of the invention, however, other embodiments of the invention may be directed to particular embodiments of each individual aspect or particular combinations of these individual aspects.

[0066]

[0075] The foregoing description of the embodiments of the present invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise forms described, and many modifications and variations are possible in light of the above teachings. The embodiments were chosen and described in order to explain the principles of the invention and its practical application, so that others skilled in the art can utilize the invention in various embodiments and with various modifications suited to the particular uses envisioned.

Claims

1. an emitter arranged to emit a pulse of light; a mirror positioned to reflect the light pulses to the environment; a detector positioned to detect light from the light pulses reflected by one or more objects in the environment, the detector comprises a first array of light sensors and a second array of light sensors; a detector positioned such that the second array of optical sensors scans a field of view in front of the first array of optical sensors; A memory device comprising instructions that, when executed by one or more processors, cause the one or more processors to adjusting the emitter output or adjusting the detector gain of the first row of photosensors in response to data from the second row of photosensors scanning the field of view; a memory device that causes generating three-dimensional point cloud data, the point cloud comprising at least one data point of the one or more objects in the environment based on light detected by one or more light sensors in the first row of light sensors; A lidar system equipped with

2. the emitter comprises a first row of lasers and a second row of lasers; The system of claim 1 , wherein the different intensities are generated by the first array of lasers emitting light at a different power than the second array of lasers.

3. the detector further comprises an image sensor positioned to produce a two-dimensional color image; the image sensor is on the same chip as the first row of photosensors and the second row of photosensors; The system of claim 1 , wherein the image sensor comprises three rows of photosensors with three different color filters.

4. an emitter arranged to emit a pulse of light; a mirror positioned to reflect the light pulses into the environment; a detector positioned to detect light from the light pulses reflected by one or more objects in the environment, the detector comprises a first array of light sensors and a second array of light sensors; a detector positioned such that a first array of light sensors detects a different intensity of light than a second array of light sensors; a memory device comprising instructions that, when executed by one or more processors, cause the one or more processors to generate a three-dimensional point cloud of data, the point cloud comprising at least one data point of the one or more objects in the environment based on light detected by the detector; and A lidar system equipped with

5. The system of claim 4 , wherein the different intensities are produced by the emitter increasing or decreasing the power of the light pulse after the second array of light sensors detects light above a threshold.

6. The system of claim 4 , wherein the different intensities are caused by one or more filters in front of the first row of light sensors and / or in front of the second row of light sensors.

7. 5. The system of claim 4, wherein the different intensities are caused by a lower sensitivity to light of the first row of light sensors or a change in gain of an amplifier receiving signals from the first row of light sensors.

8. the detector comprises a third array of photosensors; the first row of light sensors is between the second row of light sensors and the third row of light sensors; The system of claim 4 , wherein the first array of light sensors is positioned to detect a different intensity of light than the third array of light sensors.

9. The system of claim 8 , wherein the instructions cause the one or more processors to use pixel data from one or more columns of photosensors having a desired data range.

10. The system of claim 4 , wherein the different intensities are caused by a gain of the first array of photosensors being reduced compared to a gain of the second array of photosensors.

11. The system of claim 4 , wherein the detector further comprises four or more rows of light sensors, each row of light sensors detecting a different intensity of light.

12. the detector further comprising an image sensor positioned to produce a two-dimensional color image; The system of claim 4 , wherein the image sensor is on the same chip as the first array of photosensors and the second array of photosensors.

13. The system of claim 12 , wherein the image sensor, the first array of light sensors, and the second array of light sensors share a common lens.

14. the system includes an image sensor on a chip separate from the detector; The system of claim 4 , wherein the system comprises a beam splitter that transmits a first portion of the light to the image sensor and a second portion of the light to the detector.

15. The system of claim 4 , wherein the mirror is positioned to scan the light pulse in a direction perpendicular to the direction of the first row of light sensors.

16. the mirror is a first mirror; the system includes a second mirror; 16. The system of claim 15, wherein the second mirror is positioned to scan the light pulses in a direction parallel to the direction of the first row of light sensors.

17. The system of claim 4 , wherein generating the data for the three-dimensional point cloud includes adding data from the first array of optical sensors to data from the second array of optical sensors.

18. The system of claim 4 , wherein the second array of light sensors is positioned to detect stray light emitted toward and received by the first array of light sensors.

19. The system of claim 4 , wherein the detector comprises a third array of light sensors for detecting thermal light from the environment.

20. emitting light pulses using one or more lasers; reflecting said light pulses to the environment using a mirror; detecting light from the light pulses reflected by one or more objects in the environment using a detector; the detector comprises the first array of light sensors and a second array of light sensors; a first array of light sensors detecting a different intensity of light than a second array of light sensors; generating a three-dimensional point cloud data based on the light detected by the detector, the three-dimensional point cloud including at least one data point of the one or more objects in the environment; A rider's method comprising:

Images