Lidar chip with multiple detector arrays

Abstract

A lidar system uses a detector chip having a first column of light sensors and a second column of light sensors configured to detect light from laser pulses reflected by one or more objects in the environment. The second column of light sensors is configured to scan an image field in front of the first column of light sensors. A processor can adjust an emitter power or a detector gain of the first column of light sensors in response to data from the second column of light sensors scanning the image field, so that light within a dynamic range of the first column of sensors can be used to generate data for a three-dimensional point cloud.

Description

Cross-Reference to Related ApplicationsThis application claims priority to U.S. Provisional Patent Application No. 63 / 621,922, filed Jan. 17, 2024, which is incorporated by reference in its entirety for all purposes.BackgroundThree-dimensional sensors may be used in autonomous vehicles, drones, robotics, security applications, and the like. For example, the lidar projects an optical beam and detects light of the optical beam reflected by one or more objects in an environment. The lidar may be used to generate a three-dimensional map of the environment, or a portion thereof, based on the detection of the reflected light of the optical beam. Lidar scan sensors can achieve high angular resolutions suitable for such applications at affordable cost. An example of a lidar scanning system is provided in U.S. Pat. No. 10,690,754, issued Jun. 23, 2020, which is incorporated by reference for all purposes. However, improved scanning systems, apparatus, and / or methods are desirable.OverviewThis disclosure relates, without limitation, to lidar and to remedy saturation of lidar photodetectors.In certain configurations, a lidar system includes a emitter / emitter configured to emit light pulses; a mirror configured to reflect the light pulses into an environment; a detector configured to detect such light of the light pulses reflected by one or more objects in the environment, wherein: the detector includes a first column (column) of light sensors and a second column of light sensors, and the second column of light sensors is configured to scan a field of view (field of view) in front of the first column of light sensors; and / or a storage device comprising instructions that, when executed by one or more processors, cause the one or more processors to: adjust a power of the emitter, or adjust a detector gain of the first column of light sensors in response to data from the second column of light sensors sensing the image field, and / or generate data for a three-dimensional point cloud, wherein the point cloud comprises at least one data point of the one or more objects in the environment underlying light detected by one or more light sensors in the first column of light sensors.In certain configurations, a lidar system includes a radiator configured to emit light pulses; a mirror configured to reflect the light pulses into an environment; a detector configured to detect such light of the light pulses reflected by one or more objects in the environment, wherein: the detector includes a first column of light sensors and a second column of light sensors, and the first column of light sensors is configured to detect light at a different intensity than the second column of light sensors; and a storage device comprising instructions that, when executed by one or more processors, cause the one or more processors to generate data for a three-dimensional point cloud, the point cloud comprising at least one data point of the one or more objects in the environment underlying light detected by the detector.In some configurations, the different / different (different) intensity is caused by the emitter reducing a power of the light pulses after the second column of light sensors has detected light above a threshold. In some configurations, the different intensity is caused by the emitter increasing a power of the light pulses after the second column of light sensors has detected light below a threshold. In some configurations, the other intensity is caused by one or more filters before the first column of light sensors and / or before the second column of light sensors. In some configurations, the different intensity is caused by light sensors in the first column being less sensitive to light or by changing a gain of amplifiers that receive signals from light sensors in the first column. In some configurations, the detector includes a third column of light sensors; wherein the first column of light sensors is between the second column of light sensors and the third column of light sensors in some configurations; and wherein the first column of light sensors is configured to detect light at a different intensity than the third column of light sensors in some configurations. In some configurations, the mirror is configured to rotate or oscillate. In some configurations, the instructions cause the one or more processors to use pixel data from one or more columns of light sensors having a desired range of data. In some configurations, the different intensity is caused by a gain of the first column of light sensors being decreased or increased compared to a gain of the second column of light sensors. In some configurations, the emitter includes a first column of lasers and a second column of lasers; the different intensity in some configurations is caused by the first column of lasers emitting light at a different power than the second column of lasers. In some configurations, the detector further comprises four or more columns of light sensors, each column of light sensors detecting light at different intensities. In some configurations, the detector further comprises an image sensor configured to generate a two-dimensional color image; in some configurations, the image sensor is on the same chip as the first and second columns of light sensors. In some configurations, the image sensor, the first column of light sensors, and the second column of light sensors share a common lens. In some configurations, the image sensor includes three columns of light sensors with three different colored filters. In some configurations, the system includes an image sensor on a separate chip from the detector; and in some configurations, the system includes a beam splitter that sends a first portion of the light to the image sensor and a second portion of the light to the detector. In some configurations, the mirror is configured to scan with light pulses in a direction orthogonal to a direction of the first column of light sensors. In some configurations, the mirror is a first mirror; in some configurations, the system includes a second mirror; and in some configurations, the second mirror is configured to scan with the light pulses in a direction parallel to the direction of the first column of light sensors. In some configurations, generating the data for the three-dimensional point cloud includes adding data from a light sensor in the first column to data from a light sensor in the second column. In some configurations, the second column of light sensors is configured to detect stray light radiated to and received from the first column of light sensors. And / or in some configurations, the detector includes a third column of light sensors for detecting thermal radiation from the environment.In some configurations, a lidar method includes emitting light pulses using one or more lasers; reflecting the light pulses into an environment using a mirror; detecting, using a detector, such light of the light pulses reflected from one or more objects in the environment, wherein the detector includes a first column of light sensors and a second column of light sensors, and the first column of light sensors detects light at an intensity different than the two columns of light sensors; and generating data for a three-dimensional point cloud based on light detected by the detector, wherein the three-dimensional point cloud includes at least one data point of the one or more objects in the environment.In some embodiments, a detector chip includes a single column of photodetectors for detecting reflected laser light for the lidar (e.g., with an IR filter) and one, two, or three columns of photodetectors for receiving ambient light (e.g., with a red, green, and blue filter) on the same chip.Further areas of applicability of the present disclosure will become apparent from the detailed description provided hereinafter. It should be understood that the detailed description and specific examples, while indicating various embodiments, are intended for illustrative purposes only and are not necessarily intended to limit the scope of the disclosure.Brief Description of the DrawingsThe present disclosure will be described in conjunction with the appended figures. FIG. 1 shows an embodiment of a lidar sensor for a three-dimensional image. FIG. 2 shows an embodiment of a lidar sensor with a rotating mirror and a steering mirror (routing mirror). FIG. 3 shows an embodiment of a chip for lidar sensing. FIG. 4 shows an embodiment of a lidar system with a radiator and a detector using arrays (arrays). FIG. 5 illustrates an embodiment of a chip comprising multiple columns of photodetectors. FIG. 6 shows the chip of FIG. 5 included in an embodiment of a lidar system. FIG. 7 shows an embodiment of a chip comprising a lidar array and an image sensor array. FIG. 8 shows an embodiment of a system with a lidar chip that is separate from a camera chip. FIG. 9 shows a flowchart of an embodiment of a lidar method.In the appended figures, similar components and / or features have the same reference numerals. Further, various components of the same type can be distinguished by following the reference sign with a hyphen and a second sign that distinguishes the similar components from each other. When only the first reference sign is used in the description, the description is applicable to any of the similar components having the first reference sign, regardless of the second reference sign.Detailed DescriptionThe following description provides only one or more preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the following description of the one or more preferred exemplary embodiments provides a description executable by those skilled in the art for practicing a preferred exemplary embodiment. It is to be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope defined in the appended claims.Lidar sensors are used to generate a three-dimensional image of an object space within a field-of-view (FOV) of the sensor. A common difficulty with accurate imaging is that some objects that are common in driving situations, particularly retroreflective objects such as road signs or license plates, can be easily 1000 times brighter than normal objects when actively illuminated by the sensor. This can cause saturation of the photodetectors and associated electronics. This may also result in "Blooming", where small amounts of stray light from inaccuracies (imperfections) (e.g., inaccurate lenses, dust on the windshield, and reflections on interior components such as the lens barrel) may cause false detections around a retroreflector, making it appear significantly larger than it is actually. Such blooming can be annoying, in particular, when, for example, the image of a speed limit sign illuminates the lane, whereby the car brakes on the road because of a perceived obstacle, although none is located there.One possible structure for a lidar recording system comprises a 1D arrangement of sensors (one-dimensional sensor arrangement) which are installed on an integrated circuit ("IC" or "chip"). The illumination may be a vertical line of laser light (for example, a 1D array of lasers (one-dimensional laser array)). Both with the laser line and with the returning photons are scanned synchronously (for example, by separate scanning optics or jointly by the same scanning optics(s)), so that the laser photons reflected from one or more objects in the image field of the lidar system return to and are focused on a corresponding detector. Multiple columns of photodetectors may be used to increase a dynamic range of detection to allow capture of both bright and dark objects. In addition, multiple columns may be used to provide additional modes of capture, such as for camera images (color) or thermal infrared images.FIG. 1 shows an embodiment of a LiDAR sensor 100 for three-dimensional recording. The LiDAR sensor 100 includes a radiation lens 130 and a receiving lens 140. The LiDAR sensor 100 comprises a light source 110- a, which is arranged substantially in a rear focal plane of the radiation lens 130. The light source 110- ais used to emit a light pulse 120 from a corresponding emission location in the rear focal plane of the emission lens 130. The emission lens 130 is configured to collimate and direct the light pulse 120 onto an object 150 located in front of the LiDAR sensor 100. For a given emission location of the light source 110- a, the collimated light pulse 120' is directed at a corresponding angle towards the object 150.A portion 122 of the collimated light pulse 120' is reflected from the object 150 toward the receiving lens 140. The receiving lens 140 is configured to focus the portion 122' of the light pulse reflected by the object 150 onto a corresponding detection location in the focal plane of the receiving lens 140. The LiDAR sensor 100 further comprises a detector 160- aarranged substantially in a focal plane of the receiving lens 140. The detector 160- ais configured to detect the part 122' of the light pulse 120 reflected by the object at the corresponding detection location. The corresponding detection location of the detector 160- ais optically conjugate to the respective emission location of the light source 110- a.The light pulse 120 may be of short duration, such as a 10 ns pulse width. The LiDAR sensor 100 further includes a processor 190 coupled to the light source 110- aand the detector 160- a. The processor 190 is configured to determine a time off light (TOF) of the light pulse 120 from emission to detection. Since the light pulse 120 moves at the speed of light, a distance between the LiDAR sensor 100 and the object 150 may be determined based on the determined travel time.One possibility for scanning an FOV with a laser beam (for example with the light pulse 120') is to move the light source 110- alaterally relative to the emission lens 130 in the rear focal plane of the emission lens 130. For example, the light source 110- amay be moved across multiple emission locations in the rear focal plane of the emission lens 130, as shown in FIG. 1. The light source 110- acan emit a plurality of light pulses to the plurality of emission locations. Each light pulse emitted at a respective emission location is collimated by the emission lens 130 and directed at a corresponding angle to the object 150, and impinges at a corresponding point on the surface of the object 150. Since the light source 110- ais moved within a specific region in the rear focal plane of the emission lens 130, a corresponding object region on the object 150 is thus scanned. The detector 160- acan be moved (raster scanned) in a raster-like manner for positioning at a plurality of corresponding detection locations in the focal plane of the receiving lens 140, as shown in FIG. 1. Moving the detector 160- ais typically performed synchronously with moving the light source 110- asuch that the detector 160- aand the light source 110- aare optically conjugate to each other at any time.By determining the travel time for each light pulse emitted at a respective emission location, the distance from the LiDAR sensor 100 to each corresponding point on the surface of the object 150 may be determined. In some embodiments, processor 190 is coupled to a position transducer that detects the position of light source 110- aat each emission location. Based on the emission location, the angle of the collimated light pulse 120' can be determined. The X-Y coordinate of the corresponding point on the surface of the object 150 may be determined based on the angle and distance to the LiDAR sensor 100. Thus, a three-dimensional image of the object 150 may be generated based on the measured distances from the LiDAR sensor 100 to various points on the surface of the object 150. In some embodiments, the three-dimensional image may be represented by a point cloud, more specifically, as a set of X, Y, and Z coordinates of the points on the surface of the object 150.In some embodiments, the intensity of the returning light pulse 120' is measured and used to adjust the power of subsequent light pulses from the same emission point to prevent saturation of the detector, improve eye protection, or reduce overall power consumption. The power of the light pulse can be varied by varying the duration of the light pulse, the voltage or current supplied to the laser, or the charge stored in a capacitor used to energize the laser. In the latter case, the charge stored in the capacitor can be varied by varying the charging time, the charging voltage or the charging current at the capacitor. In some embodiments, reflectivity as determined by the intensity of the detected pulse may also be used to add a different dimension to the image. For example, the image may include X, Y, and Z coordinates as well as reflectivity (or brightness).The angular field of view (AKOV) of the LiDAR sensor 100 may be estimated based on the scanning range of the light source 110- aand the focal length of the radiation lens 130, where h is the scanning range of the light source 110- aalong the determined direction and f is the focal length of the radiation lens 130. For a given scan range h, shorter focal lengths produce broader A FOVs. For a given focal length f, larger scan ranges produce wider A FOVs. In some embodiments, the LiDAR sensor 100 may include multiple light sources arranged as an array in the rear focal plane of the radiation lens 130, such that a greater overall A FOV may be achieved with the scan range of each individual light source being kept relatively small. Accordingly, the LiDAR sensor 100 may include multiple detectors arranged as an array in the focal plane of the receiving lens 140, each detector being conjugate to a corresponding light source. For example, the LiDAR sensor 100 may include a second light source 110- band a second detector 160- bas shown in FIG. 1. In other embodiments, the LiDAR sensor 100 may include four light sources and four detectors, or eight light sources and eight detectors. In an embodiment, the LiDAR sensor 100 may include eight light sources arranged as a 4x2 array and eight detectors arranged as a 4x2 array such that the LiDAR sensor 101 has a wider A FOV in the horizontal direction than the A FOV in the vertical direction. According to various embodiments, the total AKOV of the lidar sensor 100 may be in a range from about 5° to about 15°, or from about 15° to about 45°, or from about 45° to about 120°, depending on the focal length of the emitting lens, the scanning range of each light source and the number of light sources.The light source 110- amay be configured to emit light pulses in near-infrared wavelength ranges. The energy of each light pulse may be on the order of microjoules, which is typically considered harmless to the eyes for repetition rates in the KHz range. For light sources operating at wavelengths greater than about 1500 nm (in the near infrared wavelength range), the energy levels may be higher because the eye does not focus at these wavelengths. The detector 160- amay include a silicon avalanche photodiode, a photomultiplier, a PIN diode, or other semiconductor sensors.FIG. 2 shows an embodiment of a lidar system 200 having a rotating mirror 204 and a steering mirror 208. The rotating mirror 204 may be a polygonal mirror having reflective surfaces 212. The rotating mirror 204 rotates (e.g., in full revolutions) about a vertical axis 216. Light is reflected by the rotating mirror 204 into a field-of-view (FOV) 220. The FOV 220 includes a horizontal component 222 and a vertical component 224. For example, a lidar system may be positioned on a car such that the vertical component 224 of the FOV 220 is in the direction of gravity and the horizontal component 222 of the FOV 220 is orthogonal to the direction of gravity. Although the rotating mirror 204 is shown as a fully rotating mirror, in some embodiments the rotating mirror may rotate back and forth in an oscillatory motion about an axis.The rotating mirror 204 is used to horizontally scan a FOV 220 of the lidar system 200 with one or more laser beams (e.g., a pulsed laser beam). The rotating mirror 204 has a plurality of planar mirror surfaces of equal size that are created on a fully rotating rotor. Although the number of areas 212 shown in FIG. 2 is six, the number of areas 212 may be greater than or equal to 2, 3, or 4 and / or less than or equal to 4, 5, 6, 7, 8, 10, or 12. A fully inverting mirror, such as rotating mirror 204 shown in FIG. 2, has low vibration, low power requirements, and linear scan behavior. When lidar imaging is used, the rotating mirror 204 scans in the horizontal direction. To achieve higher resolution in the other (e.g., vertical) direction, a vertical array of lasers (e.g., a large vertical array) and / or a galvanomirror may be used. The lidar system 200 shown in FIG. 2 includes a laser 228 and a detector 232. Although only one laser 228 and one detector 232 are shown in FIG. 2, it should be understood that more than one laser 228 and / or more than one detector 232 may be used (e.g., a laser array and / or a detector array may be used).In some embodiments, a rotating polygon mirror is used to scan in the horizontal direction with one or more lasers 228, and the steering mirror 208 is used to fold a beam path for a more compact design. The steering mirror 208 shown in FIG. 2 is also used to position light pulses from the laser 228 in a vertical direction by dynamically tilting it back and forth in the vertical direction (e.g., about a horizontal axis). This arrangement can achieve high resolution in the vertical direction while reducing and / or minimizing a number of lasers and / or cause the lidar system 200 to be more compact. As shown in FIG. 2, the steering mirror 208 may effectively increase a number of scan lines 236 in the vertical component 224 of the FOV 220. This can be achieved with a single laser and / or with a vertical arrangement of lasers. Each point on the scan line 236 represents a laser pulse.In some embodiments, the scan lines 236 are in discrete vertical steps. For example, the steering mirror 208 rotates to discrete positions for each scan line 236 (e.g., instead of continuously rotating the steering mirror 208). As the steering mirror 208 rotates continuously, the scan line is bent and laser pulses are not radiated in a straight horizontal line. That a scan line of the laser pulses is a curved line does not need to pose a problem when only one laser 228 is used. However, when multiple lasers are used, the presence of straight scan lines may help generate a scan pattern in the FOV 220 with a desired density of data points (e.g., such that light pulses from lasers are spaced apart in a desired density pattern).In some embodiments, a lidar system (e.g., lidar system 200) includes an illumination source including a plurality of lasers (e.g., an array of lasers including laser 228) and a mirror system (e.g., including rotating mirror 204 and steering mirror 208). The mirror system is configured to reflect light from the illumination source within an image field (e.g., the FOV 220) into an environment. The mirror system includes a mirror (e.g., rotating mirror 204) configured to rotate to reflect the light from the illumination source to horizontally scan the light from the plurality of lasers within the field of view of the system. The mirror system is configured to reflect light from the illumination source (e.g., using the steering mirror 208) to vertically position the light of the plurality of lasers within the image field in discrete vertical steps (e.g., the scanlines 236 are separated by discrete vertical steps). The scan lines 236 are straight (e.g., horizontal) lines.Additional lidar sensors are described in commonly owned U.S. Patent Applications 15 / 267,558 filed Sep. 15, 2016, 15 / 971,548 filed May 4, 2018, 16 / 504,989 filed July 8, 2019, 16 / 775,166 filed January 28, 2020, 17 / 032,526 filed Sep. 25, 2020, 17 / 133,355 filed Dec. 23, 2020, 17 / 205,792 filed Mar. 18, 2021, 17 / 380,872 filed July 20, 2021, and 18 / 531,507 filed Dec. 6, 2023. The disclosures of these patent applications are incorporated by reference for all purposes.FIG. 3 shows a simplified embodiment of a chip 300 for lidar detection. The chip 300 includes a column of photodetectors 304. The column of photodetectors 304 may be referred to as an array; in this case, a one-dimensional array. The photodetectors 304 are optical sensors. Two-dimensional scanning of an image field may be constructed by horizontally (and / or vertically) scanning the image field (FOV) with the receiving region of the photodetectors 104, such as by using a scanning mirror, such as a galvanometer mirror ("Galvo") and / or a rotating polygonal mirror (e.g., as shown in FIG. 2 ). A laser, or multiple lasers, is provided to provide light pulses that scan the FOV (e.g., by the same scanning mirror), thereby illuminating one or more objects in an environment imaged by the lidar detector (e.g., one or more objects within the FOV of the lidar system).Lidar photodetectors 304 are typically infrared detectors, although other wavelengths may be used. The photodetectors 304 may be simple photodiodes, avalanche photodiodes (APDs), single photon detectors (SPADs), or arrays of SPADs, often called silicon photomultipliers (SiPMs). Each photodetector 304 is coupled to a time off light detector (ToF detector) via an operational amplifier, typically a transimpedance amplifier. The ToF detector determines the time between the laser pulse and the detection event to calculate the distance between the LiDAR sensor and the detected object. The lidar system uses an active laser illumination system that emits short pulses of laser light (e.g., nanosecond duration). Each photodetector may be illuminated by its own laser, or a group of lasers, or the entire column of detectors may share a single laser. Examples of a laser include a fiber-coupled laser, an edge emitting diode laser (EEL), and a vertical cavity surface emitting laser (VCSEL).Transimpedance amplifier (TIA) 308 may amplify a signal from photodetectors 304. The amplified signal is then sent to the time-of-flight (ToF) detector 312 to calculate a distance to a reflection. Image processing circuitry 316 is used to calculate data for the three-dimensional point cloud of the environment within the FOV (image field) of the lidar sensor, which may be used to generate lidar image data. In some embodiments, a storage device includes instructions that, when executed by one or more processors (e.g., image processing circuit 316 and / or ToF detector 312), cause the one or more processors to generate data for a three-dimensional point cloud including at least one data point for one or more objects in the environment underlying light detected by the detector. The chip 100 may be used as a detector in a lidar system (e.g., the detector 232 in FIG. 2 ).FIG. 4 shows an embodiment of a lidar system 400 having a radiator 404 and a detector 408 that uses arrays. The lidar system 400 includes a rotating mirror 204 and a steering mirror 208. The emitter 404 comprises a one-dimensional laser arrangement which is provided for emitting light pulses. The steering mirror 208 and the rotating mirror 204 are configured to reflect the light pulses into an environment within a FOV 220 of the lidar system 400. The detector 408 (which comprises the chip 300, for example) is provided to detect such light of the light pulses that is reflected by one or more objects in the environment.The rotating mirror 204 (for example, a first mirror) is provided to scan the light pulses in a horizontal direction, or a direction orthogonal to a direction of a column of sensors. For example, the column of photodetectors 304 of the chip 300 in FIG. 3 is vertically disposed in the detector 408 in FIG. 4, and the rotating mirror 204 scans with the light pulses for the photodetectors along the scan lines 426 in a horizontal direction. A vertical resolution 430 may be determined by the vertical array size and / or the spacings of the photodetectors 304 on the chip 300. A horizontal resolution 432 may be determined by a scan speed of the rotating mirror 204 and / or a firing rate of the radiator 404.In some embodiments, the steering mirror 208 (e.g., a second mirror) is configured to vertically scan (e.g., in a direction parallel to the direction of the first column of sensors) with the light pulses. For example, the light pulses are scanned vertically above, below, and / or between the scan lines 426 shown in FIG. 4. Although FIG. 4 shows an array of lasers and an array of photodetectors, each of which is six, other numbers of lasers and photodetectors may be used in a column of each array. For example, greater than or equal to 2, 5, 10, 20, 30, 40, or 50 components may be present in a column and / or less than or equal to 40, 70, 100, 256, or 512 components may be present in a column. A size of the chip, a size of the components, and / or the electronics on the chip may limit the number of components in a column or array.Some lidar systems have the ability to reduce the intensity of the laser pulses when a bright object, such as a retroreflector, enters the image field so that the detectors are not saturated or dimmed. However, in a system as shown in FIGS. 3 and 4, the reduction in laser intensity is a reactive action and not a proactive action, as there is no prior knowledge (e.g., in a scanline 426) of a retroreflective object entering the FOV. When a bright object is detected and the laser power is reduced, one or more portions of the image may already have affected data.FIG. 5 illustrates an embodiment of a chip 500 having multiple columns 504 of photodetectors 304. Chip 500 is shown having a first column 504- 1 of photodetectors 304, a second column 504- 2 of photodetectors 304, and a third column 504- 3 of photodetectors 304. Although three columns 504 are shown in FIG. 5, these may be fewer (e.g., two) or more (e.g., four, five, six, or more). The chip 500 includes TIAs 308, ToF detectors 312, and image processing circuits 316.FIG. 6 shows an embodiment of the chip 500 installed in an embodiment of a lidar system 600. The lidar system 600 includes a radiator 404, a detector 608, a rotating mirror 204, and a steering mirror 208. Detector 608 includes chip 500. The chip 500 includes the first column 504- 1 of photodetectors 304 and the second column 504- 2 of photodetectors 304.The second column 504- 2 of photodetectors 304 scans the FOV 220 before the first column 504- 1 of photodetectors 304. The first column 504- 1 and the second column 504- 2 of photodetectors are scanned the same, or approximately the same, FOV (image field) 220. For example, the FOV sampled with the first column 504- 1 overlaps with the FOV sampled with the second column by a magnitude greater than or equal to 90%, 95%, 97%, 98%, 99%, or 100%; in some embodiments, the difference is less than or equal to five, three, two, or one pixel width.FIG. 6 shows pixels in a first region 611 in the FOV 220 that is captured by the first column 504- 1 of sensors and pixels captured by the second column 504- 2 of sensors in a second region 612 of the FOV 220 at the same time that the pixels in the first region 611 are captured by the first column 504- 1 of sensors. The first column 504- 1 of photodetectors 304 may receive light at a different intensity than the second column 504- 2 of photodetectors 304. For example, scanning initially performed with the second column 504- 2 of photodetectors 304 (e.g., in region 612) gives the processor time to adjust the laser illumination pulse power to prevent saturation of the first column 504- 1 of photodetectors 304 when the lidar system 600 detects an excessively bright object through the second column 504- 2 of photodetectors 304. In some situations, the second column of photodetectors 304 may detect light below a threshold power and the system may increase laser power so that more intense light is detected by the first column 504- 1 of photodetectors 304. Accordingly, the second column 504- 2 is used to scan objects in a scene before the subsequent first column 504- 1, and a processor is capable of using information from one or more photodetectors 504 in the second column 504- 2 to adjust (e.g., increase or decrease) the laser power and / or detector gain for the first column 504- 1. In some configurations, different columns 504 of detectors include different filters (e.g., different transmittance values of neutral density filters) and / or different sensitivity values of photodetectors 304.In some systems that use an oscillating mirror instead of a fully inverting mirror, a direction of the scan may be periodically reversed. In this case, depending on which column the FOV 220 is first scanned, the processor may adjust the laser power based on that column so that the column that scans an object later uses optimized laser power for illumination. A third column of detectors (e.g., 504- 3 in FIG. 5 ) may be added at an end opposite the second column 504- 2 of detectors, such that depending on the scan direction, either the second column 504- 2 or the third column 504- 3 may be used to detect an excessively bright object.In some configurations, less sensitive detectors or amplifiers with lower gain may be used in the second column 504- 2 of detectors. This enables unsaturated image data even if the laser power remains high. The processor may select pixel data from the light sensors in one or more columns 504 having a desired data range (e.g., below a high threshold and / or above a low threshold). The third column 504- 3 may include even less sensitive detectors, amplifiers with even less gain, and / or be arranged such that the second column 504- 3 is between the first column 504- 1 and the third column 504- 3. Additionally, rather than decreasing laser power (or in addition to decreasing laser power), the processor may electronically decrease the gain of a front-end detection system by techniques such as variable gain or reduced bias (bias) amplifiers of the one or more photodetectors. In some configurations, the laser intensity is switched to emit different laser powers to be detected from different detector columns, and the system selects data from one or more columns 504 (e.g., from the columns that are not saturated).In some configurations, laser illumination may be configured to illuminate primarily the first column 504- 1 and illuminate the second column 504- 2 at a significantly lower intensity. Therefore, bright objects are less likely to saturate the light sensor in the second column 504- 2. In some configurations, a second set of one or more lasers may be used to separately illuminate the second column 504- 2 at a laser power that is higher or lower than the laser power used to illuminate the first column 504- 1.In some embodiments, an array of detectors may be a two-dimensional array. One or more columns 504 may be selected to provide data in advance for adjusting the laser power. The selected column may be configurable depending on the brightness of the one or more captured objects. In some configurations, the laser intensity may be configured to decrease (e.g., gradually or in steps) across multiple columns of photodetectors. The processor may select which column or columns has the correct level of illumination for an object in the FOV 220. It should also be noted that "columns" may also be "rows" and "vertical" may also be "horizontal" without departing from the spirit of the invention. For example, scanning may take place in the vertical direction rather than the horizontal direction and the photodetectors may be arranged in rows rather than columns.In some configurations, different / different intensities of light sensors are caused by the emitter reducing the power of the light pulses after the second column of light sensors detects light above a threshold. In some configurations, the other intensity may be caused by one or more filters before the first column of light sensors or the second column of light sensors. In some configurations, the different intensity is caused by light sensors in the second column being less sensitive or having amplifiers with lower gain than the light sensors in the first column. In some configurations, the detector includes a third column of light sensors. In some configurations, the first column of light sensors is between the second column of light sensors and the third column of light sensors. In some configurations, the first column of light sensors detects light at a different intensity than the third column of light sensors. In some configurations, the mirror is configured to oscillate or rotate (e.g., rotate fully through 360°). In some configurations, the instructions cause the one or more processors to use pixel data from one or more columns of light sensors having a desired range of data (e.g., that are not saturated). In some configurations, the different intensity is caused by the gain of the first column of light sensors being decreased compared to the gain of the second column of light sensors. In some configurations, the emitter includes a first column of lasers and a second column of lasers. In some configurations, the different intensity is caused by the first column of lasers emitting light at a different power than the second column of lasers. In some configurations, the detector includes four or more columns of light sensors, each column of light sensors detecting light at a different intensity than the other columns of light sensors. In some configurations, the detector includes three, four, or more columns of light sensors, each column of light sensors detecting light at a different wavelength than the other columns of light sensors (e.g., a light sensor having four columns: R, G, B, and IR for detecting ambient light on the same chip as one, two, three, or more columns used to detect reflected laser light; wherein an IR filter for the IR ambient light that does not overlap a bandwidth of the light is used by the emitter). In some configurations, generating the data for the three-dimensional point cloud includes adding data from light sensors in adjacent columns. In some configurations, the second column of light sensors is additionally or alternatively configured to detect stray light radiated to and received from the first column of light sensors.Lidar sensor data may be combined with data from other sensors, such as cameras, to improve object detection and / or provide redundancy. Color data may also be important for traffic lights and stop light detection. A problem with such a sensor linkage is that due to the different distortion characteristics in lenses it may be difficult to perfectly superimpose the lidar image on the camera image. Another aspect of the problem is that a protective cover (in some cases, the windshield of the car) may introduce further image distortions that are different between the LiDAR sensor and the camera, and these distortions may change when the cover (or windshield) is replaced during maintenance.FIG. 7 shows an embodiment of a chip 700 including a lidar array 704 and an image sensor array 708. By integrating a camera sensor and a lidar sensor on a single sensing device (e.g., on a silicon integrated circuit) and using a common lens for capturing, difficulties in superimposing the lidar image and the camera image can be reduced or eliminated entirely. FIG. 7 shows an example of a silicon sensor, which has a 1D arrangement of lidar photodetectors and a second arrangement of photodetectors for a camera recording. The image sensor array 708 could be a single column of detectors for a black and white image, a set of three or more columns of detectors for color capture, or a combination of color sensors on a single column. The lidar sensor arrangement 704 could be one column (e.g., as described in connection with FIG. 3 ) or multiple columns (e.g., as described in connection with FIG. 5 ). In FIG. 7, the image sensor array 708 is used to capture light in the FOV using red (R), green (G), and blue (B) sensitive photodetectors, but other types may also be used.In FIG. 7, the lidar sensor assembly 704 and the image sensor assembly 708 are physically close enough to share a common lens and a scanning mechanism, with only a minimally significant offset between the images that can be removed by calibration and / or image processing. In some configurations, multiple lidar photodetector arrays may be used in combination with one or more camera photodetector arrays.In some configurations, a detector further includes an image sensor (e.g., an image sensor array 708) configured to generate a two-dimensional color image, the image sensor being on a same chip as the first column and the second column of the light sensor. For example, the chip 700 could be used as a part of the detector 608 in FIG. 6. The image sensor, the first column of the light sensor and the second column of the light sensor may share a common lens (e.g., a receiving lens). The image sensor may include three columns of light sensors with three different color filters (e.g., as shown in FIG. 7 as "R", "G", and "B").FIG. 8 shows an embodiment of a lidar system with a lidar chip 804 and a camera chip 808 on two separate chips. A beam splitter 812 (e.g., having a high pass IR light and a high visible reflection) is used to direct infrared light onto the lidar chip 804 and visible light onto the camera chip 808. For example, the lidar could be chip 804The camera chip 808 has a suitable arrangement for "ambient light" detectors or visible light detectors. Ambient light detectors are not intended to capture light from laser reflections on objects, but are used to capture objects in a scene using naturally occurring light (from the sun, for example) or light already occurring in the scene (from headlights and / or street lights, for example). Although an array of sensors (e.g., a plurality of sensors in a column) on the camera chip 808 typically matches 1:1 with an array of sensors on the lidar chip 804, it is possible that there are more or fewer sensors on the camera chip 808, thereby achieving a higher or lower resolution of the camera image relative to the lidar image. For a full color image, there may be three or more columns of sensors each having a color filter. FIG. 7 shows three columns of sensors, more precisely a column for red, a column for green and a column for blue. The sensors may be simple photodiodes, avalanche photodiodes (APDs), single photon detectors (SPADs), and / or arrays of SPADs, often called silicon photomultipliers (SiPMs). The output of the photodetectors is then amplified, digitized, and / or processed to form a camera image. In some configurations, a system includes an image sensor (e.g., 808 in FIG. 8 ) on a separate chip from the lidar detector (e.g., lidar chip 804 in FIG. 8 ) and / or a beam splitter (e.g., beam splitter 812) configured to transmit a first portion of the light to an image sensor (e.g., camera chip 808) and a second portion of the light to the detector (e.g., lidar chip 804). Pixels of the lidar chip 804 and the camera chip 808 may be matched to one another in an angular space as well as in a time period.In FIG. 8, an IR narrowband filter 820 is arranged between the beam splitter 812 and the lidar chip 804, and / or a color control mask (color control mask) 824 is arranged between the beam splitter 812 and the camera chip 808. The beam splitter 812 is (optically) disposed between a receiving optics 828 and the lidar chip 804, and the beam splitter is (optically) disposed between the receiving optics 828 and the camera chip 808.In addition to or in place of visible light sensors, thermal infrared detectors may be used which may be advantageous for detecting humans, animals and vehicles under nocturnal conditions. In some configurations, a chip includes a column of light sensors (e.g., the third column 504- 3 of photodetectors for detecting ambient thermal radiation).In some embodiments, a number of pixels in the vertical direction or the vertical FOV may be less than desired due to a size and / or cost of the silicon chip used to support as many pixels. To improve vertical resolution and / or FOV, it may be desirable to add scanning in the vertical direction and / or in the horizontal direction. In some embodiments, the chip is provided to cover half of the vertical FOV (e.g., a linear array of lasers and / or a linear array of sensors on the chip is provided to cover half of the vertical FOV). After a scan capturing the lower portion (e.g., the lower half) of the FOV, a mirror or other vertical scanning device offsets the capture such that a subsequent scan covers the upper portion (e.g., the upper half) of the FOV. For example, the steering mirror shown in FIGS. 2, 4, and 6 may be configured to tilt in or scan in the vertical direction (e.g., as in commonly owned U.S. Patent Application No. 18 / 531,507 filed December 6, 2023, which is incorporated by reference for all purposes). The rotating polygonal mirror in Figure 2 (which may be a fully rotating, oscillating, or galvo mirror) may have a tilting motion in the vertical direction (for example, as disclosed in commonly owned U.S. Patent Application No. 18 / 200,457 filed May 22, 2023, which is incorporated by reference for all purposes) in addition to horizontal scanning. In some embodiments, the upper and lower samplings of the FOV may be contiguous (e.g., with a pixel pitch similar to a distance between sensors); in some embodiments, they may overlap, providing a middle region with a higher resolution than the upper and lower regions. In some configurations, the different mirror surfaces of the rotating polygonal mirror may have different angles in the vertical direction, with each angle directing light from a different vertical portion of the FOV toward the detector array. In some embodiments, the detectors are arranged in a sparse matrix with spaces between each detector. After performing each horizontal scan, a vertical mirror or other vertical scanning device changes the capture angle slightly such that a second and / or subsequent horizontal scan implementation fills the distances between the pixels of the first horizontal scan.Signals from adjacent detectors in two or more columns may be added to each other to increase the sensitivity of detection with the same laser pulse. Adding the signals to each other can be performed in either the analog or the digital domain. Detectors may share a single laser, or each detector may use a different detector. In some embodiments, the detectors may use two different laser pulses originating from either the same or a different laser, separated in time by a time delay in accordance with the horizontal scan speed, such that each detector views the same position in the FOV when its corresponding laser is firing. In SPAD detectors, a histogram approach may be used in which multiple laser pulses are transmitted for each pixel and an object is determined to be detected if more than a threshold number of pulses are detected in a given time period. By combining multiple columns, the number of laser pulses can be reduced or the probability of reaching the detection threshold can be increased, thereby effectively increasing sensitivity.In some embodiments, the power of lasers is adjusted from a first column of detectors to a second column of detectors. In some embodiments, detectors in a second column are less sensitive to light than detectors in a first column. In some embodiments, a gain may be changed and / or different for detectors in different columns. In some embodiments, detectors in the second column may be used to detect stray light from the first column. In some embodiments, the detectors may be used to capture ambient light. In some embodiments, one or more columns of detectors may be used for thermal imaging. In some embodiments, additionally or alternatively, the scanning may be performed in the vertical dimension as well as in the horizontal dimension.Referring next to FIG. 9, a flowchart of an embodiment of a lidar process 900 is shown. The process 900 begins in step 904 with emitting light pulses using one or more lasers. For example, light is emitted by laser 228 in FIG. 2 or by emitter 404 in FIG. 4 or 6.In step 908, the light pulses are reflected into an environment using one or more mirrors. For example, mirrors 204 and 208 in FIG. 2 reflect light into FOV 220.In step 912, light from the light pulses reflected by one or more objects in the environment is detected using a detector. The detector comprises a plurality of detector arrangements. For example, the detector includes a first column of light sensors and a second column of light sensors, and the first column of light sensors detects light at a different intensity (e.g., sensitivity) than the second column of light sensors. For example, the chip 500 in FIG. 5, the chip 700 in FIG. 7, or the lidar chip 804 in FIG. 8 is used to detect IR light reflected in the environment from one or more objects within the environment. In some embodiments, the second column of light sensors samples an image field before the first column of light sensors, and one or more processors adjust (e.g., increase or decrease) a power of the emitter, or a detector gain of the first column of light sensors, in response to data from the second column of light sensors sampling the image field (e.g., in a case where light is detected by one or more sensors in the second column above or below a threshold).In step 916, data for a three-dimensional point cloud is generated based on light detected by the detector. The point cloud includes at least one data point for the one or more objects in the environment. In some configurations, a two-dimensional image is generated based on the data of the three-dimensional point cloud. For example, lidar image data in FIG. 5 is generated by the image processing circuit 316. In some configurations, the two-dimensional image is presented to a user.Various features described herein, such as methods, devices, computer readable media, and the like, may be implemented using a combination of dedicated components, programmable processors, and / or other programmable devices. Some processes described herein may be implemented on the same processor or on different processors. When some components are described as being configured to perform certain operations, such a configuration may be achieved by, for example, forming electronic circuits to perform the operation, programming programmable electronic circuits (such as microprocessors) to perform the operation, or a combination thereof. Further, although the embodiments described above reference specific hardware and software components, those skilled in the art will understand that other combinations of hardware and / or software components may also be used and that certain operations described as implemented by hardware may be implemented as software, or vice versa.In the above description, details are provided to provide an understanding of the embodiments. It is to be understood, however, that the embodiments may be practiced without some of the specific details. In some instances, well-known circuits, processes, algorithms, structures, and techniques are not shown in the figures.Although the spirit of the disclosure has been described above in connection with specific apparatuses and methods, it is to be understood that this description is merely illustrative and not intended as a limitation on the scope of the disclosure. The embodiments were chosen and described for explanation of the spirit and practical applications in order to enable one skilled in the art to utilize the invention in various embodiments and with various modifications as are suited to a particular use contemplated. It is to be understood that the description is intended to cover modifications and equivalents.It is also noted that the embodiments may be described as a process shown as a flowchart, dataflow diagram, structural diagram, or block diagram. Although a flowchart may describe the operations as a process flow, many of the operations may be performed in parallel or simultaneously. In addition, the order of the operations may be rearranged. A process is completed when its operations are completed, but may also include additional steps not included in the figures. A process may correspond to a method, function, sequence, subroutine, subroutine, etc.A use of "a," "an," "the," or "the" is intended to mean "one or more" unless expressly stated otherwise. Patents, patent applications, publications, and descriptions mentioned herein are incorporated by reference in their entirety for all purposes. None of these documents are considered prior art.The specific details of particular embodiments may be combined in any manner without departing from the spirit and scope of the embodiments of the invention. However, other embodiments of the invention may be related to specific embodiments with respect to each individual aspect, or specific combinations of these individual aspects.The above description of the embodiments of the invention has been made for illustrative and illustrative purposes. This is not intended to be exhaustive or to limit the invention to the precise form disclosed, and many modifications and alterations may be possible in light of the above teachings. The embodiments were chosen and described in order to explain the spirit and practice of the invention to enable one skilled in the art to utilize the invention in various embodiments and with various modifications as are suited to the particular use contemplated.A lidar system uses a detector chip that has a first column of light sensors and a second column of light sensors and is provided to detect such light from laser pulses that has been reflected by one or more objects in the environment. The second column of light sensors is provided to scan an image field in front of the first column of light sensors. A processor may adjust a power of a radiator or a detector gain of the first column of light sensors in response to data from the second column of light sensors scanning the image field such that light in a dynamic range of the first column of sensors may be used to generate data for a three-dimensional point cloud.References included in the specificationThis list of documents cited by the applicant has been produced in an automated manner and is only included for the better information of the reader. The list is not part of the German patent application or utility model application. The DPMA does not take any adhesion for any faults or omissions.Patent Literature citedUS 18 / 531,507

[0053]

Claims

A lidar system comprising: a radiator configured to radiate light pulses; a mirror configured to reflect the light pulses into an environment; a detector configured to detect such light of the light pulses reflected by one or more objects in the environment, wherein: the detector comprises a first column of light sensors and a second column of light sensors; and the second column of light sensors is configured to scan an image field in front of the first column of light sensors; A storage device comprising instructions that, when executed by one or more processors, cause the one or more processors to: adjust a power of the emitter, or adjust a detector gain of the first column of light sensors in response to data from the second column of light sensors sensing the image field; and generate data for a three-dimensional point cloud, wherein the point cloud comprises at least one data point of the one or more objects in the environment underlying light detected by one or more light sensors in the first column of light sensors.The system of claim 1, wherein: the emitter comprises a first column of lasers and a second column of lasers; and the different intensity is caused by the first column of lasers emitting light at a different power than the second column of lasers.The system of claim 1 or 2, wherein: the detector further comprises an image sensor configured to generate a two-dimensional color image; the image sensor is on a same chip as the first and second columns of light sensors; and the image sensor comprises three columns of light sensors having three different color filters.A lidar system comprising: a radiator configured to radiate light pulses; a mirror configured to reflect the light pulses into an environment; a detector configured to detect such light of the light pulses reflected by one or more objects in the environment, wherein: the detector comprises a first column of light sensors and a second column of light sensors; and the first column of light sensors is configured to detect light at a different intensity than the second column of light sensors; and a storage device comprising instructions that, when executed by one or more processors, cause the one or more processors to generate data for a three-dimensional point cloud, the point cloud comprising at least one data point of the one or more objects in the environment underlying light detected by the detector.The system of claim 4, wherein the different intensity is caused by the emitter increasing or decreasing a power of the light pulses after the second column of light sensors has detected light above a threshold; and / or wherein the different intensity is caused by one or more filters before the first column of light sensors and / or before the second column of light sensors; and / or wherein the different intensity is caused by light sensors in the first column being less sensitive to light, or by changing a gain of amplifiers receiving signals from light sensors in the first column; and / or wherein the different intensity is caused by a gain of the first column of light sensors being decreased compared to a gain of the second column of light sensors.The system of claim 4 or 5, wherein the detector comprises a third column of light sensors for detecting thermal radiation from the environment.The system of claim 4 or 5, wherein: the detector comprises a third column of light sensors; the first column of light sensors is between the second column of light sensors and the third column of light sensors; and the first column of light sensors is configured to detect light at a different intensity than the third column of light sensors.The system of claim 7, wherein the instructions cause the one or more processors to use pixel data from one or more columns of light sensors having a desired range of data.The system of any of claims 4 to 8, wherein the detector further comprises four or more columns of light sensors, each column of light sensors detecting light at different intensities.The system of any of claims 4 to 9, wherein: the detector further comprises an image sensor configured to generate a two-dimensional color image; and the image sensor is on a same chip as the first and second columns of light sensors.The system of claim 10, wherein the image sensor, the first column of light sensors, and the second column of light sensors share a common lens.The system of any of claims 4 to 9, wherein the system comprises an image sensor on a chip separate from the detector; and the system comprises a beam splitter that sends a first portion of the light to the image sensor and a second portion of the light to the detector.The system of any of claims 4 to 12, wherein the mirror is configured to scan with light pulses in a direction orthogonal to a direction of the first column of light sensors.The system of claim 13, wherein: the mirror is a first mirror; the system comprises a second mirror; and the second mirror is configured to scan with the light pulses in a direction parallel to the direction of the first column of light sensors.The system of any of claims 4 to 14, wherein generating the data for the three-dimensional point cloud comprises adding data from a light sensor in the first column to data from a light sensor in the second column.The system of any of claims 4 to 15, wherein the second column of light sensors is configured to detect stray light radiated to and received from the first column of light sensors.A lidar method, comprising: emitting light pulses using one or more lasers; reflecting the light pulses into an environment using a mirror; detecting, using a detector, such light of the light pulses reflected from one or more objects in the environment, wherein: the detector comprises a first column of light sensors and a second column of light sensors; and the first column of light sensors detects light at a different intensity than the second column of light sensors; and generating data for a three-dimensional point cloud based on light detected by the detector, wherein the three-dimensional point cloud comprises at least one data point of the one or more objects in the environment.

Images