LIDAR Image Processing

Abstract

Systems and methods for processing lidar data are provided. Lidar data can be obtained in a specific manner that allows reconstruction of a rectilinear image, and image processing can be applied to the rectilinear image between images. For example, kernel-based image processing techniques can be used. Such processing techniques can use adjacent lidar and / or associated color pixels to adjust various values ​​associated with the lidar signal. This image processing of lidar and color pixels can be performed by dedicated circuitry, which can be on the same integrated circuit. In addition, lidar pixels can be related to each other. For example, classification techniques can identify lidar and / or associated color pixels as corresponding to the same object. Classification can be performed by an artificial intelligence (AI) coprocessor. The image processing techniques and classification techniques can be combined into a single system.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and is a PCT application of U.S. Provisional Application No. 62 / 897,122, filed September 6, 2019, entitled “Lidar Image Processing,” the entire contents of which are incorporated herein by reference for all purposes. Background Art

[0003] Light detection and ranging (LIDAR) systems are used for object detection and ranging, for example, in vehicles such as cars, trucks, and boats. LIDAR systems are also used in mobile applications (e.g., for facial recognition), home entertainment (e.g., capturing gestures for video game input), and augmented reality. LIDAR systems measure the distance to an object by illuminating a landscape with pulses from a laser and then measuring the time it takes for photons to travel to the object and return after reflection, as measured by the LIDAR system's receiver. The detected signal is analyzed to detect the presence of reflected signal pulses amidst background light. The distance to the object can be determined based on the time of flight from the transmission of the pulse to the reception of the corresponding reflected pulse.

[0004] Providing robust distance accuracy of a few centimeters under all conditions can be difficult, especially at the economic cost of LIDAR systems. Providing reliable data with extensive information about the entire surrounding environment, especially distant objects, can be even more difficult. Obtaining prior knowledge of such distant objects can be important for vehicle navigation.

[0005] Additionally, in applications such as vehicle navigation, depth information (e.g., the distance to objects in the environment) is very useful, but not sufficient to avoid hazards and navigate safely. Specific objects must also be identified, such as traffic signs, lane markings, and moving objects that may intersect the vehicle's path. However, analysis of 3D point clouds can require significant computing resources to perform in real time for these applications. Summary of the Invention

[0006] The present disclosure provides systems and methods for analyzing lidar data. For example, lidar data can be acquired in a specific manner that allows for the reconstruction of rectilinear images, to which image processing can be applied between images. For example, kernel-based image processing techniques can be used. Such processing techniques can use adjacent lidar and / or associated color pixels to adjust various values ​​associated with the lidar signal. This image processing of lidar and color pixels can be performed by dedicated circuitry, which can be on the same integrated circuit.

[0007] In some embodiments, lidar pixels can be correlated with each other. For example, classification techniques can identify lidar and / or associated color pixels as corresponding to the same object. Classification can be performed by an artificial intelligence (AI) coprocessor. Image processing techniques and classification techniques can be combined into a single system.

[0008] These and other embodiments of the present disclosure are described in detail below.For example, other embodiments are directed to systems, devices, and computer-readable media associated with the methods described herein.

[0009] The nature and advantages of the disclosed embodiments may be better understood with reference to the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1A and 1B An automotive light ranging device, also referred to herein as a LIDAR system, according to an embodiment of the present disclosure is shown.

[0011] Figure 2 A block diagram of an exemplary lidar device for implementing various embodiments is shown.

[0012] Figure 3 The operation of a typical lidar system that may be improved by embodiments is described.

[0013] Figure 4 An illustrative example of light transmission and detection processes for an optical ranging system according to an embodiment of the present disclosure is shown.

[0014] Figure 5 Various stages of a sensor array and associated electronics are shown according to an embodiment of the disclosure.

[0015] Figure 6 A histogram according to an embodiment of the present disclosure is shown.

[0016] Figure 7 An accumulation of histograms over multiple bursts for a selected pixel is shown in accordance with an embodiment of the present disclosure.

[0017] Figure 8 A series of positions for applying a matched filter to an original histogram is shown according to an embodiment of the present disclosure.

[0018] Figure 9 A panoramic lidar image to which depth values ​​from a pixel sensor have been assigned according to an embodiment of the present disclosure is shown.

[0019] Figure 10 Shown is a simplified front view of a sensor array according to an embodiment of the present invention.

[0020] Figure 11A and11B is a simplified conceptual diagram illustrating the potential for pointing errors in a scanning system using a sensor array.

[0021] Figure 12 An example of an imaging system using an F tan θ bulk optical module according to an embodiment of the present disclosure is shown.

[0022] Figure 13 A controller configured to identify signals in a histogram and form a linear array of depth values ​​to periodically generate lidar frames is shown according to an embodiment of the present disclosure.

[0023] Figure 14 A light ranging system including a controller and a lidar image processor according to an embodiment of the present disclosure is shown.

[0024] Figure 15 A light ranging system including a controller and a lidar AI coprocessor according to an embodiment of the present disclosure is shown.

[0025] Figure 16 A light ranging system including a controller, a lidar image processor, and a lidar AI coprocessor according to an embodiment of the present disclosure is shown.

[0026] Figure 17 is a flowchart illustrating a method of performing ranging using an optical ranging system installed on a mobile device according to an embodiment of the present disclosure.

[0027] Figure 18 is a flowchart illustrating a method of performing ranging using an optical ranging system installed on a mobile device according to an embodiment of the present disclosure.

[0028] Figure 19 is a flowchart illustrating a method of performing ranging using an optical ranging system installed on a mobile device.

[0029] Figure 20 is a flowchart illustrating a method for correcting a color image according to an embodiment of the present disclosure.

[0030] Figure 21 A block diagram is shown of an exemplary computer system that may be used with systems and methods according to embodiments of the present invention.

[0031] the term

[0032] The term "ranging," particularly when used in the context of methods and devices for measuring the environment or assisting in vehicle operation, may refer to determining the distance or distance vector from one position or location to another. "Optical ranging" may refer to a type of ranging method that utilizes electromagnetic waves to perform a ranging method or function. Thus, an "optical ranging device" may refer to a device for performing an optical ranging method or function. "Lidar" or "LIDAR" may refer to a type of optical ranging method that measures the distance to a target by illuminating the target with pulsed laser light and then measuring the reflected pulses with a sensor. Thus, a "lidar device" or "lidar system" may refer to a type of optical ranging device for performing a lidar method or function. An "optical ranging system" may refer to a system that includes at least one optical ranging device (e.g., a lidar device). The system may further include one or more other devices or components in various arrangements.

[0033] A "pulse train" may refer to one or more pulses transmitted together. The emission and detection of a pulse train may be referred to as an "excitation." The excitation may occur during a "detection time interval" (or "detection interval").

[0034] A "measurement" may include N pulse trains emitted and detected in N excitations, each excitation lasting a detection interval. The entire measurement may be in a measurement interval (or just a "measurement interval"), which may be equal to N detection intervals of the measurement or longer, for example, when a pause occurs between detection intervals.

[0035] A light sensor converts light into an electrical signal. A light sensor can include multiple light detectors, such as single-photon avalanche diodes (SPADs). A light sensor can correspond to a specific resolution pixel in distance measurement.

[0036] "Histogram" may refer to any data structure representing a series of values ​​over time, such as discrete values ​​in a time bin. A histogram may have a value assigned to each time bin. For example, a histogram may store a counter of the number of photodetectors that were activated during a specific time bin in each of one or more detection intervals. As another example, a histogram may correspond to the digitization of an analog signal at different times. A histogram may include a signal (e.g., a pulse) and noise. Thus, a histogram may be viewed as a combination of a signal and noise as a photon time series or photon flux. A raw / digitized histogram (or cumulative photon time series) may contain the signal and noise digitized in memory without filtering. A "filtered histogram" may refer to the output after the raw histogram has passed through a filter.

[0037] The emitted signal / pulse may refer to an undistorted "nominal," "ideal," or "template" pulse or pulse train. The reflected signal / pulse may refer to a reflected laser pulse from an object and may be distorted. The digitized signal / pulse (or raw signal) may refer to the digitized result of the detection of one or more pulse trains from a detection interval as stored in a memory, and may therefore be equivalent to a portion of a histogram. The detected signal / pulse may refer to the location in the memory where the signal was detected. The detected pulse train may refer to the actual pulse train found by the matched filter. The expected signal profile may refer to the shape of the digitized signal caused by a specific emitted signal with specific distortion in the reflected signal.

[0038] Lidar images can be formed into a two-dimensional (2D) image consisting of a rectilinear grid of lidar pixels. The number of rows and columns can be copied from one image to another so that a set of images has the same number of rows and columns of lidar pixels. For example, a lidar (depth) pixel can have three values: a depth value, a peak value (also called a signal value), and a noise value. A series of lidar images can be collected as a set of frames that can be played back and / or analyzed together. Based on the row, column, and depth value of the pixel in the 2D image, a three-dimensional position can be defined for the lidar pixel. DETAILED DESCRIPTION

[0039] This disclosure provides systems and methods for analyzing lidar data. For example, lidar data can be acquired in a specific manner that allows for the reconstruction of rectilinear images, to which image processing can be applied between images. Sensor IDs (possibly with the location of optical ranging devices) can enable consistent mapping of signal images to lidar pixels.

[0040] For image processing, kernel-based image processing techniques can be applied to the image. This processing technique can use adjacent lidar and / or associated color pixels to adjust various values ​​associated with the lidar signal, such as depth, peak value, and detection threshold, which can be used to detect reflected pulses. This image processing of lidar and color pixels can be performed by dedicated circuitry, which can be on the same integrated circuit.

[0041] In some embodiments, lidar pixels can be correlated with each other. For example, classification techniques can identify lidar and / or associated color pixels as corresponding to the same object. Classification can be performed by an artificial intelligence (AI) coprocessor. Image processing techniques and classification techniques can be combined into a single system. Further examples can detect key points in an object, such as for image registration or velocity determination.

[0042] I. Illustrative Automotive LIDAR System

[0043] Figure 1A and 1B An automotive optical ranging device, also referred to herein as a LIDAR system, is shown according to some embodiments. The automotive application of the LIDAR system is chosen here for illustration purposes only, and the sensors described herein can be used in other types of vehicles, such as ships, airplanes, trains, etc., as well as in a variety of other applications where 3D depth images are useful, such as medical imaging, mobile phones, augmented reality, geodesy, geomatics, archaeology, topography, geology, geomorphology, seismology, forestry, atmospheric physics, laser guidance, airborne laser swath mapping (ALSM), and laser altimetry. According to some embodiments, a LIDAR system, such as a scanning LIDAR system 101 and / or a solid-state LIDAR system 103, can be mounted on the roof of a vehicle 105, such as Figure 1A and 1B As shown in .

[0044] Figure 1A The scanning LIDAR system 101 shown in FIG. 1 can employ a scanning architecture, wherein the orientation of the LIDAR light source 107 and / or detector circuitry 109 can be scanned about one or more fields of view 110 within an external field or scene outside the vehicle 105. In the case of a scanning architecture, emitted light 111 can be scanned over the surrounding environment as shown. For example, the output beam of one or more light sources (e.g., infrared or near-infrared pulsed IR lasers, not shown) positioned in the LIDAR system 101 can be scanned (e.g., rotated) to illuminate the scene near the vehicle. In some embodiments, the scanning, represented by the rotation arrow 115, can be implemented mechanically, such as by mounting the light emitter on a rotating column or platform. In some embodiments, the scanning can be implemented by other mechanical means, such as by using a galvanometer. Chip-based steering techniques can also be employed, such as by using a microchip employing one or more MEMS-based reflectors, such as a digital micromirror (DMD) device, a digital light processing (DLP) device, and the like. In some embodiments, the scanning can be achieved by non-mechanical means, such as by using electronic signals to steer one or more optical phased arrays.

[0045] For fixed architectures, such as Figure 1B 1 , one or more solid-state LIDAR subsystems (e.g., 103a and 103b) may be mounted to a vehicle 105. Each solid-state LIDAR unit may face a different direction (possibly with partially overlapping and / or non-overlapping fields of view between the units) in order to capture a larger composite field of view than each unit could capture on its own.

[0046] In either a scanning or fixed architecture, objects within the scene may reflect portions of the light pulse emitted from the LIDAR light source. One or more reflected portions then travel back to the LIDAR system and may be detected by the detector circuitry. For example, reflected portion 117 may be detected by detector circuitry 109. The detector circuitry may be housed in the same housing as the emitter. Aspects of scanning and fixed systems are not mutually exclusive and, therefore, may be used in combination. For example, Figure 1B The individual LIDAR subsystems 103a and 103b may employ steerable emitters, such as optical phased arrays, or the entire composite unit may be rotated by mechanical means to scan the entire scene in front of the LIDAR system, for example, from field of view 119 to field of view 121.

[0047] Figure 2 A more detailed block diagram illustrating a rotating LIDAR system 200 according to some embodiments. More specifically, Figure 2 Optionally described is a rotary LIDAR system that may employ a rotary actuator on a rotating circuit board, which may receive power and data (and transmit) from a fixed circuit board.

[0048] LIDAR system 200 can interact with one or more examples of a user interface 215. Different examples of user interface 215 can vary and may include, for example, a computer system with a monitor, keyboard, mouse, CPU, and memory; a touch screen in an automobile; a handheld device with a touch screen; or any other suitable user interface. User interface 215 can be local to the object on which LIDAR system 200 is mounted, but can also be a remotely operated system. For example, commands and data to and from LIDAR system 200 can be routed via a cellular network (LTE, etc.), a personal area network (Bluetooth, Zigbee, etc.), a local area network (WiFi, IR, etc.), or a wide area network such as the Internet.

[0049] The user interface 215 of hardware and software can present LIDAR data from the device to the user, but can also allow the user to control the LIDAR system 200 with one or more commands. Example commands may include commands to activate or deactivate the LIDAR system, specify light detector exposure level, bias, sampling duration, and other operating parameters (e.g., transmit pulse pattern and signal processing), and specify light emitter parameters (such as brightness). In addition, commands can allow the user to select a method for displaying results. The user interface can display LIDAR system results, which can include, for example, a single-frame snapshot image, a constantly updated video image, and / or a display of other light measurements for some or all pixels. In some embodiments, the user interface 215 can track the distance (proximity) of objects from the vehicle and potentially provide warnings to the driver or provide such tracking information for analysis of driver performance.

[0050] In some embodiments, the LIDAR system can communicate with the vehicle control unit 217 and can modify one or more parameters associated with the control of the vehicle based on the received LIDAR data. For example, in a fully autonomous vehicle, the LIDAR system can provide real-time 3D images of the car's surroundings to assist in navigation. In other cases, the LIDAR system can be used as part of an advanced driver assistance system (ADAS) or as part of a safety system, which can, for example, provide 3D image data to any number of different systems, such as adaptive cruise control, automatic parking, driver drowsiness monitoring, blind spot monitoring, collision avoidance systems, etc. When the vehicle control unit 217 is communicatively coupled to the optical ranging device 210, a warning can be provided to the driver or tracking of the proximity of trackable objects can be provided.

[0051] Figure 2 The LIDAR system 200 shown in FIG. 1 includes an optical ranging device 210. The optical ranging device 210 includes a ranging system controller 250, an optical transmission (Tx) module 240, and an optical sensing (Rx) module 230. The optical ranging device can generate ranging data by transmitting one or more light pulses 249 from the optical transmission module 240 to objects in the field of view around the optical ranging device. A reflected portion 239 of the transmitted light is then detected by the optical sensing module 230 after some delay time. Based on the delay time, the distance to the reflecting surface can be determined. Other ranging methods, such as continuous wave, Doppler, and similar methods, can also be used.

[0052] The Tx module 240 includes an emitter array 242, which can be a one-dimensional or two-dimensional array of emitters, and a Tx optical system 244, which, when combined, can form an array of micro-optical emitter channels. The emitter array 242 or individual emitters are examples of laser sources. The Tx module 240 further includes a processor 245 and a memory 246. In some embodiments, pulse coding techniques such as Barker codes and the like can be used. In such cases, the memory 246 can store a pulse code indicating when light should be transmitted. In one embodiment, the pulse code is stored as a sequence of integers stored in the memory.

[0053] The Rx module 230 may include a sensor array 236, which may be, for example, a one-dimensional or two-dimensional array of light sensors. Each light sensor or photosensitive element (also referred to as a sensor) may include a collection of light detectors, such as APDs or the like, or the sensor may be a single-photon detector (e.g., SPAD). Similar to the Tx module 240, the Rx module 230 includes an Rx optics system 237. Combined, the Rx optics system 237 and the sensor array 236 may form an array of micro-optical receiver channels. Each micro-optical receiver channel measures light corresponding to an image pixel in a distinct field of view of the surrounding volume. For example, due to the geometric configuration of the light sensing module 230 and the light transmission module 240, each sensor (e.g., a collection of SPADs) of the sensor array 236 may correspond to a specific emitter of the emitter array 242.

[0054] In one embodiment, the sensor array 236 of the Rx module 230 is fabricated as part of a monolithic device on a single substrate (using, for example, CMOS technology) and includes an array of photon detectors and an ASIC 231 for signal processing raw histograms from individual photon detectors (or groups of detectors) in the array. As an example of signal processing, for each photon detector or group of photon detectors, the memory 234 (e.g., SRAM) of the ASIC 231 can accumulate counts of photons detected over consecutive time intervals, and these time intervals, when combined, can be used to recreate a time series of reflected light pulses (i.e., photon counts versus time). This time series of accumulated photon counts is referred to herein as an intensity histogram (or just histogram). The ASIC 231 can implement matched filters and peak detection processing to identify return signals in a timely manner. In addition, the ASIC 231 can implement certain signal processing techniques (e.g., via the signal processor 238), such as multi-curve matched filtering, to help recover a photon time series that is less susceptible to pulse shape distortion due to SPAD saturation and quenching. In some embodiments, all or part of such filtering may be performed by processor 258, which may be implemented in an FPGA. In some instances, signal processor 238 and memory 234 may be considered part of the ranging circuitry. For example, signal processor 238 may count the number of photodetectors that detect photons during a measurement to form a histogram, which may be used to detect peaks corresponding to range (depth) values.

[0055] In some embodiments, the Rx optical system 237 can also be part of the same monolithic structure as the ASIC, with separate substrate layers for each receiver channel layer. For example, the aperture layer, collimating lens layer, filter layer, and photodetector layer can be stacked and bonded at the wafer level before dicing. The aperture layer can be formed by placing an opaque substrate on top of a transparent substrate or by coating a transparent substrate with an opaque film. In yet other embodiments, one or more components of the Rx module 230 can be external to the monolithic structure. For example, the aperture layer can be implemented as a separate metal sheet with pinholes.

[0056] In some embodiments, the photon time series output from the ASIC is sent to the ranging system controller 250 for further processing. For example, the data may be encoded by one or more encoders of the ranging system controller 250 and then sent as data packets to the user interface 215. The ranging system controller 250 can be implemented in a variety of ways, including, for example, by using a programmable logic device such as an FPGA as an ASIC or part of an ASIC, using a processor 258 with a memory 254, and some combination thereof. The ranging system controller 250 can operate in conjunction with a fixed base controller or independently of the base controller (via pre-programmed instructions) to control the light sensing module 230 by sending commands, including starting and stopping light detection and adjusting light detector parameters. Similarly, the ranging system controller 250 can control the light transmission module 240 by sending or forwarding commands from the base controller, including starting and stopping light emission control and controls that can adjust other light emitter parameters (e.g., pulse code). In some embodiments, the ranging system controller 250 has one or more wired interfaces or connectors for exchanging data with the light sensing module 230 and the light transmission module 240. In other embodiments, the ranging system controller 250 communicates with the light sensing module 230 and the light transmission module 240 via a wireless interconnect, such as an optical communication link.

[0057] The motor 260 may be an optional component required when system components such as the Tx module 240 and / or the Rx module 230 need to rotate. The system controller 250 controls the motor 260 and can start rotation, stop rotation, and change the rotation speed.

[0058] The optical ranging device 210 may also include a lidar image processor and an AI coprocessor. For a scanning system, the optical ranging device 210 may include an image reconstruction processor, for example, which can generate a 2D image based on measurements taken over a given cycle (e.g., a 360° rotation). Such an image reconstruction processor (e.g., the ranging system controller 250) can cache input data and assign lidar pixels to pixels in the image according to a mapping table / function, thereby reconstructing the image. The image can be constructed because the lidar pixels already exist but are arranged to form an image, for example, a straight line image with consistent resolution from one image to another. Other processors can use the reconstructed image to provide the final processed image or provide other data. These processors, the ranging system controller 250, the light sensing module 230, and the light transmission module 240 can be on the same or different integrated circuits.

[0059] II. Detection of Reflected Pulses

[0060] The light sensors can be arranged in a variety of ways for detecting the reflected pulses. For example, the light sensors can be arranged in an array, and each light sensor can include an array of light detectors (e.g., SPADs). Different patterns of pulses (pulse trains) transmitted during the detection interval are also described below.

[0061] A. Time-of-Flight Measurement and Detector

[0062] Figure 3 The operation of a typical LIDAR system, which may be improved by some embodiments, is illustrated. A laser generates a short-duration light pulse 310. The horizontal axis represents time, and the vertical axis represents power. Example laser pulse durations, as characterized by full width at half maximum (FWHM), are a few nanoseconds, with a peak power of a single emitter on the order of a few watts. Embodiments using side-emitter lasers or fiber lasers can have much higher peak powers, while embodiments with small-diameter VCSELs can have peak powers in the tens to hundreds of milliwatts.

[0063] The start time 315 of the transmission of the pulse need not coincide with the leading edge of the pulse. As shown, the leading edge of the light pulse 310 may be after the start time 315. It may be desirable for the leading edge to be different in situations where pulses of different patterns are transmitted at different times, for example, for coded pulses.

[0064] The optical receiver system can begin detecting received light simultaneously with the start of the laser, i.e., at the start time. In other embodiments, the optical receiver system can begin detecting received light at a later time, a known time after the start time of the pulse. The optical receiver system initially detects background light 330 and, some time later, detects laser pulse reflection 320. The optical receiver system can compare the detected light intensity to a detection threshold to identify laser pulse reflection 320. The detection threshold can distinguish between background light 330 and light corresponding to laser pulse reflection 320.

[0065] Time of flight 340 is the time difference between a transmitted pulse and a received pulse. This time difference can be measured by subtracting the transmission time of the pulse (e.g., as measured relative to the start time) from the reception time of the laser pulse reflection 320 (e.g., also measured relative to the start time). The distance to the target can be determined as half the product of the time of flight and the speed of light. Pulses from the laser device reflect from objects in the scene at different times, and the pixel array detects the pulses that radiate the reflections.

[0066] B. Object Detection Using Array Lasers and Light Sensor Arrays

[0067] Figure 4 An illustrative example of a light transmission and detection process for an optical ranging system is shown in accordance with some embodiments. Figure 4 An optical ranging system (eg, solid-state and / or scanning) is shown that collects three-dimensional distance data for a volume or scene surrounding the system. Figure 4 This is an idealized diagram to highlight the relationship between the emitter and the sensor, and therefore other components are not shown.

[0068] Optical ranging system 400 includes a light emitter array 402 and a light sensor array 404. Light emitter array 402 includes a light emitter array, such as a VCSEL array and the like, such as emitter 403 and emitter 409. Light sensor array 404 includes a light sensor array, such as sensors 413 and 415. The light sensor can be a pixelated light sensor that employs a collection of discrete light detectors for each pixel, such as single photon avalanche diodes (SPADs) and the like. However, various embodiments may deploy any type of photon sensor.

[0069] Each emitter can be slightly offset from its neighbors and can be configured to transmit light pulses into a different field of view than its neighbors, thereby illuminating only the respective field of view associated with that emitter. For example, emitter 403 emits an illumination beam 405 (formed by one or more light pulses) into a circular field of view 407 (which is exaggerated in size for clarity). Similarly, emitter 409 emits an illumination beam 406 (also referred to as an emitter channel) into a circular field of view 410. Although Figure 4Not shown in to avoid complexity, each emitter emits a corresponding illumination beam into its corresponding field of view, resulting in a 2D array of illuminated fields of view (in this example, 21 distinct fields of view).

[0070] Each field of view illuminated by an emitter can be considered as a pixel or spot in the corresponding 3D image generated from the ranging data. Each emitter channel can be distinct for each emitter and non-overlapping with other emitter channels, i.e., there is a one-to-one mapping between a set of emitters and a set of non-overlapping fields or viewing angles. Thus, in Figure 4 In the example of , the system can sample 21 distinct points in 3D space. A denser sampling of points can be achieved by having a denser array of emitters or by scanning the angular position of the emitter beam over time so that one emitter can sample several points in space. As described above, scanning can be achieved by rotating the entire emitter / sensor assembly.

[0071] Each sensor can be slightly offset from its neighbors, and similar to the transmitters described above, each sensor can see a different field of view of the scene in front of the sensor. In addition, the field of view of each sensor is generally consistent with the field of view of the corresponding transmitter channel, for example, overlapping with it and the same size.

[0072] exist Figure 4 In the example, the distances between corresponding emitter-sensor channels are magnified relative to the distances to objects in the field of view. In reality, the distances to objects in the field of view are much greater than the distances between corresponding emitter-sensor channels, and thus the path of light from the emitter to the object is roughly parallel to the path of light reflected from the object back to the sensor (i.e., it is almost "reflected back"). Consequently, there is a range of distances ahead of system 400 where the fields of view of individual sensors and emitters overlap.

[0073] Because the field of view of an emitter overlaps with the field of view of its corresponding sensor, each sensor channel can ideally detect the reflected illumination beam originating from its corresponding emitter channel, ideally without crosstalk, i.e., without detecting reflected light from other illumination beams. Thus, each light sensor can correspond to a respective light source. For example, emitter 403 emits illumination beam 405 into a circular field of view 407, and some of the illumination beam reflects from object 408. Ideally, reflected beam 411 is detected only by sensor 413. Thus, emitter 403 and sensor 413 share the same field of view, e.g., field of view 407, and form an emitter-sensor pair. Similarly, emitter 409 and sensor 415 form an emitter-sensor pair, sharing field of view 410. Although emitter-sensor pairs are Figure 4 are shown in the same relative orientation in their respective arrays, but any emitter can be paired with any sensor depending on the design of the optics used in the system.

[0074] During ranging measurements, reflected light from different fields of view distributed around the volume surrounding the LIDAR system is collected and processed by various sensors to derive distance information for any object in each respective field of view. As described above, time-of-flight techniques can be used, in which a light emitter emits a precisely timed pulse, and after some elapsed time, the reflection of the pulse is detected by a respective sensor. The time elapsed between emission and detection, along with the known speed of light, is then used to calculate the distance to the reflective surface. In some embodiments, additional information can be obtained from the sensor to determine other properties of the reflective surface in addition to the distance. For example, the Doppler shift of the pulse can be measured by the sensor and used to calculate the relative velocity between the sensor and the reflective surface. The pulse intensity can be used to estimate the target reflectivity, and the pulse shape can be used to determine whether the target is a hard or diffuse material.

[0075] In some embodiments, the LIDAR system may be composed of a relatively large 2D array of emitter and sensor channels and operate as a solid-state LIDAR, i.e., it may obtain frames of range data without scanning the orientation of the emitter and / or sensor. In other embodiments, the emitter and sensor may scan, e.g., rotate about an axis, to ensure that the field of view of the collection of emitters and sensors samples a full 360-degree area (or some useful portion of the 360-degree area) of the surrounding volume. For example, the range data collected from the scanning system over a predefined time period may then be post-processed into one or more data frames, which may then be further processed into one or more depth images or 3D point clouds. The depth images and / or 3D point clouds may be further processed into map tiles for use in 3D mapping and navigation applications.

[0076] C. Multiple photodetectors in each photosensor

[0077] Figure 5 The various stages of a sensor array and associated electronics according to an embodiment of the present invention are shown. Array 510 shows light sensors 515 (also referred to as pixel sensors), each corresponding to a different pixel. Array 510 can be an interleaved array. In this particular example, array 510 is an 18x4 light sensor. Array 510 can be used to achieve high resolutions (e.g., 72x1024) because the embodiment is suitable for scanning.

[0078] Array 520 shows a magnified view of a portion of array 510. As can be seen, each photosensor 515 is composed of multiple photodetectors 525. The signals from the photodetectors of a pixel collectively contribute to the measurement of that pixel. As shown, the photosensors 515 in array 510 can be staggered to form a staggered array. When array 510 is scanned in incremental steps corresponding to one column at a time (e.g., through a motion such as rotation or using a moving mirror), the photosensors 515 provide a vertical resolution of the number of rows times the number of columns. The staggered array allows the photosensor to include more photodetectors 525 than would otherwise be possible for the same vertical resolution.

[0079] In some embodiments, each pixel has a large number of single-photon avalanche diode (SPAD) cells, which increases the dynamic range of the pixel itself. Each SPAD can have analog front-end circuits for biasing, quenching, and recharging. SPADs are typically biased at a bias voltage higher than the breakdown voltage. Suitable circuitry senses the leading edge of the avalanche current, generates a standard output pulse synchronized with the avalanche buildup, quenches the avalanche by reducing the bias below the breakdown voltage, and restores the photodiode to operating levels.

[0080] SPADs can be positioned to maximize the fill factor in their local area, or a microlens array can be used, which allows for a high optical fill factor at the pixel level. Thus, an imager pixel can include an array of SPADs to increase the efficiency of the pixel detector. A diffuser can be used to diffuse the radiation passing through the aperture and collimated by the microlens. A tank diffuser is used to diffuse the collimated radiation in such a way that all SPADs belonging to the same pixel receive some radiation.

[0081] Figure 5 Further shown is a specific photodetector 530 (e.g., a SPAD) that detects a photon 532. In response to the detection, the photodetector 530 generates an avalanche current 534 of charge carriers (electrons or holes). A threshold circuit 540 regulates the avalanche current 534 by comparing it to a threshold. When a photon is detected and the photodetector 530 is functioning properly, the avalanche current 534 rises above the comparator threshold, and the threshold circuit 540 generates a temporally accurate binary signal 545 indicating the exact time of the SPAD current avalanche, which in turn is an accurate measurement of the arrival of the photon. The correlation of the current avalanche with the arrival of the photon can occur with a resolution of nanoseconds, thereby providing high timing resolution. The rising edge of the binary signal 545 can be latched by a pixel counter 550, which can be part of the ranging circuit.

[0082] Binary signal 545, avalanche current 534, and pixel counter 550 are examples of data values ​​that can be provided by a photosensor including one or more SPADs. The data value can be determined from a corresponding signal from each of a plurality of photodetectors. Each of the corresponding signals can be compared to a threshold value to determine whether the corresponding photodetector is triggered. Avalanche current 534 is an example of an analog signal, and thus the corresponding signal can be an analog signal.

[0083] Pixel counter 550 can use binary signal 545 to count the number of photodetectors for a given pixel that have been triggered by one or more photons during a specific time interval (e.g., a time window of 1, 2, 3, etc. nanoseconds) controlled by periodic signal 560. Pixel counter 550 can store a counter for each of multiple time intervals for a given measurement. The value of the counter for each time interval can start at zero and increment based on binary signal 545 indicating that a photon was detected. The counter can increment when any photodetector of the pixel provides such a signal.

[0084] The periodic signal 560 can be generated by a phase-locked loop (PLL) or a delay-locked loop (DLL), or any other method of generating a clock signal. The coordination of the periodic signal 560 and the pixel counter 550 can act as a time-to-digital converter (TDC), which is a device for identifying events and providing a digital representation of the time at which they occur. For example, the TDC can output the arrival time of each detected photon or optical pulse. The measured time can be the time elapsed between two events (e.g., the start time and the detected photon or optical pulse) rather than the absolute time. The periodic signal 560 can be a relatively fast clock that switches between a group of memories including the pixel counter 550. Each register in the memory can correspond to a histogram interval, and the clock can switch between them at sampling intervals. Therefore, when the corresponding signal is greater than a threshold, a binary value indicating a trigger can be sent to the histogram circuit. The histogram circuit can aggregate the binary values ​​across multiple photodetectors to determine the number of photodetectors that triggered during a specific time interval.

[0085] The time interval can be measured relative to the start signal, e.g. Figure 3The start time 315 of the start signal. Therefore, the counter of the time interval just after the start signal may have a low value corresponding to the background signal, such as the background light 330. The last time interval may correspond to the end of the detection time interval (also called excitation) of a given pulse train, which is further described in the next section. The number of cycles of the periodic signal 560 since the start time can serve as a timestamp when the rising edge of the avalanche current 534 indicates the detected photon. The timestamp corresponds to the time interval for a specific counter in the pixel counter 550. This type of operation is different from a simple analog-to-digital converter (ADC) following a photodiode (e.g., for an avalanche diode (APD)). Each of the counters of the time interval can correspond to a histogram, which is described in more detail below. Therefore, while the APD is a linear amplifier for the input optical signal with a finite gain, the SPAD is a trigger device that provides a binary output of yes / no for a trigger event that occurred in a time window.

[0086] D. Pulse train

[0087] Ranging can also be achieved by using a pulse train, which is defined as containing one or more pulses. Within a pulse train, the number of pulses, the width of the pulses, and the duration between pulses (collectively referred to as the pulse pattern) can be selected based on several factors, some of which include:

[0088] 1 - Maximum Laser Duty Cycle - The duty cycle is the fraction of time the laser is on. For a pulsed laser this can be determined by the FWHM as explained above and the number of pulses emitted during a given period.

[0089] 2- Eye Safety Limit - This is determined by the maximum amount of radiation that the device can emit without harming the eyes of a bystander who happens to be looking in the direction of the LIDAR system.

[0090] 3- Power Consumption - This is the power consumed by the emitter to illuminate the scene.

[0091] For example, the intervals between pulses in a pulse train may be on the order of single or 10-digit nanoseconds.

[0092] Multiple pulse trains may be transmitted during the time span of one measurement. Each pulse train may correspond to a different time interval, for example, a subsequent pulse train is not transmitted before a time limit for detecting a reflected pulse of a previous pulse train expires.

[0093] For a given emitter or laser device, the time between the transmissions of a pulse train determines the maximum detectable range. For example, if pulse train A is transmitted at time t0 = 0 ns and pulse train B is transmitted at time t1 = 1000 ns, then reflected pulse trains detected after t1 must not be assigned to pulse train A because they are more likely to be reflections from pulse train B. Therefore, the time between pulse trains and the speed of light define the maximum limit on the range of the system given in the following equation.

[0094] R max =c×(t1-t0) / 2

[0095] The time between excitations (emission and detection of the pulse train) may be approximately 1 μ8 to allow sufficient time for the entire pulse train to travel to a distant object approximately 150 meters away, and then back.

[0096] III. Histogram Signal from Photodetector

[0097] One mode of operation of a LIDAR system is time correlated single photon counting (TCSPC), which is based on counting single photons in a periodic signal. This technique works well for low levels of periodic radiation, which is appropriate in LIDAR systems. This time correlated counting can be affected by Figure 5 The periodic signal 560 is controlled and time intervals can be used, such as for Figure 5 Discussed.

[0098] The frequency of a periodic signal can specify the temporal resolution within which data values ​​of the signal are measured. For example, one measurement value can be obtained for each light sensor during each cycle of the periodic signal. In some embodiments, the measurement value can be the number of light detectors triggered during the cycle. The time period of the periodic signal corresponds to a time interval, where each cycle is a different time interval.

[0099] Figure 6 A histogram 600 is shown according to an embodiment of the present invention. The horizontal axis corresponds to a time interval as measured relative to a start time 615. As described above, the start time 615 may correspond to the start time of the pulse train. Any offset between the rising edge of the first pulse of the pulse train and the start time of either or both of the pulse train and the detection time interval may be taken into account, wherein determining the reception time will be used for the time-of-flight measurement. The vertical axis corresponds to the number of triggered SPADs. In certain embodiments, the vertical axis may correspond to the output of an ADC following the APD. For example, an APD may exhibit traditional saturation effects such as a constant maximum signal rather than the dead-time-based effects of a SPAD. Some effects may occur for both SPADs and APDs, for example, pulse tailing of extremely tilted surfaces may occur for both SPADs and APDs.

[0100] The counter for each of the time intervals corresponds to a different bar in histogram 600. The counter for the early time intervals is relatively low and corresponds to background noise 630. At some point, a reflected pulse 620 is detected. The corresponding counter is much larger and may be above the threshold for distinguishing between background and detected pulses. Reflected pulse 620 (after digitization) is shown as corresponding to four time intervals, which may be caused by laser pulses of similar width, such as 4 ns pulses when the time intervals are each 1 ns. However, as described in more detail below, the number of time intervals may vary, for example, based on the properties of the specific object in the incident angle of the laser pulse.

[0101] The temporal position of the time interval corresponding to the reflected pulse 620 can be used, for example, to determine the time of reception relative to the start time 615. As described in more detail below, a matched filter can be used to identify pulse patterns, thereby effectively increasing the signal-to-noise ratio and more accurately determining the time of reception. In some embodiments, the accuracy of determining the time of reception can be less than the temporal resolution of a single time interval. For example, for a 1 ns time interval, the resolution would correspond to approximately 15 cm. However, it may be desirable to have an accuracy of only a few centimeters.

[0102] Thus, a detected photon can cause a particular time interval of the histogram to be incremented based on its arrival time relative to a start signal, such as indicated by start time 615. The start signal can be periodic, such that multiple pulse trains are sent during a measurement. Each start signal can be synchronized to a laser pulse train, wherein multiple start signals cause multiple pulse trains to be transmitted within multiple detection intervals. Thus, a time interval (e.g., from 200 to 201 ns after the start signal) will occur for each detection interval. The histogram can accumulate counts, wherein the count for a particular time interval corresponds to the sum of the measured data values ​​across multiple excitations that all occurred in the particular time interval. When the detected photons are histogrammed based on such a technique, it results in a return signal with a signal-to-noise ratio greater than that of a single pulse train by the square root of the number of excitations made.

[0103] Figure 7 An accumulation of histograms over multiple bursts for a selected pixel is shown in accordance with an embodiment of the present invention. Figure 7 Three detected pulse trains 710, 720, and 730 are shown. Each detected pulse train corresponds to a transmitted pulse train with the same pattern of two pulses separated by the same amount of time. Thus, each detected pulse train has the same pulse pattern, as shown by the two time intervals having distinct values. Counters for other time intervals are not shown for ease of illustration, but these other time intervals may have relatively low non-zero values.

[0104] In the first detected pulse train 710, the counters for time intervals 712 and 714 are identical. This may be due to the same number of photodetectors detecting photons during both time intervals. Or, in other embodiments, approximately the same number of photons are detected during both time intervals. In other embodiments, more than one consecutive time interval may have consecutive non-zero values; however, for ease of illustration, a few non-zero time intervals are shown.

[0105] Time intervals 712 and 714 occur 458ns and 478ns after start time 715, respectively. The displayed counters for the other detected pulse trains occur at the same time intervals relative to their respective start times. In this example, start time 715 is identified as occurring at time 0, but the actual time is arbitrary. The first detection interval for the first detected pulse train may be 1 μs. Thus, the number of time intervals measured from start time 715 may be 1,000. This first detection interval then ends and a new pulse train may be transmitted and detected. The start and end of the different time intervals may be controlled by a clock signal, which may be part of a circuit acting as a time-to-digital converter (TDC), such as in Figure 5 Described in .

[0106] For the second detected pulse train 720, the start time 725 is 1 μs, at which time, for example, the second pulse train may be transmitted. Such separate detection intervals can occur so that any pulses transmitted at the beginning of the first detection interval will have already been detected and, therefore, will not cause confusion with pulses detected in the second time interval. For example, if there is no additional time between excitations, the circuit may confuse a back-reflected stop sign at 200 m with a much less reflective object at 50 m (assuming an excitation period of approximately 1 μs). The two detection time intervals for pulse trains 710 and 720 can be of the same length and have the same relationship to the respective start times. Time intervals 722 and 724 occur at the same relative times of 458 ns and 478 ns as time intervals 712 and 714. Therefore, when the accumulation step occurs, the corresponding counters can be added. For example, the counter values ​​at time intervals 712 and 722 can be added together.

[0107] For the third detected pulse train 730, the start time 735 is 2 μs, for example, during which the third pulse train may be transmitted. Time intervals 732 and 734 also occur at 458 ns and 478 ns relative to their respective start times 735. Even if the transmitted pulses have the same power, the counter values ​​at different time intervals may have different values, for example, due to the random nature of the scattering process of the light pulses leaving the object.

[0108] Histogram 740 shows the accumulation of counters from three detected bursts at time intervals 742 and 744, which also correspond to 458 ns and 478 ns. Histogram 740 may have a smaller number of time intervals measured during the corresponding detection interval, for example, due to discarding time intervals at the beginning or end, or having values ​​less than a threshold. In some embodiments, depending on the pattern of the bursts, approximately 10 to 30 time intervals may have significant values.

[0109] For example, the number of pulse trains transmitted during a measurement to create a single histogram can be approximately 1 to 40 (e.g., 24), but can also be much higher, such as 50, 100, or 500. Once a measurement is completed, the counter for the histogram can be reset, and a collection of pulse trains can be transmitted to perform a new measurement. In various embodiments, and depending on the number of detection intervals in the corresponding duration, measurements can be performed every 25, 50, 100, or 500 μs. In some embodiments, the measurement intervals can overlap, for example, so that a given histogram corresponds to a specific sliding window of pulse trains. In such instances, memory may be present for storing multiple histograms, each corresponding to a different time window. Any weights applied to the detected pulses can be the same for each histogram, or such weights can be controlled independently.

[0110] IV. Generate LIDAR (depth) images

[0111] A depth image (or lidar image or frame) can be generated from a set of histograms corresponding to different pixel sensors, which may have been taken at different times. Figure 5 The staggered array 510 can be used to obtain a set of histograms for a measurement; the array can be scanned (e.g., linearly moved or rotated), and then a new set of measurements can be taken. Each histogram for a given pixel sensor can be used to determine the depth value of a pixel in the lidar image (e.g., a linear array of pixels covering 360°).

[0112] To determine the lidar image, a matched filter can be applied to each histogram to determine the depth value. The depth value (and potentially the peak / signal value and noise value) can be assigned to a specific pixel in the rectilinear 2D array of the lidar image. The assignment can be based on both the position in the array and the angular position of the rotation. More details on generating the lidar image are provided below.

[0113] A. Matched filtering and peak detection

[0114] As mentioned above, a matched filter can be used to determine the temporal position of a detected pulse (the time of reception). The time of reception can then be used to determine the total time of flight of the pulse, which can then be converted to distance.

[0115] Figure 8 1. A series of positions for applying a matched filter to an original histogram according to an embodiment of the present disclosure is shown. The series of positions can be viewed as sliding positions because the filter slides (moves) on the original histogram. The original histogram 802 depicts a relatively small time window around the detected pulse of a single pixel. The filter 804 corresponds to the shape of the digitized pulse. Both the histogram 802 and the filter 804 have idealized shapes for ease of presentation. Figure 8 The series of plots to the left of shows different positions of the filter 804 relative to the histogram 802. The filter 804 is shifted by one time bin in each successive plot.

[0116] The filtered output resulting from the application of filter 804 to histogram 802 is Figure 8 802 at each position. The bar graph shows the level of overlap between filter 804 and histogram 802 at each position. Filtered output 810 corresponds to the first position, where the overlap between filter 804 and histogram 802 is only one time interval. The vertical axis of filtered output 810 is in arbitrary units of overlap, for example, the sum of the multiplication products of the corresponding values ​​of filter 804 and histogram 802. The values ​​of filtered output 810 are shown at the time interval corresponding to the center of filter 804. This can be done when the center of the pulse is to be detected. In other embodiments, the values ​​of the filtered output can be shown at the time corresponding to the leftmost interval of the filter. This can be done when the rising edge of the pulse is to be detected. The values ​​of the center and rising edge can be derived from each other. For example, when the pulses in the pulse train are of different widths, it can be easier to define the position of the first filter tap (essentially the rising edge). The different widths can be caused by different emission widths or due to different detected intensities.

[0117] Filtered output 820 corresponds to the second position, where the overlap is two time intervals, and therefore the resulting value is twice that of filtered output 810. The values ​​shown show different time intervals, followed by filtered output 810, because filter 804 has been shifted to the right by one time interval. Filtered output 830 corresponds to the third position, where the overlap is three time intervals. Filtered output 840 corresponds to the fourth position, where the overlap is four time intervals. Filtered output 850 corresponds to the safe position, where the overlap is five time intervals. It is easy to see that the fifth position is the highest, corresponding to a perfect overlap between filter 804 and histogram 802.

[0118] The final filtered output 890 shows the value at each of the nine locations where there is some level of overlap between the filter 804 and the histogram 802. This filtered output can be analyzed to identify the maximum corresponding to the reception time of the detected pulse. In various embodiments, this time can be recorded directly or modified (e.g., to identify where the leading edge will be) as part of the time of flight measurement.

[0119] The maximum value in the filtered output 890 (also referred to as a filtered histogram) can be identified as a peak 895 (also referred to as a peak or signal value). The time interval of the peak 895 corresponds to the detection time of the received pulse. In some embodiments, an interpolation filter can be used to determine a finer resolution than the width of the time interval. The detected time can correspond to a distance (depth value) based on when the pulse train was transmitted.

[0120] B. Assign depth values ​​to the depth image

[0121] The depth values ​​determined from the histogram can be organized into a lidar image, such as a complete 360-degree panoramic lidar image. As described above, the lidar device can be rotated and measurements can be obtained at specified angular positions. Depth values ​​at different angular positions can be aggregated to provide a lidar image. The aggregation (assignment) of depth values ​​can account for the specific position of each light sensor and when (e.g., angular position) the measurement resulting in the depth value was obtained.

[0122] Figure 9 A panoramic LIDAR image 903 is shown to which depth values ​​from pixel sensors have been assigned, in accordance with an embodiment of the present disclosure. Depth values ​​from pixel sensor 910a are assigned to depth pixels 911 (also referred to as lidar pixels) in lidar image 903. Figure 9 The assignment of some pixel sensors 910 for two angular positions of a lidar device is shown. The angular positions can be generated by a rotary encoder. The assignment of the measurement value from a given pixel sensor at a given position can use a mapping (assignment) table or function, which can be hard-coded or stored in memory. An example assignment is provided below.

[0123] The top horizontal axis 902 depicts the azimuth (angular position) of the scanning lidar device. As shown, there are seven angular positions between 0° and 1°. Each staggered pixel sensor is shifted to the right by one angular position, which corresponds to one column in the lidar image. The measurement interval can span a range of angular positions.

[0124] For simplicity, it is assumed that the light sensing module (eg, Figure 2The 230 has a 2×4 interleaved array of pixel sensors, each of which can be composed of multiple photodetectors. Measurements from the eight pixel sensors provide an image with eight rows and columns corresponding to the number of angular positions. Assuming seven angular positions per degree, the horizontal resolution is 2,520. Other pixel sensor arrays and angular position measurement frequencies will provide different vertical and horizontal resolutions, respectively.

[0125] In the example shown, the assignment is as follows. At corner position #1, the pixel sensor in the top left corner of the staggered array is assigned to the top left depth pixel. The top pixel in the second column is assigned to the pixel in the second row and second column (position 2:2). After moving to position #2, the top left pixel sensor will move to depth pixel 1:2 because it was at the top vertical position, but is now in the second column.

[0126] Once the lidar unit is in its final position (e.g., position 2,520), a 2:1 ratio of depth pixels will be allocated because the top pixel sensor in the second column will be aligned with the first column in the lidar image.

[0127] As described above, different mappings (assignments) can be used. For example, data can be streamed sequentially from the sensor array in a specific order, or with identifiers indicating which data corresponds to which pixel sensor and at which location. The data can be rearranged into a rectilinear image in the memory buffer according to its true spatial relationship (e.g., defined by a mapping table), allowing the image to be efficiently processed as an image. Therefore, a variety of techniques can be used to reconstruct the lidar image derived from the data stream source in memory.

[0128] The depth values ​​of the resulting lidar image can be filtered and processed to improve accuracy, correct for missing information or noisy data, and so on. The spatial relationships of the depth pixels correspond to the spatial relationships of the light sensors at a given location. These spatial relationships can be used to process the lidar image, for example, how to apply filter kernels, and by an AI coprocessor, which can use machine learning to classify a group of depth pixels as corresponding to the same object.

[0129] C. Color and Depth

[0130] In addition to measuring depth, embodiments can also measure the color of ambient light. Such a color sensor can be on the same sensor. Ambient light of various wavelengths can be measured, including wavelengths within or outside the visible spectrum. Here, the term color should be interpreted broadly to cover any wavelength range. Other measurements of optical properties besides color can also be used, such as polarization data, such as near birefringence, circular birefringence (also known as optical rotation or optical dispersion), linear dichroism, and circular dichroism. Color is used as an example only, and the description of color can also be applied to other measurements of optical properties.

[0131] Figure 10 1 shows a simplified front view of a sensor array 1000 according to an embodiment of the present invention. The sensor array 1000 may include a plurality of lidar sensors 1002; this example shows sixteen lidar sensors 1002, but any number of lidar sensors 202 may be included. In this example, the lidar sensors 1002 are arranged in a staggered manner; however, this is not required, and in some embodiments, the lidar sensors 1002 may be arranged in a single column (in this example, the column is parallel to the Figure 10 The z-axis extension is shown on the left side of the figure.

[0132] In this example, each lidar sensor 1002 is associated with a "row" 1004 of the sensor array 1000. (The term "row" is used herein to indicate a linear or nearly linear arrangement of elements; Figure 10 (The rows in FIG are represented by dashed lines.) In addition to lidar sensor 1002, each row of sensor array 1000 also includes one or more ambient light sensors 1006. In this example, ambient light sensor 1006R detects red light, ambient light sensor 1006G detects green light, and ambient light sensor 1006B detects blue light; however, any number and combination of ambient light sensors may be used. Additional examples are described below. Each row may include a complete set of sensors used to generate a multispectral pixel. Sensor arrays such as sensor array 1000 are referred to herein as "row-based" or "1D" sensor arrays.

[0133] Other examples might have a complete set of sensors in a 2D array, rather than in a single row. This type of 2D array might occur when the lidar unit can move in more than one direction. Regardless of the sensor arrangement, different types of optical data can be captured at each location. Furthermore, some of the data might come from an entirely different sensor system that has been spatially calibrated with respect to the primary lidar sensor, so that measurements at a certain location can be mapped to specific pixels in the image. Consequently, data can be streamed into the image buffer at different times. Image processing can wait until the buffered image is fully reconstructed, with all data layers filling the buffer for all pixels in the image area, and then process that portion of the image.

[0134] lidar and ambient light sensors can be used with Figure 9 The color pixels can be assigned to the lidar image in a similar manner to the image in

[15] . However, the ambient light sensor can be assigned to the color image, and the lidar sensor can be assigned to the lidar image. Therefore, the color pixels can be associated with the lidar pixels based on whether the sensors are in the same row or using a mapping table.

[0135] D. Form a reproducible pixel grid in the lidar image

[0136] As described above, a sensor array can be used to generate a rectilinear grid of pixels, even when the array includes multiple columns and the sensor array is moved between positions to generate an image.

[0137] In some embodiments, a sensor array of the type described above can be used in an angular scanning or rotational mode such that different sensor channels in a row of the sensor array sequentially image (i.e., sense photons therefrom) a particular region in the field of view. For purposes of this description, it is assumed that during a scanning operation, the sensor system is rotated about an axis transverse to the rows, and that the sensor channels serving as the sensor system are rotated through different angles. (It will be understood that the scanning behavior described herein can also be achieved without moving the sensor array, for example, by using MEMS mirrors to reflect light from different regions of object space onto the array at different times.) It is also assumed that the sensor array and the bulk optical module are maintained in a fixed relationship to each other in the sensor system, such that a given sensor channel has a fixed spatial relationship to the optical axis of the bulk imaging optics and "sees" through the same portion of the bulk optical module regardless of how the system is oriented in space.

[0138] To simplify image analysis, it is often desirable that a scanning sensor system uniformly sample the object space. In this case, the grid of object space pixels is considered to be arranged in rows along the scan direction and in columns transverse to the scan direction. In the scan direction, it is desirable that different sensor channels in the same row (e.g., Figure 10 In one embodiment, all sensor channels in the same row 1004 of the sensor array 1002 sample the same object space pixels (at different times). As described below, this can be achieved in part by coordinating the sampling interval with the rotation of the sensor array. However, it is also important to avoid pointing errors caused by different positions of different sensor channels relative to the optical axis of the bulk optical module. Therefore, in some embodiments, the bulk optical module used with the sensor array in the scanning sensor system is designed to provide uniform sampling in both the scanning direction and the non-scanning direction.

[0139] Figure 11A and 11B is a simplified conceptual diagram illustrating the potential for pointing errors in a scanning system using a sensor array.

[0140] Figure 11A A row of a sensor array 1100 is shown having evenly spaced sensor channels 1102a-1102d, which may correspond to, for example Figure 101000. Each sensor channel has a channel field of view, as indicated by the dashed lines, through bulk optics 1110. The evenly spaced object space pixels represented by ellipses 1104a-1104d are aligned with the channel fields of view of sensor channels 1102a-1102d.

[0141] Figure 11B Sensor array 1100 is shown after being rotated so that sensor channel 1102a is pointing approximately toward object-space pixel 1104b. Sensor channel 1102b is pointing to the left of object-space pixel 1104c, and sensor channel 1102c is pointing approximately toward object-space pixel 1104d.

[0142] from Figure 11B As can be seen in the figure, pointing errors exist. For example, the field of view of sensor channel 1102b is not pointed at object-space pixel 1104c, while the field of view of sensor channel 1102c is not precisely aligned with object-space pixel 1104d. The term "intra-pixel pointing error" is used herein to refer to differences in field of view between sensor channels that nominally point to the same object-space pixel. (These differences are "intra-pixel" relative to the object-space pixel.) In some embodiments, it is desirable to control intra-pixel pointing errors when collecting multispectral pixel data.

[0143] In addition to intra-pixel pointing errors, sensor systems may also have "inter-pixel pointing errors," which refers to the uneven spacing of object-space pixels in either the row (scanning) or column (non-scanning) directions. In a scanning sensor system, uniform pixel spacing in the scanning direction can be achieved by controlling the shutter spacing relative to the sensor system's rotation angle (e.g., as described below) and by limiting the intra-pixel pointing errors, or by having independent shutter control for each pixel to completely eliminate inter-pixel errors. In the non-scanning direction, it is desirable that object-space pixels along a column are evenly spaced, with columns in object space mapped to columns in image space. In this regard, it should also be noted that some sensor arrays (e.g., sensor array 1000) may include a collection of interleaved sensors (e.g., lidar sensor 1002). In this case, a single column of object-space pixels can be imaged by scanning the array and controlling the shutter spacing to create column alignment. For example, in the case of sensor array 1000, one column of an image may have sixteen pixels, even though the sixteen sensor channels 1002 are not aligned within a column of sensor array 1000, as described above. In embodiments without such alignment or with some alignment errors, the lidar system can identify the nearest integer rectilinear pixel coordinates to store in an image buffer. In some instances, the pointing error can be quantified and saved and used for later correction, for example, by interpolation or regression using measurements (e.g., depth, color, etc.) and the pointing error from the rectilinear pixel coordinates.

[0144] In some embodiments, the desired imaging behavior is achieved by providing a bulk optical module with a focal distortion profile in which the displacement of light rays is linear with the tangent change of the light rays' angle of incidence (θ). A lens (or lens system) with such a focal distortion profile is often referred to as an "F tan θ" lens (indicating that the displacement distance in the image plane is a linear function of tan θ) or a "flat field" lens. For small angles θ, an F tan θ lens has the property that the displacement of light rays in the image plane (i.e., the sensor array) is approximately linear with the change of the light rays' angle of incidence (θ). In the scanning direction, this provides the desired behavior of reducing intra-pixel pointing errors. In the non-scanning direction, this provides uniform object space sampling and allows columns of object space pixels to be mapped to columns of image space pixels (even if the sensors are arranged in a staggered manner).

[0145] Figure 12An example of an imaging system using an F tan θ bulk optical module according to an embodiment of the present disclosure is shown. Image plane 1202 contains a row of sensors 1204a-g separated by a uniform distance p (also referred to as "linear pitch" in the literature). Sensors 1204a-g can be, for example, a row (or portion of a row) of sensor channels in any of the multispectral sensor arrays described above, or other sensors that detect photons from a given direction. Bulk optical module 1206 is positioned above image plane 1202 at a distance f, where f is the focal point of bulk optical module 1206. In this example, bulk optical module 1206 is shown as a single biconvex lens; however, it should be understood that other lenses or multi-lens systems can be used.

[0146] Bulk optical module 1206 can be designed to focus light from the field of view (or object space) onto image plane 1202. For example, rays 1220a-1220g indicate the chief rays of sensors 1204a-1204g. (It should be understood that the actual path of light through bulk optical module 1206 is not shown.)

[0147] Bulk optical module 1206 has a focal length distortion profile of F tanθ. (Those skilled in the art will understand how to create a bulk optical module with such a profile, and detailed description is omitted.) As a result, at least for small angles, a uniform change in the angle of incidence of a light ray causes the point at which the refracted ray intersects the image plane to shift by a uniform distance, regardless of the original angle of incidence. For example, for light rays 1220a and 1220b, the difference in angle of incidence is α, and light rays 1220a and 1220b are separated at the image plane by a linear spacing p. Light rays 1220b and 1220c also have a difference in angle of incidence a, and the corresponding refracted light rays 1220b and 1220c are also separated at the image plane by a linear spacing p. Thus, if image plane 1202 and bulk optical module 1206 are rotated together by angle α, ray 1220a emanating from point 1230a will (approximately) become the chief ray for sensor 1204b, and ray 1220b emanating from point 1230b will (approximately) become the chief ray for sensor 1204c, and so on.

[0148] The rotation angle α corresponding to the linear pitch p at the image plane is referred to herein as the "angular pitch" of the scanning system, and the value of α is determined by the sensor pitch p and the properties of the bulk optical module. In a scanning ranging / imaging system (where the bulk optical module provides the angular pitch α such that scanning the system at an angle α causes the incident light to be shifted by one linear pitch unit p), different sensor channels in a row can image the same portion of the field of view by acquiring images at a series of time steps, where the sensor array is rotated by the angular pitch α at each time step (or by a smaller angle such that α is an integer multiple of the scanning pitch).

[0149] E. Controller for generating rectilinear lidar frames

[0150] As mentioned above, rectilinear lidar frames can be generated even when multiple pixels share the same bulk optical element and when the light sensing module is rotated or scanned by other means. Therefore, an image of depth pixels can be generated from a histogram and assigned to pixels in a rectilinear array. As a result, the relative positions of lidar pixels can form a pattern to which image processing techniques can be applied. In contrast, lidar data is typically formed as independent point clouds that are not uniformly structured when stored in a frame buffer.

[0151] Figure 13 FIG. 1 shows a controller 1350 configured to identify signals in a histogram and form a linear array of depth values ​​to periodically generate lidar frames according to an embodiment of the present disclosure. The controller 1350 may be configured to be coupled to a light sensing module (e.g., Figure 2 230) on the same integrated circuit or on different integrated circuits. In other examples, one or more components may reside on the light sensing module. In yet another example, different components of the controller 1350 may be on different integrated circuits. The controller 1350 may execute the same Figure 2 The controller 1350 may function as an image reconstruction processor, as described below.

[0152] As mentioned above, the light sensing module (eg, Figure 2 230) can generate a histogram of photon counts detected across multiple emissions during a measurement interval. During a measurement, a sensor array (e.g., Figure 5 Each sensor in 510 (e.g., Figure 5 A histogram is generated by the controller 1350 (e.g., 515). Each sensor can be assigned a specific ID. After the current measurement is completed, the histogram can be sent to the controller 1350. The histogram can be sent serially (e.g., the histogram for sensor ID 1 is sent first, the histogram for sensor ID 2 is sent second, and so on) or in batches, such as all sent in parallel.

[0153] The histogram can be sent as part of a data structure that includes the sensor ID. In another example, the histogram can be sent in a specified order so that the position of the histogram in the data stream can indicate the sensor ID (for example, the histogram length N can be known so that the next time interval of data (N+1) can correspond to the next sensor ID). When reconstruction of a new image begins, the assignment of sensor IDs can then be returned to the beginning. Thus, the detection circuitry (for example, all or part of the light sensing module) can provide depth values ​​in a specified order, and the image reconstruction circuitry can assign a sensor ID to a particular depth value based on the specified order.

[0154] The histogram may be provided to a matched filter 1310, which may apply a filter to provide a filtered histogram, such as Figure 8 Various matched filters can be used to provide various filtered histograms, for example, to detect different shapes of the detected pulses. The filtered histograms can be provided to a peak detector 1320. In some examples, some or all of matched filter 1310 and peak detector 1320 can be part of a ranging circuit.

[0155] The peak detector 1320 can analyze the filtered output and determine the location corresponding to the peak (e.g., time interval or higher resolution). These range (depth) values ​​can then be used to form an image, for example, for the current scan of the lidar device. The first image is shown as being filled. The first image can reside in a buffer. Once the buffer is filled or at least a specified portion of the buffer (e.g., a subset of pixels corresponding to the kernel size of the coprocessor), the frame can be sent to one or more coprocessors for further analysis. The frame buffer can be a memory bank that operates as a FIFO. Therefore, the image reconstruction circuit can determine when the specified subset of the first lidar pixels has been stored for the first lidar image and send the specified subset of the first lidar pixels to the kernel-based coprocessor.

[0156] The peak detector 1320 can also measure signal and noise values, effectively providing some signal-to-noise ratio measurement for the lidar pixel. The signal value can correspond to the number of photon counts at the peak in the histogram, and the noise value can correspond to the background level in the time interval outside the peak area. In various embodiments, the amount of light at the operating wavelength (e.g., of the emitting light source) can be used to estimate the noise, or other wavelengths can be used. Thus, a lidar pixel can include a range (depth) value, a signal value, and a noise value. The signal value corresponds to the amount of light reflected from objects in the scene. The noise value corresponds to a measurement of the ambient light level in the scene at the same wavelength as the lidar system operates.

[0157] In other embodiments, the matched filter can measure the noise and provide some signal-to-noise ratio measurement to the peak detector, which uses this information to detect the range of the signal. In some cases, the peak detector can determine that the signal is not strong enough and therefore omit the lidar pixel from the image, for example by providing a NULL value to be stored in the image. In other cases, these values ​​are initially stored so that subsequent stages of the pipeline can determine whether to provide a specific depth value for a given pixel to the user.

[0158] The position (e.g., angular position) of the light transmitting module may also be received, where each position corresponds to a different measurement of the sensing array using the light sensing module. A scan (e.g., rotation) consists of a series of measurements over a set of positions. In various embodiments, the position may be received by the controller at the start of a new measurement, may be received from the light sensing module, may be received from a different module (e.g., an encoding module), or may be obtained using a histogram, as well as suitable combinations of these options.

[0159] For a given measurement, multiple pixel sensors can each provide a histogram. Along with the histogram data, the pixel sensor can provide a pixel ID that identifies a specific sensor on the sensor circuit. The pixel ID can be used to determine which pixel in the image is filled with a given depth value determined from the corresponding histogram. As described herein, assigning data to pixels can use a lookup table in which pixels are mapped into a linear frame buffer. Assigning data to pixels can be performed using image reconstruction circuitry 1325, which can be separate from or part of peak detector 1320 and can be part of the ranging circuitry. As described above, the detection circuitry can provide the sensor ID for each depth value to the image reconstruction circuitry.

[0160] The mapping table 1330 may specify which pixel IDs correspond to which pixels in the lidar image. The mapping table 1330 may identify corresponding pixels in the image to fill in a given depth value based on the sensor ID and the current position of the light sensing device (module). Once the light sensing device moves to a new position, the controller 1350 may provide new assignments for the next set of histograms to be provided. For example, the assignments may be loaded from the mapping table 1330 to a peak detector. The mapping may be in the same order as Figure 9 For example, each row of the mapping table may have a column for the sensor ID and another column specifying the lidar pixel, such as the 2D coordinate, which may be specified as X,Y.

[0161] As shown for the first scan, the matched filter 1310 receives histograms, each histogram corresponding to a particular pixel. A pixel can be identified using a pixel ID, for example, a number that specifies which sensor in the array is used to generate the histogram. In some embodiments, the controller can assign pixel IDs based on the order in which the histograms are received in the data stream from the light sensing device. For example, the controller 1350 can track when a new image begins, for example, based on the old image buffer being full or based on a measurement received from the light sensing device, such as a rotary encoder position. The first histogram will correspond to pixel ID #1. Since the light sensing device outputs histograms in a specified order, the correspondence of the next N-1 histograms can be known.

[0162] For an example where the sensor array has 64 sensors, the second angular position will have a sensor ID range of 65-128. Figure 9 For the given mapping, sensor ID 65 would be the first (top) pixel, but shifted one column to the right (or left if rotating counterclockwise), as shown. If the sensor array had four columns of 16 sensors, sensor ID 17 would correspond to the pixel in the second column, second row, because the sensor array is staggered. Various mappings can be used, depending on the shape of the sensor array.

[0163] In some embodiments, once the first image is complete (reconstructed), it can be output, as described in more detail below. Controller 1350 can determine when the first image is complete based on knowledge of the image buffer size and / or a signal from the image buffer indicating that it is full. In other embodiments, controller 1350 can receive a signal that the light sensing device has rotated, for example, based on measurements taken by a rotary encoder. Such measurements can occur at the light sensing device, at controller 1350, or at another component of the system. In other embodiments, the first image can be output partially. For example, when a portion is complete, that portion can be output while depth values ​​are not yet assigned to other portions of the first image. In such embodiments, certain processing techniques (e.g., local kernels) can begin processing the first image even if only a portion has been fully reconstructed in memory. This local processing can reduce memory requirements and latency. However, while the system waits for the entire image to be completed, processing techniques (e.g., certain machine learning models) can be used. In some instances, some initial local processing can be performed, and then full image processing can be performed after the initial local processing.

[0164] As shown, for the second rotation, a similar mapping is applied to the sensor IDs as for the first rotation. In some embodiments, two buffers can be used, where one buffer is full once an image is sent, and control immediately fills the other buffer with the next image, eliminating the need to complete the transfer of the first buffer before the next image can be generated.

[0165] Color images can be generated in a similar manner, except that the pixel values ​​can be taken directly as sensor values ​​instead of using a histogram of data values. Figure 10 For example, three color images can be generated, each with the same resolution as the lidar image. The lidar and color images can be analyzed together or separately.

[0166] The controller 1350 can pass the cached lidar image to a coprocessor (e.g., a lidar image processor or an AI coprocessor), which can analyze the image and can send back an improved lidar image (e.g., filtered) or characterization data, such as classifying certain lidar pixels as belonging to the same object. The coprocessor can be programmed to perform various functions, such as for filtering and semantic classification. The improved lidar image and feature data can be sent to the user. An example of using a lidar image processor or an AI coprocessor is described below.

[0167] Thus, the system may include an optical distance measuring device (e.g. Figure 2 210 in ), which may be a scanning device that moves and performs measurements in multiple positions, wherein the measurements for the scan (e.g., a 360° rotation) may be combined to create an image. The optical ranging device may include a transmitting circuit (e.g., Figure 2 240) and detection circuit (eg Figure 2 230), the transmitting circuit may include multiple light sources that transmit light pulses, and the detecting circuit may include a light sensor array that detects reflected light pulses and outputs a signal measured over time.

[0168] Signal processors (e.g. Figure 2The processor 238 and / or the matched filter 1310 and the peak detector 1320) can use the light sensor array to determine the depth value from the measurement. The image reconstruction circuit (e.g., 1325) can assign a sensor ID to each depth value for use in the scanning of the light ranging device and construct a lidar image using the depth value. For example, the sensor ID can be used to access a mapping table 1330 to map the depth value to a lidar pixel in the lidar image, where the mapping table specifies the lidar pixel based on the corresponding sensor ID. Then, after the lidar pixels of the lidar image are stored in the local image buffer, the lidar pixels of the partial frame (portion of the image) or the complete frame (the entire image) can be sent to a core-based coprocessor, which can be on the same or a different integrated circuit.

[0169] As described below, the core-based coprocessor may include a classifier circuit that is communicatively coupled to the optical ranging device and is configured to receive a lidar image output by the image reconstruction circuit, analyze depth values ​​in lidar pixels of the lidar image, correlate a group of lidar pixels based on corresponding depth values ​​of the group of lidar pixels, and output classification information for the group of lidar pixels based on the correlation.

[0170] In another example described below, a core-based coprocessor can include a depth imaging circuit communicatively coupled to the optical ranging device and configured to receive first lidar pixels and apply one or more filter kernels to a subset of the first lidar pixels to generate a filtered image of the lidar pixels.

[0171] V. Apply filter kernel

[0172] Once the lidar image is generated, the controller can send the image to an image signal processor. Because lidar images are rectilinear, various image processing techniques can be used. Image processing can use proximity criteria (e.g., similarity in spatial position, color, contrast, etc.) to identify a subset of pixels on which filtering is performed. Similarity can be defined as two values ​​being within a threshold difference of each other. When the processor operates on a collection of pixels, the image signal processor can be a core-based processor.

[0173] Figure 14 14 shows an optical ranging system 1400 including a controller 1450 and a lidar image processor 1410 according to an embodiment of the present disclosure. The controller 1450 may correspond to other controllers described herein. The various components of the optical ranging system 1400 may be on the same or different integrated circuits.

[0174] The controller 1450 converts the initial lidar image 1405 (eg, Figure 13 The images #1 and #1 in FIG are sent to the lidar image processor 1410, also known as the depth imaging circuit. As described above, the controller 1450 can buffer the histogram data flowing from the sensor to the determined depth value to create an initial lidar image as a frame of the lidar image stream. As the light sensing module scans different locations, the lidar image stream can be provided to the lidar image processor 1410. As shown in the figure, two lidar images are shown to illustrate the lidar image stream that can be provided. The image stream can be in the order generated, for example, in the order of location. In some embodiments, the location can be provided as part of the metadata provided with the lidar image.

[0175] Thus, the controller 1450 (or other ranging circuitry) can be coupled to the light sensor array and configured to determine depth values ​​from measurements using the light sensor array to form a lidar image comprising a grid of lidar pixels (e.g., a rectilinear frame of the environment during a measurement interval), and periodically output the lidar image. Each lidar image can be generated during a different measurement interval and comprise rows and columns of lidar pixels.

[0176] In the lidar image processor 1410, an input buffer 1412 can buffer the initial lidar image 1405 and then send one or more images at a time to a filter kernel 1414, which can include more than one filter kernel. Because the pixels of a frame have a known spatial relationship (at least horizontally, i.e., in a 2D image), embodiments can perform processing that exploits this known spatial relationship by applying a group function to a group of lidar pixels, such as a 5x5 subgroup (or other sized group), or all lidar images in a full frame. For example, statistics of depth values ​​for a group can be used, rather than just for individual pixels. These statistics can improve accuracy, fill in missing data, provide greater resolution, etc. Various kernels can be used, as described below. Multiple kernels can be applied, and the results can be combined.

[0177] The filter kernel 1414 can provide the filtered lidar image to a signal processor 1416, which can be optional or alternative to the functionality performed in the filter kernel 1414. The filtered lidar image can include new depth values, as well as updates to any other values ​​included in the lidar image, such as new signal and noise values. The signal processor 1416 can determine whether the depth value has sufficient accuracy to be reported to subsequent stages in the pipeline, or make a second pass determination based on the new value. In some embodiments, the signal processor 1416 can analyze multiple filtered images produced by different filter kernels applied to the same lidar image and use the set of filtered images to determine the new (updated) value of the lidar pixel. For example, filter kernels of grids of different sizes can be applied to the same lidar image.

[0178] Output buffer 1418 can receive and buffer processed lidar images for transmission back to controller 1450 or other devices as processed lidar images 1407. In some embodiments, signal processor 1416 can retrieve processed images from output buffer 1418 for further analysis, for example, when a processed image from a previous frame is needed to process a filtered image for the next frame. This caching can also be performed internally within signal processor 1416. One or more processed images can then be passed to the output buffer with a flag indicating that they are ready for output.

[0179] As described above, a color image can be obtained simultaneously with the lidar image. The initial color image 1425 can be sent to the color image processor 1430. The input buffer 1432 can buffer the initial color image 1425 and then send one or more images at a time to the filter kernel 1434. Since the pixels of the frame have a known spatial relationship (at least horizontally, that is, in a 2D image), embodiments can perform processing that utilizes this known spatial relationship by performing a group function on a group of color pixels, such as a 5x5 subgroup (or other sized group) or all lidar images in a complete frame. Various kernels can be used, as described below. Multiple kernels can be applied and the results can be combined.

[0180] The signal processor 1436 can analyze the filtered color image from the filter kernel 1434. The filtered color image can include new (updated) values ​​for the colors (e.g., red, green, and blue) of the defined color pixels. The signal processor 1436 can analyze multiple filtered images generated using various filter kernels on the same initial color image. The signal processor 1436 can include logic to use the values ​​of neighboring pixels and / or multiple filtered color values ​​of the same pixel to determine new color values ​​for the color image, for example, for the purpose of defining the edges of an object. The signal processor 1436 can analyze the set of filtered images for a series of frames (e.g., for different positions of the light sensing module).

[0181] In some embodiments, signal processor 1436 and signal processor 1416 can exchange information about the depth and color values ​​of corresponding pixels to perform combined processing that can interpret the corresponding values ​​of lidar and color pixels and adjacent pixels. Such combined analysis can also use an AI coprocessor. Bus 1460 can generally transfer information between lidar image processor 1410 and color image processor 1430, and specifically between signal processor 1416 and signal processor 1436. Lidar image processor 1410 and / or color image processor 1430 can be image processing circuits, either individually or collectively.

[0182] Output buffer 1438 can receive and buffer processed lidar images for transmission back to controller 1450 or other devices as processed color images 1427. In some embodiments, signal processor 1436 can retrieve processed images from output buffer 1438 for further analysis, for example, when a processed image from a previous frame is needed to process a filtered image for the next frame. This caching can also be performed internally within signal processor 1436. One or more processed images can then be passed to the output buffer with a flag indicating that they are ready for output.

[0183] The post-processor 1440 can analyze the processed lidar image 1407 and / or the processed color image 1427. The post-processor 1440 can perform any combination of analysis of the lidar image and the color image, as well as any classification from the AI ​​co-processor, for example, the AI ​​co-processor performs pixel classification corresponding to the same object, pixel-by-pixel semantic segmentation, instance segmentation, bounding box estimation, depth interpolation, etc.

[0184] A. Lidar filtering and processing

[0185] The filter kernel can be swept across the lidar frame. As an example, the application of the filter kernel can provide range smoothing of adjacent range pixels and / or a time series of range values ​​of the current lidar pixel, edge smoothing, or noise reduction (e.g., using statistics). As an example, range smoothing can provide a smoother road than would otherwise be obtained, which results in a more realistic depth image being provided to the user. Such filters can include bilateral filtering using a combination of range data, signal data, and other passive imaging channels (e.g., color). An edge-aware blur filter can smooth (blur) values ​​that are similar to each other in space and depth values ​​(or color values) so that edges are preserved, but the smoothing can be away from edges (e.g., an edge between two objects of different depths).

[0186] The filter kernel can determine a kernel weight or "sameness" for each pixel relative to a center pixel, thereby providing a filtered value for the center pixel. The filter kernel can weight the data values ​​of neighboring pixels based on their distance from the center pixel. Values ​​that are farther away will be less spatially correlated. The locations of neighboring pixels can be identified using a rectilinear array of image frames. One example is Gaussian weighting, e.g., a 2D Gaussian weighted by the lateral (XY) distance from the center pixel. Using a range value, Gaussian weighting can also be applied in the Z direction. The filtered value of the pixel can be determined based on the weighted sum of the kernels applied to the center pixel, and an accumulated (aggregated) value (e.g., range, signal, noise, or color) can be used to determine whether the value is to be retained (e.g., above a threshold with sufficient confidence) and passed to the user or the next stage in the pipeline.

[0187] As a result of applying the filter, the strength of the signal-to-noise ratio in the signal can be increased so that the depth value can be maintained. This processing can be performed by the signal processor 1416. For example, if there are two adjacent pixels in the XY plane and both pixels have peaks at similar distances (e.g., within a distance threshold or within a cumulative value exceeding a threshold), the peak can be identified as a true peak. In this way, peak detection can use a variable (adjusted) threshold based on the signal at the adjacent pixels. The threshold and / or the underlying data can be changed (adjusted). The signal processor can make such an identification. The adjustment of the peak or detection threshold can be based on the aggregated information of the subset.

[0188] Thus, in some embodiments, one or more filter kernels of filter kernels 1414 can include a filter kernel that uses signals from other lidar pixels adjacent to the given pixel to adjust the peak value or detection threshold of the signal of the given pixel. As a further example, one or more filter kernels of filter kernels 1414 can include a filter kernel that uses signals from other lidar pixels adjacent to the given pixel to adjust the depth value of the given pixel. In various embodiments, the criteria for considering pixels as adjacent pixels (e.g., in a particular subset) can use lateral position, depth value, and / or color value.

[0189] B. Color filtering and processing

[0190] One or more filter kernels can be swept across the color frame. As an example, application of the filter kernel can provide color smoothing (or blurring) of adjacent color pixels and / or a temporal sequence of color values ​​of the current color pixel, edge smoothing, or noise reduction (e.g., using statistics). As an example, color smoothing can provide a smoother or more uniform color for an object than would otherwise be obtained, resulting in a more realistic color image being provided to the user. The filter kernel can weight the data values ​​of adjacent pixels based on their distance to a center pixel in a manner similar to filter kernels used for lidar images.

[0191] Filtering and / or processing can correct color values ​​that have spurious noise. For example, in low light conditions where few photons are detected, noise can overwhelm the true signal. An object that might be red (e.g., a staircase sign) might appear to have some green pixels. On average, the pixels are red, but not all of them actually appear red because the signal is so low that some of them appear as green objects in the surroundings. However, filtering can accumulate values ​​from neighboring pixels to increase the red contribution.

[0192] This processing can identify surrounding pixels as red (e.g., most of the surrounding pixels) so that the system can overwrite (discard) pixels that are incorrectly green and use a spatially aware cumulative filter to identify a group of pixels as all red. Thus, the adjustment of color can use the color pixel in question or only use neighboring pixels, for example, to overwrite the measured color. Thus, adjusting a first color pixel can include determining a weighted average of color pixels associated with a subset of depth pixels. The weighted average may or may not include the first color pixel. The weights can be determined based on the differences of other depth values ​​from the first depth value.

[0193] C. Combining lidar with color filtering and processing

[0194] As described above, the lidar image processor 1410 and the color image processor 1430 can transmit information to each other, including lidar and color image values, so that combined processing can be performed. For example, in some embodiments, the color values ​​in any color image (e.g., initial, filtered, or processed) can be used to estimate noise, which can then be used to determine a depth value, the accuracy of the depth value, and / or whether to report a depth value in the final lidar image, e.g., as provided to a user. For example, when ambient light levels are low, measuring only the noise in the wavelength of the light source may lead to inaccuracies, especially when the background light is not uniform over time. The intensity of the color pixel can be used instead of or in addition to the noise measurement.

[0195] As another example, the environmental data can have a higher resolution than the lidar data, for example, the color pixels can have a higher resolution. This can occur when the color sensor is located on a different sensor, for example, the sensor may not rotate. PCT publication WO 2018 / 213338, which is incorporated by reference in its entirety, provides an example of such a fixed camera that is triggered by a rotating lidar sensor, where the color pixels and lidar pixels are aligned. The trend of color changes in the color pixels can be determined, for example, as an interpolation or function fit to any set of color values ​​at the color pixels. This function can be used to determine depth values ​​between measured lidar pixels, thereby upsampling the lidar data to a higher resolution.

[0196] Thus, when the detection circuitry includes a color sensor that detects ambient light to generate an array of color pixels, the depth imaging circuitry can correlate one or more depth pixels with each color pixel. The one or more filter kernels of filter kernel 1414 can include a filter kernel that adjusts the color value of a given color pixel using color pixels correlated with other lidar pixels adjacent to the given color pixel. The other lidar pixels can meet one or more proximity criteria, including a difference between a first depth value associated with the given color pixel and other depth values ​​of the other lidar pixels.

[0197] Therefore, the depth (ranging) value can be adjusted based on similar colors or spatial positions.

[0198] VI. Classification

[0199] In addition to updating depth and color values ​​based on lidar and color images, embodiments may also analyze these images using machine learning models and other artificial intelligence (AI) models to identify and classify objects. Such classification information can be used in a variety of ways. For example, an AI coprocessor can semantically annotate a series of images. Once an object is classified and assigned to a specific lidar and / or color pixel, such information can be used to update such images. When the processor operates on a collection of pixels, the AI ​​coprocessor (or classification circuit) can be a core-based processor.

[0200] Figure 15 1 shows a light ranging system 1500 including a controller 1550 and a lidar AI coprocessor 1510 according to an embodiment of the present disclosure. The controller 1550 may correspond to other controllers described herein. The various components of the light ranging system 1500 may be on the same or different integrated circuits.

[0201] The controller 1550 converts the initial lidar image 1505 (eg, Figure 13 The lidar AI coprocessor 1510 may include images #1 and #2 in the image data (see FIG15A ) and a corresponding image stream (see FIG15B ). As described above, the controller 1550 may buffer the histogram data flowing from the sensor to the determined depth value to create an initial lidar image as a frame of the lidar image stream. As the light sensing module scans different locations, the lidar image stream may be provided to the lidar AI processor 1510. As shown, two lidar images are shown to illustrate the lidar image stream that may be provided. The image stream may be in the order in which it was generated, such as in the order of location. In some embodiments, the location may be provided as part of the metadata provided with the lidar image.

[0202] In the lidar AI coprocessor 1510, an input buffer 1512 can buffer the initial lidar image 1505 and then send it one or more images at a time to the classifier 1514. Because the pixels of a frame have a known spatial relationship (at least laterally), embodiments can perform processing that exploits this known spatial relationship by analyzing a group of lidar pixels, such as a 5x5 subgroup (or other sized group), or all lidar images in a full frame. Properties of a group of depth values ​​for different pixels can be used, as opposed to a single pixel. For example, statistics of the depth values ​​of a group can be used to determine which pixels correspond to the same object, such as a group of pixels with depth values ​​within a threshold of each other. Corresponding to the same object is a way of correlating two pixels. This classification of groups can be used to improve the accuracy of the final lidar image depth values, fill in missing data, provide higher resolution, and more. Various classification models can be used, as described below. Multiple classification models can be applied, and the results can be combined.

[0203] The classifier 1514 can provide classification information (e.g., a lidar image with classifications of certain pixels identified as corresponding to the same object) to a signal processor 1516, which can be alternative or alternative to having functionality performed in the classifier 1514. Various models can be used for the classifier 1514, including convolutional neural networks, which can include a filter kernel (e.g., Figure 14 The convolution kernel is performed by 1414). The classified color image can assign each lidar pixel to an object (e.g., using an ID) so that all pixels corresponding to the same object can be determined based on the content of the classified lidar image. Therefore, the classification information can indicate which lidar pixels of the lidar image correspond to the same object.

[0204] The signal processor 1516 can determine whether the depth value has sufficient accuracy to be reported to a later stage in the pipeline based on neighboring pixels assigned to the same object. In some embodiments, the signal processor 1516 can analyze multiple classified images generated by different classifications of the same lidar image and use these classifications to determine the new (updated) value of the lidar pixel. Additionally or alternatively, the signal processor 1516 can use multiple classified images corresponding to different positions of the light sensing module. For example, classification using various models can be applied to the same lidar image, such as decision trees and neural networks, or different types of such models, or using different parameters for the same model, such as the number of nodes or hidden layers.

[0205] Output buffer 1518 can receive and buffer processed lidar images for transmission back to controller 1550 or other devices as processed lidar images 1507. In some embodiments, signal processor 1516 can retrieve processed images from output buffer 1518 for further analysis, for example, when a processed image from a previous frame is needed to process a filtered image from a subsequent frame. This caching can also be performed internally within signal processor 1516. One or more processed images can then be passed to the output buffer with a flag indicating that they are ready for output.

[0206] Thus, a classifier circuit (e.g., a lidar AI coprocessor 1510) can be communicatively coupled to the optical ranging device and configured to receive a lidar image output by the ranging circuit, analyze depth values ​​in lidar pixels of the lidar image, correlate a group of lidar pixels based on corresponding depth values ​​of the group of lidar pixels, and output classification information for the group of lidar pixels based on the correlation. As described herein, the group of correlated lidar pixels can include a first lidar pixel in a first lidar image that is correlated with a second lidar pixel in a second lidar image because they correspond to the same point on an object in the environment.

[0207] As described above, a color image can be acquired simultaneously with the lidar image. The initial color image 1525 can be sent to the color AI coprocessor 1530. The input buffer 1532 can buffer the initial color image 1525 and then send it one at a time or multiple images at a time to the classifier 1534. Because the color pixels of the frame have a known spatial relationship (at least horizontally), embodiments can perform processing that takes advantage of this known spatial relationship by performing a group function on a group of color pixels, such as a 5x5 subgroup (or other sized group) or all lidar images in a full frame, as described with respect to lidar images.

[0208] The signal processor 1536 can analyze the classified color image from the classifier 1534. The classified color image can assign each color pixel to an object (e.g., using an ID) so that all pixels corresponding to the same object can be determined by the content of the classified color image. The signal processor 1536 can analyze multiple classified images generated using various classification models on the same initial color image. The signal processor 1536 may include logic that uses classifications of adjacent pixels and / or multiple classifications of the same pixel to determine a new color value for the color image, for example, for the purpose of defining the edges of an object. The signal processor 1536 can analyze the set of classified images for a series of frames (e.g., for different positions of the light sensing module).

[0209] In some embodiments, classifiers 1514 and 1534 (and / or signal processors 1516 and 1536) can exchange information about the depth values ​​and color values ​​of corresponding pixels (and their classifications) in order to perform combined processing that can account for the corresponding values ​​of lidar and color pixels and adjacent pixels when classifying an image or determining new values. For example, similarity in color and / or depth (e.g., within corresponding thresholds) can be used as a criterion for classifying pixels as corresponding to the same object. This combined analysis can also use lidar and color image processors. Bus 1560 can generally transfer information between lidar AI coprocessor 1510 and color AI coprocessor 1530, and specifically between signal processor 1516 and signal processor 1536.

[0210] Output buffer 1538 can receive and buffer processed lidar images for transmission back to controller 1550 or other devices as processed color images 1527. In some embodiments, signal processor 1536 can retrieve processed images from output buffer 1538 for further analysis, for example, when a processed image from a previous frame is needed to process a filtered image for the next frame. This caching can also be performed internally within signal processor 1536. One or more processed images can then be passed to the output buffer with a flag indicating that they are ready for output.

[0211] The post-processor 1540 may analyze the processed lidar image 1507 and / or the processed color image 1527. The post-processor 1540 may perform any combination of analyses on the classified lidar image and the color image.

[0212] The pixel classification of the object can be output to the user on a display. For example, a bounding box can illustrate the lidar pixels that are identified as corresponding to the same object.

[0213] A. Pre-identified obstructions

[0214] Classifiers 1514 and 1534 can use knowledge of the types of objects a particular lidar system is likely to encounter, such as objects mounted on a passenger car traveling on a standard road. For example, for a system mounted on a passenger car, a road will always be present, where the road can be constrained to the lower half of the environment and have a limited width. As another example, the size of adjacent vehicles used for classification can be constrained based on the initially perceived height in order to distinguish between adjacent vehicles while still allowing detection of a large semi-truck with a trailer, which is much taller.

[0215] In some embodiments, a user can define an image region for a particular object, such as a road. For example, a lidar device can be mounted on the bumper of a vehicle, and the vehicle manufacturer can specify a specific portion of the environment that may correspond to a road based on knowledge of the digital altitude. Such user input can be provided by drawing on the user interface, or as a numerical input of points in a rectangle or other shape to which the object is constrained.

[0216] Therefore, the detection criteria for a lidar pixel can vary depending on the location of that pixel in the lidar image. Lidar pixels located in front of the vehicle and in the lower half of the image may correspond to the road. Such pixels may be detected by a sensor whose field of view is tilted downward, for example, by a specified spacing. Therefore, a less stringent threshold for signal to noise can be allowed, allowing weak detections to still provide a depth value for a particular pixel. In contrast, lidar pixels in different parts of the image may have more stringent requirements for peak detection and depth value determination in order to determine the depth value to be output with sufficient accuracy.

[0217] B. Lidar classification and processing

[0218] The classifier 1514 can use depth values ​​from one or more lidar images to determine which pixels correspond to the same object. Using depth values ​​can improve the accuracy of the classifier because the depth values ​​of adjacent points on the same object do not change drastically from one pixel to adjacent pixels (e.g., adjacent locations on a rectilinear grid). For example, using the depth values ​​of adjacent lidar pixels can increase the confidence that two pixels are from the same object compared to using only color values.

[0219] Once a group of lidar pixels are identified as corresponding to the same object, the depth values ​​can be smoothed only for that group of pixels, for example, as performed by the lidar image processor 1410, which may receive classification information from the lidar AI coprocessor 1510. Similarly, the color image processor 1430 can make all color pixels of the same object have similar colors, or simply blend the colors of the pixels identified as corresponding to the object.

[0220] Classification can also inform the determination of depth values, or at least the confidence level of the depth values, which can be used to determine whether the depth value is included in the final lidar image. If a lidar pixel is identified as being from the same object as neighboring pixels and if the depth values ​​are similar, the depth value of the pixel can be included, even if the photon count in the histogram is relatively low and therefore the initial confidence level is low. The increased confidence level from the classification is sufficient to determine that the depth value is accurate, especially after applying a filter kernel, for example, to smooth the depth values ​​of the object. The use of classification (e.g., semantic labeling) can be performed as a second pass using a filter kernel that accumulates signal values ​​according to weights defined by a kernel function (e.g., Gaussian) as described herein. This second pass of accumulation (e.g., a first pass using only the filter kernel 1414 and signal processor 1416) can use depth values ​​from more pixels than those used based solely on lateral proximity, such as can be performed in the first pass. Thus, accumulation can be performed using both spatial and semantic (classification) information.

[0221] The classification model may consider predetermined types of objects, such as vehicles, which have multiple components, such as a windshield, headlights, doors, hood, etc. Each component may have an expected relationship between adjacent points, which may inform the kernel function used in the filtering step or classification information based on expected depth and / or color relationships (e.g., the windshield is a different color than the hood, but is part of the same vehicle object).

[0222] In some embodiments, the type of object can be used to determine the type of filter kernel used, for example, by filter kernel 1414. For example, certain types of objects may have a smoothing kernel applied to them, while other types of objects may not. For example, a group of pixels classified as corresponding to a road may be smoothed by a smoothing filter. However, a group of pixels identified as corresponding to a tree or other rough surface would not have a smoothing filter applied to it. As another example, a stop sign may be classified based on a 3D position (e.g., height above the road) and shape (octagon), as can be determined from depth values ​​in a lidar image. The range values ​​of the relevant pixels may be adjusted based on the fact that the object type is a plane. Not only which filter kernel is applied, but also the specific values ​​of the parameters used, for example, the shape and height of the Gaussian kernel, may depend on the type of object.

[0223] A type of object may have a specific expected size, which can be used by the detection technique to determine whether there is enough confidence to include the depth value in the final lidar image. For example, a road may occupy a limited portion of the environment. A low confidence level may be allowed for depth values ​​that include pixels corresponding to the expected location of the road, which can increase the distance range of the final lidar image without compromising accuracy. Therefore, if a lidar pixel corresponds to a predefined location of a specific object, more relaxed criteria (e.g., peak and / or noise thresholds) may be used, allowing the pixel to be included in the final lidar image more aggressively.

[0224] C. Color classification and processing

[0225] Upsampling of lidar data based on higher resolution color data is described above. Additionally or alternatively, lidar data can be upsampled based on classification data. For example, a particular type of object may have known parameters of its shape, or at least estimated parameters of its shape. For example, the type of vehicle can be determined based on the color and / or lidar data, for example, based on the general outline of the object. The shape of the vehicle type can then be identified, for example, from a database. The shape can have small, specific features, such as fins or lines on a door. The lidar data can be upsampled to display these features in the final lidar image (e.g., in a three-dimensional (3D) view). In addition, any missing lidar data (e.g., some lidar pixels may be missing due to noise or low signal) can be filled in based on the estimated shape of the object.

[0226] The shape of an object (e.g., determined based on the classification of a group of color pixels as belonging to the same object) can also be used to identify anomalies in lidar data, for example, due to false lenses, such as may occur during rain. The general distance of a vehicle can be estimated based on the number of color pixels assigned to the vehicle. A small number of color pixels assigned to a vehicle would indicate that the vehicle is relatively far away, for example, more than 50m). However, if the distance values ​​of the corresponding lidar pixels (i.e., lidar pixels at the same location in the image) indicate that the vehicle is relatively close, for example, within 10m, an anomaly can be identified in the color data or lidar data. If the condition is rainy (e.g., determined by weather information or a humidity sensor), the lidar data can be identified as an anomaly. In this case, the corresponding lidar data may not be registered to avoid displaying invalid data to the user, which may cause the driver to react suddenly and possibly cause an accident.

[0227] D. Combined filtering and classification

[0228] As mentioned above, Figure 15 The classification and Figure 14The filtering / processing in

[15] can be combined. For example, the classification information can be used to fill in missing lidar pixels based on other lidar pixels of the same object.

[0229] Figure 16 1 shows a light ranging system 1600 including a controller 1650, a lidar image processor 1610, and a lidar AI coprocessor 1630 according to an embodiment of the present disclosure. The controller 1650 may correspond to other controllers described herein. The various components of the light ranging system 1600 may be on the same or different integrated circuits.

[0230] The optical ranging system 1600 can operate in a similar manner to the optical ranging systems 1400 and 1500. The controller 1650 can send an initial lidar image 1605 to a lidar image processor 1610, which can operate in a similar manner to the lidar image processor 1410. The controller 1650 can also send an initial lidar image 1625 to a lidar AI coprocessor 1630, which can operate in a similar manner to the lidar AI coprocessor 1510. The initial lidar image 1605 can be the same as the initial lidar image 1625.

[0231] The bus 1670 can generally transfer information between the lidar image processor 1610 and the lidar AI coprocessor 1630, including between the various components within these processors. The communication of intermediate or final results from one processor to another can be used to complete the processing to output a processed lidar image 1607 or classification information 1627, which may include a classified image in which groups of pixels corresponding to the same object are identified. The use of processed lidar images to inform classification or the use of classified images to inform lidar image processing (e.g., updating or adding depth values) has been described above.

[0232] A similar configuration can be used for color processing. For example, the color image processor and the color AI coprocessor can communicate with each other and also with the lidar image processor 1610 and the lidar AI coprocessor 1630. The color processor 1660 can embody this configuration.

[0233] The post-processor 1640 may analyze the processed lidar image 1607 and / or the classification information 1627. The post-processor 1640 may perform any combination of analyses on the classified lidar image and the color image.

[0234] Thus, the system with the classifier circuit may further include a depth imaging circuit communicatively coupled to the optical ranging device and the classifier circuit, the depth imaging circuit configured to receive a lidar image of lidar pixels, receive classification information about groups of lidar pixels corresponding to the same object, and apply one or more filter kernels to a subset of the lidar pixels of the lidar frame based on the classification information.

[0235] VII. Keypoints (Image Registration)

[0236] In some embodiments, classification (e.g., by classifier 1514) can include identifying keypoints in an image and identifying that certain keypoints in one frame correspond to certain keypoints in other frames of a series of lidar images. Keypoints can be identified by analyzing the magnitude and direction of changes in depth values ​​(and / or color values) in a local image neighborhood to detect high contrast corners and edges where depth / color values ​​show significant changes.

[0237] Keypoints can be used in mapping or odometry problems to determine how a vehicle (e.g., a car or robot) moves in 3D space. Using two images from lidar, the system can determine how the vehicle changes position relative to a keypoint that may be stationary. By associating two keypoints as corresponding to the same point on an object in both images, image registration can be performed in 2D or 3D. Keypoints can have unique signatures, such as the edge of a building. The edge of a building can be identified by the change (or rate) of depth values ​​in a new direction before and after the edge, which can be defined by a series of points that exhibit similar changes.

[0238] Thus, the correlation of two keypoints to the same point in an object can use the difference in depth values ​​between adjacent depth (lidar) pixels. Similar differences (e.g., within a threshold) with one or more adjacent pixels in each of the two images can indicate that the two keypoints are for the same point on the same object.

[0239] As another example, peaks at specific time intervals can be used. For example, if a specific point on an object has a unique difference in depth value relative to its neighboring points, a unique peak pattern may also exist. For a given time interval, the signal at that point will be higher than the signal at neighboring pixels, where the difference in signal value will depend on the depth difference between the two points. Similar differences in signal value for each neighboring pixel can indicate that the keypoint is the same in each image.

[0240] Once two keypoints are associated between two images to identify them as the same keypoint, their 3D positions (depth combined with lateral position in the grid) can be subtracted to provide the change in the relative 3D position of the vehicle and the keypoint. If the keypoint is stationary, the subtracted value can be used to determine velocity when combined with the time interval between the two measurements. Such calculations can be performed for many keypoints between images to obtain the average motion of the lidar system mounted on the vehicle.

[0241] Thus, the lidar AI coprocessor (e.g., 1510 or 1610) can analyze multiple lidar images to identify relevant key points between lidar image frames. The lidar AI coprocessor can identify key points in one frame, cache lidar pixels corresponding to the key points (this can be done by storing the entire lidar frame or only certain pixels, including depth values ​​and pixel positions), determine key points in subsequent frames, and then derive the correlation between the two sets of key points. Thus, a second lidar image can be acquired after the first lidar image, wherein the first lidar image is cached while the second lidar image is acquired.

[0242] The classification information 1627 may include correlations between keypoints. For example, bundles of lidar pixels in separate frames (e.g., consecutive frames) may be bundled into keypoints, where the post-processor 1640 may use these bundled keypoints to determine an average velocity. Other subsequent components in the pipeline may also perform calculations to determine the average velocity, e.g., components of a separate computer, such as Figure 2 The speed values ​​over time can be used together to determine the odometry. Differences between consecutive frames and further separated frames (e.g., the first and third frames) can be used to determine the speed. Multiple differences between lidar pixels from two additional pairs of frames can be used to achieve higher accuracy.

[0243] Keypoints can also be used for visualization; for example, specific colors can be used to depict lidar pixels corresponding to keypoints. Arrows showing relative velocity can also be displayed. Such depictions can convey the relative motion of a vehicle compared to stationary and other moving objects.

[0244] In some embodiments, the classification of keypoints can be combined with the semantic classification of groups of pixels as corresponding to individual objects. This different type of classification can be performed by a separate classification engine of the classifier 1514.

[0245] As an example, keypoints can be assigned to a moving object based on pixels that are grouped together as being part of the same moving object and are keypoints. The keypoints assigned to the moving object can be used to determine the velocity of the moving object, where such velocity can be relative to the lidar system or absolute. Such absolute velocity can be determined by first determining the velocity of the lidar system relative to one or more stationary objects. The change in the keypoints of the moving object can then be used to determine the relative velocity compared to the lidar system. The velocity of the lidar system and the relative velocity of the moving object can be subtracted to determine the absolute velocity of the moving object relative to the stationary object. All pixels of the moving object can be assigned the same average velocity, which is determined by the keypoints assigned to the moving object.

[0246] Various techniques can be used to associate keypoints. The pixel ID of a keypoint (e.g., its location in a 2D grid of a lidar image) can be a starting point for identifying the same location on an object in subsequent frames. Similar depth values ​​relative to surrounding lidar pixels can indicate that the same edge or corner is reflected in the new pixel ID of the next lidar image. These two pixel IDs from the two frames can be grouped as keypoints. Machine learning techniques such as neural networks can be used. Correlation can also use color data, so the characteristic change in color from a color pixel to the surrounding color pixels can be used in conjunction with the depth of the corresponding lidar pixel to the surrounding lidar pixels. For example, two pixels can be identified as red (color proximity), close together in space, and have similar contrast metrics relative to surrounding pixels.

[0247] Before attempting correlation, it's possible to first track lidar pixels using certain criteria, such as ensuring there's sufficient depth variation between surrounding pixels. It may also be necessary to have a high degree of confidence in the depth value of a pixel. For example, a road may not have enough texture to qualify as a keypoint.

[0248] Thus, a coprocessor (e.g., a classification circuit) may receive a first frame of depth pixels from a detection circuit and identify a first set of keypoints in the first frame of depth pixels using depth value differences between adjacent depth pixels and / or peak differences between adjacent depth pixels. The first set of keypoints, including the differences used to identify the keypoints and a three-dimensional position formed by a lateral (2D) position of the depth pixels and a corresponding depth value, may be stored in a first buffer.

[0249] The coprocessor can receive a second frame of depth pixels from the detection circuit and identify a second set of key points in the second frame of depth pixels using depth value differences between adjacent depth pixels and / or peak differences between adjacent depth pixels. The differences and three-dimensional positions can then be used to associate one or more of the first set of key points with one or more of the second set of key points. For example, if the images are continuous, the depth values ​​and lateral positions should be within a threshold. The velocity of the key point can be calculated using the differences between the associated key point pairs. The velocity can be relative to the velocity of the lidar device. The velocity can be output in a variety of ways, for example, by identifying an object with a velocity above a threshold or by providing a number in the image above the object.

[0250] VIII. Methods

[0251] Various techniques are described above, for example, for reconstructing lidar images and image analysis (eg, adjusting values ​​associated with lidar images and / or color images). Such techniques can use the above-described systems.

[0252] A. Reconstructing lidar images

[0253] Reconstruction of the image can facilitate downstream analysis, such as classification or image analysis using kernels applied to the values ​​associated with the pixels.

[0254] Figure 17 17 is a flow chart illustrating a method 1700 for performing ranging using an optical ranging system installed on a mobile device according to an embodiment of the present disclosure. The mobile device may be of various types, such as a car, agricultural equipment, construction equipment, etc.

[0255] At block 1710, a transmission circuit transmits pulses from one or more light sources of a ranging system. The pulses may be reflected from one or more objects. The transmission circuit may include various circuits described above, such as a laser.

[0256] At block 1720, a detection circuit measures the signal by detecting photons of the pulse for each photosensor array. The detection circuit may include various circuits described above, for example, each photosensor is a set of SPADs.

[0257] At block 1730, a sensor ID is assigned to each signal. Each sensor ID corresponds to one of the light sensor arrays. The sensor IDs may be assigned at various points in the pipeline in a variety of ways. For example, the sensor ID may be received from the detection circuit via a signal, and the sensor ID may be assigned at that time. In another example, the sensor IDs are assigned based on a specified order in which the detection circuit provides the signals, as described above. In yet another example, the sensor ID is assigned to each signal by assigning the sensor ID to the first depth value. This assignment may be performed by the image reconstruction circuit (e.g., Figure 13 Thus, the sensor IDs may be assigned based on the specified order in which the first depth values ​​are provided to the image reconstruction circuit.

[0258] At block 1740, the signal is analyzed to determine a first depth value.The signal may be analyzed as described herein.

[0259] At block 1750, a first lidar image is constructed using the first depth value. The lidar image may be constructed using a mapping table as described above. The sensor ID may be used to map the first depth value to a first lidar pixel in a first lidar image (e.g., a rectilinear image). The mapping table may specify lidar pixels based on the corresponding sensor ID. For example, the mapping table may specify lidar pixels based on the corresponding sensor ID and the position of the optical ranging system when measuring the signal. The position may be an angular position of the optical ranging system. The sensor ID and position may specify a unique pixel within an image, for example, as described above in Figure 9 Described in .

[0260] At block 1760, an image buffer of the optical ranging system may store a first lidar pixel of the first lidar image. The image buffer may be considered local when it resides on the same integrated circuit as the detection circuit.

[0261] At block 1770, the first lidar pixel of a partial frame of the first lidar image or a complete frame of the first lidar image is sent to a kernel-based coprocessor of the optical ranging system. As described above, only a portion of the brighter image (the partial frame) may need to have values ​​before being sent, for example, because a kernel can be applied to the partial frame. Thus, a filter kernel can be applied to a portion of the first lidar image, wherein the filter kernel is applied before the first lidar image is fully constructed.

[0262] Other lidar images can be constructed and analyzed in a similar manner. For example, a second lidar image can be constructed using the second depth value. The second lidar image can also be stored in an image buffer, for example, in a different memory bank than the first letter image. One or more values ​​of the first lidar image can be adjusted based on the analysis of the first lidar image and the second lidar image to obtain one or more adjusted values. Examples of such adjustments are described in Figure 14 and 15 The one or more adjustment values ​​may include color values ​​of the first lidar image, for example, where a color pixel is associated with a lidar pixel.

[0263] B. Adjust the peak value or detection threshold

[0264] Various values ​​associated with lidar pixels and lidar images can be adjusted based on the values ​​of other pixels and proximity to the lidar pixel. For example, such values ​​can include peak values ​​or detection thresholds. The adjustment values ​​can be based on a filter kernel, for example, as described in Section V.

[0265] Figure 18 18 is a flow chart illustrating a method 1800 for performing ranging using an optical ranging system installed on a mobile device according to an embodiment of the present disclosure.

[0266] At block 1810, a transmission circuit transmits pulses from one or more light sources of a ranging system. The pulses may be reflected from one or more objects. The transmission circuit may include various circuits described above, such as a laser.

[0267] At block 1820, the pulsed photons are detected by one or more light sensors of the optical ranging system, thereby generating data values ​​for each of the depth (lidar) pixel array at a plurality of time points. The one or more light sensors may include a light sensor array. Examples of light sensors are provided herein, for example, each light sensor is a group of SPADs. The data values ​​at the plurality of time points may form a histogram for each depth pixel of the depth pixel array. The counter of the histogram at a particular time interval may correspond to one or more data values ​​at one or more time points within the particular time interval, as described above.

[0268] A first light source of the one or more light sources and a first light sensor of the one or more light sensors may be movable to provide data values ​​for at least two depth pixels of the depth pixel array. Figure 9 An example of this operation is provided in .

[0269] At block 1830, a subset of depth pixels is identified that meets one or more proximity criteria relative to a first depth pixel of the depth pixel array. The one or more proximity criteria may include a lateral distance between the first depth pixel and another depth pixel. The one or more proximity criteria may further include a difference between a first preliminary depth value of the first depth pixel and another preliminary depth value of the another depth pixel.

[0270] At block 1840, information from the data values ​​of the subset of depth pixels is aggregated to obtain aggregate information. As an example, the information from the data values ​​may include the data values ​​or a numerical value derived from the data values. Aggregation may include determining a sum (e.g., a weighted sum) of the data values ​​from corresponding time intervals. As described above, the aggregate information may be used to determine whether the signal of a given pixel is strong enough for the value to be used in analysis at a later stage.

[0271] At block 1850 , a first peak value and / or a detection threshold of a data value of a first depth pixel is adjusted using the aggregation information.

[0272] The first peak value is compared to a detection threshold at block 1860. When the first peak value exceeds the detection threshold, the reflected pulse can be identified as coming from an object in the environment.

[0273] At block 1870, based on the first peak exceeding a detection threshold, it is determined that the first peak corresponds to an object in the environment surrounding the mobile device. The detection threshold can be a fixed value or a dynamic value.

[0274] At block 1880, a first depth value for a first depth pixel is determined based on a first time associated with the first peak. Such a depth value may be determined as described herein.

[0275] Reflected pulses can be detected using peak values ​​and detection thresholds, which can be adjusted in a variety of ways. The background level of photons can be determined in a subset of depth pixels, and the peak value of data values ​​can be determined in a subset of depth pixels. The first peak value can be adjusted using the peak value, and the detection threshold can be adjusted using the background level.

[0276] In some examples, a color sensor can be used to measure ambient light to generate an array of color pixels. One or more color pixels can be associated with each depth pixel, for example, as described herein. In such examples, one or more proximity criteria can include a color similarity of one or more color pixels of a first depth pixel relative to another depth pixel.

[0277] C. Adjust the depth value

[0278] In addition to the peak or detection threshold, the depth value can also be adjusted. The adjustment value can be based on the filter kernel, for example, as described in Section V or Section VI.

[0279] Figure 19 is a flow chart illustrating a method 1900 of performing ranging using an optical ranging system installed on a mobile device.

[0280] At block 1910, a transmission circuit transmits pulses from one or more light sources of the ranging system. The pulses may be reflected from one or more objects. The transmission circuit may include various circuits described above, such as a laser. Block 1910 may be performed in a manner similar to block 1810 and other detection disclosures herein.

[0281] At block 1920, the pulsed photons may be detected by one or more light sensors of the optical ranging system, thereby generating data values ​​for each of the depth pixel array at multiple points in time. Block 1920 may be performed in a manner similar to block 1820 and other detection disclosures herein.

[0282] At block 1930, a first peak of data values ​​is determined for the first depth pixel. The peak value can be determined in various ways, such as described herein. For example, a matched filter can be used to detect a particular pulse pattern, and the signal at the matching time (e.g., as defined in a histogram) can be used to determine the first peak value.

[0283] At block 1940, a first depth value is determined for the first depth pixel based on the first time associated with the first peak. The first depth value can be determined in various ways, for example, as described herein. For example, an interpolation method can be used to determine the time with a greater resolution than the width of the time interval.

[0284] At block 1950, a subset of depth pixels that meet one or more proximity criteria relative to a first depth pixel of the depth pixel array is identified. As described herein, various proximity criteria can be used. In various embodiments, such adjacent depth pixels can be used to refine the value associated with the depth pixel or discard such measurements to account for efficiency and later stages of the pipeline.

[0285] At block 1960, respective peak values ​​of data values ​​are determined for the subset of depth pixels. The respective peak values ​​may be determined in a similar manner as for the first depth value.

[0286] At block 1970, other depth values ​​for the subset of depth pixels are determined based on the times associated with the corresponding peaks.The other depth values ​​may be determined in a similar manner as the first depth value.

[0287] At block 1980, the first depth value is adjusted based on the other depth values. For example, the adjusted first depth value can be determined as an average of the first value of the other depth values. Other examples may include classification of pixels corresponding to the same object and use of other kernel-based techniques.

[0288] Additionally, classification can be performed. Thus, method 1900 can include classifying a portion of the depth pixel array as corresponding to the same object. Classification can be based on a variety of factors, such as similar color values ​​or depth values ​​of similar adjacent lidar pixels (e.g., within a threshold in a local neighborhood). One or more proximity criteria can include a subset of depth pixels being classified as corresponding to the same object.

[0289] A set of depth pixel arrays can be identified to classify a predetermined object. For example, a specific portion (e.g., the lower portion) of the lidar image can be retained for detecting the road surface.

[0290] For example, to classify using color, a color sensor can be used to measure ambient light to generate an array of color pixels. One or more color pixels can be associated with each depth pixel. The one or more color pixels can be used to classify depth pixels as corresponding to the same object, for example based on color similarity, which can be measured as distance using colorimetry, as will be understood by those skilled in the art.

[0291] The color pixel array may have a higher resolution than the depth pixel array. In such instances, the method may further include generating additional depth values ​​for virtual depth pixels located between the depth pixels of the array, thereby increasing the ranging resolution. The additional depth values ​​may be generated using the color pixels. Further details of this upsampling are provided above.

[0292] D. Adjust color pixels

[0293] In addition to depth values, color values ​​may also be adjusted. The adjustment values ​​may be based on a filter kernel, for example, as described in Section V or Section VI.

[0294] Figure 20 is a flowchart illustrating a method 2000 for correcting a color image according to an embodiment of the present disclosure.

[0295] At block 2010, a transmission circuit transmits a pulse from one or more light sources of a ranging system. The pulse may be reflected from one or more objects. Block 2010 may be performed in a manner similar to block 1810 and other detection disclosures herein.

[0296] At block 2020, the pulsed photons may be detected by one or more light sensors of the optical ranging system, thereby generating data values ​​for each of the depth pixel array at multiple points in time. Block 2020 may be performed in a manner similar to block 1820 and other detection disclosures herein.

[0297] At block 2030, the ambient light is measured using a color sensor to generate a color pixel array. The color sensor can be on the same integrated circuit as the light sensor or on a different integrated circuit. For example, the color sensor can be in a separate camera or in the same sensor array, e.g., Figure 10 shown.

[0298] At block 2040, one or more depth pixels are associated with each color pixel. The depth pixels may be associated as described above, for example, as for Figure 9 or as described in 10.

[0299] At block 2050 , for a first color pixel, a first peak value of data values ​​is determined for a first depth pixel associated with the first color pixel. The determination of the peak value may be performed as described herein, for example, as described at block 1930 .

[0300] At block 2060 , a first depth value for the first depth pixel is determined based on the first time associated with the first peak. The first depth value may be determined as described herein, for example, as described in block 1940 .

[0301] At block 2070, corresponding peak values ​​of data values ​​are determined for the other depth pixels. The respective peak values ​​may be determined in a similar manner as for the first depth value.

[0302] At block 2080, other depth values ​​for other depth pixels are determined based on the time associated with the corresponding peak value.The individual peak values ​​may be determined in a similar manner as the first depth value.

[0303] At block 2090, a subset of depth pixels is identified that satisfies one or more proximity criteria relative to the first depth pixel of the depth pixel array. Examples of one or more proximity criteria are provided above, e.g., with respect to methods 1800 and 1900 and in Sections V and VI. The one or more proximity criteria may include a difference between the first depth value and one of the other depth values. As another example, the proximity criterion may be a classification of the subset of depth pixels as being part of the same object.

[0304] At block 2095, a first color pixel is adjusted using color pixels associated with the subset of depth pixels. Adjusting the first color pixel may include determining a weighted average of the color pixels associated with the subset of depth pixels. The weighted average may or may not include the first color pixel. The weights may be determined based on differences between other depth values ​​and the first depth value.

[0305] In some instances, adjusting the first color pixel may include discarding the first color pixel based on a difference between the first color pixel and color pixels associated with the subset of depth pixels. For example, if other colors are significantly different, the first color pixel may be discarded as an error. In this case, adjusting the first color pixel may include determining a weighted average of the color pixels associated with the subset of depth pixels, where the weighted average does not include the first color.

[0306] IX. Computer Systems

[0307] Any computer system mentioned herein (e.g. Figure 2 The user interface 215 or vehicle control unit 217) or components of the lidar system may utilize any suitable number of subsystems. Figure 21, an example of such a subsystem is shown in computer system 10. In some embodiments, a computer system includes a single computer device, wherein a subsystem may be a component of the computer device. In other embodiments, a computer system may include multiple computer devices with internal components, each of which is a subsystem. Computer systems may include desktop and laptop computers, tablet computers, mobile phones, and other mobile devices.

[0308] Figure 21 The subsystems shown in FIG. 7 are interconnected by a system bus 75. Additional subsystems are shown, such as a printer 74, a keyboard 78, one or more storage devices 79, a monitor 76 coupled to a display adapter 82 (e.g., a display screen such as an LED), etc. Peripheral devices and input / output (I / O) devices coupled to the I / O controller 71 can be connected through any number of, for example, input / output (I / O) ports 77 (e.g., USB, ) or other devices known in the art are connected to the computer system. For example, the I / O port 77 or the external interface 81 (such as Ethernet, Wi-Fi, etc.) can be used to connect the computer system 10 to a wide area network (such as the Internet), a mouse input device, or a scanner. The interconnection through the system bus 75 allows the central processing unit 73 to communicate with each subsystem and control the execution of multiple instructions from the system memory 72 or one or more storage devices 79 (for example, a fixed disk, such as a hard drive or optical disk), as well as the exchange of information between subsystems. The system memory 72 and / or the one or more storage devices 79 can be embodied as a computer-readable medium. Another subsystem is a data collection device 85, such as a camera, a microphone, an accelerometer, etc. Any of the data mentioned herein can be output from one component to another and can be output to a user.

[0309] The computer system may include multiple identical components or subsystems connected together, for example, via external interfaces 81, via internal interfaces, or via removable storage devices that can be connected to and removed from one component. In some embodiments, the computer systems, subsystems, or devices may communicate via a network. In such cases, one computer may be considered a client and another computer may be considered a server, where each computer may be part of the same computer system. The client and server may each include multiple systems, subsystems, or components.

[0310] Aspects of the embodiments may be implemented using hardware circuits (e.g., application specific integrated circuits or field programmable gate arrays) and / or in a modular or integrated manner using computer software with a substantially programmable processor in the form of control logic. As used herein, a processor may include a single-core processor, a multi-core processor on the same integrated chip, or multiple processing units on a single circuit board or networked, as well as dedicated hardware. Based on the present disclosure and the teachings provided herein, one of ordinary skill in the art will know and understand other ways and / or methods of implementing embodiments of the present invention using hardware and a combination of hardware and software.

[0311] Any software components or functions described in this application can be implemented as software code executed by a processor using any suitable computer language, such as Java, C, C++, C#, Objective-C, Swift, or a scripting language such as Perl or Python, using, for example, conventional or object-oriented techniques. The software code can be stored as a series of instructions or commands on a computer-readable medium for storage and / or transmission. Suitable non-transitory computer-readable media can include random access memory (RAM), read-only memory (ROM), magnetic media such as a hard drive or floppy disk, or optical media such as a compact disc (CD) or DVD (Digital Versatile Disc) or Blu-ray disc, flash memory, etc. The computer-readable medium can be any combination of such storage or transmission devices.

[0312] Such programs can also be encoded and transmitted using carrier signals, which are suitable for transmitting via wired, optical and / or wireless networks that meet multiple protocols including the Internet. Therefore, computer-readable media can be generated using data signals encoded with such programs. Computer-readable media encoded with program code can be packaged with compatible devices or provided separately from other devices (e.g., downloaded via the Internet). Any such computer-readable media can be present on or inside a single computer product (e.g., hard drive, CD, or entire computer system), and can be present on or inside different computer products within a system or network. A computer system can include a monitor, a printer, or other suitable displays for providing any result mentioned herein to a user.

[0313] Any method described herein can be performed in whole or in part with a computer system comprising one or more processors that can be configured to perform the steps. Therefore, embodiments may relate to a computer system configured to perform the steps of any method described herein, and the computer system may have different components that perform corresponding steps or corresponding step groups. Although presented as numbered steps, the steps of the method herein can be performed simultaneously or at different times or in different orders. In addition, the various parts of these steps can be used together with the various parts of other steps from other methods. Moreover, all or part of the steps can be optional. In addition, any one of the steps of any method can be performed using a module, unit, circuit or other component of a system for performing these steps.

[0314] The specific details of the particular embodiments may be combined in any suitable manner without departing from the spirit and scope of the embodiments of the invention. However, other embodiments of the invention may be directed to specific embodiments relating to each individual aspect or specific combinations of these individual aspects.

[0315] The above description of exemplary embodiments of the present disclosure has been presented for purposes of illustration and description. The above description is not intended to be exhaustive or to limit the disclosure to the precise form described, and many modifications and variations are possible in light of the above teachings.

[0316] Unless specifically indicated to the contrary, references to "a," "an," or "the" are intended to mean "one or more." Unless expressly indicated to the contrary, the use of "or" is intended to mean an inclusive or, not an exclusive or. Reference to a "first" component does not necessarily require the provision of a second component. Furthermore, reference to a "first" or "second" component does not limit the referenced components to specific locations, unless expressly stated. The term "based on" is intended to mean "based, at least in part, on."

[0317] All patents, patent applications, publications, and descriptions mentioned herein are incorporated by reference in their entirety for all purposes. No admission is made that they are prior art.

Claims

1. An optical ranging system, comprising: Core-based coprocessor; and A scanning light ranging device, comprising: a transmitting circuit, the transmitting circuit comprising a plurality of light sources for emitting light pulses; A detection circuit, the detection circuit comprising: A light sensor array that detects reflected light pulses and outputs a signal measured over time. Signals; and a signal processor connected to the light sensor array and configured to determine a depth value from measurements using the light sensor array; and an image reconstruction circuit communicatively coupled to the detection circuit and configured to: assigning a sensor ID to each first depth value of a first scan of the scanning light ranging device; Construct a first lidar image using the first depth value by performing the following operations: mapping the first depth value to a first lidar pixel in the first lidar image using the sensor ID, the first lidar image being a rectilinear image, wherein the mapping uses a mapping table that specifies lidar pixels based on corresponding sensor IDs; and storing the first lidar pixel of the first lidar image in a local image buffer of the scanning light ranging device; and A first lidar pixel of a partial frame of the first lidar image or a full frame of the first lidar image is sent to the core-based coprocessor.

2. The system according to claim 1, wherein: The scanning light ranging device rotates.

3. The system according to claim 1, wherein: The core-based coprocessor and the scanning light ranging device are located on the same integrated circuit.

4. The system according to claim 1, wherein: The detection circuit is configured to provide a sensor ID for each depth value to the image reconstruction circuit.

5. The system according to claim 1, wherein The detection circuitry is configured to provide the depth values ​​in a specified order, and wherein the image reconstruction circuitry is configured to assign a sensor ID to a particular depth value based on the specified order.

6. The system according to claim 1, wherein: A row of the mapping table includes a column for the sensor ID and another column specifying the coordinates of the lidar pixel in the lidar image.

7. The system according to claim 1, wherein: The image reconstruction circuit is configured to determine when a designated subset of the first lidar pixels has been stored for the first lidar image and to send the designated subset of the first lidar pixels to the core-based coprocessor.

8. The system according to claim 1, wherein: The core-based coprocessor includes a classifier circuit communicatively coupled to the scanning light ranging device and configured to: receiving the lidar image output by the image reconstruction circuit; analyzing the depth values ​​in lidar pixels of the lidar image; correlating a set of lidar pixels based on corresponding depth values ​​of the set of lidar pixels; as well as Outputting classification information of the set of lidar pixels based on the correlation.

9. The system according to claim 1, wherein: The core-based coprocessor includes a depth imaging circuit communicatively coupled to the scanning light ranging device and configured to: receiving the first lidar pixel; as well as One or more filter kernels are applied to the first subset of lidar pixels to generate a filtered image of the lidar pixels.

10. A method for performing ranging using an optical ranging system installed on a mobile device, the method comprising: transmitting pulses from one or more light sources of the optical ranging system using a transmission circuit, the pulses being reflected from one or more objects; For each photosensor in the photosensor array, measuring a signal by detecting photons of the pulse using a detection circuit; assigning a sensor ID to each of the signals, the sensor ID corresponding to one of the light sensor arrays; analyzing the signal to determine a first depth value; Construct a first lidar image using the first depth value by performing the following operations: Map the first depth value to the first lidar pixels, the first lidar image is a rectilinear image, wherein the mapping uses a mapping table, the mapping table specifying lidar pixels based on corresponding sensor IDs; and storing the first lidar pixel of the first lidar image in an image buffer of the optical ranging system; and The first lidar pixel of the partial frame of the first lidar image or the complete frame of the first lidar image is sent to a core-based coprocessor of the optical ranging system.

11. The method according to claim 10, wherein: The sensor ID and the signal are received from the detection circuit.

12. The method according to claim 10, wherein: The sensor IDs are assigned based on a specified order in which the detection circuits provide the signals.

13. The method according to claim 10, wherein: The sensor ID is assigned to each of the signals by assigning the sensor ID to the first depth value, and wherein the sensor ID is assigned based on a specified order in which the first depth values ​​are provided to an image reconstruction circuit.

14. The method according to claim 10, further comprising: constructing a second lidar image using the second depth value; storing the second lidar image in an image buffer; as well as One or more values ​​of the first lidar image are adjusted based on analysis of the first lidar image and the second lidar image, thereby obtaining one or more adjusted values.

15. The method according to claim 14, wherein The one or more adjustment values ​​include color values ​​of the first lidar image.

16. The method according to claim 10, further comprising: A filter kernel is applied to a portion of the first lidar image, wherein the filter kernel is applied before the first lidar image is fully constructed.

17. The method according to claim 10, wherein The mapping table specifies lidar pixels based on the corresponding sensor ID and the position of the light ranging system when measuring the signal.

Images