Multispectral ranging / imaging sensor array and system

Abstract

A multispectral sensor array is provided. The optical ranging / imaging system may include a combination of ranging sensor channels (e.g., LIDAR sensor channels) and ambient light sensor channels tuned to detect ambient light having channel-specific characteristics (e.g., color). The sensor channels may be positioned and spaced such that the multispectral images from the different sensors are essentially aligned with one another to provide a multispectral image of the field of view, defining an array of multispectral image pixels. Various optical elements may be provided to facilitate imaging operations. Optical ranging / imaging systems incorporating multispectral sensor arrays may be operated in rotational and / or static modes.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of the following four provisional applications: U.S. Ser. No. 62 / 716,900, filed August 9, 2018; U.S. Ser. No. 62 / 726,810, filed September 4, 2018; U.S. Ser. No. 62 / 744,540, filed October 11, 2018; and U.S. Ser. No. 62 / 877,778, filed July 23, 2019. The disclosures of all four provisional applications are incorporated herein by reference. [Background technology]

[0002] The present disclosure relates generally to optical imaging systems, and more particularly to sensor systems with multiple sensor channels, including sensor channels tuned to different light characteristics or properties and usable for ranging.

[0003] Optical Imaging, Detection, and Ranging (LIDAR) systems measure the distance to a target by shining a pulsed laser light at the target and measuring the reflected pulse with a sensor. Time-of-flight measurements can then be used to create a digital 3D representation of the target. LIDAR systems can be used in a variety of applications where 3D depth imaging is useful, including archaeology, geography, geology, forestry, mapping, construction, medical imaging, and military applications, among others. Autonomous vehicles can also use LIDAR for obstacle detection and avoidance and vehicle navigation.

[0004] In applications such as vehicle navigation, depth information (e.g., distance to objects in the environment) is very useful but not sufficient to safely navigate and avoid hazards. It is also necessary to identify specific objects, such as traffic signals, lane markings, and moving objects that may intersect the vehicle's path. Therefore, a system such as an autonomous vehicle may include both a LIDAR system and another imaging system, such as a visible light camera, that can capture ambient light, including reflected light from objects in the environment and direct light from any light sources that may be present in the environment. Each imaging system (LIDAR and visible light) independently provides images that include either depth data or spectral data. In some applications, it is beneficial to align different images with each other, for example, by performing image registration to identify the location of the same object in different images. Image registration can be a complex and computationally intensive task. For example, different imaging systems may have different resolutions and / or frame boundaries, and alignment between independently built and / or independently controlled imaging systems may be inaccurate. Summary of the Invention

[0005] Certain embodiments of the invention described herein relate to multispectral sensor arrays that incorporate multiple sensor channel types, including a depth channel (e.g., a LIDAR sensor channel) and one or more different ambient light sensor channels, within the same sensor array (which may be, for example, a monolithic ASIC sensor array). Because the different types of channels are within the same sensor array, the channels can be intrinsically aligned with one another with high precision. Different channels can be tuned (e.g., using optical filters) to be sensitive to light with specific characteristics, such as a particular wavelength range (which may be broadband or narrowband as desired), specific polarization characteristics (e.g., linearly polarized in a particular direction, circularly polarized, etc.). The sensor array can be used in combination with imaging optics to generate images containing pixel data corresponding to each channel type. Images generated from different sensor types within the same sensor array are "intrinsically" registered with one another due to the channel alignment within the sensor array. That is, the spatial relationship between different types of pixels (or channels) is established in the design of the sensor array and can be used to map pixel data from the different sensor types to the same pixel location within the field of view.

[0006] In some embodiments, some or all of the channels can have channel-specific (or channel-type-specific) compensating micro-optics that depend on the channel's location within the array and / or the particular wavelength range the channel is tuned in. Such micro-optics can be used, for example, to compensate for chromatic aberration, focal plane curvature, or other optical properties of the bulk imaging optics.

[0007] In some embodiments, different ambient light sensor channels can be tuned to different overlapping wavelength bands (e.g., using optical filters with overlapping passbands), and arithmetic logic circuitry can be used to determine the light intensities in the various wavelength bands based on measurements in the overlapping wavelength bands.

[0008] In some embodiments, a ranging / imaging system can scan a field of view using a multispectral sensor array, for example, by rotating the sensor array about an axis perpendicular to the columns. During this movement, a given location in space can be imaged sequentially by each of the channel types, thereby providing a multispectral image set with unique registration between the imaging modalities (or channels). The spatial relationship of the channels within the array, the optical properties of the imaging optics (e.g., the focal length distortion profile of the bulk imaging optics), and the imaging speed relative to the movement (e.g., rotation) of the sensor array can be selected so that data from different channels can be easily mapped to a uniform grid of pixels representing the field of view.

[0009] In some embodiments in which a multispectral sensor array is scanned, a group of two or more ambient light sensor channels in a row can have the same type of optical filter and sub-pixel sized aperture positioned differently for different ambient light sensor channels in the group. Based on the light intensity measurements (e.g., photon counts) from the ambient light sensor channels in the group, an ambient light image can be obtained with increased resolution in the scanning and / or non-scanning directions.

[0010] In some embodiments, a "2D" (two-dimensional) multispectral sensor array can be provided, where the array includes a two-dimensional arrangement of multispectral pixels. Each multispectral pixel can include a depth channel along with one or more ambient light sensor channels. Such arrays can be used in moving (e.g., rotating) ranging / imaging systems, as well as "static" systems where imaging of a field of view is achieved without moving the sensor array.

[0011] Some embodiments relate to a sensor array including sensor channels arranged in several sensor rows. Each sensor row may include a set of ranging sensor channels (e.g., LIDAR sensor channels) and one or more ambient light sensor channels. Each ambient light sensor channel may include an aperture (e.g., for determining the channel's field of view), a light sensor (e.g., one or more single-photon avalanche diodes), and a channel-specific optical filter that selectively passes light having channel-specific characteristics (e.g., desired color, polarization state, etc.). In some embodiments, some or all of the sensor channels may include channel-specific micro-optical elements that direct light having the channel-specific characteristics through the aperture to the light sensor, e.g., to compensate for chromatic aberrations in bulk imaging optics that may be positioned in front of the array. In some embodiments, the ambient light sensor channels are multispectral channels including multiple light sensors tuned to detect light having different characteristics (e.g., using patterned optical filters). In some embodiments, the sensor array may include a 2D array of "hybrid" sensor channels that includes one group of optical sensors configured for depth operations (e.g., LIDAR sensing) and one or more other groups of optical sensors configured for sensing ambient light having different characteristics. Sensor arrays of the type described herein may be incorporated into optical ranging / imaging systems and / or other optical systems.

[0012] Some embodiments relate to an optical sensor array having an arrangement of sensor channels and a corresponding arrangement of apertures in an aperture plane. A bulk optics module can be used to direct and focus light from an area being imaged onto the sensor array. If the bulk optics module has a curved focal plane, per-channel micro-optics of varying prescriptions and / or offset distances from the aperture plane can be placed in front of the apertures to compensate for the offset between the location of the apertures and their corresponding locations on the curved focal plane. Similarly, an optical emitter array can have an arrangement of emitter channels (e.g., narrowband emitters that produce light at wavelengths usable for LIDAR applications) and a corresponding arrangement of apertures in the aperture plane. A bulk optics module can be used to direct the emitted light passing through the apertures toward the area being imaged. If the bulk optics module has a curved focal plane, per-channel micro-optics of varying prescriptions and / or offset distances from the aperture plane can be placed in front of the apertures to compensate for the offset between the location of the apertures and their corresponding locations on the curved focal plane. In such embodiments, the prescription (e.g., focusing power) and / or standoff distance of the per-channel micro-optics from the aperture plane can be varied, for example, as a function of radial distance from the optical axis in the aperture plane. This can improve the efficiency of light emission and / or light collection. Per-channel micro-optics for correcting focal plane curvature of bulk optics modules can be used in optical receiving modules and / or optical transmitting modules regardless of the specific characteristics of the light emitters or light sensors. In some embodiments where different channels are tuned to emit or receive light of different wavelengths, the per-channel micro-optics can correct both focal plane curvature and chromatic aberration that may be present in the bulk optics module.

[0013] Some embodiments relate to a sensor array having sensor rows. Each sensor row includes a LIDAR sensor channel and a set of one or more ambient light sensor channels (e.g., one, three, five, six, or more). Each ambient light sensor channel includes a channel input aperture, a light sensor, and a channel-specific optical filter that selectively passes light having channel-specific characteristics to the light sensor. The light sensor of each ambient light sensor channel may be one or more photodiodes, such as one or more single-photon avalanche diodes (SPADs) operating in a photon-counting mode. In some embodiments, each LIDAR sensor channel may also include one or more SPADs operating in a photon-counting mode, and the same type of light sensor may be used for both the LIDAR sensor channel and the ambient light sensor channel.

[0014] In some embodiments, the set of ambient light sensor channels can include at least two ambient light sensor channels, each having a different channel-specific optical filter. For example, the set of ambient light sensor channels can include a red channel whose channel-specific optical filter selectively passes red light, a green channel whose channel-specific optical filter selectively passes green light, and a blue channel whose channel-specific optical filter selectively passes blue light. As another example, the set of ambient light sensor channels can include at least five different color channels, each of which has a channel-specific optical filter that selectively passes light having a different wavelength range. The different channel-specific optical filters can have overlapping or non-overlapping passbands as needed, and a particular optical filter can have a wide passband (e.g., the entire visible light spectrum) or a narrow passband (e.g., 25 nm or less, such as a passband corresponding to the emission spectrum of a typical light-emitting diode (LED)). For example, a first color channel can have a first channel-specific optical filter that selectively passes light having a first wavelength range, and a second color channel has a second channel-specific optical filter that selectively passes light having a second wavelength range. The second range can correspond to the absorption band of a particular substance, and data from the two color channels can be used to identify the substance.

[0015] In some embodiments, the ambient light sensor channels may also be selectively sensitive to properties of light other than wavelength. For example, a set of ambient light sensor channels may include one or more polarization channels in which per-channel optical filters selectively pass light having particular polarization characteristics. Color and polarization channels may be provided in combination to provide information about both the spectral and polarization properties of ambient light.

[0016] In some embodiments, a row of ambient light sensor channels may include a "multispectral" sensor channel that may include multiple photosensors and a patterned optical filter, where different portions of the patterned optical filter selectively pass light having different characteristics to different subsets of photosensors in the multispectral sensor channel. The different portions of the patterned optical filter may include, for example, a first portion that passes light in a first wavelength band, a second portion that passes light in a second wavelength band (which may be overlapping wavelength bands), a portion that passes light with particular polarization characteristics, etc.

[0017] The sensor channels in the array can be arranged as needed. For example, in an embodiment where the set of one or more ambient light sensor channels includes at least two ambient light sensor channels, each having a different channel-specific optical filter, the ambient light sensor channels in a given sensor row can be spaced apart from one another by a uniform pitch. The LIDAR sensor channels in a given sensor row can be spaced apart from the nearest ambient light sensor channel in the given sensor row by a uniform pitch or a distance that is an integer multiple of the uniform pitch. Adjacent sensor rows can also be spaced apart from one another by a uniform pitch. This allows for uniform sampling of object space when the sensor array is used in a scanning operation.

[0018] In some embodiments, the sensor array is fabricated as a single ASIC, which may also include other components, such as a data buffer disposed within the ASIC and configured to store data from two or more of the LIDAR sensor channels and two or more of the ambient light sensor channels, and / or processing circuitry disposed within the ASIC and configured to perform image processing operations on data stored in the data buffer.

[0019] Some embodiments relate to a ranging / imaging system having a fixed base, a sensor array rotatably coupled to the fixed base, a bulk optics module, and a controller. The sensor array may include sensor rows, each sensor row having a LIDAR sensor channel and a set of one or more ambient light sensor channels with per-channel optical filtering. The bulk optics module may be disposed in front of the sensor array and configured to focus incident light onto an aperture plane common to the LIDAR sensor channels and the ambient light sensor channels. The controller may synchronize the rotation of the sensor array with the operation of the optical sensors such that a given location in space relative to the fixed base is successively imaged by each of the LIDAR sensor channels and the ambient light sensor channels in one of the sensor rows. The controller may also be configured to generate multispectral image pixel data including per-pixel light intensity data determined using the ambient light sensor channels of the sensor array and per-pixel depth data determined using the LIDAR sensor channels of the sensor array. In some embodiments, the ambient light sensor channels in a given sensor row are spaced apart from one another by a uniform pitch, and the controller is further configured to rotate the ranging / imaging system so that successive imaging operations occur at angular positions spaced apart by a pitch angle corresponding to the uniform pitch. The LIDAR sensor channels in a given sensor row can be spaced apart from the nearest ambient light sensor channels in the given sensor row by the uniform pitch or a distance that is an integer multiple of the uniform pitch. In some embodiments, adjacent sensor rows are also spaced apart from one another by a uniform pitch.

[0020] Some embodiments relate to a sensor array having a two-dimensional array of hybrid sensor pixels. Each hybrid sensor pixel can include a set of a LIDAR sensor channel and one or more ambient light sensor channels, each tuned to selectively measure the intensity of light having a sensor-specific characteristic. The sensor array can also include readout electronics coupled to each hybrid sensor pixel in the two-dimensional array, the readout electronics of each hybrid sensor pixel including: a timing circuit coupled to the LIDAR sensor channel and configured to time the arrival of photons at the LIDAR sensor channel and store data representing the photon arrival times in memory; and a counter circuit coupled to the ambient light sensor channel and configured to count the number of photons detected at the ambient light sensor channel and store the photon counts in memory.

[0021] In some embodiments, the two-dimensional array of hybrid sensor pixels is formed as a single ASIC. Each hybrid sensor pixel may include a planar array of photosensors and a patterned optical filter, where different portions of the patterned optical filter selectively pass light having different characteristics to different subsets of photosensors in the planar array. The patterned optical filter may be arranged such that a first subset of photosensors receives infrared light within a narrow passband corresponding to the wavelength of a LIDAR emitter, thereby providing a LIDAR sensor channel, and a second subset of photosensors receives visible light from at least a portion of the visible light spectrum, thereby providing one of the ambient light sensor channels. In some embodiments, the first subset of photosensors is located in a central region within the pixel region of the hybrid sensor pixel, and the second subset of photosensors is located in a peripheral region around the central region within the pixel region. In some embodiments, the second subset of photosensors includes two or more photosensors, and the patterned optical filter is further arranged such that each of the two or more photosensors in the second subset receives light having different characteristics, such as different wavelength ranges or different polarization characteristics.

[0022] In some embodiments, the LIDAR sensor channels of the two-dimensional array of hybrid sensor channels are formed as a first ASIC, and the ambient light sensor channels are formed as a second ASIC overlaid on and aligned with the first ASIC, the second ASIC having a plurality of apertures formed therein to allow light to pass to the LIDAR sensor channels.

[0023] Some embodiments relate to a ranging / imaging system including a sensor array having a two-dimensional array of hybrid sensor pixels and a controller. Each hybrid sensor pixel may include a planar array of photosensors and a patterned optical filter, where different portions of the patterned optical filter selectively pass light having different characteristics to different subsets of photosensors in the planar array. The patterned optical filter may be arranged such that a first subset of photosensors receives infrared light within a narrow passband matching the wavelength of a LIDAR emitter, thereby providing a LIDAR sensor channel, and a second subset of photosensors receives visible light from at least a portion of the visible light spectrum, thereby providing one of the ambient light sensor channels. The controller may be configured to operate the LIDAR sensor channel and the ambient light sensor channel such that a given location in the field of view is imaged by the LIDAR sensor channel and the ambient light sensor channel of one of the hybrid sensor pixels. In some embodiments, the ranging / imaging system also includes an emitter for emitting light detectable by the LIDAR sensor channels, and the controller can be further configured to coordinate operation of the emitter with operation of the LIDAR sensor channels to determine depth measurements for each hybrid sensor pixel. The controller can also be further configured to operate the emitter and the LIDAR sensor channels to perform an electronic scan of the field of view, such that different portions of the field of view are imaged by different ones of the LIDAR sensor channels at different times.

[0024] Some embodiments relate to an imaging system having a fixed base, a sensor array rotatably coupled to the fixed base, a bulk optics module, and a controller. The sensor array can have multiple sensor rows, each sensor row including one or more sets of ambient light sensor channels, each including a channel input aperture, a light sensor, and a channel-specific optical filter that selectively passes light having channel-specific characteristics to the light sensor. The bulk optics module can be disposed in front of the sensor array and configured to focus incident light onto a common aperture plane for the ambient light sensor channels. The controller can be configured to synchronize the rotation of the sensor array with the operation of the light sensors to generate image pixel data including light intensity data determined using the ambient light sensor channels. In some embodiments, the set of one or more ambient light sensor channels includes at least two ambient light sensor channels, with different ambient light sensor channels having different channel-specific optical filters. The ambient light sensor channels within a given sensor row are spaced apart from one another by a uniform pitch. In some embodiments, adjacent sensor rows are also spaced apart from one another by the same uniform pitch. This can facilitate uniform sampling of the field of view. In some embodiments, the imaging system may also include a data buffer disposed within the ASIC and configured to store data from the two or more ambient light sensor channels, and a processing circuit disposed within the ASIC and configured to perform image processing operations on the data stored in the data buffer.

[0025] Some embodiments relate to an imaging system including a sensor array, a bulk optics module, a controller, and a plurality of per-channel micro-optical elements. The sensor array can have sensor channels arranged to receive light through corresponding apertures in an aperture plane. The bulk optics module can be disposed in front of the sensor array and configured to focus incident light onto the aperture plane to form an image of the field of view. The controller can operate the sensor array to generate image data of the field of view. Each of the per-channel micro-optical elements can be disposed in front of a different one of the apertures and can have different optical prescriptions for different sensor channels. The optical prescription of a particular one of the per-channel micro-optical elements can be based at least in part on optical properties of the bulk optics module, such as chromatic aberration (for a sensor channel that is color-selective) and / or focal plane curvature (where the optical prescription can be a function of radial distance from the optical axis of the bulk optics module). The optical prescription can include a focal length (or focusing power) and / or a standoff distance.

[0026] In some embodiments, the sensor channels are arranged in sensor rows, each sensor row including a LIDAR sensor channel and a set of one or more ambient light sensor channels, each including a channel input aperture, a light sensor, and a per-channel optical filter that selectively passes light having a channel-specific characteristic to the light sensor. Per-channel micro-optics can be provided for at least some of the ambient light sensor channels. For example, the per-channel micro-optics for each ambient light sensor channel can have a prescription based at least in part on the per-channel optical filter, e.g., to compensate for chromatic aberrations of the bulk optics module.

[0027] In some embodiments, the sensor channels include LIDAR sensor channels, at least some of which can have corresponding per-channel optical elements with respective optical prescriptions based in part on the LIDAR operating wavelength and in part on the optical properties of the bulk optics module.

[0028] Some embodiments relate to a LIDAR transmitter device including an emitter array, a bulk optics module, and a per-channel micro-optics element. The emitter array can have a plurality of emitter channels arranged to emit light through a corresponding plurality of apertures in an aperture plane. The bulk optics module can be disposed in front of the emitter array and configured to direct light from the aperture plane toward a field of view. The per-channel micro-optics elements can each be disposed in front of a different one of the apertures and each have a different optical prescription for a different sensor channel. The optical prescriptions of the per-channel micro-optics elements can be based at least in part on optical properties of the bulk optics module. For example, if the bulk optics module has a curved focal plane, the optical prescription of each per-channel micro-optics element can compensate for an offset between the location of the aperture and the corresponding location on the curved focal plane by using an optical prescription for each per-channel micro-optics element that is a function of the radial distance in the aperture plane from the optical axis of the bulk optics module to the corresponding aperture. The optical prescription may include a focal length (or focusing power) and / or a standoff distance, and thus channel-specific micro-optical elements disposed in front of different apertures may have optical prescriptions with different focusing powers and / or different standoff distances from the aperture plane.

[0029] Some embodiments relate to a scanning imaging system for providing an image having a fixed resolution in a scan direction. The scanning imaging system may include a sensor array, a rotational control system, and a bulk optics module. The sensor array may include a set of sensor channels arranged in two dimensions, each configured to detect light (with the same or different characteristics). The rotational control system may be configured to rotate the sensor array in the scan direction through a sequence of angular measurement positions to acquire frames of data representing an image of a field of view, such as a grid of image pixels spaced apart in the scan direction according to a uniform angular pitch. The bulk optics module may be configured to focus light toward the sensor array and may have a focal length and a focal length distortion profile, both adjusted to the arrangement of the set of sensor channels, such that rotating the sensor array along the scan direction through the uniform angular pitch shifts the location at which light rays impinge on the sensor array from one sensor channel to an adjacent sensor channel.

[0030] The set of sensor channels can include various combinations of channel types. For example, the set of sensor channels can include a staggered grid of LIDAR sensor channels defining columns extending transverse to the scan direction. Additionally (or alternatively), the set of sensor channels can include one or more ambient light sensor channels disposed along the scan direction for each of the LIDAR sensor channels.

[0031] In some embodiments, the sensor array has a fixed pitch between adjacent sensor channels along the scan direction, and the bulk optics module has either an Fθ focal length distortion profile or an F tan θ focal length distortion profile.

[0032] In other embodiments, the sensor array may have a variable distance between adjacent sensor channels. For example, if the focal length distortion profile of the bulk optics module exhibits barrel distortion, the distance between adjacent sensor channels in the sensor array may increase from the edge toward the center of the sensor array. Similarly, if the focal length distortion profile of the bulk optics module exhibits pincushion distortion, the distance between adjacent sensor channels in the sensor array may decrease from the edge toward the center of the sensor array. Such an arrangement may provide uniform sampling of the object space.

[0033] Some embodiments relate to a scanning imaging system for providing an image having a fixed resolution in a scan direction. The scanning imaging system can include a sensor array, a mirror subsystem, and a bulk optics module. The sensor array can include a set of sensor channels arranged in one or two dimensions, with each sensor channel configured to detect light (with the same or different characteristics). The mirror subsystem can direct light from different portions of a field of view onto the sensor array at different times, and the sensor array can be configured to acquire frames of data representing images of the field of view; for example, the frames of data can be a grid of image pixels spaced in the scan direction according to a uniform angular pitch. The bulk optics module can be configured to focus light toward the sensor array and can have a focal length and a focal length distortion profile, both adjusted to the arrangement of the set of sensor channels; rotating the sensor array along the scan direction through the uniform angular pitch shifts the location where light rays impinge on the sensor array from one sensor channel to an adjacent sensor channel.

[0034] The set of sensor channels can include various combinations of channel types. For example, the set of sensor channels can include a staggered grid of LIDAR sensor channels defining columns extending transverse to the scan direction. Additionally (or alternatively), the set of sensor channels can include one or more ambient light sensor channels disposed along the scan direction for each of the LIDAR sensor channels.

[0035] In some embodiments, the sensor array has a fixed pitch between adjacent sensor channels along the scan direction, and the bulk optics module has either an Fθ focal length distortion profile or an F tan θ focal length distortion profile.

[0036] In other embodiments, the sensor array may have a variable distance between adjacent sensor channels. For example, if the focal length distortion profile of the bulk optics module exhibits barrel distortion, the distance between adjacent sensor channels in the sensor array may increase from the edge toward the center of the sensor array. Similarly, if the focal length distortion profile of the bulk optics module exhibits pincushion distortion, the distance between adjacent sensor channels in the sensor array may decrease from the edge toward the center of the sensor array. Such an arrangement may provide uniform sampling of the object space.

[0037] Some embodiments relate to a raster scan imaging system for providing an image having a fixed resolution by scanning in two dimensions. The raster scan imaging system can include a sensor array, a raster scanning mechanism, and a bulk optics module. The sensor array can include a set of sensor channels arranged in one or two dimensions, each configured to detect light. The raster scanning mechanism can be configured to perform a raster scan in one or two dimensions, directing light from different portions of a field of view onto the sensor array at different times, causing the sensor array to acquire frames of data representing images of the field of view. The frames of data can be, for example, a two-dimensional grid of image pixels spaced in each of the two dimensions according to a uniform pitch, with both dimensions of the image pixel grid being larger than the dimensions of the sensor array. The bulk optics module can be configured to focus light toward the sensor array and can have a focal length and a focal length distortion profile, both adjusted to the arrangement of the set of sensor channels so that the sensor array uniformly samples the field of view.

[0038] In some embodiments, raster scanning can operate by moving the sensor array in two dimensions to direct sensor channels to different portions of the field of view, hi other embodiments, the raster scanning mechanism can include a tip-tilt mirror movable in two dimensions to direct light from different portions of the field of view onto the sensor array at different times.

[0039] The set of sensor channels can include various combinations of channel types. In some embodiments, the sensor channels can include LIDAR sensor channels and various types of ambient light sensor channels. In other embodiments, the sensor channels can include one or more “hybrid” sensor channels, each having a plurality of optical sensors and a patterned optical filter, where different portions of the patterned optical filter selectively pass light having different characteristics, and the patterned optical filter can be arranged such that different optical sensors receive light having different characteristics. The patterned optical filter can be further arranged such that a first subset of the plurality of optical sensors receives infrared light within a narrow passband matching the wavelength of the LIDAR emitter, and a second subset of the plurality of optical sensors receives visible light from at least a portion of the visible light spectrum. As another example, a hybrid sensor channel may include a LIDAR sensor channel disposed on a first sensor channel layer; an aperture layer overlying the first sensor channel layer and having an aperture therein to allow light to enter the LIDAR sensor channel; and ambient light sensor channels disposed in at least a portion of the aperture layer around the aperture, each ambient light sensor channel including a light sensor and an optical filter that selectively passes light having particular characteristics, and the optical filters of different ones of the ambient light sensor channels selectively pass light having different characteristics.

[0040] In some embodiments, the sensor array of the raster scan imaging system has a fixed pitch between sensor channels, and the bulk optics module has either an F tan θ focal length distortion profile or an Fθ focal length distortion profile.

[0041] Some embodiments relate to a sensor array having multiple sensor rows, logic circuitry, and a controller. Each sensor row can include a group of two or more high-resolution ambient light sensor channels sensitive to a range of wavelengths, and each high-resolution ambient light sensor channel in the group can include a channel-specific input aperture, where the channel-specific input apertures of different high-resolution ambient light sensor channels in the group expose different portions of the channel area, and a light sensor. The logic circuit can determine multiple subpixel light intensity values ​​based on intensity data from the light sensor in the group of high-resolution ambient light sensor channels. The controller can be configured to perform a scanning operation that exposes the sensor array to different areas in the field of view at different times, so that each ambient light sensor channel in the group of two or more high-resolution ambient light sensor channels is exposed to the same pixel area in the field of view at different times.

[0042] In some embodiments, each high-resolution ambient light sensor channel in the group can include an optical filter that selectively passes light having particular characteristics, the particular characteristics being the same for every high-resolution ambient light sensor channel in the group.

[0043] In some embodiments, the different portions of the channel area exposed by the apertures of different high resolution ambient light sensor channels in a group are non-overlapping portions of the channel area. For example, a group of high resolution ambient light sensor channels may include four high resolution ambient light sensor channels, and the non-overlapping portions may correspond to different quadrants of the channel area.

[0044] In other embodiments, the different portions of the channel areas exposed by the apertures of different high-resolution ambient light sensor channels in a group can include overlapping portions of the channel areas. Arithmetic and logic circuitry can be provided to decode intensity values ​​of the set of non-overlapping portions of the channel areas based on sensor data from a group of two or more high-resolution ambient light sensor channels. To facilitate decoding, one (or more) of the high-resolution ambient light sensor channels in the group can have an aperture that exposes the entire channel area.

[0045] In some embodiments, each sensor row further includes a LIDAR sensor channel spatially registered to the group of high-resolution ambient light sensor channels, wherein the LIDAR sensor channel can provide a depth image having a first resolution, while the high-resolution ambient light sensor channel provides an intensity image having a second resolution higher than the first resolution in a row-by-row direction and / or in a direction perpendicular to the sensor row.

[0046] Some embodiments relate to a scanning imaging system including a sensor array, an arithmetic logic circuit, and a controller. The sensor array can include a group of two or more high-resolution ambient light sensor channels sensitive to a range of wavelengths, each of which can include a channel-specific input aperture, where the channel-specific input apertures of different high-resolution ambient light sensor channels in the group expose different portions of the channel area; a light sensor; and two or more registers for accumulating photon counts from the light sensor during a time interval subdivided into two or more time bins, each register accumulating photon counts during a different one of the time bins. The arithmetic logic circuit can calculate multiple subpixel light intensity values ​​based on the photon counts accumulated in the multiple registers of all of the high-resolution ambient light sensor channels in the group. The controller can be configured to perform a scanning operation that exposes the sensor array to different regions in the field of view at different times, such that each ambient light sensor channel in the group of two or more high-resolution ambient light sensor channels is exposed to the same pixel region in the field of view at different times.

[0047] In some embodiments, each high-resolution ambient light sensor channel in the group can include an optical filter that selectively passes light having particular characteristics, the particular characteristics being the same for every high-resolution ambient light sensor channel in the group.

[0048] In some embodiments, the scanning imaging system also includes a LIDAR sensor channel spatially registered to the group of high-resolution ambient light sensor channels, wherein the LIDAR sensor channel can provide a depth image having a first resolution, while the high-resolution ambient light sensor channel provides an intensity image having a second resolution in one or two dimensions that is higher than the first resolution.

[0049] The different portions of the channel area exposed by the apertures of different high-resolution ambient light sensor channels in the group can include overlapping and / or non-overlapping portions of the channel area. For example, the group of two or more high-resolution ambient light sensor channels can include four ambient light sensor channels, the two or more registers can include four registers, and the arithmetic logic circuit can calculate 16 sub-pixel light intensity values. For example, the per-channel input aperture of a first one of the high-resolution ambient light sensor channels exposes one-quarter of the channel area, and the per-channel input apertures of each of the second, third, and fourth high-resolution ambient light sensor channels each expose a different portion of the one-quarter of the channel area, and the 16 sub-pixel light intensity values ​​expose different portions of the 4×4 grid corresponding to the channel area.

[0050] Some embodiments relate to a sensor array having multiple sensor rows. Each sensor row can include a set of at least two ambient light sensor channels, each of which can include a channel input aperture, a light sensor, and a channel-specific optical filter that selectively passes light having a channel-specific characteristic to the light sensor. The set of at least two ambient light sensor channels in each sensor row can include at least two overlapping ambient light sensor channels with respective channel-specific optical filters whose channel-specific characteristics of light overlap. The sensor array can also include an arithmetic logic circuit that can decode signals from the three or more ambient light sensor channels into respective light intensity levels of light having multiple non-overlapping characteristics.

[0051] In some embodiments, the channel-specific characteristics include wavelength ranges of light. The set of at least two overlapping ambient light sensor channels includes a first color channel having a first channel-specific optical filter that selectively passes light having a first wavelength range, a second color channel having a second channel-specific optical filter that selectively passes light having a second wavelength range, and a third color channel having a third channel-specific optical filter that selectively passes light having a third wavelength range, wherein the first wavelength range and the second wavelength range partially overlap, and the third wavelength range encompasses both the first wavelength range and the second wavelength range. For example, the third wavelength band can correspond to the visible light spectrum.

[0052] In some embodiments, the channel-specific characteristics may be different characteristics such as polarization characteristics of the light.

[0053] In some embodiments, each sensor row further includes a LIDAR sensor channel, and depth data determined from the LIDAR sensor channel may be inherently registered to intensity data determined from the ambient light sensor channel.

[0054] Some embodiments relate to an imaging system including a sensor array, a controller, and an arithmetic logic circuit. The sensor array can have multiple sensor rows. Each sensor row can include a set of at least two ambient light sensor channels, each including a channel input aperture, a light sensor, and a channel-specific optical filter that selectively passes light having a channel-specific characteristic to the light sensor. The set of at least two ambient light sensor channels in each sensor row can include at least two overlapping ambient light sensor channels with respective channel-specific optical filters whose channel-specific characteristics of light overlap. The controller can operate the sensor array so that each of the three or more ambient light sensor channels is exposed to light from the same portion of the field of view. The arithmetic logic circuit can decode signals from the at least two overlapping ambient light sensor channels into respective light intensity levels of light having multiple non-overlapping characteristics.

[0055] Some embodiments relate to a sensor array including a plurality of sensor channels, including a multispectral sensor channel. Each multispectral sensor channel can include a channel input aperture, at least three light sensors, and a patterned optical filter having at least three distinct portions, the distinct portions of the patterned optical filter selectively passing light having different characteristics to different subsets of the at least three light sensors. The distinct portions of the patterned optical filter can include at least a first portion that passes light to a first subset of the at least three light sensors and a second portion that passes light to a second subset of the at least three light sensors, the respective characteristics of the light passed by the first and second portions overlapping. An arithmetic logic circuit can decode signals from the first and second subsets of light sensors into respective light intensity levels of the light having the plurality of non-overlapping characteristics. As with other embodiments, the characteristics can include wavelength ranges and / or polarization characteristics.

[0056] In some embodiments, the sensor channel may include multiple LIDAR sensor channels arranged such that each LIDAR sensor channel forms a sensor row with a different one of the multispectral sensor channels, and depth data determined from the LIDAR sensor channels may be inherently registered to intensity data determined from the ambient light sensor channels.

[0057] In some embodiments, each multispectral sensor channel can include a LIDAR optical sensor, and the patterned optical filter can include a fourth portion that selectively passes light having a wavelength corresponding to the LIDAR emitter to the LIDAR optical sensor.

[0058] The following detailed description will provide a better understanding of the nature and advantages of the claimed invention. [Brief explanation of the drawings]

[0059] [Figure 1A] 1 shows a simplified cross-sectional view of a single micro-optical sensor channel that may be included in a sensor array according to one or more embodiments described herein. [Figure 1B] 1 shows a simplified cross-sectional view of a single micro-optical sensor channel that may be included in a sensor array according to one or more embodiments described herein.

[0060] [Figure 2] FIG. 1 shows a simplified front view of a sensor array according to one or more embodiments described herein.

[0061] [Figure 3] 3 shows a simplified cross-sectional side view of a row of sensors of the sensor array of FIG. 2.

[0062] [Figure 4] 1 shows a simplified top view of another sensor array according to one or more embodiments.

[0063] [Figure 5] 1 shows a simplified top view of another sensor array according to one or more embodiments.

[0064] [Figure 6] 1 shows a simplified top view of another sensor array according to one or more embodiments.

[0065] [Figure 7] FIG. 7 shows a simplified top view of a multispectral sensor channel that may be included in the sensor array of FIG. 6.

[0066] [Figure 8] 7 shows a schematic side view of a portion of the sensor array of FIG. 6.

[0067] [Figure 9] 1 shows a simplified cross-sectional side view of a portion of a sensor array, according to an embodiment of the present invention.

[0068] [Figure 10] FIG. 10 shows a simplified top view of the sensor array of FIG. 9.

[0069] [Figure 11] 1 illustrates an example of three non-overlapping passbands of filters that can be used to provide ambient light intensity measurements in some embodiments.

[0070] [Figure 12] 1 shows an example of overlapping passbands of three filters that can be used to provide ambient light intensity measurements using encoded spectral information, in some embodiments.

[0071] [Figure 13] 1 shows a simplified front view of a sensor array according to some embodiments.

[0072] [Figure 14A] 1 illustrates an example of a multispectral sensor channel with a patterned optical filter, according to some embodiments. [Figure 14B] 1 illustrates an example of a multispectral sensor channel with a patterned optical filter, according to some embodiments.

[0073] [Figure 15] 3 shows a simplified side view of an optical ranging / imaging device that can incorporate the sensor array of FIG. 2.

[0074] [Figure 16] 1 is a simplified cross-sectional view of a portion of a sensor array with per-channel compensation micro-optical elements according to one or more embodiments.

[0075] [Figure 17] 1 is a simplified cross-sectional view of a portion of a sensor array with per-channel compensation micro-optical elements according to one or more embodiments.

[0076] [Figure 18]1 is a simplified cross-sectional view of a portion of a sensor array with per-channel compensation micro-optical elements according to one or more embodiments.

[0077] [Figure 19] FIG. 2 is a simplified cross-sectional view of a portion of a sensor array including an achromatic bulk optics module according to one or more embodiments.

[0078] [Figure 20] FIG. 10 is a simplified cross-sectional view of a portion of another sensor array including an achromatic bulk optics module according to one or more embodiments.

[0079] [Figure 21] 1 illustrates an example sensor module with micro-optics per channel to compensate for the focal length of a bulk optics module, according to one or more embodiments.

[0080] [Figure 22] 1 illustrates another example of a receiver module with per-channel micro-optical elements, according to one or more embodiments.

[0081] [Figure 23] 1 illustrates an example of a transmitter module with per-channel micro-optical elements, according to one or more embodiments.

[0082] [Figure 24A] FIG. 1 is a simplified conceptual diagram illustrating the possibility of pointing errors in a scanning system using a sensor array. [Figure 24B] FIG. 1 is a simplified conceptual diagram illustrating the possibility of pointing errors in a scanning system using a sensor array.

[0083] [Figure 25] FIG. 1 is a simplified optical diagram illustrating focal length distortion characteristics of a bulk optics module according to one or more embodiments.

[0084] [Figure 26] 10 shows an example of intra-pixel pointing error that can be quantified and constrained in some embodiments of the scanning system.

[0085] [Figure 27] 1 illustrates an example of sensor channel positioning to compensate for barrel and pincushion distortions in a bulk optics module, according to one or more embodiments. [Figure 28] 1 illustrates an example of sensor channel positioning to compensate for barrel and pincushion distortions in a bulk optics module, according to one or more embodiments.

[0086] [Figure 29] 1 illustrates an example of raster scanning using a sensor array, according to one embodiment of the present invention.

[0087] [Figure 30] 1 illustrates the non-uniform sampling pattern that can result from raster scanning using an array with bulk optical elements that exhibit pincushion distortion.

[0088] [Figure 31A] 31A and 31B show an example of a rotational imaging / LIDAR system according to one or more embodiments, with FIG. 31A being a simplified top view and FIG. 31B being a simplified side view. [Figure 31B] 31A and 31B show an example of a rotational imaging / LIDAR system according to one or more embodiments, with FIG. 31A being a simplified top view and FIG. 31B being a simplified side view.

[0089] [Figure 32] 1 illustrates a block diagram of a rotational imaging / LIDAR system according to one or more embodiments.

[0090] [Figure 33A] 1 illustrates an example of a hyperspectral imaging operation using a sensor array, according to one or more embodiments. [Figure 33B] 1 illustrates an example of a hyperspectral imaging operation using a sensor array, according to one or more embodiments.

[0091] [Figure 34] 1 shows a simplified front view of a sensor array according to some embodiments.

[0092] [Figure 35] 1 illustrates a set of four ambient light sensor channels with spatially encoded sub-pixel apertures according to some embodiments.

[0093] [Figure 36] 1 shows a simplified schematic diagram of a read datapath with multiple accumulation registers according to some embodiments.

[0094] [Figure 37] 1 illustrates ambient light measurement using multiple integrating registers, according to some embodiments.

[0095] [Figure 38A] 1 illustrates a set of ambient light sensor channels that provide spatially encoded sub-pixel apertures, according to some embodiments.

[0096] [Figure 38B] 38B illustrates the effect of temporal segmentation using the ambient light sensor channel of FIG. 38A.

[0097] [Figure 39] 1 illustrates an example of a static imaging / LIDAR system, according to one or more embodiments.

[0098] [Figure 40] 1 illustrates an exemplary automotive application of a static imaging / LIDAR system, according to one or more embodiments.

[0099] [Figure 41]1 illustrates another exemplary automotive application of a static imaging / LIDAR system, according to one or more embodiments.

[0100] [Figure 42] 1 illustrates an example of a still imaging / LIDAR system with an extended field of view, according to one or more embodiments.

[0101] [Figure 43] 1 illustrates a block diagram of a static imaging / LIDAR system according to one or more embodiments.

[0102] [Figure 44] 1 illustrates an example of a hyperspectral image that can be acquired using a multispectral ranging / imaging sensor array, according to one or more embodiments.

[0103] [Figure 45] 1 shows an example of an image that has been annotated to identify the materials it contains. DETAILED DESCRIPTION OF THE INVENTION

[0104] 1. Multispectral Sensor Array As used herein, a multispectral sensor array refers to an array of sensors, each configured to image a portion of a field of view (a pixel) at a different wavelength. Data from different sensors imaging the same pixel can be combined to provide a multispectral pixel in an image. Examples of multispectral sensor arrays are now described. These examples illustrate and embody various principles and concepts related to building multispectral sensor arrays. It will be apparent that many other implementations of multispectral sensor arrays are possible, and the examples provided are not intended to be limiting.

[0105] 1.1.Sensor Channel Example Examples of multispectral sensor arrays described herein include arrays constructed from sensor channels formed or disposed on a monolithic semiconductor device, such as an application-specific integrated circuit (ASIC). FIG. 1A shows a cross-section of a single micro-optical sensor channel 100 that can be used in some embodiments of a multispectral sensor array. The sensor channel 100 accepts an input cone of light potentially containing a wide range of wavelengths and filters out all but a selected subset of wavelengths (including a selected subset according to a particular channel), allowing the optical sensors 152 (sometimes referred to as “pixels”) to detect only, or substantially only, photons within the selected subset of wavelengths. Embodiments of the present invention are not limited to a particular configuration of sensor channels, and the sensor channel 100 is merely one example of a sensor channel that can be implemented in a sensor array 200.

[0106] In some embodiments, the sensor channel 100 includes an input aperture layer 110 that includes an optically transparent aperture 112 and an optically non-transparent aperture region 114. As used herein, the term "optically transparent" refers to a material that allows most or all incident light to pass through. As used herein, the term "optically non-transparent" refers to a material that allows little or no light to pass through, e.g., a reflective or absorbing surface. The aperture 112 is shaped and sized to define a narrow field of view when positioned at the focal plane of the imaging optics, examples of which are described below. The aperture layer 110 is configured to receive an input light cone, as illustrated by input marginal rays 120. In a multispectral sensor array, examples of which are described below, the aperture layer 110 may include an array of optically transparent apertures and optically non-transparent aperture regions constructed on a single monolithic piece, such as an optically transparent substrate. In some embodiments, the aperture layer 110 can be formed from an optically non-transparent material that forms the aperture region 114, and the aperture 112 can be a hole or opening in the layer 110.

[0107] In some embodiments, sensor channel 100 includes an optical lens layer 130 that includes a collimating lens 132 characterized by a focal length. Collimating lens 132 may be offset from the plane of aperture 112 and aperture region 114 by the focal length and axially aligned with aperture 112 (i.e., the optical axis of the collimating lens is aligned with the center of the aperture). In this manner, collimating lens 132 may be configured to collimate light rays passing through aperture 112 such that the light rays travel approximately parallel to the optical axis of collimating lens 132. Optical lens layer 130 may optionally include apertures, optically opaque regions, and tubular structures to reduce crosstalk between nearby sensor channels 100 in the sensor array.

[0108] In some embodiments, the sensor channel 100 includes an optical filter layer 140 that includes optical filters 142. In some embodiments, the optical filter layer 140 is disposed on the detector side (opposite the aperture side) of the optical lens layer 130. The optical filter layer 140 can be configured to pass normally incident photons at a particular operating wavelength and passband. The optical filter layer 140 can include any number of optical filters 142. The optical filter(s) in a particular instance of the sensor channel 100 can be selected based on the intended use of that particular instance of the sensor channel 100, for example, as described below. The optical filter layer 140 can optionally include apertures, optically opaque regions, and tubular structures to reduce crosstalk.

[0109] In some embodiments, the sensor channel 100 includes a photosensor layer 150 including one or more individual photosensors 152 disposed behind the optical filter layer 140. Each photosensor 152 may be a photosensor capable of detecting photons, with a detector active area consisting of, for example, one or more standard photodiodes, avalanche photodiodes (APDs), single-photon avalanche diodes (SPADs), resonant cavity photodiodes (RCPs), optical nanoantennas, microbolometers, or other suitable photodetectors. The photosensors 152 often have higher dynamic ranges, faster response times, or other beneficial properties compared to a single large photon detection area, and may be composed of several photon detector areas (e.g., each a different SPAD) working together to function as a single sensor. In addition to the photosensors 152 for any number of sensor channels, the photosensor layer 150 may include optional structures to improve detection efficiency and reduce crosstalk with adjacent sensor channels. The photosensor layer 150 may optionally include diffusers, converging lenses, apertures, optically non-transparent tubular spacer structures, optically non-transparent conical spacer structures, and the like.

[0110] Stray light can be caused by roughness of optical surfaces, imperfections in transparent media, back reflections, etc., and can be generated by many features within or external to the sensor channel 100. Stray light can be directed through the optical filter layer 140 along paths non-parallel to the optical axis of the collimating lens 132, reflect between the aperture layer 110 and the collimating lens 132, and generally take other paths or trajectories that may involve many reflections and refractions. When multiple receiver channels are arranged adjacent to each other, stray light in one receiver channel can be absorbed by the optical sensors in another channel, thereby contaminating the timing, phase, intensity, or other information about the received photons. Therefore, the sensor channel 100 may also feature structures to reduce crosstalk and increase the signal between the receiver channels. Examples of such structures and other suitable receiver channels are described in U.S. patent application Ser. No. 15 / 979,295, filed May 14, 2018, entitled "Micro-optics for Imaging Module with Multiple Converging Lenses per Channel," the disclosure of which is incorporated herein by reference in its entirety for all purposes.

[0111] The components and arrangement of the sensor channel 100 can be modified as needed. By way of example, FIG. 1B shows a cross section of a single micro-optical sensor channel 100′ that can be used in some embodiments of a multispectral sensor array. The micro-optical sensor channel 100′ is generally similar to the micro-optical sensor channel 100 of FIG. 1A, and components are correspondingly numbered. In this example, the lens element 132′ has a different configuration than the lens 132 shown in FIG. 1A, with its planar surface oriented toward the aperture and its convex surface oriented toward the optical filter layer 140. Like the lens element 132, the lens element 132′ collimates incident light, as indicated by the marginal rays 120, and directs the collimated light toward the optical filter layer 140. It should be understood that other modifications are possible. For example, the optical lens layer 130 can include a light guide in addition to or instead of the lens element, an optical filter can be disposed on the aperture side of the optical lens layer 130, etc. As another example, a sensor channel may include any micro-optical element and may be as simple as a light sensor (or group of light sensors) with an optical filter disposed thereon. In some cases, the optical filter may be fabricated within the metal layer of the light sensor (e.g., in the case of the polarization channel described below). Additional examples of alternative sensor channel configurations are provided below. It should also be understood that different sensor channels within the same sensor array may have different configurations.

[0112] 1.2. Exemplary Multispectral Sensor Array In some embodiments, a multispectral sensor array incorporates a group of aligned sensor channels fabricated on a common substrate. Sensor channels (also referred to herein as "sensor types") tuned to different wavelengths or wavelength ranges can be located at different locations on the substrate, with the locations selected so that a given portion of the field of view can be seen by the different sensor channels, either simultaneously or at different times. Many specific arrangements are possible, and examples are described below.

[0113] 1.2.1. Column-Based Multispectral Sensor Array 2 shows a simplified front view of a sensor array 200 according to an embodiment of the present invention. The sensor array 200 can include a number of LIDAR sensor channels 202; this example shows 16 LIDAR sensor channels 202, but can include any number of LIDAR sensor channels 202. In this example, the LIDAR sensor channels 202 are arranged in a staggered pattern; however, this is not required, and in some embodiments, the LIDAR sensor channels 202 can be arranged in a single column (in this example, the column runs parallel to the z-axis shown on the left side of FIG. 2).

[0114] In this example, each LIDAR sensor 202 is associated with a “row” 204 ​​of sensor array 200. (The term “row” here is used to indicate a linear or near-linear arrangement of elements; rows in FIG. 2 are indicated by dashed lines.) In addition to LIDAR sensors 202, each row of sensor array 200 includes one or more ambient light sensor channels 206. In this example, ambient light sensor channel 206R detects red light, ambient light sensor channel 206G detects green light, and ambient light sensor channel 206B detects blue light; however, any number and combination of ambient light sensor channels can be used. Additional examples are described below. Each row can include a complete set of sensors for generating a multispectral pixel; sensor arrays such as sensor array 200 are referred to herein as “row-based” or “1D” sensor arrays.

[0115] 3 shows a simplified cross-sectional side view of a row 204 of the sensor array 200 of FIG. 2. Each sensor channel 206R / G / B, 202 of the sensor array 200 can be implemented as a separate instance of the sensor channel 100 described above. In some embodiments, the different sensor channels 206R / G / B, 202 differ in having different optical filters. For example, the optical filter 342L of a LIDAR sensor channel 202 can include, for example, a Bragg reflector-type filter for passing light at a LIDAR signal wavelength with a narrow passband. The optical filter of a given ambient light sensor channel can include a bandpass filter that passes light within a given region of the spectrum and blocks light outside the bandpass region. For example, in the case of the red light sensor channel 206R, the optical filter 342R can pass light having wavelengths in the red region of the spectrum (e.g., wavelengths of approximately 620 nm to approximately 750 nm), in the case of the green light sensor channel 206G, the optical filter 342G can pass light having wavelengths in the green region (e.g., wavelengths of approximately 495 nm to approximately 570 nm), and in the case of the blue light sensor channel 206B, the optical filter 342B can pass light having wavelengths in the blue region (e.g., wavelengths of approximately 450 nm to approximately 495 nm). Those skilled in the art will understand that specific bandpass filters of a given color can be selected as needed, and that different embodiments can include sensor channels "tuned" to any desired range of optical wavelengths (by application of appropriate optical filters), including non-visible light wavelengths such as ultraviolet, near-infrared (NIR), short-wave infrared (SWIR), mid-wave infrared (MWIR), or long-wave infrared (LWIR, i.e., thermal imaging), and that the different wavelength ranges associated with different types of sensor channels within a given sensor array may or may not overlap. Optical systems of the type described herein can operate over a wavelength range spanning 300 nm to 20 μm, provided that the optical elements are selected to function at the operating wavelengths and the optical sensors are capable of sensing electromagnetic energy at those same wavelengths. Suitable materials and sensors for all wavelengths in this range are known in the art, and the same optical principles (ray optics, refraction, etc.) apply.Other ambient light sensor channels can be tuned to detect other characteristics of light, examples of which are described below.

[0116] 1A (or FIG. 1B), where a single input aperture layer 310 can provide apertures 312R / G / B, 312L for each sensor channel 206R / G / B, 206R of the sensor array 200 such that these apertures lie in the same plane. In some embodiments, the aperture layer 310 can have a thickness d, and the apertures 312R / G / B / L can be formed with tapered openings at an exit surface 360 ​​of the aperture layer 310 such that the exit aperture width can be wider than the aperture, e.g., the same width as the respective sensor channel (shown as 362R / G / B / L). Alternatively, the taper direction can be reversed, such that the apertures are widest on the input side and narrow toward the sensor channel. The apertures can follow the ray cones defined by the marginal rays of each channel, thereby defining a numerical aperture for the channel that matches the numerical aperture of the channel optics 332 and the bulk optics that direct the light onto the sensor array (examples of which are described below). The particular thickness and structure of the aperture layer 310 can be varied as desired.

[0117] In some embodiments, the per-channel compensation micro-optical elements 370R, 370G, and 370B can be positioned directly in front of the input apertures 312R / G / B. As described below, such per-channel micro-optical elements can provide improved light collection efficiency, for example, by compensating for chromatic aberrations of the bulk optics of the system.

[0118] In some embodiments, the sensor array 200 can be fabricated as part of a monolithic device on a single substrate, for example, using CMOS technology. The monolithic device can include an array of photosensors 152 along with a processor and memory (not shown in FIGS. 2-3 ) for processing raw signals from individual photosensors 152 (or groups of photosensors 152) in the sensor array 200. The monolithic device including the sensor array 200, processor, and memory can be fabricated as a dedicated ASIC. In some embodiments, the sensor array 200 can be fabricated using 3D stacking technology and can include two or more monolithic devices, each fabricated on a single substrate and stacked with electrical connections running between them. The top monolithic device includes the array of photosensors 152 and can be tuned for optimal light sensing, while the underlying substrate can include the processor and memory and be optimized for digital logic. In some embodiments, sensor array 200 can be divided into multiple monolithic devices, each optimized for sensing a different wavelength (or multiple different wavelengths) of light, or optimized for depth sensing versus ambient light imaging, and the monolithic devices can be arranged side by side and associated with different channels of the sensor array shown in FIG. 3 . In some embodiments, sensor array 200 can also include micro-optical components (e.g., micro-optical elements 332R / G / B / L and / or per-channel compensation micro-optical elements 370R / G / B) as part of the monolithic structure. In such cases, the micro-optical components can be formed on the same ASIC that comprises sensor array 200, or can be fabricated on a separate wafer substrate and bonded to the sensor array ASIC at the wafer level so that they are part of a monolithic structure with a separate substrate layer for each layer of sensor channels. For example, compensation micro-optical layers, aperture layers, collimating lens layers, optical filter layers, and photodetector layers can be stacked and bonded to multiple ASICs at the wafer level before dicing. The aperture layer can be formed by placing a non-transparent substrate on a transparent substrate or by coating a transparent substrate with an opaque film.In such an embodiment, the dicing step forms multiple ASICs, each with its own unique micro-optical element structure bonded directly to it. As another example, the micro-optical element components may be formed as separate monolithic structures that can be bonded directly to the ASICs after the ASICs are separated from the larger wafer by a dicing process. In this manner, the ASICs and micro-optical element structures can be bonded together to form a single monolithic structure. In yet other embodiments, one or more components of the sensor array 200 may be external to the monolithic structure. For example, the aperture layer 310 may be implemented as a separate metal sheet with pinholes.

[0119] In the above example, three ambient light channels (tuned to red, green, and blue light, respectively) are provided. This is for ease of explanation, and embodiments of the present invention are not limited to a particular number or combination of ambient light channels. In some embodiments, a sensor row can have fewer than three ambient light channels; for example, a sensor row can have one ambient light channel with an optical filter that passes "white light" (e.g., including the entire visible light spectrum) or no optical filter (in which case the spectral selectivity is determined by the sensitivity of the light sensor). In other embodiments, a sensor row can have more than three ambient light channels. By way of illustration, FIG. 4 shows a simplified top view of a sensor array 400 according to an embodiment of the present invention, with several ambient light channels in each row to provide additional multispectral imaging capabilities. Again, the number and combination of sensor channels is for illustration purposes.

[0120] The sensor array 400 includes 64 LIDAR sensor channels 402. In this example, the LIDAR sensor channels 402 are arranged in a staggered grid, although this arrangement is not required. 32 of the 64 LIDAR sensor channels have an associated row 404 of ambient light sensors 406, although in other embodiments, every LIDAR sensor channel 404 can have an associated row of ambient light sensors 406. In this example, each row of ambient light sensors 406 includes eight spectral color channels 410, each defined by a different bandpass filter, two IR-band color channels 412, four polarization channels 414, and two ultra-narrow absorption band channels. Each channel can have an internal structure as described above with reference to FIG. 1A or 1B, and the sensor array 400 can be fabricated using these or other techniques.

[0121] The spectral color channels 410 can be created by using appropriate bandpass filters as the optical filters 142. In addition to red, green, and blue channels, the spectral color channels 410 in this example include channels tuned to wavelength ranges corresponding to orange, yellow, cyan, indigo, and violet. Other example spectral channels may include infrared, ultraviolet, and / or white (e.g., broad spectrum) channels, as well as channels tuned to any portion of the visible, infrared, or ultraviolet spectrum. In some embodiments, each spectral color channel 410 may have a compensating per-channel micro-optical element (similar to micro-optical elements 370R / G / B in FIG. 3 ), whose optical characteristics are based, at least in part, on the wavelength range to which the channel is tuned; examples of per-channel micro-optical elements are described below.

[0122] The IR band color channel 412 may be an additional spectral color channel with a bandpass filter tuned to the infrared portion of the spectrum. In some embodiments, it may be desirable to avoid LIDAR operating frequencies so that stray LIDAR radiation is not confused with ambient IR. In some embodiments, each IR band color channel 412 may have per-channel compensation micro-optics whose optical characteristics are based at least in part on the wavelength range to which the channel is tuned.

[0123] The polarization channels 414 can be created by using optical polarization filters, such as diffraction gratings, instead of or in addition to the optical bandpass filters 142. The polarization filters in each channel 414 of a group can be tuned to different angles relative to linear polarization by orienting the polarization filters of different channels at different angles. In one embodiment, the four polarization channels 414 have orientations of 0, 90, 45, and 135 degrees, respectively. The polarization filters can also be tuned to other forms of polarization, such as circular and / or spiral polarization. Polarization filters can be applied to different surfaces of the micro-optical sensor channel 200 in a manner similar to bandpass filters, or they can be fabricated as metal gratings directly within the metal layer of the optical sensor(s) 152. In some embodiments, each polarization channel 414 can have channel-specific compensation micro-optical elements. In some cases, for example, when the polarization channels 414 are not limited to a specific wavelength band, the compensation micro-optical elements may be omitted or tuned to a central wavelength within the band.

[0124] Each absorption band channel 416 can be defined by a narrowband optical filter corresponding to an absorption band characteristic of a particular substance of interest. In this case, the absence of a signal in an absorption band channel can be interpreted (e.g., in combination with information from other spectral color channels) as indicating the presence of a substance that absorbs light in that band. For example, in some applications, it may be useful to distinguish leaves (e.g., trees, grass, other plants) from other categories of objects (e.g., cars, buildings). Chlorophyll, commonly associated with leaves, has multiple narrow absorption bands in the IR spectrum, and absorption band channels can be tuned to some or all of these bands. As another example, many gases have absorption bands in the short-, mid-, and long-wave IR regions, and absorption band channels can be tuned to these bands to identify gaseous air pollutants. The system also provides distance to the object; in the case of gas absorption detection, this distance information can be used to calculate the distance in the atmosphere at which the absorption measurement was made, which can help determine the confidence level of the detection and / or the concentration of the pollutant. Like the other channels, in some embodiments, each absorption band channel 416 can have a channel-specific compensation micro-optical element whose optical characteristics are based at least in part on the band to which the channel is tuned.

[0125] These examples of ambient light sensor channels are illustrative and may be modified. The modifier ambient light applied to a sensor or sensor channel should be understood to generally refer to a sensor that operates to measure the amount (intensity) of incident light having the characteristic(s) (e.g., wavelength range and / or polarization) for which the channel is modulated. Ambient light sensor channels do not rely on intentional illumination of the field of view (in contrast to LIDAR sensor channels, which are designed to detect intentionally emitted light); however, intentional illumination (such as the use of automobile headlights or camera flashes) is not excluded.

[0126] In addition to a LIDAR sensor channel (or multiple LIDAR sensor channels each operating at a different wavelength), a row of the sensor array can include any number and combination of ambient light sensor channels, including one or more visible light sensor channels tuned to any desired color or range of colors, one or more polarization sensor channels, one or more infrared sensor channels, one or more ultraviolet sensor channels, one or more absorption band sensor channels, etc. As another example, the ambient light sensor channels in a given row may include two or more sensor channels tuned to the same wavelength range but with different attenuation filters, allowing for a higher dynamic range in the image.

[0127] Furthermore, it is not necessary for every LIDAR sensor channel in the sensor array to have an associated row of ambient light sensor channels, or for every row of ambient light sensor channels to have an associated LIDAR sensor channel. As described below, arranging a set of LIDAR and ambient light sensor channels in a single row facilitates registration between images and depth data captured at different wavelengths during a scanning operation, but interpolation can be used to generate multispectral image pixels as long as the offsets between the different sensor channels are fixed and known.

[0128] 1.2.2. Sensor Array with Multispectral Sensor Channels In the above embodiments, each sensor type of the multispectral pixel is provided as a separate sensor channel. It is also possible to combine multiple sensor types in a single sensor channel. For example, a LIDAR sensor channel may use multiple SPADs as light sensors, with depth measurements based on the number of SPADs triggered in a given time interval. An ambient light channel may use a single SPAD or standard photodiode that occupies a smaller area of ​​the semiconductor device. Thus, some embodiments may include one or more "multispectral" sensor channels in a row of sensors.

[0129] 5 illustrates a simplified top view of a sensor array 500 incorporating multispectral sensor channels in accordance with an embodiment of the present invention. The sensor array 500 includes 64 LIDAR sensor channels 502. In this example, the LIDAR sensor channels 502 are arranged in a staggered grid, although this arrangement is not required. 32 of the 64 LIDAR sensor channels have an associated multispectral sensor channel 506, although in other embodiments, every LIDAR sensor channel 504 can have an associated multispectral sensor channel 506. In this example, as seen in inset 510, the multispectral sensor channel 506 can incorporate a red sensor 512, a green sensor 514, a blue sensor 516, polarization sensors 518, 520, and 522, and an IR-band color channel 524.

[0130] In some embodiments, each multispectral sensor channel 506 can be implemented as a single instance of a sensor channel, as described above with reference to FIG. 1A or 1B. The photosensor layer 150 can include a different photosensor 152 for each type of light to be detected. In this context, each photosensor 152 can be, for example, a standard photodiode with an amplifier, coupled to a capacitive charge bucket, and read using an analog-to-digital converter (ADC). Alternatively, each photosensor 152 can be one or more SPADs with an analog front end and an integrating register for counting photons. One or more patterned optical filters can be used in the optical filter layer 140 to direct light with desired characteristics to a particular photosensor 152. Each photosensor 152 can be separately readout (using appropriate electronics), thereby providing multiple outputs. As used herein, the term "multispectral sensor channel" refers to a configuration in which a single optical channel provides separate data outputs for different photosensors disposed therein, each of which can be conditioned (e.g., via optical filters) to detect light with different characteristics. As shown, the use of multispectral sensor channels can reduce the area consumed by a given set of sensor types.

[0131] It should be understood that the specific number and combination of sensor types included in a multispectral sensor channel may differ from those shown. For example, a multispectral sensor channel may include any or all of the above ambient light sensor types, including visible, ultraviolet, infrared, polarized light, broadband, and / or narrowband sensors. In some embodiments, a row in a sensor array such as sensor array 500 may include any number of multispectral sensor channels, each incorporating a different combination of sensor types, in addition to one or more LIDAR sensor channels (operating at different wavelengths). A row in a sensor array such as sensor array 500 may also include one or more “single-type” ambient light sensor channels (such as any of the sensor channels shown in FIG. 4 ) in combination with one or more multispectral sensor channels.

[0132] 1.2.3. Sensor Array with Hybrid Sensor Channels 5, the LIDAR (ranging) sensor channels are separate from the multispectral sensor channels (which measure ambient light). In other embodiments, the sensor array may include channels that incorporate both ranging (e.g., LIDAR) sensors and one or more ambient light sensors. Such channels are referred to herein as "hybrid sensor channels" or "hybrid sensor pixels."

[0133] 6 shows a simplified top view of a sensor array 600 incorporating hybrid sensor channels, in accordance with an embodiment of the present invention. The sensor array 600 includes 128 hybrid sensor channels 602 arranged in a rectilinear grid. It should be understood that the number and arrangement of sensor channels can vary.

[0134] As shown in inset 610, each hybrid sensor channel 602 can include a set of LIDAR light sensor elements 650 and several ambient light sensors tuned (e.g., using optical filters) to detect light having specific characteristics. In this example, the ambient light sensors include a red sensor 612, an orange sensor 614, a yellow sensor 616, a green sensor 618, an ultraviolet color sensor 620, a cyan color sensor 622, a violet color sensor 624, a blue sensor 626, polarization sensors 628, 630, 632, and an IR-band color sensor 634. In the example shown, the LIDAR light sensor elements 650 occupy a central region within the channel region of the hybrid sensor channel 602, while the ambient light sensors are disposed in peripheral regions of the channel region surrounding the central region. Other configurations are possible.

[0135] In some embodiments, similar to the multispectral sensor channel 506, each hybrid sensor channel 602 can be implemented as a single instance of a sensor channel, as described above with reference to FIG. 1A or 1B. The optical sensor layer 150 can include a different optical sensor 152 (or group of optical sensors 152) for each type of light to be detected. One or more patterned optical filters can be used in the optical filter layer 140 to direct light having desired characteristics onto a particular optical sensor 152 (or group of optical sensors 152). Each optical sensor 152 (or group of optical sensors 152) can be read separately using appropriate electronics, thereby providing multiple outputs.

[0136] A "hybrid sensor channel" can be understood as a special case of a multispectral sensor channel that includes a light sensor and associated readout circuitry configured to determine the time-of-flight of emitted / reflected light, as well as another light sensor and associated readout circuitry configured to measure light intensity. FIG. 7 shows a simplified schematic diagram of a hybrid sensor channel 602, illustrating the associated readout circuitry. In this example, each ambient light sensor 612-634 is implemented using a standard photodiode with an amplifier coupled to a capacitive charge bucket 712-734. The capacitive charge buckets 712-734 are each connected to a multichannel counter circuit 750, which can determine the number of photons detected by each ambient light sensor 612-634 (e.g., during a shutter interval).

[0137] The LIDAR light sensor element 650 in this example can be implemented using a SPAD connected to a timing circuit 760, which can time the arrival of photons and store the arrival times in a memory bank of photons over time, thereby enabling depth measurements.

[0138] As mentioned above, each hybrid sensor channel 602 can be implemented as a single instance of sensor channel 100 of FIG. 1A (or sensor channel 100′ of FIG. 1B). FIG. 8 is a simplified schematic side view of a portion of sensor array 600 showing the channel structure. Each hybrid sensor channel 602 has an aperture 812 (in aperture layer 810), an optical layer 830, a filter layer 840, and a photosensor layer 850 (shown in perspective for clarity of illustration). Filter layer 840 can include a patterned filter 842 (shown in perspective for clarity of illustration), which can be located on a filter wafer or deposited directly on the photosensor ASIC over appropriate photosensor elements.

[0139] In operation, light 860 is directed into aperture 812 and propagates through channel 602, as indicated by arrow 862. Patterned filters 842 direct light with desired characteristics onto individual photosensors 852 in photosensor layer 850. As described above, appropriate readout electronics can be used to extract time-of-arrival information from the LIDAR photosensors, as well as color, polarization, and / or cumulative photon counts for other ambient light sensors.

[0140] It should be understood that the specific number and combination of sensor types included in a hybrid sensor channel may differ from those shown. For example, a hybrid sensor channel may include any or all of the above ambient light sensor types, including visible, ultraviolet, infrared, polarized, broadband, and / or narrowband sensors, in addition to a LIDAR sensor. Furthermore, while sensor array 600 is shown as a 2D array of identical sensor channels 602, this is not required. A hybrid sensor channel may be included in a 1D array or in a row with other sensor types, similar to sensor array 400 of FIG. 4 or sensor array 500 of FIG. 5. The arrangement and configuration of the sensor channels may vary as desired.

[0141] 1.2.4. Dual-Planar Multispectral Sensor Array In the above embodiment, it is assumed that the light sensors of the various channel types are arranged in one plane. In other embodiments, the different light sensors may be in different planes.

[0142] 9 shows a simplified cross-sectional side view of a portion of another embodiment of a sensor array 900. The sensor array 900 includes one or more LIDAR channels 902, each of which may be a separate instance of the sensor channel 100 of FIG. 1A (or the sensor channel 100′ of FIG. 1B). The LIDAR channels 902 are fabricated on an ASIC 904, which includes one or more light sensors 906 for each LIDAR channel 902. An aperture layer 910 overlies the LIDAR channels 902 and has apertures 912 formed therein for directing light to each LIDAR channel 902. In these respects, the sensor array 900 may be similar to the other embodiments described above.

[0143] In this example, aperture layer 910 is a second ASIC having photosensors 916R, 916G, and 916B fabricated or otherwise disposed in or on its upper surface in a location that does not obstruct aperture 912. Photosensors 916R / G / B are located in the same plane as aperture 912, which may be the focal plane of the bulk imaging optics of the sensor array. Color filters 918R, 918G, and 918B, each of which may be bandpass filters that accept light within a different wavelength band (red, green, and blue in this example), are disposed above photosensors 916R, 916B, and 916G. This arrangement provides ambient light sensor channels 920R, 920G, and 920B. Aperture layer 910 may be electrically connected to readout and / or control circuitry (e.g., processor and memory) located in ASIC 904, as indicated schematically by wirebonds 922. (Wire bonding is not required and can be replaced by other techniques for establishing electrical connections between the ASICs, or the two ASICs can each be connected to readout and control circuits located on separate devices.)

[0144] FIG. 10 shows a simplified top view of a sensor array 900. The sensor array 900 provides a 2D array of multispectral pixels 1020. The size and dimensions of the sensor array 900 can be varied as needed. As shown in inset 1010, each multispectral pixel 1020 can include a LIDAR sensor channel 902 and one or more ambient light sensor channels 920. The ambient light sensor channels 920 can be fabricated in an ASIC that covers and provides an aperture for the LIDAR sensor channel 902 (as shown in FIG. 9). Any number and combination of ambient light sensor channels 920 can be provided, including any of the specific channel types described above (e.g., color channels including infrared, visible, and / or ultraviolet channels, polarization channels, narrowband absorption channels, etc.).

[0145] In some embodiments, the aperture layer ASIC 910 can have a significantly higher density of light sensors (or channels) 920 than the "base" ASIC 904 that supports the LIDAR sensor channels 902. For example, the LIDAR sensor channels have spacing of 100-400 μm and apertures with a diameter of 30 μm. The sensor channels (light sensors or pixels) in the aperture layer ASIC 910 can be significantly smaller (e.g., in the size range of 1-10 μm), which means that each hybrid pixel 1020 can include a large number of ambient light pixels. This may allow for an increased number of sensor types per multispectral pixel and / or multiple multispectral pixels with higher resolution for the ambient light imaging channels than for the LIDAR channels.

[0146] Multispectral images obtained using the aperture layer ASIC 910 may contain gaps corresponding to the locations of the apertures 912 or LIDAR channels 902. In some embodiments, interpolation may be used to fill the gaps.

[0147] 1.2.5. Multispectral Pixels with Encoded Spectrally Selective Passbands In the above example, different ambient light sensor channels may include optical filters with different passbands. In some embodiments, the passbands of different ambient light sensor channels may generally not overlap, and thus different ambient light sensor channels sample different portions of the light spectrum (including infrared, visible, and / or ultraviolet). FIG. 11 shows an example of non-overlapping passbands of three filters that may be used to provide ambient light intensity measurements in some embodiments, such as the multispectral sensor array of FIG. 2. In this example, the "blue" (B) filter 1102 has a passband from about 425 nm to about 515 nm, the "green" (G) filter 1104 has a passband from about 515 nm to about 610 nm, and the "red" (R) filter 1106 has a passband from about 610 nm to about 700 nm. It should be understood that these ranges and boundaries are exemplary and may vary. In some embodiments, the passbands of different filters may have some overlap. For example, B filter 1102 may have a passband from about 410 nm to about 510 nm, while G filter 1104 has a passband from about 490 nm to about 620 nm, and R filter 1106 has a passband from about 600 nm to about 700 nm. As another example, B filter 1102 may have a passband from about 410 nm to about 440 nm, while G filter 1104 has a passband from about 490 nm to about 620 nm, and R filter 1106 has a passband from about 600 nm to about 700 nm. Other variations are possible. The filter set shown in FIG. 11 can provide "R," "G," and "B" spectral intensity measurements for a multispectral pixel. (The names R, G, and B are used herein to suggest red, green, and blue, but the passbands of filters with these names need not correspond to the passbands associated with any particular color.)

[0148] In some embodiments, different ambient light sensor channels may have overlapping passbands selected to encode spectral information. FIG. 12 shows an example of the overlapping passbands of three filters that can be used to provide ambient light intensity measurements using encoded spectral information in some embodiments. In this example, a first filter 1202 has a "W" passband that encompasses nearly the entire visible light spectrum (wavelengths from about 425 nm to about 700 nm). A second filter 1204 has a "Cb" passband from about 425 nm to about 610 nm, and a third filter 1204 has a "Cr" passband from about 515 nm to about 700 nm. Intensity measurements from ambient light sensor channels with passbands shown in FIG. 12 can be used to extract R, G, and B spectral information corresponding to the spectral measurements from the filter set of FIG. 11. For example, if the intensity measurements from filters 1202, 1204, and 1206 are designated as W, Cb, and Cr, respectively, the intensities in the R, G, and B bands identified in FIG. 11 can be calculated as follows: R=W-Cb (1a) B=W-Cr (1b) G=W-(R+B)=Cb+Cr-W (1c) These calculations can be performed using arithmetic and logic circuits of conventional design that can be fabricated, for example, on the same ASIC as the sensor array.

[0149] In this way, either the non-overlapping filter set of Figure 11 or the spectrally encoded filter set of Figure 12 can provide equivalent spectral information. The encoding scheme of Figure 12 allows each channel to accept more light, which may improve measurement accuracy.

[0150] The filter set of FIG. 12 can be incorporated into various multispectral sensor arrays. FIG. 13 shows a simplified front view of a sensor array 1300 according to some embodiments. The sensor array 1300 can be similar to the sensor array 200 of FIG. 2 and can include LIDAR sensor channels 202 (as described above), each of which can be associated with a row 1304 including ambient light sensor channels 1306 a (W passband), 1306 b (Cb passband), and 1306 c (Cr passband), where the W, Cb, and Cr passbands are defined as shown in FIG. 12. Sensor data from the ambient light sensor channels 1306 a, 1306 b, 1306 c of a given row 1304 can be provided to an on-chip arithmetic logic circuit 1310 that implements equations (1a)-(1c) to generate R, G, and B output signals. It should be understood that sensor row 1304 may also include other types of ambient light sensor channels, for example, as described above with reference to FIG.

[0151] Spectrally encoded passbands can also be implemented in sensor arrays having multispectral or hybrid sensor channels. FIG. 14A shows a simplified front view of a multispectral sensor channel 1400 according to some embodiments. The multispectral sensor channel 1400 has a patterned optical filter including a region 1402 having a W passband (shown in FIG. 12), a region 1404 having a Cb passband, and a region 1406 having a Cr passband. In this example, the regions are square, but a particular filter geometry is not required. As described above with reference to FIG. 5, a separate optical sensor (e.g., one or more SPADs) can be positioned behind each region. While FIG. 14A shows three regions for each passband, it should be understood that any number of regions can be provided for a given passband (as long as a separate optical sensor is provided for each region).

[0152] All photosensors associated with the same passband can provide ambient light intensity measurements (e.g., in the form of electronic signals representing photon counts) to the same integration register. Thus, for example, register 1412 can accumulate (or integrate) photon counts from photosensors in region 1402, register 1414 can accumulate photon counts from photosensors in region 1404, and register 1416 can accumulate photon counts from photosensors in region 1406. Registers 1412, 1414, and 1416 can provide the accumulated photon counts as input to on-chip arithmetic logic circuit 1420, which implements equations (1a)-(1c) to generate R, G, and B output signals. It should be understood that multispectral sensor channel 1400 can also include other regions with different types of optical filters, for example, as described above with reference to FIG. 5. Furthermore, while FIG. 14A illustrates optical filters with the same passband occupying adjacent regions within a channel region, this is not required. For example, Figure 14B shows an alternative patterned optical filter 1400' in which regions 1402, 1404, 1406 having the same passband are distributed throughout the channel area, which may further improve measurement accuracy. As in Figure 14A, intensity measurements (e.g., photon counts) from different optical sensors associated with the same type of optical filter can be accumulated (or integrated) in the same integration register.

[0153] The foregoing examples of optical filters with spectrally encoded passbands and ambient light sensor channels incorporating such filters are illustrative and not limiting. Spectrally encoded passbands can be incorporated into any of the multispectral sensor arrays described above, including 1D arrays, 2D arrays, arrays with multispectral pixels, and arrays with hybrid pixels. While the examples herein use three passbands to encode three color channels, it will be understood that any number of different optical filters with overlapping passbands can be used to encode spectral information with any desired granularity. This encoding technique is not limited to the spectral characteristics of light. For example, a similar arrangement can be implemented using polarizing filters (e.g., in combination with non-polarizing filters) to encode polarization information with any desired granularity.

[0154] It should be understood that the above multispectral sensor arrays are exemplary and that many variations and modifications are possible. A given multispectral sensor array can include any combination of depth channels (e.g., LIDAR sensor channels or hybrid sensor channels), ambient light sensor channels, multispectral sensor channels, and / or hybrid sensor channels, which can be constructed using any of the techniques described above or other techniques. Components described with reference to one example or embodiment can be used in other embodiments.

[0155] 2. Optical elements for multispectral sensor arrays The various sensor arrays described above operate in response to light passing through an aperture associated with each channel. In some embodiments, optics are provided to focus the light onto the aperture plane. Examples of optics and optical elements that can be used in connection with a multispectral sensor array (e.g., sensor arrays 200, 400, 500, 600, and / or 900) are described below.

[0156] As used herein, the term bulk optic(s) refers to a single lens and / or lens assembly that has a focal plane and transmits light simultaneously from or to all micro-optical channels in an array. In some embodiments, bulk optics can have sizes (e.g., diameters) on the order of millimeters or centimeters or greater, such as those used in commercially available camera and microscope lenses. In this disclosure, the term bulk optic is contrasted with the term micro-optic, which refers to an optical element or array of optical elements provided for a specific sensor channel. In some embodiments, micro-optics can have individual element diameters that correspond to the size of a single sensor channel (e.g., on the order of a few micrometers to a few millimeters in size, or smaller). In general, micro-optics can modify light differently for different emitters and / or different detectors in an array of emitters or an array of sensor channels, while bulk optics modify light across the entire array.

[0157] 2.1.Bulk Optical Module A multispectral sensor array (such as any of the sensor arrays described above) can be incorporated into an optical ranging / imaging device 1500, as shown in FIG. 15. The optical ranging / imaging device 1500 includes an optical transmitting (Tx) module 1510 and an optical sensing (Rx) module 1540, which may include an implementation of the sensor array 200 (or any other sensor array described above). Additional examples of configurations of the optical transmitting module 1510 and the optical sensing module 1540 are described in U.S. patent application Ser. No. 15 / 979,235, filed May 14, 2018, entitled "Optical Imaging Transmitter with Brightness Enhancement," and U.S. patent application Ser. No. 15 / 979,266, filed May 14, 2018, entitled "Spinning LIDAR Unit with Micro-optics Aligned behind Stationary Window," the disclosures of each of which are incorporated herein by reference in their entirety for all purposes.

[0158] 15, the Tx module 1510 can include a Tx-side micro-optics package 1520 and a bulk optics 1530. The Tx-side micro-optics package 1520 includes several light emitters 1522, optionally including a microlens layer 1524 and an aperture layer 1526. The emitters 1522 can be arranged in a one-dimensional or two-dimensional array of transmitter channels, such as the channels 1525 shown in the boxed area. Each transmitter channel has one or more light emitters 1522, such as near-infrared (NIR) vertical cavity semiconductor lasers (VCSELs), capable of emitting narrowband light, and optionally a microlens from the lens layer 1524 and an aperture from the aperture layer 1526.

[0159] During operation, the Tx module 1510 provides active illumination of objects within an area surrounding the LIDAR system by transmitting pulses of narrowband light, e.g., NIR light having a spectral width of 10 nm, 2 nm, 1 nm, 0.5 nm, 0.25 nm, or smaller, within one or more fields of view. The Rx module 1540, in particular its LIDAR sensor channels 202, detects reflected portions of the transmitted narrowband light reflected by objects within the scene. At the same time, each ambient light sensing channel 206R / G / B of the Rx module 1540 can detect ambient light in its specific wavelength band.

[0160] Light emitted from each transmitter in the transmitter array diverges as it approaches one of the micro-optical elements in the Tx-side micro-optical element lens layer 1524. The micro-lenses from the micro-lens layer 1524 capture the diverging light and refocus it onto a focal plane that coincides with the apertures in the aperture layer 1526, which contains an array of apertures corresponding to the positions of the array of micro-optical elements and the array of emitters. The aperture array 1526 can reduce crosstalk in the system. After exiting the micro-lenses, the focused light diverges again into a cone and then impinges on the Tx-side bulk imaging optics module 1530. In some embodiments, the separation between the microlens layer 1524 and the Tx-side bulk imaging optics module 1530 is equal to the sum of their focal lengths, such that the light focused by the aperture array 1526 appears as collimated light at the output of the Tx-side bulk imaging optics module 1530, with each collimated ray bundle exiting the Tx-side bulk imaging optics module 1530 at a different chief ray angle. Thus, light from each emitter is directed toward a different field of view in front of the device. In some embodiments, the Tx-side bulk imaging optics 1530 is telecentric on the image side (which is the emitter side) of the lens, i.e., the chief rays on the image side of the bulk imaging optics 1530 are substantially parallel to each other and perpendicular to the image plane (emitter plane) for every position on the image plane. In this configuration, the emitter array advantageously operates as a telecentric light source, i.e., the optical element captures substantially all light generated by the emitter array, even light emitted from emitters on the outer edge of the array. Without a telecentric design, only the portion of the emitted ray cone that coincides with the lens's oblique ray cone would be captured by the lens, potentially reducing the light captured by the outer emitters. The LIDAR sensing channels 202 of the Rx module 1540 can be positioned to coincide with the Tx-side micro-optics package 1520, with a LIDAR sensor channel 202 corresponding to each micro-optics transmitter channel 1525.

[0161] The Rx module 1540 includes an Rx-side bulk imaging optics module 1560 and a sensor array 200. Portions of emitted light that reflect off objects in the field, shown as light rays 1505, enter the Rx-side bulk imaging optics module 1560 from multiple directions. The Rx-side bulk imaging optics module 1560 can include a single lens or multiple lens groups that focus the light rays 1505 into a plane coincident with the Rx-side input aperture layer 310, allowing the light to enter the LIDAR sensor channels 202. In some embodiments, the Rx module 1540 includes a LIDAR sensor channel for each emitter 1522, with the field of view of each individual LIDAR sensor channel 202 coinciding with the field of view of the respective emitter 1522.

[0162] The Rx-side bulk imaging optics module 1560 can also collect ambient light. As used herein, "ambient" light refers to any light ray that may be propagating in the environment and that did not originate from the Tx module 1510. Ambient light includes direct light from any light source that happens to be present in the environment (e.g., the sun, artificial lighting fixtures, traffic lights, etc.) and light reflected or scattered by objects in the environment (e.g., light reflected by road signs, vehicles, road surfaces, trees, etc.). Ambient light can propagate in any direction, and ambient light that happens to propagate in a similar direction to light ray 1505 can enter and pass through the Rx-side bulk imaging optics module 1560.

[0163] 2.2. Compensating Micro-Optical Elements for Each Channel In some embodiments, the Rx-side bulk imaging optics module 1560 can be designed as a monochromatic lens optimized to focus a specific narrow wavelength band, e.g., a LIDAR operating wavelength, onto a target plane, e.g., the input aperture plane 310. The Rx-side bulk imaging optics module 1560 can exhibit chromatic aberration (i.e., a wavelength-dependent focal length). This can reduce the collection efficiency of the ambient light sensor channels; if an implementation of the Rx-side bulk imaging optics module 1560 with chromatic aberration focuses light at the LIDAR operating wavelength onto the input aperture plane 310, light at wavelengths other than the LIDAR operating wavelength will not be focused onto the input aperture plane 310, and some of that light will be blocked by the aperture stop rather than entering the ambient light sensor channels 206R / G / B. Furthermore, the amount of light lost due to this effect is wavelength-dependent, which can complicate analysis of imaging data. Furthermore, the spatial resolution of these channels is reduced (the field of view becomes larger and less clearly defined, i.e., "blurred") because the aperture 310 is not in the focal plane of those wavelength bands or because the monochromatic lens is unable to provide a small focused spot for the out-of-band light.

[0164] Accordingly, some embodiments of sensor array 200 (or other multispectral sensor arrays described herein) include per-channel compensation micro-optical elements that can be placed in front of the input aperture plane to enable more efficient light capture. FIG. 16 is a simplified cross-sectional view of a portion of sensor array 200, with annotations to illustrate the behavior of incident light. (In this example, the aperture taper is reversed relative to FIG. 3, with apertures 312R / G / B / L shown at bottom surface 360 ​​of aperture layer 310. However, the same principles apply regardless of the exact location of the aperture plane.)

[0165] In the example of Figure 16, the dashed lines illustrate the effects of chromatic aberration. The converging dashed lines on channels 206R, 206G, and 206B indicate the respective marginal rays for red, green, and blue light rays focused by a bulk optic with chromatic aberration (e.g., Rx-side bulk imaging optics module 1560 of Figure 15). As shown, LIDAR light rays 1620L converge at aperture plane 360, whereas shorter wavelength light (visible light in this example) converges in front of aperture plane 360 ​​at a distance dependent on wavelength. Thus, in this example, red light focal point 1612R is slightly in front of input aperture 312R, green light focal point 1612G is further in front of input aperture 312G, and blue light focal point 1612B is even further in front of input aperture 312B. Without the corrective optics, the focused red, green, and blue light rays (dashed lines) diverge before reaching aperture plane 360, resulting in varying degrees of light loss at apertures 312R, 312G, and 312B.

[0166] In some embodiments, per-channel compensation micro-optical elements can be used to correct for such effects. For example, as shown in FIG. 16 , a first compensation micro-optical element, in this example a first plano-concave lens 1650R, is positioned in front of the aperture layer 310 and aligned with the aperture 362R of the red channel 206R. The plano-concave lens 1650R has an optical prescription (e.g., surface curvature or focal length) that reduces the divergence of incident light and shifts the focus of red light from its uncorrected focus 1612R to the aperture 312R. A second compensation micro-optical element, in this example a second plano-concave lens 1650G, is aligned with the aperture 362G. The plano-concave lens 1650G has a prescription that reduces the divergence of incident light more strongly than the plano-concave lens 1650R and shifts the focus of green light from its uncorrected focus 1612G to the aperture 312G. A third compensation micro-optical element, in this example a third plano-concave lens 1650B, is aligned with the aperture 362B. The plano-concave lens 1650B has a prescription that reduces the divergence of incident light more strongly than the plano-concave lens 1650G and shifts the focus of the blue light from the uncorrected focus 1612B to the aperture 312B. It should be understood that the plano-concave lenses 1650R, 1650G, 1650B in this example each have a different prescription that is optimized for the wavelength (or wavelength range) that the corresponding sensor channel 206R, 206G, 206B is tuned to detect. In this example, no compensating micro-optics are provided to the LIDAR channel 202 because the Rx-side bulk imaging module 1560 already focuses light at the LIDAR operating wavelength to the aperture 312L.

[0167] In other embodiments, the specific wavelengths at which the Rx-side bulk imaging module focuses light onto the input aperture plane may be different. By way of illustration, FIG. 17 shows an example of a per-channel compensation micro-optical element for an embodiment in which an Rx-side bulk imaging module 1560 with chromatic aberration focuses blue light onto the aperture plane 360. In this example, the blue channel 202B does not use a compensation micro-optical element; however, without the compensation micro-optical element (due to chromatic aberration), the focal points of desired light wavelengths for the other channels would be located beyond the aperture plane 360, again resulting in wavelength-dependent light loss and spatial selectivity. To compensate for this, per-channel compensation micro-optical elements—in this example, plano-convex lenses 1750R, 1750G, and 1750L—can be placed in front of the channel apertures of the red channel 206R, green channel 206G, and LIDAR channel 202. In this example, the plano-convex lens has a prescription that increases the divergence of the incident light, shifting the focal point toward aperture plane 360 ​​so that, for a given sensor channel, the focal point of light for the color that the sensor channel is tuned to coincides with aperture plane 360. As in the previous example, the per-channel compensation micro-optics for each channel have a different prescription that brings the focal point of that particular channel onto aperture plane 360.

[0168] In the examples of Figures 16 and 17, light is focused onto an aperture plane and then collimated by optical elements in the sensor channels (e.g., as shown in Figures 1A or 1B). Another option is to provide per-channel compensation micro-optics in the ambient light channels that collimate light at per-channel wavelengths. Figure 18 shows an example of a sensor array 1800 with collimating per-channel compensation micro-optics that can be used in some embodiments. In this example, substrate array 1800 is generally similar to substrate array 200, but the apertures 1812R, 1812G, 1812B in each ambient light channel are substantially the same width as the channel. (The aperture 1812L of the LIDAR channel 1802 can be narrower, for example, as shown.) In this configuration, at least for the ambient light channels 1806R, 1806G, 1806B, the optical element 132 (shown in FIG. 1A or 1B) can be omitted and is otherwise similar to the ambient light channels 206R, 206G, 206B described above. Note that in this arrangement, the ambient light channels 1806R, 1806G, 1806B can be smaller and more tightly packed than the LIDAR channel 1802. For the ambient light channels, a narrow channel width can provide spatial selectivity without requiring an aperture narrower than the channel width; however, the collimation angle increases, thereby increasing the lower limit on the width of the bandpass filter.

[0169] Similar to the example of FIG. 16, the Rx-side bulk imaging module 1560 focuses light at the LIDAR operating wavelength (ray 1822L) onto aperture 1812L. Light with shorter wavelengths is focused at different distances from the back surface 1814, as indicated by the dashed lines. Plano-convex lenses 1850R, 1850G, and 1850B reduce the divergence of red, green, and blue light, respectively, and collimate the light of the desired wavelengths as it enters the channels, as indicated by the colored lines. As in the previous example, the channel-specific correction micro-optics for the different color channels have different prescriptions that compensate for the wavelength-dependent focal length of the incident light.

[0170] These examples are illustrative and not limiting. The Rx-side bulk imaging module with chromatic aberration can be adapted to focus light of any desired wavelength onto the aperture plane, and channels sensitive to other wavelengths can have compensation micro-optics with wavelength-specific (or channel-specific) positive (focusing) or negative (defocusing) prescriptions placed in front of the aperture. For ease of assembly, the compensation micro-optics for all channels in the sensor array (or all channels containing such elements) can be placed on the same plane (e.g., above the aperture layer). The specific shape of the compensation micro-optics can vary; for example, the compensation micro-optics for a given channel can include a plano-convex lens, a plano-concave lens, a biconvex lens, a biconcave lens, a convex-concave lens, a freeform lens, or a combination of multiple lenses. Different shapes can be used for different channel types as needed. As the above examples illustrate, it is not necessary to provide correction micro-optics for all sensor channels in a given sensor array; in some embodiments, the Rx-side bulk imaging module can be designed so that light having a desired wavelength for one of the sensor channel types is focused onto the channel's aperture. However, a specific design of the Rx-side bulk imaging module is not required; in some embodiments, for example, if the system includes a window or housing with the optical power that needs to be corrected, it can have per-channel compensation micro-optics to compensate for any aberrations in the system. A sensor array can include multiple sensor channels associated with a given wavelength, for example, as described above. In some embodiments, different sensor channels of the same channel type (e.g., wavelength range) can be designed identically, so that the compensation micro-optics prescription only needs to be determined once per channel type. Alternatively, because different channels are at different locations relative to the Rx-side bulk imaging module and aberration effects (including chromatic aberration) within the optical module may depend on the distance from the optical axis of the module, it may be desirable to design the compensation micro-optics for each channel individually. In either case, conventional optical modeling techniques can be applied to the specific channel design and the specific design of the Rx-side bulk imaging module to determine the appropriate prescription for the compensation micro-optics for a given channel.

[0171] The per-channel compensation micro-optical elements can be fabricated from any material that is optically transparent at the relevant wavelength. Molding or other processes can be used to form the micro-optical elements. In some embodiments, the micro-optical elements for all channels of a sensor array can be fabricated as a single structure with surface features (e.g., areas of local convex or concave curvature) that define the per-channel micro-optical elements and assembled with other layers of a monolithic sensor array. Furthermore, the prescription of the per-channel micro-optical elements can be selected based on any optical property of the bulk optical element, not limited to chromatic aberration. An example of using per-channel micro-optical elements to compensate for the focal plane curvature of a bulk optical element is described below.

[0172] 2.3.Achromatic Bulk Optics In some embodiments, the compensation micro-optics for each channel may be omitted. For example, the bulk optics module may have negligible (or no) chromatic aberration, so that light of all relevant wavelengths is focused onto the same aperture plane. Achromatic bulk optics modules may be particularly useful for sensor arrays that include multispectral sensor channels (e.g., sensor array 500) and / or hybrid sensor channels (e.g., sensor array 600), as well as sensor arrays in which some of the optical sensors are located at the aperture plane (e.g., sensor array 900).

[0173] 19 shows an example of a sensor array 200 in a system with an achromatic bulk optics module that focuses all colors within the aperture plane 310. In this example, no corrective micro-optics per channel are used. For channels with wide passbands (e.g., channels 206R / G / B), the optical filters can be located anywhere within the channel. For polarization channels (not shown), one or more polarization gratings can be included in the stack (e.g., optical filter layer) or in the metal layer of the underlying ASIC optical sensor(s).

[0174] FIG. 20 shows another example of a sensor array 2000 in a system with an achromatic bulk optics module that focuses all colors within an aperture plane 2010. In this example, a LIDAR sensor channel 2002 and ambient light sensor channels 2004R, 2004G, and 2004B are fabricated as separate monolithic devices arranged side by side. Each sensor channel 2002, 2004R / G / B has an aperture 2012L, 2012R / G / B located in the same aperture plane 2010. The LIDAR sensor channel 2002 includes a collimating optic 2020. The ambient light sensor channels 2004R / G / B in this example do not include a collimating optic. Instead, non-refractive optical elements (e.g., light guides) can be used to direct light through the channels to the light sensors 2030R, 2030G, and 2030B. For wide passbands, color filters can be placed anywhere in the channels. Although shown for a sensor channel having a single sensor type, a channel configuration with a non-refractive optical element may also be useful for a multispectral sensor channel (e.g., multispectral sensor channel 506 in FIG. 5) or a hybrid sensor channel (e.g., hybrid sensor channel 602 in FIG. 6).

[0175] 2.4. Micro-optics to compensate for focal plane curvature The above examples assume that the bulk optics module focuses light (of a given wavelength) onto a (flat) image plane, regardless of where the light passes through the bulk optics module. In the examples shown above (e.g., Figures 19 and 20), the image plane coincides with the aperture plane.

[0176] In some embodiments, a bulk optics module can focus light of a given wavelength onto a curved surface (referred to as a "curved focal plane") rather than a flat surface. When this is the case, per-channel micro-optics similar to the example above can be used to compensate for the offset between the curved focal plane and the (flat) aperture plane at each aperture location. FIG. 21 shows an example of per-channel micro-optics for correcting the focal length of a bulk optics module that can be used in some embodiments. A sensor array 2100 has rows of sensor channels 2102 arranged in a plane. (A one-dimensional sensor array is shown for ease of illustration; it should be understood that the same principles apply to two-dimensional sensor arrays.) A planar aperture layer 2104 has apertures 2106 arranged such that each aperture 2106 passes light to a corresponding sensor channel 2102. In this example, a bulk optics module 2108 has a curved focal plane, represented by dotted line 2110. In front of each aperture 2106 is a per-channel micro-optics element 2112 that compensates for the curvature of the focal plane 2110. For example, each per-channel micro-optical element 2112 can have a prescription that compensates for the offset between the location of the corresponding aperture 2106 and the corresponding location on the curved focal surface 2110, such that light is focused at the corresponding aperture 2106 (rather than in front of or behind the aperture 2106). In this example, for most of the apertures 2106, the corresponding location on the curved focal surface 2110 is in front of the planar aperture layer 2104, and the corresponding per-channel micro-optical element 2112 has a positive focusing power. In this example, the magnitude of the focusing power of the different per-channel micro-optical elements 2112 increases with radial distance r from the optical axis 2114 of the bulk optics module 2108. In other examples (not shown), the curved focal surface 2110 can be behind the planar aperture layer 2104 at some or all of the aperture locations 2106, and any particular per-channel micro-optical element 2112 can have a positive or negative focusing power as desired.In some embodiments, the curved focal plane 2110 of the bulk optics module 2108 may coincide with the aperture plane of one or more sensor channels, and the channel-specific micro-optics 2112 of such sensor channels may be omitted or may be provided with micro-optics with zero focusing power.

[0177] FIG. 22 shows another example of a receive (Rx) module 2200 with per-channel micro-optics. The Rx module 2200 can be similar to the Rx module 1540 of FIG. 15 above and can include any combination of sensor channel types. For example, all channels 2202 can be LIDAR sensor channels, all channels can be ambient light sensor channels, all channels 2202 can be hybrid sensor channels, or there can be a combination of different sensor channel types. In this example, per-channel micro-optics 2204 are provided in front of the aperture plane 2206 to compensate for the curvature of the focal plane of the bulk optics module 2208. As in the example of FIG. 21, the prescription of the per-channel micro-optics 2204 can be a function of radial distance from the optical axis, corresponding to the curvature of the focal plane of the bulk optics module 2208. In this example, the channel-specific micro-optical elements 2204 have a positive focusing power that increases with radial distance from the optical axis; however, as noted above, some or all of the channel-specific micro-optical elements 2204 can have a negative or zero focusing power.

[0178] In some embodiments, per-channel micro-optics that compensate for focal length can be used in LIDAR transmitter arrays and sensor arrays. FIG. 23 shows an example of a transmit (Tx) module 2300 with per-channel micro-optics. The Tx module 2300 can be similar to the Tx module 1510 of FIG. 15 above and can include a 1D or 2D array of emitter channels 2302. In this example, per-channel micro-optics 2304 are provided to compensate for the curvature of the focal plane of the bulk optics module 2308. As in the examples of FIGS. 21 and 22, the prescription of the per-channel micro-optics 2304 can be a function of radial distance from the optical axis, corresponding to the curvature of the focal plane of the bulk optics module 2308.

[0179] These examples are illustrative and not limiting. For example, in the above example, the prescription (focusing power) of the per-channel micro-optical element is varied to compensate for the curvature of the focal plane of the bulk optical element. In other embodiments, similar per-channel compensation can be achieved by using per-channel micro-optical elements with the same prescription and a variable standoff distance between the per-channel micro-optical element and the aperture plane, where the standoff distance can be based on the radial distance from the optical axis. A combination of varying the prescription and standoff distance can also be used.

[0180] It should be understood that per-channel micro-optics that compensate for the focal plane curvature of a bulk optics module may be useful in situations other than multispectral sensor arrays. For example, LIDAR systems that do not include ambient light sensor channels may also benefit from the sharper imaging associated with correcting the focal plane curvature of a bulk optics module. Per-channel micro-optics may be incorporated into the transmitter module, the receiver module, or both, depending on the characteristics of the bulk optics provided in each module. Imaging systems that include only ambient light sensor channels may also benefit, and the presence of a transmitter module is not required. Because bulk lens systems without focal plane curvature are generally larger and more complex than bulk lens systems with focal plane curvature, the use of per-channel micro-optics to compensate for the focal plane curvature of a bulk optics module may enable a reduction in the cost and / or size of the bulk optics module.

[0181] In some embodiments of a multispectral sensor array (e.g., any of the examples above) or other system in which the bulk optics module exhibits chromatic aberration and focal plane curvature, the per-channel micro-optics for any given channel can be designed to compensate for both effects so that light of the desired wavelength for the given channel is focused onto the aperture plane. More generally, the per-channel micro-optics can have a prescription designed to compensate for any optical property (or optical characteristic) of the bulk optics module that has different effects on channels at different positions in the array.

[0182] 2.5. Uniform Sampling of Object Space Sensor arrays of the type described herein can be incorporated into a variety of ranging / imaging systems that produce images composed of multispectral image pixels, each containing data obtained from a different type of sensor. It is often desirable for such images to represent a uniform sampling of the field of view (also referred to as "object space") of the sensor system. Specifically, it is desirable to define a regular "grid" of sampled regions in object space (referred to herein as "object-space pixels") that can be arranged in rows and columns, and to design the sensor system and its operation to produce a grid of image pixels, each pixel corresponding to a single object-space pixel imaged by each sensor type in the sensor array. In some embodiments of ranging / imaging systems, bulk optics are designed to support this uniform sampling of object space.

[0183] 2.5.1. Optical Elements of Static Systems In some embodiments, multispectral sensor arrays of the above type may be used in “static” ranging / imaging systems. Such systems incorporate a 2D sensor array (e.g., sensor array 600 or sensor array 900 described above) and acquire images on the surface of the sensor array without moving the array, as described below. The image pixels of such systems may correspond to hybrid sensor channels (e.g., hybrid sensor channel 602) or multispectral pixels (e.g., multispectral pixel 1020). Such arrays can uniformly sample object space if there are no local distortions in the bulk imaging optics. In some embodiments, the use of a flat-field focal length distortion profile may be desirable, so that light is focused to the aperture plane across the entire array.

[0184] 2.5.2. Scanning System Optics In some embodiments, multispectral sensor arrays of the type described above are used in an angular scan or rotational mode, where different sensor channels in a row of the sensor array sequentially image specific regions in the field of view (i.e., sense photons from specific regions in the field of view). An example of a scanning operation is described below. For purposes of this description, it is assumed that during a scanning operation, the sensor system rotates about an axis that is perpendicular to the rows, and the sensor channels operate as the sensor system rotates at different angles. (It should also be understood that scanning behavior can be achieved without moving the sensor array, for example, by using MEMS mirrors to reflect light from different regions in object space onto the array at different times.) Because the sensor array and bulk optics module are held in a fixed relationship to each other within the sensor system, it is also assumed that a given sensor channel has a fixed spatial relationship to the optical axis of the bulk imaging optics and "sees" through the same portion of the bulk optics module, regardless of the system's orientation in space.

[0185] To simplify image analysis, it is generally desirable for a scanning sensor system to uniformly sample object space. In this context, a grid of object space pixels can be considered to be arranged with rows along the scanning direction and columns perpendicular to the scanning direction. In the scanning direction, it is desirable for different sensor channels within the same row (e.g., all sensor channels within the same row 204 of sensor array 202 in FIG. 2 ) to sample the same object space pixel (at somewhat different times) as the sensor array rotates. This can be achieved in part by aligning the sampling interval with the rotation of the sensor array, as described below. However, it is also important to avoid pointing errors due to differences in the location of different sensor channels relative to the optical axis of the bulk optics module. Therefore, in some embodiments, the bulk optics module used with the sensor array in a scanning sensor system is designed to provide uniform sampling in both the scanning and non-scanning directions.

[0186] Figures 24A and 24B are simplified conceptual diagrams illustrating the possibility of pointing error in a scanning system using a sensor array. Figure 24A shows a row of a sensor array 2400 having uniformly spaced sensor channels 2402a-2402d, which may correspond, for example, to the sensor channels in row 204 of sensor array 200 in Figure 2. Each sensor channel has a channel field of view through a bulk optic 2410, as indicated by the dashed lines. Uniformly spaced object space pixels, represented by ovals 2404a-2404d, are aligned with the channel fields of view of sensor channels 2402a-2402d. Figure 24B shows sensor array 2400 after sensor channel 2402a has been rotated at an angle such that it points approximately at object space pixel 2404b. Sensor channel 2402b points to the left of object space pixel 2404c, which points approximately to object space pixel 2404d.

[0187] As can be seen in Figure 24B, pointing errors are present. For example, the field of view of sensor channel 2402b is not pointing to object space pixel 2404c, and the field of view of sensor channel 2402c is not precisely aligned with object space pixel 2404d. The term "intra-pixel pointing error" is used herein to refer to differences in field of view between sensor channels nominally aimed at the same object space pixel. (These differences are "intra-pixel" with respect to the object space pixel.) In some embodiments, it is desirable to control in-pixel pointing error when collecting multispectral pixel data.

[0188] In addition to intra-pixel pointing error, a sensor system may have an “inter-pixel pointing error,” which refers to non-uniform spacing between object space pixels in the row (scanning) or column (non-scanning) directions. In a scanning sensor system, uniform pixel spacing in the scanning direction can be achieved by controlling the shutter spacing (e.g., as described below) relative to the rotation angle of the sensor system and by limiting the intra-pixel pointing error. In the non-scanning direction, it is desirable for the object space pixels along a column to be evenly spaced, such that columns in object space map to columns in image space. In this regard, it should also be noted that some sensor arrays (e.g., sensor array 200) may include a set of staggered sensor channels (e.g., LIDAR channels 202). In this case, a single column of object space pixels can be imaged by scanning the array and controlling the shutter spacing to create a columnar arrangement. For example, for sensor array 200, an image column can have 16 pixels even though the 16 sensor channels 202 are not aligned with the columns on sensor array 200.

[0189] In some embodiments, the desired imaging behavior is achieved by providing a bulk optics module with a focal length distortion profile in which the displacement of light rays is linear with respect to tangential changes in the ray's angle of incidence (θ). A lens (or lens system) with this type of focal length distortion profile is commonly referred to as an "F tan θ" lens (meaning that the displacement distance at the image plane is a linear function of tan θ) or a "flat-field" lens. For small angles θ, an F tan θ lens has the property that the displacement of light rays on the image plane (i.e., the sensor array) is approximately linear with respect to changes in the ray's angle of incidence (θ). In the scanning direction, this provides the desired behavior of reducing intra-pixel pointing error. In the non-scanning direction, this provides uniform sampling in object space of sensor rows spaced at a uniform pitch and allows for mapping of columns of object-space pixels to columns of image-space pixels even when the sensors are staggered.

[0190] FIG. 25 illustrates an example of an imaging system using an F tan θ bulk optics module. Image plane 2502 includes rows of sensors 2504a-g spaced apart by a uniform distance p (also referred to herein as "linear pitch"). Sensors 2504a-g may be, for example, rows (or portions of rows) of sensor channels in any of the multispectral sensor arrays described above, or other sensors that detect photons from a given direction. Bulk optics module 2506 is positioned a distance f above image plane 2502, where f is the focal length of bulk optics module 2506. In this example, bulk optics module 2506 is depicted as a single biconvex lens; however, it should be understood that other lenses or multi-lens systems may be used.

[0191] Bulk optics module 2506 can be designed to focus light from the field of view (or object space) onto image plane 2502. For example, rays 2520a-2520g represent the chief rays of sensors 2504a-2504g. (It should be understood that the actual path of the light through bulk optics module 2506 is not shown.)

[0192] Bulk optics module 2506 has an F tan θ focal length distortion profile. (Those skilled in the art will understand how to create a bulk optics module with this profile, and a detailed description will be omitted.) As a result, at least for small angles, a uniform change in the angle of incidence of a ray shifts the point where the refracted ray intersects the image plane by a uniform distance, regardless of the original angle of incidence. For example, for rays 2520a, 2520b, the difference in angles of incidence is α, and rays 2520a, 2520b are separated by a linear pitch p at the image plane. Rays 2520b, 2520c also have a difference in angles of incidence of α, and the corresponding refracted rays 2520b, 2520c are also separated by a linear pitch p at the image plane. Thus, if image plane 2502 and bulk optics module 2506 are rotated together by angle α, ray 2520a originating from point 2530a will (approximately) become the chief ray of sensor 2504b, while ray 2520b originating from point 2530b will (approximately) become the chief ray of sensor 2504c, and so on. The rotation angle α, corresponding to the linear pitch p at the image plane, is referred to herein as the "angular pitch" of the scanning system, and the value of α is determined based on the sensor pitch p and the characteristics of the bulk optics module. In a scanning ranging / imaging system, where the bulk optics module provides an angular pitch α such that scanning the system through angle α shifts the incident ray by one linear pitch unit p, different sensor channels in a row can image the same portion of the field of view by acquiring images in a sequence of time steps, with the sensor array rotating by the angular pitch α (or a smaller angle such that α is an integer multiple of the scan pitch) at each time step. Examples of this type of scanning operation are described in more detail below.

[0193] The use of an F tan θ lens can reduce in-pixel pointing error to a negligible level, where “negligible” can be quantified based on the size of the sensor channel’s field of view. FIG. 26 shows an example of in-pixel pointing error that can be quantified and constrained in some embodiments of a scanning system. Circle 2602 represents the nominal location of an object-space pixel, and point 2604 is the center of circle 2602. Circle 2612 (dashed line) represents the field of view sampled by a particular sensor channel when nominally pointed in the direction of circle 2602. As shown, center point 2614 of circle 2612 is offset from center 2604 of object-space pixel 2602 by an offset ε. This offset can be used to quantify the in-pixel pointing error. In some embodiments, if the offset ε for any given sensor channel in a sensor row is less than 50% of the diameter of the channel’s field of view, the in-pixel pointing error is considered negligible. In other embodiments, a stricter definition is used, for example, the intra-pixel pointing error is less than 10% of the diameter of the channel field of view. Whether a given sensor system meets this constraint can be determined, for example, by imaging a test pattern. Other definitions can also be used.

[0194] The bulk optics of a scanning sensor system can also have an F tan θ focal length distortion profile in the non-scanning direction. Thus, in the example shown in FIG. 25, sensors 2504a-g can also be understood as corresponding to columns (or portions of columns) of sensor channels in any of the multispectral sensor arrays described above. When some or all of the columns of sensor channels are staggered (e.g., LIDAR sensor channel 202 in FIG. 2), a bulk optics with an F tan θ focal length distortion profile in both directions can enable the staggered columns of sensor channels to sample uniformly spaced columns of object space pixels via a scanning operation, allowing different sensor channels in the same row to have negligible intra-pixel pointing error.

[0195] It should be noted that F tan θ bulk optics modules are useful in situations other than multispectral sensor arrays. For example, a scanning LIDAR sensor array may include an array of staggered LIDAR channels arranged in a column, which may be operated in a scan / rotation mode to image a field of view. Examples of such systems are described, for example, in U.S. Patent Application No. 15 / 685,384, filed August 24, 2017 (published as U.S. Patent Application Publication No. 2018 / 0059222), the disclosure of which is incorporated herein by reference in its entirety. F tan θ bulk optics modules can be used to provide vertical alignment (i.e., along the column in image space) of object space pixels imaged by sensor channels located in different columns of a staggered array and / or to provide uniform spacing of sampled locations along the column.

[0196] It should be understood that the bulk optics module of a sensor array (multispectral or LIDAR only) need not have an F tan θ focal length distortion profile or any other particular focal length distortion profile. For example, lenses used in some laser scanning systems have focal length distortion profiles where the displacement is a linear function of θ (rather than tan(θ)), and such lenses are sometimes referred to as “Fθ” lenses. For small angles of incidence θ, tan(θ) is approximately equal to θ, and Fθ lenses can provide approximately the desired behavior. Thus, in some embodiments, the bulk optics can have an Fθ focal length distortion profile. Furthermore, the focal length distortion profiles in the scan and non-scan directions need not be the same.

[0197] In some embodiments, non-uniformity in the size or location of the areas sampled by different sensors in the sensor array can be accounted for using image processing techniques. For example, image processing algorithms can interpret images with fisheye distortion, etc., as long as the distortion profile of the bulk optics is not affected by local deviations (e.g., high-frequency noise).

[0198] Alternatively, the sensor channels can be arranged in a non-uniform array rather than a linear array in a pattern that compensates for the distortion profile of the bulk optics so that uniform sampling of object space and consistent pointing behavior is achieved. For example, FIG. 27 shows a non-uniform array pattern 2750 that compensates for barrel distortion. The sensor channels can be arranged at vertices 2752, for example. (Some vertices 2752 are highlighted as red dots, but it should be understood that sensor channels can be arranged at any vertex 2752.) In this example, the spacing between adjacent sensor channels increases toward the center of the array. FIG. 28 shows a non-uniform array pattern 2860 that compensates for pincushion distortion. The sensor channels can be arranged at vertices 2862, for example. In this example, the spacing between adjacent sensor channels decreases toward the center of the array.

[0199] More generally, based on the design of a particular bulk optic, the distortion profile in the image plane can be mapped, and the sensor channels can be arranged non-uniformly so that the sampling density is uniform in object space. (Note that this technique can complicate the design and fabrication of the sensor array, which may require adapting the sensor array to the particular bulk optic.)

[0200] Additionally, in some embodiments, the shutter intervals can be controlled independently for different sensor channels, so that different sensor channels can begin and end data collection for a given pixel at different times. Individual shutter control can be used to compensate for in-pixel pointing errors of particular channels along the scan direction. (Note that this can complicate the design of the sensor electronics.)

[0201] 2.5.3. Optical Elements in Raster Scan Systems In some embodiments, multispectral sensor arrays of the type described above can be used in raster scan mode. In raster scan mode, a sensor array with a relatively small number of sensor channels can scan a field of view in two directions to generate an image with a greater number of pixels than the number of sensor channels. For convenience, the scan directions are referred to herein as "horizontal" and "vertical," however, those skilled in the art will understand that the spatial orientation of raster scanning is arbitrary. Raster scanning can be performed using a sensor array including a 2D array of hybrid sensor channels (e.g., sensor array 600) or multispectral pixels (e.g., sensor array 900), or a row-based scan sensor array (e.g., sensor array 200) that also scans in a column-by-column direction.

[0202] FIG. 29 illustrates an example of raster scanning using a sensor array, according to an embodiment of the present invention. The sensor array 2900 includes several sensor channels 2902 arranged in a regular sensor grid. In this example, the sensor grid is 3×3; however, the dimensions can be varied as needed. The sensor channels 2902 can include any of the sensor channel types described above. Arrow 2904 indicates the motion path of the sensor array 2900. As shown, the sensor array 2900 can move to the right along a horizon line through a series of imaging positions within a field of view 2920, including positions 2912 and 2914. At each imaging position, the sensor channels 2902 can be operated to capture an image. At the end of the horizon line (position 2914), the sensor array 2900 can shift down to position 2916, for example, by a pitch distance based on the number of rows in the sensor array 2900, to capture the next image. The sensor array 2900 can then be moved left to capture the next horizontal line of images. The captured images can be accumulated into a larger image that covers the entire field of view 2920.

[0203] The sensor array 2900 can be, for example, any of the multispectral sensor arrays described above. If the sensor array 2900 is a 2D array (e.g., the sensor array 600 or the sensor array 900), the distance the sensor array 2900 moves along a horizontal scan line between successive images can be based on the horizontal size of the array to provide uniform, non-overlapping samples as shown in FIG. 29. If the sensor array 2900 is a row-based array with rows oriented along horizontal scan lines, the distance between successive images along the horizontal scan lines can be equal to the channel pitch, allowing different sensors in the same row of the sensor array to image the same object space pixel. The vertical shift between scan lines can be determined based on the number of rows in the array.

[0204] The motion pattern for raster scanning may differ from that shown in FIG. 29. For example, a "horizontal retrace" pattern may be used, where at the end of a horizontal scan line, the sensor array 2900 returns to the left edge and shifts down to the next horizontal scan line, so that the image for each horizontal scan line is captured using the same direction of travel. As another example, for a row-based sensor array, the rows may be oriented vertically, with the vertical distance between horizontal scan lines equal to the channel pitch within the row. (As noted above, "vertical" and "horizontal" are arbitrary.) Raster scanning may be implemented by physically moving the array in two dimensions or by providing an optical system with tip-tilt mirrors that can steer light in a raster pattern.

[0205] Some embodiments of a raster scanning system can include a sensor array 2900 and a bulk optics module that supports uniform sampling of the field of view 2920. If the bulk optics module introduces global distortion (e.g., barrel distortion or pincushion distortion), the resulting image of the field of view 2920 will not be uniformly sampled. By way of example, FIG. 30 shows a non-uniform sampling pattern that can result from raster scanning using a sensor array with bulk optics that exhibits pincushion distortion. Each grid 3001-3006 represents a location imaged by the sensor at a different location in the raster pattern. As shown, the distortion is subject to local deviations. This type of local distortion pattern can pose significant challenges to subsequent image processing and analysis (much more so than global pincushion distortion).

[0206] As described above, the use of a bulk optics module with an F tan θ focal length distortion profile can provide uniform sampling across the sensor array. Therefore, the F tan θ bulk optics module can be used in a raster-scanned system. Alternatively, the sensor channels of the sensor array of the raster-scanned system can be arranged to compensate for the distortion profile of the bulk optics module, for example, as described above with reference to Figures 27 and 28.

[0207] It should be understood that the foregoing examples of optical elements and optical modules are illustrative and subject to change and modification. Furthermore, optical elements shown in connection with one type of sensor array can also be used with other types of sensor arrays. For example, achromatic bulk optic modules can be used with both row-based (or 1D) and 2D multispectral sensor arrays. Achromatic bulk optic modules can have an F tan θ focal length distortion profile, an F theta focal length distortion profile, or a different profile, as desired. Similarly, bulk optic modules with chromatic aberration can have an F tan θ focal length distortion profile, an F theta focal length distortion profile, or a different profile, as desired. As noted above, achromatic bulk optic elements may be desirable for sensor arrays that include multispectral and / or hybrid sensor channels; however, this is not required.

[0208] 3. Ranging / imaging system with multispectral sensor array Multispectral sensor arrays of the types described above can be incorporated into ranging / imaging systems that provide multispectral images (e.g., color images, absorption images, polarization images, and / or other images extracted from ambient light sensor channels) of a field of view that are essentially registered with each other and registered to depth information (e.g., extracted from LIDAR sensor channels in the multispectral sensor array). The specific implementation of a multispectral ranging / imaging system will depend in part on the particular multispectral sensor array. For illustrative purposes, two types of ranging / imaging systems are described. The first type, referred to herein as an “angular scanning” (also called “rotating” or “spinning”) ranging / imaging system, rotates the sensor array (and its associated optical elements) to point to different portions of the field of view at different times, or uses controllable optical elements (e.g., MEMS galvanometers) to direct light from different portions of the field of view onto the array at different times. In either case, the angular scanning system allows different sensor channels on the same array (e.g., different sensors in a column of sensor array 200 in FIG. 2) to image (detect photons from) a given area in the field of view at different times. A second type, referred to herein as a "static" (or "solid-state") ranging / imaging system, uses a 2D multispectral sensor array that can image the field of view with multiple channels without moving the sensor array.

[0209] 3.1.Angular scanning ranging / imaging system 31A illustrates an example of an automotive application of an angle-scanning (e.g., rotating or spinning) imaging / LIDAR system 3100 incorporating a sensor array described herein. The automotive application is chosen here for illustrative purposes only; the sensors described herein may be used in other types of vehicles, such as boats, aircraft, and trains, as well as in a variety of other applications where 3D depth images spatially and temporally registered to spectral images are useful, such as medical imaging, geodesy, geomatics, archaeology, geography, geology, topography, seismology, forestry, atmospheric physics, laser guidance, airborne laser swath mapping (ALSM), and laser altimetry. According to some embodiments, the scanning imaging / LIDAR system 3100 may be mounted on the roof of a vehicle 3105, as shown. In other embodiments, one or more LIDAR and / or imaging sensors may be mounted elsewhere on the vehicle, including, but not limited to, the front or rear of the vehicle, the sides of the vehicle, and / or the corners of the vehicle.

[0210] The scanning imaging / LIDAR system 3100 shown in Figure 31A can incorporate a light source module 3102 for emitting laser pulses, such as the transmit module 1510 of Figure 15, and / or a light-sensing module 3104, such as the receive module 1540 of Figure 15, which can incorporate a sensor array (e.g., any of the multispectral sensor arrays described above) that includes both LIDAR sensor channels and ambient light sensor channels. In some embodiments, the light-transmitting module 3102 can be disposed within the same housing as the light-sensing module 3104.

[0211] The scanning imaging / LIDAR system 3100 can employ a scanning architecture, where the orientation of the LIDAR optical transmitting module 3102 and the optical sensing module 3104 can scan around one or more fields of view 3110 (e.g., in some embodiments, a 360-degree field of view) within an external field or scene outside the vehicle 3105. With a scanning architecture, the emitted light 3112 can be scanned across the surrounding environment as shown. For example, the output beam(s) of one or more light sources (e.g., infrared or near-infrared pulsed IR lasers, not shown) located within the scanning imaging / LIDAR system 3100 can scan, e.g., rotate, to illuminate the scene around the vehicle. In some embodiments, the scanning, represented by the rotation arrow 3115, can be performed by mechanical means, e.g., by mounting the light emitter and sensor on a rotating column or platform. In some embodiments, the scanning can be performed through other mechanical means, such as the use of a galvanometer. For example, chip-based steering techniques can also be used by using a microchip employing one or more MEMS-based reflectors, such as a digital micromirror (DMD) device, digital light processing (DLP) device, etc. In the case of an emitter, such a mirror subsystem can be controlled to direct light onto different portions of the field of view at different times, and in the case of a sensor, such a mirror subsystem can be controlled to direct light from the field of view onto different portions of the sensor array at different times. In some embodiments, scanning can be achieved via non-mechanical means, such as by steering one or more optical phased arrays using electronic signals.

[0212] An object in the scene (e.g., object 3110) can reflect a portion of the light pulse emitted from the LIDAR light source. One or more reflected portions can then return to the imaging / LIDAR system and be detected by a detector circuit. For example, reflected portion 3114 can be detected by the light sensor module 3104. Additionally, ambient light 3116 can enter the detector circuit 3104.

[0213] 31B is a side view illustrating a simplified example of the structure of a scanning imaging / LIDAR system 3100, according to some embodiments. The scanning imaging / LIDAR system 3100 may include a fixed base 3120 that may be mounted, for example, to the roof of a vehicle 3105. A rotating housing 3122 that holds the emitter module (Tx) 3102 and the light sensor module (Rx) 3104 may be rotatably coupled to the fixed base 3120.

[0214] 32 illustrates a block diagram of a rotating imaging / LIDAR system 3200 (e.g., implementing the scanning imaging / LIDAR system 3100 of FIG. 31 ), according to some embodiments. The rotating imaging / LIDAR system 3200 can optionally use a rotary actuator with wireless data and power transmission and reception capabilities. In some embodiments, the rotary actuator includes a rotor integrated on the surface of a rotating circuit board and a stator integrated on the surface of a stationary circuit board, with both board assemblies equipped with wireless power and data transfer capabilities.

[0215] The rotating imaging / LIDAR system 3200 shown in FIG. 32 includes two main modules: an optical ranging / imaging (R / I) device 3220 and a rotational actuator 3215. The rotating imaging / LIDAR system 3200 can also interact with one or more instantiations of user interface hardware and software 3205. Different instantiations of the user interface hardware and software 3205 can vary and include, for example, a computer system with a monitor, keyboard, mouse, CPU, and memory, a touchscreen in an automobile, a handheld device with a touchscreen, or any other suitable user interface. The user interface hardware and software 3205 can be local to the object on which the rotating imaging / LIDAR system 3200 is mounted, or it can be a remotely operated system. For example, commands and data to / from the rotating imaging / LIDAR system 3200 can be routed through a cellular network (e.g., LTE), a personal area network (e.g., Bluetooth, Zigbee), a local area network (e.g., Wi-Fi, IR), or a wide area network such as the Internet.

[0216] The user interface hardware and software 3205 can present LIDAR data from the device to a user and / or allow a user or a higher-level program to control the rotating imaging / LIDAR system 3200 with one or more commands. Example commands can include commands to activate or deactivate the imaging / LIDAR system, specify photodetector exposure levels, biases, sampling periods, and other operating parameters (e.g., emission pulse patterns and signal processing), and specify light emitter parameters such as brightness. In addition, the commands can allow a user or a higher-level program to select a method for displaying or interpreting the results. The user interface can display the results of the imaging / LIDAR system, including, for example, a single-frame snapshot image, a continuously updated video image, and / or other light measurements for some or all pixels. Examples of other light measurements for LIDAR pixels include ambient noise intensity, return signal strength, calibrated target reflectivity, target classification (hard target, diffuse target, retroreflective target), range, signal-to-noise ratio, target line-of-sight velocity, return signal temporal pulse width, etc. In some embodiments, the user interface hardware and software 3205 can track the distance (proximity) of objects from the vehicle and / or analyze visual features determined from the ambient light sensor channel. Based on the visual features and distance information, the user interface hardware and software can, for example, identify and track objects within the field of view, potentially providing a warning to the driver or providing such tracking information for analysis of driver behavior.

[0217] In some embodiments, the imaging / LIDAR system can communicate with the vehicle control unit 3210, and one or more parameters associated with controlling the vehicle can be modified based on the received LIDAR data and / or ambient light data. For example, in a fully autonomous vehicle, the imaging / LIDAR system can provide real-time 3D hyperspectral images of the environment surrounding the vehicle to aid in navigation. In other cases, the imaging / LIDAR system can be used as part of an advanced driver assistance system (ADAS) or as part of a safety system that can provide 3D hyperspectral image data to any number of different systems (e.g., adaptive cruise control, automatic parking, driver drowsiness monitoring, blind spot monitoring, collision avoidance systems, etc.). When the vehicle control unit 3210 is communicatively coupled to the optical ranging / imaging device 3220, alerts can be provided to the driver or the proximity of objects can be tracked and / or displayed.

[0218] The optical ranging / imaging device 3220 includes an optical sensing module 3230, an optical transmitting module 3240, and an optical ranging / imaging system controller 3250. The optical sensor module 3230 may be similar to the optical sensing module 1540 described above and may include a sensor array such as sensor array 200 of FIG. 2 or sensor array 400 of FIG. 4. The optical transmitting module 3240 may be similar to the optical transmitting module 1510 described above. The rotary actuator 3215 includes at least two circuit board assemblies: a lower circuit board assembly 3260 (also referred to herein as a base subsystem) and an upper circuit board assembly 3280 (also referred to herein as a turret subsystem). The lower circuit board assembly 3260 may be mechanically attached to a stationary portion of an enclosure or housing (not shown), while the upper circuit board assembly 3280 is typically free to rotate about an axis of rotation defined by a shaft (not depicted in FIG. 32 ) that is also attached (directly or indirectly) to the enclosure. The optical ranging / imaging device 3220 may be mechanically mounted to a rotatable upper circuit board assembly 3280 and is therefore free to rotate within the housing.

[0219] 32 shows one particular arrangement of components within the optical ranging / imaging device 3220 and the rotary actuator 3215, in some embodiments, certain components may be integrated into one or the other module differently than shown. As an example, the ranging / imaging system controller 3250, which may be, for example, an FPGA, an ASIC, or a more general computing device similar to an embedded system or system-on-a-chip (SOC), may be mounted (e.g., soldered) directly to a printed circuit board that is part of the upper circuit board assembly 3280. In other words, in some embodiments, parts of the rotary actuator may be integrated within the optical ranging / imaging device 3220, or vice versa.

[0220] The rotary actuator 3215 includes several different systems integrated onto one or more printed circuit boards of the lower circuit board assembly 3260 and the upper circuit board assembly 3280. For example, the rotary actuator 3215 may include a brushless electric motor assembly, an optical communications subsystem, a wireless power transmission subsystem, and a base controller. These systems are formed by pairs of cooperative circuit elements, with each pair including one or more circuit elements on the lower circuit board assembly 3260 operating in cooperation (e.g., having complementary functions) with one or more circuit elements on the upper circuit board assembly 3280. As described in more detail below, the complementary functions include, for example, transmission (Tx) and reception (Rx) of power and / or data communications signals.

[0221] The brushless electric motor assembly includes a stator assembly 3262 integrated onto the printed circuit board of the lower circuit board assembly 3260 and a rotor assembly 3282 integrated onto the printed circuit board of the upper circuit board assembly 3280. Rotation of the rotor assembly 3282 is driven based on a drive signal, e.g., a three-phase drive current, generated by a motor driver circuit 3264. In some embodiments, one or more motor control lines connect the motor driver circuit to the coils of the stator assembly 3262, enabling the drive signal to be provided to the motor stator. Furthermore, the motor driver circuit 3264 can be electrically connected to the base controller 3266 such that the base controller 3266 can control the rotational speed of the rotor assembly and therefore the rotational speed (i.e., frame rate) of the optical ranging / imaging device 3220.

[0222] In some embodiments, the rotor assembly 3282 can rotate at speeds between 10 and 30 Hz. In some embodiments, the rotor assembly 3282 can be a passive device including a series of permanent magnets attached to the circuit board of the upper circuit board assembly. These permanent magnets are either attracted or repelled by electromagnetic, e.g., magnetic, forces generated by the coils of the stator assembly to drive the rotation of the upper circuit board assembly 3280 relative to the lower circuit board assembly 3260. The direction of rotation of the upper circuit board assembly 3280 can be tracked by a rotary encoder receiver 3294, which can track the angular position of the upper circuit board assembly by detecting the passage of one or more features on a rotary encoder 3274. A variety of rotary encoder technologies can be used. In some embodiments, the rotary encoder 3274 is integrated directly into the surface of the circuit board of the lower circuit board assembly 3260.

[0223] The rotary actuator 3215 may also include a wireless power system including a wireless power transmitter 3272 and a wireless power receiver 3292 in a configuration referred to herein as a rotary transformer. Power transmitted from the transmitter 3272 to the wireless power receiver 3292 may be consumed by the optical ranging / imaging device 3220 and / or any circuitry on the turret / upper circuit board assembly that requires power. In some embodiments, all power required by the optical ranging / imaging device 3220 is provided through the wireless power receiver 3292, thus eliminating the need for rotary electrical couplers similar to slip rings or mercury-based devices, thereby increasing overall system reliability and reducing costs.

[0224] The rotary actuator 3210 may also include an optical communications subsystem including several optical transmitters (e.g., optical transmitters 3278 and 3296) and several optical receivers (e.g., optical receivers 3276 and 3298) used for bidirectional, contactless data transmission between the rotary actuator 3215 and the optical ranging / imaging device 3220 (or to / from any other device or system mechanically connected to the upper circuit board assembly 3280 of the rotary actuator 3215). More specifically, the optical communications subsystem may include a set of base optical communications components mounted (e.g., soldered) to a lower circuit board assembly 3260 that is part of the fixed base of the imaging / LIDAR system 3200, and a set of turret optical communications components mounted (e.g., soldered) to a rotating upper circuit board assembly 3280 that is part of the rotating turret of the LIDAR system 3200. These optical communication components provide an uplink data channel for providing optical signals containing control signals to the optical ranging / imaging device 3220, and also provide a downlink data channel for providing optical signals containing ranging and operational data from the optical ranging / imaging device 3220 to the base controller 3266, the user interface hardware and software 3205, and / or the vehicle control unit 3210.

[0225] A downlink optical communication channel from the upper circuit board assembly 3260 to the lower circuit board assembly 3280 can be created between the optical downlink transmitter 3296 and the optical downlink receiver 3276. The optical ranging / imaging device 3220 can be directly connected to the upper circuit board assembly 3280 and thus access the downlink optical communication channel to pass ranging and operational data to the lower circuit board assembly 3260 for further use. In some embodiments, the data passed in the optical signal over the optical downlink can include range data for individual points (pixels) in the field (or, in some cases, multiple ranges for a single pixel and angle when looking through a glass window during fog / rain, for example), azimuth and zenith angle data, signal-to-noise ratio (SNR) of the return or signal strength, target reflectivity, ambient near-infrared (NIR) levels coming from each pixel's field of view, diagnostic operational information from the optical ranging / imaging device such as temperature, voltage levels, etc. Additionally, data from any other systems connected to the rotary actuator's upper circuit board 3280 can be passed over the optical downlink. For example, data from high-speed RGB or thermal cameras, line scan cameras, etc.

[0226] An uplink optical communication channel from the lower circuit board assembly 3260 can be created between the optical uplink transmitter 3278 and the optical uplink receiver 3298. In some embodiments, control signals from the base controller 3266 can be passed to the optical ranging / imaging device 3220 via the optical uplink communication channel. For example, in some embodiments, the base controller 3266 can monitor various temperatures in the device (received from the downlink channel) and, in the event of an overheating condition, can send an emergency shutdown signal to the optical ranging / imaging device 3220 via the uplink channel. In some embodiments, the base controller can be a mobile computer, e.g., a programmable system-on-chip using an ARM+FPGA architecture with associated memory and I / O capabilities (e.g., Ethernet, etc.).

[0227] Ranging data can be generated by the optical ranging / imaging device 3220 by transmitting one or more light pulses from the optical transmitting module 3240 to objects within a field of view surrounding the optical ranging / imaging device. Reflected portions of the transmitted light are then detected by the optical sensing module 3230 after some delay time. Based on the delay time, commonly referred to as the "time of flight," the distance to the reflecting surface can be determined. Other ranging methods, such as continuous wave, Doppler, etc., can also be used.

[0228] In addition to ranging data, the optical ranging / imaging device 3220 can generate light intensity data based on the ambient light. For example, the light sensing module 3230 can include one or more ambient light sensor channels tuned to various wavelength bands (e.g., as described above), where the ambient light sensor channels can operate to count photons of the channel wavelength band detected during a specific time interval (referred to herein as a "shutter interval"). The photon count of a particular channel indicates the intensity of light in that wavelength band. Other ambient light sensor channels can be used to measure other characteristics of the ambient light, such as polarization (e.g., by determining the difference in photon counts detected by polarization channels with different orientations) and / or absorption at specific wavelengths (e.g., by comparing the number of photons of a channel tuned to an absorption band to the number of photons of another channel tuned to a broader band that includes the absorption band, with the absence of the absorption band indicating absorption).

[0229] The optical transmission module 3240 may include an emitter array 3242 (e.g., the emitter array 1520 described above) and transmit (Tx) optics 3244 (e.g., including the Tx optics module described above). The optical transmission module 3240 may further include a processor 3246 and a memory 3248, although in some embodiments, these computing resources may be incorporated into the ranging / imaging system controller 3250. In some embodiments, a pulse coding technique such as a Barker code may be used. In such a case, the memory 3248 may store a pulse code that indicates when to transmit light. In one embodiment, the pulse code is stored as a sequence of integers stored in memory.

[0230] Light-sensing module 3230 can include a sensor array 3232 and receiver (Rx) optics 3234. Sensor array 3232 can be, for example, an implementation of sensor array 200 or sensor array 400 (or a similar sensor array) and can include rows of sensor channels incorporating both LIDAR sensor channels (or other ranging sensor channels) and ambient light sensor channels, as described above.

[0231] As described above, the processor 3236 and memory 3238 (e.g., SRAM) can perform signal processing. As an example of signal processing for a ranging sensor channel, for each photon detector, or group of photon detectors, the memory 3238 of the light-sensing module 3230 can accumulate counts of detected photons over successive time bins, which can be used in combination to recreate a time series of reflected light pulses (i.e., photon counts versus time). This time series of aggregated photon counts is referred to herein as an intensity histogram (or simply a histogram). In addition, the processor 3236 can apply certain signal processing techniques, such as matched filtering, to help recover a photon time series that is less susceptible to pulse shape distortions that can occur due to SPAD saturation and quenching. As an example of signal processing for an ambient light sensor channel, for each light sensor, or group of light sensors, the memory 3238 of the light-sensing module 3230 can accumulate counts of detected photons over a single time interval (referred to herein as a “shutter interval”). The shutter interval can be, for example, as long as the total length of the time bins used to construct the intensity histogram of the ranging sensor channel, or it can be a longer or shorter time interval. The photon counts accumulated by a particular ambient light sensor channel during the shutter interval can indicate the intensity of light received by that ambient light sensor channel. In some embodiments, the processor 3236 can apply signal processing techniques, such as calibration-based corrections to reduce noise and / or compensate for channel-to-channel variations in intensity measurements. In some embodiments, one or more components of the ranging / imaging system controller 3250 can be integrated into the same ASIC as the sensor array 3232, processor 3236, and memory 3238, thereby eliminating the need for a separate ranging controller module.

[0232] In some embodiments, output from the processor 3236 is sent to the ranging / imaging system controller 3250 for further processing. For example, data may be encoded by one or more encoders in the ranging / imaging system controller 3250 and then sent as data packets via an optical downlink to the lower circuit board assembly 3260. The ranging / imaging system controller 3250 may be implemented in several ways, including, for example, using a programmable logic device such as an ASIC or an FPGA as part of an ASIC, using a processor 3252 with memory 3254, and any combination of the above. The ranging / imaging system controller 3250 may operate in cooperation with a base controller 3266 or independently (via pre-programmed instructions) from the base controller to control the light-sensing module 3230 by sending commands to adjust light detector parameters, including starting and stopping light detection. Similarly, the ranging / imaging system controller 3250 can control the optical transmission module 3240 by sending commands or relaying commands from the base controller 3266, including controls to start and stop light emission and controls that may adjust other optical emitter parameters such as emitter temperature control (for wavelength tuning), emitter drive power and / or voltage.

[0233] If the emitter array 3242 has multiple independent drive circuits, there may be multiple on / off signals that can be appropriately sequenced by the ranging / imaging system controller 3250. Similarly, if the emitter array includes multiple temperature control circuits to differently regulate different emitters in the array, the transmitter parameters may include multiple temperature control signals. In some embodiments, the ranging / imaging system controller 3250 has one or more wired interfaces or connectors for exchanging data with the light-sensing module 3230 and the light-transmitting module 3240. In other embodiments, the ranging / imaging system controller 3220 communicates with the light-sensing module 3230 and the light-transmitting module 1840 via a wireless interconnection, such as an optical communication link.

[0234] While specific examples of scanning ranging / imaging systems are described in detail, those skilled in the art with access to this disclosure will recognize that other implementations are possible, including scanning ranging / imaging systems that perform raster scanning in two dimensions. The raster scanning mechanism may include, for example, an electric motor for moving the sensor array in two dimensions (e.g., rotary motion about one axis combined with linear or rotary motion along or about an orthogonal axis), a tip-tilt mirror system that is rotatable about two or more orthogonal axes, or a combination of sensor array and mirror system motions (e.g., a raster scanning mechanism may move the sensor array in one direction and move a mirror to provide scanning in an orthogonal direction).

[0235] 3.2. Operation of the Scanning Ranging / Imaging System In an example imaging operation, the rotation (or other scan) of the optical ranging / imaging device 3220 can be coordinated with the shutter interval (which may correspond to the LIDAR active sensing interval) so that a given location in the field of view is successively imaged by each sensor channel in a row of the sensor array. That is, the time between shutter intervals can be based on the angular distance between adjacent image pixels divided by the rotational speed of the imaging / LIDAR sensor array. Because the sensor channels image the same point in space (at slightly different times), registration between images acquired from different channels is unique, and no object identification or point-mapping algorithms are required. Furthermore, if the speed of the imaging operation is fast enough, it can be assumed that little change occurs between imaging on successive channels, and the images correspond to the same scene. Thus, in some embodiments, a row-based sensor array such as sensor array 200 or sensor array 400 can enable multispectral imaging over a wide field of view (e.g., up to 360 degrees).

[0236] 33A and 33B illustrate an example of multispectral imaging with inherent registration between imaging channels using an imaging / LIDAR sensor array similar to sensor array 200 or sensor array 400 described above. FIG. 33A shows a scanned field of view 3300 (e.g., a 360-degree field of view). For illustrative purposes, the description of the imaging process refers to a particular image region 3302 within field of view 3300; however, the same principles can be applied to all portions of field of view 3300.

[0237] FIG. 33B illustrates the progression of data collection at successive stages in a scanning operation using a row-based imaging / LIDAR sensor array (e.g., any of sensor arrays 200, 400, or 500) to create a set of essentially registered images of area 3302. In this example, the sensor array (not explicitly shown) is assumed to have five ambient light channels and one LIDAR channel in each sensor row, tuned to different colors (or wavelength regions). It is also assumed that the ambient light sensor channels in the columns of the sensor array are spaced apart by a uniform linear pitch p, and that the bulk optics module provided to the imaging / LIDAR sensor array has an F tan θ focal length distortion profile (e.g., as described above) such that rotation of the imaging / LIDAR system through a pitch angle α shifts the field of view by approximately the linear pitch p.

[0238] At a first time (t=1), the sensor array operates at a first shutter interval. Each channel collects data corresponding to a different location (or object space pixel) within region 3302, as indicated by the representative colored dots 3305. For example, the object space pixel indicated by box 3309 is sampled (or imaged) by green sensor channel 3306G. It should be understood that the actual number of sensor channels may be significantly greater than the number of colored dots 3305 shown in FIG. 33B ; for example, there may be more than five rows of sensors, and the row density may be significantly higher than what is shown.

[0239] At time t=2, the sensor array moves through pitch angle α, which shifts each channel to the right by a distance equal to liner pitch p relative to region 3302, with colored dots 3305 shifting one pitch to the right. At this time, the sensor array operates on a second shutter interval, in which object space pixel 3309 is sampled by yellow sensor channel 3306Y. (From time t=2 onwards, white dots 3307 indicate locations that were sampled by at least one sensor channel corresponding to white dots 3305 in the previous shutter interval, but are not currently being sampled by the channel corresponding to any of colored dots 3305.)

[0240] At time t=3, the sensor array again moves through the same pitch angle α, shifting each channel to the right by another pitch so that at time t=3, object space pixel 3309 is sampled by orange sensor channel 3306O. Similarly, at time t=4, object space pixel 3309 is sampled by red sensor channel 3306R. Proceeding in this manner, object space pixel 3309 (and elsewhere in region 3302) may ultimately be sampled by every sensor channel present in a particular row of the sensor array, including LIDAR sensor channel 3312. It should be understood that the channel pitch may be small and the number of sampling intervals per 360-degree rotation may be large (e.g., 1024, 2048, or 4096 sampling intervals per rotation) to provide higher image resolution than suggested by FIG. 33B. The size and shape of the object space pixel are determined in the non-scanning direction by the spacing of the sensor array rows (and the size of the field of view) and in the scanning direction by the angle between successive sampling operations. Depending on the particular system design, the object space pixels can have a simple aspect ratio (eg, 1:1 or 2:1, etc.) to facilitate image processing and analysis.

[0241] In this example, adjacent ambient light sensor channels within a row have a uniform pitch p, which facilitates unique registration of images captured using different sensors. As shown, the LIDAR sensor channels 3302 are spaced apart more than the uniform pitch of the ambient light sensor channels. In some embodiments, the spacing between LIDAR sensor channels 3312 and adjacent ambient light sensors within a row can be an integer multiple of the uniform pitch p of the ambient light sensor channels (in the example, the top row of sensors has LIDAR sensor channels 3312 spaced 2p from the nearest ambient light sensor channel), which further enables unique registration between the LIDAR and ambient light sensor channels. (This is shown in FIG. 33B.) More generally, data from different sensor channels within a row can be essentially registered to the same location in the field of view, provided the angular pitch of the sensor channels is an integer multiple of the angular displacement (or measurement angle) between successive shutter intervals, which is the case when the bulk optics module provided in the imaging / LIDAR sensor array has an F tan θ focal length distortion profile. In embodiments where this condition is not met, imaging operations can be performed and data from different sensor channels can be used to generate reliably registered images (because the spatial relationships between the different sensor channels are fixed); however, image processing may be more complex.

[0242] In some embodiments, the rotary ranging / imaging system can rotate continuously (e.g., at a rate of 10-30 Hz) and can determine when to start and stop collecting data based on the current angle of rotation. For example, as described above with reference to FIG. 32, the rotary actuator 3215 can include a rotary encoder 3274, and a rotary encoder receiver 3294 can track the angular position of the upper circuit board assembly 3280 (which is rigidly connected to the sensor array 3232). A set of M "measurement angles" φ corresponding to uniformly spaced angular positions can be calculated. i (for i=1,2...M) is φ for an integer N. i -φ i-1= α / N, where α is the pitch angle of the sensor array. In some embodiments, N=1. The number of measurement angles, M, can be selected as M=360° / (α / N) (or more generally Θ / (α / N), where Θ is the angle the sensor array moves during a scan). In one example, the rotary encoder 3274 has 2048 steps, and the sensor array and bulk optics module are designed such that α=360° / 2048.

[0243] The sensor array 3232 (along with the rest of the optical ranging / imaging device 3220) can be continuously rotated at a uniform angular velocity, and the LIDAR sensor channels can continuously generate signals. The memory 3238 can accumulate counts of detected photons over successive time bins, which can be used to create an intensity histogram as described above. A controller (e.g., R / I system controller 3250 in FIG. 32) can determine when the encoder position is at one of the measurement angles φ. i A signal indicating when the histogram corresponds to the time of receipt of the reflected LIDAR pulse can be received. This signal, also referred to as a "marker" signal, indicates the boundary between successive measurement periods of the LIDAR sensor channel. In response to this signal, the histogram data collected in memory 3238 can be sent to a digital signal processor (DSP) (e.g., processor 3236) for analysis, which may include, for example, applying a filter to the histogram data to determine the exact time of receipt of the reflected LIDAR pulse. In response to the same signal, memory 3238 can begin accumulating data for the next histogram. In some embodiments, memory 3238 can include two (or more) banks dedicated to storing photon counts, and photon count data from alternate measurement periods can be stored in alternate banks.

[0244] In some embodiments, the marker signal can also be used as a trigger to initiate a shutter interval for the ambient light sensor channels. During the shutter interval, a single photon count (accumulated across the entire shutter interval) can be determined from the signal received on each ambient light sensor channel. The photon counts from each ambient light sensor channel can be transmitted to the DSP along with histogram data from the LIDAR sensor channel. The shutter interval can have the same duration as the measurement period or a different (e.g., shorter) duration, as desired. In some embodiments, the shutter interval can be dynamically variable, for example, based on the current light level of one or more ambient light sensor channels, with a shorter shutter interval selected to avoid saturating the light sensor and a longer shutter interval selected under low light conditions.

[0245] Continuous rotation during measurement can be used with multispectral sensor arrays, as described above. Continuous rotation during measurement can also be used with other types of sensor arrays, such as LIDAR-only sensor arrays that include multiple rows of LIDAR channels (e.g., which may be staggered as shown in FIG. 1 and / or tuned to different emission frequencies). It should also be understood that continuous rotation is not required. In some embodiments, the rotating ranging / imaging system can rotate and collect data in stages, for example, rotating to a first measurement angle, collecting data for a measurement period, then rotating to the next measurement angle, and repeating the data collection.

[0246] 3.3. Increased Resolution Scanning in Ambient Light Channel 33A-33B, a scanning ranging / imaging system using a multispectral sensor array produces images with the same spatial resolution for all channel types. In some applications, it may be desirable to increase the spatial resolution of the ambient light sensor channels relative to the number of LIDAR channels. Next, we describe an example of a multispectral sensor array that can improve (increase) the spatial resolution of the ambient light sensor channels in both the scanning and non-scanning directions.

[0247] FIG. 34 shows a simplified front view of a sensor array 3400, according to some embodiments. The sensor array 3400 may be a 1D sensor array similar to the sensor array 200 of FIG. 2 above, including LIDAR sensor channels 202, each associated with a row 3404 including ambient light sensor channels 3406a-d. In this example, the ambient light sensor channels 3406a-d each have the same type of optical filter, which may be, for example, a broad-spectrum visible light filter (e.g., having a passband from about 425 nm to about 700 nm). Various types of optical filters (e.g., polarization filters, color filters, etc.) can be used, and in some embodiments, the ambient light sensor channels 3406a-d may not have an optical filter, in which case the wavelength range detectable by the ambient light sensor channels 3406a-d is determined by the wavelength range of the light sensors within the ambient light sensor channels 3406a-d. Ambient light sensor channels 3406a-d have different "subpixel" apertures (indicated by dark squares 3410). It should be understood that the dark squares 3410 indicate openings in the aperture plane, which is opaque over other portions of the area associated with channels 3406a-d. In this example, each subpixel aperture exposes a different quadrant of the channel area.

[0248] In operation, the sensor array 3400 can perform a scan as described above with reference to Figures 33A and 33B. As the ambient light sensor channels 3406a-d scan across the object space pixel, each channel 3406a-d samples a different "subpixel" (i.e., a subset of the total area of ​​the object space pixel) using the same type of optical filter. In this manner, an ambient light image can be generated that has four times the resolution of the LIDAR channel 202. Therefore, ambient light sensor channels such as channels 3406a-d are also referred to as "high-resolution" ambient light sensor channels.

[0249] In the example of FIG. 34, each ambient light sensor channel 3406a receives one-quarter of the incident light. In other embodiments, a spatial encoding scheme can be used to accept more light while still providing data with sub-pixel resolution. For example, FIG. 35 shows a set of four ambient light sensor channels 3506a-d with spatially encoded sub-pixel apertures, according to some embodiments. In this example, the aperture of channel 3506a (dark area) exposes the entire channel area, while the apertures of channels 3506b, 3506c, and 3506d each occlude a different quadrant of the channel area (white areas). The intensity measurements (e.g., photon counts) C0-C3 from channels 3506a-d can be provided to an arithmetic logic circuit 3520, which can implement the following equation to calculate sub-pixel values ​​for the sub-pixels (S0, S1, S2, S3) of pixel 3524: S1=C0-C3 (2a) S2=C0-C2 (2b) S3=C0-C1 (2c) S0=C0-(C1+C2+C3)=C1+C2+C3-2C0 (2d)

[0250] The examples in Figures 34 and 35 show subpixels as quadrants of the channel area, doubling the image resolution in each direction. Other embodiments can provide different increases in resolution. For example, higher resolution can be achieved by providing more ambient light channels 3406 or 3506 with smaller (relative to the channel area) subpixel apertures, with the upper resolution limit based on the aperture size required to measure intensity with acceptable accuracy. In some embodiments, the subpixel apertures are arranged so that the subpixels form a square grid (e.g., as shown in Figures 34 and 35), but this is not required, and other sampling patterns (including rectangular rather than square patterns) can be used. Furthermore, while the apertures shown in Figures 34 and 35 are either square or six-sided regions (with a square indentation in one corner), this is also not required, and circular apertures or apertures with other shapes can be used. It is assumed that all ambient light sensor channels within a group of ambient light sensor channels used for subpixel sampling have the same type of optical filter, so that the same spectral information is sampled in each channel, the effect of which is to increase the spatial resolution of the sampling. The particular filter type may be selected as desired, including broad spectrum filters, narrow band filters, or other types of optical filters.

[0251] 34 and 35, spatial resolution is increased in both the scanning and non-scanning directions by using subpixel apertures. This approach (with or without spatial encoding) involves using one ambient light sensor channel per subpixel, e.g., using four ambient light sensor channels to increase spatial resolution by four times, or using sixteen ambient light sensor channels to increase spatial resolution by sixteen times. In other embodiments, sampling resolution in the scanning direction can be increased by using temporal subdivision, and sampling resolution in the non-scanning direction can be increased by using subpixel apertures. This can allow, for example, four ambient light sensor channels to increase spatial resolution by sixteen times.

[0252] In some embodiments, temporal resolution can be provided by accumulating intensity data (e.g., photon counts) for each ambient light sensor channel using multiple integration registers, with different integration registers active during different portions of the shutter interval (the shutter interval is described above with reference to FIG. 33B). Assuming the sensor array is continuously rotating during the shutter interval, this has the effect of separately measuring the intensity of different portions along the scan direction of the area occupied by the object space pixel (conveniently referred to as the "column area").

[0253] Figure 36 shows a simplified schematic diagram of a read data path comprising multiple accumulation registers 3602, according to some embodiments. In this example, it is assumed that the light sensor 3604 of a particular ambient light sensor channel provides data (e.g., photon counts) for each time bin (as described above with reference to Figure 32), where the time bin is shorter than the shutter interval. The photon counts for each time bin are delivered to a selected accumulation register 3602 in a bank of accumulation registers 3610, which adds the photon count received from the light sensor 3604 to its currently stored value. A select signal is provided by bank select logic 3606 to select one of the accumulation registers 3602.

[0254] In the example shown, the accumulation registers operate as follows: at each clock cycle, multiplexer 3620, controlled by selection logic 3606, reads the stored value from one of the currently selected accumulation registers 3602. The current value 3622 thus selected is sent to arithmetic logic unit (ALU) 3624, which also receives a new photon count from photosensor 3604. ALU 3624 adds the new photon count to current value 3622 and delivers the result to accumulation register bank 3610. Selection logic 3606 selects the current one of accumulation registers 3602 to receive the new value. Other implementations may also be used.

[0255] In some embodiments of a scanning ranging / imaging system (e.g., system 3200 described above) with (N) accumulating registers, the selection logic 3606 divides the shutter interval into N sub-intervals (each sub-interval comprising one or more clock cycles) and selects a different one of the accumulating registers 3602 for each sub-interval, so that each accumulating register 3602 accumulates pixel counts for a different time portion (1 / N) of the sub-interval. For example, the selection logic 3606 can define the sub-intervals using a rotary encoder 3274 (as shown in FIG. 32), or the sub-intervals can be defined based on a timer used as a proxy for position based on a known scan speed. At the end of the shutter interval, each accumulating register 3602 can be read to provide N intensity measurements per pixel.

[0256] Temporally subdividing each shutter interval in this manner can increase the sampling resolution in the scan direction. Figure 37 illustrates ambient light measurement using multiple integrating registers for a vehicle 3703, according to some embodiments. In this example, a scanning ranging / imaging system 3701 (which may be, for example, an implementation of system 3200 described above) may be mounted on top of the vehicle 3703. The scanning ranging / imaging system 3701 may be configured to rotate around its central axis many times per second (e.g., at 30 Hz) to scan the surrounding area and generate a multispectral image, as described above with reference to Figure 33B.

[0257] In embodiments in which the scanning ranging / imaging system provides temporal subdivision of the ambient light sensor channels, the spatial resolution of the ambient light image in the scan direction can be increased based on the number of accumulating registers. In the example of FIG. 37, accumulating register bank 3710 (which can operate similarly to accumulating register bank 3610 described above) includes four accumulating registers 3712a-d. The increment of angular rotation 3702 corresponding to the shutter interval can be subdivided into four angular increments, during each of which received photon counts are accumulated in a corresponding one of accumulating registers 3712a-d (indicated by arrows 3716a, 3716b). This increases the spatial resolution of the ambient light image by a factor of four in the scan direction.

[0258] It may also be desirable to increase the spatial resolution of the ambient light image in the non-scanning direction. In some embodiments, multiple ambient light sensor channels with spatially encoded sub-pixel apertures can be used for this purpose. FIG. 38A shows a set of four ambient light sensor channels 3806a-d providing spatially encoded sub-pixel apertures, according to some embodiments. Using ambient light sensor channels 3806a-d, the resolution can be increased by a factor of four in both the scanning and non-scanning directions. In this example, the aperture (shaded area) of ambient light sensor channel 3806a exposes one-quarter of the total channel area, while the apertures of ambient light sensor channels 3806b-d each expose 3 / 16 of the total channel area. The intensity measurements (e.g., photon counts) C0-C3 from channels 3806a-3806d can be provided to an arithmetic logic circuit 3820, which can implement the following equation to calculate sub-pixel values ​​for the four sub-pixels (S0, S1, S2, S3) of pixel 3824: S1=C0-C2 (3a) S2=C0-C3 (3b) S3=C0-C1 (3c) S1=C0-(S1+S2+S3)=C2+C3+C1-2C0 (3d) As shown for pixel 3824, the four sub-pixels S0, S1, S2, S3 correspond to four pixels occupying different rows within a column area that is one-quarter the width (in the scan direction) of the total area of ​​pixel 3824.

[0259] To fully position subpixels within all column regions of a pixel, temporal subdivision as illustrated in FIG. 37 can be used to allow a single ambient light sensor channel to sequentially sample different column regions during a shutter interval. FIG. 38B shows the effect of temporal subdivision of ambient light sensor channel 3806a of FIG. 38A, according to some embodiments. In this example, the shutter interval is divided into four subintervals as described above with reference to FIG. 37. The shutter interval is assumed to last from t=0 to t=1. During the first subinterval (starting at t=0), the aperture of ambient light channel 3806a is exposed to column region 3832a of object space pixel 3824, and intensity C00 of column region 3832a is measured. During the second subinterval (starting at t=0.25), the aperture of ambient light channel 3806a is exposed to column region 3832b of object space pixel 3824, and intensity C01 of column region 3832b is measured. During the third subinterval (starting at t=0.5), the aperture of ambient light channel 3806a is exposed to column region 3832c of object space pixel 3824, and intensity C02 of column region 3832c is measured. During the fourth subinterval (starting at t=0.75), the aperture of ambient light sensor channel 3806a is exposed to column region 3832d of object space pixel 3824, and intensity C03 of column region 3832d is measured. Thus, using the temporal division of the shutter interval, ambient light sensor channel 3860a can successively sample each column region of object space pixel 3824 to provide four intensity values. As described above with reference to FIG. 33B, ambient light sensor channel 3806b can traverse object space pixel 3824 in the same manner as shown in FIG. 38B, with an offset of one shutter interval (or some other integer number of shutter intervals), to generate four intensity values, and similarly for ambient light sensor channels 3806c and 3806d. Applying the computational logic of FIG. 38A and equations (3a)-(3d) separately to the four intensity values ​​in each column provides a total of 16 subpixel samples using the four ambient light sensor channels. Thus, the combination of spatial and temporal subdivision of the object space pixel can provide an ambient light image with increased resolution in both the scanning and non-scanning directions.The example shown here increases resolution by a factor of four in each direction, but other embodiments may provide greater or lesser increases as desired.

[0260] It will be understood that the spatial and temporal subdivision examples described herein are illustrative. The specific number, shape, and size of apertures assigned to a particular ambient light sensor channel can be varied, and any enhancement factor can be achieved (subject to physical constraints such as light sensor size and the minimum size of apertures that can be manufactured). Thus, spatial resolution in the scanning and / or non-scanning directions can be increased to any desired degree, and the enhancement in the scanning and non-scanning directions need not be equal. The spatial resolution enhancement described herein can be applied to any type of ambient light sensor channel, regardless of the optical filter used.

[0261] 3.4.Static Ranging / Imaging System Rotating ranging / imaging systems such as those described above can be implemented using a multispectral sensor array, such as sensor array 200, sensor array 400, or sensor array 500, in which different types of sensor channels are arranged along rows that are scanned across the field of view. Other sensor array examples discussed above (e.g., sensor array 600, sensor array 900) provide 2D arrays of identical multispectral and / or hybrid sensor channels (or pixels). While such arrays may be used in rotating systems, a 2D array of multispectral or hybrid pixels does not require rotation or other scanning motion to image a two-dimensional field of view. Accordingly, some embodiments provide static (or "solid-state") ranging / imaging systems in which the sensor array does not move to perform imaging operations. It should be understood that static ranging / imaging systems can be mobile. For example, one or more static ranging / imaging systems can be mounted on a vehicle.

[0262] 39 is a side view illustrating a simplified example of the structure of a static imaging / LIDAR system 3900, according to some embodiments. The imaging / LIDAR system 3900, which is an example of a static ranging / imaging system, can include a housing 3922, which holds an emitter module (Tx) 3902 and a light sensor module (Rx) 3904. The housing 3922 can be mounted to a vehicle or any other location where a ranging / imaging sensor is desired.

[0263] 40 and 41 are simplified illustrations of example implementations of on-board static electronic ranging / imaging systems according to various embodiments. Specifically, FIG. 40 illustrates an implementation 4000 in which static ranging / imaging systems 4002a-d are implemented in an exterior region of a road vehicle 4005, such as an automobile, and FIG. 41 illustrates an implementation 4100 in which static ranging / imaging systems 4102a-b are implemented on top of the road vehicle 4105. In each implementation, the number of LIDAR systems, the placement of the LIDAR systems, and the field of view of each LIDAR system may be selected to capture most, if not the entire, 360-degree view of the environment surrounding the vehicle. An automotive implementation of a LIDAR system is chosen herein for illustrative purposes only; the sensors described herein may be utilized in other types of vehicles, such as boats, aircraft, trains, and in various other applications in which 3D depth images are useful, such as any of the applications discussed above with reference to FIG. 32. It should also be understood that static and rotational ranging / imaging systems can be used together, and that some ranging / imaging systems can be configured for selectable operation in either static or rotational modes.

[0264] Referring to FIG. 40 , static ranging / imaging systems 4002a-d can be mounted on the exterior regions of the vehicle, near the front and back fenders. Each of the static ranging / imaging systems 4002a-d can be positioned at a respective corner of the vehicle 4005, such that each is positioned near the outermost corner of the vehicle 4005. In this manner, the static ranging / imaging systems 4002a-d can better measure the distance of the vehicle 4005 from objects within the fields of regions 4006a-d. Each static ranging / imaging system can face in a different direction (possibly with overlapping and / or non-overlapping fields of view between units) to capture a larger combined field of view than each unit could capture by itself. Objects within the scene can reflect portions of the light pulse 4010 emitted from the LIDAR Tx module 4008. One or more reflected portions 4012 of the light pulse 4010 can then return to the static ranging / imaging system 4002a and be received by the Rx module 4009, which can be disposed in the same housing as the Tx module 4008. The Rx module 4009 can include a multispectral sensor array (e.g., as described above) that receives ambient light as well as reflected light from the LIDAR Tx module 4008.

[0265] In some embodiments, each of the static ranging / imaging systems 4002a-d can image its entire field of view (shown as regions 4006a-d, respectively) at once. In other embodiments, the static ranging / imaging systems 4002a-d can electronically scan a scene to capture images of the scene. As used herein, "electronic scanning" refers to collecting data for different portions of a scene at different times without physically moving (e.g., reorienting) the sensor array; thus, electronic scanning is distinguished from the rotating / spinning operations described above. Electronic scanning can be performed, for example, by activating different portions of the LIDAR emitter array and corresponding subsets of LIDAR sensor channels at different times, or by other means, such as chip-based beam steering techniques, for example, by using a microchip employing one or more MEMS-based reflectors, such as a digital micromirror (DMD) device, digital light processing (DLP) device, etc., to steer and reflect light from the Tx module 4008 onto different portions of the sensor array at different times. Thus, static ranging / imaging system 4002a can electronically scan between points 4020 and 4022 to capture objects within the field of area 4006a, and similarly for systems 4002b-d and areas 4006b-d.

[0266] While FIG. 40 illustrates four static electronically scanned LIDAR systems mounted at the four corners of the vehicle, embodiments are not limited to such a configuration. Other embodiments may have fewer or more static ranging / imaging systems mounted in other areas of the vehicle. For example, as shown in FIG. 41, the static ranging / imaging systems 4102a-b may be mounted on the roof of the vehicle. In such embodiments, the static ranging / imaging systems 4102a-b may have a higher vantage point to better observe the areas 4107a-b around the vehicle 4105.

[0267] As mentioned, the number of static ranging / imaging systems, the placement of the static ranging / imaging systems, and the field of view of each static ranging / imaging system may be selected to capture most, if not the entire, 360-degree field of view of the environment surrounding the vehicle. Thus, each static ranging / imaging system 4002a-d may be designed to have a field of view of approximately 90 degrees so that, when all four systems 4020a-d are implemented, a substantial majority of the 360-degree field of view around the vehicle 4005 can be observed. In embodiments in which each static ranging / imaging system 4002a-d has a field of view less than 90 degrees, for example, a 45-degree field of view, one or more additional static ranging / imaging systems may be implemented to expand the field of view and achieve a combined field of view that is larger than that of a single static ranging / imaging system.

[0268] 42 is a simplified plan view of an exemplary static ranging / imaging system 4200 including multiple sets of emission and detection systems to achieve an expanded field of view, in accordance with some embodiments of the present disclosure. As shown in FIG. 42 , the static ranging / imaging system 4200 can include sets of emission and detection systems 4202a-i mounted on a central support structure 4204, with each set of emission and detection systems including a respective optical emission system, e.g., optical transmission system 1510 of FIG. 15 , and optical detection system, e.g., optical detection system 1540 of FIG. 15 . Each set can be positioned radially outward from the center of the support structure 4204, and the sets can be positioned side-by-side such that their fields of view are adjacent to one another to form a combined field of view 4206 that is many times larger than the field of view of a single set of single emission and detection systems. All of the multiple emission and detection systems can be synchronized and controlled by a common LIDAR controller, allowing the end user to interact with what appears to be a single system. In addition, the individual emission detection systems are all aligned to a fixed pixel grid so that the data simulates a larger field-of-view, high-resolution system operating on a fixed field-of-view grid.

[0269] FIG. 43 illustrates a block diagram of an exemplary static ranging / imaging system 4300 according to some embodiments of the present disclosure. The static ranging / imaging system 4300 may include an optical ranging / imaging device 4302 and a user interface 4350. The optical ranging / imaging device 4302 may include a ranging / imaging system controller 4304, an optical transmitting (Tx) module 4306, and an optical sensing (Rx) module 4308. Ranging data may be generated by the optical ranging / imaging device 4302 by transmitting one or more optical pulses 4310 from the optical transmitting module 4306 to an object within a field of view surrounding the optical ranging / imaging device 4302. A reflected portion 4312 of the transmitted light is then detected by the optical sensing module 4308 after some delay time. Based on the delay time, the distance to the reflective surface may be determined. Other ranging methods may also be used, such as continuous wave, optical demodulation, Doppler, etc. Spectral image data can be generated by the optical ranging / imaging device 4302 by operating the ambient light sensor channels included in the sensor array 4308 in a photon counting mode.

[0270] The optical transmission module 4306 includes an emitter array 4314, which can be a one-dimensional or two-dimensional array of emitters, and Tx optics 4316, which, together with the emitter array 4314, can form an optical transmission system 4338 similar to the optical transmission system 1510 of FIG. 15 . The Tx module 4306 can further include an optional processor 4318 and memory 4320, although in some embodiments, these computing resources can be incorporated into the ranging / imaging system controller 4304. In some embodiments, a pulse coding technique, such as a Barker code, can be used. In such cases, the memory 4320 can store a pulse code that indicates when to transmit light. In some embodiments, the pulse code is stored as a sequence of integers stored in memory.

[0271] The light-sensing module 4308 can include a sensor array 4326, which can be any of the 2D multispectral sensor arrays described above, such as, for example, sensor array 600 or sensor array 900.

[0272] In some embodiments, the optical ranging / imaging device 4302 can operate in an electronic scanning mode, where at least a LIDAR image of a scene is captured by activating only a subset of emitters at a time and reading out only a corresponding subset of LIDAR sensor channels simultaneously with the emitter firing. Different subsets of emitters can be activated at different times and corresponding subsets of LIDAR channels can be read out simultaneously, so that eventually all emitters can be activated and all LIDAR channels in the sensor array can be read out through one emission cycle. As an example, the emitter array can emit light by activating one column at a time in sequence from left to right in each emission cycle, while the sensor array can be configured to read out corresponding LIDAR channels in a corresponding sequence. Ambient light channels can be read out synchronously with LIDAR channels corresponding to the same multispectral pixel, or in some other manner (e.g., all ambient light channels can be read out simultaneously).

[0273] To facilitate electronic scanning, some embodiments of the static ranging / imaging system may include one or more components for synchronizing light emission and sensing. In some embodiments, the light detection system 4336 may include a sensor controller 4325 coupled to the sensor array 4326 and configured to control the operation of the sensor array 4326. The sensor controller 4325 may be any suitable component or group of components capable of selecting one or more light sensors to sense light, such as an ASIC, a microcontroller, an FPGA, or any other suitable processor coupled to selection circuitry such as a multiplexer. Similarly, the light emission system 4338 may include an emitter controller 4315 coupled to the emitter array 4314 and configured to control the operation of the sensor array 4326. The emitter controller 4315 may also be any suitable processor described above for the sensor controller 4325 and may include one or more drive components for operating the emitter array 4314.

[0274] In some embodiments, the sensor controller 4325 and the emitter controller 4315 are synchronized so that the sequence of light emissions in the emitter array 4314 is synchronized with the sequence of readouts of the light sensors (of all sensor types or only LIDAR channels) in the sensor array 4326. As an example, both the sensor controller 4325 and the emitter controller 4315 can be coupled to a clock 4317 so that both controllers can operate based on the same timing scheme. The clock 4317 can be an electrical component that generates a specific signal that oscillates between high and low states at a specific rate to coordinate the operation of digital circuits. Optionally, the sensor controller 4325 and the emitter controller 4315 can include their own clock circuits to coordinate their own operation. In such embodiments, the sensor controller 4325 and the emitter controller 4315 can be communicatively coupled to each other via communication line 4319 so that the sensor controller 4325 can synchronize its clock with the emitter controller 4315. In that way, the sensor controller 4325 and the emitter controller 4315 can operate the sensor array 4326 and the emitter array 4314, respectively, in synchronization to achieve image capture.

[0275] In some further embodiments, instead of or in addition to the sensor controller 4325 and the emitter controller 4315, the ranging / imaging system controller 4304 can be configured to synchronize the operation of the light-sensing module 4308 and the light-transmitting module 4306, so that the sequence of light emission by the emitter array 4314 is synchronized with the sequence of light sensing by the sensor array 4326. For example, the ranging / imaging system controller 4304 can instruct the emitter array 4314 of the light-transmitting module 4306 to emit light by sequentially activating one column at a time from left to right for each emission cycle, and correspondingly instruct the sensor array 4326 of the light-sensing module 4308 to sense light in the same order, one column at a time. In such embodiments, the ranging / imaging system controller 4304 can have its own clock signal on which to base its sequence instructions to the light-sensing module 4308 and the light-transmitting module 4306. It should be understood that other forms of sequences for light detection are envisioned, and such sequences are not limiting. Additionally, the collection of (intensity) data for an ambient light sensor channel of a given multispectral pixel can be, but need not be, timed to coincide with the operation of the LIDAR sensor channel of that multispectral pixel.

[0276] The optical ranging / imaging system 4300 may also include other components that may be similar to corresponding components in Figure 32. Signal processing by the processor 4322 and memory 4324 may be similar to the processing operations described above with reference to Figure 32. The user interface 4350 and its operation may be similar to the user interface described above with reference to Figure 32. Furthermore, any of the ranging / imaging systems described herein may interact with other systems (e.g., vehicle control units) rather than directly (or indirectly) with a user, and such systems may control the operation of the ranging / imaging system by exchanging appropriate control instructions, data, or other signals with the ranging / imaging system controller 4304.

[0277] 3.5. Operation of the Static Ranging / Imaging System As noted above, imaging operations using the static ranging / imaging system 4300 can be performed in a variety of modes. In one mode, referred to as “full frame” mode, all sensor channels in the array (or all sensor channels of a given type) can be operated simultaneously. In another mode, referred to as “electronic scan” mode, different subsets of channels can be operated at different times. For example, as noted above, the Tx module 4306 can be operated to emit light that is reflected onto different portions of the sensor array in the Rx module 4308 at different times, e.g., by activating different emitters in the Tx module 4306 or by using the same emitter in combination with a MEMS-based beam steering component (e.g., a MEMS mirror galvanometer, sometimes referred to as a “galvo”) to control the direction of the emitted light. Different subsets of the LIDAR sensor channels can be selectively activated (e.g., by selective emission and / or steering) when light is directed to those channels.

[0278] A particular ambient light sensor channel (or a particular ambient light sensor of a multispectral or hybrid sensor channel) can also operate in either full-frame or electronic scan mode. In full-frame mode, all ambient light sensor channels can be activated simultaneously, or different types of sensor channels can be activated at different times. In electronic scan mode, different subsets of ambient light sensor channels corresponding to different regions within the sensor array can be activated at different times. For example, ambient light sensor channels corresponding to a particular group of multispectral pixels can be activated when the corresponding subset of LIDAR sensor channels is activated, or ambient light sensor channels corresponding to a particular group of multispectral pixels can be activated when the corresponding subset of LIDAR sensor channels is inactive.

[0279] In some embodiments, the operating mode of the LIDAR and / or ambient light sensor channels may be selectable. Furthermore, the LIDAR and ambient light sensor channels may operate in different modes. For example, the LIDAR channel may operate in an electronic scan mode, and the ambient light sensor channel may operate in a full frame mode to capture one spectral image during each scan period.

[0280] In any of these and other modes of operation, data can be collected for each sensor type for each multispectral pixel in the sensor array. Buffering can be used to collect data from different channels or sensor types corresponding to the same multispectral pixel. Thus, similar to the rotating ranging / imaging system described above, an image can be acquired containing a set of multispectral image pixels spanning the entire field of view.

[0281] 4. Multispectral Image Processing As described above, both rotating and static ranging / imaging systems can generate multispectral images of a field of view. A multispectral image can include an array (which can be a linear array) of multispectral image pixels, and for each image pixel, it can include depth information extracted from one or more LIDAR sensor channels, as well as information extracted from an ambient light sensor, such as intensity values ​​for various bands within the light spectrum (including visible, infrared, and ultraviolet), polarization-filtered light intensity, and / or other measurements described above. Multispectral imaging provides a rich data set for a specific location within the imaged area. For example, in the case of sensor array 400 of FIG. 4, the data set for a given image pixel can include the distance to the imaged object (i.e., any object that happens to be visible in the specific direction associated with the image pixel), the color signature (e.g., the intensity or amount of light collected in different wavelength bands), polarization signature, and absorption signature of the imaged object across the visible and near-infrared spectrum. Other combinations of image data per pixel are possible, depending on the specific combination of sensor channel types included in the sensor array.

[0282] By way of example, Figure 44 shows an example of multispectral image data that may be acquired for an area 4402 using either the rotational or static multispectral ranging / imaging system described above (or other similar systems). Image group 4402 includes spectral images acquired at different wavelength bands. Image group 4404 includes polarization images (intensity of light with a particular polarization direction). Image group 4406 represents a depth image based on data provided by a LIDAR sensor channel.

[0283] The images of image groups 4402, 4404, and 4406 can be inherently registered with one another due to the fixed spatial arrangement of the different sensor types. In the case of a sensor array in a rotational ranging / imaging system, the array can be arranged and operated so that all sensors in a given row sequentially image the same area (e.g., as described above), providing unambiguous (or inherent) registration. In the case of a 2D multispectral sensor array in a static ranging / imaging system, image pixels can be defined based on the area occupied by each group of different types of sensors. For example, in sensor array 600 of FIG. 6, each hybrid sensor channel 602 can correspond to an image pixel, and in sensor array 900, each multispectral pixel 1020 (shown in FIG. 10) can correspond to an image pixel. In these examples, due to a (small) spatial offset between sensors, different sensor types may sample different locations within a multispectral image pixel. In some embodiments, this offset can be ignored, and the data can be treated as if all sensors were located at the center of the image pixel. Alternatively, offset compensation can be applied as needed, for example, by interpolating from nearby sensor locations to the geometric center of each image pixel.

[0284] In some embodiments, the sensor array ASIC can stream pixel data to another system component (or another device) as it is acquired, and all image processing can be performed by the other system component. In other embodiments, the sensor array ASIC can include an “on-board” data buffer that can accumulate data for different image pixels (including a single channel per pixel or multiple channels per pixel). Depending on the implementation, the on-board data buffer can hold data for any number of multispectral image pixels, from one or two pixels to full image size. The buffered pixel data can be used to reconstruct a “local image” of the scene (which can be a 1D or 2D image and can be smaller than the full image size), and a processor within the sensor array ASIC or external to the sensor array can perform various image processing operations on the local image, including both pixel-by-pixel analysis and local or full-scene inference. The size of the on-board data buffer can be varied as needed, depending on the amount of data to be accumulated and the functionality required. Thus, image processing and image analysis operations can be performed on-chip or off-chip as needed.

[0285] In some embodiments, multispectral image analysis can include training an automatic classifier using a machine learning algorithm and a training set of images containing known (and labeled) objects. The machine learning algorithm can include an artificial neural network or other classifier (e.g., a classifier based on classical statistical techniques). Once training is complete, one or more automatic classifiers can be deployed within the sensor array ASIC (e.g., a machine learning coprocessor) or to a client system that receives data from the sensor array ASIC.

[0286] A variety of image processing and analysis operations can be performed on multispectral images, examples of which are provided below.

[0287] 4.1. Pixel-by-pixel analysis of multispectral image pixels In some embodiments, a rich data set per image pixel can enable advanced analysis, such as identifying materials within an image. To illustrate, FIG. 45 shows an example of an image annotated to identify the materials contained therein. In some cases, different materials may have similar colors to the human eye (e.g., a green car and green bushes), but the materials may have subtly different spectral signatures, different polarization characteristics, and / or absorption signatures that make them distinguishable based on analysis of each image pixel. Combining spectral response information from multiple ambient light channels (including any absorption band channels) with depth channel data can enable classification of hard, soft, and diffuse objects such as rock, plants, asphalt, metal, glass, water, skin, fur, and clothing, as well as various gases and particles such as methane, carbon dioxide, and black carbon. Multispectral pixel information can also be used to classify different narrow-spectrum and broad-spectrum light sources to provide other environmental factors, such as the type of lighting present, based on the pixel's spectral pattern. Such classification can be performed in real time, pixel by pixel. In some embodiments, an artificial neural network or other machine learning system (which can be implemented on-sensor or off-sensor as desired) can be trained to classify materials from multispectral image data based on a combination of depth, color, and polarization and / or absorption features, using manually annotated images as training input. Once training is complete, the machine learning system can identify the types and locations of objects present in the environment in real time.

[0288] As another example, real-time polarization imaging can occur in a sensor processor, which can combine data from multiple polarization channels to calculate the polarization angle and / or degree. Polarimetric measurements can be used, for example, to provide real-time glare removal on a vehicle windshield or on water surfaces, enhance contrast in shadow areas, enhance imaging in the presence of haze or other atmospheric obstructions, and / or provide real-time identification and classification of water, ice, and other polarized materials in the environment, or more specifically on road surfaces.

[0289] 4.2. System for Scene Inference from Multispectral Images In some embodiments, scene-level inference can be extracted by analyzing multispectral image data across a set of image pixels, which can include anywhere from two pixels to the entire image field of view. Scene-level inference can be performed on-chip using an on-board data buffer in the sensor ASIC and / or off-chip, for example, in another system component or another device. Many types of scene-level inference can be implemented.

[0290] For example, identification of distinct objects within the field of view can be based on identifying changes in color, polarization, and / or distance. In some embodiments, the results of a pixel-by-pixel analysis of possible material compositions can be used to identify objects based in part on the likely material compositions. Objects can be further evaluated to determine distance, composition, etc. This, combined with depth information from multispectral pixels, can reliably identify content (e.g., cars, walls, bushes, roads) and locations within the image. It is believed that machine learning systems can reliably determine the types and locations of objects present within an environment based on multispectral image data (including depth data) acquired using ranging / imaging systems of the type described herein. Such information has a variety of uses and applications, including, but not limited to, driver assistance and / or autonomous driving technologies.

[0291] Other inferences can also be made. For example, in some cases, the sun or moon may be identifiable as an object in the field of view. Multispectral image data can be used to identify and distinguish the sun and moon from one another, which may provide clues about the time of day and / or general lighting conditions. Even when the sun or moon is not in the field of view, the different spectral characteristics of different light sources may provide clues about whether ambient lighting significantly contrasts with artificial lighting (indicating nighttime or indoor situations, such as a tunnel or parking lot) or natural light (indicating daytime or outdoor situations). As another example, xenon-based or LED headlights on modern cars can be distinguished from sodium vapor street lights. As yet another example, LED-based traffic signals emit relatively narrow (approximately 50 nm) spectra of red, yellow, or green, and these spectra can be distinguished from the broader spectra of objects such as stop signs, green grass, and yellow lane marks.

[0292] 5. Additional Embodiments While the present invention has been described with reference to specific embodiments, those skilled in the art with access to this disclosure will recognize that numerous variations and modifications are possible. For example, multispectral sensor arrays of the type described herein can be fabricated to include any number of rows and any number of sensor channels per column. (The terms "row" and "column" are used to distinguish between the two dimensions of a sensor array, particularly in the context of an array used in a scanning mode, and do not imply any particular spatial orientation of the array.) The configuration of specific sensor channels, including per-channel micro-optical elements, can be varied. The combination of ambient light sensing channels used in each row can be varied as needed, and in some embodiments, different rows can have different combinations of ambient light sensing channels. Furthermore, the ambient light sensing channels are not limited to the specific examples given above; other types of optical filters can be used to create a variety of ambient light sensing channels that can be used to collect image data.

[0293] The term "ambient light sensing channel" is used herein to indicate that a sensor channel measures light intensity (as opposed to timing or other ranging data). Such a channel may provide useful data in the absence of intentional illumination emitted from the sensor system. However, this does not preclude intentional illumination of the field of view. For example, white light may be directed at the field of view (e.g., from a car headlight or a camera flash). As another example, in an application using an absorption channel, light having wavelengths that encompass an absorption band may be directed at the field of view, and the absence of light in the absorption channel may indicate that materials in the field are absorbing light.

[0294] In addition to the ambient light-sensing channel(s), the sensor array may include one or more LIDAR sensor channels (and / or other depth-sensing channels) that provide timing data (e.g., the histogram described above) or other data usable to derive distances to objects within the field of view. The LIDAR sensor channels may operate at various wavelengths, including near-infrared, short-wave infrared (e.g., 1600 nm), mid-wave infrared, and / or long-wave infrared (e.g., up to 15 μm). Furthermore, in some embodiments, additional sensor channels (e.g., LIDAR sensor channels) may be included between sensor rows, or there may be some sensor rows that do not include LIDAR sensor channels (or other depth-sensing channels), and images from different sensor channels (or sensor types) may have the same resolution, but need not be the same. The multispectral array may be a row-based (or “1D”) array operable in a scanning mode to image the field of view, or it may be a 2D array comprising multispectral sensor channels or multispectral pixels.

[0295] Sensor arrays of the type described herein can be incorporated into a variety of sensing systems, including, but not limited to, combined imaging / LIDAR systems such as those described above, which can be implemented using rotating and / or static platforms, and can be used in any application where it is desirable to simultaneously collect ambient light and ranging data.

[0296] The systems described herein can generate multispectral image data that can include both light intensity data for various portions of the light spectrum (including visible, infrared, and ultraviolet light with broad and / or narrow passbands as needed, light with various polarization states, and other examples mentioned above) and depth information for the entire field of view (which can be as wide as needed, up to 360 degrees in some embodiments). Images captured by various sensor types (including ranging sensors such as LIDAR) can inherently register with each other as a result of the alignment of the different sensor types on the sensor array. In some embodiments, this inherent registration can facilitate the generation of multispectral pixel data for the image.

[0297] Multispectral image data can be analyzed using a variety of computer-implemented algorithms operating on any portion of the data. In some embodiments, the multispectral image data can be used to generate images for display to a user, which can include rendering the image data directly and / or rendering an image of a scene (or portion thereof) based on algorithmic inferences from the data. While the above examples relate to vehicle navigation and / or driver assistance, the present invention is not limited to any particular data analysis or any particular application of multispectral image data.

[0298] The foregoing description of exemplary embodiments of the present invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form described, as many modifications and variations are possible in light of the above teachings. The embodiments were chosen and described in order to explain the principles of the invention and its practical application, thereby enabling those skilled in the art to use the invention in various embodiments, with various modifications suited to the particular uses contemplated. Thus, while the invention has been described with reference to specific embodiments, it should be understood that the invention is limited only by the scope of the following claims.

Claims

1. A single ASIC with an array of sensor channels arranged in multiple sensor rows. A sensor device comprising, Each sensor row extends along the scanning direction in a 2D plane. LIDAR sensor channel and A plurality of ambient light sensor channels positioned along the aforementioned sensor array, It has, The LIDAR sensor channel is, A LIDAR channel input aperture is located within the aperture plane, A LIDAR light sensor is positioned within the light sensor surface, The optical path between the LIDAR channel input aperture and the light sensor, A LiDAR optical filter in the optical path that allows light of the LiDAR signal wavelength to pass through to the LiDAR light sensor, Includes, The aforementioned light sensor surface is separated from the aperture surface, Each ambient light sensor channel is: An ambient light channel input aperture arranged within the aperture surface, An ambient light sensor arranged within the aforementioned light sensor surface, A light guide between the channel input aperture and the photosensor, A channel-specific optical filter that selectively passes light having channel-specific characteristics to the light sensor, Includes, The LIDAR sensor channels are arranged in a staggered grid, each LIDAR sensor channel is located in one of several different sensor rows, and each LIDAR sensor channel in adjacent sensor rows is offset from each other along the scanning direction. The sensor device further, A data buffer disposed within the ASIC and configured to store data from two or more of the LIDAR sensor channels and two or more of the ambient light sensor channels, A processing circuit disposed within the ASIC and configured to perform image processing operations on the data stored in the data buffer, A sensor device characterized by comprising the following features.

2. In each sensor array, the plurality of ambient light sensor channels are A first ambient light sensor channel having a first channel-specific optical filter that selectively passes light in a first wavelength range related to a first portion of the visible light spectrum to a first sensor among the plurality of ambient light sensors, A second ambient light sensor channel having a second channel-specific optical filter that selectively passes light in a second wavelength range related to the second part of the visible light spectrum to the second sensor among the plurality of ambient light sensors, A third ambient light sensor channel having a third channel-specific optical filter that selectively passes light in a third wavelength range related to the third part of the visible light spectrum to the third sensor among the plurality of ambient light sensors, including The sensor device according to feature 1.

3. The first wavelength range and the second wavelength range partially overlap, The second wavelength range and the third wavelength range partially overlap. The sensor device according to claim 2.

4. The first wavelength range is the red wavelength, The second wavelength range is the green wavelength, The third wavelength range is the blue wavelength. The sensor device according to claim 3, characterized in that it is a sensor device.

5. In each sensor array, the plurality of ambient light sensor channels are further, A fourth ambient light sensor channel having a fourth channel-specific optical filter that selectively passes light in the wavelength range covering the visible range of the electromagnetic spectrum to the fourth sensor among the plurality of ambient light sensors. including The sensor device according to claim 2.

6. Within a set of different sensor arrays, the LIDAR sensor channel occupies a central region along the scanning direction. A first subset of the plurality of ambient light sensor channels is arranged on one side of the central region along the scanning direction. A second subset of the plurality of ambient light sensor channels is located on the other side of the central region along the scanning direction. The sensor device according to feature 1.

7. The corresponding ambient light sensor channels in adjacent sensor rows are position-aligned within the row that crosses the scanning direction. The sensor device according to feature 1.

8. Each LIDAR sensor channel further, A collimating lens is positioned within the optical path between the LIDAR channel input aperture and the light sensor. including The sensor device according to feature 1.

9. The LIDAR light sensor and the ambient light sensor are single-photon avalanche diodes (SPADs) operating in photon counting mode. The sensor device according to feature 1.

10. Fixed base and A sensor array rotatably coupled to the aforementioned fixed base, A ranging / imaging system comprising, The aforementioned sensor array is A single ASIC with an array of sensor channels arranged in multiple sensor rows. Equipped with, Each sensor row extends along the scanning direction in a 2D plane. LIDAR sensor channel and A plurality of ambient light sensor channels positioned along the aforementioned sensor array, It has, The LIDAR sensor channel is, A LIDAR channel input aperture is located within the aperture plane, A LIDAR light sensor is positioned within the light sensor surface, The optical path between the LIDAR channel input aperture and the light sensor, A LiDAR optical filter in the optical path that allows light of the LiDAR signal wavelength to pass through to the LiDAR light sensor, Includes, The aforementioned light sensor surface is separated from the aperture surface, Each ambient light sensor channel is: An ambient light channel input aperture arranged within the aperture surface, An ambient light sensor arranged within the aforementioned light sensor surface, A light guide between the channel input aperture and the photosensor, A channel-specific optical filter that selectively passes light having channel-specific characteristics to the light sensor, Includes, The LIDAR sensor channels are arranged in a staggered grid, each LIDAR sensor channel is located in one of several different sensor rows, and each LIDAR sensor channel in adjacent sensor rows is offset from each other along the scanning direction. The recovered array further, A data buffer disposed within the ASIC and configured to store data from two or more of the LIDAR sensor channels and two or more of the ambient light sensor channels, A processing circuit disposed within the ASIC and configured to perform image processing operations on the data stored in the data buffer, A bulk optical module is disposed in front of the sensor array and configured to focus incident light onto the aperture surface, A controller for synchronizing the rotation of the sensor array with the operation of the optical sensor so that a given location in space relative to the fixed base is continuously imaged by the LIDAR sensor channel and the ambient light sensor channel in one of the sensor rows, A distance measuring / imaging system characterized by having the following features.

11. The controller is further configured to generate multispectral image pixel data including per-pixel light intensity data determined using the ambient light sensor channel of the sensor array, and per-pixel depth data determined using the LIDAR sensor channel of the sensor array. The distance measuring / imaging system according to feature 10.

12. In each sensor array, the plurality of ambient light sensor channels are A first ambient light sensor channel having a first channel-specific optical filter that selectively passes light in a first wavelength range related to a first portion of the visible light spectrum to a first sensor among the plurality of ambient light sensors, A second ambient light sensor channel having a second channel-specific optical filter that selectively passes light in a second wavelength range related to the second part of the visible light spectrum to the second sensor among the plurality of ambient light sensors, A third ambient light sensor channel having a third channel-specific optical filter that selectively passes light in a third wavelength range related to the third part of the visible light spectrum to the third sensor among the plurality of ambient light sensors, including The distance measuring / imaging system according to feature 10.

13. The first wavelength range and the second wavelength range partially overlap, The second wavelength range and the third wavelength range partially overlap. The distance measuring / imaging system according to feature 12.

14. The first wavelength range is a red wavelength between approximately 600 nm and approximately 700 nm. The second wavelength range is the green wavelength between approximately 490 nm and approximately 620 nm. The third wavelength range is the blue wavelength between approximately 410 nm and approximately 510 nm. The distance measuring / imaging system according to feature 13.

15. In each sensor array, the plurality of ambient light sensor channels are further, A fourth ambient light sensor channel having a fourth channel-specific optical filter that selectively passes light in the wavelength range of approximately 425 nm to approximately 700 nm to the fourth sensor among the plurality of ambient light sensors. including The distance measuring / imaging system according to feature 12.

16. Within a set of different sensor arrays, the LIDAR sensor channel occupies a central region along the scanning direction. A first subset of the plurality of ambient light sensor channels is arranged on one side of the central region along the scanning direction. A second subset of the plurality of ambient light sensor channels is located on the other side of the central region along the scanning direction. The distance measuring / imaging system according to feature 10.

17. The corresponding ambient light sensor channels in adjacent sensor rows are position-aligned within the row that crosses the scanning direction. The distance measuring / imaging system according to feature 10.

18. Each LIDAR sensor channel further, A collimating lens is positioned within the optical path between the LIDAR channel input aperture and the light sensor. including The distance measuring / imaging system according to feature 10.

19. Emitter subsystem having an array of pulse emitters that emit infrared light at the LIDAR signal wavelength. Furthermore, The controller is further configured to synchronize the operation of the pulse emitter with the operation of the optical sensor, thereby enabling time-of-flight measurement using the LIDAR sensor channel. The distance measuring / imaging system according to feature 10.

Images