Stereoscopic image capture system

Abstract

A stereoscopic imager system includes a sensor array including a first plurality of photosensors and a second plurality of photosensors spaced apart from the first plurality of photosensors by a gap, the first plurality of photosensors and the second plurality of photosensors configured to detect ambient light in a scene; a movement assembly coupled to the sensor array and operable to move the sensor array between a first position and a second position within a full rotation image capture cycle; and a system controller coupled to the sensor array and the movement assembly. The system controller can be configured to: move a field of view of the sensor array by instructing the movement assembly to capture, with the first plurality of photosensors, a first image of an object in the scene from a first perspective at the first position, and capture, with the second plurality of photosensors, a second image of the scene of the object in the scene from a second perspective at the second position; and calculate a distance to the object using an optical baseline defined by the gap based on the first image and the second image.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 016,118, filed April 27, 2020, and U.S. Application No. 17 / 229,671, filed April 13, 2021, each of which is incorporated herein by reference in its entirety. Background Technology

[0003] Image capture systems capture images of a scene by sensing light. Light is typically sensed by image sensors, such as charge-coupled devices (CCDs) or complementary metal-oxide-semiconductor (CMOS) devices, which convert the sensed light into electrons. These electrons can then be read and interpreted to construct the captured image. Images captured by image sensors typically do not provide depth perception for constructing a stereoscopic image of the scene, nor can they calculate distances to objects within the scene. Summary of the Invention

[0004] Some embodiments of this disclosure relate to a stereo image capture system that can capture depth information using a sensor array. A stereo image capture system can be configured to move its sensor array, for example, by rotating the array about an axis transverse to its rows, such that a given location in space can be continuously imaged by at least two photoelectric sensors spaced apart from each other. The distance between the two photoelectric sensors can be used as an optical baseline to calculate depth information to the given location in the field, thereby enabling the stereo image capture system not only to construct an image using depth perception from a 2D sensor array, but also to determine depth information to enhance the ranging accuracy of the depth sensors.

[0005] Some embodiments relate to a stereo image capture system incorporating both a ranging photoelectric sensor and an imaging photoelectric sensor. The stereo imager system may include: a sensor array comprising: a plurality of ranging photoelectric sensors that detect light once it has been reflected from an object in the scene by a transmitter array; a first plurality of imaging photoelectric sensors positioned on a first side of the ranging photoelectric sensors; and a second plurality of imaging photoelectric sensors positioned on a second side of the ranging photoelectric sensors opposite to the first side. The first plurality of imaging photoelectric sensors and the second plurality of imaging photoelectric sensors can detect ambient light in the scene and may be spaced apart. The system may further include: a moving component coupled to the sensor array and operable to move the sensor array between a first position and a second position within a complete rotational image capture cycle; and a system controller coupled to the sensor array and the moving component. The system controller can be configured to: determine a first distance to an object in the scene using the plurality of ranging photoelectric sensors via time-of-flight calculation; capture a first image of the scene at a first location using the first plurality of imaging photoelectric sensors, and capture a second image of the scene at a second location using the second plurality of imaging photoelectric sensors; and calculate a second distance to the object based on the first image and the second image and an optical baseline determined by the gap.

[0006] In some implementations, embodiments may include one or more of the following features: The plurality of ranging photoelectric sensors may be arranged in a diagonally staggered manner. The first plurality of imaging photoelectric sensors and the second plurality of imaging photoelectric sensors may each be arranged in a rectangular manner. At least some of the first plurality of imaging photoelectric sensors and at least some of the second plurality of imaging photoelectric sensors may be positioned along the same horizontal line. The moving component may be a motor that rotates the sensor array about a central axis. The moving component may be a microelectromechanical system (MEMS) device that reflects light to move the field of view. The system controller may be further configured to calculate the final distance to the object based on the first distance and the second distance.

[0007] According to some embodiments, a stereo imaging system includes: a sensor array comprising a first plurality of photoelectric sensors and a second plurality of photoelectric sensors spaced apart from the first plurality of photoelectric sensors, the first plurality of photoelectric sensors and the second plurality of photoelectric sensors being configured to detect ambient light in a scene; a moving component coupled to the sensor array and operable to move the sensor array between a first position and a second position within a complete rotational image capture cycle; and a system controller coupled to the sensor array and the moving component. The system controller may be configured to: move the field of view of the sensor array by instructing the moving component to capture a first image of an object in the scene from a first perspective using the first plurality of photoelectric sensors at the first position, and to capture a second image of the scene of the object in the scene from a second perspective using the second plurality of photoelectric sensors at the second position; and to calculate the distance to the object based on the first image and the second image using an optical baseline defined by the gap.

[0008] In some embodiments, a distance measurement method is provided, wherein the method comprises: moving the field of view of a sensor array, the sensor array including a first imaging photoelectric sensor and a second imaging photoelectric sensor spaced apart from the first imaging photoelectric sensor; as the field of view moves, capturing a first image of an object in a scene from a first viewpoint using the first imaging photoelectric sensor in a first time instance; capturing a second image of the scene of the object in the scene from a second viewpoint using the second imaging photoelectric sensor in a second time instance as the field of view moves; and calculating a first distance to the object based on the first image and the second image using an optical baseline defined by the gap.

[0009] In various embodiments, the method may include one or more of the following: Moving the field of view may include rotating the sensor array about a central axis. Moving the field of view may include reflecting light to move the field of view while the sensor array is stationary. The first imaging photoelectric sensor array and the second imaging photoelectric sensor array may each be a two-dimensional imaging photoelectric sensor array. The sensor array may be formed of a two-dimensional imaging photoelectric sensor array, and the first imaging photoelectric sensor array and the second imaging photoelectric sensor array may each be a subset of the two-dimensional imaging photoelectric sensor array. The method may further include comparing common features of the object captured in the first image and the second image, and using the result from the comparison to calculate a first distance to the object. The method may further include measuring a second distance to the object using a ranging photoelectric sensor, and determining a final distance to the object based on the first distance and the second distance. Moreover, the ranging photoelectric sensor may be located in a two-dimensional ranging photoelectric sensor array, and the first imaging photoelectric sensor array and the second imaging photoelectric sensor array may be located on opposite sides of the ranging photoelectric sensor array.

[0010] The nature and advantages of the embodiments of this disclosure can be better understood by referring to the following detailed description and accompanying drawings. However, it should be understood that each drawing is provided for illustrative purposes only and is not intended to limit the scope of this disclosure. Furthermore, as a general rule, and unless expressly contrary to the description, where elements in different figures use the same reference numerals, these elements are generally identical or at least similar in function or purpose. Attached Figure Description

[0011] Figure 1 This is a block diagram of an example passive stereo imaging system according to some embodiments of the present disclosure.

[0012] Figure 2 This is a simplified top view of an example light detection system configured to perform stereo imaging according to some embodiments of the present disclosure.

[0013] Figures 3A-3B This is a simplified top view of a rotating light detection system according to some embodiments of the present disclosure during different time instances of an image capture sequence.

[0014] Figure 3C The light detection system according to some embodiments of this disclosure is in Figures 3A-3B The top view shown is an overlay of different time instances.

[0015] Figure 4This is a simplified illustration of an example of two offset images captured by a first photoelectric sensor array and a second photoelectric sensor array respectively in a first time instance and a second time instance, according to some embodiments of the present disclosure.

[0016] Figure 5A This is a simplified illustration of an example sensor array of two separate linear photoelectric sensor arrays configured to be spaced apart from each other for stereo imaging, according to some embodiments of the present disclosure.

[0017] Figure 5B This is a simplified illustration of an example sensor array of two separate m x n photoelectric sensor arrays configured to be spaced apart from each other for stereo imaging, according to some embodiments of the present disclosure.

[0018] Figure 5C This is a simplified illustration of an example sensor array of a single mxn photoelectric sensor array configured to have two subsets of photoelectric sensors spaced apart from each other for stereo imaging, according to some embodiments of the present disclosure.

[0019] Figure 6 This is a block diagram of an example active stereo system according to some embodiments of the present disclosure.

[0020] Figure 7 This is a simplified diagram of an example light detection system according to some embodiments of the present disclosure.

[0021] Figure 8 This is a simplified illustration of an example sensor array for an active stereo imaging system according to some embodiments of the present disclosure.

[0022] Figure 9 This is a block diagram of an example method for operating a stereo imager system to perform stereo imaging using a light detection system, according to some embodiments of the present disclosure.

[0023] Figure 10A This is a simplified cross-sectional view of a portion of an optical detection system with no crosstalk between channels.

[0024] Figure 10B This is a simplified cross-sectional view of a part of an optical detection system where crosstalk exists between channels.

[0025] Figure 11 This is a simplified cross-sectional view of an example micro-optical receiver channel structure according to some embodiments of the present disclosure.

[0026] Figure 12A-12B This is a simplified illustration of an example embodiment of a stereo imager system according to some embodiments of the present disclosure. Detailed Implementation

[0027] Stereo imaging is a technique that creates or enhances depth perception using two offset two-dimensional (2D) images of the same field of view. The distance to a position within the field of view can be determined using data generated by stereo imaging and a known distance (i.e., an “optical baseline”) between the sensors that capture the 2D images. The imager system that performs stereo imaging to capture depth perception and determine distances is referred to herein as a “stereo imager system.”

[0028] Some embodiments of this disclosure relate to a stereo imager system with an image sensor whose field of view can move / rotate across a scene. The sensor array may comprise one or more photoelectric sensor arrays, or one or more subsets of photoelectric sensors in a single two-dimensional array. In some cases, the sensor array includes a first photoelectric sensor array spaced apart from a second photoelectric sensor array, wherein the first and second photoelectric sensor arrays are configured to capture images of the scene from two different viewpoints for stereo imaging purposes. To enable the sensor array to capture images from two different viewpoints, the viewing direction of each photoelectric sensor in the first photoelectric sensor array may intersect with the viewing direction of the corresponding photoelectric sensor in the second photoelectric sensor array, as described herein. Figure 2 Further discussion follows. Thus, when the viewing orientation of the first and second photoelectric sensor arrays moves across the scene, the field of view captured by the first photoelectric sensor array at a first position can overlap with the field of view of the second photoelectric sensor array at the second position after the sensor array's field of view moves from the first position to the second position. For example, the sensor arrays can rotate about an axis such that, while the sensor arrays rotate about the axis, the first and second photoelectric sensor arrays can capture images of the field from two different viewpoints at different times for calculating distances to objects in the scene, as will be discussed in this paper. Figures 3A-3C Further discussion is needed.

[0029] In some embodiments, the stereo imaging system may be a passive system, which does not actively illuminate the scene or instead detects ambient light reflected from one or more objects in the scene. The passive stereo imaging system may include a light sensing module for receiving ambient light in the field. The light sensing module may include one or more volume receiver optics, a micro-optical receiver system, and a system controller for operating the light sensing module. The micro-optical receiver system may include one or more micro-optical receiver layers and one or more photoelectric sensors capable of measuring the received light, as will be discussed herein. Figure 11 Further discussion is needed.

[0030] In some alternative embodiments, the stereo imager system can be an active system that emits light into a field and then detects the reflected light after the emitted light has been reflected from the surface of an object in the field. In addition to a light sensing module, the active stereo imager system may also include a light emitting module and can be configured as an optical ranging device. The light emitting module may comprise an emitter layer consisting of an array of individual emitters (e.g., vertical cavity surface-emitting lasers (VCSELs)), each emitter in which can be paired with a corresponding micro-optical receiver channel in the light sensing module and a corresponding photoelectric sensor in the sensor array; or the light emitting module may be a uniform illuminator that propagates light uniformly across a scene without specific pairing between individual emitters and receiver channels. In some cases, the light emitting module may include an array of micro-optical emitter channels to enhance the light output from the emitter array. During operation, the light output from the emitter array (e.g., a laser pulse) passes through the micro-optical emitter channel array and enters a volume emitter optics device with a large numerical aperture to better capture the light from the micro-optical emitter channel array. Then, the light leaves the body emitter optics and illuminates multiple spots in the far field.

[0031] In such an active stereo imaging system, the 2D sensor array can include a ranging photoelectric sensor array and an imaging photoelectric sensor array. The ranging photoelectric sensor array is used to receive light emitted by the transmitter array, such as narrowband laser, to calculate distance via time-of-flight measurement, etc. The imaging photoelectric sensor array is used to receive ambient red-green-blue (RGB) light for stereo imaging purposes, as will be discussed herein. Figure 7 and 8 Further discussion follows. By using both a ranging photoelectric sensor and an imaging photoelectric sensor, distance accuracy can be enhanced using distance data calculated from stereo images captured by the imaging photoelectric sensor.

[0032] To better understand the functionality and configuration of the passive stereo imaging system and the active stereo imaging system according to embodiments of this disclosure, each system will be discussed in detail herein.

[0033] I. Passive Stereo Imaging System

[0034] Figure 1 This is a block diagram of an example passive stereo imaging system 100 according to some embodiments of the present disclosure. The passive stereo imaging system 100 may include a system controller 104 and a light sensing module 106. Imaging data may be generated by the passive stereo imaging system 100 by receiving light present in the scene where the passive stereo imaging system 100 is located. In contrast to light emitted from a transmitter within the system 100, the received light may be naturally present light in the field, i.e., ambient light.

[0035] The light sensing module 106 may include a sensor array 108, which may be, for example, a two-dimensional photoelectric sensor array. Each photoelectric sensor may be a CCD, CMOS, or any other suitable sensor for detecting ambient light. The light sensing module 106 includes an optical sensing system 110, which, together with the sensor array 108, can form a light detection system 112. In some embodiments, the optical sensing system 110 may include a volume receiver optics 114 and optical components 116 such as an aperture layer, a collimating lens layer, and a filter. The optical sensing system may be combined with the sensor array 108 to form a micro-optical receiver channel array, wherein each micro-optical receiver channel measures light corresponding to image pixels in different fields of view of the surrounding field where the system 100 is located. Further details of various embodiments of the micro-optical receiver channels according to this disclosure are incorporated herein by reference. Figure 11 Let's discuss this in detail.

[0036] The volume imaging optics defined herein can be one or more optical surfaces that may contain multiple lens elements, have a transparent aperture greater than one millimeter, and be positioned to receive light projected from a micro-optical emitter / receiver layer or to focus received light onto a micro-optical emitter / receiver layer. Volume imaging optics that project light received from an optical emitter, such as a micro-optical emitter layer, are sometimes referred to herein as volume emitter optics or output volume imaging optics. Volume optical layers that focus light received from the field onto an optical detector, such as micro-optical receiver layers, are sometimes referred to herein as volume receiver optics or input volume imaging optics. Input image-space telecentric volume imaging optics allow the system to measure narrowband light uniformly located across a wide field of view (FOV).

[0037] In some embodiments, the sensor array 108 of the light sensing module 106 (using, for example, CMOS technology) is fabricated as part of a monolithic device on a single substrate, the monolithic device comprising both a photodetector array and a processor 118 and a memory 120 for signal processing of measured light from individual photodetectors (or groups of photodetectors) in the array. The monolithic structure comprising the sensor array 108, processor 118, and memory 120 can be fabricated as a dedicated ASIC. In some embodiments, the optical component 116 can also be part of a monolithic structure in which the sensor array 108, processor 118, and memory 120 are part. In this case, the optical component 116 can be formed on an ASIC, for example, by bonding (irreversibly) with epoxy resin, making it part of the monolithic structure. As mentioned above, the processor 118 (e.g., a digital signal processor (DSP), microcontroller, field-programmable gate array (FPGA), etc.) and memory 120 (e.g., SRAM) can perform signal processing. As an example of signal processing, for each photoelectric sensor or group of photoelectric sensors, the memory 120 of the light sensing module 106 can accumulate detected photons over time, and these detected photons can be used to reconstruct an image of the field.

[0038] In some embodiments, output from processor 118 is sent to system controller 104 for further processing. For example, data may be encoded by one or more encoders of system controller 104 and then sent as data packets to user interface 115. System controller 104 may be implemented in various ways, including, for example, by using a programmable logic device such as an FPGA as part of an ASIC, using processor 122 with memory 124, and some combinations thereof. According to some embodiments of this disclosure, processor 122 may receive stereo images from light sensing module 106 and use these images to generate depth-aware images and calculate distances to objects in the scene, as will be discussed further herein. System controller 104 may cooperate with or operate independently of a fixed base controller (via pre-programmed instructions) to control light sensing module 106 by sending commands including start and stop light detection and adjustment of photodetector parameters. In some embodiments, system controller 104 has one or more wired interfaces or connectors for exchanging data with light sensing module 106. In other embodiments, system controller 104 communicates with light sensing module 106 via a wireless interconnect such as an optical communication link.

[0039] According to some embodiments of this disclosure, the passive stereo imager system 100 may further include a moving component 126 coupled to a system controller 104 and a light sensing module 106. The moving component 126 may be controlled by the system controller 104 to move the field of view of the sensor array 108. In some cases, the movement of the sensor array 108 may be achieved by physically moving the light sensing module 106 or by redirecting the field of view of the light sensing module 106 by means of light reflection. For example, the moving component 126 may be a motor (e.g., an electric motor) that rotates the light sensing module 106 about an axis perpendicular to the row of photoelectric sensors in the sensor array 108, as will be discussed herein. Figure 4 Further discussion in A-4C. Alternatively, the moving part 126 can be a light redirection part, such as a microelectromechanical system (MEMS) device, which can be modulated to reflect light in different directions from a two-dimensional light emitter array to capture two offset 2D images of the field. The two offset 2D images can then be used to generate stereo images of the scene for calculating distance measurements and / or for generating depth-aware images.

[0040] A. Optical detection system for stereo imaging

[0041] Figure 2 This is a top view of an example light detection system 200 configured to perform stereoscopic imaging according to some embodiments of the present disclosure. The light detection system 200 may include a sensor array 202 mounted on a heat sink 204 for dissipating heat generated by the sensor array 202 during operation. The sensor array 202 may be positioned behind a volume receiver optics 206 such that light propagates through the volume receiver optics 206 and is then exposed on the sensor array 202. The light detection system 200 may be enclosed within a light-transmitting housing 208 to protect the light detection system 200 from environmental influences.

[0042] In some embodiments, sensor array 202 may include a plurality of photoelectric sensors arranged to capture two offset 2D images of a scene. For example, sensor array 202 may include more than one photoelectric sensor array spaced apart from each other, or a single photoelectric sensor array having a subset of photoelectric sensors spaced apart from each other, such that two offset 2D images can be captured as the field of view of the photoelectric sensors moves. For example, sensor array 202 may include a first photoelectric sensor array 210 and a second photoelectric sensor array 212 separated by gap 214. The first photoelectric sensor array 210 and the second photoelectric sensor array 212 may each be an m×n photoelectric sensor array, and the field of view size of both the first photoelectric sensor array 210 and the second photoelectric sensor array 212 may be the same.

[0043] In some cases, the first photoelectric sensor array 210 and the second photoelectric sensor array 212 can capture light from a principal ray propagating in different directions. For example, the first photoelectric sensor array 210 can capture light from a principal ray 220 propagating in a first direction, and the second photoelectric sensor array 212 can capture light from a principal ray 222 propagating in a second direction, wherein the first and second directions intersect each other. Thus, when the field of view of sensor array 202 moves, for example, by a physical rotation 216 about the central axis 218, the field of view of the first photoelectric sensor array 210 can overlap with the field of view of the second photoelectric sensor array 212 at different time instances during the rotation of the light detection system 200, and each array can capture the same image of the scene, but using a gap 214 as an optical baseline for stereo imaging and distance calculation purposes to capture from different viewpoints. A better understanding of this operation can be found in the references herein. Figures 3A-3C To understand.

[0044] Figures 3A-3B This is a simplified top view of a rotating light detection system 200 according to some embodiments of the present disclosure during different time instances of an image capture sequence, and Figure 3C This is a top view of a light detection system 200 according to some embodiments of the present disclosure during different time instances superimposed on each other. Specifically, Figure 3A A first-time instance of the light detection system 200 is shown, and Figure 3B A second time instance is shown after the first time instance, but before the light detection system 200 has performed a full 360° rotation around its central axis, such as central axis 218 in Figure 3. During operation, the light detection system 200 can continuously rotate 360° while its photoelectric sensors are activated at high frequency to capture images of its surrounding environment, such as its scene, for stereo imaging and distance calculation.

[0045] like Figure 3A As shown, the optical detection system 200 can be as follows: Figure 3A The example shown is positioned while rotating around its central axis 218 in the first moment. At this time, the first photoelectric sensor array 210 of the sensor array 202 can be positioned such that its field of view 300 captures an image 301 of the tree 302 from a first viewing angle via the main ray 220 passing through the body receiver optics 206. Simultaneously, the second photoelectric sensor array 212 can be positioned such that its field of view 304 captures an image of the region to the right of the tree 302 via the main ray 222 passing through the body receiver optics 206.

[0046] As the optical detection system 200 continues to rotate, it can... Figure 3BThe second time instance is shown, but positioning occurs before the complete rotation of the positioning light detection system 200 begins in the first time instance. In the second time instance, the second photoelectric sensor array 212 can be positioned such that its field of view 304 captures an image 303 of the tree 302 from a second viewpoint via the main ray 222 passing through the volume receiver optics 206. Simultaneously, the first photoelectric sensor array 210 can be positioned such that its field of view 300 captures an image of the region to the left of the tree 302 via the main ray 220 passing through the volume receiver optics 206. The second viewpoint can differ from the first viewpoint, such that images 301 and 303 form an offset image pair of the tree 302.

[0047] In some embodiments, the field of view 300 of the first photoelectric sensor array 210 has the same size as the field of view 304 of the second photoelectric sensor array 212. In such embodiments, the size and shape of the first photoelectric sensor array 210 may be equal to the size and shape of the second photoelectric sensor array 212. For example, the first photoelectric sensor array 210 and the second photoelectric sensor array 212 may be m×n photoelectric sensor arrays with a certain spacing suitable for capturing images of trees 302 in the scene. Therefore, the field of view of the second photoelectric sensor array 212 in a second time instance may completely overlap with the field of view of the first photoelectric sensor array 210 in a first time instance. Thus, the image 301 of the tree 302 captured by the first photoelectric sensor array 210 in the first time instance and the image 303 of the tree 302 captured by the second photoelectric sensor array 212 in the second time instance may be images of the same field of view of the scene but captured from two different perspectives. Image pairing at different fields of view can be continuously repeated as the light detection system 200 continuously rotates about its central axis to capture images of the surrounding scene.

[0048] The difference in perspective can be achieved by relatively positioning the first photoelectric sensor array 210 and the second photoelectric sensor array 212 relative to a tree 302 where images 301 and 303 are captured. For example... Figure 3C As shown, the position of the image 301 of the tree 302 captured by the first photoelectric sensor array 210 is offset from the position of the image 303 of the tree 302 captured by the second photoelectric sensor array 212 by a distance 308 defined by the gap 214. Therefore, the distance 308 is an optical baseline that can be used to triangulate the distance from the surface of the tree 302 to the light detection system 200 and can be used for depth sensing purposes. (See reference...) Figure 3C Understandably, due to the fact that the rotation axis, such as the central axis 218 in Figures 3 and 3A-3B, is not located at the center of the sensor array 202, but rather below it, the distance 308 may be greater than the gap 214. Therefore, in Figures 3A-3CIn the illustrated embodiment, distance 308 is defined by, but not equal to, 214. However, this is not intended to be limiting, as other embodiments may position the axis of rotation at the center of sensor array 202, where the optical baseline is equal to the gap between the photoelectric sensor arrays.

[0049] As can be understood from the above discussion, the two images captured in the first time instance by the first photoelectric sensor array 210 and in the second time instance by the second photoelectric sensor array 212 can be images of the same field of view of the scene, but captured from two different perspectives, thus forming two offset images suitable for stereo imaging and distance calculation purposes. Distance can be calculated by comparing the two images and measuring the separation distance between identical features of obvious objects in the two images. An example of this comparison is discussed herein with reference to Figure 5.

[0050] Figure 4 This is a simplified illustration of two offset images 301 and 303 captured by a first photoelectric sensor array 210 and a second photoelectric sensor array 212 respectively in a first time instance and a second time instance, according to some embodiments of this disclosure. Considering the difference in viewing angle, image 301 captured by the first photoelectric sensor array 210 in the first time instance may show a different tree 302 than the tree in image 303 captured by the second photoelectric sensor array 212 in the second time instance. For example, tree 302 may appear laterally compressed in image 303. In the case where the two images 301 and 303 are superimposed on each other (for clarity, see image 303),... Figure 4 (As shown in the image, which is taken next to each other in opposite directions), the distances between the same features of objects in the image, such as the distances between the same branches, trunks and roots, 400, 402 and 404 respectively, can be used for triangulation purposes.

[0051] It should be understood that Figures 3A-3C The rotational movement of the light detection system 200 shown is merely one way to move the field of view of the light detection system 200, and the embodiments are not intended to be so limiting. Other types of mechanisms for moving the field of view to achieve stereoscopic imaging are envisioned herein. For example, a movable mirror, such as a rotating mirror / MEMS device, can be used to reflect the main light rays captured by the photoelectric sensors of the sensor array, allowing the photoelectric sensors to capture two offset images of the scene as discussed herein.

[0052] B. Sensor array for passive stereo imaging

[0053] As mentioned earlier in this document, a sensor array for achieving stereo imaging can include photoelectric sensors spaced apart from each other such that two offset 2D images can be captured as the field of view of the sensor array moves. According to some embodiments, this sensor array can be constructed in various ways. For example, the sensor array can include more than one photoelectric sensor array spaced apart from each other, or the sensor array can include a single photoelectric sensor array having a subset of photoelectric sensors spaced apart from each other, as will be discussed herein. Figures 5A-5C Further discussion is needed.

[0054] Figures 5A-5C This is a simplified illustration of example sensor arrays configured in various ways. Specifically, Figure 5A This is a simplified illustration of an example sensor array 500, which consists of two separate linear photoelectric sensor arrays configured to be spaced apart from each other for stereo imaging, according to some embodiments of the present disclosure. Figure 5B This is a simplified illustration of an example sensor array 501, consisting of two separate m×n photoelectric sensor arrays configured to be spaced apart from each other for stereo imaging, according to some embodiments of this disclosure. Figure 5C This is a simplified illustration of an example sensor array 503, which is a single mxn photoelectric sensor array configured to have two subsets of photoelectric sensors spaced apart from each other for stereo imaging, according to some embodiments of the present disclosure.

[0055] like Figure 5A As shown, the sensor array 500 may include a first photoelectric sensor array 502 and a second photoelectric sensor array 504 separated by a gap 506. In some embodiments, the first photoelectric sensor array 502 and the second photoelectric sensor array 504 may each be a linear photoelectric sensor array arranged in a vertical orientation. To achieve stereoscopic image capture, each photoelectric sensor 508 in the first photoelectric sensor array 502 may be located in the same rotation path as the corresponding photoelectric sensor 510 in the second photoelectric sensor array 504. For example, when the sensor array 500 is rotated around a vertical axis 512, for example... Figure 2 When the central axis 218 rotates, the corresponding photoelectric sensors 508 and 510 can be positioned on the same horizontal rotation path. Thus, when the field of view of sensor array 500 moves, for example by physical rotation, the field of view of the first photoelectric sensor array 502 can overlap with the field of view of the second photoelectric sensor array 504 to capture the same image of the scene, but from different perspectives, as discussed above. Figures 3A-3B As mentioned.

[0056] The gap 506 can be defined by the distance between corresponding photoelectric sensors positioned along the direction of movement between the first photoelectric sensor array 502 and the second photoelectric sensor array 504. For example, if the sensor array 500 rotates about the vertical axis 512, causing its field of view to move horizontally across the scene, the gap 506 can be defined by corresponding photoelectric sensors positioned along the horizontal line, such as the fourth photoelectric sensor starting from the top of the first photoelectric sensor array 502 and the fourth photoelectric sensor starting from the top of the second photoelectric sensor array 504. Figure 5A As shown. Thus, when the sensor array 500 rotates about the vertical axis 512 to capture images for stereoscopic imaging purposes, the fields of view of those photoelectric sensors can overlap. As discussed herein, gap 506 can be used to determine an optical baseline for calculating distances to objects in the scene and generating a depth-aware image. Therefore, gap 506 can be any distance suitable for providing an optical baseline that can be used for triangulation to calculate distances to objects in the scene. For example, gap 506 can range from 4 cm to 8 cm, such as approximately 6 cm.

[0057] It should be understood that gap 506 is not limited to corresponding photoelectric sensors positioned along a horizontal line. For example, if the axis of rotation is positioned diagonally, such as diagonal axis 514, then the gap defining the optical baseline can be defined by the distance between corresponding photoelectric sensors positioned along a line transverse to diagonal axis 514, such as diagonal rotation line 516. Therefore, the corresponding photoelectric sensors would be the third photoelectric sensor from the bottom in the first photoelectric sensor array 502 and the third photoelectric sensor from the top in the second photoelectric sensor array 502, as... Figure 5A As shown. Thus, when the sensor array 500 rotates about the diagonal axis 514 to capture images for stereo imaging purposes, the fields of view of those photoelectric sensors can overlap. It should be understood that the sensor array 500 can rotate in any direction for stereo imaging, and therefore can be used to define the gap for establishing an optical baseline based on the direction of field of view movement without departing from the spirit and scope of this disclosure.

[0058] In addition to linear arrays, some sensor arrays can also have photoelectric sensors arranged in the form of two separate m x n arrays, where m and n are greater than 1. For example, such as Figure 5BAs shown, the first photoelectric sensor array 522 and the second photoelectric sensor array 524 can each be formed by 5x15 photoelectric sensor arrays spaced apart by a distance 526. Having a greater number of photoelectric sensors in each array compared to sensor array 500 allows each photoelectric sensor array 522 and 524 to have a larger field of view than photoelectric sensor arrays 502 and 504, making it possible to capture fewer images to image the scene when sensor array 501 is rotated 350°. The gap used to determine the optical baseline may not be limited by the distance 526 between arrays 522 and 524, because the field of view of the photoelectric sensor located at the right edge of the first photoelectric sensor array 522 may not correspond to the field of view of the photoelectric sensor located at the left edge of the second photoelectric sensor array 522 for stereo imaging and distance calculation. Instead, the distance between corresponding photoelectric sensors in the first photoelectric sensor array 522 and the second photoelectric sensor array 524 can represent the gap used for distance calculation and depth perception.

[0059] For example, the gap 528 defined by the distance between corresponding photoelectric sensors, such as the upper left photoelectric sensor 520 of the first photoelectric sensor array 522 and the upper left photoelectric sensor 522 of the second photoelectric sensor array 524, can be used to determine an optical baseline to calculate distance and generate depth perception. Furthermore, since the fields of view of the two arrays overlap, the length of the gap 528 can be common across all corresponding photoelectric sensors. For example, the length of the gap 534 defined by the distance between corresponding photoelectric sensors, such as the lower right photoelectric sensor 536 of the first photoelectric sensor array 522 and the lower right photoelectric sensor 538 of the second photoelectric sensor array 524, and all other gaps between corresponding photoelectric sensors, can be equal to the length of the gap 528. In this way, the field of view of the first photoelectric sensor array 522 can overlap with the field of view of the second photoelectric sensor array 522 to capture two offset images of the scene for stereo imaging and distance calculation purposes.

[0060] although Figure 5A and 5B A sensor array is shown with two photoelectric sensor arrays separated by a sensor array that does not have a photoelectric sensor, but the embodiments are not limited to this configuration. Instead, embodiments can be implemented in an imager system in which the sensor array is configured as a single two-dimensional photoelectric sensor array. For example, refer to Figure 5CSensor array 503 may comprise a single 20x15 photoelectric sensor array 540. Sensor array 503 may be configured to perform stereo imaging and distance calculation by allocating images captured by a subset of photoelectric sensors for stereo imaging and distance calculation purposes. Those subsets may be strategically positioned such that two offset images can be captured as the field of view of sensor array 503 moves (e.g., by rotation).

[0061] For example, the photoelectric sensor array 540 may comprise two photoelectric sensor array subsets: a first photoelectric sensor subset 542 and a second photoelectric sensor subset 544 spaced apart from each other. The size, configuration, and operation of the first photoelectric sensor subset 542 and the second photoelectric sensor subset 544 may be consistent with the description herein. Figure 5B The dimensions, configuration, and operation of the first photoelectric sensor array 522 and the second photoelectric sensor array 524 discussed correspond to each other. The distance between the first photoelectric sensor subset 542 and the second photoelectric sensor subset 544 can be filled with other photoelectric sensors 546 that may not be used for stereo imaging and distance calculation purposes but only for capturing 2D images of the scene. As will be understood herein, some embodiments can utilize image sensors typically used for 2D image capture for stereo imaging and distance calculation in a unique manner, and may therefore be more cost-effective and simpler in design.

[0062] although Figures 5A-5C Example sensor arrays with linear, 5x15, and 20x15 arrays are shown, but the embodiments are not limited to such configurations. It should be understood that the embodiments herein can have any number, size, and arrangement of photoelectric sensors suitable for stereo imaging and distance calculation purposes without departing from the spirit and scope of this disclosure.

[0063] II. Active Stereo Imaging System

[0064] As discussed herein, stereo imaging systems can also be configured as active stereo imaging systems. Active stereo imaging systems can differ from passive stereo imaging systems because they can also emit their own light into the field and detect the reflected light after it has been reflected from the surface of an object in the field. In some embodiments, active stereo imaging systems can also function as LIDAR devices in which reflected light can be correlated during emission and reception to determine the distance to the object from which the emitted light is reflected. Active stereo imaging systems can collect a large number of distance data points, and these distance data points can be processed to form a three-dimensional point cloud representing a scene in the field of view of the system, as captured by a LIDAR device. A better understanding of active stereo imaging systems can be found in [reference needed]. Figure 6 To determine.

[0065] Figure 6 This is a block diagram of an active stereo system 600 according to some embodiments of the present disclosure. The active stereo system 600 may include an optical ranging device 602 and a user interface 615. The optical ranging device 602 may include a ranging system controller 604, a light emitting (Tx) module 606, and a light sensing (Rx) module 608. Ranging data can be generated by the optical ranging device 602 by emitting one or more light pulses 610 from the light emitting module 606 onto objects in the field of view surrounding the optical ranging device 602. The light sensing module 608 then detects the reflected portion 612 of the emitted light after a certain delay time. Based on the delay time, the distance to the reflecting surface can be determined. Other ranging methods, such as continuous wave, Doppler, etc., may also be used.

[0066] Tx module 606 includes an emitter array 614 and a Tx optical system 616. The emitter array can be a one-dimensional or two-dimensional emitter array, and the Tx optical system, together with the emitter array 614, can form a light emission system 638. The Tx optical system 616 may include an image-spatial telecentric volume emitter optics. In some embodiments, the Tx optical system 616 may further include one or more micro-optical structures that increase the brightness of the beam emitted from the volume emitter optics and / or are used for beam shaping, beam manipulation, etc. The emitter array 614 or a single emitter may be a laser source. Tx module 606 may further include an optical processor 618 and a memory 620, but in some embodiments, these computing resources may be incorporated into the ranging system controller 604. In some embodiments, pulse coding techniques, such as Barker codes, may be used. In such cases, the memory 620 may store pulse codes indicating when light should be emitted. In some embodiments, the pulse codes are stored as a sequence of integers stored in the memory.

[0067] The optical sensing module 608 can be constructed similarly to the one referenced in this article. Figure 1 The light sensing module 106 discussed is essentially similar. Therefore, details of the processor 622, memory 624, sensor array 626, and Rx optical system 628 (which, together with the sensor array 626, can form a light detection system 636) can be found in this document. Figure 1For reference, and for the sake of brevity, this document only discusses differences regarding those components. For the active stereo system 600, sensor array 626 may include ranging photoelectric sensors. Each ranging photoelectric sensor may be a miniature array of multiple single-photon avalanche detectors (SPADs) or a single photon detector (e.g., an APD). In some embodiments, the ranging photoelectric sensors of sensor array 626 may correspond to a specific emitter of emitter array 614, for example, as a result of the geometric configuration of light sensing module 608 and Tx module 606. For example, in some embodiments, emitter array 614 may be arranged along the focal plane of the volume emitter optics such that each illumination beam projected from the volume emitter optics into the field in front of the system has substantially the same size and geometry as the field of view of the corresponding receiver channel at any distance from the system exceeding an initial threshold distance. In addition to ranging photoelectric sensors, sensor array 626 may also include imaging photoelectric sensors. For example, imaging photoelectric sensors such as CCD or CMOS sensors may be positioned and configured to capture images for stereo imaging and distance calculation purposes. Such sensor arrays are discussed herein with respect to... Figure 8 Further discussion.

[0068] In some embodiments, processor 618 may perform signal processing on the raw histograms from individual photon detectors (or groups of detectors) in the array. As an example of signal processing, for each photon detector or group of photon detectors, memory 624 (e.g., SRAM) may accumulate counts of detected photons within consecutive time selections, and these time selections together may be used to reconstruct the time series of reflected light pulses (i.e., the photon counts over time). This time series of accumulated photon counts is referred to herein as an intensity histogram (or histogram only). Processor 618 may implement matched filtering and peak detection processing to timely identify the returned signal. Additionally, processor 618 may perform certain signal processing techniques (e.g., by processor 622), such as multi-profile matched filtering, to help recover the photon time series that is less susceptible to pulse shape distortion that may occur due to SPAD saturation and quenching. In some embodiments, all or part of such filtering may be performed by processor 458, which may be embodied in an FPGA.

[0069] In some embodiments, the photon time series output from processor 618 is sent to ranging system controller 604 for further processing. For example, the data may be encoded by one or more encoders of ranging system controller 604 and then sent to user interface 615 in the form of data packets. Ranging system controller 604 can be implemented in various ways, including, for example, using a programmable logic device such as an FPGA as part of an ASIC, using a processor 630 with memory 632, and some combinations thereof. Ranging system controller 604 can cooperate with or operate independently of a fixed base controller (via pre-programmed instructions) to control light sensing module 608 by sending commands including start and stop light detection and adjustment of photodetector parameters. Similarly, ranging system controller 604 can control light emitting module 606 by sending commands or relaying commands from a base controller that includes start and stop light emission controls and controls for adjusting other light emitting parameters (e.g., pulse codes). In some embodiments, the ranging system controller 604 has one or more wired interfaces or connectors for exchanging data with the light sensing module 608 and the light emitting module 606. In other embodiments, the ranging system controller 604 communicates with the light sensing module 608 and the light emitting module 606 via a wireless interconnect, such as an optical communication link.

[0070] An optical ranging device 602 can be used in the scanning architecture in which the Rx module 608 and Tx module 606 physically rotate together via an electric motor 634, or the field of view rotates via a mirror device such as a MEMS device while the Rx module 608 and Tx module 606 are stationary. Therefore, the electric motor 634 is an optional component in the active stereo imager system 600 that can be used to rotate system components, such as the Tx module 606 and Rx module 608, which are part of the LIDAR and stereo imaging capture architecture. The system controller 604 can control the electric motor 634 and can start, stop, and change the rotation speed as needed to implement the scanning system.

[0071] The active stereo imaging system 600 can interact with one or more instances of the user interface 615. These instances can vary and may include, but are not limited to: a computer system with a monitor, keyboard, mouse, CPU, and memory; a touchscreen in an automobile or other vehicle; a handheld device with a touchscreen; or any other suitable user interface. The user interface 615 can be located locally on the object where the active stereo imaging system 600 is mounted, but it can also be a remotely operated system. For example, commands and data arriving at / from the active stereo imaging system 600 can be routed via cellular networks (LTE, etc.), personal area networks (Bluetooth, Zigbee, etc.), local area networks (WiFi, IR, etc.), or wide area networks such as the Internet.

[0072] The hardware and software user interface 615 can present LIDAR data from the device to a user or vehicle control unit (not shown), but can also allow the user to control the active stereo imaging system 600 with one or more commands. Example commands may include: activating or deactivating the active stereo imaging system; specifying the photodetector exposure level, bias, sampling duration, and other operating parameters (e.g., emitted pulse pattern and signal processing); specifying light emitter parameters, such as brightness. Additionally, commands may allow the user to select a method for displaying results. The user interface can display results from the active stereo imaging system, which may include, for example, a single-frame snapshot image, continuously updated video images, and / or the display of other light measurements of some or all pixels. In some embodiments, the user interface 615 may track the distance (proximity) of an object to the vehicle and potentially provide alerts to the driver or provide such tracking information for analyzing driver performance.

[0073] In some embodiments, such as when the active stereo imaging system 600 is used for vehicle navigation, the user interface 615 may be part of a vehicle control unit that receives output from or communicates with the optical ranging device 602 and / or the user interface 615 via a network such as the wired or wireless networks described above. The vehicle control unit may modify one or more parameters associated with vehicle control based on the received ranging data. For example, in a fully autonomous vehicle, the active stereo imaging system 600 may provide real-time 3D images of the environment surrounding the vehicle to aid navigation in conjunction with GPS and other data. In other cases, the active stereo imaging system 600 may be used as part of an advanced driver assistance system (ADAS) or as part of a safety system that may provide 3D image data to any number of different systems, such as adaptive cruise control, automatic parking, driver fatigue monitoring, blind spot monitoring, collision avoidance, etc. When the user interface 615 is implemented as part of a vehicle control unit, it may provide alerts to the driver or track the proximity of objects.

[0074] Figure 7 This is a simplified diagram illustrating detailed views of an exemplary active stereo imager system 700 having a wide field of view and capable of narrowband imaging according to some embodiments of the present disclosure. Unlike passive stereo imager systems, the active stereo imager system 700 may include both a light detection system 701 and a light emission system 702. The light emission system 702 provides active illumination of at least a portion of the field in which the system 700 is positioned using narrowband light 704. The light detection system 701 detects the light after the narrowband light emitted from the light emission system 702 has been reflected as a reflected ray 706 by an object in the field.

[0075] A. Optical emission system

[0076] In some embodiments, the light emitting system 702 includes a volume emitter optics 718 and a light emitting layer 720 formed by an array of one-dimensional or two-dimensional light emitters 722. Each light emitter 722 can be configured to generate a discrete narrowband beam. In some embodiments, the light emitting layer 720 is configured to project discrete beams through the volume emitter optics 718 according to an illumination pattern that matches the field of view of a corresponding receiver channel in the micro-optical receiver channel array 714 across a range of distances in size and geometry from the light emitting system 702. The light emitters 722 can be any suitable light emitting device, such as a vertical cavity surface-emitting laser (VCSEL) integrated on one or more monolithic chips or any other type of laser diode. The light emitters 722 can generate a narrowband light cone 724 that is guided to the volume emitter optics 718, which can collimate the light cone 724 and then output the collimated light as an emitted ray 704 to a distant target within the field. In some embodiments, the volume emitter optics 718 is image-spatial telecentric.

[0077] B. Optical Detection System

[0078] When the light emitting system 702 emits light into a field, once the emitted light has been reflected from an object in the field, the corresponding ranging photoelectric sensor in the light detection system 701 can receive the reflected light. The received reflected light can be used to determine the distance to the object in the field. According to some embodiments of this disclosure, the light detection system 701 may also include an imaging photoelectric sensor for performing stereoscopic imaging, as will be discussed further herein.

[0079] The light detection module 701 can represent the above regarding Figure 1 The discussed light detection system 112 may include a volume receiver optics 708 and a micro-optical receiver (Rx) layer 714. During operation, light rays 706 enter the volume receiver optics 708 from multiple directions and are focused by the volume receiver optics 708 to form a light cone 710. The micro-optical receiver layer 714 is positioned such that an aperture 726 coincides with the focal plane of the volume receiver optics 708. In some embodiments, the micro-optical receiver layer 714 may be a one-dimensional or two-dimensional array of micro-optical receiver channels 712, wherein each micro-optical receiver channel 712 is positioned along the same axis in the direction of light flow, for example... Figure 2 The diagram shows a horizontally positioned aperture 726, collimating lens 728, and photoelectric sensor 716 arranged from left to right. Furthermore, each micro-optical receiver channel 712 can be configured in various ways to mitigate interference from stray light from between the photoelectric sensors, as will be discussed further herein. During operation, each micro-optical receiver channel 712 measures the light information (i.e., position in the field) of a different pixel.

[0080] At the focal point of the volume receiver optics 708, light 706 is focused and passes through the aperture 726 in the aperture layer 711, entering the corresponding collimating lens 728. Each collimating lens 728 collimates the received light so that the light rays all enter the filter at approximately the same angle, for example, parallel to each other. The aperture and the focal length of the volume receiver optics 708 determine the cone angle of the corresponding light rays focused at the aperture 726. The aperture size and the focal length of the collimating lens 728 determine the degree of collimation allowed, which determines how narrow the bandpass can be implemented in the filter 730 to block unwanted wavelengths of light. During the operation of the light detection system 701, the aperture 726 can serve various functions. For example, aperture 726 can (1) constrain the pixel FOV so that it has tight spatial selectivity despite the large spacing at the photoelectric sensor plane, (2) provide a small point source at the focal plane of the collimating lens to achieve tight collimation of light before it passes through the filter, wherein better collimation produces a tighter band that can pass through the filter, and (3) repel stray light.

[0081] In some embodiments, the photodetector 716 is positioned on the side opposite the collimating lens 728, such that light 706 first passes through the collimating lens 728 and the filter 730 before being exposed to the photodetector 716. Some photodetectors 716 may be ranging photodetectors configured to receive the emitted light, such as a miniature array of multiple photodetectors, for example, multiple single-photon avalanche detectors (SPADs). An array of SPAD miniature arrays can be fabricated on a single monolithic chip, thereby simplifying manufacturing. In some alternative embodiments, each photodetector 716 may be a single photodetector, such as a standard photodiode, an avalanche photodiode, a resonant cavity photodiode, or another type of photodetector. Other photodetectors may be configured as imaging photodetectors for stereoscopic imaging purposes, as will be discussed further herein.

[0082] C. Sensor array for active stereo imaging

[0083] As discussed above, an active stereo imager system can have a sensor array comprising a ranging photoelectric sensor and an imaging photoelectric sensor. The ranging photoelectric sensor can be associated with an emitter, allowing it to capture light emitted by the emitter, such as a narrowband laser, to calculate distance via time-of-flight measurements, etc. The imaging photoelectric sensor may not be configured to capture the emitted light, but instead can be configured to capture an offset red-green-blue (RGB) image of the scene from ambient light for stereo imaging and distance calculation purposes. In such embodiments, the distance calculated via stereo imaging can be used to enhance measurements performed by the ranging photoelectric sensor and / or fill measurement gaps where the ranging photoelectric sensor may be less accurate, such as measurements at very close ranges (e.g., 0-2 meters). Examples of such sensor arrays are discussed herein. Figure 8 Let's have a discussion.

[0084] Figure 8 This is a simplified illustration of an example sensor array 800 for an active stereo imager system according to some embodiments of the present disclosure. Sensor array 800 may include a ranging photoelectric sensor array 802 positioned between two imaging photoelectric sensor arrays: a first imaging photoelectric sensor array 804 and a second imaging photoelectric sensor array 806. The ranging photoelectric sensor array 802 may be a photoelectric sensor of the same type as the first imaging photoelectric sensor array 804 and the second imaging photoelectric sensor array 806. For example, the ranging photoelectric sensor 802 and the imaging photoelectric sensors 804 and 806 may be SPADs or other avalanche diode sensors configured to measure narrowband lasers. However, the light sensed by either photoelectric sensor array may be different. For example, the ranging photoelectric sensor 802 may sense light emitted from an emitter array, while the imaging photoelectric sensors 804 and 806 are configured to sense ambient light. To enable the imaging photoelectric sensors 804 and 806 to measure light, filters in the receiver channels of the corresponding photoelectric sensors 804 and 806 may be configured to allow certain wavelengths of visible light.

[0085] In some additional and alternative embodiments, the ranging photoelectric sensor array 802 may be a different type of photoelectric sensor than the first imaging photoelectric sensor array 804 and the second imaging photoelectric sensor array 806. For example, the ranging photoelectric sensor array 802 may be a SPAD or other avalanche diode sensor, while the first imaging photoelectric sensor array 804 and the second imaging photoelectric sensor array 806 may be CCDs, CMOS sensors, and other similar types of photoelectric sensors configured to measure wider bandwidth light, such as visible RGB ambient light.

[0086] Due to its correlation with the transmitter array, the ranging photoelectric sensor array 802 can have an array size and photoelectric sensor spacing suitable for receiving light emitted by the transmitter array, while the first imaging photoelectric sensor array 804 and the second imaging photoelectric sensor array 806 can have array sizes and photoelectric sensor spacings different from both the transmitter array and the ranging photoelectric sensor array 802. For example, as Figure 8 As shown, the ranging photoelectric sensor array 802 can have a diagonally staggered arrangement shifted downwards to the right, and the first imaging photoelectric sensor array 804 and the second imaging photoelectric sensor array 806 can each have a two-dimensional rectangular mxn arrangement of imaging photoelectric sensors that are more densely packed than those in the ranging photoelectric sensor array 802. In such an embodiment, each ranging photoelectric sensor, such as ranging photoelectric sensor 808, can correspond to an imaging photoelectric sensor group, such as imaging photoelectric sensor group 810, located in the same row as the corresponding ranging photoelectric sensor 808, such that when the sensor array 800 rotates around to image the scene, the imaging photoelectric sensor group 810 can capture ambient light present in the same field of view as the ranging photoelectric sensor 808 when each photoelectric sensor in the imaging photoelectric sensor group 810 is correspondingly positioned. Therefore, the light captured by the sensor array 800 can be used to construct a topographic map of the scene, as well as an RGB image of the scene that is highly correlated with the topographic map of the scene.

[0087] Aside from differences in device type, spacing, and arrangement, the ranging photoelectric sensor array 802 can be constructed from photoelectric sensors of a different size than the first imaging photoelectric sensor array 804 and the second imaging photoelectric sensor array 806. For example, the ranging photoelectric sensor can have a larger size than the image photoelectric sensor. In this way, the ranging photoelectric sensor can be better equipped to measure the emitted light reflected from objects in the scene, such as by constructing it from a SPAD.

[0088] According to some embodiments of this disclosure, a subset of imaging photoelectric sensors from the first imaging photoelectric sensor array 804 and the second imaging photoelectric sensor array 806 can be allocated to provide sensors that can be used for stereoscopic imaging purposes, as described herein. Figures 3A-3C Data for stereoscopic imaging purposes discussed in 5A-5B. For example, a first imaging optosensor subset 812 from the first optosensor array 804 and a second imaging optosensor subset 814 from the second optosensor array 804 can be used for stereoscopic imaging purposes. In some embodiments, the first imaging optosensor subset 812 and the second imaging optosensor subset 814 may each comprise a portion of the entire imaging optosensor array in the first optosensor array 804 and the second imaging optosensor array 806, such as... Figure 8As shown. This may be because those imaging photoelectric sensors that are not part of the subset of imaging photoelectric sensors (e.g., the top two rows of the first imaging photoelectric sensor array 804 and the bottom two rows of the second imaging photoelectric sensor array 806) may not have corresponding photoelectric sensors in the other array 804 / 806 positioned on the same rotational path (e.g., the same horizontal line in an example where sensor array 800 rotates about a vertical axis). Therefore, those imaging photoelectric sensors may not have corresponding offset images for stereo imaging purposes.

[0089] For reference Figure 8 Understandably, imaging optoelectronic sensor subset 812 can be spaced apart from imaging optoelectronic sensor subset 814, and therefore can be corresponding imaging optoelectronic sensor subsets used to generate two offset images of a scene to calculate distance and produce depth perception. Imaging optoelectronic sensor subsets 812 and 814 can be related to the subsets discussed in this paper. Figure 5B and 5C The photoelectric sensor arrays 622 and 624 discussed, as well as the photoelectric sensor subsets 642 and 644, have similar features and arrangements.

[0090] In some embodiments, the accuracy of the distance measured by the ranging photoelectric sensor array 802 can be enhanced and / or replaced by distance measurements calculated based on analysis of two offset stereo images captured by the imaging photoelectric sensor subsets 812 and 814. For example, the accuracy of measurements taken by the ranging photoelectric sensor array 802 at very close distances (e.g., 0-2 meters) may be lower than its accuracy at more distant distances (e.g., 2+ meters). Therefore, those distances calculated by the ranging photoelectric sensor array 802 can be enhanced by distances calculated through analysis of the stereo image pair. One way to enhance the measurement results is by averaging the two distances together. This allows for the determination of more accurate distance measurements. In some additional and alternative conventional embodiments, those distances calculated by the ranging photoelectric sensor array 802 can be enhanced by distances calculated through analysis of the stereo image pair. In this case, those distances calculated by the ranging photoelectric sensor array 802 can simply be ignored. Either form of enhancement can be performed by default, or in response to determining that the difference between the first and second distances is greater than a threshold.

[0091] III. Methods for operating a stereo imaging system

[0092] Figure 9 This is a block diagram of an example method 900 for operating a stereo imager system to perform stereo imaging using a light detection system, according to some embodiments of the present disclosure. At step 902, the field of view of the sensor array can be moved to capture an image of the scene. For example, the sensor array can be rotated about a central axis, as described herein. Figures 3A-3CThe discussed, or one or more mirrors, such as MEMS devices, can be moved to change the field of view of the sensor array. The sensor array may include a first photoelectric sensor and a second photoelectric sensor spaced apart from the first photoelectric sensor to capture an image of the scene. For example, the first photoelectric sensor may be any photoelectric sensor from the first photoelectric sensor array / subset 502, 522, 542, and 804, and the second photoelectric sensor may be any photoelectric sensor from the second photoelectric sensor array / subset 504, 524, 544, and 806 corresponding to the first photoelectric sensor, as described herein. Figures 5A-5C And as discussed in section 8. In some embodiments, the first photoelectric sensor and the second photoelectric sensor may be separated by a certain gap, for example, as discussed herein. Figures 5A-5B The discussion interval is 506 or 528.

[0093] At step 904, as the field of view of the sensor array moves, a first image of an object in the scene can be captured from a first viewpoint in a first instantaneous instance. For example, as the sensor array rotates about its central axis, a first photoelectric sensor array including a first photoelectric sensor can capture an image of an object in the scene, as described herein. Figure 3A The discussion then continues. Then, at step 906, while the sensor array continues to move, but before completing a full 360° rotation after capturing the first image, a second image of an object in the scene can be captured from a second perspective in a second time instance. For example, while the sensor array rotates about its central axis, a second photoelectric sensor array containing a second photoelectric sensor can capture images of objects in the scene, as discussed herein. Figure 3B The first and second photoelectric sensor arrays can have the same field of view size, allowing the two images to form a stereo image pair.

[0094] At step 910, the first and second images can be analyzed to calculate a first distance to an object in the scene using an optical baseline based on the gap between the first and second photoelectric sensors. In some embodiments, the processor of the stereo imager system 100, such as those described herein, Figure 1 The processor 122 of the system controller 104 discussed herein can receive a first image and a second image, compare common features of objects that are apparent in the first image and the second image, and use the distances between these common features to determine the distance to the objects using triangulation, as discussed herein with respect to FIG5. Therefore, embodiments of this disclosure can use a two-dimensional sensor array to capture the distance to objects in a scene.

[0095] As this article is about Figure 7 and 8The stereo imaging system discussed can be an active stereo imaging system, which can also use an emitter array and a ranging photoelectric sensor for ranging to measure the light after the light emitted by the emitter array has been reflected from an object in the scene. Therefore, method 900 may optionally include step 912, in which a second distance to the object can be measured. The second distance can be measured by the ranging photoelectric sensor, which measures the light once the light emitted by the corresponding light emitter has been reflected from an object in the scene. The distance to the object can be calculated using time-of-flight measurements, etc.

[0096] Subsequently, at step 914, the final distance to the object can be determined based on the first distance and the second distance. For example, the first distance can be used to enhance the accuracy of the second distance, especially for measurements at very close ranges, as discussed herein. Figure 7 and 8 This is under discussion. In some other cases, the first distance can be used in place of the second distance. Either form of enhancement can be applied by default, or in response to determining that the difference between the first and second distances is greater than a threshold.

[0097] IV. Mitigating receiver channel crosstalk

[0098] As can be understood from the disclosure herein, the channels in a micro-optical receiver are positioned very close to each other, typically within a few micrometers. This small spacing between each channel can increase the chance of problems. For example, light propagating through a volumetric imaging optics can sometimes cause stray light to seep into adjacent channels, resulting in inaccurate readings of reflected light for each pixel in the field. Ideally, no channel should receive stray light, such as... Figure 10A As shown.

[0099] Figure 10A This is a simplified cross-sectional view of a portion of a light detection system 1000 in which there is no crosstalk between channels. During operation, the perpendicular ray 1002 and the principal ray 1004 enter the volume imaging optics 1006 and produce a light cone 1008. Rays 1002 and 1004 enter the aperture of the aperture layer 1010 and then enter the collimating lens 1011. The collimating lens 1011 accepts a limited range of incident light angles. For example, the collimating lens 1011 can accept rays with incident angles of +25 to -25 degrees relative to the vertical. Figure 10A A light cone 1008 with an incident angle between +25 and -25 degrees is shown. The principal ray 1004 is the ray that passes through the center of the aperture. In this example, the principal ray 1004 has an incident angle of 0 degrees on the collimating lens 1011.

[0100] Figure 10BThis is a simplified cross-sectional view of a portion of a light detection system 1001 where crosstalk exists between channels. In this case, during operation, the tilted ray 1012 and the main ray 1014 enter the volume receiver optics 1016 and then the collimating lens 1021. In this example, the collimating lens 1021 belongs to a micro-optical channel corresponding to a photodetector far from the image center. In this example, the main ray 1014 has an incident angle of -12 degrees, and the focused cone has an incident angle of +12 to -35 degrees. The collimating lens 1021 repels some of the light rays because it only accepts light with an incident angle of +25 to -25 degrees. Additionally, light rays outside the collimating lens receiving cone can travel to other optical surfaces and become stray light. Therefore, the non-telecentric volume imaging optics delivers significantly fewer signal photons to the photodetector while potentially contaminating other channels with the deviated ray 1022. On the other hand, telecentric imaging optics will produce a light cone with an incident angle of approximately +25 to -25 degrees and a principal ray with an incident angle of approximately 0 degrees on the collimating lens, regardless of the angles of the tilted ray 1012 and the principal ray 1014. Telecentric imaging optics offer similar benefits to the emitter when the laser is telecentric (its principal rays are all parallel), as is the case with VCSELs or side-emitting diode laser bars.

[0101] In some embodiments, the light detection system of the light sensing module uses an input image-space telecentric imaging optics. In some other embodiments, such as when cost or increased field of view is more important than performance, the light detection system may use a more standard input volume imaging optics, such as a biconvex lens. For any given input field entering the image-space telecentric lens, the resulting principal rays are parallel to the optical axis, and the image-side ray cones all span approximately the same set of angles. This allows micro-optical channels in the light detection system that are far from the optical axis to achieve similar performance to coaxial micro-optical channels. The light detection system does not require perfect image-space telecentricity to operate, but the closer to perfect telecentricity, the better. For micro-optical receiver lens layers that can only accept + / -25 degrees of light, it is preferable that the input volume imaging optics generate image-side rays with an angle of no more than 25 degrees for each point on the focal plane.

[0102] According to some embodiments of this disclosure, each channel of the micro-optical receiver channel array can be specifically configured to minimize the intrusion of stray light into the corresponding photodetector, thereby reducing or eliminating any harmful effects caused by the presence of stray light. Figure 11 This is a simplified cross-sectional view of an example micro-optical receiver channel structure 1100, which is also referred to herein as a micro-optical receiver channel. Receiver channel 1100 can represent... Figure 7The micro-optical receiver channel 712 shown is used to receive an input light cone containing a wide range of wavelengths, filtering out all wavelengths outside the narrow band centered on the operating wavelength, and allowing the photoelectric sensor 1171 to detect only or substantially only photons within the aforementioned narrow band wavelengths. According to some embodiments of this disclosure, a micro-optical receiver channel structure, such as receiver channel 1100, may include the following layers:

[0103] ● Input aperture layer 1140, which includes a light-transmitting aperture 1144 and an opaque aperture region 1146, is configured to be placed in an imaging optical device such as a body receiver optical device 708. Figure 7 As shown in the image; Figure 11 A narrow field of view is defined at the focal plane (not shown). Aperture layer 1140 is configured to receive input edge rays 1133. The term "transparent" herein refers to allowing most or all light to pass through. Light here refers to the spectrum in the near-ultraviolet, visible, and near-infrared ranges (e.g., 300 nm to 5000 nm). Opaque herein refers to allowing little or no light to pass through, instead absorbing or reflecting light. Aperture layer 1140 may contain transparent apertures spaced apart from each other by opaque aperture regions. Apertures and aperture regions may be constructed on a single monolithic element, such as a transparent substrate. Aperture layer 1140 may optionally contain a one-dimensional or two-dimensional array of apertures 1144.

[0104] ● Optical lens layer 1150, comprising a collimating lens 1151 characterized by a focal length offset from the plane of aperture 1144 and stop region 1146, axially aligned with aperture 1144, and configured to collimate photons transmitted through the aperture such that the photons travel approximately parallel to the axis of collimating lens 1151, which is aligned with the optical axis of receiver channel 1100. Optical lens layer 1150 may optionally include an aperture, an opaque region, and a tubular structure to reduce crosstalk.

[0105] ● Filter layer 1160, comprising filters 1161, typically Bragg reflector type filters, adjacent to collimating lens 1151 and opposite aperture 1144. Filter layer 1160 can be configured to allow normal incident photons of a specific operating wavelength and passband to pass through. Filter layer 1160 can contain any number of filters 1161. Filter layer 1160 may optionally include an aperture, opaque regions, and tubular structures to reduce crosstalk.

[0106] ● Photodetector layer 1170, which includes a photodetector 1171 adjacent to filter layer 1160 and configured to detect photons incident on photodetector 1171. Photodetector 1171 herein refers to a single photodetector capable of detecting photons, such as an avalanche photodiode, SPAD (single-photon avalanche detector), RCP (resonant cavity photodiode), etc., or to several photodetectors working together to act as a single photodetector, such as a SPAD array, which typically have higher dynamic range, lower dark count rate, or other beneficial characteristics compared to a single large photon detection area. Photodetector 1171 can also refer to a visible light photodetector, such as a CCD or CMOS sensor used to capture stereoscopic images. Each photodetector can be a photon-sensing, i.e., an active region of light. Photodetector layer 1170 refers to a layer made of photodetectors and contains optional structures for improving detection efficiency and reducing crosstalk with adjacent receiver structures. The photoelectric sensor layer 1170 may optionally include a diffuser, a converging lens, an aperture, an opaque tube spacer structure, an opaque tapered spacer structure, etc.

[0107] Stray light can be caused by roughness of optical surfaces, defects in transparent media, back reflections, etc., and can be generated at many features inside or outside receiver channel 1100. Stray light can be guided: through filter region 1161 along a path not parallel to the optical axis of collimating lens 1151; reflected between aperture 1144 and collimating lens 1151; and generally takes any other path or trajectory that may contain many reflections and refractions. If multiple receiver channels are arranged adjacent to each other, such stray light in one receiver channel may be absorbed by the photodetector in another channel, thereby contaminating the inherent timing, phase, or other information of the photons. Therefore, receiver channel 1100 can be characterized by several structures to reduce crosstalk between receiver channels.

[0108] V. Implementation of the Stereo Imaging System

[0109] Figure 12AThis is a simplified top view of an example stereoscopic imager system 1200 according to some embodiments of the present disclosure, implemented for a vehicle 1205, such as an automobile, and capable of continuous 360-degree scanning. The output beams of one or more light sources (such as infrared or near-infrared pulsed IR lasers, not shown) located in the stereoscopic imager system 1200 can be scanned, for example, rotated, to illuminate a continuous scene 1220 around the vehicle. In some embodiments, the scan, indicated by the rotating arrow 1215, can be implemented by any suitable mechanical means, such as by mounting a light emitter to a rotating column or platform, or any other mechanical means, such as by using a galvanometer or chip-based control technology. During operation, objects around the vehicle 1205 in any direction and within the field of view of the stereoscopic imager system 1200 can reflect portions of light pulses 1211 emitted from the emitting module 1208 in the stereoscopic imager system 1200. One or more reflected portions 1217 of the light pulses 1211 then travel back to the stereoscopic imager system 1200 and can be detected by its sensing module 1209. Additionally, ambient light from scene 1220 can be captured to generate two offset images for distance calculation and depth perception purposes. In some cases, the sensing module 1209 can be housed in the same enclosure as the transmitting module 1208.

[0110] although Figure 12A A solid-state stereo imaging system mounted on the roof of vehicle 1205 is shown, but the embodiment is not limited to this configuration. Other embodiments may have solid-state stereo imaging systems mounted in other areas of the vehicle. For example, the stereo imaging system may be mounted in a corner of the vehicle, such as... Figure 12B As shown. Figure 12B An embodiment 1201 according to some embodiments of the present disclosure is shown, wherein solid-state stereo imager systems 1204a-d are implemented at the external area of ​​a road vehicle, such as a car. In this embodiment, each stereo imager system 1204a-d may be a rotating stereo imager system capable of measuring distances around the entire 360 ​​degrees. However, since at least some of those measurements will be taken relative to the vehicle 1205, those measurements may be ignored. Therefore, each stereo imager system 1205a-d may utilize a subset of the measurements from the 360-degree scan, for example, only utilizing angles of the coverage area 1219a-d that do not capture the vehicle 1205.

[0111] Although this disclosure has been described with respect to specific embodiments, it should be understood that this disclosure is intended to cover all modifications and equivalents within the scope of the following claims.

Claims

1. A stereo imaging system, comprising: Sensor array, the sensor array comprising: Multiple ranging photoelectric sensors detect light once it has been reflected from an object in the scene by a transmitter array; A plurality of imaging photoelectric sensors are positioned on a first side of the ranging photoelectric sensor; and The second plurality of imaging photoelectric sensors are positioned on the second side of the ranging photoelectric sensor opposite to the first side. The first plurality of imaging photoelectric sensors and the second plurality of imaging photoelectric sensors detect ambient light in the scene and are spaced apart by a certain gap. A moving component, coupled to the sensor array and operable to move the sensor array between a first position and a second position within a complete rotational image capture cycle; and A system controller, coupled to the sensor array and the moving part, is configured to: The first distance to the object in the scene is determined by using the multiple ranging photoelectric sensors through time-of-flight calculation; A first image of the scene is captured at the first location using the first plurality of imaging photoelectric sensors, and a second image of the scene is captured at the second location using the second plurality of imaging photoelectric sensors; A second distance to the object is calculated based on the first image and the second image, and an optical baseline determined by the gap; and The final distance to the object is determined based on the first distance enhanced by the second distance.

2. The stereo imaging system according to claim 1, wherein the plurality of ranging photoelectric sensors are arranged in a diagonally staggered manner, and the first plurality of imaging photoelectric sensors and the second plurality of imaging photoelectric sensors are each arranged in a rectangular manner.

3. The stereo imaging system according to claim 2, wherein at least some of the first plurality of imaging photoelectric sensors and at least some of the second plurality of imaging photoelectric sensors are positioned along the same horizontal line.

4. The stereo imaging system according to claim 1, wherein the moving component is an electric motor that rotates the sensor array about a central axis.

5. The stereo imaging system of claim 1, wherein the moving component is a microelectromechanical system (MEMS) device that reflects light to move the field of view.

6. The stereo imaging system according to any one of claims 1 to 5, wherein determining the final distance to the object based on the first distance enhanced by the second distance comprises: In response to determining that the difference between the first distance and the second distance is greater than a threshold, the final distance to the object is determined based on the first distance enhanced by the second distance.

7. A stereo imaging system, comprising: A transmitter array configured to emit light into a field outside the stereoscopic imager system; A sensor array, comprising a plurality of ranging photoelectric sensors, a first plurality of imaging photoelectric sensors, and a second plurality of imaging photoelectric sensors spaced apart from the first plurality of imaging photoelectric sensors, wherein the first plurality of imaging photoelectric sensors and the second plurality of imaging photoelectric sensors are configured to detect ambient light in a scene. A movable component, which is coupled to the sensor array and operable to move the sensor array between a first position and a second position within a complete rotational image capture cycle; A system controller, coupled to the sensor array and the moving part, is configured to: The first distance to the object in the scene is determined using the plurality of ranging photoelectric sensors; The field of view of the sensor array is moved by instructing the moving component to capture a first image of the object in the scene from a first perspective using the first plurality of imaging photoelectric sensors at the first position, and to capture a second image of the scene of the object in the scene from a second perspective using the second plurality of imaging photoelectric sensors at the second position. A second distance to the object is calculated based on the first image and the second image using an optical baseline defined by the gap; as well as The final distance to the object is determined based on the first distance enhanced by the second distance.

8. The stereo imaging system of claim 7, further comprising a light detection system, the light detection system comprising: A volume receiver optics device configured to receive light from a field originating outside the stereo imaging system; as well as An optical assembly having multiple micro-optical receiver channels defining multiple discrete, non-overlapping field of view in the field, the optical assembly comprising: An aperture layer having a plurality of discrete apertures arranged along the focal plane of the volume receiver optical device, wherein the first plurality of imaging photoelectric sensors and the second plurality of imaging photoelectric sensors are disposed behind the aperture layer; as well as A non-uniform filter layer, which is configured to allow different micro-optical channels to measure different ranges of wavelengths.

9. The stereo imaging system according to claim 7 or 8, wherein the moving component is an electric motor that rotates the sensor array about a central axis.

10. The stereo imager system of claim 7 or 8, wherein the moving component is a microelectromechanical system (MEMS) device that reflects light to move the field of view.

11. A distance measurement method, comprising: The field of view of a moving sensor array, the sensor array including a first imaging photoelectric sensor and a second imaging photoelectric sensor spaced apart from the first imaging photoelectric sensor by a certain gap; The first distance to the object is measured using a distance-measuring photoelectric sensor; During the field of view movement, a first image of an object in the scene is captured from a first perspective using the first imaging photoelectric sensor at a first time. During the field of view movement, a second image of the scene of the object in the scene is captured from a second perspective using the second imaging photoelectric sensor in a second time instance; A second distance to the object is calculated based on the first image and the second image using an optical baseline defined by the gap; as well as The final distance to the object is determined based on the first distance enhanced by the second distance.

12. The method of claim 11, wherein moving the field of view comprises rotating the sensor array about a central axis.

13. The method of claim 11, wherein moving the field of view comprises reflecting light to move the field of view while the sensor array is stationary.

14. The method of claim 11, further comprising: The common features of the object captured in the first image and the second image are compared; as well as The first distance to the object is calculated using the result of the comparison.

15. The method of claim 11, wherein the first imaging photoelectric sensor is included in a first imaging photoelectric sensor array, and the second imaging photoelectric sensor is included in a second imaging photoelectric sensor array.

16. The method of claim 15, wherein the first imaging photoelectric sensor array and the second imaging photoelectric sensor array are each a two-dimensional imaging photoelectric sensor array.

17. The method of claim 15, wherein the sensor array is formed by a two-dimensional imaging photoelectric sensor array, and the first imaging photoelectric sensor array and the second imaging photoelectric sensor array are each subsets of the two-dimensional imaging photoelectric sensor array.

18. The method of any one of claims 11 to 17, wherein determining the final distance to the object based on the first distance enhanced by the second distance comprises: In response to determining that the difference between the first distance and the second distance is greater than a threshold, the final distance to the object is determined based on the first distance enhanced by the second distance.

19. The method of claim 18, wherein the ranging photoelectric sensor is located in a two-dimensional ranging photoelectric sensor array, and the first imaging photoelectric sensor array and the second imaging photoelectric sensor array are located on opposite sides of the ranging photoelectric sensor array.

20. The method of claim 19, wherein the ranging photoelectric sensor array is arranged in a diagonally staggered manner, and the first imaging photoelectric sensor array and the second imaging photoelectric sensor array are arranged in a rectangular manner.

Images