Three-dimensional sensor including a bandpass filter having multiple passbands

By using multi-pass bandband-pass filters and RGB+IR sensors in the distance sensor, the noise interference problem during the acquisition of two-dimensional and three-dimensional images in the prior art is solved, which enhances the three-dimensional sensing capability and reduces the cost.

CN113272624BActive Publication Date: 2024-11-26MAGIK EYE INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202080009889.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-01-20
Filing Date
2020-01-13
Publication Date
2024-11-26
Estimated Expiration
2040-01-13

AI Technical Summary

Technical Problem

Existing distance sensors have noise interference when acquiring two-dimensional images and three-dimensional depth mappings at the same time, resulting in difficulty in detecting invisible light wavelengths. There are cost and size problems in replacing narrowband interference filters with infrared bandpass filters.

Method used

Using a bandpass filter with multiple passbands, including visible and invisible passbands, an imaging sensor is configured to detect infrared photodetectors but prevent other wavelengths, combined with an RGB+IR sensor or CMY+IR sensor, through bandpass filtering The device significantly cuts off light at different wavelengths to improve signal-to-noise ratio and three-dimensional sensing capabilities.

Benefits of technology

The signal-to-noise ratio of the distance sensor is improved, ensuring that the infrared photodetector receives infrared light while the RGB photodetector does not, enhancing the three-dimensional sensing capability and reducing noise interference, realizing the ability to acquire two-dimensional and three-dimensional images at the same time.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113272624B_ABST
    Figure CN113272624B_ABST
Patent Text Reader

Abstract

An example device includes a light projection system, a light receiving system, and a controller. The light projection system projects light beams of invisible wavelengths. When each beam is incident on a surface, the beam forms a pattern on the surface. The light receiving system captures an image of the pattern on the surface. The controller calculates the distance to the surface based on the image. The light receiving system includes a lens, an imaging sensor, and a bandpass filter. The imaging sensor includes a first subset of photodetectors sensitive to visible light wavelengths and a second subset of photodetectors sensitive to invisible light wavelengths. The bandpass filter includes a first passband whose range corresponds to the visible light wavelengths and a second passband whose range corresponds to the invisible light wavelengths.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application Serial No. 62 / 794,631, filed on January 20, 2019, the entire contents of which are incorporated herein by reference. Background Art

[0003] U.S. Patent Applications Serial Nos. 14 / 920,246, 15 / 149,323, and 15 / 149,429 describe various configurations of distance sensors. Such distance sensors may be useful in a variety of applications including security, gaming, unmanned vehicle control, and other applications.

[0004] The distance sensors described in these applications include a light projection system (e.g., including a laser, a diffractive optical element, and / or other cooperating components) that projects a light beam at a wavelength that is substantially invisible to the human eye (e.g., infrared light) into the field of view. The light beam spreads to create a pattern (a pattern of dots, dashes, or other man-made objects) that can be detected by an appropriate light receiving system (e.g., a lens, an image capture device, and / or other components). When the pattern is incident on an object in the field of view, the distance from the sensor to the object can be calculated based on the appearance of the pattern in one or more images of the field of view that can be captured by the sensor's light receiving system (e.g., the positional relationship of dots, dashes, or other man-made objects). The shape and size of the object can also be determined.

[0005] For example, the pattern appearance may change with distance to the object. As an example, if the pattern includes a pattern of dots, the dots may appear closer to each other when the object is closer to the sensor, and may appear farther away from each other when the object is farther away from the sensor. Summary of the invention

[0006] An example device includes a light projection system, a light receiving system, and a controller. The light projection system projects multiple light beams of wavelengths that are invisible to the human eye. When the multiple light beams are incident on a surface, the multiple light beams form a pattern on the surface. The light receiving system acquires an image of the pattern on the surface. The controller calculates the distance to the surface based on the image of the pattern on the surface. The light receiving system includes a lens; an imaging sensor; and a bandpass filter located between the lens and the imaging sensor. The imaging sensor includes a first subset of photodetectors that are sensitive to wavelengths of light visible to the human eye and a second subset of photodetectors that are sensitive to wavelengths of light that are invisible to the human eye. The bandpass filter includes at least a first passband whose range corresponds to wavelengths of light visible to the human eye and a second passband whose range corresponds to wavelengths of light that are invisible to the human eye.

[0007] In another example, a method includes: controlling, by a processing system of the distance sensor, a light projection system of the distance sensor to project a plurality of light beams, wherein the plurality of light beams include light of wavelengths that are not visible to the human eye, and wherein the plurality of light beams form a pattern on the surface when the plurality of light beams are incident on the surface; controlling, by the processing system, a light receiving system of the distance sensor to acquire an image of the pattern on the surface; and calculating, by the processing system, a distance to the surface based on the image of the pattern on the surface. The light receiving system includes: a lens; an imaging sensor including a plurality of photodetectors, wherein the plurality of photodetectors include a first subset of photodetectors sensitive to wavelengths of light visible to the human eye and a second subset of photodetectors sensitive to wavelengths of light not visible to the human eye; and a first bandpass filter located between the lens and the imaging sensor, wherein the first bandpass filter includes at least a first passband whose range corresponds to wavelengths of light visible to the human eye and a second passband whose range corresponds to wavelengths of light not visible to the human eye.

[0008] In another example, a non-transitory machine-readable storage medium is encoded with instructions executable by a processor of a distance sensor, wherein when executed, the instructions cause the processor to perform operations. The operations include: controlling a light projection system to project a plurality of light beams, wherein the plurality of light beams include light of wavelengths that are not visible to the human eye, and wherein when the plurality of light beams are incident on a surface, the plurality of light beams form a pattern on the surface; controlling a light receiving system to acquire an image of the pattern on the surface; and calculating a distance to the surface based on the image of the pattern on the surface. The light receiving system includes: a lens; an imaging sensor including a plurality of photodetectors, wherein the plurality of photodetectors include a first subset of photodetectors sensitive to wavelengths of light visible to the human eye and a second subset of photodetectors sensitive to wavelengths of light not visible to the human eye; and a first bandpass filter located between the lens and the imaging sensor, wherein the first bandpass filter includes at least a first passband whose range corresponds to wavelengths of light visible to the human eye and a second passband whose range corresponds to wavelengths of light not visible to the human eye. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a block diagram illustrating an example distance sensor of the present disclosure.

[0010] Figure 2 Graphics Figure 1 An example of an imaging sensor.

[0011] FIG. 3A to FIG. 3B It is shown in Figure 1 Graph of example wavelength transmission characteristics of a filter of an optical receiving system of an example distance sensor and an imaging sensor illustrated in FIG.

[0012] Figure 4 Illustrated example timing diagram showing the Figure 1The relative timing of the emission of the light projection system and the lighting system of the distance sensor illustrated in FIG.

[0013] Figure 5 is a flow chart illustrating one example of a method for operating a range sensor for three-dimensional depth sensing and two-dimensional image capture according to the present disclosure, and .

[0014] Figure 6 A high-level block diagram depicts an example electronic device for operating a range sensor for three-dimensional depth sensing and two-dimensional image capture. DETAILED DESCRIPTION

[0015] The present disclosure broadly describes apparatus, methods, and non-transitory computer-readable media for operating a three-dimensional sensor including a bandpass filter having multiple passbands. As discussed above, distance sensors such as those described in U.S. patent applications with serial numbers 14 / 920,246, 15 / 149,323, and 15 / 149,429 determine the distance to an object (and possibly the shape and size of the object) by projecting a light beam that spreads to create a pattern (e.g., a pattern of dots, dashes, or other artifacts) in a field of view that includes the object. The light beam may be projected from one or more laser sources that emit light that is substantially invisible to the human eye but visible to an appropriate detector (e.g., a light receiving system). The three-dimensional distance to the object may then be calculated based on the appearance of the pattern to the detector.

[0016] In some examples, the detector of the light receiving system may include a red, green, blue, infrared (RGB+IR) sensor. The RGB+IR sensor typically includes a separate sensing element (e.g., a photodetector combined with a bandpass filter) for capturing images of both visible light and invisible (e.g., infrared) light. This allows the distance sensor to simultaneously generate a two-dimensional (e.g., RGB) image and a three-dimensional (e.g., IR) depth map of an object. In such applications, the infrared sensing element of the RGB+IR sensor typically includes a narrow bandwidth filtering action centered on the wavelength of the infrared light emitted by the light projection system.

[0017] However, since the main purpose of an RGB+IR sensor is often to acquire normal infrared images, the transmission bandwidth of the infrared sensing element (and more specifically, the infrared bandpass filter) may be too wide to allow simultaneous acquisition of a two-dimensional image and a three-dimensional image. For example, noise from visible light wavelengths (used to capture a two-dimensional image) may make detection of invisible light wavelengths (used to calculate distances for three-dimensional depth mapping) difficult. Furthermore, replacing the infrared bandpass filter with a narrowband interference filter (e.g., a coated filter) may not be feasible due to size and / or cost considerations.

[0018] Examples of the present disclosure include a distance sensor having a light receiving system, wherein the imaging sensor includes a bandpass filter including at least two passbands (i.e., a range of frequencies or wavelengths that can pass): a first passband of visible light (i.e., light visible to the human eye, such as light in RGB wavelengths) and a second passband of invisible light (i.e., light invisible to the human eye, such as light in infrared wavelengths). Thus, the invisible light passes to a photodetector of the imaging sensor that is configured to detect the invisible light, but is blocked from passing to a photodetector that is not configured to detect the invisible light.

[0019] Although the examples of the present disclosure are described in the example context of an RGB+IR sensor, the detector of the light receiving system disclosed herein may alternatively include a cyan, magenta, yellow (CMY)+IR sensor. Therefore, although the examples described below may refer to an RGB+IR sensor or a wavelength of light within the RGB range, it will be appreciated that any such reference is equally applicable to a CMY+IR sensor and a wavelength of light within the CMY range.

[0020] Figure 1 1 is a block diagram illustrating an example distance sensor 100 of the present disclosure. The distance sensor 100 can be used to detect the distance of an object or surface 102. In one example, the distance sensor 100 shares many components of the distance sensors described in U.S. Patent Applications Serial Nos. 14 / 920,246, 15 / 149,323, and 15 / 149,429. For example, in one example, the distance sensor includes a light projection system 104, a light receiving system 106, and a controller 108. Optionally, the distance sensor 100 can further include an illumination system 110.

[0021] In one example, the light projection system can be arranged in a manner similar to any of the arrangements described in U.S. patent application Ser. No. 16 / 701,949. For example, the light projection system 104 can generally include a laser emitter 112, a lens 114, and a diffractive optical element (DOE) 116. The laser emitter 112 can include at least one laser light emitting diode (LED) that emits a light beam at a wavelength that is substantially invisible to the human eye (e.g., infrared light). In one example, the laser emitter 112 includes at least one vertical cavity surface emitting laser (VCSEL) or at least one edge emitting laser (EEL).

[0022] The lens 114 may include a lens configured to magnify the light beam emitted by the laser emitter 112. For example, the lens 114 may include a converging lens (e.g., a biconvex or plano-convex lens) having positive optical power. In this case, the collimated light beam passing through the lens 114 may converge to a focal point behind the lens 114 before spreading or diverging from the focal point.

[0023] The diffractive optical element 116 may include a conical mirror, a holographic film, a grating, or another phase element that uses interference and diffraction to create a beam distribution from a collimated (e.g., single) beam. The diffractive optical element 116 may be positioned at or near the focal point of the lens 114, for example, between the lens 114 and the surface 102. Thus, the beam may pass through the diffractive optical element 116 as each beam converges at the focal point of the lens 114 or is collimated at the focal point of the lens 114. The diffractive optical element 116 may then split the collimated light into a plurality of beams that are distributed to produce a projection pattern on the surface 102.

[0024] Although Figure 1 A single laser emitter 112, lens 114, and diffractive optical element 116 combination is shown, but it should be noted that the light projection system 104 may include any number of laser emitter / lens / diffractive optical element combinations. For example, multiple laser emitter / lens / diffractive optical element combinations may be arranged around the light receiving system 106, such as so that the light receiving system 106 is located at the center of the multiple laser emitter / lens / diffractive optical element combination. In this case, each of the multiple laser emitter / lens / diffractive optical element combinations may be arranged in a manner similar to that in Figure 1 The laser emitter 112, lens 114 and diffractive optical element 116 are configured in the manner shown in the figure.

[0025] In one example, the multiple light beams emitted by the light projection system 104 are spread out from the central axis in multiple different directions. The angles between each pair of beams in the multiple light beams can be equal. When the multiple light beams are incident on the surface 102, each beam can create a light point (e.g., a dot, a dash, etc.) on the surface 102. In general, the light points created by the multiple light beams form the pattern described above, and the distance to the surface 102 can be calculated based on the pattern. For example, the pattern can include a grid in which the multiple points are arranged into multiple rows and columns.

[0026] The light receiving system 106 of the distance sensor 100 is positioned to capture a reflected image of a pattern projected onto the surface 102 by the light projecting system 104. In one example, the light receiving system 106 may generally include a lens 118, a filter 120, and an imaging sensor 122.

[0027] In one example, lens 118 may be a wide angle lens, such as a fisheye lens, that creates a field of view of up to 180 degrees (eg, a hemispherical field of view).

[0028] In one example, the filter 120 may include a first bandpass filter (e.g., an interference filter) having a plurality of passbands, wherein at least a first passband and a second passband of the plurality of passbands are configured to allow light of different wavelengths to pass (but block light of all other wavelengths). For example, the first passband may transmit light of one or more wavelengths visible to the human eye (e.g., RGB wavelengths), while the second passband may transmit light of one or more wavelengths invisible to the human eye (e.g., infrared wavelengths). In one example, a portion of the first bandpass filter corresponding to the second passband may be coated.

[0029] In one example, the imaging sensor 122 includes a plurality of photodetectors (or pixels), where the plurality of photodetectors may be arranged in an array. Figure 2 Graphics Figure 1 An example of an imaging sensor 122. Figure 2 As shown in FIG. 1 , the imaging sensor 122 may include a plurality of photodetectors 2001-200 n (hereinafter individually referred to as "photodetector 200" or collectively referred to as "photodetector 200"), where the photodetectors 200 are arranged in a square (e.g., 6×6) array. However, in other examples, the array can be larger or smaller and can have a different shape. The photodetectors 200 can include complementary metal oxide semiconductor (CMOS) photodetectors.

[0030] As Figure 2 As indicated by the keys in , the plurality of photodetectors 200 may include photodetectors that are sensitive to light of different wavelengths. For example, the plurality of photodetectors may include photodetectors that are sensitive to wavelengths of light visible to the human eye, such as a red light photodetector (e.g., photodetector 2001), a green light photodetector (e.g., photodetector 2002), and a blue light photodetector (e.g., photodetector 2003). n-1 The plurality of photodetectors may also include photodetectors sensitive to wavelengths of light that are invisible to the human eye, such as infrared photodetectors (eg, photodetector 200 n ). Thus, the plurality of photodetectors may include a first subset of photodetectors (e.g., RGB photodetectors) that are sensitive to wavelengths of light visible to the human eye and a second subset of photodetectors (e.g., IR photodetectors) that are sensitive to wavelengths of light not visible to the human eye.

[0031] The passband of the filter 120 may be positioned accordingly. For example, a passband that allows visible red light to pass may be located above the red light photodetector 200 of the imaging sensor 122; a passband that allows visible green light to pass may be located above the green light detector 200 of the imaging sensor 122; a passband that allows visible blue light to pass may be located above the blue light detector 200 of the imaging sensor 122; and a passband that allows invisible infrared light (or other wavelengths of light that are invisible to the human eye) to pass may be located above the infrared photodetector 200 of the imaging sensor 122. Therefore, infrared light emitted for three-dimensional distance sensing may be input only to the photodetector 200 having an IR filter, while light of other wavelengths may be recognized as a color image by the photodetector having an RGB filter. Therefore, the imaging sensor 122 may detect red light, green light, blue light, and infrared light, and may detect only infrared light, or may detect only red light, green light, and blue light. Because three-dimensional distance sensing depends on the intensity of the projected pattern of light, and two-dimensional imaging depends on external brightness, the optimal exposure times for the IR portion and the RGB portion of the imaging sensor 122 may be different.

[0032] The illumination system 110 , if included in the distance sensor 100 , may be configured to provide light at a wavelength that is not visible to the human eye (eg, infrared light). In one example, the illumination system 110 may include a light source 124 , a reflector 126 , and a filter 128 .

[0033] The light source 124 may include an LED light source, such as one or more white light LEDs operated by a pulsed current. The light source 124 may be capable of emitting a spectrum including both visible wavelengths and invisible (e.g., infrared) wavelengths. The reflector 126 may include a reflective surface (e.g., a mirror) configured to spread the light emitted by the light source 124 in a uniform manner. For example, the reflector 126 may include a flash diffuser. The reflector 126 may have a cone, a parabola, or other shapes. The filter 128 may include a second bandpass filter that allows light of wavelengths invisible to the human eye (e.g., infrared light) to pass while blocking light of other wavelengths emitted by the light source 124.

[0034] The controller 108 may be operable to send signals to the light projection system 104, the light receiving system 106, and the illumination system 110. For example, the controller 108 may send a signal to the light projection system 104 to cause the light projection system 104 to project a light pattern onto the surface 102. The controller 108 may also control the duration during which the light projection system 104 projects the light pattern onto the surface 102, as well as the timing with which the light projection system 104 projects the light pattern onto the surface 102. For example, the controller 108 may control the duration of a pulse emitted by the laser emitter 112 of the light projection system 104.

[0035] The controller 108 may also send a signal to the light receiving system 106 to cause the light receiving system 106 to capture an image of the light pattern projected onto the surface 102 and / or to capture a two-dimensional (e.g., RGB) image of the surface 102. In a further example, the controller 108 may control the exposure time of the light receiving system 106 (e.g., the duration during which the shutter of the light receiving system is open) and the timing of the light receiving system 106 capturing images (including images of the surface 102) with it. In one example, the controller 108 may set two separate exposure durations for the light receiving system 106: a first exposure duration, during which an image of the surface 102 is captured while the light projection system 104 projects a pattern or light onto the surface 102 (e.g., for three-dimensional distance sensing), and a second exposure duration, during which an image of the surface 102 is captured at a time when the light projection system 104 does not project a light pattern onto the surface 102 (e.g., for two-dimensional image acquisition). In one example, the controller 108 may alternate between the first exposure duration and the second exposure duration.

[0036] The controller 108 may also send a signal to the lighting system 110 to cause the lighting system 110 to emit light. The controller 108 may also control the duration during which the lighting system 110 emits light and the timing at which the lighting system 110 emits light. For example, the controller 108 may control the duration of a pulse emitted by the light source 124 of the lighting system 110.

[0037] In a further example, the controller 108 can process the images acquired by the light receiving system 106. For example, the controller 108 can be configured to process the captured image data (e.g., three-dimensional distance data and two-dimensional image data) to calculate the distance to the surface 102. For example, the distance can be calculated according to the methods described in U.S. patent applications serial numbers 14 / 920,246, 15 / 149,323, and 15 / 149,429. Thus, in one example, the controller 108 can include a processor, such as in Figure 6 Processor 602 is illustrated in and described in further detail below.

[0038] FIG. 3A to FIG. 3B It is shown in Figure 1 1 is a line graph of example wavelength transmission characteristics of the filter 120 and imaging sensor 122 of the light receiving system 106 of the example distance sensor 100. More specifically, FIG. 3A to FIG. 3B Quantum efficiency (as a percentage of light transmitted) versus wavelength (in nanometers) is plotted for an example filter 120 and imaging sensor 122 .

[0039] exist Figure 3A In the example illustrated in , the infrared passband of filter 120 is substantially matched to or narrower than the passband of the infrared photodetector of imaging sensor 122. In addition, the wavelength of the coherent light projected by light projection system 104 matches the overlapping region of the infrared passband of filter 120 and the passband of the infrared photodetector of imaging sensor 122.

[0040] exist Figure 3B In the example illustrated in , the infrared passband of filter 120 partially matches the passband of the infrared photodetector of imaging sensor 122. In addition, the wavelength of the coherent light projected by light projection system 104 matches the overlapping region of the infrared passband of filter 120 and the passband of the infrared photodetector of imaging sensor 122.

[0041] As discussed above, coherent light emitted by the light projection system 104 is reflected by the surface 102, imaged onto the imaging sensor 122 of the light receiving system 106, and captured only by the infrared photodetector of the imaging sensor 122 (where the imaging sensor 122 also includes an RGB photodetector). In addition, when a bandpass filter such as the filter 120 is included in the light receiving system 106, wavelengths of light (e.g., visible or non-infrared wavelengths) other than the wavelengths emitted by the light projection system 106 that may be received by the infrared photodetector can be significantly cut off, which can improve the signal-to-noise ratio (SNR) of the light receiving system 106 and the image acquired by the light receiving system 106. At the same time, other photodetectors (e.g., RGB photodetectors) of the imaging sensor 122 can capture light of visible wavelengths and acquire a typical two-dimensional (e.g., RGB) image.

[0042] Therefore, including a bandpass filter 120 having multiple passbands (including both visible light and invisible light) in the distance sensor's light receiving system 106 ensures that the infrared photodetector 200 of the imaging sensor 122 receives infrared light emitted by the light projection system 104, while the RGB photodetector 200 does not receive infrared light emitted by the light projection system 104. In addition, by cutting off wavelengths around the wavelength(s) emitted by the light projection system 104, the bandpass filter 120 can also reduce the SNR of the light receiving system 106 and improve the three-dimensional sensing capability of the distance sensor 100.

[0043] In further examples of the present disclosure, the disclosed configuration (including the bandpass filter 120) may be implemented in combination with many different kinds of RGB+IR imaging sensors.

[0044] In a further example, the disclosed configuration may also be used in combination with a distance sensor configuration that produces an infrared two-dimensional image (e.g., a dark field image) rather than an RGB two-dimensional image. In this case, the infrared illumination for two-dimensional imaging may be emitted separately from the infrared light emitted for three-dimensional distance sensing (e.g., emitted from a separate infrared light source and / or emitted at different times to minimize interference). The exposure time for the infrared illumination for two-dimensional imaging and the exposure time for the infrared light emitted for three-dimensional distance sensing may also be different (and the difference may vary with changes in the object(s) being imaged). However, the infrared illumination for two-dimensional imaging may still include infrared wavelengths within the infrared passband of the bandpass filter 120.

[0045] In still further examples, the distance sensor 100 of the present disclosure may be implemented in a stereo camera configuration configured to detect distance using projected images as feature points.

[0046] Figure 4 Illustrated example timing diagram showing the Figure 1 1 and 2. The relative timing of the emissions of the light projection system 104 and the illumination system 110 of the distance sensor 100 is illustrated in FIG. As described above, the wavelengths of light emitted by the illumination system 110 may include visible wavelength regions and non-visible (eg, infrared) wavelength regions.

[0047] When the wavelength of light emitted by the illumination system 110 is infrared, the infrared photodetector of the imaging sensor 122 of the light receiving system may detect the light emitted by the illumination system 110. In this case, the light emitted by the illumination system 110 may interfere with the light beam (which may also include infrared wavelengths) emitted by the light projection system 104. This may impair the ability of the light receiving system to accurately detect the light pattern projected by the light projection system 104.

[0048] Therefore, if Figure 4 As shown in FIG. 1 , the light projection system 104 and the illumination system 110 may be controlled (eg, by the controller 108 ) such that the light projection system 104 and the illumination system 110 emit their respective lights at different times in order to minimize interference at the imaging sensor 122 .

[0049] In addition, as illustrated by the timing offsets O1 and O2, the timing of the shutter of the light receiving system can be controlled so that the time during which the shutter is open is not exactly the same as the time during which light is emitted (by the light projection system 104 or the illumination system 110). For example, the shutter can be opened just before the light emission starts or just after the light emission starts, and / or the shutter can be closed just before the light emission starts or just after the light emission starts.

[0050] Figure 5 is a flow chart illustrating one example of a method 500 for operating a distance sensor for three-dimensional depth sensing and two-dimensional image capture according to the present disclosure. The method 500 may be, for example, Figure 1 The controller 108 shown in the figure may be Figure 6 The method 500 is executed by a processor 602 illustrated in and discussed in further detail below. For purposes of example, the method 500 is described as being performed by a processing system.

[0051] Method 500 may begin at step 502. In step 504, a processing system may control a light projection system of a distance sensor to project a light pattern onto a surface (e.g., an object whose distance is to be detected), wherein the light pattern is created by a plurality of light beams having wavelengths that are not visible to the human eye. For example, the plurality of light beams may include infrared light. As discussed above, the light pattern may include light spots arranged to form a pattern such as a grid on the surface.

[0052] In step 506, the processing system may control a light receiving system of the distance sensor to acquire a three-dimensional image of the light pattern on the surface, wherein the light receiving system comprises an RGB+IR imaging sensor and a bandpass filter having multiple passbands, including at least one passband that allows invisible light wavelengths to pass.

[0053] For example, an RGB+IR imaging sensor may include multiple photodetectors (or pixels), wherein the multiple photodetectors may include photodetectors sensitive to light of different wavelengths. As an example, the multiple photodetectors may include photodetectors sensitive to wavelengths of light visible to the human eye, such as a red light photodetector, a green light photodetector, and a blue light photodetector. The multiple photodetectors may also include photodetectors sensitive to wavelengths of light not visible to the human eye, such as infrared photodetectors.

[0054] In this case, the plurality of passbands of the bandpass filter may include at least a first passband and a second passband. The first passband and the second passband may be configured to allow light of different wavelengths to pass (but block light of all other wavelengths). For example, the first passband may transmit light of one or more wavelengths visible to the human eye (e.g., RGB wavelengths), while the second passband may transmit light of one or more wavelengths invisible to the human eye (e.g., infrared wavelengths).

[0055] Thus, in the case of step 506, the invisible (e.g., infrared) light emitted in step 504 may be received by (may be directed to) a photodetector of the imaging sensor that is configured to detect invisible light; however, the invisible light may not be received by (may be blocked from going to) other photodetectors (e.g., RGB photodetectors) of the imaging sensor.

[0056] In step 508, the processing system may calculate the distance to the surface or may calculate a three-dimensional depth map of the surface based on the image acquired in step 506. The calculations may be performed as described in U.S. Patent Application Serial Nos. 14 / 920,246, 15 / 149,323, and 15 / 149,429.

[0057] In optional step 510 (illustrated in phantom), the processing system may control an illumination system of the distance sensor to illuminate the surface, wherein light emitted by the illumination system may include light having wavelengths invisible to the human eye and light having wavelengths visible to the human eye.

[0058] In optional step 512 (illustrated in dashed boxes), the processing system may control the light receiving system of the distance sensor to acquire a two-dimensional image of the illuminated surface. Thus, in the case of step 512, the visible (e.g., RGB) light emitted in step 510 may be received by (and may be directed to) a photodetector of the imaging sensor configured to detect visible light; however, the visible light may not be received by (and may be blocked from going to) other photodetectors (e.g., infrared photodetectors) of the imaging sensor.

[0059] Method 500 may end at step 514 .

[0060] It should be noted that, although not explicitly stated, some blocks, functions, or operations of the method 500 described above may include storage, display, and / or output for specific applications. In other words, depending on the specific application, any data, records, fields, and / or intermediate results discussed in the method 500 may be stored, displayed, and / or output to another device. Furthermore, Figure 5 The description of a determination operation or a block, function or operation involving a decision in the embodiment does not imply the practice of two branches of the determination operation. In other words, depending on the result of the determination operation, one of the branches of the determination operation may not be executed.

[0061] Figure 6 A high-level block diagram depicts an example electronic device 600 for operating a distance sensor for three-dimensional depth sensing and two-dimensional image capture. As such, the electronic device 600 may be implemented as a processor of an electronic device or system, such as a distance sensor (e.g., Figure 1 Controller 108 in.

[0062] As in Figure 6As depicted in FIG. 1 , the electronic device 600 includes: a hardware processor element 602, such as a central processing unit (CPU), a microprocessor, or a multi-core processor; a memory 604, such as a random access memory (RAM) and / or a read-only memory (ROM); a module 605 for operating a distance sensor for three-dimensional depth sensing and two-dimensional image capture; and various input / output devices 606, such as storage devices, including but not limited to a tape drive, a floppy disk drive, a hard drive, or an optical drive, a receiver, a transmitter, a display, an output port, an input port; and user input devices, such as a keyboard, a keypad, a mouse, a microphone, a camera, a laser source, and an LED light source, etc.

[0063] Although one processor element is shown, it should be noted that multiple processor elements may be employed in the electronic device 600. Further, although one electronic device 600 is shown in the figure, if the method(s) as discussed above are implemented in a distributed or parallel manner for a particular illustrative example, i.e., blocks of the method(s) above or the overall method(s) are implemented across multiple or parallel electronic devices, then the electronic device 600 of the figure is intended to represent each of those multiple electronic devices.

[0064] It should be noted that the present disclosure may be implemented by machine-readable instructions and / or in a combination of machine-readable instructions and hardware, for example, using an application specific integrated circuit (ASIC), a programmable logic array (PLA) including a field programmable gate array (FPGA), or a state machine deployed on a hardware device, a general-purpose computer, or any other hardware equivalent, for example, computer-readable instructions related to the (multiple) methods discussed above may be used to configure a hardware processor to perform the blocks, functions, and / or operations of the (multiple) methods disclosed above.

[0065] In one example, instructions and data (e.g., machine-readable instructions) for the present module or process 605 for operating a range sensor for three-dimensional depth sensing and two-dimensional image capture may be loaded into the memory 604 and executed by the hardware processor element 602 to implement blocks, functions, or operations as discussed above in connection with the method 500. Further, when the hardware processor executes instructions to perform an “operation,” this may include the hardware processor directly performing the operation and / or facilitating, directing, or cooperating with another hardware device or component (e.g., a coprocessor, etc.) to perform the operation.

[0066] A processor that executes machine-readable instructions related to the method(s) described above may be considered a programmed processor or a dedicated processor. As such, the present module 605 of the present disclosure for operating a distance sensor for three-dimensional depth sensing and two-dimensional image capture may be stored on a tangible or physical (broadly non-transitory) computer-readable storage device or medium, such as a volatile memory, non-volatile memory, ROM memory, RAM memory, magnetic or optical drive, device or disk, etc. More specifically, a computer-readable storage device may include any physical device that provides the ability to store information, such as data and / or instructions to be accessed by a processor or electronic device (such as a computer or controller of a security sensor system).

[0067] It will be appreciated that the above disclosed variations and other features and functions or their replacements may be combined into many other different systems or applications. Various currently unforeseen or unanticipated replacements, modifications or variations thereof may subsequently be made, which are also intended to be covered by the subsequent claims.

Claims

1. A device comprising: a light projection system configured to project a plurality of light beams, wherein the plurality of light beams include light of wavelengths that are not visible to the human eye, and wherein when the plurality of light beams are incident on a surface, the plurality of light beams form a pattern on the surface; A light receiving system configured to acquire a first image of a pattern on a surface and a second two-dimensional image of the surface without the pattern, wherein the light receiving system comprises: lens; an imaging sensor comprising a plurality of photodetectors, wherein the plurality of photodetectors comprises a first subset of photodetectors sensitive to wavelengths of light visible to the human eye and a second subset of photodetectors sensitive to wavelengths of light not visible to the human eye; and A first bandpass filter located between the lens and the imaging sensor, wherein the first bandpass filter includes at least a first passband and a second passband, the range of the first passband corresponds to a wavelength of light visible to the human eye, and the range of the second passband corresponds to a wavelength of light invisible to the human eye; An illumination system configured to emit light detectable by the light receiving system, wherein the illumination system comprises: a light source configured to emit wavelengths of light visible to the human eye and wavelengths of light invisible to the human eye; and A controller configured to: controlling the light projection system to project the plurality of light beams during a first time period; controlling the lighting system to emit light having a wavelength visible to the human eye and a wavelength invisible to the human eye during a second time period not overlapping with the first time period; controlling the light receiving system to acquire a first image during a first exposure duration occurring during the first time period, the first time period having a duration different from the first exposure duration; controlling the light receiving system to acquire the second two-dimensional image for a second exposure duration separate from the first exposure duration during the second time period, the second time period having a duration different from the second exposure duration; and The distance to the surface is calculated based on an image of the pattern on the surface.

2. The device of claim 1, wherein the wavelength of light invisible to the human eye comprises infrared light.

3. The apparatus of claim 1, wherein the wavelengths of light visible to the human eye include red light, green light, and blue light.

4. The apparatus of claim 1, wherein the wavelengths of light visible to the human eye include cyan, magenta, and yellow light.

5. The apparatus of claim 1, wherein the second passband overlaps with a wavelength of light to which the second subset of photodetectors is sensitive.

6. The apparatus of claim 1, wherein the second passband is narrower than a wavelength of light to which the second subset of photodetectors is sensitive.

7. The apparatus of claim 1, wherein the first passband is the same as the range of light wavelengths to which the first subset of photodetectors is sensitive.

8. The apparatus of claim 1, wherein the plurality of light beams comprises coherent light.

9. The apparatus of claim 1, wherein the light projection system comprises a vertical cavity surface emitting laser.

10. The apparatus of claim 1, wherein the first bandpass filter comprises an interference filter.

11. The apparatus of claim 3, wherein the plurality of photodetectors comprises: a first plurality of photodetectors sensitive to visible red light; a second plurality of photodetectors sensitive to visible green light; a third plurality of photodetectors sensitive to visible blue light; as well as A fourth plurality of photodetectors are sensitive to infrared light.

12. The apparatus of claim 11, wherein the first bandpass filter comprises: a first plurality of passbands allowing visible red light to pass, wherein the first plurality of passbands are located above the first plurality of photodetectors; a second plurality of passbands allowing visible green light to pass, wherein the first plurality of passbands are located above the second plurality of photodetectors; a third plurality of passbands that allow visible blue light to pass, wherein the third plurality of passbands are located above the third plurality of photodetectors; as well as A fourth plurality of passbands allows infrared light to pass, wherein the fourth plurality of passbands are located above the fourth plurality of photodetectors.

13. The apparatus of claim 1 , wherein the illumination system comprises a second bandpass filter, and wherein the second bandpass filter allows wavelengths of light emitted by the light source that are invisible to the human eye to pass through to illuminate the surface while blocking wavelengths of light emitted by the light source that are visible to the human eye.

14. The apparatus of claim 1, wherein the first bandpass filter is positioned to allow light in wavelengths of light that are not visible to the human eye to pass to the second subset of photodetectors but is blocked from passing to the first subset of photodetectors.

15. The apparatus of claim 14, wherein the first bandpass filter is further positioned to allow light in wavelengths of light visible to the human eye to pass to the first subset of photodetectors but is blocked from passing to the second subset of photodetectors.

16. A method comprising: controlling, by a processing system of the distance sensor, a light projection system of the distance sensor to project a plurality of light beams during a first time period, wherein the plurality of light beams include light of a wavelength that is not visible to a human eye, and wherein when the plurality of light beams are incident on a surface, the plurality of light beams form a pattern on the surface; controlling, by the processing system, a lighting system of the distance sensor to emit light of a wavelength visible to the human eye and a wavelength invisible to the human eye during a second time period that does not overlap with the first time period; controlling, by a processing system, a light receiving system of a distance sensor to acquire a first image of a pattern on a surface while the projection system projects the plurality of light beams during a first exposure duration, wherein the first exposure duration occurs within a first time period having a duration different from the first exposure duration; The processing system controls the light receiving system to acquire a second two-dimensional image of the surface, while the illumination system emits light of wavelengths visible to the human eye and wavelengths invisible to the human eye during a second exposure duration, wherein the second exposure duration occurs within the second time period, the second time period having a duration different from the second exposure duration, wherein the light receiving system comprises: lens; an imaging sensor comprising a plurality of photodetectors, wherein the plurality of photodetectors comprises a first subset of photodetectors sensitive to wavelengths of light visible to the human eye and a second subset of photodetectors sensitive to wavelengths of light not visible to the human eye; and A first bandpass filter located between the lens and the imaging sensor, wherein the first bandpass filter includes at least a first passband and a second passband, the first passband having a range corresponding to wavelengths of light visible to the human eye and the second passband having a range corresponding to wavelengths of light invisible to the human eye; and The distance to the surface is calculated by a processing system based on the image of the pattern on the surface.

17. The method of claim 16, wherein the wavelength of light invisible to the human eye comprises infrared light.

18. The method of claim 16, wherein the plurality of photodetectors comprises: a first plurality of photodetectors sensitive to visible red light; a second plurality of photodetectors sensitive to visible green light; a third plurality of photodetectors sensitive to visible blue light; as well as A fourth plurality of photodetectors are sensitive to infrared light.

19. A method according to claim 16, wherein the first bandpass filter is positioned to allow light in wavelengths of light that are invisible to the human eye to pass through the second subset of photodetectors but is blocked from passing through the first subset of photodetectors, and wherein the first bandpass filter is further positioned to allow light in wavelengths of light that are visible to the human eye to pass through the first subset of photodetectors but is blocked from passing through the second subset of photodetectors.

20. A non-transitory machine-readable storage medium encoded with instructions executable by a processor of a distance sensor, wherein when executed, the instructions cause the processor to perform operations comprising: controlling a light projection system to project a plurality of light beams during a first time period, wherein the plurality of light beams include light of a wavelength that is not visible to a human eye, and wherein when the plurality of light beams are incident on a surface, the plurality of light beams form a pattern on the surface; controlling the lighting system to emit light of wavelengths visible to the human eye and wavelengths invisible to the human eye during a second time period that does not overlap with the first time period; controlling a light receiving system to acquire a first image of a pattern on a surface while the light projecting system projects the plurality of light beams during a first exposure duration, wherein the first exposure duration occurs within a first time period, the first time period having a duration different from the first exposure duration, controlling the light receiving system to acquire a second two-dimensional image of the surface while the illumination system emits light of wavelengths visible to the human eye and wavelengths invisible to the human eye during a second exposure duration, wherein the second exposure duration occurs within a second time period, the second time period having a duration different from the second exposure duration, wherein the light receiving system comprises: lens; an imaging sensor comprising a plurality of photodetectors, wherein the plurality of photodetectors comprises a first subset of photodetectors sensitive to wavelengths of light visible to the human eye and a second subset of photodetectors sensitive to wavelengths of light not visible to the human eye; and a first bandpass filter located between the lens and the imaging sensor, wherein the first bandpass filter includes at least a first passband and a second passband, the first passband having a range corresponding to wavelengths of light visible to the human eye and the second passband having a range corresponding to wavelengths of light invisible to the human eye; and; The distance to the surface is calculated based on an image of the pattern on the surface.