System and method for utilizing SWIR sensing in cleaning machine

Abstract

A system and method for sensing with SWIR in an automatic cleaning machine, comprising: a first illumination source for emitting radiation of a first wavelength in SWIR toward an FOV; and a receiver operating in the SWIR to acquire SWIR image data based on radiation reflected from an element located in the FOV, and to determine the presence of the aqueous-based liquid in the FOV based on the SWIR image data.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority from U.S. Provisional Patent Application No. 63 / 345,594, filed on May 25, 2022, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to cleaning machines and more particularly to an automatic cleaning machine having short range infrared (SWIR) sensing capability. Background Art

[0004] Autonomous or semi-autonomous cleaning machines (such as robotic devices) are commonly used in domestic, public and industrial environments. Such autonomous or semi-autonomous devices may include vacuum cleaners, lawn mowers, sweeping robots, floor washing machines, window washing machines and other devices.

[0005] Recently, the field of autonomous cleaning machines or robotic cleaning machines, especially vacuum cleaners and floor cleaning machines, has developed rapidly, with the main goal of autonomously navigating the user's home while cleaning floors and / or carpets. In order to work autonomously or with minimal external control while navigating an area, autonomous devices can generally utilize mapping, positioning, object recognition and path planning methods. Some of the methods utilized may include a visual system that can capture still or moving images of the surrounding area. However, such systems may be affected by insufficient light and poor quality of the images captured, and may encounter great problems in identifying specific obstacles or situations (such as splashed liquids on the floor). Improved methods and systems are needed to overcome the shortcomings of common automatic cleaning machines that use traditional visual systems operating in the visible spectrum. Summary of the invention

[0006] Some embodiments may include an automated cleaning system and method for utilizing SWIR sensing in automated cleaning. The system may include: a first illumination source for emitting radiation of a first wavelength in the SWIR toward a field of view (FOV); a receiver operating in the SWIR to acquire SWIR image data based on radiation reflected from an element located in the FOV, and determining the presence of a water-based liquid in the FOV based on the SWIR image data.

[0007] Some embodiments may include a second illumination source for emitting radiation at a second wavelength in the SWIR toward a target within the FOV and determining a spectral characteristic of the target based on absorption of the target at the first wavelength and absorption of the target at the second wavelength.

[0008] According to some embodiments of the present disclosure, the second illumination source may emit radiation toward the target by emitting radiation of a second wavelength in SWIR toward a second FOV included in the first FOV.

[0009] According to some embodiments of the present disclosure, the second illumination source may include a low-power illumination source.

[0010] According to some embodiments of the present disclosure, the receiver may include one or more germanium (Ge) photodiodes (PD).

[0011] According to some embodiments of the present disclosure, the first illumination source may include a laser source.

[0012] According to some embodiments of the present disclosure, the first wavelength may be between 1300nm-1400nm.

[0013] According to some embodiments of the present disclosure, the second illumination source may include a light emitting diode (LED).

[0014] According to some embodiments of the present disclosure, the second wavelength is between 800nm-1000nm.

[0015] Some embodiments may include a third illumination source for emitting radiation at a third wavelength in the SWIR toward a target within the FOV and determining a spectral characteristic of the target based on absorption by the target at the first wavelength, absorption by the target at the second wavelength, and absorption by the target at the third wavelength. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The subject matter regarded as the present disclosure is particularly pointed out and distinctly claimed in the concluding portion of the specification. Figure 1 The present disclosure, both as to organization and method of operation, together with objects, features, and advantages thereof, may be best understood by reference to the following detailed description when read together with the accompanying drawings, wherein:

[0017] Figure 1 is a schematic diagram of components of an exemplary non-limiting architecture of an automated cleaning system according to an embodiment of the present disclosure;

[0018] Figure 2A , 2B and 2C are schematic diagrams of an exemplary automated cleaning system according to an embodiment of the present disclosure;

[0019] Figure 3 is a flow chart of a method for using a SWIR vision system in an automated cleaning system according to some embodiments of the present disclosure;

[0020] Figure 4shows the timing of three different detection signals arriving from the same direction within the FOV according to an embodiment of the present disclosure; and

[0021] Figure 5 is a block diagram depicting a computing device that may be included in embodiments of a system for utilizing SWIR sensing in a cleaning machine, according to some embodiments of the present disclosure.

[0022] It should be understood that, in order to make the illustration simple and clear, the elements shown in the figures are not necessarily drawn to scale. For example, for the sake of clarity, the size of some elements may be enlarged relative to other elements. In addition, where appropriate, the reference numerals in the figures may be repeated to indicate corresponding or similar elements. DETAILED DESCRIPTION

[0023] Those skilled in the art will recognize that the present disclosure may be implemented in other specific forms without departing from the spirit or essential characteristics of the present disclosure. Therefore, the foregoing embodiments are considered in all respects to be illustrative rather than limiting of the disclosure described herein. Therefore, the scope of the present disclosure is represented by the appended claims rather than the foregoing description, and therefore, all changes within the meaning and equivalent range of the claims are intended to be included therein.

[0024] In the detailed description below, in order to provide a thorough understanding of the present disclosure, many specific details are set forth. However, it will be appreciated by those skilled in the art that the present disclosure may be practiced without these specific details. In other cases, well-known methods, procedures and components are not described in detail to avoid blurring the present disclosure. Some features or elements described about an embodiment may be combined with features or elements described about other embodiments. For the sake of clarity, the discussion of the same or similar features or elements may not be repeated.

[0025] Although the embodiments disclosed herein are not limited in this regard, discussions utilizing terms such as, for example, "process," "compute," "calculate," "determine," "establish," "analyze," "examine," etc. may refer to operations and / or processes of a computer, computing platform, computing system, or other electronic computing device that manipulates and / or transforms data represented as physical quantities (e.g., electronic) within the computer's registers and / or memories into other data similarly represented as physical quantities within the computer's registers and / or memories, or other information non-transitory storage media that may store instructions that, when executed by a processor, cause the processor to perform the operations and / or processes.

[0026] Although the embodiments disclosed herein are not limited in this regard, the terms "plurality" and "multiple" used herein may include, for example, "more" or "two or more". The terms "plurality" or "multiple" may be used throughout the specification to describe two or more components, devices, elements, units, parameters, etc. When used herein, the term "group" may include one or more items.

[0027] Unless explicitly stated, the method embodiments described herein are not limited to a particular order or sequence. Additionally, some of the described method embodiments or their elements may occur or be performed simultaneously, at the same point in time, or concurrently.

[0028] Embodiments disclosed herein can overcome the shortcomings of common visual systems used in automatic cleaning machines. Automatic or robotic cleaning machines using cameras or sensors that detect visible light can use light wavelengths of 400 to 700 nanometers (nm: nano-meter). Such cameras or sensors have some disadvantages because they may not be able to detect certain obstacles on the floor or the cleaned area. For example, liquids such as water, cleaning fluid, urine, or any other liquid may not be detected by a visible camera due to the transparency of the liquid. An automatic cleaning machine using a SWIR imaging system can easily detect and identify liquids, such as liquids containing water. The detection of water-based liquids can be based on specific characteristics of water, for example, water absorbs wavelengths in the 1450nm region, so dark areas are returned in the captured image. Other advantages of cleaning machines using SWIR imaging systems may include specific material detection. Material detection can be used to identify materials in the cleaned area, such as wooden floors, parquet floors, marble floors, tile floors, etc. As described in the embodiments of the present disclosure, the use of a SWIR imaging system in a cleaning device can improve the current system and overcome the significant shortcomings of cleaning machines known in the art.

[0029] Figure 1 A schematic diagram showing components of an exemplary non-limiting architecture of an automated cleaning system according to an embodiment of the present disclosure. Automated cleaning system 100 may include short wave infrared (SWIR) imaging system 110, control system 120, and processor 130. Automated cleaning system 100 (also referred to herein as "system 100") may be any automated or semi-automated cleaning machine, such as a mobile robot, a robotic vacuum cleaner, an automated floor cleaning machine, a ride-on floor scrubber, or any other automated or semi-automated cleaning machine, sweeper, scrubber, vacuum cleaner, or any other cleaning machine.

[0030] The SWIR imaging system 110 may include one or more illumination sources 112, one or more SWIR receivers or sensors 111, and optics 113. The control system 120 may include an imaging system control unit 121, an analysis unit 122, and a navigation unit 123. A processor 130 may be connected to the SWIR imaging system 110 and the control system 120. In some embodiments of the present disclosure, the processor 130 may be implemented as an internal processor in each of the control system 120 and / or the imaging system 110, or as an external processor, for example, external to the system 100. The SWIR imaging system 110 is operable to detect light reflected from the FOV and provide one or more images of the FOV or a portion thereof based on the light reaching the imaging system 110.

[0031] The illumination source 112 may include one or more illumination sources for illuminating or emitting radiation in the SWIR band toward an object, area, or scene, all of which are referred to herein as a target 140. The term "target" refers to any object, area, subject, element, surface, content, and / or anything, such as solids, liquids, flexible objects, and rigid objects, that includes or is located in the FOV of the imaging sensor. The illumination emitted from the system 100 is represented by 141, and the illumination reflected from the target 140 back toward the system 100 is represented by 142. Portions of the emitted radiation 141 may also be reflected, deflected, or absorbed by the target 140 in other directions (not shown).

[0032] The illumination source 112 may include any light source or illumination source that can provide a certain level of illumination to the area around the system 100. Any suitable type of illumination source 112 may be used, such as a light-emitting diode (LED), a laser diode, a quasi continuous wave (QCW) laser, a vertical cavity surface emitting laser (VCSEL), a Q-switched (QS) laser, etc. Any other illumination source or any combination of illumination sources may be included in the illumination source 112. The illumination source 112 may emit light of any bandwidth that can be detected by the receiver 111 to generate an image. For example, the light 141 emitted by the illumination source 112 may be in the SWIR region of the electromagnetic spectrum.

[0033] The illumination source 112 may include a first illumination source for emitting radiation of a first wavelength in the SWIR toward the FOV, for example, a wavelength in the range between 1300nm-1500nm. For example, emitting radiation or transmitting light at a specific wavelength (e.g., 1450nm or close to 1450nm) may allow the presence of water-based liquids within the FOV to be detected, observed, identified, or monitored because water absorbs radiation at a wavelength of 1450nm. The receiver 111 may acquire SWIR image data based on the radiation reflected from the target 140. Based on the SWIR image data, the presence of water-based liquids in the FOV may be determined by identifying dark areas in the image. For example, dark areas or black areas in the SWIR image may indicate the presence of water, water-based liquids, and any material containing water.

[0034] According to an embodiment of the present disclosure, the illumination source 112 can operate near 1400nm in the SWIR spectrum, also known as the "solar blind region", in which the SWIR becomes immune to ambient noise from the sun. Illumination near 1400nm (e.g., between 1300-1500nm) can result in a higher signal-to-noise ratio (SNR) and an overall improvement in image quality in common low-visibility scenes (such as glare, rain, smoke, and fog). This can allow the SWIR camera 111 to generate high-resolution images and expand the detection range.

[0035] The illumination source 112 may also include a second illumination source for emitting radiation of a second wavelength in the SWIR toward a target within the FOV or toward a second FOV included in the first FOV. According to an embodiment of the present disclosure, the second wavelength is predetermined to be different from the first wavelength (illuminated by the first illumination source) to allow classification or detection of materials on a surface, plane, object, obstacle, or any other target, area, object, or surface in the FOV. For example, the first illumination source may be a VCSEL emitting 1450nm light, while the second illumination source may be an LED emitting 940nm light. Accurate material classification based on spectral analysis may require any additional illumination source other than the first illumination source.

[0036] The receiver 111 may include one or more SWIR sensors or detectors for receiving and collecting radiation reflected from the target 140 and / or from the illuminated FOV. The receiver 111 may operate in the SWIR to acquire SWIR image data based on radiation reflected from an element, segment, or region located in the illuminated FOV. The receiver 111 may generate or generate an electrical signal in response to the detected electromagnetic radiation representing the imaging of the illuminated FOV. The receiver 111 is capable of capturing images of an area surrounding, adjacent to, or close to the location of the automatic cleaning system 100. For example, the receiver 111 may be or may be implemented as a forward-facing camera to capture images in the forward travel direction of the automatic cleaning system 100. In some embodiments, the processing of the output of the receiver 111 may be performed by the processor 130, and additionally or alternatively by an external processor (not shown).

[0037] Receiver 111 and / or illumination source 112 may utilize optics 113. Optics 113 may include one or more optical elements, such as reflectors, lenses, diffusers, or any other element to improve, enhance, and / or support receiver 111 and / or illumination source 112. For example, optics 113 may be arranged to collect, concentrate, and optionally filter reflected radiation 142, and focus the electromagnetic radiation onto a focal plane of receiver 111.

[0038] According to some embodiments of the present disclosure, the receiver 111 may include a plurality of photodetection devices, such as a photodetector array or "PDA", which may include a plurality of photosites, each photosite including one or more photodiodes for detecting incident light and a capacitor for storing charge provided by the photodiode. In the following, "photosite" is generally replaced by the acronym "PS". The term "photosite" refers to a single sensor element of a sensor array. Each PS may include one or more photodiodes. In addition to the photodiodes, the PS may also include some circuits or additional components.

[0039] According to some embodiments of the present disclosure, the receiver 111 may include a plurality of germanium (Ge) photodetectors (PDs) operable to detect reflected SWIR radiation. In some embodiments, the receiver 111 may include a SWIR focal plane array (FPA), and the plurality of Ge PDs may be part of or form a SWIR FPA. The receiver 111 may generate an electrical signal for each of the plurality of Ge PDs, the electrical signal representing an amount of incident SWIR light within a detectable spectral range of each Ge PD. The amount of light detected at the receiver 111 may include an amount of SWIR radiation reflected from the target 140, and may also include an additional amount of SWIR light arriving, for example, from sunlight or from an external light source.

[0040] The term "Ge PD" relates to any PD in which light-induced electron excitation (which can subsequently be detected as a photocurrent) occurs within Ge, within a Ge alloy (e.g., silicon germanium), or at an interface between Ge (or a Ge alloy) and another material (e.g., silicon, silicon germanium). In particular, the term "Ge PD" relates to both pure Ge PDs and Ge-silicon PDs. When a Ge PD comprising Ge and silicon is used, different concentrations of germanium can be used. For example, the relative proportion of Ge in the Ge PD (whether alloyed with silicon or adjacent to it) can be in the range of 5% to 99%. For example, the relative proportion of Ge in the Ge PD can be between 15% and 40%. Note that materials other than silicon can also be part of the Ge PD, such as aluminum, nickel, silicide, or any other suitable material. In some embodiments of the present disclosure, the Ge PD can be a pure Ge PD (comprising greater than 99.0% Ge).

[0041] In some embodiments of the present disclosure, the receiver 111 may be implemented as a PD array fabricated on a single chip. The PDs (e.g., Ge PDs) may be arranged in any suitable arrangement, such as a rectangular matrix (straight rows and columns of Ge PDs), a honeycomb tiling, and even an irregular configuration. Preferably, the number of Ge PDs in the receiver 111 may allow for the generation of high-resolution images. For example, the number of PDs may be on the order of 1 million pixels, 10 million pixels, or more.

[0042] Exemplary embodiments disclosed herein relate to systems and methods for high SNR active SWIR imaging using a receiver including a Ge-based PD. The main advantage of Ge receiver technology over InGaAs technology is compatibility with CMOS processes, allowing the fabrication of the receiver as part of a CMOS production line. For example, Ge PDs can be integrated into a CMOS process, such as in Si photonics, by growing a Ge epitaxial layer on a silicon (Si) substrate. Therefore, Ge PDs are also more cost-effective than equivalent InGaAs optical receivers.

[0043] The control system 120 may control the operation of the imaging system 110 through the imaging system control unit 121. The imaging system control unit 121 may control the operation of the receiver 111, the illumination source 112, and the optical device 113. The control unit 121 may trigger the activation of the illumination source 112 according to a predetermined method to match the operation of the receiver 111. For example, in some embodiments of the present disclosure, the control unit 121 may trigger the illumination source to operate only during a time period when the receiver 111 is turned on to detect the reselected light. In some embodiments of the present disclosure, the control unit 121 may trigger the illumination source 112 to operate continuously in pulses or in any other method during a specific time period.

[0044] According to some embodiments of the present disclosure, the control unit 121 may be configured to control the activation of the receiver 111 within a relatively short integration time in order to limit the impact of the accumulation of dark current noise on the quality of the generated signal. For example, the control unit 121 may be operable to control the activation of the receiver 111 within an integration time in which the accumulated dark current noise does not exceed the readout noise that is independent of the integration time.

[0045] The control system 120 may control and / or perform analysis of the imaging data via the analysis unit 122. The analysis unit 122 may be connected to the imaging system 110 and may specifically receive information from the receiver 111. The analysis unit 122 may receive an electrical signal generated in response to electromagnetic radiation detected by the receiver 111. The received signal represents an imaging of an illuminated scene in the FOV. In some embodiments of the present disclosure, the signal detected by the receiver 111 may be processed by the analysis unit 122 and / or may be transmitted to the processor 130 via the analysis unit 122 to be processed into a SWIR image of the target 140 or the illuminated FOV. The analysis unit 122 may further process the electrical signal generated in response to the electromagnetic radiation to detect or observe the target 140, perform spectral analysis of the target 140, determine spectral characteristics of the target 140, and / or determine the presence of one or more materials included in the target 140.

[0046] According to some embodiments of the present disclosure, spectral analysis may be performed based on absorption of the target 140 at a first wavelength and absorption of the target at a second wavelength. The target 140 may be illuminated at two different SWIR wavelengths, for example, a first illumination source may emit radiation at a first wavelength, and a second illumination source may emit radiation at a second wavelength different from the first wavelength. Based on the absorption of the target 140 at the first wavelength and the absorption of the target 140 at the second wavelength, spectral analysis of the target 140 may be performed, and spectral characteristics of the target 140 may be derived or determined.

[0047] According to some embodiments of the present disclosure, the illumination source 112 may emit light of a specific wavelength, which may be selected to allow detection, observation, or determination of the presence of a specific material, such as for detecting water or any liquid including water. For example, the illumination source 112 may emit radiation of a wavelength of 1400 nm or 1450 nm, because water has a significant absorption peak near a wavelength of 1450 nm.

[0048] The control system 120 can also control the navigation of the cleaning system 100 through the navigation unit 123. According to some embodiments of the present disclosure, based on the generated SWIR image of the FOV, the analysis unit 122 can detect or observe the target 140. For example, any obstacles or elements detected in the FOV can be identified and used to control the navigation of the cleaning system 100. The imaging system 110 may include additional types of sensors to provide the cleaning system 100 with information about its surroundings for navigation purposes. For example, the imaging system 110 may include one or more position sensitive devices, one or more physical contact sensors, infrared (IR) sensors, one or more cameras or sensors that detect visible light, or any other type of sensor. The navigation unit 123 can receive signals from any sensor of the imaging system 110, and can adjust the position and trajectory of the cleaning system 100 based on the received signals.

[0049] All information and data collected by the imaging system 110 can be fed into the control system 120, and specifically fed into the analysis unit 122 and the navigation unit 123. The analysis unit 122 can receive images taken by the imaging system 110, and can analyze the images to find significant or landmark features within the FOV (e.g., within the area around the cleaning system 100). Significant or landmark features may include high-contrast features that can be detected within the image, such as spilled water, table legs, shoes, cables, carpets, or any other objects. The navigation unit 123 can use the detected landmark features to triangulate and determine the location of the cleaning system 100 within the local environment, and avoid obstacles without contacting them. For example, by identifying spilled liquids, such as a puddle of water, soapy water, urine, or any other water-based liquid, the cleaning system 100 can avoid driving into the liquid, can bypass the liquid, etc.

[0050] Figure 2A , 2B 2C are schematic diagrams of exemplary automated cleaning systems according to embodiments of the present disclosure. Figure 2A is a top view of the cleaning system 100, Figure 2B and 2C is a side view of the cleaning system 100. The cleaning system 100 may be Figure 1 The cleaning system 100 may include or may utilize Figure 1 The cleaning system 100 may include a body 200, a cleaning head 213, and other components that allow the system 100 to move, such as a motor, a transmission unit, a power unit, a battery, one or more wheels, and / or any other elements required for movement and cleaning (not shown). It should be understood by those skilled in the art that Figure 2A-2C The shapes and designs of the body 200 and any other components shown in FIG. 1 are intended as exemplary designs, and the cleaning system 100 may have any shape, design, structure, or appearance.

[0051] According to an embodiment of the present disclosure, the main body 200 may include a camera 210 to capture images of the area around the cleaning system 100. The camera 210 may include Figure 1 The imaging system 110. The camera 210 may be any type of camera, which may include Figure 1 A SWIR device or receiver 111. Figure 1 The control system 120 may be embodied in software and electronics contained within the cleaning system 100 that may be capable of processing images captured by the camera 210 , thereby allowing the cleaning system 100 to understand, interpret, and autonomously navigate the local environment.

[0052] Figure 2A and 2B The exemplary illustration of 200 shows a FOV 220 having a horizontal FOV 205 of 45 degrees and a vertical FOV 223 of 5 degrees. A first illumination source included in the cleaning system 100 can illuminate a segment located at a minimum distance 221 (e.g., 25 centimeters (cm)) and a maximum distance 222 (e.g., 50 cm) from the cleaning system 100. As described in an embodiment of the present disclosure, the first illumination source can emit light having a wavelength between 1300 nm and 1500 nm (e.g., at 1450 nm) to identify or detect water-based liquids on a surface illuminated by the first illumination source.

[0053] According to an embodiment of the present disclosure, to allow for high accuracy material classification, at least one additional illumination source may be used to illuminate the second FOV 230 at a second wavelength different from the first wavelength (used by the first illumination source). Illuminating the second FOV 230 within the FOV 220 using the second illumination source using the second wavelength may allow for improved material classification, such as material classification of the object or area 215 included in the second FOV 230. Since each material has a unique spectral response, illuminating the inspected material (e.g., the object 215) at more than one wavelength may allow for identification, determination, or detection of the spectral response of the inspected material.

[0054] Embodiments disclosed herein may allow for the use of a first primary illuminator, such as a VCSEL, for illuminating the primary FOV 220, and for each additional wavelength, adding one or more additional illumination sources with limited FOV 230. This may allow for a reduction in the total power required for the cleaning system 100, since the additional illumination sources may require lower energy to illuminate a limited portion of the FOV 230 within the primary FOV 220. Each of the one or more additional illumination sources may include, for example, an IR LED that illuminates at a specific, predetermined, selected wavelength.

[0055] According to an embodiment of the present disclosure, the material of the target 215 (e.g., a floor) can be determined based on at least two images of a specific area, target, or space including the detected material. Since each material has a unique spectral response, such as a unique spectral reflectance, this unique spectral response can be used to determine the material of the illuminated area. The embodiments disclosed herein can determine the material of the detected target based on at least two images taken at two different wavelengths in the SWIR. For example, a first image can be acquired based on irradiating a larger main FOV 220 at a first wavelength, and a second image can be acquired based on irradiating a portion of the FOV 230 included in the FOV 220 at a second wavelength. Other examples may include irradiating the same FOV with two or more wavelengths from two or more illumination sources.

[0056] Embodiments disclosed herein may include illuminating a detected material (e.g., FOV 230) at a first wavelength and a second wavelength, and obtaining or identifying two values ​​of reflectivity and / or absorptivity of the detected material (a first value associated with the first wavelength and a second value associated with the second wavelength). The two values ​​of reflectivity and / or absorptivity may match the spectral response of a particular material, which may be identified or determined as the detected material.

[0057] According to embodiments of the present disclosure, two values ​​of reflectivity and / or absorptivity may be the minimum number of values ​​based on which the spectral response may be determined, however, embodiments disclosed herein may allow any number of additional illumination sources to emit radiation at any number of additional wavelengths. For example, a third illumination source for emitting radiation at a third wavelength in the SWIR toward a target within the FOV may be used, and the spectral characteristics of the target may be determined based on the absorption of the target at the first wavelength, the absorption of the target at the second wavelength, and the absorption of the target at the third wavelength.

[0058] refer to Figure 2C , which is a side view of cleaning system 100 according to an embodiment of the present disclosure. System 100 can provide an indication of any obstacles that may be near body 200 of cleaning system 100, or located in an area close to body 200 of cleaning system 100. This can allow cleaning system 100 to avoid obstacles, such as obstacle 228, without contacting them. Based on identifying obstacles in an image captured by imaging system 110, Figure 1 The controller 120 can generate control signals and can instruct the cleaning system 100 to stop and / or adjust its position and trajectory. This can prevent the cleaning system 100 from causing any damage to itself or to obstacles.

[0059] Figure 2C The exemplary illustration of shows a FOV 220 including a vertical FOV 223 of 17 degrees. The first illumination source can illuminate a segment located at a minimum distance 221 (e.g., 25 cm) and a maximum distance 224 (e.g., 200 cm) from the cleaning system 100. For example, an obstacle 228 can be detected in an image captured by the imaging system 110 of the cleaning system 100, the obstacle 228 being located within a distance 224 of approximately 150 cm from the cleaning system 100 and having a reflectivity difference of at least 20% with the background.

[0060] For example, in some embodiments of the present disclosure, a 17 degree vertical FOV 223 may be selected to allow for capturing a floor or other item at a distance 222 of 50 cm from the cleaning system 100 and at a height 227 of 10 cm. Figure 1 If a single illumination source is implemented in the imaging system 110 of FIG. 10 , the illumination source may have a Gaussian distribution to have a clear image at a distance 222 of 50 cm from the cleaning system 100 and a height 227 of 10 cm. Adding one or more illumination sources may expand the detection range and the total power, respectively.

[0061] The images captured by imaging system 110 may be analyzed to find notable or landmark features within the FOV, such as obstacle 228, which may have high contrast features that may be detected within the SWIR image. Figure 1The navigation unit 123 can use the detected obstacle 228 to determine the position of the cleaning system 100 in the local environment to avoid contact with the obstacle 228, such as bypassing the obstacle 228, etc.

[0062] Reference Figure 3 , which is a flow chart of a method for using a SWIR vision system in an automatic cleaning system according to some embodiments of the present disclosure. An embodiment of the method for using a SWIR vision system in an automatic cleaning system may be, for example, Figure 1 The example system shown is implemented.

[0063] In operation 310, a first illumination source may emit radiation of a first wavelength in the SWIR toward a first FOV. For example, the first illumination source (e.g., Figure 1 The illumination source 112 may emit radiation having a wavelength in the range of 1300 nm to 1500 nm. In some embodiments of the present disclosure, emitting radiation or transmitting light of a specific wavelength (e.g., 1450 nm or close to 1450 nm) may allow for the detection, observation, identification, or monitoring of the presence of a water-based liquid within the first FOV.

[0064] In operation 320, a receiver operating in the SWIR may receive radiation reflected from an element located in the FOV (e.g., from an area, object, item, or any other component located in the FOV). The receiver (e.g., Figure 1 The receiver 112 may include one or more SWIR sensors or detectors for receiving and collecting radiation reflected from elements located in the FOV. The receiver may generate an electrical signal in response to the detected electromagnetic radiation representing the image of the illuminated FOV.

[0065] In operation 330, SWIR image data may be acquired based on radiation reflected from an element located in the FOV. For example, an element located in the FOV may refer to any object, region, subject, surface, content, and / or anything included in or located in the FOV of the imaging sensor, such as a solid, liquid, flexible object, and rigid object.

[0066] In operation 340, the presence of water-based liquid in the FOV may be determined based on the SWIR image data. For example, because water absorbs radiation having a wavelength of 1450 nm, the SWIR image data may be acquired based on radiation reflected from the illuminated FOV, and the presence of water-based liquid in the FOV may be determined by identifying dark areas in the image.

[0067] In operation 350, a second illumination source may emit radiation of a second wavelength in the SWIR toward a target within the FOV and / or toward a second FOV included in the first FOV. According to an embodiment of the present disclosure, the second wavelength is selected to be different from the first wavelength to allow for detection and spectral classification of materials of a surface, plane, object, obstacle, or any other target in the FOV.

[0068] In operation 360, a spectral characteristic of the target may be determined based on the absorption of the target at the first wavelength and the absorption of the target at the second wavelength. Based on the absorption of the target at the first wavelength and the absorption of the target at the second wavelength, a spectral analysis of the target may be performed and the spectral characteristic of the target may be derived or determined.

[0069] Unless explicitly stated, the method embodiments described herein are not limited to a particular order or sequence. In addition, all operations described herein are intended only as examples, and other or different operations may be used. Additionally, some of the described method embodiments or their elements may occur or be performed at the same time point.

[0070] In some embodiments, receiver 111 is activated multiple times to create "time slices," each covering a specific distance range. Image processor 130 can combine these slices to create a single image with greater visual depth to allow the distance of objects in the FOV to be determined or measured.

[0071] According to some embodiments of the present disclosure, different techniques may be used to detect the presence of a light source based on one or more light-sensitive points (e.g., included in Figure 1 The depth is determined by the output of the photosensitive point in the receiver 111. The embodiments disclosed herein can be used to determine the depth of a SWIR electro-optical system (e.g., Figure 1 The distance of an object in the FOV of an imaging system 110 (as well as other electro-optical systems sensitive to other parts of the electromagnetic spectrum).

[0072] According to some embodiments of the present disclosure, the method for determining depth may be used, for example, by the control system 120 of the cleaning system 100 to provide the cleaning system 100 with information about its surroundings for navigation purposes. Figure 1 The navigation unit 123 can adjust the position and trajectory of the cleaning system 100 based on the depth measurements, which can indicate the distance of one or more obstacles, landmarks, or any other elements located near the cleaning system 100.

[0073] Embodiments of the present invention may generate a depth image of a scene based on detection by a SWIR electro-optical imaging system (SEI system). According to examples of the presently disclosed subject matter, the SEI system described herein may be implemented in any system of the cleaning system 100, for example, Figure 1 The SEI system may be any of the systems discussed above, or any other suitable SWIR electro-optical system (e.g., sensor, camera, lidar, etc.). The methods described herein may be executed by one or more processors of the SEI system, such as Figure 1 processor 130, and / or one or more processors external to the SEI system, or a combination of both.

[0074] A plurality of detection signals of the SEI system may be obtained. Each detection signal may indicate the amount of light captured from a particular direction within the FOV of the SEI system by at least one FPA detector of the SEI system during a corresponding detection time frame (i.e., during which the corresponding detection signal is captured, e.g., measured from the start of illumination triggered by an associated light source, such as a laser). The at least one FPA may include a plurality of individual photosites, each of which may include a Ge element in which an incident photon is converted into a detected charge. Any type of photosite characterized by a high dark current may be implemented in the SEI system, even if Ge is not included but other elements are included.

[0075] For each of the plurality of directions within the FOV, different detection signals (of the plurality of detection signals) indicate levels of reflected SWIR illumination from different distances along that direction.

[0076] refer to Figure 4 , which shows the timing of three different detection signals arriving from the same direction within the FOV according to an embodiment of the present disclosure. Figure 4 Graph 571 of shows the timing of three different detection signals arriving from the same direction within the FOV. The y-axis (ordinate) in the graph represents the level of response of the detection system to reflected photons arriving from the direction of interest. The reflected illumination originates from one or more light sources (e.g., lasers, LEDs), which are optionally controlled by the same processor that controls the FPA, and is reflected from a portion of the FOV (e.g., corresponding to the volume of space detectable by a single photosite). Note that different detection signals can be associated with similar, but not completely overlapping, portions of the FOV (e.g., if the sensor, scene, or intermediate optics between the two are shifted in time, then detection signals from the same photosite may be reflected from slightly different angles within the FOV at different detection time windows associated with the different detection signals).

[0077] Note that graph 571 does not show the detection level of each signal, but rather shows the response of the detection signal to photons that reflect from a perfect reflector at different times from the start of light emission. Graph 572 shows three objects positioned at different distances from the SEI system (e.g., from cleaning system 100). Note that in many cases, only one object is detected at a time in each direction, which is the object closest to the SEI system. However, in some scenarios, more than one object may be detected (e.g., if the foreground object is partially transparent, or does not block light from the entire photosite).

[0078] Graph 573 shows the levels of three return signals in directions where one of the objects is present - in this example, a dog in the near field, a shoe in the mid-range, and a table in the far field (the choice of objects is arbitrary, and typically only a portion of the light reflected from each object is detected by a single photosite). For the three different detection signals (corresponding to different detection timing windows and different ranges from the SEI system), the light returned from an object at distance D1 is represented by a dog shape. Similarly, the levels of the detection signals corresponding to light reflected from objects at distances D2 and D3 are represented by shoe and table symbols, respectively.

[0079] As shown in graph 574, reflections from an object located at a given distance can be converted into a tuple (or any other representation of data, such as any suitable form of a direction-associated data-structure (DADS)) that indicates the relative levels of the signal detected over different time windows. In the example shown, each number in the tuple indicates the signal level detected in one detection window. The indication of the detection level in the tuple can be corrected for the distance from the sensor (because the reflected light from the same object decreases with distance), but this is not required. Although three partially overlapping time windows are used in the example shown, any number of time windows may be used. The number of time windows can be the same for different areas of the FOV, but this is not required.

[0080] Embodiments disclosed herein may include processing multiple detection signals to determine a three-dimensional (3D) detection map, which includes multiple 3D positions of objects detected in the FOV. The processing may include compensating for dark current (DC) levels accumulated during the collection of multiple detection signals generated by the Ge element, and the compensation includes applying different degrees of dark current compensation to the detection signals detected by different photosites of at least one focal plane array. With reference to the examples of the accompanying drawings, different detection signals can be obtained at different times by different readout structures of any applicable photosites discussed above. Alternatively, the detection signal can be obtained by a group of interconnected photosites, as discussed in more detail below. Other embodiments may also be used.

[0081] In addition to or instead of compensating for the accumulated dark current, the processing may include compensating for high integrated noise levels and / or read noise levels during readout of the plurality of detection signals. The compensation may include applying different degrees of noise level compensation to the detection signals detected by different photosites of the at least one focal plane array.

[0082] The collection of dark current, the compensation of read noise and / or the integrated noise can be performed in any suitable manner, such as by using any combination of one or more of the following: software, hardware and firmware. In particular, the compensation of the collection of dark current can be implemented using any combination of any one or more of the systems, methods and computer program products discussed above, or any part thereof. Some non-limiting examples of systems, methods and computer program products that can be used to compensate for dark current and apply a certain degree of dark current compensation to the detection signals detected by different photosites.

[0083] In some implementations, compensation can be performed during acquisition of multiple detection signals (eg, in the hardware level of the sensor), and processing can be performed on the detection signals that have been compensated for dark current accumulation.

[0084] In some embodiments of the present disclosure, compensation may optionally include: subtracting a first dark current compensation offset from a first detection signal detected by a first photosensitivity point corresponding to a first detection range, and subtracting a second dark current compensation offset different from the first dark current compensation offset from a second detection signal detected by the first photosensitivity point corresponding to a second detection range, wherein the second detection range is farther away from the SEI system than the first detection range.

[0085] Embodiments disclosed herein may include coordination of active illumination (e.g., by at least one light source of the SEI system) and acquisition of detection signals. Optionally, embodiments disclosed herein may also include: (a) triggering emission of a first illumination (e.g., laser, LED) in coordination with initiation of exposure of a first gated image, in which a plurality of first detection signals are detected for different directions in a plurality of directions; (b) triggering emission of a second illumination (e.g., laser, LED) in coordination with initiation of exposure of a second gated image, in which a plurality of second detection signals are detected for different directions in the second gated image; and (c) triggering emission of a third illumination (e.g., laser, LED) in coordination with initiation of exposure of a third gated image, in which a plurality of third detection signals are detected for different directions in the third gated image.

[0086] In this case, the processing may optionally include determining the presence of a first object in a first 3D position within a first direction among different directions based on at least one detection signal from each of the first image, the second image, and the third image, and determining the presence of a second object in a second 3D position within a second direction among different directions based on at least one detection signal from each of the first image, the second image, and the third image, wherein a distance of the first object from the SEI system is at least twice a distance of the second object from the SEI system.

[0087] Optionally, applying different degrees of DC compensation to detection signals detected by different photosites of the at least one FPA may include using detected dark current levels of different reference photosites that are shielded from light from the FOV.

[0088] Optionally, the compensation may include applying different degrees of DC compensation to detection signals detected simultaneously by different photosites of at least one FPA.

[0089] With reference to integrated noise and with reference to readout noise, it is noted that compensation for such noise may be operated by at least one processor executing an embodiment to be associated with the number of illumination pulses used to illuminate a portion of the FOV during acquisition of the corresponding detection signal. Different numbers of illumination pulses may result in significant non-linearity of the detected signal, which may optionally be corrected as part of the processing prior to determining the distance / 3D position of different objects in the FOV.

[0090] With reference to using DADS to determine the distance / 3D position of different objects in the FOV, note that different translation functions of DADS (e.g., tuples) versus distance can be used for different directions within the FOV, for example, to compensate for non-uniformity of detection channels (e.g., of sensors and / or detection objects) throughout the FOV, non-uniformity of illumination (e.g., use of multiple light sources, non-uniformity of light sources, or non-uniformity of optics), etc.

[0091] According to some embodiments of the present disclosure, different detection signals from the same direction within the FOV correspond to different detection windows, and the detection windows may have the same distance, or different distances. For example, the detection window may correspond to a distance range of approximately 50 cm (e.g., between 80 cm from the SEI system and 130 cm from the SEI system). In different examples, some or all of the detection windows used to determine the distance / 3D position of an object in the FOV may have a distance range between 0.01m-0.1m, between 0.1m-10m, between 5m-25m, between 20m-50m, between 50m-100m, between 100m-250m, etc. The distance ranges associated with different detection signals may overlap. For example, a first detection window may detect return light from an object with a distance between 0cm and 50cm from the SEI system, a second window may correspond to an object between 25cm and 75cm, and a third window may correspond to an object between 50cm and 150cm.

[0092] Embodiments disclosed herein may include a system for generating a depth image of a scene based on detection by a short wave infrared (SWIR) electro-optical imaging system (SEI system), the system comprising at least one processor configured to: obtain a plurality of detection signals of the SEI system, each detection signal indicating an amount of light captured by at least one FPA detector of the SEI system from a particular direction within a FOV of the SEI system within a corresponding detection time frame, the at least one FPA comprising a plurality of separate photosites, each photosite comprising a Ge element in which incident photons are converted into detection charges, wherein for each of the plurality of directions within the FOV, a different detection signal indicates a level of SWIR illumination reflected from a different range of distances along the direction; and process the plurality of detection signals to determine a three-dimensional (3D) detection map comprising a plurality of 3D positions of detected objects in the FOV, wherein the processing comprises compensating for a dark current (DC) level accumulated during collection of the plurality of detection signals generated by the Ge element, wherein the compensation comprises applying different degrees of DC compensation to the detection signals detected by different photosites of the at least one FPA.

[0093] In some embodiments of the present disclosure, compensation may include: subtracting a first DC compensation offset from a first detection signal detected by a first DE corresponding to a first detection range; and subtracting a second DC compensation offset different from the first DC compensation offset from a second detection signal detected by the first DE corresponding to a second detection range, the second detection range being farther away from the SEI system than the first detection range.

[0094] In some embodiments of the present disclosure, at least one processor may be further configured to: (a) trigger emission of a first illumination in coordination with initiation of exposure of a first gated image, wherein a plurality of first detection signals are detected for different directions in the plurality of directions in the first gated image; (b) trigger emission of a second illumination in coordination with initiation of exposure of a second gated image, wherein a plurality of second detection signals are detected for different directions in the second gated image; and (c) trigger emission of a third illumination in coordination with initiation of exposure of a third gated image, wherein a plurality of third detection signals are detected for different directions in the third gated image. In this case, as part of determining the 3D detection map, at least one processor may be further configured to determine: (a) presence of a first object in a first 3D position within a first direction in the plurality of directions based on at least one detection signal from each of the first image, the second image, and the third image, and (b) presence of a second object in a second 3D position within a second direction in the plurality of directions based on at least one detection signal from each of the first image, the second image, and the third image, wherein a distance of the first object from the SEI system is at least twice a distance of the second object from the SEI system. A gated image (or its equivalent) may be achieved by utilizing different readout configurations of the photosites of the PDA, for example, in any of the ways discussed above.

[0095] In some embodiments of the invention, applying different degrees of DC compensation to detection signals detected by different photosites of the at least one FPA includes using dark current levels detected by different reference photosites, the different reference photosites being shielded from light arriving from the FOV. Optionally, the compensation may include applying different degrees of DC compensation to detection signals detected simultaneously by different photosites of the at least one FPA. Optionally, one or more (and possibly all) of the at least one processor may be part of the SEI system.

[0096] Reference now Figure 5 , which is a block diagram depicting a computing device according to some embodiments of the present disclosure, which may be included in an embodiment of a system utilizing SWIR sensing in a cleaning machine.

[0097] The computing device 1 may include a controller 2, which may be, for example, a central processing unit (CPU), a processor, a chip, or any suitable computing or computing device, an operating system 3, a memory 4, an executable code 5, a storage system 6, an input device 7, and an output device 8. The controller 2 (or one or more controllers or processors, possibly across multiple units or devices) may be configured to perform the methods described herein, and / or to perform or act as various modules, units, etc. According to an embodiment of the present disclosure, more than one computing device 1 may be included in the system, and one or more computing devices 1 may act as components of the system. For example, the computing device 1 may include Figure 1 In the imaging system 1, the control system 120 and / or the processor 130 of the cleaning system 100.

[0098] Operating system 3 may be or may include any code segment (e.g., a code segment similar to executable code 5 described herein) that is designed and / or configured to perform tasks involving coordination, scheduling, arbitration, supervision, control, or otherwise managing the operation of computing device 1, such as scheduling the execution of software programs or tasks or enabling software programs or other modules or units to communicate. Operating system 3 may be a commercial operating system. Note that operating system 3 may be an optional component, for example, in some embodiments, the system may include a computing device that does not require or include operating system 3.

[0099] For example, the memory 4 may be or may include a random access memory (RAM), a read only memory (ROM), a dynamic RAM (DRAM), a synchronous DRAM (SD-RAM), a double data rate (DDR) memory chip, a flash memory, a volatile memory, a non-volatile memory, a cache memory, a buffer, a short-term storage unit, a long-term storage unit, or other suitable memory unit or storage unit. The memory 4 may be or may include a plurality of possibly different storage units. The memory 4 may be a computer or processor non-transitory readable medium, or a computer non-transitory storage medium, such as RAM.

[0100] The executable code 5 may be any executable code, such as an application, a program, a process, a task or a script. The executable code 5 may be executed by the controller 2 under the control of the operating system 3. For example, the executable code 5 may be an application that allows the automatic acquisition of SWIR images. Although for clarity, Figure 4A single executable code 5 is shown in FIG. 1 , but the system according to some embodiments includes multiple executable code segments similar to the executable code 5 , which can be loaded into the memory 4 and cause the controller 2 to perform the methods described herein.

[0101] The storage system 6 may be or may include, for example, a flash memory known in the art, a memory built into or embedded in a microcontroller or chip known in the art, a hard drive, a CD-Recordable (CD-R) drive, a Blu-ray disk (BD), a USB (universal serial bus) device, or other suitable removable and / or fixed storage unit. Content may be stored in the storage system 6 and may be loaded from the storage system 6 into the memory 4, where it may be processed by the controller 2. In some embodiments, Figure 4 Some of the components shown may be omitted. For example, the memory 4 may be a non-volatile memory having the storage capacity of the storage system 6. Thus, although the storage system 6 is shown as a separate component, the storage system 6 may be embedded or included in the memory 4.

[0102] The input device 7 may be or may include any suitable input device, component or system, such as a detachable keyboard or keypad, a mouse, etc. The output device 8 may include one or more (possibly detachable) displays or monitors, speakers and / or any other suitable output devices. Any applicable input / output (I / O: input / output) device may be connected to the computing device 1, as shown in boxes 7 and 8. For example, a wired or wireless network interface card (NIC: network interface card), a universal serial bus device or an external hard drive may be included in the input device 7 and / or the output device 8. It should be appreciated that any suitable number of input devices 7 and output devices 8 may be operably connected to the computing device 1, as shown in boxes 7 and 8.

[0103] Systems according to some embodiments may include components such as, but not limited to, multiple central processing units (CPUs) or any other suitable multi-purpose or special-purpose processors, controllers (e.g., controllers similar to controller 2), microprocessors, microcontrollers, field programmable gate arrays (FPGAs), programmable logic devices (PLDs), or application-specific integrated circuits (ASICs). Systems according to some embodiments may include multiple input units, multiple output units, multiple memory units, and multiple storage units. The system may additionally include other suitable hardware components and / or software components. Where applicable, the modules or units described herein may be similar to or may include components of device 1 described herein.

[0104] Unless otherwise specified, the above functions herein may be performed by executable codes and instructions stored in computer-readable media and running on one or more processor-based systems. In addition, those skilled in the art will appreciate that the present disclosure may be practiced with other computer system configurations, including multiprocessor systems, microprocessor-based electronic devices, minicomputers, mainframe computers, etc.

[0105] The terms "for example," "such as," and "optionally" as used herein are intended to introduce non-limiting examples. Although certain example system components or algorithms are mentioned herein, other components and algorithms may also be used, and / or the example components may be combined into fewer components and / or divided into more components.

[0106] Although certain features of the present disclosure have been shown and described herein, many modifications, substitutions, changes and equivalents may occur to those skilled in the art. It should therefore be understood that the appended claims are intended to cover all modifications and changes that fall within the true spirit of the present disclosure.

[0107] Various embodiments have been proposed. Of course, each of these embodiments may include the features of the other embodiments described, and embodiments not specifically described may also include various features described herein.

Claims

1. An automatic cleaning system, comprising: a first illumination source for emitting radiation at a first wavelength in a shortwave infrared range (SWIR) towards a field of view (FOV); a receiver operative in the SWIR to acquire SWIR image data based on radiation reflected from elements located in the FOV; and A processor is provided for determining the presence of a water-based liquid in the FOV based on the SWIR image data.

2. The automatic cleaning system according to claim 1, further comprising: A second illumination source for emitting radiation at a second wavelength in the SWIR toward a target within the FOV, wherein the processor is operable to determine a spectral characteristic of the target based on absorption of the target at the first wavelength and absorption of the target at the second wavelength.

3. The automatic cleaning system according to claim 2, wherein: The second illumination source emits radiation toward the target by emitting radiation of the second wavelength in the SWIR toward a second FOV included in the first FOV.

4. The automatic cleaning system according to claim 2, wherein: The second illumination source comprises a low power illumination source.

5. The automatic cleaning system according to claim 1, wherein: The receiver includes one or more germanium (Ge) photodiodes (PDs).

6. The automatic cleaning system according to claim 1, wherein: The first illumination source includes a laser source.

7. The automatic cleaning system according to claim 1, wherein: The first wavelength is between 1300nm-1400nm.

8. The automatic cleaning system according to claim 1, wherein: The second illumination source includes a light emitting diode (LED).

9. The automatic cleaning system according to claim 1, wherein: The second wavelength is between 800nm-1000nm.

10. The automatic cleaning system according to any one of claims 1 to 9, further comprising: a third illumination source for emitting radiation of a third wavelength in the SWIR toward the target within the FOV, wherein the processor is further operable to determine a spectral characteristic of the target based on absorption of the target at the first wavelength, absorption of the target at the second wavelength, and absorption of the target at the third wavelength.

11. A method of using a short wave infrared range (SWIR) vision system in an automated cleaning system, the method comprising: emitting radiation at a first wavelength in the SWIR toward a field of view (FOV) by a first illumination source; receiving, by a receiver operating in the SWIR, radiation reflected from an element located in the FOV; acquiring SWIR image data based on radiation reflected from elements located in the FOV; as well as The presence of a water-based liquid in the FOV is determined based on the SWIR image data.

12. The method according to claim 11, further comprising: emitting radiation of a second wavelength in the SWIR toward a target within the FOV from a second illumination source; as well as A spectral characteristic of the object is determined based on the absorption of the object at the first wavelength and the absorption of the object at the second wavelength.

13. The method according to claim 12, wherein: Emitting radiation at a second wavelength in the SWIR toward a target within the FOV includes emitting radiation at the second wavelength in the SWIR toward a second FOV included in the first FOV.

14. The method according to claim 12, wherein: The second illumination source comprises a low power illumination source.

15. The method according to claim 11, wherein: The receiver includes one or more germanium (Ge) photodiodes (PDs).

16. The method according to claim 11, wherein: The first illumination source includes a laser source.

17. The method according to claim 11, wherein: The first wavelength is between 1300nm-1400nm.

18. The method according to claim 12, wherein: The second illumination source includes a light emitting diode (LED). The method of claim 12 , wherein the second wavelength is between 800 nm and 1000 nm.

20. The method according to any one of claims 11 to 19, further comprising: emitting radiation of a third wavelength in the SWIR toward the target within the FOV from a third illumination source; as well as A spectral characteristic of the object is determined based on absorption of the object at the first wavelength, absorption of the object at the second wavelength, and absorption of the object at the third wavelength.

Images