Detection of glass wall and door and distance measurement for robot, and method for controlling the same
Infrared-based detection and triangulation methods enable precise glass wall and door detection, addressing transparency issues in existing sensors, enhancing navigation and collision avoidance in autonomous systems.
Patent Information
- Application Number
- JP2024068804
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-04-27
- Filing Date
- 2024-04-22
- Publication Date
- 2024-11-13
AI Technical Summary
Existing sensors struggle to reliably detect glass walls and doors due to their transparency to lasers, LIDAR, and microwaves, leading to ineffective collision avoidance in autonomous systems.
Utilizing infrared (IR) imagers and emitters to detect IR heat signatures reflected from glass surfaces, employing triangulation to calculate distance and angle, and using resistor arrays to form thermal signatures for precise detection.
Effectively detects and maps glass walls and doors, enabling accurate navigation and collision avoidance in autonomous systems, while being cost-effective and compatible with existing systems.
Smart Images

Figure 2024160211000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to robots that require detection of glass walls and doors for navigation and collision avoidance. The present invention relates to delivery robots, cleaning robots, home robots, human aid robots and other robots. [Background technology]
[0002] Autonomous mobile objects such as suction cleaning robots, delivery robots, and flying drones are intended to navigate inside facilities or outdoors in unmanned or remote-controlled mode. For this purpose, robots are usually equipped with many sensors (ultrasonic sensors, laser range finders, LIDAR sensors, microwave sensors, image sensors, etc.) to avoid collisions with various kinds of obstacles.
[0003] Although glass is a very common material in buildings today, the above mentioned sensors have significant difficulties in detecting glass walls and doors: glass is transparent to lasers, LIDAR and microwaves, and in most cases glass walls / doors cannot be reliably detected by these sensors, whereas ultrasonic sensors cannot provide sufficient accuracy, especially to detect glass edges.
[0004] A very common solution to alleviate this problem is to equip the robot with touch sensors, buttons, or other sensors to detect collisions with obstacles after they have occurred, but this solution does not prevent collisions.
[0005] Therefore, through the present invention, it is desired to provide a system and method for an autonomous system to detect see-through objects such as glass panels / walls.
[0006] The present invention provides a glass wall and door detection and distance measurement device for a robot and a method for controlling the same. It is also hoped that the present invention provides a cost-effective solution that can be implemented or added to any existing autonomous system. Summary of the Invention
[0007] The present invention seeks to provide a method and system for detecting glass walls / reflective surfaces for an autonomous system. In one aspect of the present invention, a method for detecting glass walls with reflective surfaces for an autonomous system is provided. The method includes disposing at least one infrared (IR) imager having a field of view (FOV) and at least one IR emitter in the autonomous system, emitting an IR heat signature located outside the FOV from the autonomous system by the IR heat emitter, detecting the presence of the IR heat signature reflected by the reflective surface, and determining the distance and angle of the autonomous system from the reflective surface based on the reflected IR heat signature within the FOV. The reflective surface is identified as a glass wall.
[0008] In one embodiment, each IR emitter includes a number of resistors arranged to radiate a thermal signature.
[0009] In another embodiment, the IR emitter includes an array of resistors arranged to form a thermal signature.
[0010] In a further embodiment, the autonomous system determines distance and angle based on triangulation techniques.
[0011] In yet other embodiments, the IR heat emitter emits a thermal signature in the long wavelength IR band.
[0012] In another aspect of the invention, a sensor assembly of an autonomous system is provided that is adapted to detect a glass wall. The sensor assembly includes at least one infrared (IR) heat sensor having a field of view (FOV) and at least one IR emitter that, in operation, emits a heat signature. In operation, when the IR heat signature is reflected off a reflective surface and the reflected IR heat signature is captured within the FOV, the sensor assembly determines a distance and angle of the IR sensor from the reflective surface, and the reflective surface is identified as a glass wall.
[0013] In one embodiment, each of the IR emitters includes a resistor array, with each resistor unit in the resistor array forming a pixel of the thermal signature.
[0014] In yet other embodiments, the thermal signature is in the long wavelength IR spectrum.
[0015] In a further embodiment, the distance and angle are determined through triangulation.
[0016] In yet another embodiment, an autonomous system is provided having a sensor assembly as described above. [Brief description of the drawings]
[0017] The invention will now be described by way of non-limiting embodiments thereof with reference to the accompanying drawings, in which:
[0018] [Figure 1] FIG. 1 illustrates a method for detecting a glass wall for an autonomous system according to one embodiment of the present invention.
[0019] [Figure 2A] FIG. 2A illustrates a side view showing an autonomous system according to one embodiment of the present invention facing a glass wall.
[0020] [Figure 2B] FIG. 2B shows a plan view of the autonomous system of FIG. 2A facing a glass wall.
[0021] [Diagram 3] FIG. 3 shows a top view of an autonomous system having a sensor assembly for detecting a glass wall according to one embodiment of the present invention.
[0022] [Figure 4] 4A, 4B, and 4C illustrate three thermal signature patterns according to an embodiment of the present invention.
[0023] [Diagram 5] FIG. 5 illustrates a sensor assembly of an autonomous system according to an embodiment of the present invention.
[0024] [Figure 6] FIG. 6 illustrates another sensor assembly for an autonomous system according to another embodiment of the present invention.
[0025] [Figure 7] Figure 7 shows a graph of the reflectance of different types of glass taken from researchgate.net. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0026] In accordance with the above summary, the following provides a description of a number of specific and alternative embodiments so that the inventive features of the present invention can be understood. However, it will be apparent to those skilled in the art that the present invention can be practiced without such specific details. Some details may not be described in detail so as not to obscure the present invention. For ease of reference, common reference numerals are used throughout the figures when referring to the same or similar features common to the figures.
[0027] The present invention is based on the reflectance of window glass within the infrared (IR) spectrum, at least in the range of 8-15 μm (i.e., long wavelength IR), which corresponds to object temperatures of -80°C to +89°C.
[0028] According to Wikipedia, this region is the "thermal imaging" region, where sensors can obtain completely passive images of objects that are only slightly warmer than room temperature, such as the human body, based only on thermal radiation and without the need for illumination such as the sun, moon, or infrared illuminators. This region is also called "thermal infrared". ["Infrared", Wikipedia.org(online),<URL:https: / / en.wikipedia.org / wiki / Infrared> [see references]
[0029] Figure 7 shows a graph of the reflectance of various types of glass. Raul J Martin-Palma, et.al. “Silver-based low-emissivity coatings for architectural windows: Optical and structural properties”. ResearchGate.net[online], May, 1998,[<URL:https: / / www.researchgate.net / publication / > Retrieved March 16, 2023 from . As shown in the figure, typical glass has a reflectance of over 20% for wavelengths in the band around 8.5 to 11.5 μm. In other words, objects that emit wavelengths in this range can be detected by very common thermal imaging devices and cameras. For example, the SeekThermal CompactPRO TM The low-cost thermal sensor used in is sufficient to detect objects with temperatures from -40°C to +330°C, which covers most of the long-wavelength IR (LWIR) band. The thermal sensor is, for example, a 206x156 thermal sensor with a 36° wide-field lens.
[0030] Any object with a temperature between approximately -20°C and +65°C can be a source of IR radiation in the 8.5 to 11.5 μm band. [sspickle “Blackbody wavelength from temperature”, May, 2017, [URL:<https: / / www.vcalc.com / wiki / sspickle / > Retrieved March 16, 2023 from]. In view of the above, it is an object of the present invention to provide a glass wall detection system and method, in which an autonomous device is equipped with an IR camera and an IR emitter, and obtains directional information regarding the IR thermal signature emitted by the IR emitter and reflected by the glass wall, and further calculates the distance from the autonomous device to the glass wall.
[0031] The IR camera can be any device capable of acquiring a thermal image of an object, which may (or may not) include an image processor that searches for the signature of an IR emitter and transmits information about the detected glass wall (direction and / or distance) to the robot, or transmits raw (or pre-processed) image data for eventual processing by an external device (e.g., the robot's computer).
[0032] An IR emitter can be any device whose temperature differs from that of its surroundings, and that difference in temperature can be detected by an IR camera. A hot IR source can be as simple as a resistor that converts electrical power into heat.
[0033] It is understood that the system of the present invention requires a processing device (built-in or dedicated), such as a computing device, to process the data captured from the IR camera. Based on an understanding of the laws of reflection, the system uses triangulation to calculate the distance between the device and the reflective surface. It involves calculating the distance between the camera and the reflective surface using the obtained angle of incidence and the distance between the imager and the emitter.
[0034] 1 illustrates a method for detecting a glass wall for an autonomous system, such as a robot, according to an embodiment of the present invention. The method includes disposing an infrared (IR) imager and an IR emitter in the autonomous system in step 110, emitting an IR heat signature from the autonomous system via the IR emitter in step 120, detecting the IR heat signature reflected from a reflective surface within a field of view (FOV) of the IR imager in step 130, and determining a distance and angle of the autonomous system relative to the reflective surface based on the IR heat signature in step 140.
[0035] In step 110, the autonomous system is equipped with one or more IR imagers and scans the area of interest to determine whether a glass panel / wall is present within the FOV of the imager. In step 120, one or more devices may be provided to emit one or more IR heat signatures from the autonomous system. The IR heat signature is preferably uniquely identifiable as being from the autonomous system. The IR heat signature can be reflected by any reflective surface such as a glass wall. The advantage of utilizing heat signatures in the IR spectrum is that the heat signature is reflected even if the glass is transparent glass, and therefore, the disclosed method allows for the detection of transparent glass panels / walls for the autonomous system / device. In step 130, if the IR heat signature is reflected by any glass panel / wall and the reflected IR heat signature falls within the FOV of the IR imager, the autonomous system recognizes that the glass panel / wall is within the FOV of the IR imager. Thereby, in step 140, the distance and angle / direction from the IR imaging device to the glass panel / wall can be calculated accordingly.
[0036] FIG. 2A illustrates a side view of an autonomous system 200 according to an embodiment of the present invention facing a glass wall 210. FIG. 2 illustrates a side view of the autonomous system 200 positioned such that its front side faces the glass wall 210, allowing the distance between the front side of the autonomous system 200 (here, IR sensor S) and the glass wall 210 to be determined. In this embodiment, the IR sensor S is mounted below the LWIR emitter E to form a vertical axis at the front side of the autonomous system 200, such that the front side (ES) of the autonomous system 200 is parallel to the glass wall 210. The IR sensor S has a field of view (FOV) for detecting / capturing IR heat signatures. It should be noted that the position and orientation of the IR emitter E relative to the IR sensor S is configured such that, in operation, the IR heat signature emitted by the IR emitter E is not directly detected by the IR sensor S, i.e., outside the FOV. L represents the distance between the LWIR sensor S and the LWIR emitter E, and R represents the center point between S and E on the vertical axis. The vertical glass wall 210 is marked with E'R' and S', corresponding to E, R and S, as reference points for illustration. A thermal signature is emitted from an IR emitter E, reflected off the glass wall 210 at R' and reaches the IR sensor S. L and α are known. Thus, the distance between the autonomous system 200 and the wall can be determined as follows:
[0037]
number
[0038] It is obvious to those skilled in the art that the angle α can be easily obtained through the specifications of the IR sensor S (eg, the resolution and FOV of the image array), so further details will be omitted here.
[0039] Based on the description in FIG. 2A and the description in FIG. 2B below, the IR sensor S should be a two-dimensional (2D) thermal imaging sensor for determining angles α and β, where angle β is needed to calculate the direction to the wall and angle α is needed to calculate the distance to the wall.
[0040] In this embodiment, the IR sensor S is an array sensor, which allows the system to further detect the position (x,y) of the thermal signature (image) on the array sensor. With the understanding that the thermal signature may occupy multiple pixels on the IR sensor S, the position (x,y) may simply be the reference position of the thermal signature, or may be the position of one signal pixel of the thermal signature. Thus, for each x and y coordinate (of the array sensor), the angle β x and angle α y can be determined appropriately.
[0041] FIG. 2B further shows a plan view of FIG. 2A. In this plan view, points E, R, and S are shown on the same point. As shown by the reference dotted line ER′, the only direction in which the signal from the IR emitter E can be detected by the IR sensor S is when the line ER′ is perpendicular to the glass wall 210, i.e., ER′ is the shortest distance from the device to the glass wall 210, and the direction is determined as angle β. Similarly, it is also obvious to those skilled in the art that angle β can also be obtained based on the sensor specifications.
[0042] In other embodiments of the present invention, the autonomous system 200 may include two or more IR emitters E. In further embodiments, the multiple IR emitters E may emit different thermal signal signatures, which may help to obtain more information or features of the glass wall, such as the edges of the glass wall. With multiple IR emitters E, the IR sensor S can receive multiple responses when the thermal signatures are reflected by the glass wall. The presence / absence of the thermal signature that can be detected by the IR sensor from the glass wall according to the location / position of the IR emitters E relative to the IR sensor S in the autonomous system can be used as a reference to determine the characteristics and contours of the glass wall.
[0043] FIG. 3 is a plan view of an autonomous system 300 according to an embodiment of the present invention, with a sensor assembly detecting a glass wall 310. In this view, the glass wall 310 is tilted with respect to the direction in which the autonomous system 300 is facing. In this embodiment, the sensor assembly includes an IR sensor S and four IR emitters E1, E2, E3, and E4, with two emitters arranged in a line on each side of the IR sensor S. The glass wall 310 has an opening or window W, shown in dotted lines, which is a void or non-reflective surface. The opening W can be a door opening. As further shown, the autonomous system 300 is positioned in front of the glass wall 310. When the sensor assembly is operating, the IR emitters each emit a heat signature onto the glass wall 310, and the IR sensor S scans the glass wall 310 for the heat signature. As shown in Figure 3, the heat signatures from E1, E2 and E4 are reflected off the glass panel 310 (shown as E1', E2' and E4' in Figure 3) and detected by the IR sensor S. E3 is not detected because it falls on a non-reflective surface, and therefore is considered to have no glass surface at that point.
[0044] It is understood by those skilled in the art that the autonomous system 300 typically includes some detection means for scanning for non-reflective obstacles. If the other detection means also do not detect any obstacles in the opening W, it can be determined that it is a void. More specifically, the present embodiment can estimate that the opening W is between E2' and E4'.
[0045] In this embodiment, the positions of the IR sensors S are arranged in a line and parallel to the ground. It will be appreciated that by increasing the number of emitters E and / or sensors S, the accuracy of the above estimation can be improved.
[0046] 3, as the autonomous system 300 rotates (pans) about its current position in front of the glass wall 310, the IR sensor S detects the thermal signals E1', E2', E3', and E4' when they are emitted onto a reflective surface (i.e., the glass wall 310) and detects their absence when they strike the opening W. Such continuous scanning and detection can further determine the shape, including the edges, of the glass wall. It should be noted that the emitter E and the sensor S as required by the present invention are very low-cost components and are very cost-effective to implement in any autonomous system according to an embodiment of the present invention. Operationally, as the sensor assembly pans over an area on the autonomous system, the captured images can be collectively used to map the area by determining the presence and absence of thermal signatures. The accuracy of the mapping results can be affected by the resolution and / or frame rate of the IR sensor S.
[0047] Combining concepts from computational geometry and computer vision, simultaneous localization and mapping can be achieved by an autonomous system. Utilizing embodiments of the present invention, an autonomous system can build a map of its surroundings by analyzing the distances to objects in its field of view, and use the map to navigate and avoid obstacles, including glass walls / panels.
[0048] There are other systems and methods known in the art and therefore, for the sake of brevity, the details of these are omitted here.
[0049] It should also be noted that the distance between the emitter E and the sensor S can vary depending on the design of the autonomous system 300.
[0050] In one embodiment, the emitter E can be any thermal element that can be controlled to emit a thermal signature. These emitters E preferably emit a thermal signature that can be used as a distinguisher of the autonomous system from the surroundings. The thermal element can be any controllable thermal device or component capable of emitting heat or cold in the range of -20°C to +65°C. It can be a resistor or a Peltier element. The advantage of using a controllable device by the system is that the device can be adjusted and controlled to emit a thermal signature that is distinguishable from the ambient temperature. This thermal signature can be distinguished through different temperatures. In other embodiments, the thermal signature can be a pattern of some shape that is visually distinguishable by the sensor S. For example, the emitter E may comprise an array of resistors or Peltier elements arranged to form a pattern upon actuation. Figures 4A, 4B and 4C illustrate three thermal signature patterns. These are not limiting, but are merely for illustrative purposes.
[0051] As described herein, resistor array refers to a plurality of resistors arranged to emit a desired thermal signature as required by the present invention. These resistor arrays or resistors can be an inexpensive and scalable solution to manufacture an emitter that emits a desired thermal signature. It can be packaged as one device or it can be a plurality of independent resistors arranged on an autonomous system. Each resistor unit can form one (thermal) pixel of the thermal signature, and the number of resistor units included in the resistor array is highly customizable depending on the application. This allows the thermal signature to be controlled not only by heat or its wavelength range, but also to be presented as a unique pattern to the autonomous system. These resistors are small in size and low cost. For the avoidance of doubt, resistor arrays, individual resistor units, etc. are not limiting of the present invention and should be understood as possible means of emitting a thermal signature for detecting glass walls for an autonomous system.
[0052] 5 is a plan view of a sensor assembly of an autonomous system 500 according to an embodiment of the present invention. The autonomous system 500 includes an IR sensor S and two IR emitters E1 and E2. This configuration allows a calculation to determine the distance D from the autonomous system 500 to a glass wall 510 as shown. The two emitters E and the sensor S are placed in a linear arrangement, with the two emitters E placed a distance L on each side of the sensor S. If the sensor assembly is oriented parallel to the glass wall, e.g., L1=L2 and α1=α2, then the distance D from the autonomous system 500 to the glass wall 510 can be determined.
[0053] In other embodiments, the sensor assembly of the autonomous system 500 may include two sensors S and one emitter E arranged similarly to the previously described embodiments and may detect the distance between the autonomous system 500 and the glass wall 510.
[0054] 6 shows another sensor assembly 600 according to another embodiment of the present invention. The sensor assembly 600 comprises two sensors S1, S2 and nine emitters E. The sensors S1 and S2 are arranged side by side and form a reference axis (not shown). The emitters E0 1 , E0 2 , E0 3 , E0 4 and E0 5 are all positioned alongside S1 and S2 on the same reference axis, in the following order:
[0055] E0 1 -S1-E0 2 -E0 3 -E0 4 -S2-E0 5
[0056] Emitter E0 n The arrangement is configured to scan and detect the edges and contours of the glass wall as described in the above embodiments.
[0057] Emitter E1 1 and E1 2 are located above and below the sensor S1, respectively. 1 and E2 2 are located above and below S2, respectively. 1 and E1 2 is provided to measure the distance and direction to S1, and emitter E2 1 , E2 2 is provided to measure the distance and direction to S2.
[0058] According to the present invention, there is no limit to the number of sensors S and / or emitters E required in a sensor assembly. Those skilled in the art will appreciate that the number required will depend on the application. The emitters can be of any suitable size, shape and temperature emission required for the application. The thermal signature may further be operatively altered to provide a unique thermal signature for each emitter to improve its applicability.
[0059] Of the wide range of infrared spectrum that can be detected through any thermal imaging camera, it is preferred that long wavelength infrared, also known as far infrared, is used. In other words, in other embodiments of the present invention, the system may employ long wavelength IR (LWIR) sensors and long wavelength infrared (LWIR) emitters to detect the presence of glass walls. Also, as explained above, the autonomous system may be equipped with two or more LWIR sensors and / or two or more LWIR emitters. Multiple sensors and / or emitters provide adaptability and accuracy for detecting glass walls.
[0060] The present invention provides a solution for an autonomous system to detect glass walls by using a thermal imaging device that detects a reflected image of the thermal signature from the autonomous system.
[0061] While particular embodiments have been described and illustrated, it will be understood that many variations, modifications, variations and combinations thereof can be made to the invention without departing from the scope of the invention.
Claims
1. A method for detecting a glass wall having a reflective surface for an autonomous system, comprising: disposing at least one infrared (IR) imager having a field of view (FOV) and at least one IR emitter on the autonomous system; emitting an IR thermal signature from the autonomous system with an IR thermal emitter, the IR thermal signature being located outside the FOV; detecting the presence of the IR thermal signature reflected off the reflective surface; determining a distance and angle of the autonomous system from the reflective surface based on the reflected IR thermal signature within the FOV; Thereby, the reflective surface is identified as the glass wall.
2. each IR emitter includes a plurality of resistors arranged to emit the thermal signature; The method of claim 1.
3. the IR emitter includes an array of resistors arranged to form the thermal signature; The method of claim 1.
4. the autonomous system determines the distance and angle based on triangulation. The method of claim 1.
5. the IR thermal emitter emits the thermal signature in the long wavelength IR band; The method of claim 1.
6. A sensor assembly of an autonomous system adapted to detect a glass wall, comprising: at least one infrared (IR) thermal sensor having a field of view (FOV); at least one IR emitter that operatively emits a thermal signature; Thus, in operation, when an IR thermal signature is reflected off a reflective surface and the reflected IR thermal signature is captured within the FOV, the sensor assembly determines the distance and angle of the IR sensor from the reflective surface, and the reflective surface is identified as a glass wall.
7. each of the IR emitters includes a resistor array, each resistor unit of the resistor array forming a pixel of the thermal signature; The sensor assembly of claim 6.
8. the thermal signature is in the long wavelength IR spectrum; The sensor assembly of claim 6.
9. The distance and the angle are determined through triangulation. The sensor assembly of claim 6.
10. An autonomous system comprising a sensor assembly according to any one of claims 6 to 9.
Citation Information
Cited By
Panel for forming a floor covering
US12359446B2
Floor panel
US12442196B2
