Systems and methods for enabling navigation in environments containing dynamic objects

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to mobile robotic systems, and more particularly to systems and methods for improved autonomous navigation of indoor mobile robotic systems within an operating environment.BACKGROUND

[0002] Indoor mobile industrial robots are programmable machines configured to autonomously navigate within an indoor industrial setting, often while performing one or more tasks. One recognized deficiency with autonomous mobile robotic systems is the general inability of such machines to recognize their surroundings and adequately react to changes within a given operating environment. There are two key functions involved in autonomously driving a mobile industrial robot; one relates to navigation within the operating environment, and the other relates to obstacle avoidance. Regarding navigation, one particular challenge is seen when features previously relied on for navigation within the operating environment are no longer there or have been moved. In general, indoor mobile industrial robot navigation systems struggle with an inability to cope with objects that are not of a static nature. Many algorithms for Simultaneous Localization and Mapping (SLAM) initially assume that what is detected is static and may be expected to be at the same position next time that position is visited.

[0003] Relevant technology may be seen in: EP3508937, US2016 / 302639, US2019 / 369640, US2019 / 265721 and WO2020 / 098456.

[0004] Embodiments of the present disclosure address this concern.SUMMARY OF THE DISCLOSURE

[0005] In a first aspect, the invention relates to a navigation system according to claim 1.

[0006] In this context, a navigation system is a system capable of determining a position of the robot unit and / or determining in which direction to move the robot unit.

[0007] The robot unit is for navigating in a scene or venue, which may be any type of environment, such a room, facility, storage room, production space, terminal, warehouse, store, waiting room, terminal, sports arena, indoor, outdoor, in the air or underwater or the like.

[0008] A robot unit often is a self-propelled unit which needs no human operator to navigate the scene or venue. The robot unit may have means for moving the robot unit, such as wheels, tracks, or the like. Robot units may be configured to move on land, under water, on the water, and / or in the air. The unit may be driven by one or more motors, engines or the like, such as an electrical motor. The robot unit may be powered by e.g. a cable, or the unit may comprise its own power source, such as a battery, fuel container, solar cells, fuel cell or the like.

[0009] The robot unit comprises one or more remote sensors for determining features, often in a vicinity of the unit. Features may be static or dynamic, where a static feature is a feature which the unit may assume will not move or change position within a predetermined period of time, such as one second, one minute, one hour, one day, one week or the like. Static features are fixed structures, such as walls, pillars, racks, fences, building structures, heavy furniture or elements, storage elements, or the like.

[0010] Dynamic features, on the other hand, may be features which may be moving, such as in relation to the earth or the ground, or which may be expected to move or change position / rotation within one second, one minute, one hour, one day, one week or the like. A dynamic feature may be a feature which cannot be expected to be in the same position the next time the robot unit reverts to the present position of the feature. Dynamic features may be persons / animals, luggage, packets, pallets, bicycles, cars, vehicles, trailers, campers, but also lightweight furniture and the like, like chairs, tables, plants, and the like.

[0011] The time frame may vary, so that in some embodiments, even lightweight furniture or the like may be seen as static structures, such as if the same position is re-visited within minutes, whereas even more heavy furniture, such as sofas and tables, may be seen as dynamic features if the position is revisited only once a month. Clearly, "static" need not be completely and perpetually immovable. Further below, methods of handling situations with partially static features are described.

[0012] The remote sensor(s), which are capable of sensing features remote from the sensors, may be based on any technology, such as stereo vision, lidar, radar, sonar, or the like. Any combination of such technology may be used. Preferably, the sensor(s) is / are configured to determine a position of a feature vis-à-vis the robot. Such technology may also be termed collision detection technology, as it is often used for assisting a robot in not colliding with obstacles in its vicinity during navigation in a scene or venue.

[0013] The system comprises a controller which may be any type of controller, software controlled or hardwired. The controller may be a processor, ASIC, DSP, or the like. The operation of the controller may be divided so that different portions of the functionality are performed on different hardware or even different types of hardware, which then usually would be in communication. Part of the functionality may be performed remotely, such as on a server, such as in a cloud solution.

[0014] The controller is configured to receive an output from the sensor(s). This output may be received via wires and / or wirelessly. The output may have any desired form. The output may be more or less raw sensor output, such as images from a stereo camera set-up, the received signals output from a lidar, or the like. Alternatively, the sensor(s) may comprise their own processor or the like for converting sensor data into more processed data, such as data representing positions, often relative to the robot or in a predetermined coordinate system, which may more easily be interpreted by the controller.

[0015] In this respect, a feature may be a physical object, such as a piece of furniture, a wall, structural element, such as a pillar, a piece of luggage, a storage rack or the like. A feature, however, may also be a characteristic of a physical object, such as an element attached to the physical object. A feature thus may be a surface color, surface shape, surface reflectivity, physical appearance, emission characteristics of the physical object or the element attached thereto, such as if having a fluorescent surface.

[0016] The features may be determined in any manner, such as from a presence thereof. A feature may be any element detected by the sensor(s). Often combinations of sensors are used; often called sensor fusion. The sensor output may be filtered or otherwise processed so that features determined fulfill certain requirements, such as a size thereof, a dimension thereof or the like. Clearly, also other characteristics may be used, such as a reflectivity thereof, a shape thereof, or even a color thereof.

[0017] Between the determined feature(s), one or more static features are recognized. A static feature is recognized by a fluorescent or reflective element applied thereto.

[0018] From the static feature(s), the position of the robot vis-à-vis this feature or these features may be determined. Often, position determination based on dynamic features is problematic, especially if the position desired is a position in a fixed coordinate system, such as a coordinate system of the scene or venue.

[0019] Clearly, a collision avoidance sensing scheme would be useful irrespective of whether the features detected are static or dynamic, but when determining a position of the robot in the scene or venue, relying on dynamic features could be problematic.

[0020] Determining the position based on the static features acts to render the reliability of the position determination high.

[0021] In one embodiment, the controller is further configured to determine, from the output, one or more dynamic objects and, in, the determining step, the position only from the static feature(s). As mentioned, the dynamic features may be relevant for collision avoidance, but the position is determined, in this embodiment, leaving out the dynamic features.

[0022] The system further comprises a storage comprising information representing a position of each of a number of the static features. In one situation, the information may be a map or may represent a map of the scene or venue and comprising therein the positions of the static features. Thus, the controller isthen configured to determine a position of the robot unit in the scene or venue based on the information of the storage and the determined position of the robot unit vis-à-vis the recognized static feature(s).

[0023] A map may be represented in any desired manner, such as by a plurality of positions at which static features are present or expected. Different manners of representing such information are provided below.

[0024] Clearly, in the above situations and in the below situations, embodiments and aspects, the positions may be 2D or 3D positions. A position may be two-dimensional such as in the situation of a robot driving on a plane floor. Robots may, however, move in three dimensions, and a position may be three-dimensional.

[0025] Even in situations where the robot moves only or predominantly in two dimensions, information may be taken into account also in a third dimension, such as if a feature is recognized which has a wider outline at one height and a more narrow outline at another height. In that situation, the extent of the robot at these heights may be taken into account when determining e.g. a relative distance between the feature and the robot.

[0026] When the features determined or recognized are not in complete coherence with the features of the map, robots may have problems. Thus, using only recognized features when searching in the map for a correspondence, to identify a position of the navigation system in the map, dynamic features are ruled out so that a more probable or correct position is determined. In the search in the map for correspondence with features comprising also dynamic features, an erroneous position might be determined. Clearly, "static" and "dynamic" need not be a binary determination. Further below, variations thereof are described as is how to incorporate this into the method.

[0027] The controller may be configured to update the storage with information representing a recognized static feature. Thus, if the recognized feature is not in the storage, it may be added to the information so as to be represented in the future.

[0028] In one embodiment, the same map or information is shared by multiple robots, some of which are not configured to update the information or map. Thus, from the updating performed by the present system or robot, other robots may now receive an updated map and may thus navigate using the updated map.

[0029] Another aspect of the invention relates to a method according to claim 4.

[0030] Naturally, all aspects, embodiments, situations and the like may be interchanged. The above description is equally relevant to the below description.

[0031] The robot / navigation system, features, static nature, dynamic nature, sensors, information output and the like may be as described above and below.

[0032] The method comprises the sensor(s) outputting the information representing surroundings of the robot. Often, the sensors are configured to determine any features, obstacles or the like in the surroundings of the robot. In addition to determining whether features are static or dynamic, the sensor output may be used for ensuring that the robot does not collide with a feature or obstacle.

[0033] The surroundings may comprise a surface on which the robot moves, or a medium through which the robot moves, and / or features detectable by / from the robot. The sensors may be based on vision or emission / reflection of radiation of any type. Thus, the surroundings detected may be visible from the robot, so that line of sight is desired to at least the nearest features.

[0034] From the information output from the sensor(s), a plurality of features is determined. A feature may be a separate, coherent or interconnected, feature in the information. Multiple features may in fact be a single feature, if the full extent of that feature was known. A U-shaped storage rack, for example, may initially be seen as two different features if the U is viewed from above and it is not possible to detect or sense the bottom thereof. If that feature is recognized or determined to be static, so would the portions thereof, so that it may not be problematic that not all of a feature is sensed at the same time.

[0035] As described, one or more static features are recognized, and the position is determined vis-à-vis these features.

[0036] In one embodiment, the method further comprises determining, from the information, one or more dynamic features, and wherein the determining step comprises determining the position only from the static feature(s). Thus, when dynamic features are detected or determined, these are not, in this embodiment, taken into account when determining the position.

[0037] In one embodiment, the method further comprises providing a storage comprising information representing a position of each of a number of the static features. As mentioned above, this information may represent or be in the form of a map of the scene or venue or a portion thereof, where the positions of the static features are represented. A robot or navigation system may use this map for navigating in the scene or venue. The robot may determine its position in the scene or venue or map by comparing its relative position(s) to the recognized static features, the positions of which are represented in the information.

[0038] This information may be provided in any manner. In one manner, the information may be generated or updated with or by information as to the position of a static feature, such as if the position of the robot is known. Then, the information of a static feature may be added or updated in the information, as is described below.

[0039] The invention comprises the step of determining a position of the robot unit in the scene or venue based on the information of the storage and the determined position of the robot unit vis-à-vis the recognized static feature(s). This may be obtained using knowledge of the positions of the static features in the scene or venue. This may be found from the above information and / or the map.

[0040] One embodiment further comprises the step of updating the storage with information representing a recognized static feature. In this manner, newly recognized static features may be added to the information, or information may be updated if a static feature is no longer there or has moved. As described above, static features may not be required to be at the same position all the time. A static feature may be moved and the information updated.

[0041] In one embodiment, the controller is configured to recognize a static feature as a determined feature which: emits or reflects an amount of radiation exceeding a predetermined minimum intensity or emits or reflects radiation at a predetermined wavelength.

[0042] In addition or alternatively, the feature may emit or reflect radiation in a predetermined spatial or temporal pattern.

[0043] A static feature is determined due to its emission or reflection. Static features may be marked, tagged or the like with elements or surfaces which have the desired characteristics.

[0044] Reflection may be determined from the intensity received when the intensity output is known. Instead of reflection, scattering may be determined.

[0045] A surface may be fluorescent or may comprise a fluorescent material or compound which will absorb radiation and output radiation at one or more predetermined wavelengths. Such material / compound may be selected so that it outputs radiation in a wavelength which is not present or only present in a rather low intensity, so that detection of the wavelength points to a static feature being present. Alternatively, a filtering or correction may be performed if the wavelength in question is also emitted by other sources, such as lamps, the sun or the like, so that any contribution from such sources is subtracted to identify the static features.

[0046] Clearly, the wavelength may be a wavelength interval around or comprising the wavelength in question. Optical filters of most types are able to only define a wavelength interval. The width of the wavelength interval may be decided based on e.g. the wavelength of other radiation in the vicinity of the static feature(s).

[0047] A static feature may alternatively be recognized based on the size or amount / intensity of its emission or reflection. A static feature may have a large emission / reflection or may be provided with an element having such emission / reflection, so that it may be recognized based on the amount of radiation, such as within a particular wavelength, reflected and then usually detected by the sensor(s). In a simple embodiment, the static features may be provided with reflective elements or surfaces. Then, the initial determination that a feature is static may be made by e.g. an operator simply providing that surface, such as in the form of a fluorescent or reflective surface or element, such as a sticker or mirror. Subsequently, the feature may be recognized based on the emission / reflection.

[0048] A suitable sensor for identifying reflective or fluorescent elements or surfaces may be a sensor emitting radiation of a predetermined wavelength. Then, reflection may be determined at the same wavelength, where fluorescence may be determined at another wavelength, as the fluorescent material will convert radiation at one wavelength to radiation at another wavelength.

[0049] Naturally, a feature need only have a portion thereof having the sought-for characteristic, such as the reflection or emission. Once a sufficient emission / reflection / scattering has been identified, the complete feature preferably is recognized as a static feature. Thus, a complete extent of the feature may be determined, or as much of the feature as can be sensed by the sensors.

[0050] Alternatively, a color may be determined. All features with a predetermined color may then be assumed to be static and features of another color may be assumed to be dynamic.

[0051] In one embodiment, the step of recognizing a static feature comprises determining, as static, features which: emit or reflect an amount of radiation exceeding a predetermined minimum intensity, or emit or reflect radiation at a predetermined wavelength.

[0052] As mentioned above, the wavelength may be selected in a number of manners, for example as a wavelength which is not present or not present to any significant degree in the surroundings. A radiation source may be provided for directing radiation on to the features, such as with a predetermined radiation intensity so that a reflection percentage may be determined. The distance to the feature may be taken into account in this respect.

[0053] A radiation emitter may also be useful for e.g. causing fluorescence emission if desired.

[0054] Optical filters, lenses and the like may be used as known in the art for setting up the detection and any irradiation.

[0055] All features with a predetermined color may then be assumed to be static or a particular type of object, such as a storage rack. Then, if the predetermined color is red, red objects are taken as static, whereby e.g. storage racks may be red. Alternatively, lightweight furniture may be selected with a particular color so that they may easily be determined as dynamic.

[0056] Computer vision can be applied in the discipline of robotic mapping. The goal for an autonomous robot is to be able to construct a map / floor plan of an operating environment, and to localize itself for navigation within the operating environment. Autonomous navigation often utilizes simultaneous localization and mapping (SLAM), which involves constructing or updating a map of an unknown environment while simultaneously keeping track of the machine's location within the environment. In that regard, the machine can be guided or influenced by a navigation module based, at least in part, upon data related to distance measurements acquired by an optical assembly (e.g., a LiDAR, 2-D camera, 3-D / stereo camera, etc.).

[0057] In some embodiments, localization can be based upon a creation of a multiplicity of data points, which collectively form a representation of the operating environment. Each discrete data point can represent a node in a data structure and can include data related to an estimation of the current position of the machine. In some embodiments, the machine can utilize the data points to formulate a respective version of a map of the operating environment. Representation of such maps may, for example, be in the form of a grid delineating boundaries or otherwise describing features of the operating environment, such as an occupancy grid or a Truncated Signed Distance Field (TSDF) map.

[0058] As the machine traverses the operating environment, motion of the machine can be measured with an odometer, inertial measurement unit, GPS or other instrument. Each respective data point can then be updated with actual information based upon the known rotation of wheels. With respect to traversing the operating environment, additional distance sensor readings (e.g., acquired by one or more optical assemblies) can be used to update the map, as well as to increment or decrement a data point weight, relevance score, or some other factor representing an estimated likelihood of the detected object to which the data point belongs as being either of a static feature or a dynamic feature.

[0059] Maps can be two-dimensional or three-dimensional. Two-dimensional maps can be generated based on the scene or venue at a predetermined height (e.g., 10-30 cm from a floor or bottom, 20cm from a floor or bottom, etc.). Alternatively, a two-dimensional map can be formed by projecting the obstacles to a predetermined, often horizontal, plane. Three-dimensional maps can take into account the different shapes of obstacles and the robot, as a robot may reach or impact on an obstacle at one height while not at another height. Thus, if the map is generated at "another" height, the robot could impact on the obstacle even when a two-dimensional map would indicate that there is still space between the obstacle and the robot.

[0060] In some embodiments, a map can represent an illustration of the boundaries of all obstacles, features and / or landmarks within a given scene / venue projected on to a horizontal surface. Maps often are illustrated in a scaled-down version if displayed, for example to an operator. Different types of maps exist, such as occupancy grids and truncated signed distance fields (TSDFs). In an occupancy grid, the space is divided into a grid of map cells, where each cell contains an estimate of the probability that that cell is occupied by a landmark or feature. Thus, the map may be represented by a number of map cells each representing a probability of there being a landmark or feature within that cell. Often this probability is represented as a number of between 0 and 1.

[0061] A TSDF is like an occupancy grid, but here each cell contains the distance to the nearest landmark or feature, if there is a landmark or feature within some truncation distance. This map may be represented with distance measures for each cell or more simply as a color indicating distances (e.g. green indicating a large distance to red indicating no distance).

[0062] Other types of maps are also contemplated. One embodiment of the present disclosure provides an indoor mobile industrial robot system configured to classify a detected object within an operating environment as being either one of a static feature or a dynamic feature. Classification of a detected object can occur locally (e.g., via an onboard computer) or remotely (e.g., in the cloud). The indoor mobile industrial robot system can include a mobile robotic platform, a LiDAR unit, positional module, and processor. The mobile robotic platform can be configured to self-navigate within an operating environment. The LiDAR unit can be operably coupled to the mobile robotic platform and configured to emit light energy and receive reflected light energy from the detected object. The positional module can be configured to account for at least one of a position and / or rotation angle of the LiDAR unit with respect to the mobile robotic platform. The processor can be configured to translate the received reflected light energy and information from the positional module into a set of data points representing the detected object having at least one of Cartesian and / or polar coordinates, and an intensity, wherein if any discrete data point within the set of data points representing the detected object has an intensity at or above a defined threshold the entire set of data points is classified representing a static feature, otherwise such set of data points is classified as representing a dynamic feature.

[0063] In one embodiment, the defined threshold can be a receipt of at least 70% of the emitted light energy from the LiDAR unit. In one embodiment, the LiDAR unit can be configured to at least one of emit a continuous beacon of light energy or emit discrete pulses of light energy. In one embodiment, the LiDAR unit can be configured to rotate with respect to the mobile robotic platform.

[0064] In one embodiment, the system can further include a navigation module, including one or more sensors configured to sense a positional movement of the robotic platform within the operating environment (e.g., wheel encoder, wheel direction sensor, etc.) configured to account for navigational movements of the mobile robotic platform within the operating environment. In one embodiment, the system can further include a memory configured to store the set of data points. In one embodiment, the processor can be configured to utilize the set of data points to provide an estimation of a shape of the detected object. In one embodiment, the processor can further be configured to produce a map of the operating environment, including relative positions of any detected static features and / or dynamic features within the operating environment, as well as the position of the robotic platform within the operating environment. In one embodiment, the system can further include one or more proximity sensors configured to detect a proximity of the mobile robotic platform to obstacles within the operating environment.

[0065] Another embodiment of the present disclosure provides an active light-based sensor device configured to classify a detected object within an operating environment as being either one of a static feature or a dynamic feature. The device can include a LiDAR unit, positional module, and processor. The LiDAR unit can be configured to emit light energy and to receive reflected light energy from the detected object. The positional module can be configured to account for at least one of a position and / or rotation angle of the LiDAR unit within the operating environment. The processor can be configured to translate the received reflected light energy and information from the positional module into a set of data points representing the detected object having at least one of Cartesian and / or polar coordinates, and an intensity, wherein if any discrete data point within the set of data points representing the detected object has an intensity at or above a defined threshold the entire set of data points is classified representing a static feature, otherwise such set of data points is classified as representing a dynamic feature.

[0066] Another technology of the present disclosure provides a method of classifying a detected object within an operating environment as being either one of a static feature or a dynamic feature, including: identifying one or more static feature within an operating environment; applying a reflective material to the identified static feature; emitting a light energy from a light-based sensor device towards the static feature; receiving reflected light energy from the static feature; and translating the received reflected light energy into a set of data points representing a detected object having at least one of Cartesian and / or polar coordinates, and an intensity, wherein if any discrete data point within the set of data points representing the detected object has an intensity at or above a defined threshold indicating the presence of a reflective material the entire set of data points is classified representing a static feature, otherwise such set of data points is classified as representing a dynamic feature. In some embodiments, the reflective material can be at least one of a sticker, tape, emblem, paint, or the like configured to reflect at least 50% of the light energy directed at it.

[0067] In one embodiment, the optical assembly can include a left camera and a right camera separated from one another by a fixed distance. In one embodiment, the distances to detected objects within the operating environment can be determined by way of binocular disparity. In one embodiment, the optical assembly can be one or more of a LiDAR unit, 2-D optical assembly and / or 3-D optical assembly with range finding capabilities. In one embodiment, the optical assembly is configured to rotate with respect to the mobile robotic platform. In one embodiment, the navigation module can be configured to account for navigational movements of the mobile robotic platform within the operating environment.

[0068] In one technology, the processor can be further configured to designate individual cells among the plurality of cells of one or more grids as one of being occupied by a detected object, being unoccupied, or not being within the line of sight of the optical assembly. In one embodiment, the individual cells designated as being occupied by a detected object are assigned a first initial probability, and individual cells designated as being unoccupied are assigned second initial probability, wherein the second initial probability is lower than the first initial probability, thereby indicating that an object later detected in the operating environment corresponding to the individual cells designated as being unoccupied is more likely to be a dynamic feature and / or an estimated distance to a detected feature. In one embodiment, the processor can be configured to assign initial probabilities to individual cells previously being designated as not being within the line of sight of the optical assembly upon receiving images of a portion of the operating environment corresponding to the individual cells previously being designated as not being within the line of sight of the camera. In one embodiment, a first grid captures featured deemed static, while a second grid captures features deemed dynamic.

[0069] The summary above is not intended to describe each illustrated embodiment or every implementation of the present disclosure. The figures and the detailed description that follow more particularly exemplify these embodiments.BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The disclosure can be more completely understood in consideration of the following detailed description of various embodiments of the disclosure, in connection with the accompanying drawings, in which: FIG. 1 is a perspective view depicting a mobile industrial robot system, in accordance with an embodiment of the disclosure. FIG. 2 is a system architecture diagram depicting a light detection and ranging (LiDAR) unit, in accordance with an embodiment of the disclosure. FIG. 3 depicts use of the unit depicted in FIG. 2 to detect and classify objects within an operating environment, in accordance with an embodiment of the disclosure. FIG. 4A is a system architecture diagram depicting a 3-D camera unit, in accordance with an embodiment of the disclosure. FIG. 4B depicts left and right images taken by the 3-D camera unit of FIG. 4A, in accordance with an embodiment of the disclosure. FIG. 5A-C depict the use of the 3-D camera of FIG. 4A to detect and classify objects within an operating environment, in accordance with an embodiment of the disclosure. FIG. 6 is an overall review of a scene or venue in which the robot system may navigate.

[0071] While embodiments of the disclosure are amenable to various modifications and alternative forms, specifics thereof shown by way of example in the drawings will be described in detail. It should be understood, however, that the intention is not to limit the disclosure to the particular embodiments described.DETAILED DESCRIPTION

[0072] Referring to figure 1, a perspective view of an mobile industrial robot system 10 having one or more optical assemblies 14 and 16 (e.g., a LiDAR, 2-D camera, 3-D / stereo camera, etc.) configured to detect and classify objects within an operating environment as either being a "static feature" or a "dynamic feature," is depicted in accordance with an embodiment of the disclosure. Figure 6 illustrates a scene or venue having therein the robot system 10 and a number of features or obstacles comprising a wall 1, a pillar 2, a table 3 and a chair 4. The wall 1 and pillar 2 are stationary, whereas the table 3 may sometimes be slightly displaced (thus the two hatched outlines) and the chair 4 is often in different positions (thus multiple hatched outlines)

[0073] Examples of static features can include fixed structures such as columns, walls, buildings, bridges, etc., as well as typically stationary equipment, such as semi-permanent walls, shelving, fire-extinguishers, safety equipment, etc. Dynamic features (also referred to as "dynamic objects" or "dynamic entities" refer to objects, features and / or structures that may move within the operating environment, particularly over the span of two or more navigations within the operating environment. Examples of dynamic features can include objects such as furniture, crates, etc.

[0074] When navigating in the space, the robot system 10 will determine its position in the space from its relative position vis-à-vis the features. Now, the robot will preferably rely more on, or solely on, the static features, as the robot's position, in the space, determined in relation to static features is more certain. Determining a position, in the space, of the robot vis-à-vis a dynamic feature such as the chair will give a less precise certain determination, as the position of the chair, in the space, is not constant.

[0075] The robot will usually, however, determine or detect all obstacles or features in its vicinity, so it is desired to be able to determine which features are static, and perhaps to which degree these are static, and which are dynamic and thus perhaps are not to be used - or on which less emphasis may be put, when determining the position of the robot and / or navigating in the space.

[0076] Naturally, determination or detection of also the dynamic features may be desired, not the least to prevent the robot from colliding therewith.

[0077] A feature may be determined to be static in a number of manners, some of which are described further below. In one manner, the static features are visibly marked in a manner so that a detection system may distinguish between marked features and un-marked features.

[0078] In one embodiment, the indoor mobile industrial robot system 10 can include a mobile robotic platform 12 configured to clean, treat, scrub or polish a floor surface or perform other similar actions using, for example, a trailing mop system, scrubber and / or squeegee. Other mobile robotic platform 12 applications include social interacting and guiding robots (e.g., airports, stock-keeping and monitoring for warehouses and supermarkets), detecting and cleaning contaminated surfaces, moving goods, etc. In some embodiments, an operator can stand on the mobile robotic platform and control the system 10 using a steering wheel. Alternatively, the one or more optical assemblies 12A / B, 14A / B, 16, can enable the system 10 to autonomously drive itself. The present disclosure describes various features that enable detection of objects, and classification of the objects as being one of either a dynamic feature or a static feature for improved navigation within an operating environment. Although the features described in the present application may be utilized in connection with industrial robotic systems, other applications of the static and dynamic feature classification systems and methods are contemplated. For example, static and dynamic feature classification systems and methods may be used on other types of user driven, semi-autonomous, and autonomous vehicles, or may be used as a standalone system (e.g., independent from any form of a mobile platform) for gathering information about an operating environment.

[0079] Successful identification of static and dynamic features play an important role in enabling successful navigation in both manual and autonomous modes. Static features can serve as landmarks to estimate ego-motion within the operating environment. Proper identification of objects as static features creates a more robust localization. More robust localization in turn, improves the accuracy of any new map entries. Accordingly, knowledge of dynamic features inhibits pollution of maps and enables embodiments of the present disclosure to make more informed decisions about the operating environment, thereby enabling safer, more efficient and elegant operation within the operating environment.

[0080] Embodiments of the present disclosure can use the various systems and methods to classify detected objects as being either a static feature or a dynamic feature. Such systems and methods can include, for example, the use of retro-reflectors as aids in properly classifying features as being either static or dynamic. Although such systems and methods are described herein as being distinct (e.g., standalone), such systems and methods, in whole or in part, may be optionally combined for improved classification.

[0081] In some embodiments, operating environments in which embodiments of the present disclosure are intended to operate can be augmented with retro-reflectors applied to known static features (as e.g. identified by a user). Retro-reflector markers can include highly reflective materials configured to direct a large portion of received light energy back to its source. Accordingly, navigation systems including active light-based sensors will generally receive a distinctly stronger signal upon interaction with retro-reflective markers, thereby improving the positive identification of static features.

[0082] In one embodiment, the operator identifies static features within the operating environment, and applies retro-reflective markers to surfaces of the static features, such as at an appropriate height. Thereafter, while operating within the operating environment, one or more active light-based sensors of the robot can be used to determine distances of objects relative to embodiments of the present disclosure, as an aid in navigation within the operating environment. Groups of data including distinctive signals corresponding to returns from retro-reflectors can be classified as static features within the operating environment. All other groups of data collected by the light-based sensors can be classified as dynamic features.

[0083] With reference to figure 2, in some embodiments, the active light-based sensor can include a light detection and ranging (LiDAR) unit 100 configured to detect both a distance and returned light intensity from objects within the operating environment, which in combination with positional information of the unit 100 can enable the collection of a dataset useful in constructing a multidimensional understanding of the distances of surrounding objects. In some embodiments, the dataset can include an x- and y-component or a relative distance and angle according to a polar coordinate system, for example, where the LiDAR unit is configured as a rotating beacon emitting a continuous source or pulses of light energy along a plane at a given distance from the floor (e.g., 8 inches above the ground). In some embodiments, the dataset can include a limited z-component, where the LiDAR unit is configured to additionally operate within a vertically oriented band, perpendicular to the floor. Such an understanding of the distances of surrounding objects is useful in mapping the operating environment for navigational purposes. In some embodiments, the LiDAR unit 100 can further be combined with other navigational and obstacle avoidance aids, such as one or more proximity sensors (e.g., a 3-D camera, or the like) configured to detect when embodiments of the present disclosure are in close proximity to an object within the operating environment, and / or is at risk of inadvertently impacting the object.

[0084] In some embodiments, the LiDAR unit 100 can include a laser unit 102, optical receiver 104, navigation module 106, positional module 108, and processor / database 110 / 112. The laser unit 102 can be configured to emit a light energy, for example in the form of a burst (e.g., a pulse) or continuous beacon as the unit 100 rotates with respect to the operating environment. In one embodiment, the emitted light energy can have a wavelength of approximately 1064 nm; although other wavelengths are also contemplated. The optical receiver 104 can be configured to detect light energy emitted by the laser unit 102 that is reflected back to the LiDAR unit 100 by the surrounding objects. In some cases, multiple returns corresponding to different surfaces of the objects can be received by the optical receiver 104.

[0085] The navigation module 106 can be configured to account for navigation of the navigational vehicle on which the LiDAR unit 100 is mounted within the operating environment, while the positional module 108 can be configured to account for rotation and / or other orientation factors of the LiDAR unit 100 relative to the navigational vehicle. The processor 110 can be configured to calculate distances to the surrounding objects based on the travel time of the emitted light and reflected return energy.

[0086] Frequently, emitted light may reflect off several different surfaces of a surrounding object, thereby indicating structural components and / or dimensional complexity of the object. The amount of energy received by the optical receiver 104 can be referred to as the "intensity." The areas where more photons or more light energy returns to the receiver 104 create peaks in a distribution (e.g., waveform curve) of the received energy. In some embodiments, these peaks in the waveform can be considered to represent surfaces in which the light energy has reflected. Accordingly, identifying multiple peaks representing different reflective surfaces associated with the object, can provide an estimation of the shape of the object.

[0087] In some embodiments, the processor 110 can be configured to translate the received light energy reflections into a collection of discrete points corresponding to the return peaks in the waveform curve. The collection of discrete return LiDAR points can be referred to as a LiDAR point cloud, which may in some embodiments include Cartesian and / or polar coordinate location values. Additionally, each of the discrete points may have an intensity value, representing the amount of light energy recorded by the receiver 104. The data can be stored by the memory 112.

[0088] Accordingly, where the amount of light energy recorded by the receiver 104 indicates the presence of a retro-reflector, the dataset associated with that object or feature can be classified as a static feature, thereby indicating that the object may be considered as a reliable navigational aid in future operations within the operating environment. All other datasets associated with other objects can be classified as dynamic features, thereby indicating that said objects have a likelihood of being moved within the operating environment between subsequent operations.

[0089] With reference to figure 3, an example operating environment 200 in which static features (e.g., walls 202A, 202B, & 202C and columns 204A & 204B) have been augmented with retro-reflectors 206, is depicted in accordance with an embodiment of the disclosure. A two-dimensional LiDAR unit 100, generally positioned in the center of the operating environment 200 receives distinctly more intense (e.g., brighter) returns from the marked static features, as compared to unmarked dynamic features (e.g., couch 208). As such, the distinctive difference in intensities of the range data enable embodiments of the present disclosure to distinguish between static and dynamic features.

[0090] Clearly, other types of visible or optically determinable features may be used instead of or in addition to retro reflectors, such as a particular color of the feature. A simple manner of indicating to the robot which features are static would be to paint these with a particular color which no other features in the scene or venue are allowed to have.

[0091] Autonomous machines typically use a variety of sensors as navigational aids while performing assigned tasks within an operating environment. Such sensors can include wheel encoders, LiDAR and vision systems such as 2-D and 3-D (e.g., stereo) cameras. Data streams from these sensors can be utilized in various combinations to build maps of the operating environment, which in some cases can be iteratively refined over multiple observations of or navigations through the same operating environment.

[0092] For example, with reference to figure 4A, in one embodiment, a 3-D camera unit 300 can be employed to gather positional data for detected objects based on a rotation angle of the unit 300 and / or other orientation factors with respect to a navigational vehicle on which the unit 300 is mounted, as well as a calculated distance to the detected objects, which can be based on the principle of binocular disparity.

[0093] In a typical 3-D camera unit 300, a left camera 302A and a right camera 302B are separated from one another by a fixed horizontal distance, typically referred to as a "baseline." With additional reference to figure 4B, observations by the left and right cameras 302A / B will have a slightly different view of the operating environment, which can be observed by an apparent horizontal shift of objects captured in the left and right images. Binocular disparity refers to the horizontal shift (e.g., difference in coordinates measured in pixels) of an identifiable feature of a detected object between the two images, which in turn can be used to determine a relative distance to the detected object.

[0094] In some embodiments, the unit 300 can employ a processor 304 to scan both the left and right images for a matching identifiable feature (e.g., a left edge of a detected object). Thereafter, the processor 304 can compute a disparity between the images as a general shift of the identifiable feature to the left in the right image. For example, an identifiable feature that appears in the nth pixel along the x-axis of the left image may be present in the nth-3 pixel along the x-axis of the right image. Accordingly, the disparity of the identifiable feature in the right image would be three pixels.

[0095] It should be noted that the use of a 3-D camera unit 300 to estimate distances to detected objects represents one exemplary embodiment of the present disclosure. Other mechanisms for estimating distances to detected objects are also contemplated. For example, embodiments of the present disclosure may use LiDAR (as discussed above), a 2-D camera unit with range finding capabilities, or other suitable approaches for determining distances to detected objects, which optionally can be used in combination for improved distance estimation capabilities.

[0096] Thereafter, the unit 300 can use the estimated distance in combination with positional information of the unit 300 gathered from a navigation module 308 and / or positional module 310 to construct a multidimensional understanding of the operating environment. In some embodiments, the multidimensional understanding can include the creation of a map including Cartesian and / or polar coordinate components. For example, in one embodiment, an operating environment can be broken up into an array of cells for mapping classification purposes.

[0097] With reference to figure 5A, an initial map 400A (representing map n) of an operating environment 401, in which the operating environment 401 has been divided into an array of cells 402, is depicted in accordance with an embodiment of the disclosure. For simplicity, the operating environment 401 is divided into a 10 x 10 array of cells 402; in operation, the operating environment 401 may be divided into thousands of individual cells 402.

[0098] As depicted in figure 5A, during an initial scan, the unit 300 detects a plurality of objects 404 within the operating environment. Although the line of sight of the unit 300 may not extend to the full boundaries of the operating environment, the unit 300 can begin to designate individual cells 402 within the map 400A as being one of occupied, unseen or empty; although other designations of individual cells 402 are also contemplated, such as a probability that a feature or obstacle is positioned at the corresponding position. Although not depicted in figure 5A for simplicity, in some embodiments, as the unit 300 moves about the room during the course of navigation, the designations of the cells 402 can be updated, including re-designation of previously designated unseen cells as being either occupied or empty to create a more complete map 400 of the operating environment. Thereafter, cells 402 designated as either occupied or empty can be assigned an initial value representing a probability of being a static feature, with empty cells being assigned a low probability indicating that any object detected within that cell in a subsequent observation is likely a dynamic feature (which is subject to move). By contrast, the occupied cells can be assigned an initial, neutral value indicating some uncertainty as to whether the object detected within that cell is a static or dynamic feature.

[0099] With reference to figure 5B, a subsequent map 400B (representing map n+1) can be created based on a subsequent observation within the operating environment, with the cells 402 again being designated as one of occupied, unseen or empty. In comparing the initial and subsequent maps 400A-B, it can be seen that one object 404A has disappeared, and two new objects 404G-H have appeared, with no apparent movement in the other previously detected objects 404B-F. Based on the comparison, the values previously assigned to the cells 402 representing a probability of the cell containing a static feature can be updated. The value for the now empty cells containing the previously detected object 404A can be decreased to indicate that the previously detected object 404A (as well as any object detected within those cells in a subsequent observation) is likely a dynamic feature. Newly detected objects 404G-H occupying cells previously designated as unoccupied or not observed can be assigned the initial, neutral value.

[0100] With reference to figure 5C, a second subsequent map 400C (representing map n+2) can be created based on additional observations within the operating environment, with the cells 402 again being designated as one of occupied, unseen or empty. In comparing the previous and subsequent maps 400B-C, it can be seen that object 404H has disappeared, and new object 404I has appeared, with no apparent movement in the other previously detected objects 404B-G. Accordingly, based on the comparison, the updated values previously assigned to the cells 402 representing a probability of the cell containing a static feature can be updated. In particular, the value for the now empty cells containing the previously detected object 404H can be decreased to indicate a low probability that those cells contain a static feature. Newly detected object 404I can be assigned the initial, neutral value, and the values for cells containing objects which have and continue to remain static can increase (eventually to a maximum) to reflect the overall level of confidence that the detected object occupying the cell represents a static feature.

[0101] Further, in some embodiments, various combinations of the described retroreflectors, trained model and automatic object recognition methods can be employed to properly classify features as being either static or dynamic. For example, in some embodiments, the retro-reflector and trained model approaches can be utilized to provide training data for the automatic object recognition method over the course of multiple training epochs. Feature classification can gradually shift towards automatic object recognition, after the respective weights and biases of the deep learning algorithm have been appropriately tuned. In some embodiments, various elements of the described LiDAR unit 100, 3-D camera unit 300, and automatic object recognition system 600 can be shared. For example, in some embodiments, a single unit comprising one or more of the LiDAR unit, 3-D camera unit and / or automatic recognition system can be constructed, in which the various features share a common processor database and / or optical unit. The invention is further illustrated by the following embodiments:

[0102] An indoor mobile industrial robot system configured to classify a detected object within an operating environment as likely being either one of a static feature or a dynamic feature, the indoor mobile industrial robot system comprising: a mobile robotic platform configured to self-navigate within an operating environment; a LiDAR unit operably coupled to the mobile robotic platform and configured to emit light energy and receive reflected light energy from the detected object; a positional module configured to account for at least one of a position and / or rotation angle of the LiDAR unit with respect to the mobile robotic platform; and a processor configured to translate the received reflected light energy and information from the positional module into a set of data points representing the detected object having at least one of Cartesian and / or polar coordinates, and an intensity, wherein if any discrete data point within the set of data points representing the detected object has an intensity at or above a defined threshold, the entire set of data points is classified representing a static feature, otherwise such set of data points is classified as representing a dynamic feature.

[0103] Various embodiments of systems, devices, and methods have been described herein. These embodiments are given only by way of example and are not intended to limit the scope of the claimed inventions. It should be appreciated, moreover, that the various features of the embodiments that have been described may be combined in various ways to produce numerous additional embodiments. Moreover, while various materials, dimensions, shapes, configurations and locations, etc. have been described for use with disclosed embodiments, others besides those disclosed may be utilized without exceeding the scope of the claimed inventions.

[0104] Persons of ordinary skill in the relevant arts will recognize that the subject matter hereof may comprise fewer features than illustrated in any individual embodiment described above. The embodiments described herein are not meant to be an exhaustive presentation of the ways in which the various features of the subject matter hereof may be combined. Accordingly, the embodiments are not mutually exclusive combinations of features; rather, the various embodiments can comprise a combination of different individual features selected from different individual embodiments, as understood by persons of ordinary skill in the art. Moreover, elements described with respect to one embodiment can be implemented in other embodiments even when not described in such embodiments unless otherwise noted.

[0105] In one or more examples, the described techniques may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include non-transitory computer-readable media, which corresponds to a tangible medium such as data storage media (e.g., RAM, ROM, EEPROM, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer).

[0106] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term "processor" as used herein may refer to any of the foregoing structure or any other physical structure suitable for implementation of the described techniques. Also, the techniques could be fully implemented in one or more circuits or logic elements.

Claims

1. A navigation system (10) for navigating a robot unit, the robot unit comprising one or more sensors (12A / B, 14A / B, 16, 100, 300) capable of sensing features remote from the sensors, in one or more of a scene and a venue (200), the scene / venue comprising a number of static features (1, 2, 202A, 202B, 202C, 204A / B) and a number of dynamic features (3, 4, 208), where the static features are walls, pillars, racks, fences, building structures, heavy furniture or storage elements, the system comprising a controller (106, 110) configured to: - receive an output from the sensor(s), - determining, from the output, a plurality of static and / or dynamic features (1, 2, 3, 4, 202A, 202B, 202C, 204A / B, 208), - recognizing, between the determined features, one or more static features (1, 2, 202A, 202B, 202C, 204A / B), - determining a position of the robot unit vis-à-vis the recognized static feature(s), - determining a position of the robot unit in the scene or venue based on information of a storage of the navigation system and the determined position of the robot unit vis-à-vis the recognized static feature(s), the storage comprising information representing a position of each of a number of the static features (1, 2, 202A, 202B, 202C, 204A / B), wherein the information of the storage represents a map of the scene or venue comprising therein the positions of the static features, characterized in that the complete static features are recognized as being static by having a fluorescent or reflective element applied thereto at a portion thereof, and in that the determination of the position of the robot unit in the scene or venue is based on a search in the map for correspondence with features, using only recognized static features and based on the complete extent of the recognized static features, to identify a position of the robot unit in the map.

2. The navigation system according to claim 1, wherein the controller is further configured to determine, from the output, one or more dynamic features and, in, the determination of the plurality of features, to determine the position only from the static feature(s).

3. The navigation system according to claim 1, wherein the controller is configured to recognize a static feature as a determined feature which: - emits or reflects an amount of radiation exceeding a predetermined minimum intensity, or - emits or reflects radiation at a predetermined wavelength.

4. A method of operating a navigation system for navigating a robot unit in one or more of a scene and a venue, the scene / venue comprising a number of static features and a number of dynamic features, where the static features are walls, pillars, racks, fences, building structures, heavy furniture or storage elements, the method comprising: - one or more sensors of the robot unit, the sensors capable of sensing features remote from the sensors, outputting information representing surroundings of the robot unit, - determining, from the information, a plurality of static and / or dynamic features, - recognizing, between the determined features, one or more static features, - determining a position of the robot unit vis-à-vis the recognized static feature(s), - determining a position of the robot unit in the scene or venue based on information of a storage of the navigation system and the determined position of the robot unit vis-à-vis the recognized static feature(s), the storage comprising information representing a position of each of a number of the static features, wherein the information of the storage represents a map of the scene or venue comprising therein the positions of the static features, characterized in that the complete static features are recognized as being static by having a fluorescent or reflective element applied thereto at a portion thereof, and in that the determination of the position of the robot unit in the scene or venue is based on a search in the map for a correspondence with features, using only recognized static features, and based on the complete extent of the recognized static features, to identify a position of the robot unit in the map.