Texture segmentation and obstacle detection for robotic navigation

Abstract

This disclosure generally relates to intelligent image segmentation and object recognition in digital images, and is particularly directed to real-time identification / classification of terrain texture and detection of obstacles and drivable area in an environment via object recognition and image segmentation for robotic and navigational applications. In particular, the disclosed Drivable Area Segmentation (DAS) framework uses a ground plane or environmental terrain segmentation algorithm to make a rough estimation of the drivable area or usable terrain area. This area is further refined using a modified fast One-Shot-Texture Retrieve (F-OSTR) model to exclude low profile objects in the area. The F-OSTR model includes a fast texture-oriented one-shot segmentation neural network trained using a Perlin-noise-based collage synthesis technique and thus does not need manually labeled training data. For an area with non-uniform textures, a plurality of texture masks may be generated based on, for example F-OSTR and then used to more accurately classify the texture of the area. In addition, LiDAR (Light Detection And Ranging) sensors may be employed to generate density point cloud to supplement RGB image based obstacle detection in order to reduce false obstacle detection due to varying lighting conditions.

Description

TEXTURE SEGMENTATION AND OBSTACLE DETECTION FOR ROBOTIC NAVIGATION INCORPORATION BY REFERENCE

[0001] This application is based on and claims the benefit of priority to U.S. Provisional Patent Application No.63 / 529,977 filed on July 31, 2023 and entitled " Texture Oriented and Self-Supervised One-Shot Drivable Area Segmentation," which is herein incorporated by reference in its entirety. BACKGROUND Technical Field

[0002] This disclosure generally relates to intelligent image segmentation and object recognition in digital images, and is particularly directed to real-time identification / classification of terrain texture and detection of obstacles and drivable area in an environment via object recognition and image segmentation for robotic and navigational applications. Background Technologies

[0003] Reasonably accurate robot perception of an operating environment may be one of the essential prerequisites for mobile robotic navigation. Specifically, indoor mobile robots often need to operate in cluttered and confined spaces initially designed for human access or spaces with many obstacles and drastically variable lighting conditions, and these environments may include terrains having variable and non-uniform textures, which raises some unique challenges for robotic perception. In addition, many indoor or outdoor mobile robotic applications require real-time determination of their environments but yet may be limited by their computation capabilities. As such, fast models for accurately deriving navigable areas from perception data in real-time is critical. BRIEF SUMMARY

[0004] This disclosure generally relates to intelligent image segmentation and object recognition in digital images, and is particularly directed to real-time identification / classification of terrain texture and detection of obstacles and drivablearea in an environment via object recognition and image segmentation for robotic and navigational applications. In particular, the disclosed Drivable Area Segmentation (DAS) framework uses a ground plane or environmental terrain segmentation algorithm to make a rough estimation of the drivable area or usable terrain area. This area is further refined using a modified fast One-Shot-Texture Retrieve (F-OSTR) model to exclude low profile objects in the area. The F-OSTR model includes a fast texture-oriented one-shot segmentation neural network trained using a Perlin-noise-based collage synthesis technique and thus does not need manually labeled training data. A validation dataset is further created to quantify the efficacy of the fast DAS framework. The validation test shows that the F-OSTR approach has a competitive accuracy and much higher inference speed compared to some other benchmarking semantic segmentation models. Furthermore, the framework is deployable on a low-power SBC and capable of achieving real-time inference performance.

[0005] In some other aspects of this disclosure, LiDAR (Light Detection And Ranging) sensors in addition to images (such as stereo RGB images with or without depth dimension) may be further employed to capture dense point cloud data of the environment using reflected laser beams, which may be unaffected by external lighting condition of the environment. The LiDAR data may enable a detection of terrain slope, distance, and obstacles even in completely dark environments and thus may be used as complement or alternatives to approaches based on image segmentation and object recognition in RGB images, which may be particularly helpful in reducing false detection of obstacles and / or terrain patterns in the presence of light reflections and shadows.

[0006] In some other aspects for further addressing issues associated with non- uniform texture in some terrains (particularly terrains covering wide areas with multiple terrain types), the one-shot texture segmentation model above may be used to provide semantic masks form the terrain image before identifying the terrain type. Each semantic mask may contain a single texture appearance. The masks may then be fed into an image clarifier based on, for example, a pre-trained neural network (e.g., a convolutional neural network). Such an approach may allow for a precise classification of terrain types based on pre-trained categories. In a practical application, a proprioception-based neural network model may further identify theclosest terrain type surrounding a robot (e.g., a ground vehicle) using data from its control unit, encoders, and feedback from the motor of the robot.

[0007] In an example implementation, a method for navigating a robot is disclosed. The method may include receiving, from a camera, an RGB image, a point cloud, and a depth image of a surrounding of the robot, and an inertia measurement of the camera; automatically generating an orientation-normalized and ground-plane segmented point cloud based on the received point cloud and the inertia measurement; automatically generating a ground-plane mask based on the orientation-normalized and ground-plane segmented point cloud and the depth image; automatically generating a masked image by applying the ground-plane mask to the received RGB image to mask out non-ground-plane obstacles in the received RGB image; receiving a single-shot reference textual image; automatically generating a drivable area map for the robot using an offline-trained one-shot texture retrieve (OSTR) model to further segment the masked image to exclude additional on-the-ground obstacles; and automatically navigating the robot to avoid the non- ground-plane obstacles and on-the-ground obstacles according to the drivable area map.

[0008] Obstacle maps generated above based on RGB stereo images may be fused with other obstacle maps generated from the LiDAR data. Through appropriate sensor calibration and integration of a proprioceptive devices, the RGB image sensors and LiDAR sensors may be aligned to the same coordinate system. The fusion of the obstacle maps may provide improved detection accuracy of, for example, drivable or navigable areas of the environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0010] Fig.1 illustrates a general data and logic flow for Drivable Area Segmentation (DAS) in robotic navigational applications.

[0011] Fig.2 illustrates an example general framework for efficiently utilize sensor readings from a RGB-Depth (RGB-D) camera to derive drivable areas in the form of traversability maps for robotic applications.

[0012] Fig.3 illustrates further details of an example general DAS framework for efficiently utilize sensor readings from RGB-D camera to derive drivable areas in the form of traversability maps for robotic applications.

[0013] Fig.4 illustrates an example coordinate configuration for an RGB-D camera.

[0014] Fig.5 illustrates an example RGB image take from the RGB-D camera of Fig.4.

[0015] Figs.6-9 illustrate example processing of point clouds and RGB-D data taken from the RGB-D camera of Fig.4 and example steps as part of a DAS framework for deriving drivable areas for robotic applications.

[0016] Figs.10-11 illustrate a comparison between two example textural data generation techniques.

[0017] Fig.12 illustrates an example Perlin-Noise-based collage generation process.

[0018] Fig.13 illustrates example benchmark results of a backbone models on ImageNet with input size 224 x 224.

[0019] Fig.14 illustrates ablation testing of the variously drivable area segmentation models with input image size of 256 x 256.

[0020] Fig.15 illustrates example RGB images used for validation of the example DAS framework of this disclosure.

[0021] Fig.16 illustrates quantitative results from the validation testing of Fig.14.

[0022] Figs.17-19 illustrate example qualitative results from validation test with drivable areas highlighted in blue (see color drawings of the as-filed application) and obstacles highlighted in red (see color drawings of the as-filed application), with reference images and depth images shown in smaller size (second column from the left).

[0023] Fig.20 illustrates example extension of the disclosed DAS framework to various example outdoor applications.

[0024] Fig.21 illustrates an example architecture for a faster one-shot texture retrieval (F-OSTR) model.

[0025] Fig.22 illustrates a texture encoder component of the example F-OSTR of Fig.21.

[0026] Fig.23 illustrates an example global context generator of the example F- OSTR of Fig.21.

[0027] Fig.24 illustrates an example handheld device for collecting Indoor Small Objects Dataset (ISOD).

[0028] Fig.25 shows an example data and logic flow for fusing RGB image based obstacle mapping and LiDAR based obstacle mapping. DETAILED DESCRIPTION

[0029] Various aspects for intelligent image processing will now be described in detail hereinafter with reference to the accompanied drawings, which form a part of the present disclosure, and which show, by way of illustration, various example implementations and embodiments. The imaging processing and robotic navigational devices and systems disclosed herein may, however, be embodied in a variety of different forms and, therefore, the disclosure herein is intended to be construed as not being limited to the embodiments set forth below. Further, the disclosure may be embodied as methods, components, and / or platforms in addition to the disclosed devices and systems. Accordingly, embodiments of the disclosure may, for example, take the form of hardware, software, firmware or any combination thereof.

[0030] In general, terminology may be understood at least in part from usage in its context. For example, terms, such as "and", "or", or "and / or," as used herein may include a variety of meanings that may depend at least in part upon the context in which such terms are used. Typically, the term "or", if used to associate a list, such as A, B or C, is intended to mean A, B, and C, here used in the inclusive sense, as well as A, B or C, here used in the exclusive sense. In addition, the term "one or more" or "at least one" as used herein, depending at least in part upon context, may be used to describe any feature, structure, or characteristic in a singular sense or may be used to describe combinations of features, structures or characteristics in a plural sense. Similarly, terms, such as "a", "an", or "the", again, may be understood to convey a singular usage or to convey a plural usage, depending at least in part upon context. In addition, the term "based on" or "determined by" may beunderstood as not necessarily intended to convey an exclusive set of factors and may, instead, allow for the existence of additional factors not necessarily expressly described, again, depending at least in part on context.

[0031] Many other modifications of the implementations above may be made to adapt a particular situation or material to the teachings without departing from the scope of the current disclosure. Therefore, it is intended that the present methods and systems not be limited to the particular embodiments disclosed, but that the disclosed methods and systems include all embodiments falling within the scope of the appended claims. General Implementations

[0032] By ways of introduction, reasonably accurate robot perception may be one of the essential prerequisites for indoor or outdoor mobile robot navigation applications. Depending on the complexity of the robot’s locomotion mechanism and the expected operating environment, mobile robots are designed to have specialized perception capabilities to navigate and interact with the environment appropriately and in safe manners.

[0033] Specifically, indoor mobile robots often need to operate in cluttered and confined spaces initially designed for human access and can dynamically change, which raises some unique challenges for robotic perception. For example, these robots usually have relatively small form factors such that they can access narrow doorways or under-counter spaces. This characteristic restricts the battery, computation power, and sensor systems that can be carried on the robot. Likewise, indoor mobile robots generally have low ground clearance and small actuation wheels. As a result, they are more susceptible to jamming and tipping caused by small obstacles, which may be driven over by larger outdoor robots. Hence, the perception systems for the smaller indoor robots need to be capable of detecting and identifying small obstacles and identifying the corresponding regions as not drivable. In addition, complex floor plan, layouts, and scattered interior objects can create a large number of occlusions, reflections, and confusing situations that hinder the performance of robot sensors and limit their FOVs (Field of View). As a result, the robot’s perception system must be fast and responsive enough to react to sudden obstacles and to facilitate avoidance maneuvers.

[0034] Such indoor navigational robots may include but are not limited to smart wheelchairs, service rotes, and the like. For example, a service robot may be implemented in the form of a smart vacuum machine, smart floor mopping machine, mobile hospitality service stations, and the like. For another example, a smart wheelchair may include an assistive rideable ballbot system referred to as MiaPURE (Modular, interactive, and adaptive Personalized Unique Rolling Experience), operated as an omnidirectional, dynamically balanced device driven by a single spherical wheel. Its under-actuated dynamics induce tilting motions during accelerations. An example MiaPURE system is described in an earlier U.S. Non- Provisional Patent Application No.17 / 464,446 filed on September 1, 2021 by the same applicant. The content of that patent application is herein incorporated by reference in its entirety. Such types of robot system may require accurate determination of drivable areas in real-time because they must be able to handle sudden change of surroundings and usually have very low ground clearance and is easily tipped over by small undetected objects.

[0035] The perception system for such indoor robotic applications may rely on processing of vision input data. Traditional vision-based methods for segmenting obstacle-free floor areas from monocular images may be based on the assumption that the floor has a uniform texture. For example, supervised Deep Neural Networks (DNNs) may be deployed to effectively recognize low-lying objects and thus exclude them from a final drivable area prediction. However, training of such DNNs usually requires a large amount of expensive manually labeled datasets. Other homography-based floor segmentation methods often fail to capture low-lying small obstacles (typically ≤ 3 ^^ height). Accurate and low-noise information for low-lying objects may also be difficult to obtain from other direct depth sensors such as laser range finders and stereo cameras.

[0036] In some example implementations, separate texture-based training may be used to achieve One-Shot Texture segmentation (or Retrieve) (OSTR). For example, texture generation techniques such as Voronoi-based collage generation may be used to automatically generate synthesized images with pixel-wise textual annotations for the training of one-shot texture segmentation models. The trained models can identify, from an input image, regions that contained a targeted texture when given a reference image of that texture. The training of such model would theninvolve no manually labeled textural datasets. In an OSTR implementation, only a single reference image of the target texture needs to be provided for the model to perform segmentation to identify regions in an input image occupied by the same reference texture, hence the term “one-shot”, as opposed to needing many training images of that texture in order for the model to be capable of performing segmentation of an input image.

[0037] The various example implementations below describe an efficient indoor Drivable Area Segmentation (DAS) framework capable of detecting small unknown obstacles in real-time (~30 Frame-Per-Second, FPS) on an example low-power Single Board Computer (SBC), such as UpXtreme-i7. The disclosed framework may be configured to approach the problem of DAS in several example steps. As shown in the data and logic flow 100 of Fig.1. As an essential component of the disclosed DAS framework, a Faster OSTR (F-OSTR) model may be generated by performing a series of backbone optimizations and model compressions of traditional OSTR models to ensure adequate accuracy and inference speed for the application of indoor robot navigation. Additionally, the disclosed DAS framework uses a Perlin- Noise-based collage generation method for automatic training data generation that avoids expensive manual labeling and significantly improves the accuracy and robustness of the resulting F-OSTR model.

[0038] As shown in 102 of Fig.1, a data and logic flow 100 for the example DAS framework first acquires a composite image of a surrounding of the robot. The image may be taken from one or more built-in camera of the robot, e.g., an RGB-D (RGB and Depth) camera. An example composite image is shown in 202 of Fig.2 as including, for example, a three-color-channel RGB image with depth information, and a point cloud. In step 104, a real-time and motion-robust ground plane segmentation circuitry of the framework may be employed to segment the point cloud collected by the RGB-D camera and generate a rough estimation of the drivable area, as shown in 202 of Fig.2.

[0039] As further shown in step 106 of Fig.1, a one-shot segmentation circuitry may be employed to refine the drivable area by filtering out small obstacles and anomalies in the RGB image. The outputs of the example framework may include a simplistic 2D obstacle map, shown as an example in 206 of Fig.2, that encodes all the large obstacles (e.g., walls, furniture, pedestrians, etc.) for collision avoidancepurposes, and a traversability map shown as 208 of Fig.2, which indicates the surrounding regions that are free of small obstacles or anomalies. Given a general texture reference and characteristics of the traversable surface, the example framework aims to detect all obstacles that might endanger the robot system including the small unknown objects that are not detectable using point clouds or depth images.

[0040] Fig.3 provides further example specifics to the overall DAS framework above and provides a visualization of the data and logic flow of various data stream therein. As shown in Fig.3, an RGB-D camera 302 (e.g., Intel RealSense D455) may output four types of real-time sensor data: an RGB image 304, a depth image 306, a point cloud 308, and an Inertial Measurement Unit (IMU) data 310. The point cloud 308 may represent 3D structures of the captured scene. The IMU data 310 may be used to describe a spatial / angular orientation of the RGB-D camera 302. The example framework may feed the point cloud 310 and the IMU data 310 to a ground plane segmentation node or circuitry 312. The ground plane segmentation node or circuitry 312 may be configured to separate the raw point cloud into a ground plane group 314 and a non-ground group 316. The segmented point clouds may be used by an OctoMap construction node / circuitry 318 to construct a 3D OctoMap 320 (i.e., a grid map constructed by 3D voxels) that describes the 3D scene. Then, the 3D voxels are projected to the ground to generate a more simplistic 2D occupancy grid 320. The 3D OctoMap 320 may not be able to provide information of the small objects on the ground plane, but it does provide the course large object arrangement to assist the robot in subsequent high-level control tasks like path planning and navigation, in, for example, avoiding collisions with these larger objects (such as furniture).

[0041] The ground plane model from the ground plane segmentation node or circuitry 312 and the depth image 306 may be further used to mask the raw RGB image 304 using a ground plane masking node / circuitry 321, such that most of the obstacle pixels are masked out by painting them black. This pre-processing step prepares query images including the masked image 322 (with the non-ground portion of the RGB image masked) to the OSTR node / circuitry 324 which may be implemented as a fast OSTR, referred to as F-OSTR. The F-OSTR 324 further segments the ground regions into drivable areas and anomalies based on a targetterrain surface given by a single-shot reference image 326, which embeds features of the ground texture (e.g., tiled or carpeted floor). Finally, the outcome of the F- OSTR 324 may be re-projected back to the robot workspace coordinate to form the traversability map 328.

[0042] As compared to conventional supervised learning models, the training of the F-OSTR model for the generation of the traversability map follows a self- supervised methodology rather than relying on any dense (pixel-wise annotations) manually labeled datasets. In the example implementations described below, a novel Perlin-Noise-based collage generation technique is used for generating automatically training datasets each with known ground truth. In particular, the training dataset generating starts with a set of classes of texture pattern library. Training images are generated by using randomly generated combinations of mask patterns and the texture patterns with arbitrary length scale (including length as small as the small objects the final F-OSTR is configured to segment). Each of these images together with a particular texture pattern as reference, and the corresponding mask of the particular texture pattern for generating the image form a triple in the training datasets. The corresponding mask function as the ground truth (see Fig.12 and the corresponding description below for detail). As such, an arbitrarily large number of training datasets may be generated in a fast manner to allow for an offline training of the F-OSTR model. Such offline-trained F-OSTR model may then be included in the example DAS framework of Fig.3 for drivable area segmentation in an inference mode based on an input image and a corresponding one-shot reference.

[0043] In addition, the F-OSTR model, as described in further detail below, may be derived from a full version OSTR model architecture via an ablation process to remove or lighten components that may not significantly affect segmentation accuracy in indoor environment, but may facilitate reduction of inference speed and suitable for being run on less powerful computing system (such as the example UpXtreme-i7 SBC system). As such, the resulting F-OSTR model provides a short inference time that enables the example framework to be deployment on an SBC (with e.g., ~70 FPS (frames per second) for the neural network model alone and about 30 FPS for the entire framework as limited by a sampling rate of the camera).

[0044] Furthermore, due to the nature of one-shot segmentation, the example framework may be easily reconfigured and adapt to different working environments or robotic mission by changing the reference input image.

[0045] As indicated above and described in further detail below, the key contributions of the disclosed example DAS framework of Fig.3 include at least the following: • A real-time and motion-robust ground plane segmentation algorithm for detecting large and negative obstacles in point cloud measurements. • A modified F-OSTR model optimized for the generation of the traversability map in an embedded, real-time DAS application. • A Perlin-Noise-based collage generation technique for automatically generating training datasets for training the robust texture segmentation models (F-OSTR models). • From a model testing standpoint, an Indoor Small Objects Dataset (ISOD) that contained diverse and densely labeled images created for validating the performance of the example DAS framework.

[0046] The example implementations above are not described in the indoor context, but they are not so limited. These examples may be further applicable to outdoor environment.

[0047] In some other aspects of this disclosure, LiDAR sensors in addition to optical images (such as stereo RGB images with or without depth dimension) may be further employed to capture dense point cloud data of the environment using reflected laser beams, which may be unaffected by external lighting condition of the environment. The LiDAR data may enable a detection of terrain slope, distance, and obstacles even in completely dark environments and thus may be used as complement or alternatives to approaches based on image segmentation and object recognition in RGB images, which may be particularly helpful in reducing false detection of obstacles and / or terrain patterns / textures in the presence of light reflections and shadows and the like.

[0048] In some other aspects for further addressing issues associated with non- uniform texture in some terrains (particularly terrains covering wide areas withmultiple terrain types), the one-shot texture segmentation model above may be used to provide multiple semantic masks for the terrain image before identifying the terrain type. Each of the multiple semantic masks may contain a single texture appearance. The plurality of masks may then be fed into an image clarifier based on, for example, a pre-trained neural network (e.g., a convolutional neural network). Such an approach may allow for a precise classification of terrain types based on pre-trained categories. In a practical application, a proprioception-based neural network model may further identify the closest terrain type surrounding a robot (e.g., a ground vehicle) using data from its control unit, encoders, and feedback from the motor of the robot.

[0049] As a result, this invention provides a terrain recognition architecture by fusing multiple machine learning models. In a bird's-eye view image captured by a stereo camera, the proposed invention can identify the surrounding terrain type using both visual and proprioceptive feedback, and can estimate farther areas with distinct terrain textures. The proposed architecture has undergone rigorous testing using off- road dataset and quantitative analysis. All hardware and software integration tasks have been meticulously completed by our research team.

[0050] Further details of the example implementation above are provided below. Ground Plane Modeling

[0051] The example DAS framework above include performing ground plane segmentation (e.g., via node / circuitry 312 of Fig.3). The ground plane segmentation of 3D point clouds 308 constitutes an essential step to filter away points of less interest. In the context of the example DAS framework disclosed here, a ground plane segmentation may be used as a pre-processing step of an input RGB-D image to obtain a preliminary estimation of the drivable areas. As the example DAS framework is expected to operate primarily indoors, an approximation may be made such that the geometry of an indoor ground plane can be roughly modeled using a point-normal equation: ax + by + cz + d = 0 (1)

[0052] where [^, ^, ^]^contains constant values that form the normal vector ^^ of the ground plane (e.g., assuming a ground plane without curving), and d represents an orthogonal offset distance between the ground plane and the point cloud’s frame.The position vector of the ^^^point in the point cloud that belongs to the ground plane may be written as ^^^^^^^,^= [^, ^, ^]^. Equation (1) above may then be rewritten into matrix form: ^^^^^^^^^^,^= − (2)

[0053] The objective of the ground plane segmentation node or circuitry 312 of Fig.3 is to estimate the values of the normal vector ^^ and offset distance d such that they can be used to determine whether a given point belongs to the ground plane ^^^^^^^or not. Filtering

[0054] In some example implementations, a ground plane segmentation algorithm may be based on a RANdom SAmple Consensus (RANSAC) based method. A reasonably suitable initial guess (candidate points) may be selected for the targeted ground plane. This can reduce the need for more RANSAC iterations, and the algorithm is more likely to coverage to the true plane. In reliably obtaining these candidate points, the example algorithm may apply a series of filters and operations to the input point cloud, as show in Fig.4-9 with the input point cloud having a coordinate system shown in Fig.4.

[0055] First, an example voxel grid filter may be used to down sample the raw point cloud data taken by the camera and shown in Fig.4 by taking a spatial approximation of the points in each voxel (i.e., a 3D cube with a defined size). This filter generates voxel indexes ! over the input point cloud ^^^"^^:

[0056] where ^@^^, ^@^^, and ^@^^are the minimum point position values of ^^^"^^. In some example implementations, the size of the voxel A may be empirically defined to be, for example, 0.045m × 0.15m × 0.045m or 0.03m × 0.2m × 0.03m (X-Y-Z convention) such that the output point cloud has a lower resolution on the vertical axis (i.e., y-axis, the coordinate frame is defined in Fig.4). Voxel points will then be sorted based on their voxel index. The set of points within the same voxel share thesame voxel index and may be replaced by their centroid. This filter can significantly reduce the number of points for subsequent processing without losing vital features. An example down sampled point cloud is shown in Fig.6.

[0057] Second, the synchronized IMU reading (pitch and roll angles) from the camera may be utilized to align the point cloud orientation with the gravity vector to cancel the robot’s tilting motion as shown by Figs.6 and 7. The raw IMU reading may consist of angular rates D = [EF0, EF7, EF:] as measured by a gyroscope, and linear accelerations G = [α0, α7, α:] as measured by an accelerometer in three Euclidean axes. The Euler angles (^^I: K7, L^M^ℎ: K0, O(PP: K:) that describe the camera orientation may be calculated using an advanced complementary filter that fuses D and G. The pitch K0and roll K:angles may be used to construct a rotation matrix Q∈ ℝS×S. The yaw angle K7may be set to zero as this operation has no interest in changing the yaw rotation. Q follows a Y-X-Z convention such that gimbal lock only happens if the camera points straight up or down (which rarely happens for a forward-facing camera). If ^U^0VW∈ ℝS×^denotes the 3D positions of all the points processed by the voxel grid filter, this operation is simply represented by:

[0058] After the alignment, the resulting ground plane in ^XW^^^V^may appear mostly upright in the camera frame, as shown in Fig.7.

[0059] There may be two main benefits of performing an orientation alignment. First, the alignment can simplify the problem as if the robot will never tilt. And as the ground (floor) is orthogonal to the gravity vector in most indoor spaces, RANSAC would only needs to determine one parameter, the offset distance d in this simplified problem. Second, the alignment may allow the downstream mapping algorithms like Simultaneous Localization and Mapping (SLAM) to consistently map the environment in a unified planar frame.

[0060] For a third example filtering process, a box crop operation may be applied to retain the down sampled points in Fig.7 within a predefined number (e.g., 7) of meters of planar range to the camera frame, where the measurements are most reliable. The points that are at or more than another predefined number (e.g., 1) of meters above the camera may also be removed to discard ceiling points, which serve little value in ground robot navigation.

[0061] Finally, an example radius outlier removal filter may be used. This filter may implement, for example, an Kd-Tree radius search to determine a number of neighbors Y that lie within a radius of a query point. If Y is less than a predetermined threshold, then this query point may be removed. This example filter helps reduce artifact points created by reflective surfaces. Also, as the filtered point cloud has a lower resolution on the vertical axis (y-axis), points in vertical structures (e.g., walls, large furniture) may be sparse and may have fewer neighboring points. Hence, they are more likely to be filtered out. Most remaining points are thus likely to belong to large planes orthogonal to the gravity vector, as shown in Fig.8 and would provide good RANSAC candidates. Ground Plane Segmentation

[0062] In some example implementations, once the candidate points are determined, as shown in FIG.8, they may be further sorted based on their heights and the lowest, for example, 50% of points may be sampled to form the initial seed for the RANSAC algorithm, a pseudo code of which is shown in Algorithm 1 below. The RANSAC algorithm may then iteratively estimate the ground plane model ^^ and d using the candidate points of Fig.8. Ideally, ^^ can be derived directly from the IMU readings if the relation between ^^ and the IMU reference frame (e.g., gravity vector) is fixed and known. However, there is no guarantee that the floor plane is perfectly orthogonal to the gravity vector (e.g., the ground plane may have a slight tilt), and the IMU readings are also susceptible to strong vibrations during operation. Therefore, a Principal Component Analysis (PCA) may be used to refine the normal vector ^^ by finding the singular vector corresponding to the smallest singular value. As the candidates are pre-filtered using the IMU, RANSAC is thus less likely to converge onto false planes (e.g., walls, large furniture) compared to using the PCA method alone. Algorithm 1: Pseudocode of the ground plane segmentation node Inputs: ^^^"^^: the raw input pointsK7, K0, K:: the Euler angles estimated by

[0043] A: the size of the voxel gridZ: the size of the box cropY@^^: the threshold for radius outlier removalO: the radius for the radius search[: the RANSAC iteration budget, default = 3]^": the upper bound of ground plane thickness]W^^: the lower bound of ground plane thicknessOutputs: ^^^^^^^: the ground points^^_`^X\WV: the obstacle / non-ground points^^ \^^^V\^V^: the normal vector of the ground plane: the orthogonal distance to the ground plane1. ^U^0VW= abcdefghijhekdg2. Q = gbknkhb^jgbopqedgr^sedt(K7, K0, K:) 3. bghd^knkhb^rehs^od^k(^U^W, Q) 4. ubcvgb>5. = gnihqtwqkehdgQdobane(^\^^""V^) 6.(^\X^^^^X^V, ^X0^`^xy* ^*z (O yO) 7. = {h^i|dn^(}^OxM 400 yPy^y*Mx (} ^`^^^V^) 8. jbg each point >^in ^`^^^V^do: 9. 10. 11.12. p^i {bg 13. ^^^^^^^= ^`VV^14. jbg ^ = 0; ^ < [; ^^^: 15. ^^ , = dtkhonkd^en^d22. petd:28. petd: 29. 30. p^i31. p^i {bg 32. p^i h{ 33. p^i {bg 34.

[0063] At the final iteration, the algorithm above may use the estimated ^^ and d to infer the unfiltered point cloud ^\^^""V^and segment it into two groups: the obstaclegroup (red, upper of Fig.9) and the ground plane group (green, lower of Fig.9). Due to the imperfectness in the point cloud measurement, the ground points might not form a clear-cut surface. Therefore, a set of threshold values ]^"and ]W^^may be used to determine the ground plane’s upper and lower padding thickness (tolerance). Any point within this tolerance range may be considered a ground point; others may be considered obstacle points. In some example implementations, ]^"and ]W^^may be empirically determined to be, for example, 0.03^ and 0.06^, respectively.

[0064] In some example implementations, there may be two settings for the function estimatePlane(∗) in Algorithm 1. Users can either choose to estimate the value of d alone or to estimate the value of ^^ as well. As mentioned earlier, the orientation alignment can keep the ground plane mostly upright, so the normal vector estimation may be unnecessary. However, intense accelerations and vibrations might compromise the IMU’s ability to estimate orientation accurately and cause the aligned ground plane to be slightly tilted. In this case, a Principal Component Analysis (PCA) can be used to estimate the normal vector of the aligned ground plane and refine the result. First, a covariance matrix v∈ ℝS×Smay need to be computed from the candidate points ^\∈ ℝS×@: v = (^\−^^^^\)(^\−^^^^\)^

[0065] where P¡\∈ ℝS×@consists of the mean position of the candidate point cloud along the row direction. The covariance matrix C∈ ℝS×Srepresents the dispersion of the candidate points, and the direction that accounts for the least variance is likely to be orthogonal to the ground plane (a flat surface) and parallel to the normal vector.

[0066] Singular Value Decomposition (SVD) may be used to identify this direction by finding the singular vector corresponding to the smallest singular value. In one example, the entire segmentation algorithm takes 7~13^x computation time on the single board computer (SBD) described above (depending on the size of the point cloud, the results might vary). Typically, this setup is able to recognize floor objects higher than 5^^. If users choose to skip the SVD calculation to preserve computation power, estimatePlane(∗) will return [0,1,0] as the default value of n^ . Note that, in both cases, n^ only reflect the normal vector of the ground plane afterorientation alignment. Therefore, an inverse operation may be needed to obtain the actual normal vector n^\^^^V\^V^for subsequent algorithms. Ground Plane Masking

[0067] The output from the ground plane segmentation node or circuitry 312 above may be then used to mask the RGB image 304 and form the “masked image” 322 in Fig.3, which served as one of the inputs to the F-OSTR 324.

[0068] For example, the ^^\^^^V\^V^and estimated above may be sent to the ground plane masking node 321 of Fig.3, where the depth image may be used in combination to mask the RGB image. First, a homography transformation may be applied to wrap the depth image to align with the RGB image. This step may be necessary as the depth sensor and the RGB sensor may have different FOVs and coordinate frames. Second, the ^^\^^^V\^V^and d may be used to screen the pixel values in the depth image to build a mask that blocks all the pixels belonging to the obstacle group. Finally, this mask may be applied to the RGB image and form the masked ground image, which may serve as one of the inputs to the F-OSTR 324.

[0069] There may be two benefits of using the masked image instead of the raw RGB image:

[0070] 1) vision-based DNNs are often influenced by contextual correlations. Therefore, blocking out non-task-related background pixels may mitigate their effect and guide the model’s attention to the subject of interest (the ground areas); and

[0071] 2) the F-OSTR 324 may be designed to segment the regions in the query image that share a similar texture to the reference image and does so without any scene understanding. When the surrounding walls or furniture have a similar texture as the ground / floor, the F-OSTR 324 might mistake those regions as drivable areas. However, using the masked ground image can prevent these edge cases. Texture Modeling in the F-OSTR

[0072] In some example implementations, a traditional OSTR model may be modified for faster execution for real-time and normal frame rate applications. The F-OSTR model may be configured to process two inputs, i.e., the query image referred to as Q (322 of Fig.3), which may contain regions of different textures, and the reference image referred to as R (326 of Fig.3), which embodies the targettexture class. The output of OSTR may be a probability map that indicates regions of the query image that correspond to the target texture class. To train such a model, a triple T^of training samples may be needed: T^= (Q, R^, G^) (6) where Q is the query image that contains at less one texture class, G^is the pixel- wise ground truth label corresponding to the ^^^class texture in Q, andis the reference image of the ^^^class.

[0073] The neural network model }(∗) can be presented in the following form: P^= }(Q, R^, θ) (7) represents the predicted segmentation result and θ is the set of learnable parameters in function }(∗). The goal of the training process is to learn the optimized θ∗given the training triple T^such that the loss function is minimized: θ∗= argmin ­(xx(P^, G^) (8)

[0074] For the loss function, choices may include but are not limited to Focal Loss, Focal Tversky Loss, Balanced-Binary Cross Entropy Loss (B-BCE), Dice Loss, Batch-Dice Loss, and Combo Loss. An example Combo Loss may be used as the loss function in the example implementations of this disclosure. It may essentially be constructed as a weighted sum of a B-BCE Loss and a Dice Loss. The combination of these two loss functions is smooth and may be able to handle class imbalance. Such weighted combination for the loss function may be represented by: ­\^@_^= ®­¯ / ¯°±+ ­²^\V(9)

[0075] where ® represents the weighting factor in favor of B-BCE Loss. In some example implementations, the value of ® may be set to 10. The B-BCE Loss can be written as follows:

[0076] −(1 − ·)(1 − Y)log³1 −µY¶ (10)

[0077] whereµY denotes the prediction made by the neural network, Y represents the ground truth label, · = 1is the weighting factor that balances the contribution of the positive and negative pixel labels, and ^^denotes the pixel that contains a positive label. Similarly, Dice Loss may be written as:

[0078] where À is the smoothing factor that prevents the denominator from ever becoming zero when|Y|= ¾µY¾ = 0. While the Dice Loss is non-convex in nature, the combination of these two loss functions is smooth and able to handle class imbalance. Perlin-Noise-based Collage Generation & Original OSTR Model Training

[0079] The traditional OSTR relies on a benchmark texture database referred to as the Describable Textures Dataset (DTD) to for training. DTD is a dataset that contains 5640 images organized into, for example, a number (e.g.47) of texture categories based on human perception. A random Voronoi-based collage generation technique may be used, for example, to form synthetic query images, as shown in Fig.10. However, query images generated in this manner generally have three biases: 1) there is no discontinuous region of the same textures; 2) there are only straight edges separating different textures; 3) there is no small region in the collage, and region areas are relatively balanced. As a result, the model trained using these synthetic query images likely lacks the ability to deal with arbitrary class boundaries in the real world. As such, rather than using Voronoi-based collage generation techniques, the example F-OSTR implementations of the current DAS framework may be trained using texture data generated based on a novel Perlin- Noise-based collage generator, which help synthesize pseudo-random query images with pseudo random combination of textual patches, as illustrated in Fig.11.

[0080] Perlin noise approach is a type of procedural generation technique used in, for example, the game-development and visual effect industries to generate pseudo-random textures or terrain maps. The smoothness and the amount of detail in the generated shapes can be manipulated by a grid size and a number of Octaves used in the Perlin noise function. This allows the user to have some degree of control over the random generation process. This example image generator may create a pseudo-random binary map that masks each texture candidate in the query image. For each texture candidate in the query image, for example, the generator may create a 32 × 32 Perlin noise map (pixel values range from 0 to 1) with an Octave count of * (* = 2 by default, * = 3 if the number of texture candidates > 3). A cutoff threshold that ranges from 0.1 to 0.2 is randomly chosen to convert the 2DPerlin noise map to a binary mask. The binary map may then be resized to match the size of the candidate texture image. A gaussian blur filter may then be used to smooth the resized map before it is used to mask the candidate texture image. Finally, all the masked candidates may be concatenated to form the synthetic query image.

[0081] A visualization of this process is shown in Fig.12. In particular, a random number (e.g., 5) of textures may be chosen with corresponding random 5 complementary binary masks being generated. Yellow regions of the masks in Fig. 12 represent textured areas. As such, the yellow regions of the 5 example random masks are complementary in that they do not overlap (e.g., the same area would not be occupied with two different textures). Then, all masked candidates may be concatenated to form the final synthetic query image, as illustrated by the image labeled as “query” in Fig.12. The generated query, and a selected reference texture (from one of the 5 example component textures, e.g., the botchy reference) and the correponding binary map (the first birnary map correcpoinding to the “botchy” textural component) forms the one tripple among the training datasets for the F-OSTR model as described in further detail below.

[0082] During training, all training images may be resized to 256 × 256, and up to a predefined number (e.g., 5) of texture candidates may be randomly sampled to form the query image. In some example implementations, the disclosed filtering allows the case when only one texture candidate is sampled (the target texture class itself). This prepares the model for the situation when no foreign texture is presented during test time. A reference imagethat corresponds to the ^^^class (the target class) in Q may be randomly chosen. The pseudo-random mask of the ^^^class may be used as G^. Together, Q, R^, and G^form the triple T^in Equation (6).

[0083] All T triples may be randomly generated at each training or evaluation step. All random processes may be tightly controlled by random seeds and random generators to ensure reproducibility. The random seeds and generators are updated such that the training triples are different in every epoch and every data loader worker (if parallel processing is enabled); however, the evaluation triples are always the same to provide a consistent and fair comparison. Among the predetermined number (e.g., 47) of texture classes in DTD, a subset of them (e.g., 42 of them) maybe used to form the training dataset, and the remaining five classes may be used as an evaluation set. OSTR Model Streamlining & Ablation Study

[0084] In some example implementations, to ensure that the neural network of the OSTR model has low latency (high FPS throughput) during deployment for real- time situations, the traditional OSTR may be modified to only retain a series of model essential components. Like many one-shot learning models, the original OSTR has a typical dual-branch encoder structure (e.g., a Siamese encoder), as shown in the example F-OSTR model of Fig.21.

[0085] As shown in Fig.21, the example F-OSTR model 2100 includes an encoding portion 2102 and a decoding portion 2104. The encoder portion 2102 includes a backbone of encoders having two branches (the query image branch and the reference branch) each containing a series of reduction encoder layers (or blocks). The final encoder block in each branch is then processed by a textual encoder 2110. The two branches basically extract textual features from the input image and the reference and these features are then compared to extract their differences by a global context extractor 2120. The differences are then processed by the decoder portion 2104. The decoder portion contains a series expanding decoding layers which gradually expand the difference from the global context module or circuitry 2020 back to original image resolution by also incorporating feature information from the encoder portion at corresponding layers, as shown by the collecting arrows from the encoder layers to the decoder layers. The output of the F-OSTR, as shown by 2130, is the segmented drivable map.

[0086] The throughput of the encoder 2102 containing the backbone encoders accounts for a substantial portion of the model inference time. Therefore, the key to achieving real-time performance may be to find an efficient backbone model as the encoder for each encoding layer of Fig.21. A benchmark study on, for example, seven backbone models amongst the various choices for building efficient segmentation models, including ResNet-based models (i.e., ResNet-18, 34, and 50), EfficientNet, EfficientNet-v2, MobileNet-v2, and MobileNet-v3. All these example models may be pre-trained on the ImageNet dataset and converted into a Float16 Intermediate Representation (IR, a model format optimized for model deployment)using a model optimizer from Intel OpenVINO (ver 2022.1). As the SBC underlying the various example implementations is powered by mostly Intel hardware (Intel i7- 8665UE CPU and Intel UHD Graphics 610 GPU), OpenVINO IR can provide optimal performance. Fig.13 illustrates a bubble plot that shows the example benchmark result of these models. The vertical axis shows the ImageNet Top 1 accuracy of the converted models, and the horizontal axis is the FPS evaluated on the single-board computer. The sizes of the bubbles in Fig.13 indicate the sizes of the model relative to each other. According to Fig.13, the selected example selected model, MobileNet-v3 appears to be over two times faster than the most accurate model (EfficientNet-v2), yielding only ~3% lower in accuracy.

[0087] Fig.14 provides a result of the ablation study. The vertical axis shows the Mean Intersection over Union (MIoU) of the trained models on DTD, and the horizontal axis is the FPS evaluated on the single board computer. The original OSTR (referred to as the “OSTR baseline”) is shown in FIG.14 as a performance baseline. “OSTR mbnv3” in Fig.14 represents the OSTR’s performance after changing the Siamese encoder to the example MobileNet-v3 encoder. It is noted that the rest of the models all use the example MobileNet-v3 encoders. The term “TexEnc” and “GloCon” are the abbreviations for the texture encoder module and the global context module, respectively. According to an example F-OSTR architecture shown in Fig.21, the new backbone greatly improved the inference speed and marginally increased the segmentation accuracy.

[0088] Besides adapting the encoder backbone for accuracy for the indoor navigational applications, an example ablation study that dissects each component’s contribution toward accuracy and speed is performed. There are two main components in the OSTR, namely the texture encoder module or circuitry (alternatively referred to as “TexEnc”) 2110 and global context (or global grouping, (referred to as “GloCon”) module or circuitry 2120.

[0089] An example texture encoder 2110 is illustrated in Fig.22. For example, the example texture encoder 2110 may be configured to capture the directional texture features in the query and reference branches. In the example implementation of Fig.22, eight directional maps may guide the neural network to register these directional features into eight groups. A learnable scaling module maybe configured to emphasize the most effective directional feature groups for different inputs.

[0090] On the other hand, as shown by an example implementation in Fig.23, the global context module / circuitry 2120 may be used to compare the corresponding directional feature groups from the query branch and the reference branch. The same set of network layers may be used to compare each feature group pair, allowing the model to access all the feature groups globally using relatively small channel dimensions.

[0091] The example final model architecture as modified in Figs.21-23, denoted as “F-OSTR” 324 of Fig.3, may be implemented as a combination of “Reduced TexEnc I” and “BottleNeck GloCon”. The former reduces the channel dimensions of the hidden layers in the texture encoder module for the reference branch, and the latter adds a bottleneck layer in the global context module. The example F-OSTR, as tested, provides an average FPS of 70.21 and an MIoU of 68.3%, which is about 393% faster than the baseline OSTR with a marginally higher (~0.2%) accuracy, as shown in Fig.14.

[0092] All the models in the example ablation study above are trained using the DTD dataset for 100 epochs with a batch size of 16. Five texture classes: "banded", "blotchy", "braided", "bubbly", and "bumpy" are used as the evaluation set. The initial learning rate is set to 1 × 10 / Sand the training optimizer uses SGD with a momentum of 0.9 and a weight decay of 1 × 10 / Â. A step learning rate scheduler may be used with a step size of 10 and a gamma value of 0.8. The training may be performed on an example pre-built PC with 16G of RAM, an Intel i7-11700 CPU, and an NVIDIA GeForce RTX 3060 GPU (12Gb). Validation Testing - Validation Dataset

[0093] To quantify the efficacy of the proposed framework, an Indoor Small Objects Dataset (ISOD) is collected and used as a validation dataset. The ISOD contained 2,000 diverse and manually labeled images collected from 20 different sites, including over 30 kinds of randomly placed small objects in this dataset in addition to the items already in the scenes. For example, these small objects include LEGO blocks, rags, slippers, gloves, shoes, cables, crayons, chalk, glasses, smartphones (cases), fake banana peels, fake pet waste, piles of toilet paper, etc.The heights of objects are typically ≤ 3^^, which are difficult to detect in the point cloud 306 and thus would be part of ground plane in ground plane segmentation 312 if scattered on the drivable surface (e.g., floor) of the scene. These objects are chosen because they either pose threats to the safe operation of indoor mobile robots or they may leave a mess behind when run over. Example images of the ISOD are shown in Fig.15. Besides the RGB images, the ISOD also record the corresponding depth images and IMU readings, as needed in the data flow of Fig.3. To preserve storage space, instead of logging the raw point clouds 308, the ISOD directly include output (the “masked image” 322 in Fig.3) of the ground plane masking node / circuitry 321 as the query images for the F-OSTR 324. A reference image 326 of each floor type for the ISOD may also be recorded using a separate camera.

[0094] All the data in the ISOD may be collected using a handheld device, as shown in Fig.24. This device consisted of the Intel D455 RGB-D camera, the Up Xtreme single-board computer, a portable monitor, and battery units, as indicated in Fig.22. One human operator may be designated for all the data collection. The collection process involved video-tapping the entire scene with the handheld device while moving around at walking speed (≤ 1^ / x). During collection, the handheld device may be held at different heights (ranging from 0.2^ to 1^) to include more perspective height variance. Tilting motions of the handheld device may be incorporated to simulate the operations of dynamically balanced robots. The recorded IMU readings show that the camera’s pitch ranged from −30°to −10°(tilts slightly downward to capture more floor areas), and the roll angles ranged from −30°to 30°to test the robustness of the proposed framework. Each collection session may last, for example, for 3 minutes per scene. For example, one hundred samples may be randomly selected from the raw data recording and added to the ISOD. Finally, the ISOD data may be annotated (or segmented) with the drivable regions in the RGB images with positive labels (pixel value =1), and the remaining areas with negative labels (pixel value =0), thereby generating manual traversability maps as labels for 328 of Fig.3. Validation Testing - Benchmark Model Preparation / Training

[0095] The ISOD described above may only be used as a validation dataset. The F-OSTR 324 of Fig.3 may be trained using the DTD dataset and the method asdescribed above with respect to training data generated via Perlin-Noise-based generation process in relation to Fig.12. In some example implementations, additional data may be augmented in the process above including but not limited to: 1) random blackened-out regions in the synthetic query image such that the model learned to ignore the blackened-out regions in the masked ground image during test time; 2) an illumination-based data augmentation technique may be used to improve F-OSTR’s performance in the presence of cast shadows and light spots; and 3) standard data augmentations like horizontal & vertical flips, random rotation & affine transformation, and color jittering. In some example implementations, the F-OSTR 324 may be trained for 500 epochs with a batch size of 16. For example, the initial learning rate may be 1 × 10 / Sand the training optimizer may be based on Adam optimizer. The step size of the learning rate scheduler may be, for example, 35, and the gamma value may be, for example, 0.9. The performance and efficacy of the resulting F-OSTR may be compared to several benchmarking segmentation models for driving the traversability maps.

[0096] It may be difficult to directly compare the performance of the DAS framework described above to similar existing indoor DAS methods. However, various RGB-D semantic segmentation models may be used as a basis for DAS and as benchmarks for the DAS framework described above. These semantic segmentation models are part of a set of powerful and versatile general computer vision tools. They may be adapted for use on robots or embedded systems and used as benchmarking herein. Such adapted models may enrich the robot’s visual perception and facilitate specific subsequent processing steps, like DAS, based on semantic information. In some example implementations, the DAS segmentation by these models may be based on texture difference between the floor and non-floor areas rather than relying on semantic or contextual information from the scenes, and therefore, may be more generalizable to unknown environments and objects. Such adapted RGB-D semantic segmentation models may be used as performance indicator and benchmarks to quantify the efficacy of the DAS framework F-OSTR node / circuitry 324 as described above.

[0097] The indoor semantic segmentation, being the main purpose of these RGB- D semantic segmentation models, is arguably a more complicated task compared to DAS. Thus, the fully built and trained RGB-D semantic segmentation models mightpay unnecessary attention to objects unrelated to the DAS task. Therefore, to generate the benchmarking models for a fairer comparison, these semantic segmentation models are modified and retrained using modified data from, e.g., SUNRGB-D. SUNRGB-D is a particular indoor semantic segmentation benchmark dataset that contains 10,335 RGB-D images collected from 37 scene categories. For modification of this semantic dataset, a label reduction process is conducted to turn this semantic dataset into a drivable area dataset. First, for example, samples in SUNRGB-D whose areas contain less than 1% of “floor” or “carpet” labels are filtered out. Second, for example, the labels of the resulting 7,946 samples are converted to binary maps (“floor” & “floor_mat” =1, all others =0) and randomly divided into a training set and evaluation set with a ratio of 4:1. Two example RGB- D semantic segmentation models, RedNet (referred to as the first RGB-D semantic segmentation model) and ESANet (referred to as the second RGB-D semantic segmentation model), are then trained using this modified dataset to generate their DAS version, RedNet-DAS and ESANet-DAS, and to achieve training accuracies (e.g., Mean Intersection over Union (MIoU)) of, for example, 82.4% and 88% respectively. Furthermore, the unmodified original OSTR, and the RedNet, and ESANet trained using the un-modified dataset are added to the set of benchmark models as baselines.

[0098] While the RGB and depth images are used as inputs to the retrained RGB-D semantic segmentation models during the test below, the RGB (masked) and texture reference images are used as inputs to the original OSTR as a benchmark and the F-OSTR of the current DAS framework. All the image data may be resized to, for example, 256 × 256. The outputs of these models are all binary maps with the same size that predicted the drivable areas, except for the vanilla RedNet and ESANet. Their outputs required an additional mapping ("floor" & "floor_mat" =1, all others =0) for benchmarking. For a fair comparison, all the tested neural network models may be converted into Float16 IR and deployed to the same SBC for estimating the inference speed. Validation Testing - Results

[0099] The metrics used in the validation test may include accuracy (measured in MIoU, Mean Intersection over Union) and inference speed (measured in FPS). Fig. 16 shows that the 52.2% MIoU of the original OSTR achieved 73% accuracy onISOD. An increase in accuracy (+10.5%) is observed when the OSTR is trained using the Perlin-Noise-based collage generator (instead of the Voronoi-based). A decrease in accuracy (-6.5%) is observed when the original OSTR is trained using the Voronoi-based collage generator. This implies the synthetic training examples generated by the Perlin-Noise-based method better prepare the models for the real- world scenes. The two pre-trained original RGB-D semantic models, the RedNet and ESANet models, achieve accuracies of 71.0% and 80.7%, respectively. Their retrained DAS versions, RedNet-DAS and ESANet-DAS, show improved accuracies of 85.1% and 83.0%, respectively. This suggests that one of these models, the ESANet, has higher generalizability, while the other, the RedNet, appears to overfit the training data as it has higher training accuracy.

[0100] By design, the second of the two benchmarking RGB-D semantic segmentation models, the ESANet model, is more speed optimized. The inference speed results indeed suggest that the second model is about two times faster than the first model. With an inference speed of 21.71 FPS, as shown in Fig.16, the ESANet-DAS model will suffice for most indoor robot perception needs. However, in comparison, the current DAS framework using the F-OSTR is over three times faster than the ESANet-DAS segmentation benchmarking model while having a competitive accuracy, leaving even more computation resources for subsequence robot control algorithms.

[0101] This result suggests that the F-OSTR is a superior choice for indoor DAS. To dissect the factors that contribute to the improved performance, the testing process further benchmarks the F-OSTR with ResNet-34 and ResNet-50 as backbones. As expected and as shown in Fig.16, the boost in inference speed is largely contributed by the faster backbone. On the other hand, the accuracies of these ResNet-based F-OSTRs are comparable to the current F-OSTR based on MobNetv3. F-OSTR-34 has the highest accuracy in this test and a similar FPS to the ESANet-DAS model. The F-OSTR-50’s architecture is virtually identical to the original OSTR except for the changes added in the ablation study. Evidentially, the model compression improved both the accuracy and inference speed of the original OSTR.

[0102] The validation study also included the accuracy of the ground plane segmentation algorithm in 312 of Fig.3. This accuracy may be calculated bycomparing the ground truth with the masked image whose blackened pixels are mapped to 0 and all others to 1. Fig.16 shows that it can achieve an MIoU of 82.7% even though the algorithm cannot identify small objects (≤ 5^^ in height). This is because the pixel areas of those small objects only account for a small fraction of the entire scene. Therefore, completely neglecting those objects does not necessarily result in a devastating MIoU for grand plane identification. The same logic applies to the results of the neural network models. Note that the test for the F-OSTR above used the masked RGB images logged in the ISOD dataset rathe rather than using masked images as generated based on the ground plane segmentation node / circuitry 312. To gain better insights and demonstrate the performance differences between the tested methods, further qualitative results from the validation test may be generated.

[0103] Figs.17-19 show that ISOD is a challenging dataset for indoor semantic models. Both the original RedNet and the ESANet models struggle to identify the floor containing small objects. The DAS versions of them have clear improvement since they use a more general open set to categorize non-floor objects. Both of these models can capture the floor regions even when the scene is tilted. However, they mostly ignored the smaller objects presented in the scenes as their training data SUNRGB-D does not contain enough small object examples. On the contrary, the current F-OSTR model can identify small low-lying objects regardless of shape, color, and size, as shown by the last columns of Figs.17-19. As this model is trained completely using synthetic images, it does not require manually labeled small object training examples like the semantic models. Furthermore, as the current DAS framework method first estimates the ground plane using an analytic algorithm, it rarely produces false positive labels on walls and furniture. This added safety can help mitigate the uncertainty of data-driven methods like neural networks. General Applicability of DAS

[0104] While the description above focus on indoor application of the example DAS framework, the various disclosed models and framework may be applicable other environments. The underline principles may be used and adapted to building a DAS framework that applies to outdoors environment. In addition, the potential of the F-OSTR as described can go beyond refining DAS. For example, Fig.20 shows that such F-OSTR can effectively detect trash and litter on outdoor terrain like sandand snow. In that situation, the outdoor “ground plane” would be, for example, textured terrain (e.g., asphalt). For example, when given an image of well- maintained asphalt, the F-OSTR framework may identify cracks on a runway image, demonstrating that F-OSTR can facilitate applications such as autonomous cleaning robots and robot runway patrol. Note that the same F-OSTR model used in the validation test above may be directly transferred (without any retraining or fine- tuning) to generate the predictions in Fig.20, showing that F-OSTR can be easily repurposed and adapted to a new environment or task. LiDAR Sensors for Improving False Obstacle Detection

[0105] As described above, the various indoor and outdoor environment may be subject to varying lighting conditions, including amount of lighting, light reflection, shadowing that depends on location of light sources. Such lighting conditions would render variable appearances of an entirety or portions of the terrain. For example, shadowed areas of a terrain may be falsely identified as obstacles when there are, in fact, no obstacles.

[0106] In some aspects of this disclosure, LiDAR (Light Detection And Ranging) sensors in addition to images (such as stereo RGB images with or without depth dimension) may be further employed to capture dense point cloud data of the environment using reflected laser beams emitted from the robot. Such dense point cloud information derived from the LiDAR data may enable a detection of terrain slope, distance, and obstacles even in completely dark environments. Such data may be unaffected by external lighting condition of the environment and thus may be used as complement or alternatives to approaches based on the environmental lighting dependent image segmentation and object recognition in RGB images. Such data may be particularly helpful in reducing false detection of obstacles and / or terrain patterns in the presence of light reflections and shadows.

[0107] For example, obstacle maps generated above based on RGB stereo images may be fused with other obstacle maps generated from the LiDAR data. Through appropriate sensor calibration and integration of a proprioceptive devices, the RGB image sensors and LiDAR sensors may be aligned to the same coordinate system. The fusion of the obstacle maps may provide improved detection accuracy of, for example, drivable or navigable areas of the environment.

[0108] FIG.25 shows an example implementation 2500 involving acquiring real- time composite RGB images from RGB cameras from a robot in 2502 and generate a first obstacle map based on the composite RGB images in 2504; acquiring LiDAR point cloud from by scanning laser beams and detecting reflected laser beams at the robot at 2506 and generating a second obstacle map based on the LIDAR point cloud; fusing the first obstacle map and the second obstacle map to reduce false detection of obstacles at 2510. For example, only the object aligned in both the first obstacle map and the second obstacle map are registered as detected obstacles. Alignment of the obstacles may be defined as alignment of a centers of mass of the obstacles in the first map and the second map. The final mask of a confirmed obstacle may be defined as the union of the corresponding masks for the obstacle in the first map and the second map. Masking and Classification of Terrain with Non-Uniform Texture

[0109] As described above, some indoor or outdoor terrains, particular the ones with large coverage areas, may contain varying and non-uniform or diverse textures or terrain types. An overall classification of the texture of the terrain may then be inaccurate. In some example implementations, to address this issue, a one-shot texture segmentation model, such as the example disclosed above, may be used to provide multiple semantic masks from the terrain image before identifying the terrain type. Each of the multiple semantic masks may be associated with a single texture appearance. The multiple masks associated with multiple texture appearances may then be fed into a specially pretrained model such as a CNN-based image classifier. This may allow for the precise classification of terrain types based on pre-trained categories.

[0110] Additionally, a proprioception-based model, e.g., CNN model, can be pretrained to further identify the closest terrain type surrounding the robot or ground vehicle using data from its IMU, encoders, and feedback from the robot's motor.

[0111] As a result of the above implementations, a terrain recognition architecture may be provided by fusing multiple machine learning models. In a bird's-eye view image captured by a stereo camera, such implementations may be used to identify surrounding terrain type using both visual and proprioceptive feedback, and may be used to estimate farther areas with distinct terrain textures.

[0112] Such an ability to accurately recognize and classify different terrain types using advanced machine learning models may open up numerous opportunities across multiple fields where terrain analysis and adaptation are crucial. Such terrain recognition technologies can be integrated into autonomous vehicles, particularly off- road, all-terrain vehicles, and drones. It also has applications in agricultural machinery, construction and mining, environmental monitoring, and defense. In summary, the disclosure above provides a fast and motion-robust RGB-D DAS framework for robot navigation. The framework uses a ground plane or environmental terrain segmentation algorithm to make a rough estimation of the drivable area or usable terrain area. This area is further refined using F-OSTR to exclude low profile objects in the area. The F-OSTR includes a modified and reduced fast texture-oriented one-shot segmentation neural network trained using a novel Perlin-noise-based collage synthesis technique and thus does not need labeled training data. A validation dataset ISOD is created to quantify the efficacy of the proposed framework. The validation test showed that the F-OSTR approach has a competitive accuracy and much higher inference speed compared to some other benchmarking semantic segmentation models. Furthermore, the framework is deployable on a low-power SBC and capable of achieving real-time inference performance.

[0113] The output segmented traversibility map 328 may then be incorporated by the robot in its navigational controller to avoid the areas including the small object detected by the F-OSTR indicated as non-drivable in the traversibility map 328 in addition to avoiding collision with larger obstacles.

[0114] In some other implementations, the ground plane segmentation algorithm above may be refined as described in Appendix A with example updated pseudo- code included therein.

[0115] In some example implementations, instead of the OSTR and improved OSTR models described above, an example Miniature One-Shot Texture Segmentation (MOSTS) model may be used. The MOSTS model is described in the attached additional disclosure including its design, justification, and benefit. Further benchmark studies on the DTD dataset comparing MOSTS, OSTR, and prior One- Shot Texture Segmentation (OSTS) models are also included in the attachedadditional disclosure. In addition, the validate studies above on ISOD using the MOSTS model are also included in the attached additional disclosure.

[0116] It is to be understood that the various implementations above are not limited in its application to the details of construction and the arrangement of components set forth above and in the accompanying drawings. The disclosure is intended to cover other embodiments that may be practiced or carried out in various ways following the underlying principles disclosed herein.

[0117] It should also be noted that a plurality of hardware and software-based devices, as well as a plurality of different structural components may be used to implement the various embodiments of the disclosure. In addition, it should be understood that embodiments of this disclosure may include hardware, software, and electronic components or modules that, for purposes of discussion, may be illustrated and described as if the majority of the components are implemented solely in hardware. However, one of ordinary skill in the art, and based on a reading of this disclosure, would recognize that, in at least one embodiment, the electronic based aspects of the invention may be implemented in software (e.g., stored on non- transitory computer-readable medium) executable by one or more processors. As such, it should be noted that a plurality of hardware and software-based devices, as well as a plurality of different structural components may be utilized to implement the invention. Furthermore, and as described in subsequent paragraphs, the specific mechanical configurations illustrated in the drawings are intended to exemplify embodiments of the invention and that other alternative mechanical configurations are possible. For example, “controllers” described in the specification can include standard processing components, such as one or more processors, one or more computer-readable medium modules, one or more input / output interfaces, and various connections (e.g., a system bus) connecting the components. These controllers may be implemented as dedicated processing circuitry or in general- purpose processors, in combination of various software and / or firmware, and in combination of other wired or wireless communication interfaces.

Claims

CLAIMS We claim:

1. A method for navigating a robot, comprising: receiving, from a camera, an RGB image, a point cloud, and a depth image of a surrounding of the robot, and an inertia measurement of the camera; automatically generating an orientation-normalized and ground-plane segmented point cloud based on the received point cloud and the inertia measurement; automatically generating a ground-plane mask based on the orientation- normalized and ground-plane segmented point cloud and the depth image; automatically generating a masked image by applying the ground-plane mask to the received RGB image to mask out non-ground-plane obstacles in the received RGB image; receiving a single-shot reference textual image; automatically generating a drivable area map for the robot using an offline- trained one-shot texture retrieve (OSTR) model or miniature one-shot texture segmentation (MOSTS) model to further segment the masked image to exclude additional on-the-ground obstacles; and automatically navigating the robot to avoid the non-ground-plane obstacles and on-the-ground obstacles according to the drivable area map.

2. The method of claim 1, wherein the OSTR or MOSTS model is offline-trained using a machine-generated training dataset.

3. The method of claim 2, wherein the machine-generated training dataset for the OSTR or MOSTS model is synthesized using a Perlin-noise-based collage synthesis procedure.

4. The method of claim 3, wherein the Perlin-noise-based collage synthesis procedure is performed on a set of textual images belonging to a plurality of textural classes.

5. The method of claim 4, wherein the machine-generated training dataset comprises a plurality of triples each comprising a synthesized query textual image, areference textual class image, and a ground truth mask indicating the reference textual class within the synthesized query textual image.

6. The method of claim 5, wherein the Perlin-noise-based collage synthesis procedure comprises generating each of the plurality of triples by: determining one or more candidate textural classes among the set of textual classes; generating one or more pseudorandom Perlin-noise binary maps corresponding to the one or more candidate textural classes; concatenating corresponding textual images among the set of textual images using the one or more pseudorandom Perlin-noise binary maps to form the synthesized query textual image; choosing the textual image corresponding to one of the one or more candidate texture classes as the reference textual class image; and choosing a Perlin-noise binary map among the one or more pseudorandom Perlin-noise binary maps corresponding to the reference textual class image as the ground truth mask.

7. The method of claim 1, wherein the OSTR or MOSTS model comprises an encoder network followed by a decoder network.

8. The method of claim 7, wherein the encoder network comprises: two branches of reducing backbone encoders, each branch being followed by a textual encoder; and a global context extraction network for combining outputs of the textual encoders of the two branches of backbone encoders.

9. The method of claim 8, wherein a first branch of reducing backbone encoders are configured to process a query image whereas a second branch of reducing backbone encoders are configured to process a reference image of a reference texture.

10. The method of claim 9, wherein each of the two branches of reducing backbone encoders comprises a MobileNet-v3 encoder.

11. The method of claim 9, wherein the textual encoders are configured to extract textual features from the two branches of reducing backbone encoders and the global context extraction network is configured to compare the textual features from the two branches of reducing backbone encoders as extracted from the textual encoders.

12. The method of claim 9, wherein the decoder network comprises a cascading series of expanding decoding blocks following the global context extraction network and coupled to the first branch of reducing backbone encoders, the decoder network being configured to generate a predicted segmentation mask of the query image for the reference texture.

13. The method of claim 1, wherein automatically generating the orientation- normalized and ground-plane segmented point cloud based on the received point cloud and the inertia measurement comprises: down-sampling the received point cloud; performing an orientation normalization of the down-sampled point cloud based on the inertia measurement; and removing a set of outliers from the down-sampled and orientation normalized point cloud to generate the orientation-normalized and ground-plane segmented point cloud.

14. A controller for navigating a robot, comprising a processor for executing a set of computer instructions to: receive, from a camera, an RGB image, a point cloud, and a depth image of a surrounding of the robot, and an inertia measurement of the camera; automatically generate an orientation-normalized and ground-plane segmented point cloud based on the received point cloud and the inertia measurement; automatically generate a ground-plane mask based on the orientation- normalized and ground-plane segmented point cloud and the depth image;automatically generate a masked image by applying the ground-plane mask to the received RGB image to mask out non-ground-plane obstacles in the received RGB image; receive a single-shot reference textual image; automatically generate a drivable area map for the robot using an offline- trained one-shot texture retrieve (OSTR) model or miniature one-shot texture segmentation (MOSTS) model to further segment the masked image to exclude additional on-the-ground obstacles; and automatically navigate the robot to avoid the non-ground-plane obstacles and on-the-ground obstacles according to the drivable area map.

15. The controller of claim 14, wherein the OSTR or MOSTS model is offline- trained using a machine-generated training dataset.

16. The controller of claim 15, wherein the machine-generated training dataset for the OSTR or MOSTS is synthesized using a Perlin-noise-based collage synthesis procedure.

17. The controller of claim 16, wherein the Perlin-noise-based collage synthesis procedure is performed on a set of textual images belonging to a plurality of textural classes.

18. The controller of claim 17, wherein the machine-generated training dataset comprises a plurality of triples each comprising a synthesized query textual image, a reference textual class image, and a ground truth mask indicating the reference textual class within the synthesized query textual image.

19. The controller of claim 18, wherein the Perlin-noise-based collage synthesis procedure comprises generating each of the plurality of triples by: determining one or more candidate textural classes among the set of textual classes; generating one or more pseudorandom Perlin-noise binary maps corresponding to the one or more candidate textural classes;concatenating corresponding textual images among the set of textual images using the one or more pseudorandom Perlin-noise binary maps to form the synthesized query textual image; choosing the textual image corresponding to one of the one or more candidate texture classes as the reference textual class image; and choosing a Perlin-noise binary map among the one or more pseudorandom Perlin-noise binary maps corresponding to the reference textual class image as the ground truth mask.

20. The controller of claim 14, wherein the OSTR or MOSTS model comprises an encoder network followed by a decoder network.

Images