System and method for performing simultaneous localization and mapping using machine vision system

Abstract

The invention provides a mobile robot configured to navigate an operating environment, the mobile robot including a controller circuit that directs a driver of the mobile robot through the environment, navigates the mobile robot using a camera-based navigation system, and a camera coupled to the controller circuit. And a camera including optics defining a camera field of view and a camera optical axis, where the camera is positioned within the recessed structure and tilted such that the camera optical axis is aligned at an acute angle above a horizontal plane consistent with the top surface and aimed in a forward driving direction of the robot body, and the camera is configured to capture an image of an operating environment of the mobile robot.

Description

[0001] This application is a divisional application of the invention patent application with the application number 2021111329351, the application date of November 17, 2015, and the invention title of "Systems and Methods for Performing Simultaneous Localization and Mapping Using a Machine Vision System".

[0002] The invention patent application with the application number 2021111329351 is a divisional application of the invention patent application with the application number 2015800407063, the application date of November 17, 2015, and the invention title of "Systems and Methods for Performing Simultaneous Localization and Mapping Using a Machine Vision System". Technical Field

[0003] Systems and methods for capturing images are described herein for obtaining visual measurements for use in simultaneous localization and mapping. Background Art

[0004] Many robots are electro - mechanical machines controlled by computers. Mobile robots have the ability to move around in their environment and are not fixed to a physical location. Examples of commonly used mobile robots today are automated guided vehicles or automatic guided vehicles (AGVs). AGVs are typically considered mobile robots that follow markers or wires in the floor or use vision systems or lasers for navigation. Mobile robots can be found in industrial, military, and security environments. They also appear as consumer products for entertainment or for performing specific tasks such as vacuum cleaning and home assistance.

[0005] To achieve full autonomy, mobile robots typically need to have the ability to explore their environment, build a reliable map of the environment, and localize themselves within the map without user intervention. Considerable research has been conducted in the field of simultaneous localization and mapping (SLAM) to address this problem in mobile robotics. The development of more accurate sensors and better navigation algorithms has enabled significant progress towards building better robots. Summary of the Invention

[0006] The present invention provides a mobile robot configured to navigate an operating environment, the mobile robot comprising: a body having a top surface; a drive mounted to the body; a recessed structure below the plane of the top surface and near the geometric center of the body; a controller circuit in communication with the drive, wherein the controller circuit guides the drive to navigate the mobile robot through the environment using a camera-based navigation system; and a camera including optics defining a camera field of view and a camera optical axis, wherein: the camera is positioned within the recessed structure and is tilted such that the camera optical axis is aligned at an acute angle of 30 - 40 degrees above a horizontal plane coinciding with the top surface and is aimed in the forward driving direction of the robot body, the field of view of the camera spans a frustum of 45 - 65 degrees in the vertical direction, and the camera is configured to capture images of the operating environment of the mobile robot.

[0007] In some embodiments, the camera is protected by a lens cap aligned at an acute angle with respect to the camera optical axis.

[0008] In certain embodiments, the lens cap is retracted rearward relative to the opening of the recessed structure and is at an acute angle with respect to the camera optical axis, the acute angle being closer to a right angle than the angle formed between the plane defined by the top surface and the camera optical axis.

[0009] In some embodiments, the acute angle is between 15 and 70 degrees.

[0010] In some embodiments, the angle formed between the plane defined by the opening in the recessed structure and the camera optical axis ranges between 10 and 60 degrees.

[0011] In some embodiments, the field of view of the camera aims at static features at a distance of 3 feet to 10 feet from the static features, which are located in the range of 3 feet to 8 feet from the floor surface.

[0012] In some embodiments, the camera image contains approximately 6 - 12 pixels per inch, and features at the top of the image move upward between consecutive images faster than the mobile robot moves, and features at the bottom of the image move downward between consecutive images slower than the mobile robot moves, and wherein the controller is configured to determine the speed of the mobile robot and the position of features in the image in identifying the parallax between consecutive images.

[0013] In certain embodiments, the mobile robot moves at a speed of 220 mm per second to 450 mm per second, and features below 45 degrees relative to the horizontal direction will be tracked more slowly than approximately 306 mm per second, and features above 45 degrees will be tracked faster than 306 mm per second.

[0014] In some embodiments, the optics define an f - number between 1.8 and 2.0.

[0015] In some embodiments, the optical device defines a focal length of at least 40 cm.

[0016] In some embodiments, the body further includes: a memory in communication with the controller circuit; and an odometry sensor system in communication with the driver, wherein the memory further includes a visual measurement application, a simultaneous localization and mapping (SLAM) application, a landmark database, and a map of landmarks, and wherein the controller circuit guides the processor to: actuate the driver and capture odometry data using the odometry sensor system; obtain visual measurements by providing at least the captured odometry information and the captured images to the visual measurement application; determine an updated robot pose within the updated map of landmarks by providing at least the odometry information and the visual measurements as inputs to the SLAM application; and determine robot behavior based on inputs including the updated robot pose within the updated map of landmarks.

[0017] In some embodiments, the landmark database includes: descriptions of a plurality of landmarks; landmark images of each of the plurality of landmarks and associated landmark poses from which the landmark images were captured; and descriptions of a plurality of features associated with a given landmark from the plurality of landmarks, the description including the 3D positions of each of the plurality of features associated with the given landmark.

[0018] In certain embodiments, the visual measurement application guides the processor to: identify features within the input image; identify landmarks from the landmark database within the input image based on the similarity of the features identified in the input image to matching features associated with the landmark images of the landmarks identified in the landmark database; and estimate the most likely relative pose by determining a rigid transformation of the 3D structure of the matching features associated with the identified landmark that results in the highest degree of similarity to the features identified in the input image, wherein the rigid transformation is determined based on the estimated relative pose and an acute angle at which the optical axis of the camera is aligned above the direction of motion of the mobile robot.

[0019] In some embodiments, identifying a landmark in the input image includes comparing an uncorrected patch from the input image and a landmark image within the landmark database.

[0020] In some embodiments, the SLAM application guides the processor to: estimate the position of the mobile robot within the map of landmarks based on a previous position estimate, odometry data, and at least one visual measurement; and update the map of landmarks based on the estimated position of the mobile robot, odometry data, and at least one visual measurement.

[0021] In some embodiments, the controller circuit further directs the processor to: actuate the drive to translate the robot towards the landmark identified in the previous input frame; and the visual measurement application further instructs the processor to search for the features of the landmark identified in the previous input image at a location above the location where the features were identified in the previous input image.

[0022] In some embodiments, the visual measurement application further directs the processor to generate a new landmark by: detecting features within an image in a sequence of images; identifying from the sequence of images a set of features that form a landmark in multiple images; estimating the 3D structure and relative robot pose of the set of features that form the landmark when capturing each of the multiple images using the set of identified features that form the landmark in each of the multiple images; recording the new landmark in a landmark database, where recording the new landmark includes storing: an image of the new landmark, at least the set of features that form the new landmark, and the 3D structure of the set of features that form the new landmark; and notifying the SLAM application of the creation of the new landmark.

[0023] In some embodiments, the SLAM application directs the processor to: determine the landmark pose as the pose of the mobile robot when capturing an image of the new landmark stored in the landmark database; and record the landmark pose for the new landmark in the landmark database.

[0024] In some embodiments, the visual measurement application further directs the processor to: estimate the 3D structure of the set of features that form the landmark and the relative robot pose by minimizing the reprojection error of the 3D structure of the set of features that form the landmark onto each of the multiple images of the relative robot pose used for estimation when capturing each of the multiple images.

[0025] In some embodiments, the visual measurement application further directs the processor to: identify at least one landmark in multiple images from the image sequence by comparing uncorrected image patches from the images in the image sequence.

[0026] In some embodiments, the at least one processor is a single processor directed by the visual measurement application, the behavior controller application, and the SLAM application.

[0027] In some embodiments, the at least one processor includes at least two processors, where the behavior controller application directs the first of the at least two processors, and the virtual measurement application and the SLAM application direct the other of the at least two processors.

[0028] In some embodiments, the machine vision sensor system further includes a plurality of cameras below the top surface of the robot and having a focal axis angled upward relative to the horizontal plane of the top surface, and where at least one camera faces a reverse driving direction opposite to the forward driving direction.

[0029] In some embodiments, the machine vision sensor system includes a second camera, the second camera including optics that define a camera field of view and a camera optical axis; and the second camera is positioned such that the optical axis of the optics of the second camera is aligned at an angle above the direction of motion.

[0030] In some embodiments, the machine vision system includes a stereo camera pair having overlapping fields of view.

[0031] In some embodiments, the camera is positioned within a recessed structure at a distance of up to 6 inches from the floor surface.

[0032] In some embodiments, the camera field of view is aimed at static features located in the range of 3 to 8 feet from the floor surface on a planar wall.

[0033] In some embodiments, the camera image includes approximately 6 - 12 pixels per inch.

[0034] In certain embodiments, the camera is a 320×420 VGA, 3 - megapixel camera without an IR filter.

[0035] Some embodiments of the invention provide a mobile robot configured to navigate an operating environment, including a body that includes a top surface located at a height of no more than 6 inches from the bottom surface, the body including a recessed structure below the top surface, the body comprising: at least one processor; a memory containing a behavior controller application: wherein the behavior controller application guides the processor to navigate the environment based on captured images; a machine vision sensor system including a camera configured to capture images of the operating environment of the mobile robot, the camera including optics that define a camera field of view and a camera optical axis, wherein the camera is positioned within the recessed structure and is tilted such that the optical axis is aligned at an acute angle between 10 and 60 degrees above the forward driving motion direction of the mobile robot.

[0036] In some embodiments, the body further includes: a lens cover that protects the camera, wherein the lens cover is retracted rearward relative to the opening of the recessed structure and forms an acute angle with respect to the optical axis of the camera, the acute angle being closer to a right angle than the angle formed between the plane defined by the opening in the recessed structure and the optical axis of the camera.

[0037] In some embodiments, the camera is a 320×420 VGA, 3 - megapixel camera without an IR filter. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1A Front perspective view of the illustrated mobile robot.

[0039] Figure 1B Illustration of the bottom of the mobile robot.

[0040] Figure 1C Block diagram of the controller circuit of the illustrated robot.

[0041] Figure 2A Top perspective view of a mobile robot showing an inclined camera included within a recessed structure.

[0042] Figure 2B Cross-sectional view of the inclined camera of the illustrated mobile robot.

[0043] Figures 3A - 3C Illustration of a lens holder that prevents lens distortion that may occur in a camera lens due to transmission of mechanical stress from the mobile robot to the lens.

[0044] Figure 4 Illustration of a mobile robot configured with cameras that tilt forward and backward, which are contained within separate recesses in the top of the body of the mobile robot and are protected using lens caps.

[0045] Figure 5 Illustration of Figure 4 Cross-sectional view of the forward- and backward-tilting cameras of a mobile robot configured in the manner illustrated in

[0046] Figure 6 Illustration of a mobile robot controller circuit that can be used for VSLAM using an enhanced navigation system.

[0047] Figure 7 Conceptual illustration of a mobile robot behavior control application configured to enable navigation within an environment based on (but not limited to) a VSLAM process.

[0048] Figure 8 Is a flowchart illustrating a process that can optionally be used by a mobile robot to navigate an environment.

[0049] Figure 9 Is a flowchart illustrating a process that can optionally be used by a mobile robot to identify new landmarks for use in navigation.

[0050] Figure 10 Is a flowchart illustrating a process that can optionally be used by a mobile robot to determine a relative pose using previously created landmarks.

[0051] Figures 11A - 11C Illustration of an example of a field of view captured by a mobile robot having a camera configured such that the optical axis is aligned with the forward direction of movement of the mobile robot.

[0052] Figures 12A - 12CIllustration of an example view of a scene captured by a camera that is tilted such that its optical axis forms an acute angle above the direction of motion of a mobile robot.

[0053] Figure 13A Illustration of the parallax that can be used to determine the distance to a feature.

[0054] Figure 13B Illustration of an example of the upward movement of a feature for which parallax is determined.

[0055] Figure 14A Illustration of an image generated by simulating the viewpoint of a forward-facing camera of a simulated mobile robot in a simulated virtual indoor environment.

[0056] Figure 14B Illustration of an image generated by simulating the viewpoint of a camera that has the same field of view as the camera used in the simulation for generating the image shown in Figure 14A (except that the camera is tilted such that the optical axis of the camera forms an angle of 30 degrees above the forward motion direction of the simulated mobile robot).

[0057] Figure 15 Is a graph showing a comparison of the accuracies with which a mobile robot can determine its relative pose in the floor plane in each of the above simulations.

[0058] Figure 16A Illustration of the sampling grid of a camera of a mobile robot using a wide-angle lens configuration.

[0059] Figure 16B Illustration of a sampled scene using a tilted camera with a narrow field of view.

[0060] Figure 17A Illustration of an example of a specific type of occlusion and the resulting image captured by the occluded camera.

[0061] Figure 17B Illustration of an example of an image captured by a camera in which a translucent occlusion results in a blurred portion of the image.

[0062] Figure 18A Illustration of an example of an opaque occlusion.

[0063] Figure 18B Illustration of an example of an image captured by a camera in which an opaque occlusion results in a complete occlusion of a portion of the scene.

[0064] Figure 19 Is a flowchart illustrating an occlusion detection process that can optionally be performed by a mobile robot.

[0065] Figure 20 Illustration of a communication diagram that illustrates the communication between a mobile robot, an external server, and a user device.

[0066] Figure 21 A system for notifying a user device of an occlusion is illustrated.

[0067] Figure 22 A robot is illustrated that has a top surface no more than 4 inches from the floor surface, a camera mounted below the top surface of the mobile robot; the robot has a field of view across a frustum that has a field of view angle δ of approximately 50 - 60 degrees in the vertical direction and an optical axis 155 that is at an acute angle φ of approximately 30 degrees above the horizontal direction. DETAILED DESCRIPTION

[0068] Turning now to the drawings, and particularly Figures 1A - 1C FIGS. 4 and 5, systems and methods for performing visual simultaneous localization and mapping (VSLAM) using a machine vision system of a mobile robot 10 that includes one or more tilted cameras 125 are illustrated. In an implementation, one or more tilted cameras 125 are recessed into a central portion of the body 100 of the mobile robot 10 at a fixed angle relative to the top surface 108 of the robot, and each of the one or more tilted cameras 125 is protected by a lens cover 135 that is aligned at an acute angle with respect to the optical axis 155 of the camera. In the case where the camera 125 is recessed into a portion of the body 100 of the mobile robot 10, the lens cover 135 may also be set back relative to the opening of the recessing 130. In an implementation, the navigation system 120 is part of the mobile robot 10, which is a house cleaning robot that is no more than 4 inches from the floor surface to the top surface 108 of the robot. In this way, the house cleaning robot 10 can navigate into tight spaces (e.g., under chairs and under the face frames of kitchen cabinets). In several implementations, the mobile robot 10 has a diameter of 13.9 inches and weighs 8.4 pounds and moves at a speed of 220 mm per second to 450 mm per second. In yet another alternative aspect of the invention, one or more tilted cameras 125 are tilted such that the house cleaning robot 10 can capture images of reliable static feature-rich objects (such as picture frames hanging on the walls of a home and other features that do not move). As discussed further below, the mobile robot 10 can use the features of reliable static objects located within a specific height range above the floor to build a map of the environment and use vision-based sensors and vision-based simultaneous localization and mapping (or VSLAM) to navigate.

[0069] The combination of SLAM and visual sensors is often referred to as VSLAM 740. The VSLAM 740 process is typically vision- and odometry-based and enables reliable navigation in feature-rich environments. Such vision technologies can be used by vehicles such as mobile robot 10 to autonomously navigate an environment using a continuously updated self-generated map. A variety of machine vision systems have been proposed for use in VSLAM 740, including machine vision systems incorporating one or more cameras 125.

[0070] The systems and methods according to embodiments of the invention use camera 125 to perform VSLAM 740, the camera 125 being mounted below the top surface 108 of mobile robot 10 and having an optical axis 155 aligned at an acute angle above the top surface 108 of mobile robot 10. Many VSLAM 740 processes analyze the parallax of features captured in a series of images in order to estimate the distance to the features and / or triangulate a position. Generally, the amount of parallax for a set of feature observations between a series of images that have been captured from different vantage points determines the accuracy with which those features are mapped within the environment. The greater the observed parallax, the more accurate the distance measurement.

[0071] When mobile robot 10 employs a forward-facing camera 125 having an optical axis 155 with a lens 140 aligned parallel to the forward direction of motion, there is generally only a minimal amount of parallax, which can be observed in a set of features directly positioned in front of mobile robot 10 as the robot moves towards the features. As discussed further below, as mobile robot 10 moves towards a feature, the features visible at the center of the forward-facing camera's field of view may increase in scale, with little distinguishable parallax between successive images of the features. Accordingly, the 3D structure of features within the center of the forward-facing camera's field of view may be difficult to determine from a series of images captured as mobile robot 10 moves forward towards the features. This problem is particularly acute for small mobile robots such as house cleaning robots, where the robot form factor dictates a camera placement close to the ground (e.g., 4 inches or less above the ground). The positional accuracy of the forward-facing camera can be improved by increasing the field of view of the forward-facing camera. However, increasing the field of view reduces the angular resolution of the image data captured for a given image sensor resolution. Additionally, increasing the field of view enables the observation of parallax with respect to the peripheral portion of the camera's field of view or the off-axis field of view of the camera, where the distortion of a wide-angle lens is typically greatest and the angular resolution of the images captured by the camera is lowest.

[0072] Tilting the camera at an angle above the forward motion direction can increase the parallax observed across the camera's field of view as the camera moves towards an object. In particular, tilting the camera at an angle above the forward motion direction increases the parallax observed within the center of the field of view, where the portion of the camera's field of view has the highest angular resolution. Features observable within the center of the camera's field of view that are tilted such that their optical axis 155 forms an acute angle with the horizontal axis aligned with the forward motion direction will move upward in a series of images captured as the camera moves towards the feature. Accordingly, objects such as (but not limited to) picture frames and televisions that are frequently hung on the walls of a residence and have features that are easily distinguishable at any scale and in various lighting conditions provide excellent navigation landmarks for a mobile robot such as a house cleaning robot operating in a residential environment.

[0073] Tilting the camera 125 upward can also enable the mobile robot 10 to more precisely determine the 3D structure of the lower side of an object hanging on a wall. Additionally, tilting the camera 125 allows the mobile robot 10 to focus on areas within a typical indoor environment where features are invariant (such as those imaged around door frames, picture frames, and other static furniture and objects), allowing the mobile robot 100 to repeatedly identify reliable landmarks, thereby precisely positioning and mapping within the environment. In addition, in an implementation, the camera 125 on the mobile robot 10 is a 320×240 QVGA, 0.0768 mP camera (or 640×480 VGS, 0.3 MP camera) that transmits images at a rate of less than 30 milliseconds and with an image processing rate of 3 frames per second. In an implementation, the camera 125 does not have an IR filter for better feature detection in low-light environments. In an implementation, if the number of detectable features falls below a threshold for the minimum number of features for detecting landmarks, the mobile robot 10 will create a new landmark. In an embodiment, the threshold number of landmarks is a cluster of identifiable features detected at a rate of approximately 1 ft per second or approximately 306 mm per second, at a rate of 1 - 10 landmarks per foot of travel (and preferably 3 landmarks per foot of robot travel). In an implementation, if the environment is too dark and the lighting is insufficient for feature detection, the mobile robot 10 will rely on another sensor for positioning, such as an optical dead reckoning sensor aimed at the floor (e.g., an LED or laser-illuminated mouse sensor).

[0074] Parallax measurements for features visible within the center of the field of view of camera 125 can be made more precisely by sacrificing field of view for increased angular resolution. In several embodiments, by utilizing a tilt camera 125 employing a view lens 140 with a horizontal field of view of, for example, 78 degrees, the angular resolution achieved using a specific sensor resolution is increased relative to a forward-facing camera employing a wide-angle lens. A typical difference between a narrow field of view lens 140 and a wide-angle lens is that perspective projection is generally a good approximation of the true imaging characteristics of a narrow field of view lens, whereas a wide-angle lens introduces distortion. Accordingly, the angular resolution and modulation transfer function (MTF) of a lens tend to be more uniform over the field of view of a narrow field of view lens when compared to the variations in angular resolution and MTF experienced across the field of view of a wide-angle lens. Distortion introduced by a wide-angle lens can be corrected using computationally expensive distortion correction operations, and parallax can be determined after the image captured by the camera is corrected. When utilizing a narrow field of view camera 125, the uncorrected image can be utilized during subsequent image processing. Accordingly, using a narrow field of view lens in conjunction with an appropriate VSLAM process can provide advantages relative to more conventional VSLAM processes using a wide field of view lens, as they avoid performing additional computationally expensive correction steps and can localize features and / or landmarks (i.e., a set of features) with higher precision. The following refers to Figure 16A Examples of implementations for capturing an environment using a narrow field of view lens are described in detail.

[0075] Many VSLAM processes rely on having a wide baseline between observed features to perform localization. As discussed further below, an alternative approach involves identifying correlated features with distinctive 3D structures as landmarks. Based on the 3D structure of the features forming the landmarks and the appearance of the features in the images captured by the mobile robot 10, the relative pose of the mobile robot 10 can be readily determined. Accordingly, the accuracy of relative pose estimation depends largely on the angular accuracy with which the distances between features observed within the images captured by the camera 125 of the mobile robot 10 can be measured. Accordingly, configuring the mobile robot 10 with a tilt camera 125 having a narrow field of view lens can achieve improved performance over equivalent systems equipped with a forward-facing camera or a camera with a wider field of view and the same resolution. The following refers to Figure 1A and 2A -B for a detailed description of the implementation using the tilt camera 125.

[0076] In many embodiments, the field of view of the camera 125 of the mobile robot 10 and the specific angle formed between the optical axis 155 of the camera and the forward movement direction are determined based on dedicated requirements, which include (but are not limited to) the frame rate of the camera 125, the speed of the mobile robot 10, and the processing capabilities of one or more processors for performing image processing within the mobile robot 10. As can be easily understood, the greater the speed of the mobile robot 10, the greater the observed parallax between the captured frames. The parallax is observed by overlapping the FOV of the frames, and the mobile robot 10 moves at a speed between 220 mm per second and 450 mm per second, and preferably at a speed of 306 mm per second. Increasing the tilt of the camera 125 can further increase the parallax. The ability to observe the parallax depends on the rate at which the mobile robot 10 can capture and process frames of image data in real time. A lower frame rate can be compensated for by a larger field of view. However, as noted above, increasing the field of view may incur a computational cost for correcting the captured images. Thus, in many alternative configurations of the mobile robot 10, the tilted camera 125 and the field of view are selected to meet the requirements of the specific mobile robot and its operating environment.

[0077] By adding additional cameras 125, 410 positioned at different locations around the mobile robot 10, the effective field of view of the navigation system 120 can be increased without reducing the angular resolution. In an alternative aspect of the invention, each camera 125, 410 is housed within a separate recess 130 in the body 100 of the mobile robot 10 having a separate protective lens cover 135. Certain embodiments may include both a forward-facing tilted camera 125 and one or more rear-facing tilted cameras 410. Referring below to Figure 4 and 5Describe in detail the implementation using multiple cameras. The use of multiple cameras 125, 410 may allow the mobile robot 10 to observe more features and / or landmarks for use in VSLAM while maintaining the angular accuracy with which it detects these features. Easily identifying landmarks in one or more directions around the robot 10 helps with rapid re-localization to resume the cleaning task and continue building a map of the environment after a kidnapping event in which the robot 10 has been moved or lost. Additionally, the rearward-tilted camera 410 may allow the VLSAM process to use the same landmarks regardless of the direction of motion. This may be useful in a mobile robot 10 navigating an environment in a "corn-row" pattern. In this case, whenever the mobile robot 10 turns, it can use features and / or landmarks visible to the forward-facing camera 125 to navigate on the return path using the image data captured by the rearward-tilted camera 410. Additionally, given the increased spatial resolution of a narrower field-of-view lens (assuming comparable sensor resolution), the rearward-tilted camera 410 can detect landmarks with a higher angular accuracy than in the case where the mobile robot 10 uses, for example, a 360-degree omnidirectional camera.

[0078] During operation, one or more cameras 125 mounted on the mobile robot 10 may become blocked for any of a variety of reasons. For example, over time and with the use of the mobile robot 10, dust and debris may accumulate on the camera lens 140 or the lens cap 135, and thereby obscure portions of the image being captured. The mobile robot 10 is capable of detecting when a certain type of obstruction is obscuring one or more portions of the camera lens 140. When an obstruction is detected, the mobile robot 10 may provide a notification, such as notifying the user to clean the lens cap 135 protecting the camera lens 140 to remove the obstruction. To detect the presence of an obstruction blocking a portion of the field of view of the camera 125, some embodiments analyze specific portions of the image that provide useful information for the VSLAM process and, based on that analysis, are able to determine certain other portions that may be obscuring the field of view. In particular, some embodiments may maintain histograms of different portions of the field of view and the frequency with which each portion is capturing image data that is being used to generate new landmarks and / or identify existing landmarks in the environment during navigation of the mobile robot 10 using VSLAM. Areas used in conjunction with low frequencies may be flagged as obstructed and a notification generated accordingly.

[0079] While most of the following discussion describes camera configurations used in conjunction with a specific VSLAM process, the techniques disclosed herein may be utilized by a mobile robot 10 configured with any of a variety of different mapping and navigation mechanisms. Accordingly, various alternative configurations of a mobile robot 10 incorporating one or more tilted cameras 125 for use in navigating an environment are further discussed below.

[0080] Mobile Robot with Enhanced Vision Sensor Configuration

[0081] Mobile robot 10 incorporates a navigation system 120 that includes a camera 125, which can capture image data used by a VSLAM process in the navigation of mobile robot 10 and mapping of the environment surrounding mobile robot 10. Figures 1A - 2B Illustrated in is the tilt camera 125 used in the navigation system 120 of mobile robot 10. In particular, FIG. 1 illustrates a front perspective view of mobile robot 10, and FIG. 2 illustrates a recess 130 substantially disposed in the middle of the body 100 of mobile robot 10 that houses the tilt camera 125. As can be readily understood, the camera 125 is protected by a lens cap 135 that is configured such that one edge of the lens cap 135 is closer to the opening of the recess 130 than a second edge of the lens cap 135. In this way, the lens cap 135 forms an acute angle α with the optical axis 155 of the camera 125, and the acute angle α is closer to a right angle than the angle φ formed between a plane defined by the opening in the recessed structure 130 within the body 100 of mobile robot 10 and the optical axis 155 of the camera 125. In some embodiments, the acute angle α can range between 15 and 70 degrees, and the acute angle φ can range between 10 and 60 degrees. Angling the lens cap 135 in this way is an optional aspect of the invention, and the lens cap 135 can be implemented in or recessed into the plane of the opening in the recess 130 of the body 100 of the mobile robot that houses the tilt camera 125, but parallel to the plane of the opening in the recess 130 of the body 100 of the mobile robot. The configuration of the tilt camera 125 within the recess 130 in the body 100 of mobile robot 10 is further discussed below.

[0082] As shown of the bottom of robot 10 Figure 1B and a block diagram depicting the controller circuit 605 of robot 10 and the systems of robot 10 that can operate with the controller circuit 605 Figure 1C As shown in, mobile robot 10 includes a body 100 supported by a drive 111 located beneath the body 100, and the drive 111 includes a left drive wheel module 111a and a right drive wheel module 111b that can maneuver robot 10 across a floor surface. In an implementation, the drive is the drive of the robot described in U.S. Patent Application Publication No. 2012 / 0317744, the entire content of which is incorporated herein by reference. In many implementations, mobile robot 10 is no more than 4 inches above the floor surface to allow mobile robot 10 to navigate through tight spaces within a typical indoor environment.

[0083] The mobile robot 10 can be configured to actuate its drive 111 based on drive commands. In some embodiments, the drive commands can have x, y, and θ components, and the commands can be issued by the controller circuit 605. The mobile robot body 100 can have a forward portion 105 corresponding to the front half of the shaped body, and a rearward portion 110 corresponding to the rear half of the shaped body. The drive includes a right drive wheel module 111a and a left drive wheel module 111b that can provide an odometer to the controller circuit 605. The wheel modules 111a, 111b are arranged substantially along the transverse axis X defined by the body 100, and include respective drive motors 112a, 112b for driving the respective wheels 113a, 113b. The drive motors 112a, 112b can be releasably connected to the body 100 (e.g., via fasteners or tool-free connections), where the drive motors 112a, 112b are optionally positioned substantially above the respective wheels 113a, 113b. The wheel modules 111a, 111b can be releasably attached to the chassis and forced into engagement with the cleaning surface by springs. The mobile robot 10 can include casters 116 that are arranged to support a portion of the mobile robot body 100, here the front portion of the circular body 100. In other implementations with a cantilever cleaning head (such as a square-fronted or tombstone-shaped robot body 100), the casters are arranged in the rearward portion of the robot body 100. The mobile robot body 100 supports a power source (e.g., battery 117) for powering any electrical components of the mobile robot 10.

[0084] Referring again to Figure 1A and 1B , the mobile robot 10 can move across the cleaning surface by various combinations of movements relative to three mutually perpendicular axes (the transverse axis X, the longitudinal axis Y, and the central vertical axis Z) defined by the body 100. The forward movement direction along the longitudinal axis Y is designated as F (sometimes referred to hereinafter as "forward"), and the rearward drive direction along the longitudinal axis Y is designated as A (sometimes referred to hereinafter as "backward"). The transverse axis X extends substantially along the axis defined by the center points of the wheel modules between the right side R and the left side L of the robot.

[0085] In many embodiments, the forward portion 105 of the body 100 carries a bumper 115 that can be used to detect (e.g., via one or more sensors of the bumper sensor system 550) events that include, but are not limited to, obstacles in the drive path of the mobile robot 10. Depending on the behavioral programming of the mobile robot 10, the behavioral programming of the mobile robot 10 can dispatch the robot 10 in response to events (e.g., backing away from a detected obstacle) by controlling the wheel modules 111a, 111b to respond to events (e.g., obstacles, cliffs, walls) detected by the bumper 115, the cliff sensors 119a - 119f, and one or more proximity sensors 120a - 120n.

[0086] As illustrated, the user interface 126 is disposed on the top portion of the body 100 and can be used to receive one or more user commands and / or display the status of the mobile robot 10. The user interface 126 communicates with the controller circuit 605 carried by the robot 10 such that one or more commands received by the user interface 126 can initiate a cleaning routine to be performed by the robot 10.

[0087] The mobile robot 10 may also include a camera 125 and a navigation system 120 embedded within the body 100 of the robot 10 under the top cover 108. The navigation system 120 may include one or more cameras 125 (e.g., standard cameras, volumetric point cloud imaging cameras, three - dimensional (3D) imaging cameras, cameras with depth map sensors, visible light cameras, and / or infrared cameras) that capture images of the surrounding environment. In an alternative configuration, the camera 125 captures images of the environment positioned at an acute angle relative to the axis of motion (e.g., F or A) of the mobile robot 10. For example, as Figure 22 illustrated, in an implementation of the robot 10 having a top surface 108 no more than 4 inches from the floor surface, the camera 125 mounted below the top surface 108 of the robot 10 will detect features in the environment at a height generally between 3 - 14 feet. The camera 125 has a field of view in the shape of a frustum that spans approximately 50 - 60 degrees in the vertical direction and an optical axis 155 that is angled approximately 30 degrees above the horizontal direction. For example, a mobile robot 10 of these dimensions with these camera settings will view objects at a distance of 3 feet, at a height of approximately 6 inches to 4.5 feet; at a distance of 5 feet, at a height of approximately 9 inches to 7.5 feet; and at a distance of 10 feet, at a height of approximately 1.2 feet to 14 feet. By focusing the undistorted central portion of the camera 125's field of view on areas where features remain constant (such as those imaged around door frames, picture frames, and other static furniture and objects), the robot 10 repeatedly identifies reliable landmarks, thereby enabling precise positioning and mapping within the environment.

[0088] In these embodiments, the lens 140 of camera 125 ( Figure 2A and 2B ) is angled in an upward direction such that it primarily captures an image having a reliable undistorted portion that is focused on a feature-rich invariant region of a wall, a wall-ceiling intersection, and a portion of the ceiling surrounding the mobile robot 10 in a typical indoor environment. As noted above, many environments, including (but not limited to) residential accommodation environments, include varying and static objects hanging and clustered on walls that provide features and / or landmarks useful for performing navigation. Objects (such as door and window frames, pictures, and large furniture) typically available in an area of about 2 - 15 feet (e.g., 3 - 10 feet, 3 - 9 feet, 3 - 8 feet) have little displacement and thus provide a feature geometry for reliable imaging that results in the creation of landmarks that lead to more accurate and reliable identification. By aggregating reliable features in a region of a narrow field of view, landmark determination is improved and thus localization determination is improved. By aggregating varying and distinct features imaged from items with no displacement, the mobile robot 10 builds a map of reliable landmarks.

[0089] The camera 125 can be optionally tilted such that the lower periphery of the field of view of the camera 125 is not blocked by the body 100 of the mobile robot 10. Alternatively, in an implementation, the body 100 of the mobile robot 10 partially blocks the lower portion of the field of view of the tilted camera 125, and when imaging features, the controller circuit 605 discards that portion of the field of view. As noted above, as the mobile robot 10 moves through the environment, the tilted camera 125 can increase the amount of parallax observed across the field of view of the camera 125. In an implementation, the mobile robot 10 employs a tilted camera 125 with optics having a narrow enough field of view such that perspective projection can be assumed to be a good approximation of the true imaging characteristics of the narrow field of view lens. Subsequent image processing can be performed without correcting the image, and the camera 125 can observe features with a higher angular resolution than a wider-angle lens that would also introduce distortion.

[0090] The images captured by the camera 125 can be used by the VSLAM process in order to make intelligent decisions regarding the actions to be taken to dispatch the mobile robot 10 around the environment operation. Although the camera 125 of the navigation system 120 is illustrated in Figures 1A - 1C and 2A - 2B as being included within a centrally located recess 130 below the top surface 108 of the mobile robot 10, the camera 125 forming part of the navigation system 120 of the mobile robot 10 can additionally or alternatively be arranged at any one of one or more locations and orientations on the mobile robot 10 (including on or within the front bumper and along the sides of the mobile robot).

[0091] In addition to the camera 125 of the navigation system 120, the mobile robot 10 may include different types of sensor systems 500 to enable reliable and robust autonomous movement. The additional sensor systems 500 may be used in combination with each other to create a perception of the environment of the mobile robot 10 that is sufficient to allow the robot to make intelligent decisions about the actions to be taken in that environment. The various sensor systems may include one or more types of sensors supported by the robot body 100, including but not limited to obstacle detection obstacle avoidance (ODOA) sensors, communication sensors, navigation sensors, rangefinder sensors, proximity sensors, contact sensors (such as bumper sensors), sonar, radar, LIDAR (light detection and ranging, which may require optical remote sensing of the properties of scattered light to find the range and / or other information of distant targets) and / or LADAR (laser detection and ranging). In some implementations, the sensor system includes a rangefinder sonar sensor, a proximity cliff detector 119a-119f, proximity sensors 120a-120n (e.g., "n" is an infinite number in an array of proximity sensors on the sidewalls of the mobile robot 10), contact sensors in the bumper sensor system 550, a laser scanner, and / or an imaging sonar.

[0092] There are several challenges involved in placing sensors on a robotics platform. First, sensors are typically placed such that they have maximum coverage of the area of interest around the mobile robot 10. Second, sensors are typically placed in such a way that the robot itself causes an absolute minimum of occlusion of the sensors; in essence, sensors should not be placed such that they are obscured by the robot itself. Third, the placement and installation of the sensors should not intrude on the rest of the industrial design of the platform. In terms of aesthetics, it can be assumed that a robot with sensors that are not visibly mounted is more appealing than otherwise. In terms of functionality, the sensors should be mounted in a way so as not to interfere with normal robot operation (e.g., creating an obstruction on an obstacle).

[0093] Additional options that can be employed in the implementation of the navigation system 120 of the mobile robot 10 are discussed further below.

[0094] Machine vision system

[0095] To navigate through the environment, the mobile robot 10 may use information collected from various different types of sensors to determine the characteristics of its surrounding environment. As noted above, the mobile robot 10 uses a navigation system 120 that includes one or more cameras 125 that capture images of the surrounding environment. The images may be provided to a VSLAM process for use in localizing and mapping the mobile robot 10 within the environment. Figure 2A Illustrates a top perspective view, and Figure 2B Illustrates atFigure 1A Cross-sectional view of the tilt camera 125 of the mobile robot 10 shown in the figure. In particular, Figure 2A (and the corresponding Figure 2B ) shows the tilt camera 125 housed within the recessed structure 130 in the body 100 of the mobile robot 10 and covered by the lens cap 135. As Figure 2B and 22 depicted, the camera 125 includes a camera lens 140 having an optical axis 155 that forms an acute angle φ with respect to the horizontal axis defining the direction of movement of the mobile robot 10. Thus, the lens 140 is primarily aimed in a direction that will capture walls, wall-ceiling intersections, and to a lesser extent, ceilings within a typical indoor environment. For example, as Figure 22 shown in the figure, in an implementation of the robot 10 having a top surface 108 no more than 4 inches from the floor surface 2205, the camera 125 mounted below the top surface 108 of the mobile robot 10 will detect features 2215 in the environment at a height generally between 3 - 14 feet. The camera 125 has a field of view across a frustum and an optical axis 155 that has a field of view angle δ of approximately 50 - 60 degrees in the vertical direction, and the optical axis 155 forms an acute angle φ of approximately 30 degrees above the horizontal direction. For example, a mobile robot 10 of these dimensions with these camera settings will see an object at a distance Dw of 3 feet, at a height of approximately 6 inches to 4.5 feet; at a distance Dw of 5 feet, at a height of approximately 9 inches to 7.5 feet; and at a distance Dw of 10 feet, at a height of approximately 1.2 feet to 14 feet. By focusing the undistorted field of view of the tilt camera 125 on feature-rich areas where the features are invariant (such as those imaged at door frames, picture frames 2215, and other static furniture and objects with easily imaged feature geometries), the robot 10 repeatedly identifies reliable landmarks, thereby precisely and effectively localizing and mapping within the environment. This is particularly useful for repositioning after an abduction event that interrupts the robot 10's task. A robot 10 that has been moved from one location to another or into a dark room or under a furniture area quickly identifies the unique set of features at these heights and easily and precisely knows its location. A camera aimed at an invariant ceiling or a ceiling with repetitive features (such as a dropped ceiling with tiles or a ceiling with evenly spaced lighting features) will not easily know its location and will have to find the unique features on which to localize, such as the corners of the room. The mobile robot 10 of the present invention is thus effective and precisely picks up its cleaning routine at useful locations without having to randomly move to areas where landmarks are identifiable on the map.

[0096] In Figure 22In the implementation, the field of view of camera 125 projects onto a vertical planar surface such as a home wall at an upper angle β of 35 degrees and a lower angle ψ of 95 degrees, and the optical axis 155 intersects the vertical planar surface at a lower angle τ of 60 degrees. The closer a wall feature 1 is to camera 125 with a smaller field of view, the more useful it becomes. For example, in the implementation, robot 10 includes a tilted camera 125 such as the one just described above that aims at wall 2210 and has a field of view of 50 degrees, which is equivalent to 3 times the resolution of a directly forward-looking camera (e.g., at a distance Dw of 8 feet from the wall, for features directly forward (about 12 pixels per inch), and for features 8 feet tall (about 6 pixels per inch)). The feature with the highest resolution will be the lowest feature on the wall, and thus camera 125 is aimed so as not to miss those features. Figure 22 The configuration of camera 125 creates more trackable optical flow, but maintains a high pixel count by having at least half of the field of view greater than 45 degrees. A feature at 45 degrees to robot 10 in the field of view of camera 125 will instantaneously track the exact middle of the field of view of camera 125 at the same speed as mobile robot 10. In an embodiment, mobile robot 10 moves at a speed between 220 mm per second and 450 mm per second, and preferably at a speed of about 306 mm per second or 1 ft per second. As mobile robot 10 moves towards wall 2210, all objects will increase in height in the field of view. Features below 45 degrees will track more slowly and accelerate upward and laterally compared to features at 45 degrees, and features above 45 degrees will track faster and accelerate more quickly compared to features at 45 degrees. In the implementation, features below 45 degrees will track more slowly than about 1 ft per second, and features above 45 degrees will track faster than 1 ft per second. In the implementation, the first set of edge pixels tracks vertically at a rate that appears faster than the self-motion of mobile robot 10, and the second set of edge pixels tracks vertically at a rate that appears equal to or slower than the robot's self-motion. In the implementation, camera 125 is arranged to be less than 6 inches from the floor, where the 50-degree field of view is oriented to span a 50-degree range defined between -10 degrees and 90 degrees above the horizontal direction (e.g., extending from 30 - 80 degrees), and camera 125 is oriented to be limited to viewing the forward wall portion of mobile robot 10 at a height between 3 and 8 feet, as long as mobile robot 10 is at a distance Dw between 3 - 10 feet from wall 2210.

[0097] Return to Figure 2A and 2BIn the implementation method, the lens cover 135 is also pressed into the recessed structure 130 and is positioned below the top surface 108 of the mobile robot 10. Additionally, the lens cover 135 is aligned at an acute angle α with respect to the optical axis 155 of the camera, and this acute angle α is greater than the acute angle φ between the plane forming the opening of the recessed structure 130 and the optical axis 155 of the camera 125. In an embodiment, the acute angle α ranges between 15 and 70 degrees, and the acute angle φ ranges between 10 and 60 degrees. Angling the lens cover 125 with respect to the optical axis 155 prevents unwanted imaging problems, such as light reflection and / or refraction that may prevent the camera 125 from effectively imaging features.

[0098] As noted above, the mobile robot 10 may optionally include a narrow - field - of - view lens 140 that provides an image, where perspective projection can be assumed to be a good approximation of the true imaging characteristics of the narrow - field - of - view lens. In the case where the narrow - field - of - view lens 140 is utilized by the mobile robot 10, the transfer of mechanical stress from the mobile robot 10 to the lens 140 can distort the lens by introducing complex distortion correction processing as part of the image - processing pipeline, eliminating some of the benefits of using the narrow - field - of - view lens 140. The design of the lens holder can play an important role in preventing the transfer of mechanical stress from the mobile robot 10 to the lens 140 and avoiding distortion of the lens 140. The implementation of the lens holder that can be optionally utilized in one or more cameras 125 of the mobile robot 10 is further discussed below.

[0099] Lens holder

[0100] Figures 3A - 3C Depicts the implementation of a lens holder 310 that prevents lens deformation, which may occur in the camera lens 140 due to the transfer of mechanical stress from the mobile robot 10 to the lens 140. Figure 3A Illustrates the lens holder 310 for holding the camera lens 140. The lens holder 310 is connected to a set of screw bosses 320 supported by a set of star ribs 330. Figure 3B Illustrates a side view of the lens holder 310. In particular, this figure illustrates that the bottom of the screw boss 320 is positioned at a distance above the lens holder body 310. In the illustrated lens holder 310, the screw boss 320 is positioned 0.5 mm above the lens holder body 310. The specific distance at which the screw boss 320 is positioned above the lens holder body 310 typically depends on the requirements of a given application. Figure 3CThe illustration experiences almost all of the deformation (e.g., the stress shown in the dark regions) at the star rib 330, where little deformation occurs in the lens holder body 310. A design that directly connects the lens holder body 310 to the screw boss 320 without the star rib 330 often experiences a significant amount of deformation throughout the lens holder body 310, thereby deforming the lens 140 being held within the body 310. By using the star rib 330, the lens holder 310 is able to redirect the force applied from screwing the lens holder 310 onto the star bracket 330 while maintaining the structure of the lens holder body 310. Although the above refers to Figures 3A - 3C the scope of the lens holder design with a star rib, any of a variety of star rib design configurations can be constructed, including three or more star ribs positioned at different locations along the lens holder body and at different heights relative to the screw boss.

[0101] Although the mobile robot 10 is seen as having a single camera 125 embedded within the top cover of the mobile robot body 100, the mobile robot 10 can include any of a variety of optional camera configurations, including (but not limited to) one or more cameras 125 positioned at different locations along the mobile robot body 100 and at one or more acute viewing angles. Figures 1A - 2B

[0102] Mobile robot with front and rear cameras

[0103] The mobile robot 10 can optionally include a plurality of cameras distributed around the body 100 of the mobile robot. A particularly advantageous configuration involves using a forward-tilted camera 125 and a rearward-tilted camera 410. The forward- and rearward-tilted cameras 125, 410 can optionally be contained within separate recesses 130a, 130b within the top 108 of the body 100 of the mobile robot 10 and are protected using lens caps 135a, 135b configured in a manner similar to those described above with reference to Figure 1A and 2A -2B for mounting a single recess 130 and a tilted camera 125 behind the lens cap 135. Figure 4 The mobile robot 10 illustrated in configured with forward- and rearward-tilted cameras 125, 410, the forward- and rearward-tilted cameras 125, 410 are contained within separate recesses 130a, 130b within the top 108 of the body 100 of the mobile robot 10 and are protected using lens caps 135a, 135b. In particular, Figure 4Illustrated is a mobile robot 10, which is configured with a forward-tilted camera 125 aimed in a direction of capturing the environment in front of the mobile robot 10 and a rearward-tilted camera 410 aimed in a direction of capturing the environment behind the mobile robot 10. Much like the embodiments described above with respect to Figure 22 the embodiments, the cameras 125, 410 are angled at about 30 degrees (e.g., 25 degrees, 35 degrees) to focus a high frustum of vision of approximately 50 degrees (e.g., 45 degrees, 55 degrees) on static features located in a range of 3 - 8 feet high. In an embodiment, the cameras focus a high frustum of vision of approximately 60 degrees (e.g., 55 degrees, 65 degrees) on static features located in a range of 3 - 8 feet high. The front and rear cameras 125, 410 are embedded within separate recessed structures 130a, 130b within the mobile robot body 100, and each is covered by a respective lens cap 135a, 135b. Each lens cap 135a, 135b retracts rearward from the opening of the corresponding recessed structure 130a, 130b that contains the tilted camera 125, 410, and is aligned at an acute angle α with respect to the plane of the opening of the recessed structure 130a, 130b (e.g., the top surface 108).

[0104] Figure 5 the conceptual map shown in Figure 4Cross-sectional view of the forward- and rear-facing tilt cameras 125, 410 of the mobile robot 10 configured in the manner illustrated. The tilt cameras 125, 410 are positioned such that respective optical axes 155a, 155b of each of the cameras 125, 410 are aligned at an acute angle φ above the top surface 108 in the forward and backward motion directions of the mobile robot 10. By setting the cameras 125, 410 at these angles, the forward-facing camera 125 primarily aims at the tops of walls, ceilings, and largely immovable furniture (such as a TV cabinet, a chaise longue, and a workbench directly in front of the mobile robot 10 in a typical indoor environment), and as the mobile robot 10 travels in the forward direction, the rear-facing camera 410 primarily aims at the tops of walls, ceilings, and largely immovable furniture (such as a TV cabinet, a chaise longue, and a workbench behind the mobile robot 10). Using the tilted forward- and rear-facing cameras 125, 410 allows the mobile robot 10 to observe a high concentration of reliable static and unchanging features to establish repeatable and uniquely identifiable landmarks within the surrounding environment for use in VSLAM while maintaining the angular accuracy for detecting these features. Additionally, when these landmarks are no longer within the field of view of the front camera 125, the tilted rear camera 410 may allow the VLSAM process to use the same landmarks previously used by the tilted front camera 125. This can be particularly useful, for example, when the mobile robot 10 is configured to navigate its environment in a "row of corn" pattern and when the mobile robot 10 resumes localization after a kidnapping event (in which the mobile robot 10 is forcibly moved to a new location or loses track of its pose due to lighting changes). In such cases, whenever the mobile robot 10 turns, the same landmarks observed using the front camera 125 are captured by the rear-facing camera 410 for navigation on the return path. Additionally, assuming the same sensor resolution, the rear-facing camera 410 can detect landmarks with a higher angular accuracy than in the case where the mobile robot 10 would use, for example, a 360-degree omnidirectional camera. This is due to the increased spatial resolution that can be obtained with a narrower lens when capturing features within the surrounding environment.

[0105] In some embodiments, both cameras 125, 410 can capture images of the surrounding environment and provide these images to the VSLAM process. In certain embodiments, only one of the cameras 125 or 410 provides input images to the VSLAM process. For example, the mobile robot 10 can use the forward-facing camera 125 to detect and track a set of features associated with a landmark while moving towards the landmark in the forward direction, and use the rear-facing camera 410 to detect and track the same set of features while moving away from the landmark when switching directions.

[0106] The mobile robot 10 can use both the tilted front and rear cameras 125, 410 to simultaneously capture images of the surrounding environment, thereby capturing a larger portion of the surrounding environment in less time than the robot 10 enabled by a single camera 125. The mobile robot 10 can optionally utilize a wide-angle, omnidirectional, panoramic, or fisheye-type lens, thereby capturing more of the surrounding environment at the cost of reduced angular resolution. However, by providing input images using two cameras 125, 410 each having a narrower field of view compared to, for example, a panoramic camera, the VSLAM process is able to detect a similar number of features as would be achieved using a panoramic or similar wide-field-of-view lens, but each feature is captured at a higher angular resolution with a narrower field-of-view lens (assuming comparable sensor resolution). In particular, the narrowed field of view spans a frustum of approximately 50 - 60 degrees in the vertical direction and is able to detect features in the environment at a height generally between 3 - 14 feet. As discussed below, providing the VSLAM process with a higher-precision measurement of the positions of features visible within the images captured by the machine vision sensor system 120 enables the VSLAM process to map the environment and accurately locate the position of the mobile robot 10.

[0107] While the above refers to Figure 4 and 5 describe various alternative configurations of the mobile robot 10 involving the tilted front- and rear-facing cameras 125, 410, the mobile robot 10 can optionally be configured using any one of a variety of camera configurations, which include a tilted front camera 125 combined with a forward camera (not shown) aligned in the direction of motion, multiple front-facing cameras tilted at different angles, a stereo camera pair, two or more tilted cameras having adjacent or partially overlapping fields of view, and / or front and rear cameras 125 and 140 angled at different angles to accommodate, for example, the angled top surface 108 of the mobile robot 10. The process of performing VSLAM using the image data captured by one or more of the tilted cameras 125 in the navigation system 120 of the mobile robot 10 is typically performed by the controller circuit 605, which may also be responsible for implementing other behaviors supported by the mobile robot 10. The robot controller 605 and the VSLAM process performed by the robot controller according to various embodiments of the invention are further discussed below.

[0108] Robot controller

[0109] Based on the characteristics of the surrounding operating environment of the mobile robot 10 and / or the state of the mobile robot 10, the behavior of the mobile robot 10 is typically selected from a number of behaviors. In many embodiments, the characteristics of the environment can be determined from the images captured by the navigation system 120. The captured images can be used by one or more VSLAM processes to map the environment surrounding the mobile robot 10 and to locate the position of the mobile robot 10 within the environment.

[0110] Figure 6 The figure shown can be used for a mobile robot controller circuit 605 (hereinafter referred to as "controller circuit 605") for VSLAM using an enhanced navigation system 120. The robot controller circuit 605 includes a processor 610 that communicates with a memory 625, a network interface 660, and an input / output interface 620. The processor 610 can be a single microprocessor, multiple microprocessors, a multi-core processor, a microcontroller, and / or any other general-purpose computing system that can be configured by software and / or firmware. The memory 625 contains a visual measurement application 630, a SLAM application 635, one or more maps of landmarks 640, a behavior control application 645, and a landmark database 650. The memory 625 can optionally contain any one of a variety of software applications, data structures, files, and / or databases (suitable for the requirements of a specific application).

[0111] The landmark database 650 contains information about many previously observed landmarks that the mobile robot 10 can use to perform visual measurements from which relative poses can be determined. A landmark can be considered a collection of features with a specific 3D structure. Any of a variety of features can be used to identify a landmark, including (but not limited to) 2D features, 3D features, features identified using Scale-Invariant Feature Transform (SIFT) descriptors, features identified using Speeded-Up Robust Features (SURF) descriptors, and / or features identified using Binary Robust Independent Elementary Features (BRIEF) descriptors. When the mobile robot 10 is configured as a house cleaning robot, the landmarks may be (but not limited to) a set of features identified based on the 3D structure of the corners of a picture frame, or a set of features identified based on the 3D structure of a door frame. Such features are based on the static geometry within a room, and while the features have some illumination and scale variation, they are generally easier to discern and identify as landmarks than objects located in the lower regions of an environment with frequent positional displacements (such as chairs, trash cans, pets, etc.). In an implementation, the camera 125 on the mobile robot 10 is a 320×240 QVGA, 0.0768 MP camera (or 640×480 VGP, 0.3 MP camera) without an IR filter for better detection of features in low-light environments. In an implementation, particularly when the robot 10 is starting a new task without storing data between runs or entering a previously unprobed area, the mobile robot 10 will create new landmarks. In an implementation, if the lighting change makes previously viewed features indiscernible and the number of detectable features falls below a threshold of the minimum number of features for detecting landmarks, the mobile robot 10 will also create new landmarks. In an embodiment, the threshold number of landmarks is a cluster of identifiable features detected at a rate of approximately 1 ft per second or approximately 306 mm per second, at a rate of 1 - 10 landmarks per foot of travel (and preferably 3 landmarks per foot of robot travel). The robot 10 thus builds a useful localization map of discernible features at that lighting intensity, and in an implementation, the robot stores one or more persistent maps with landmarks viewed at various light intensities (e.g., intensities associated with data including calendar dates and times of day associated with seasonal lighting changes). In still some other implementations, if the environment is too dark and the lighting is insufficient for feature detection, the mobile robot 10 will depend on another sensor or a combination of sensors, such as wheel odometry and optical dead reckoning drift detection sensor 114 that aims at the floor (e.g., an LED or laser-illuminated mouse sensor) for localization ( Figure 1B ). In one implementation, the landmark database 650 includes landmark images captured from a specific pose that can be referred to as the landmark pose. Visual measurements involve determining a pose relative to the landmark pose. To facilitate visual measurements, the landmark database 650 stores a set of features associated with each landmark and the 3D structure of that set of features.

[0112] The vision measurement application 630 matches a portion of the input image with the landmark image and then determines the relative pose based on the spatial relationship between the features from the landmarks identified in the input image and the 3D structure of the features identified from the landmarks retrieved from the landmark database 650. There are various options for determining the relative pose based on the features from the landmarks identified in the input image and the 3D structure of the identified features, including (but not limited to) determining the relative pose based on a rigid transformation of the 3D structure that generates the spatial relationship of the features most similar to the features observed in the input image, thereby minimizing the reprojection error. Alternatively or additionally, given the observed spatial relationship of the characteristics and knowledge of the error sources within the vision measurement system, the rigid transformation produces an estimate of the most likely relative pose. Regardless of the specific process used to determine the relative pose, the accuracy of the relative pose estimate is improved by more precisely measuring the spatial relationship between the 3D structure of the features that form the landmarks and / or the features identified within the input image. The processes for creating new landmarks for use in vision measurement and for determining the relative pose using landmarks are discussed further below.

[0113] Referring again to Figure 6 , based on the previous position estimate, the odometry data, and at least one vision measurement received from the vision measurement application, the SLAM application 635 estimates the position of the mobile robot 10 within the map of the landmarks. As noted above, the vision measurement utilized by the SLAM application 635 can optionally be a relative pose estimate that is determined relative to the landmark pose associated with the landmarks identified within the landmark database 650. In an embodiment, the SLAM application 635 uses the relative pose and the odometry data and / or the mouse sensor drift data to update the position estimate of the robot 10 relative to the map of the landmarks. The mobile robot 10 utilizes any one of various SLAMs, including (but not limited to) the processes described in the U.S. Patent Publication 2012 / 0121161, entitled "Systems and Methods for VSLAM Optimization," published on May 17, 2013, and the U.S. Patent Publication 2011 / 0167574, entitled "Methods and Systems for Complete Coverage of a Surface By An Autonomous Robot," published on July 14, 2011, the relevant disclosures of which are hereby incorporated by reference in their entirety. The SLAM application 635 can then update the map 640 of the landmarks based on the newly estimated position of the mobile robot 10.

[0114] The map 640 of landmarks includes a map of the environment around the mobile robot 10 and the positions of the landmarks relative to the positioning of the mobile robot within the environment. The map 640 of landmarks may include pieces of information describing each landmark in the map, including (but not limited to) references to the data of the landmarks described in the landmark database.

[0115] The behavior control application 630 controls the actuation of different behaviors of the mobile robot 10 based on the surrounding environment and the state of the mobile robot 10. In some embodiments, as images are captured and analyzed by the SLAM application 635, the behavior control application 645 determines how the mobile robot 10 should behave based on an understanding of the environment around the mobile robot 10. The behavior control application 645 may select from a number of different behaviors based on specific characteristics of the environment and / or the state of the mobile robot 10. Behaviors may include but are not limited to wall-following behavior, obstacle avoidance behavior, evasion behavior, and many other primitive behaviors that can be actuated by the mobile robot 10.

[0116] In several embodiments, the input / output interface 620 provides the ability to communicate with devices such as (but not limited to) sensors with the processor and / or the memory. In some embodiments, the network interface 660 provides the mobile robot 10 with the ability to communicate with remote computing devices (such as computers and smart phone devices) via wired and / or wireless data connections. Although Figure 6 illustrates various robot controller 605 architectures, any one of a variety of architectures may be utilized in the implementation of the robot controller circuit 605, including architectures in which the robot behavior controller application 645 is located in non-volatile solid-state memory or some other form of storage device and is loaded into the memory at runtime, and / or architectures in which the robot behavior controller application is implemented using various software, hardware, and / or firmware. The conceptual operation of the mobile robot 10 when configured by a robot controller similar to the robot controller circuit 605 described above is further discussed below.

[0117] Mobile Robot Behavior Control System

[0118] The mobile robot 10 may include a behavior control application 710 for determining the behavior of the mobile robot based on the surrounding environment and / or the state of the mobile robot. The mobile robot 10 may include one or more behaviors activated by specific sensor inputs, and an arbiter determines which behaviors should be activated. The inputs may include images of the environment around the mobile robot 10, and behaviors may be activated in response to characteristics of the environment determined from one or more captured images.

[0119] Figure 7The conceptual map illustration is configured to enable a mobile robot behavior control application 710 for navigation within an environment based on (but not limited to) a VSLAM process. The mobile robot behavior control application 710 can receive information about its surrounding environment from one or more sensors 720 (e.g., machine vision systems, collision, proximity, wall, stagnation, and / or cliff sensors) carried by the mobile robot 10. The mobile robot behavior control application 710 can control the utilization of robot resources 725 (e.g., wheel modules) in response to the information received from the sensors 760, causing the mobile robot 10 to actuate behaviors based on the surrounding environment. For example, when the mobile robot 10 is used to clean an environment, the mobile robot behavior control application 710 can receive images from the navigation system 120 and guide the mobile robot 10 to navigate through the environment while avoiding obstacles and clutter detected in the images. The mobile robot behavior control application 710 can be implemented using one or more processors in communication with a memory containing non-transitory machine-readable instructions that configure the one or more processors to implement programmed behaviors 730 and a control arbiter 750.

[0120] The programmed behaviors 730 can include various modules that can be used to actuate different behaviors of the mobile robot 10. In particular, the programmed behaviors 730 can include a VSLAM module 740 and a corresponding VSLAM database 744, a navigation module 742, and a number of additional behavior modules 743.

[0121] The VSLAM module 740 manages the mapping of the environment in which the mobile robot 10 operates and the localization of the mobile robot with respect to the mapping. The VSLAM module 740 can store data about the mapping of the environment in the VSLAM database 744. The data can include a map of the environment and characteristics of different regions of the map, such as regions containing obstacles, other regions with traversable floors, regions that have been traversed, the borders of regions that have not been traversed, the date and time of information describing specific regions, and / or additional information suitable for the requirements of a specific application. In many instances, the VSLAM database 744 also includes information about the boundaries of the environment (including the locations of stairs, walls, and / or doors). As can be easily understood, many other types of data can optionally be stored and utilized by the VSLAM module 740 in order to map the operating environment of the mobile robot 10. In cases where the VSLAM module 740 performs visual measurements and uses the visual measurements to provide relative pose as an input to the SLAM module, the VSLAM database 744 can include a landmark database similar to the landmark database 650 described above.

[0122] The navigation module 742 actuates the mobile robot 10 in a manner based on the characteristics of the environment in which it will navigate. For example, in an implementation, the navigation module 742 may direct the mobile robot 10 to change direction, drive at a speed of approximately 306 mm per second, and then slow down when approaching an obstacle, drive in a certain manner (e.g., in a swinging manner to wipe the floor, or in a manner of pushing against a wall to clean the sidewall), or navigate to a home charging station.

[0123] Other behaviors 743 for controlling the behavior of the mobile robot 10 may also be specified. Additionally, to make the behaviors 740 - 743 more powerful, arbitration may be performed between the outputs of multiple behaviors and / or the outputs of multiple behaviors may be linked together to the inputs of another behavior module to provide complex combined functions. The behaviors 740 - 743 are intended to implement a manageable portion of the overall cognition of the mobile robot 10.

[0124] Referring again to Figure 7 , the control arbiter 750 helps to allow the modules 740 - 743 of the programmed behaviors 730 to each control the mobile robot 10 without the need to know about any other behaviors. In other words, the control arbiter 750 provides a simple prioritized control mechanism between the programmed behaviors 730 of the robot and the resources 725. The control arbiter 750 can access the behaviors 740 - 743 of the programmed behaviors 730 and control access to the robot resources 760 among the behaviors 740 - 743 at runtime. The control arbiter 750 determines which module 740 - 743 has control of the robot resources 760 as required by that module (e.g., the priority hierarchy among the modules). The behaviors 740 - 743 can start and stop dynamically and run completely independently of each other. The programmed behaviors 730 also allow complex behaviors that can be combined together to assist each other.

[0125] The robot resources 760 can be a network of functional modules (e.g., actuators, drivers, and their sets) having one or more hardware controllers. The commands of the control arbiter 750 are typically dedicated to the resources for performing a given action. The specific resources with which the mobile robot 10 is configured typically depend on the requirements of the specific application to which the mobile robot 10 is adapted.

[0126] Although specific robot controllers and behavior control applications have been described above with respect to Figures 6 - 7 , any of a variety of robot controllers can optionally be utilized within the mobile robot 10, including controllers that do not rely on a behavior - based control paradigm. Referring again to FIGS. 1 - 4, the mobile robot 10 includes a navigation system 120 that may include one or more tilt cameras 125. The manner in which the mobile robot 10 can optionally perform a VSLAM process using the input images captured by the one or more tilt cameras 125 to facilitate its programmed behaviors is discussed further below.

[0127] Overview of VSLAM

[0128] The mobile robot 10 can continuously detect and process information from various on-board sensors in order to navigate through the environment. Figure 8 Process 800 that can optionally be used by the mobile robot 10 to navigate the environment is illustrated. The process determines (805) the robot behavior. The robot behavior can be determined by the robot controller circuit 605 based on various factors, which include, among various other considerations, the specific characteristics of the environment around the mobile robot 10, the current state of the mobile robot 10, the specific operation being performed by the mobile robot, and the state of the power source used by the mobile robot 10. The mobile robot 10 determines (810) whether the behavior involves a change in the pose of the robot 10. When the mobile robot 10 determines that no pose change is required, the process completes the behavior and determines a new behavior (805). Otherwise, the process actuates (815) the drive 111 of the mobile robot 10. In some embodiments, the drive 111 uses two or more wheels 113a, 113b to move the mobile robot across the surface. In other embodiments, the drive 111 may use one or more rotating pads or grooved tracks, and the one or more rotating pads or grooved tracks move the mobile robot 10 across the surface based on the rotation of the pad or track along the surface.

[0129] As the mobile robot 10 moves, image and odometry data are captured (820). In an alternative aspect of the invention, the mobile robot 10 captures each new image after the capture of a previous image followed by traveling a threshold distance (such as, but not limited to, 20 cm between images). The specific distance between image captures typically depends on factors including, but not limited to, the speed of the robot 10, the field of view of the camera 125, and the real-time processing capabilities of the specific mobile robot 10 configuration. In several embodiments, the odometry data is provided by one or more different types of odometers, which include wheel odometers that capture odometry data based on the rotation of the wheels, or optical flow odometry systems that capture images of the tracking surface and determine the traveled distance (including correcting any heading drift) by observing the optical flow between consecutive images. Other embodiments may use additional odometry sensors or combinations of these sensors as suitable for the requirements of the specific application.

[0130] Visual measurements can be generated by the mobile robot 10 based on the odometry data and the captured images captured by the tilted camera 125. In some embodiments, process 800 matches the new image with a set of landmark images stored in the landmark database, and for each match, estimates the relative pose determined with respect to the landmark given the 3D structure and the feature correspondences between the new view and the landmark image.

[0131] Then, visual measurement data and odometry data can be used to perform the SLAM process (step 830). The mobile robot 10 can optionally maintain a map of landmarks and perform the SLAM process to estimate the position of the mobile robot 10 within the map. The SLAM process can also update the map of landmarks.

[0132] The mobile robot 10 determines whether the process has been completed (835), and if so, the process is completed. Otherwise, the process determines a new behavior (805).

[0133] Similar to the process described above with reference to Figure 8 The execution of the navigation process can be significantly enhanced by utilizing the camera 125, which is tilted such that its optical axis 155 forms an acute angle with the forward movement direction of the mobile robot 10, as described above with reference to Figure 2A 、 2B and 22. Similarly, increasing the angular resolution of the camera 125 (by utilizing a narrow field of view camera and / or increasing the resolution of the camera sensor) can increase the effectiveness of the navigation process. The sensitivity of the navigation process of the mobile robot 10 to observed parallax and / or spatial resolution is further discussed below with reference to the measurement of the 3D structure during landmark creation and the measurement of the spatial relationship of features identified in the input images captured during navigation.

[0134] Landmark Creation and Landmark-Based Navigation

[0135] The mobile robot 10 can utilize landmarks to perform navigation, where a landmark is a collection of features with a specific visually distinguishable 3D structure. The mobile robot 10 creates landmarks by capturing images of the environment and observing and aggregating common features between the images. By overlapping the images and measuring the parallax between each of the features and estimating the pose change between the captures of the images, the mobile robot 10 moving at a known speed can measure the distance to each of the features. These distances can then be used to determine the 3D structure of a set of features that define the landmark. As discussed above, when some or all of the features forming the landmark are subsequently observed, the knowledge of the 3D structure of the features can be used to estimate the relative pose of the mobile robot 10. In an implementation, the mobile robot 10 can employ two or more images over the distance traveled for localization.

[0136] Figure 9The middle illustration can optionally be used by the mobile robot 10 for the process of identifying new landmarks for use in navigation. Process 900 includes obtaining (902) a sequence of input images. The image sequence can include as few as two input images. Correspondences can be established (904) between features in two or more of the input images. Any of a variety of features can be utilized, including (but not limited to) 2D features, 3D features, features identified using Scale-Invariant Feature Transform (SIFT) descriptors, features identified using Speeded-Up Robust Features (SURF) descriptors, and / or features identified using Binary Robust Independent Elementary Features (BRIEF) descriptors. In one implementation, the robot 10 uses FAST SLAM and Brief descriptors. The hypothesized correspondences can be generated using only feature descriptors or by leveraging the motion estimates provided by odometry data. The process of establishing (904) correspondences can optionally involve applying geometric constraints to eliminate features that are mis-identified as correspondences between two or more input images.

[0137] Given a specific estimate of the 3D structure of the features to be identified, the 3D structure of the identified features can then be determined (906) by minimizing the reprojection error between the positions of the features observed in each of the input images and the predicted positions. Due to the uncertainty in the relative pose of the mobile robot 10 when the input images are captured, the mobile robot 10 can use techniques including (but not limited to) bundle adjustment to simultaneously determine (906) the 3D structure of the identified features and estimate the relative motion of the mobile robot during the capture of the input images. The mobile robot 10 can optionally use any of a variety of processes for determining structure from motion, including (but not limited to) the trifocal tensor method.

[0138] Information about the newly identified landmarks can be added (908) to the landmark database. The mobile robot 10 can optionally associate one or more of the input images or portions of the input images, along with an estimate of the pose from which the specific input image was captured, with the landmarks in the landmark database as landmark images and corresponding landmark poses. The mobile robot 10 can also optionally store the features and / or descriptors of the features associated with the landmarks, as well as a description of the 3D structure of the features. As can be readily understood, the specific structure of the database is typically determined based on the requirements of the specific application and can include (but not limited to) the use of a set of kd-trees for performing an efficient approximate nearest neighbor search based on the observed features. The information added by the mobile robot 10 to the landmark database can then be utilized by the mobile robot 10 to determine relative pose during subsequent navigation.

[0139] Figure 10The figure illustration can optionally be used by the mobile robot 10 for the process of determining the relative pose using previously created landmarks. Process 1000 includes obtaining (1002) one or more input images. The mobile robot 10 can compare (1004) the features within the input images and the features associated with various landmarks in the landmark database. The process for identifying the features in the input images is typically the same process used for identifying features during the creation of new landmarks.

[0140] When there is sufficient similarity between the features in the input image and the features of one or more landmarks in the landmark database, the mobile robot 10 can determine (1006) that a known landmark from the landmark database is visible within the input image. Based on the reprojection of the 3D structure of the landmark whose features most closely match the spatial relationship of the features observed within the input image where the landmark is visible, the mobile robot 10 can then estimate (1008) the pose of the robot relative to the landmark pose associated with the landmark in the landmark database. The relative pose can optionally be a description of the translation and / or rotation of the mobile robot 10 relative to the landmark pose associated with the landmark in the landmark database. The specific way in which it represents the relative pose depends largely on the requirements of the particular application. The process for determining the most likely relative pose estimate can vary depending on the configuration of the mobile robot 10. In an alternative configuration of the mobile robot 10, a cost metric is used to minimize the reprojection error of the landmark features visible within the input image. In another alternative configuration of the mobile robot 10, the estimation process takes into account one or more possible sources of error associated with the process of estimating the 3D structure of the features forming the landmark and / or the position of the landmark pose. In a specific configuration of the mobile robot 10, the reprojection of the 3D structure of the features forming the landmark is used to form an initial estimate of the relative pose, and then a maximum similarity estimate is performed assuming that the 3D structure of the features and the relative pose can vary. In other configurations of the mobile robot 10, any of a variety of techniques can be used to determine the relative pose based on one or more of odometry data, the position of the features in the images used to create the landmark, the 3D structure of the landmark, and / or the spatial relationship of the features associated with the landmark visible within the input image.

[0141] When the mobile robot 10 does not detect the presence of a known landmark within the input image, the mobile robot can optionally attempt to create (1010) a new landmark, or can simply obtain another input image.

[0142] The mobile robot 10 can provide (1012) an estimated relative pose and the identification of recognized landmarks to the SLAM process, which can determine a global pose estimate and / or update a global map of landmarks maintained by the mobile robot. As noted above, any one of a variety of SLAM processes can optionally be used to configure the mobile robot 10, which relies on relative pose estimates determined as inputs relative to known landmarks. In an implementation, the robot 10 uses FAST SLAM and BRIEF descriptors.

[0143] The above processes for creating new landmarks and for navigating based on previously created landmarks rely on spatial relationships that can be established using re-projections of the 3D structure of a set of features. As further discussed below with reference to FIGS. 11-16B, the alignment of the camera optical axis 155 relative to the forward motion direction of the mobile robot 10 can play an important role in the accuracy with which the mobile robot can navigate. Aligning the camera 125 directly in the forward motion direction of the mobile robot 10 can negatively affect the accuracy of the re-projection process. When the camera moves along its optical axis 155 towards a landmark, little or no parallax is observed, and the re-projection process relies solely on changes in scale. Configuring the mobile robot 10 with at least one camera 125 that is tilted so that its optical axis 155 forms an angle above the forward motion direction of the mobile robot 10 can increase the observed parallax and the accuracy of measurements made using re-projection.

[0144] Figures 11A - 11C illustrates an example of the field of view captured by the mobile robot 10 having a camera 125 that is configured so that the optical axis 155 is aligned with the forward motion direction of the mobile robot. The mobile robot 10 is in Figure 11A shown in an indoor environment. The mobile robot 10 is configured with a forward-facing camera 125 having a field of view, and the forward-facing camera 125 is capturing portions of two walls 1102, 1104 that meet at a corner 1106 and various portions of the ground surface 1108 of the environment. The mobile robot 10 is shown moving towards the corner 1106 along the optical axis 155 of the camera. Successively smaller portions of the scenes 1112, 1114, and 1116 are illustrated as being captured in Figure 11A 、 11B and 11C. In each of the three views 1112, 1114, 1116 shown in Figure 11A 、 11B and 11C, the features that remain visible appear larger the closer the mobile robot 10 is to the corner 1106. However, little parallax is observed within the center of the field of view of the camera 125 between the successive views 1112, 1114, and 1116. Figure 13AThe lack of parallax between views of a feature as the conceptual map shows the camera 125 moves directly toward the feature in the direction of motion along the optical axis 155 of the camera 125. As the camera 125 moves closer to the feature, the size of the feature increases in each of three consecutive views 1300, 1302, 1304. However, the feature remains centered within the field of view of the camera 125. Thus, the distance to the feature is difficult to determine with any precision. Additionally, features such as corners (which are typically very distinct for navigation) can look very similar at different scales. Referring again to Figures 11A - 11C corner 1106 will look very similar in each of views 1112, 1114, and 1116.

[0145] As described above with respect to Figure 2A and 2B and 22, tilting the camera 125 such that its optical axis 155 forms an acute angle above the direction of motion of the mobile robot 10 increases the parallax observed across the field of view of the camera. When the mobile robot 10 is configured as a house cleaning robot, the mobile robot 10 is constrained to move across a generally planar floor. Even when the floor is sloped, the mobile robot 10 does not move in the direction along the optical axis 155 of the camera 125. Thus, the translation of the mobile robot 10 results in an observable parallax between different views of the scene.

[0146] Figures 12A - 12C Examples of views of a scene captured by a camera 125 tilted such that its optical axis 155 forms an acute angle above the direction of motion of the mobile robot 10 are illustrated in Figures 11A - 11C The mobile robot 10 captures views of a scene similar to the scene shown in Figures 11A - 11C involving two walls 1202, 1204 meeting at a corner 1206. The mobile robot 10 moves along the floor 1208 in the direction of motion 1210 toward the corner 1206. Since the mobile robot 10 configured with a forward-facing camera is configured in the manner described above with respect to Figures 11A - 11C the mobile robot 10 configured with a tilted camera 125 captures views 1212, 1214, 1216 of successive smaller portions of the scene. However, the tilt of the camera 125 causes the field of view of the camera to shift downward along the walls 1202, 1204 in each of the successive views. Thus, the features visible in each of the three views appear to move upward between the successive views 1212, 1214, 1216. Figure 13B The conceptual map in illustrates the upward movement of the feature. As the mobile robot 10 moves along the floor in the direction toward the feature, the feature size increases and gradually moves upward in each of three consecutive views 1310, 1312, 1314.

[0147] To illustrate the observable parallax, three consecutive views 1310, 1312, 1314 are superimposed. When a feature is not on the center line of the field of view of camera 125, the feature will move upward and away from the center line of the camera's field of view. As can be readily understood, the presence of significant parallax can be used to determine the distance to a feature with a much higher precision than can be achieved by the features observed in the consecutive views 1300, 1302, 1304 illustrated in Figure 13A When considering a set of features across which the parallax increases, more information about the 3D structure of the set of features forming the landmark is provided. For example, if a set of features corresponds to an object in an environment having a 3D structure such as a door frame rather than just a single circle on a wall, then as the mobile robot 10 moves closer to the door frame, it will be able to determine the 3D structure of the door frame with higher precision. As noted above, the precision with which the 3D structure of a landmark is determined significantly affects the precision of the relative pose estimation made using the landmark during subsequent navigation. The tilt of camera 125 can also enable the mobile robot 10 to see more of the underside of objects. Many of the objects that the mobile robot 10, configured to clean indoor objects, relies on for navigation are relatively flat objects that are suspended or mounted to walls such as (but not limited to) picture frames, flat panel TVs, and speaker systems. Tilting camera 125 increases the portion of the camera's field of view dedicated to imaging the underside of such objects. Accordingly, the 3D structure of the underside of the object can be determined more precisely, resulting in more precise subsequent navigation based on the 3D structure of the landmark.

[0148] As illustrated by the series of simulations described below with reference to Figure 14A and 14B and 15, the effect of increasing the observable parallax when determining the 3D structure of a landmark and estimating the relative pose is shown. The simulations involve simulating the precision with which a mobile robot 10 moving towards a corner of a room can use its front camera to determine its relative pose (which is compared to the precision achieved using a camera tilted such that the optical axis of camera 125 forms an acute angle above the direction of motion of the mobile robot 10).

[0149] Figure 14AIt is an image 1400 generated by simulating the viewpoint of the front-facing camera of a simulated mobile robot in a virtual indoor environment. The virtual indoor environment is a room that includes textures on its walls and ceiling, in the form of images of cheetahs 1402 and zebras 1404 on two walls that form a corner, and an image of a sunrise 1406 on the ceiling 1406 of the virtual room. In the simulation, the virtual room is a 4-unit by 4-unit room with walls that are 2 units high. The simulated mobile robot camera is 1 unit above the floor, and the front-facing camera is pointed at the corner. In the simulation, the mobile robot determines the shift in relative pose based on a comparison of features recognized in the image obtained after moving forward 0.5 units towards the corner. Features recognized by the simulated mobile robot using a process similar to those described above are illustrated as multiple crosses 1408. Figure 14B It is an image 1450 generated by simulating the viewpoint of a camera that has the same field of view as the camera used in the simulation for generating the image 1400 shown in FIG. 14 (except that the camera is tilted such that the optical axis of the camera is at a 30-degree angle above the forward movement direction of the simulated mobile robot). Features recognized by the simulated mobile robot and used for performing localization are illustrated as multiple crosses 1452.

[0150] Figure 15 It is a chart showing a comparison of accuracies with which a mobile robot can determine its relative pose in the floor plane in each of the above simulations. Chart 1500 illustrates the uncertainty 1502 in the relative pose determined using a simulated mobile robot with a front-facing camera, and the uncertainty 1504 in the relative pose determined using a simulated mobile robot with a camera (the camera is tilted such that the optical axis of the camera is at a 30-degree angle above the movement direction of the mobile robot). As noted above, the simulation involves the simulated mobile robot moving 0.5 units towards the corner of the virtual room. The uncertainty is represented in terms of the error to the left or right of the true position of the simulated mobile robot and in front of or behind the true position of the simulated mobile robot defined with respect to the forward movement direction. The left-right uncertainty can be considered as the uncertainty in the dimension perpendicular to the movement direction, and the front-back uncertainty can be considered as the uncertainty in the dimension parallel to the movement direction. As can be understood by viewing Figure 15 it, the use of a camera tilted such that the optical axis of the camera is at a 30-degree angle above the movement direction of the mobile robot achieves a reduction in uncertainty with respect to the front-facing camera in two dimensions. The reduction in uncertainty in the dimension perpendicular to the movement direction is slight, however, a significant reduction in uncertainty is achieved in the dimension along the movement direction.

[0151] While the above simulations compare a front camera and a camera tilted such that its optical axis is at a 30-degree angle above the direction of motion of the mobile robot, a similar reduction in positional uncertainty can be achieved using a camera tilted such that its optical axis is aligned at other acute angles above the direction of motion of the mobile robot. Additionally, while the above was referenced Figure 9 and Figure 10 to describe the benefits of using a tilted camera 125 to increase the parallax observed in successive images captured by a camera using a process for creating landmarks and determining relative pose based on landmarks, similar benefits can be obtained by using one or more tilted cameras 125 to capture images of a scene with a mobile robot 10 configured using any of a variety of alternative VSLAM navigation processes.

[0152] Cameras with wide-angle lenses are typically used in mobile robots configured to perform VSLAM and CV-SLAM. A significant benefit of using a tilted camera 125 is that features observed within the center of the camera's field of view can be used to obtain accurate depth estimates. As discussed further below, the accuracy with which the mobile robot 10 can determine the 3D structure of landmarks can be further increased by using a tilted camera 125 configured with a narrow field of view lens 140, which has increased angular resolution relative to a camera using a sensor with the same resolution and a wider field of view lens.

[0153] Increasing navigation accuracy with increased angular resolution [[ID= 12]]

[0154] Configuring the mobile robot 10 with a tilted camera 125 having a narrow field of view and increased angular resolution can increase the accuracy with which the mobile robot 10 can determine the 3D structure of landmarks and the accuracy of relative pose estimates determined based on the reprojection of the 3D structure of landmarks. Additionally, the use of a narrow field of view lens can enable the mobile robot 10 to perform image processing without the computational cost (e.g., time, processing power, etc.) of correcting the images acquired by the camera 125. As can be readily understood, reducing the processing load can enable the mobile robot 10 to process more images at a higher frame rate (enabling the mobile robot to move faster), reduce power consumption (increasing battery life), and / or provide additional processing power to the mobile robot 10 to perform other functions.

[0155] Figure 16A and 16B illustrates the increase in accuracy with which landmarks can be located when the mobile robot 10 is configured with a tilted camera 125 having a narrow field of view compared to when the mobile robot is configured with a front-facing camera having a relatively wide field of view. Figure 16AThe sampling grid 1600 of camera 125 of mobile robot 10 using a wide-angle lens configuration is shown. The sampling grid 1600 conceptually maps the portions of the scene sampled by each pixel in the direct forward-view camera 125. In real-world applications, mobile robot 10 is typically configured with cameras of much higher resolution. However, a very low-resolution sampling grid is used to illustrate the effects of angular resolution and distortion on subsequent image processing. The sampling grid 1600 samples a scene that is a typical living room, including a couch 1620, a coffee table 1625, a television 1630, a framed picture 1635, and a window 1640. When mobile robot 10 moves in the direction along the optical axis 155 of camera 125, very little parallax is observed within the central portion of the camera's field of view, and the most reliable depth estimates of the 3D structure are obtained from features at the periphery of the camera 125's field of view. The distortion caused by the wide-angle lens is conceptually shown as curves within the sampling grid 1600. Due to the distortion, the angular resolution of camera 125 at the periphery of the camera's field of view is lower compared to the angular resolution of the camera at the center of the camera's field of view. The lower angular resolution is depicted as a larger spacing between vertices within the sampling grid 1600. Thus, features observed within the periphery of camera 125's field of view are observed with relatively low precision. A feature that may be used in navigation is the front corner of the couch. Shading 1610 is used to indicate groupings of pixels that may indicate landmarks. As can be seen by comparison with Figure 16B (discussed below), the relatively low resolution means that significant shifts and rotations of the sampling grid 1600 can occur before features are observed using different groups of pixels. Accordingly, the accuracy of the localization of mobile robot 10 is much lower.

[0156] In contrast, Figure 16B the mobile robot 10 shown in uses a tilted camera 125 with a narrow field of view to sample the scene. Due to the tilted camera 125, mobile robot 10 is able to accurately determine the 3D structure of features across the camera's field of view. Relative to Figure 16A the angular resolution of the camera shown in, narrowing the field of view of camera 125 significantly increases the angular resolution of the camera. Figure 16BThe sampling grid 1650 of the mobile robot 10 illustrated in the figure samples the scene with a much higher angular resolution or sampling density. Therefore, the accuracy with which the position of specific features can be determined is much higher. In addition, the narrow field of view lens 140 introduces very little distortion. Therefore, the angular resolution does not decrease significantly across the narrower field of view, and the mobile robot 10 can process the image data without first performing correction. The ability to use a narrower field of view lens and measure parallax with finer precision can result in a more accurate estimate of the 3D structure of the features that form the landmarks and the distances of those structural elements to the mobile robot 10, which moves relative to the stationary landmarks at a known speed. Additionally, the ability to more accurately measure the distances between the features identified within the image based on the minimization of the reprojection error of the 3D structure into a new viewpoint can result in a more accurate relative pose estimate. Accordingly, the combination of the tilted camera 125 and the narrow field of view lens 140 can result in a much higher positioning accuracy during navigation based on the VSLAM process. A feature that may be used in navigation is the upper right corner of the photo frame 1635. The shadow 1660 is used to indicate the grouping of pixels that may indicate a landmark. As can be seen by comparison with Figure 16A (discussed above), the relatively high resolution means that a small shift and / or rotation of the sampling grid will result in the feature being observed by a different group of pixels. Accordingly, the accuracy of the positioning of the mobile robot 10 is much higher.

[0157] Although different fields of view are illustrated in Figure 16A and 16B for the purpose of conceptually illustrating the benefits of using the tilted camera 125 with a narrow field of view relative to the forward-looking camera with a wide-angle lens, the mobile robot 10 can be configured with the tilted camera 125 having any one of a variety of fields of view. A narrow field of view lens is typically considered a lens in which the perspective projection is generally a good approximation of the true imaging characteristics of the lens, while a wide-angle lens introduces distortion. In an alternative configuration, the tilted camera 125 is configured with a narrow field of view lens having a horizontal field of view selected from the range of 65 - 75 degrees and a vertical field of view selected from the range of 45 - 65 degrees. In an implementation, the tilted camera 125 lens has a horizontal field of view of 72 degrees and a vertical field of view of 50 degrees. In another alternative configuration, the tilted camera 125 is configured with a narrow field of view lens having a field of view that is 70 and 80 degrees in the horizontal direction and 50 and 60 degrees or between in the vertical direction. As can be readily understood, the specific lens 140 and / or the specific field of view lens 140 utilized in one or more cameras 125 of the mobile robot 10 depends largely on the requirements of a given application.

[0158] When some or all of the field of view of camera 125 is blocked or severely blurred, the extent of the field of view of camera 125 utilized by mobile robot 10 may be largely irrelevant. Accordingly, mobile robot 10 may optionally be configured to detect the occlusion and notify the user that mobile robot 10 needs to be inspected to attempt to ignore the occlusion from the comparison of consecutive images. The occlusion detection process that may optionally be employed by mobile robot 10 is discussed further below.

[0159] Occlusion Detection

[0160] Due to the ongoing exposure to elements in the environment around mobile robot 10, one or more cameras 125, 140 in navigation system 120 of mobile robot 10 may experience degraded functionality. In particular, within an indoor environment, dust, debris, fingerprints, hair, food particles, and various other objects may collect and remain on one or more camera lens covers 135 as mobile robot 10 cleans the environment. These obstructions may reduce the quality of the images captured by camera 125 for use in the VSLAM process and thus reduce the accuracy of navigation of mobile robot 10 through the indoor environment. To maintain an adequate level of navigation performance, mobile robot 10 may provide a notification to the user when it is determined that mobile robot 10 is no longer receiving useful information from some or all views of at least one camera 125.

[0161] A portion of the field of view of one of cameras 125 of mobile robot 10 may vary with the environment in which mobile robot 10 may be configured to operate in a manner in which it becomes occluded. When a portion of the field of view is obscured, camera 125 may not provide any useful image data that can be used by navigation processes including, but not limited to, the VSLAM process. Figures 17A - 18B Conceptually illustrates an example of a specific type of occlusion and the resulting image captured by the occluded camera 125. In Figure 17A illustrates a translucent occlusion 1706. Camera 125 is shown recessed within the body 100 of mobile robot 10 and having a lens cover 135 located within the opening 1704 of the recess. A translucent occlusion 1706 (e.g., a finger smudge or a water droplet) is present on the lens cover 135. Figure 17B Conceptually illustrates the image 1710 captured by camera 125, where the translucent occlusion 1706 results in a blurred portion of image 1712. As can be readily understood, the blur may result in a lack of feature correspondence for features that would otherwise be identified within the portion of the camera field of view distorted by the translucent occlusion. Thus, the presence of occlusion 1706 will become apparent over time since navigation processes such as, but not limited to, visual measurement processes cannot identify landmarks within the affected portion of the camera field of view.

[0162] In Figure 18AThe figure shows an opaque occlusion 1806. The camera 125 is similarly shown as being recessed within the body 100 of the mobile robot 10 and having a lens cap 135 located within the opening 1804 of the recess 130. An opaque occlusion 1806 (such as a dust particle, an ink droplet, or a piece of paper) is present on the lens cap 135. Figure 18B The figure conceptually shows an image 1810 captured by the camera 125, where the opaque occlusion 1806 results in a complete occlusion of a portion of the scene 1812. In terms of the mobile robot 10 detecting features within the portion of the scene affected by the opaque occlusion 1806, the mobile robot 10 will not be able to observe any parallax of those features in a continuous view of the scene. The opaque occlusion 1806 will typically appear as a false landmark in the same position with the same dimensions in consecutive images. This lack of parallax prevents the use of any features associated with the occlusion 1806 in the creation of landmarks because the mobile robot 10 will not be able to determine any 3D structure for the landmarks. Accordingly, over time, the presence of the opaque occlusion 1806 will become apparent over time because navigation processes such as (but not limited to) the visual measurement process cannot identify landmarks within the affected portion of the field of view of the camera 125. The mobile robot 10 can optionally detect the opaque occlusion 1806 more quickly by detecting features that appear in the same position, regardless of the rotation and / or translation of the mobile robot 10.

[0163] Figure 19 The figure shows an occlusion detection process that can optionally be performed by the mobile robot 10. The process 1900 includes capturing (1905) an input image for use in navigation. As noted above, the mobile robot 10 can optionally use the camera 125 located within the recess 130 in the top cover of the body 100 of the mobile robot 10 to configure and / or the camera 125 can be tilted such that the optical axis 155 of the camera forms an acute angle above the forward movement direction of the mobile robot 10.

[0164] The mobile robot 10 determines (1910) different portions of the field of view of one or more cameras 125 in which features are identified and utilized during a navigation process including (but not limited to) a VSLAM process, and accordingly updates (1915) the occlusion detection data. When the mobile robot 10 is configured in such a way that the VSLAM process is performed during the navigation process, certain portions of the image may contain features that the VSLAM process can utilize to generate landmarks for use in performing visual measurements and mapping the environment. The mobile robot 10 can collect occlusion detection data describing portions of the image used to generate and / or detect landmarks, and these portions of the image may correspond to different portions of the camera's field of view. In an alternative aspect of the invention, the mobile robot 10 can maintain histograms of the various portions of the field of view for identifying features. When there is an occlusion, the histogram will reflect that these portions of the field of view are not being used by the VSLAM to generate and / or detect landmarks.

[0165] The mobile robot 10 can determine (1920) whether a certain threshold number of input images have been captured. In many embodiments, the threshold number of input images may vary, and the accuracy of occlusion detection generally increases with a larger number of input images. When the threshold number of images has not been captured, the process continues (1905) to capture additional images. In an embodiment, the threshold number of images is 1 - 10 images per foot of travel at a rate of approximately 1 ft per second or approximately 306 mm per second, and preferably 3 images per foot of robot travel. When the threshold number of images has been captured, the process determines (1925) whether the occlusion detection data identifies one or more portions of the field of view of the camera 125 that are capturing image data not being used by the navigation process. When the occlusion detection data does not identify one or more portions of the field of view that are capturing image data not being used by the navigation process, the mobile robot 10 assumes that the field of view of one or more cameras 125 is unoccluded, and the occlusion detection process is complete.

[0166] In one implementation, when the occlusion detection data identifies one or more portions of the field of view that are capturing image data not being used by the navigation process, the mobile robot 10 provides (1930) a camera occlusion notification to one or more users associated with the mobile robot 10 so that the users can clear the occlusion from the field of view of the camera 125. In an alternative aspect of the process, the notification is delivered in the form of an electronic message to a user who is using the user account information located on a server with which the mobile robot 10 can communicate. For example, the user can be notified via email, text message, or other communication. In another alternative aspect, the user can be notified by some form of indicator on the mobile robot 10, such as a flash light, sound, or other suitable alert mechanism. The mobile robot 10 can also optionally be configured with a wiper blade, fan, air knife, and / or another suitable cleaning mechanism that the mobile robot can use to attempt to eliminate the detected occlusion.

[0167] Figure 20 The figure shows a communication diagram illustrating communication between a mobile robot 10, an external server, and a user device according to an alternative aspect of the invention. In particular, Figure 20 The figure shows the mobile robot 10 sending an occlusion notification to the external server 2020. The mobile robot 10 may send a notification after it detects the presence of an occlusion that is obscuring a portion of the field of view of a camera on the mobile robot 10 such that the unobscured portion of the camera image is no longer useful for localizing the robot 10. In an alternative aspect of the invention, the mobile robot 10 may send a notification directly to a user device using a short-range communication protocol such as, for example, Bluetooth. When receiving a notification from the mobile robot 10, the external server 2020 sends a message to one or more user devices registered to receive the notification. In an alternative aspect of the invention, the external server may send any one of an SMS text message, an automated voicemail, and / or an email message. In another alternative aspect of the invention, the mobile robot 10 may send a notification on a periodic basis while detecting the continued presence of the occlusion. For example, the mobile robot 10 may send a notification reminder based on a daily, weekly, or other schedule while detecting the occlusion. In other alternative aspects, the mobile robot 10 may send a notification whenever it begins cleaning an environment.

[0168] Figure 21 The figure shows a system for notifying a user device of an occlusion according to an embodiment of the invention. System 2100 includes a server 2104 that receives and processes messages from one or more mobile robots. In particular, when the server receives a message from the mobile robot 10, it may provide the message to one or more user devices 2105 - 2107.

[0169] In several embodiments, various user devices may use HTTP, SMS text, or another suitable protocol to receive messages via a network 2108 such as the Internet. In the illustrated embodiment, the user devices include personal computers 2105 - 2106 and a mobile phone 2107. In other embodiments, the user devices may include consumer electronic devices such as DVD players, Blu-ray players, televisions, set-top boxes, video game consoles, tablet computers, and other devices capable of connecting to the server via HTTP and receiving messages.

[0170] Although the above includes descriptions of many specific alternative aspects of the invention, these should not be construed as limitations on the scope of the invention, but rather as examples of its different configurations. Accordingly, the scope of the invention should not be determined by the illustrated examples, but by the appended claims and their equivalents.

Claims

1. A mobile robot, comprising: A main body; A driver for supporting the main body; A controller circuit in communication with the driver, the controller circuit configured to direct the driver to navigate the mobile robot through an operating environment; And A camera configured to capture an image of the operating environment of the mobile robot, the camera aiming in the forward driving direction of the mobile robot, the camera including optics that define a camera field of view spanning vertical and horizontal directions.

2. The mobile robot according to claim 1, wherein the optics further define a camera optical axis, the optical axis forming an acute angle above the forward driving direction.

3. The mobile robot according to claim 1, wherein the camera field of view is oriented to detect features in the operating environment at a height ranging from 3 feet to 14 feet.

4. The mobile robot according to claim 3, wherein the camera includes a narrow field of view lens configured to provide a camera field of view spanning 45 - 65 degrees in the vertical direction and 65 - 75 degrees in the horizontal direction.

5. The mobile robot according to claim 4, wherein the controller circuit is configured to direct a processor to construct a map of the operating environment and use visual simultaneous localization and mapping (VSLAM) based on the images captured by the camera to direct the driver to navigate the mobile robot through the operating environment.

6. The mobile robot according to claim 2, wherein the main body includes a lens cover covering the camera, The lens cover being oriented at an acute angle with respect to the camera optical axis.

7. The mobile robot according to claim 6, wherein the acute angle of the lens cover with respect to the camera optical axis is greater than the non - zero angle of the camera optical axis with respect to the forward driving direction of the mobile robot.

8. A mobile cleaning robot, comprising: A main body defining a tombstone shape; A driver for supporting the main body; A cleaning head assembly; A controller circuit in communication with the driver and configured to direct the driver to navigate the mobile cleaning robot through an operating environment; And A camera aiming in the forward driving direction of the mobile cleaning robot, the camera configured to capture an image of the operating environment, the camera including optics that define a camera field of view spanning vertical and horizontal directions.

9. The mobile cleaning robot according to claim 8, wherein the optics further define a camera optical axis, The optical axis being inclined at an acute angle above the forward driving direction of the mobile cleaning robot.

10. The mobile cleaning robot according to claim 8, wherein the camera is tilted to orient the camera field of view to detect features in the operating environment at a height ranging from 3 feet to 14 feet.

11. The mobile cleaning robot according to claim 10, wherein the main body includes a lens cover covering the camera, the lens cover being oriented at an acute angle with respect to the camera optical axis.

12. The mobile cleaning robot according to claim 8, wherein the cleaning head assembly is cantilevered.

13. The mobile cleaning robot of claim 8, wherein the mobile cleaning robot is a robotic vacuum cleaner, the robotic vacuum cleaner further comprising a brush adjacent a periphery of the body.

14. The mobile robot of claim 1, wherein the optical device further defines a camera optical axis, the camera optical axis aligned parallel to the forward drive direction.

15. The mobile robot according to claim 1, wherein: the body including a recess, the body having a front portion and a rear portion relative to the forward driving direction; The camera is configured to be mounted in the recess. The mobile robot according to claim 15 , wherein the recess is located at a front portion of the main body.

17. The mobile robot of claim 8, wherein the optical device further defines a camera optical axis, the camera optical axis aligned parallel to the forward drive direction.

18. The mobile robot of claim 1, wherein the camera is configured to be tilted to orient the camera field of view to detect features below a horizontal plane containing the top surface of the body.

19. The mobile robot of claim 1, wherein the camera is configured to be tilted to orient the camera field of view to not be obstructed by the main body.

20. The mobile robot of claim 1, wherein the camera is configured to be tilted to orient the camera field of view to detect features at a wall-ceiling intersection or a portion of a ceiling surrounding an area directly above the mobile robot.

Images