Photomosaic soil mapping

Abstract

A method for providing a floor plan from images captured by a floor-moving robot, comprising: providing a camera mounted high enough in a robot housing to allow an inclined downward view of a floor; capturing a series of images of the floor with the camera; determining a position of the robot for each of the images; labeling each image in the series of images with a position of the robot at the time of image capture; cropping each of the images to identify at least one segment in each of the images; transforming the segment from a robot camera view into a planar view; filling the floor plan with each of the segments at the robot position in the planar view, with an offset corresponding to a difference between the robot position and the position of the segment on the floor; displaying the floor plan to a user;and accepting user input on the floor plan to change an operating property of the robot.;

Description

BACKGROUND OF THE INVENTION The present disclosure relates generally to robotic cleaners and in particular to the creation of floor plans. Cleaning robots often contain cameras for various purposes. The camera can capture images of an area to be cleaned and display objects within that area. Collected image data can be used to create a map using a SLAM (Simultaneous Location and Mapping) algorithm. A VSLAM (Visual SLAM using image sensors) localization method can also be used with image data. A camera can capture images of a ceiling and use them for navigation. The map can be created by stitching together images. The iRobot application with publication number US 2017 / 0052033A1 describes such a visual SLAM method. The Carnegie Mellon patent with number US 5999866A describes positioning based on unique aspects of images, such as carpet patterns. The published iRobot application with number...US 2017 / 0203446A1 shows a floor plan and camera image capture of objects and provides an indication of the floor plan's location, with a link to the image. iRobot patent number US 9014848B2 describes the correction of distortions in a target layout map compared to a robot-generated map. DE 10 2010 029 241 A1 discloses a method for cultivating a soil surface using a soil cultivation device comprising a soil cultivation unit and at least one detection unit, wherein the soil cultivation device is moved over the soil surface, images of the soil surface are created at successive times by means of the at least one detection unit, and a map of the soil surface is generated by the soil cultivation device based on the images. DE 10 2012 109 004 A1 discloses a mobile, self-driving robot for autonomously performing tasks. The robot comprises a drive module for moving the robot across the floor; a processing module for performing the tasks during a processing operation; and a navigation module, which is designed to navigate the robot across the floor during the processing operation using a map of the environment, and to store and manage one or more maps of the environment. DE 103 13 309 A1 discloses an automatic cleaning device, an automatic cleaning system, and a method for controlling them. The aim is to efficiently execute a command-controlled activity by detecting the travel distance and direction of the cleaning device independently of wheel slippage or floor irregularities. DE 101 64 280 A1 discloses a cleaning robot system comprising a remotely controllable cleaning robot and an external control unit for controlling the cleaning robot, wherein the transmission of control data takes place via a mobile communication network. Furthermore, a mobile communication terminal is provided for transmitting a command for operations to the cleaning robot via a mobile communication relay device. Piotr Jasiobedzki reveals further robot cleaning systems in Detecting driveable floor regions. In: Proceedings 1995 IEEE / RSJ international conference on intelligent robots and systems. human robot interaction and cooperative robots, August 5-9, 1995, Pittsburgh, PA, USA. Vol. 3. Piscataway, NJ: IEEE, 1995. pp. 264-270. ISBN 0-8186-7108-4. Robot cleaning maps and floor plans are optimized for use by the robot to determine its position and take other actions, but not for use by humans. It would be desirable to have a map optimized for human user interaction to control a cleaning robot. SUMMARY OF THE INVENTION In one embodiment, a robot (e.g., a cleaning robot) generates a planar map from a bird's-eye view by transforming the view from a robot camera and stitching together images labeled according to position. The planar map can then be displayed on a user interface as a floor plan showing the position of objects (e.g., carpets). A camera is mounted sufficiently high within the cleaning robot's housing to allow for a downward angled view of the floor. The camera's field of view is captured as an image, and a portion or segment of this image is cropped or masked. The cropped segment is transformed from the camera perspective into a planar view (before or after cropping) and combined with other images to create a floor plan map. In one embodiment, a lens (e.g., a fisheye lens) is used in front of the camera. Before (or after) the transformation from the robot perspective view to the planar view, a distortion algorithm is applied to correct the image distortion caused by the lens. The transformation from the robot perspective view to the planar view utilizes the known mounting height of the camera from the ground and the known downward angle of the camera to correlate a pixel position with a planar ground position. The position relative to the robot is used as an offset to the localized position of the robot. Any tilt of the robot is detected and used to correct the pixel position or simply discard the image. In one embodiment, the floor plan is displayed to a user on a smartphone or other display device. The user provides input on the floor plan to modify an operating property of the robot. For example, the user can specify areas that the robot should not enter, areas that require more intensive cleaning, etc. In one embodiment, segments are captured for an area equal to or larger than the minimum resolution of the floor plan to be generated. For example, an area of ​​just a few millimeters might correspond to a single pixel of a floor plan displayed on a smartphone, and thus a finer resolution would be a waste of computing power. Likewise, a less expensive camera with only such a required resolution can serve the purposes of the present invention. Overlapping segments are captured, allowing for alignment by comparing the overlapping sections. The same segment area can be captured multiple times from different distances within the camera's field of view or from different passes over the same area. Image quality can be weighted, with a higher-quality image replacing a lower-quality one.Weighting factors include, for example, the speed of the robot when capturing the image, vibrations, an angle (a tilt), a match with overlapping parts of other segments, the proximity of the segment to the camera, and lighting. In one embodiment, the illumination of the image is determined and deviations in the illumination of different segments are corrected. The illumination can vary due to fluctuations in the ambient light or due to fluctuations in the intensity of a light on the cleaning robot within the camera's field of view. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a representation of a cleaning robot with a LiDAR tower according to one embodiment. Fig. 2 is a representation of a cleaning robot and a charging station according to one embodiment. Fig. 3 is a representation of the underside of a cleaning robot according to one embodiment. Fig. 4 is a representation of a smartphone control application display for a cleaning robot according to one embodiment. Fig. 5 is a representation of a smartwatch control application display for a cleaning robot according to one embodiment. Fig. 6 is a representation of an electronic system for a cleaning robot according to one embodiment. Fig. 7 is a simplified block diagram of a representative computer system and a client computer system suitable for implementing certain embodiments of the present invention.Figure 8 shows a camera embedded in a cleaning robot housing according to one embodiment. Figure 9 shows a cleaning robot with a camera, showing the calculated planar position of a point in the camera image according to one embodiment. Figure 10 shows an image from a robot's camera distorted by a fisheye lens according to one embodiment. Figure 11 shows an image from a robot's camera after a transformation to correct the distortion by a fisheye lens according to one embodiment. Figure 12 shows an image from a robot's camera showing the trimming of a portion of the floor according to one embodiment. Figure 13 shows an image from a robot's camera that has been transformed into a planar shape and shows a captured segment of the image according to one embodiment.Figure 14 shows a representation of a user device display that shows joined segments to form a planar image with objects according to one embodiment. Figure 15 shows a representation of a user device display that shows joined segments to form a planar image with no-go sections that are marked by a user on the display, according to one embodiment. Figure 16 is a flowchart of the method for generating a planar view according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION Overall architecture Fig. 1 is a representation of a cleaning robot with a LIDAR turret or LIDAR turret according to one embodiment. A cleaning robot 102 has a LIDAR (Light Detection and Ranging) turret 104 that emits a rotating laser beam 106. Detected reflections of the laser beam from objects are used to calculate both the distance to objects and the location or position of the cleaning robot. One embodiment of the distance calculation is shown in U.S. Patent No. 8,996,172 B2, "Distance Sensor System and Method," the disclosure of which is incorporated herein by reference. Alternatively, VSLAM (Visual SLAM with Image Sensors) or other localization methods can be used. The collected data are also used to create a map using a SLAM (Simultaneous Location and Mapping) algorithm. One embodiment of a SLAM algorithm is shown in U.S. Patent No.US 8 903 589 B2 , “Method and apparatus for simultaneous localization and mapping of the environment of a mobile robot”, described, the disclosure of which is incorporated herein by reference. Fig. 2 shows a cleaning robot and a charging station according to one embodiment. The cleaning robot 102 with tower 104 is shown. Also shown is a cover 204, which can be opened to access a dirt collection bag and the top of a brush. The buttons 202 enable basic functions of the robot cleaner, such as starting a cleaning process. A display 205 provides information to the user. The cleaning robot 102 can dock with a charging station 206 and draw power via the charging contacts 208. Fig. 3 shows the underside of a cleaning robot according to one embodiment. The wheels 302 move the cleaning robot, and a brush 304 helps to vacuum up loose dirt into the dirt bag. Fig. 4 shows a representation of a smartphone control application display for a cleaning robot according to one embodiment. A smartphone 402 has an application or app that is downloaded to control the cleaning robot. An easy-to-use interface has a start button 404 to initiate cleaning. Fig. 5 shows a representation of an intelligent clock control application display for a cleaning robot according to one embodiment. Example displays are shown. A display 502 provides an easy-to-use start button. A display 504 offers the possibility to control multiple cleaning robots. A display 506 gives the user feedback, e.g., a message that the cleaning robot has completed a task. Fig. 6 is an overview diagram of an electronic system for a cleaning robot according to one embodiment. A cleaning robot 602 includes a processor 604, which operates a program downloaded into the memory 606. The processor communicates with other components via a bus 634 or other electrical connections. In a cleaning mode, the wheel motors 608 control the wheels independently of each other to move and steer the robot. Brush and vacuum motors 610 clean the floor and can be operated in various modes, such as an intensive cleaning mode with higher power or a mode with normal power. A LIDAR module 616 includes a laser 620 and a detector 616. Alternatively, an image sensor with VSLAM operation can be used. A turret motor 622 moves the laser and the detector to detect objects up to 360 degrees around the cleaning robot. It rotates several times per second, for example, about 5 revolutions per second. Various sensors provide inputs for the processor 604, such as a shock sensor 624, which indicates contact with an object; a proximity sensor 626, which indicates proximity to an object; and acceleration and tilt sensors 628, which indicate a fall (e.g., down stairs) or a tipping of the cleaning robot (e.g., when climbing an obstacle). Examples of the use of such sensors for navigation and other controls of the cleaning robot are described in US Patent No.US 8,855,914 B1, "Method and apparatus for traversing corners of a floor surface with a robotic surface treatment device," the disclosure of which is incorporated herein by reference. Other sensors can be included in other embodiments, such as a dirt sensor to detect the amount of dirt to be vacuumed, a motor current sensor to detect when the motor is overloaded, e.g., because it is entangled in something, a floor sensor to detect the type of floor, and an image sensor (camera) to provide images of the surroundings and objects. A battery 614 supplies power to the rest of the electronics via power connections (not shown). A battery charging circuit 612 supplies a charging current to the battery 614 when the cleaning robot is docked to the charging station 206 of Fig. 2. Input buttons 623 enable direct control of the robot cleaner 602 in conjunction with a display 630. Alternatively, the cleaning robot 602 can be remotely controlled and send data to remote locations via transceiver 632. One or more cameras 642 are connected to bus 634. A lens 644 is located in front of the camera. A light 646 can be provided to illuminate the camera's field of view. The images can be stored in memory 606 and / or uploaded to the cloud via a wireless or wired connection and the internet. The cleaning robot can be controlled via the Internet 636 and / or other networks and send information back to a remote user. A remote server 638 can provide commands and process data uploaded by the cleaning robot. A wearable smartphone or watch 640 can be operated by a user to send commands either directly to the cleaning robot 602 (via Bluetooth, direct RF, a WiFi LAN, etc.) or via an Internet 636 connection. The commands can be sent to the server 638 for further processing and then forwarded in modified form to the cleaning robot 602 via the Internet 636. Computer systems for a media platform and client system Various processes described herein can be performed on computer systems. Fig. 7 shows a simplified block diagram of a representative computer system 702 and a client computer system 704 suitable for implementing certain embodiments of the present invention. In various embodiments, the computer system 702 or similar systems can implement the cleaning robot processor system, the remote server, or another computer system described herein, or parts thereof. The client computer system 704 or similar systems can implement user devices, such as a smartphone or a watch, with a robot cleaning application. The 702 computer system can be one of several types, including processor and memory, a portable handheld device (e.g., an iPhone® mobile phone, an iPad® computer tablet, a PDA), a portable device (e.g., a Google Glass® head-mounted display), a personal computer, a workstation, a mainframe computer, a kiosk, a server rack, or any other data processing system. The 702 computer system can include a 710 processing subsystem. The 710 processing subsystem can communicate with a number of peripheral systems via the 770 bus subsystem. These peripheral systems can include an 730 I / O subsystem, a 768 storage subsystem, and a 740 communication subsystem. The Bus Subsystem 770 provides a mechanism by which the various components and subsystems of the Server Computer System 704 can communicate with each other as intended. Although the Bus Subsystem 770 is schematically represented as a single bus, alternative embodiments of the Bus Subsystem can use multiple buses. The Bus Subsystem 770 can form a local area network that supports communication in processing the Subsystem 710 and other components of the Server Computer System 702. The Bus Subsystem 770 can be implemented using various technologies, including server racks, hubs, routers, etc. The Bus Subsystem 770 can be one of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local area bus, using any variety of bus architectures.Such architectures can include, for example, an ISA bus (Industry Standard Architecture), an MCA bus (Micro Channel Architecture), an Enhanced ISA bus (EISA), a local bus (Video Electronics Standards Association, VESA), and a PCI bus (Peripheral Component Interconnect), which can be implemented as a mezzanine bus according to IEEE standard P1386.1 and the like. The I / O subsystem 730 can include devices and mechanisms for inputting information into the computer system 702 and / or for outputting information from or about the computer system 702. In general, the term "input device" is used to encompass all possible types of devices and mechanisms for inputting information into the computer system 702. User interface input devices, for example, can be a keyboard, pointing devices such as a mouse or trackball, a touchpad or touchscreen integrated into a display, a scroll wheel, a click wheel, a dial, a button, a switch, a keyboard, audio input devices with speech recognition systems, microphones, and other types of input devices.User interface input devices can also include motion sensor and / or gesture recognition devices, such as the Microsoft Kinect® motion sensor, which allows users to control and interact with an input device, the Microsoft Xbox® 360 game controller, or devices that provide an interface for receiving input via gestures and spoken commands. User interface input devices can also include eye gesture detection devices, such as the Google Glass® blink detector, which detects eye activity (e.g., blinking while taking pictures and / or selecting a menu) and converts the eye gestures into input for an input device (e.g., Google Glass®). Furthermore, user interface input devices can include speech recognition devices, which allow users to interact with speech recognition systems (e.g., Siri® navigator) using voice commands. Other examples of user interface input devices include three-dimensional (3D) mice, joysticks or pointing sticks, gamepads and graphics tablets, as well as audio / visual devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. Furthermore, user interface input devices can include, for example, input devices for medical imaging such as computed tomography (CT), magnetic resonance imaging (MRI), positional emission tomography (PEM), and medical ultrasound devices. User interface input devices can also include audio input devices such as MIDI keyboards, digital musical instruments, and the like. User interface output devices may include a display subsystem, indicator lights, or non-visual displays such as audio output devices, etc. The display subsystem may be a cathode ray tube (CRT), a flat-panel device such as a liquid crystal display (LCD) or plasma display, a projection device, a touchscreen, and the like. In general, the term "output device" encompasses all possible types of devices and mechanisms for outputting information from the Computer System 702 to a user or another computer. For example, without limitation, user interface output devices may include a variety of display devices that visually transmit text, graphics, and audio / video information, such as monitors, printers, loudspeakers, headphones, automotive navigation systems, plotters, speech output devices, and modems. The processing subsystem 710 controls the operation of the computer system 702 and may include one or more processing units 712, 714, etc. A processing unit may include one or more processors, including single-core or multi-core processors, one or more processor cores, or combinations thereof. In some embodiments, the processing subsystem 710 may include one or more special co-processors, such as graphics processing units (GPUs), digital signal processors (DSPs), or the like. In some embodiments, some or all of the processing units of the processing subsystem 710 may be implemented using custom circuitry, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored on the circuit itself.In other embodiments, the processing unit(s) can execute instructions stored in local memory, e.g., in local memory 722, 724. Any type of processor in any combination can be contained in the processing unit(s) 712, 714. In some embodiments, the 710 processing subsystem can be implemented in a modular design that includes any number of modules (e.g., blades in a blade server implementation). Each module can include processing units and local memory. For example, the 710 processing subsystem can include the 712 processing unit and the corresponding 722 local memory, as well as the 714 processing unit and the corresponding 724 local memory. The local memory 722, 724 can include volatile storage media (e.g., conventional DRAM, SRAM, SDRAM, or the like) and / or non-volatile storage media

[0001] (e.g., magnetic or optical disks, flash memory, or the like). Storage media integrated into the local memory 722, 724 can be fixed, removable, or expandable, as desired. The local memory 722, 724 can be physically or logically divided into various subunits, such as system memory, ROM, and a permanent storage device. The system memory can be a read / write storage device or a volatile read / write memory, such as a dynamic random-access memory. The system memory can store some or all of the instructions and data required by the processing unit(s) 712, 714 at runtime.The ROM can store static data and instructions required by the processing units 712 and 714. The permanent storage device can be a non-volatile read / write storage device capable of storing instructions and data even when a module containing one or more processing units 712 and 714 and a local memory 722 and 724 is powered off. The term "storage medium" as used herein includes any medium on which data can be stored indefinitely (subject to overwriting, electrical disturbances, power failure, or the like), but excludes carrier waves and transient electronic signals propagating wirelessly or via wired connections. In some embodiments, the local memory 722, 724 can store one or more software programs to be executed by the processing units 712, 714, such as an operating system and / or programs implementing various server functions, such as functions of the UPP system 102 or other servers associated with the UPP system 102. "Software" generally refers to sequences of instructions that, when executed by the processing unit(s) 712, 714, cause the computer system 702 (or parts thereof) to perform various operations, thereby defining one or more specific machine implementations that execute and perform the operations of the software programs.The instructions can be stored as firmware located in read-only memory and / or as program code stored on non-volatile memory media and read into volatile memory for execution by the processing unit(s) 712, 714. In some embodiments, the instructions can be stored by the memory subsystem 768 (e.g., computer-readable memory media). In various embodiments, the processing units can execute a variety of programs or code instructions and manage multiple concurrently running programs or processes. At any given time, some or all of the program code to be executed may be stored in local memory 722, 724 and / or in the memory subsystem, possibly including one or more memory media.Software can be implemented as a single program or as a collection of separate programs or program modules that interact as desired. From the local memories 722, 724 (or the non-local memories described below), the processing units 712, 714 can retrieve program instructions for execution and data for processing in order to perform various operations described above. The storage subsystem 768 provides a repository or data storage for storing information used by the computer system 702. The storage subsystem 768 is a tangible, non-volatile, computer-readable storage medium for storing the basic programming and data constructs that provide the functionality of some embodiments. Software (programs, code modules, instructions) that, when executed by the processing subsystem 710, provides the functionality described above can be stored in the storage subsystem 768. The software can be executed by one or more processing units of the processing subsystem 710. The storage subsystem 768 can also provide a repository for storing data used according to the present invention. The memory subsystem 768 can include one or more non-volatile memory devices, including both volatile and non-volatile storage devices. As shown in Fig. 7, the memory subsystem 768 includes a system memory 760 and a computer-readable storage medium 752. The system memory 760 can include several memories, including volatile main memory for storing instructions and data during program execution and non-volatile ROM or flash memory in which fixed instructions are stored. In some implementations, a basic input / output system (BIOS), containing the basic routines that help transfer information between elements within the computer system 702, for example, at startup, is typically stored in ROM. The RAM typically contains data and / or program modules that are currently operated and executed by the processing of the subsystem 710.In some implementations, the 760 system memory can include several different memory types, such as static random-access memory (SRAM) or dynamic random-access memory (DRAM). The 768 memory subsystem can be based on magnetic, optical, semiconductor, or other data storage media. Direct-attached storage, storage area networks, network-attached storage, and the like can be used. All data stores or other data collections described herein as being produced, consumed, or maintained by a service or server can be stored in the 768 memory subsystem. A system memory 760, for example, can store application programs 762, which may include client applications, web browsers, mid-tier applications, relational database management systems (RDBMS), etc., program data 764, and one or more operating systems 766, as shown in Fig. 7. For example, a sample operating system may include various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems, a variety of commercial UNIX® or UNIX-like operating systems (including, but not limited to, the variety of GNU / Linux operating systems, Google Chrome® OS, and the like), and / or mobile operating systems such as iOS, Windows® Phone, Android® OS, BlackBerry® 10 OS, Palm® OS, and Palm® OS. Computer-readable storage media 752 can store programming and data constructs that provide the functionality of some embodiments. Software (programs, code modules, instructions) that, when executed by processing the subsystem 710, is provided by a processor that provides the functionality described above, can be stored in the memory subsystem 768. For example, computer-readable storage media 752 can include non-volatile memory such as a hard disk drive, a magnetic disk drive, an optical drive such as a CD-ROM, DVD, Blu-ray® Disc, or other optical media. Computer-readable storage media 752 can include, but are not limited to, Zip® drives, flash memory cards, USB flash drives, secure digital (SD) cards, DVD discs, digital videotapes, and the like.Computer-readable storage media 752 can also be solid-state drives (SSDs) based on non-volatile memory, such as flash memory-based SSDs, enterprise flash drives, solid-state ROMs, and the like; SSDs based on volatile memory, such as solid-state RAM, dynamic RAM, static RAM, DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs that use a combination of DRAM and flash memory-based SSDs. Computer-readable media or storage devices 752 can enable the storage of computer-readable instructions, data structures, program modules, and other data for the computer system 702. In certain embodiments, the storage subsystem 768 may also include a computer-readable storage media reader 750, which can further be connected to the computer-readable storage medium 752. Together and optionally in combination with the system memory 760, computer-readable storage media 752 can comprehensively represent remote, local, fixed and / or removable storage devices and storage media for storing computer-readable information. In certain embodiments, the Computer System 702 can provide support for running one or more virtual machines. The Computer System 702 can run a program, such as a hypervisor, to facilitate the configuration and management of the virtual machines. Memory, processing power (e.g., processors, cores), I / O, and network resources can be allocated to each virtual machine. Each virtual machine typically runs its own operating system, which may be the same as or different from the operating systems run by other virtual machines running on the Computer System 702. Accordingly, multiple operating systems can be run concurrently by the Computer System 702. Each virtual machine generally runs independently of the other virtual machines. A Communications Subsystem 740 provides an interface to other computer systems and networks. The Communications Subsystem 740 serves as an interface for receiving data from and transmitting data to other systems from the Computer System 702. For example, the Communications Subsystem 740 can enable the Computer System 702 to establish a communication channel to one or more client computer devices over the Internet for receiving and sending information to and from the client computer devices. The Communication Subsystem 740 can support both wired and wireless communication protocols. For example, in certain embodiments, the Communication Subsystem 740 can include radio frequency (RF) transceiver components for accessing wireless voice and / or data networks (e.g., using cellular technology, advanced data network technology such as 3G, 4G, or EDGE (increased data rates for global development), Wi-Fi (IEEE 802.11 family standards or other mobile communication technologies, or any combination thereof), Global Positioning System (GPS) receiver components, and / or other components. In some embodiments, the Communication Subsystem 740 can provide a wired network connection (e.g., Ethernet) in addition to or instead of a wireless interface. The Communications Subsystem 740 can receive and send data in various formats. For example, in some configurations, the Communications Subsystem 740 can receive incoming communications in the form of structured and / or unstructured data feeds, event streams, event updates, and the like. For instance, the Communications Subsystem 740 can be configured to receive (or send) real-time data feeds from users of social media networks and / or other communication services, such as Twitter® feeds, Facebook® updates, web feeds like Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources. In certain embodiments, the 740 communication subsystem can be configured to receive data in the form of continuous data streams, which may include event streams of real-time events and / or event updates that can be continuous or unlimited without an explicit end. Examples of applications that generate continuous data include sensor data applications, financial tickers, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, automotive traffic monitoring, and the like. The Communication Subsystem 740 can also be configured to output the structured and / or unstructured data feeds, event streams, event updates, and the like to one or more databases that can be connected to one or more streaming data source computers connected to the Computer System 702. The 740 communication subsystem can provide a 742 communication interface, such as a WAN interface, which enables data communication between the local network (770 bus subsystem) and a larger network, such as the internet. Conventional or other communication technologies can be used, including wired (e.g., Ethernet, IEEE 802.3 standards) and / or wireless technologies (e.g., Wi-Fi, IEEE 802.11 standards). The 702 computer system can operate in response to requests received through the 742 communication interface. Furthermore, in some embodiments, the 742 communication interface can connect multiple 702 computer systems to each other, providing scalable systems capable of handling high activity volumes. Conventional or other techniques can be used to manage server systems and server farms (collections of cooperating server systems), including dynamic resource allocation and redistribution. The Computer System 702 can interact with various user-owned or user-controlled devices over a wide-area network such as the Internet. An example of a user-controlled device is shown in Fig. 7 as the Client Computer System 702. The Client Computer System 704 can be implemented, for example, as a consumer device such as a smartphone, another mobile phone, a tablet computer, a wearable computing device (e.g., a smartwatch, glasses), a desktop computer, a laptop, etc. For example, the client computer system 704 can communicate with the computer system 702 via the communication interface 742. The client computer system 704 can include conventional computer components such as processing unit(s) 782, storage device 784, network interface 780, user input device 786, and user output device 788. The client computer system 704 can be a computer device implemented in a variety of form factors, such as a desktop computer, a laptop computer, a tablet computer, a smartphone, other mobile computing devices, portable computing devices, or the like. The processing unit(s) 782 and the storage device 784 can be similar to the processing units 712, 714 and the local storage 722, 724 described above. Suitable devices can be selected depending on the requirements of the client computer system 704; for example, the client computer system 704 can be implemented as a "thin" client with limited processing capabilities or as a high-performance computer device. Program code can be provided to the client computer system 704 that is executable by the processing units 782 to enable various interactions with the computer system 702 of a message management service, such as accessing messages, performing actions on messages, and other interactions described above. Some client computer systems 704 can also interact with a message service independently of the message management service. A network interface 780 can establish a connection to a wide area network (e.g., the Internet) to which the communication interface 740 of the computer system 702 is also connected. In various embodiments, the network interface 780 can include a wired interface (e.g., Ethernet) and / or a wireless interface that implements various RF data transmission standards such as Wi-Fi, Bluetooth, or cellular data network standards (e.g., 3G, 4G, LTE, etc.). A user input device 786 can include any device (or devices) by which a user can provide signals to the client computer system 704; the client computer system 704 can interpret the signals as indicating special user requests or information. In various embodiments, the user input device 786 can include a keyboard, a touchpad, a touchscreen, a mouse or other pointing device, a scroll wheel, a click wheel, a dial, a button, a switch, a microphone, and so on. A user output device 788 can include any device by which the client computer system 704 can provide information to a user. For example, the user output device 788 can include a display for showing images generated by or supplied to the client computer system 704. The display can include various image-generating technologies, such as a liquid crystal display (LCD), a light-emitting diode (LED), including organic light-emitting diodes (OLEDs), a projection system, a cathode ray tube (CRT), or the like, together with supporting electronics (such as digital-to-analog or analog-to-digital converters, signal processors, or the like). Some embodiments can include a device, such as a touchscreen, that functions as both an input and output device.In some embodiments, other user output devices 788 may be provided in addition to or instead of a display. Examples include flashing lights, loudspeakers, tactile "display" devices, printers, etc. Some embodiments include electronic components, such as microprocessors, storage, and memory, which store computer program instructions on a computer-readable storage medium. Many of the functions described in this description can be implemented as processes, which are specified as a set of program instructions encoded on a computer-readable storage medium. When these program instructions are executed by one or more processing units, they cause the processing unit(s) to perform various operations specified in the program instructions. Examples of program instructions, or computer code, include machine code, as generated by a compiler, and files of higher-level code, which are executed by a computer, electronic component, or microprocessor using an interpreter.Through appropriate programming, the processing units 712, 714 and 782 can provide various functions for the computer system 702 and the client computer system 704, including any of the functions described herein performed by a server or client, or other functionality related to message management services. It should be noted that computer system 702 and client computer system 704 are for illustrative purposes only and that variations and modifications are possible. Computer systems used in conjunction with embodiments of the present invention may have other capabilities not expressly described herein. Although computer system 702 and client computer system 704 are described with reference to certain blocks, it should be understood that these blocks are defined for convenience only and are not intended to implicitly involve a specific physical arrangement of components. For example, different blocks may be located in the same facility, in the same server rack, or on the same motherboard, but need not be. Furthermore, the blocks need not correspond to physically distinct components. Blocks can be configured to perform different operations, e.g.,This is achieved by programming a processor or providing a suitable control circuit, and various blocks may or may not be reconfigurable depending on how the initial configuration is obtained. Embodiments of the present invention can be implemented in a variety of devices, including electronic devices, using any combination of circuitry and software. Photomosaic soil mapping Fig. 8 shows a camera embedded in a cleaning robot housing according to one embodiment. One embodiment uses a single camera 804 in the front housing 802 of a cleaning robot. An alternative embodiment adds a second camera 806. As shown, both cameras are mounted in a recess 812 on the front of the housing 802. The cameras 804 and 806 are mounted so that they point slightly downwards towards a base 810. In one embodiment, a light 808 is mounted between the cameras to illuminate the cameras' field of view. Fig. 9 is a representation of the cleaning robot with a camera that shows the calculated planar position of a point in the camera's image according to one embodiment. The cleaning robot 902 has a camera 904 mounted on its front. The camera has a field of view defined by the area between the upward line 916 and the downward line 918. The camera is mounted at a known distance 906 from a floor or other support surface 908. For each individual pixel in an image within the camera's field of view, the planar position can be calculated from the position and the camera height 906. For example, the camera views point 912 at a pixel position illustrated by a line 914 touching the camera. This pixel in the image can then be assigned a position corresponding to an axial distance 910 from the camera to point 912 on the floor 908. The position of robot 902 is known from a LiDAR module 920. Therefore, if the offset distance between the detector in LiDAR module 920 and camera 904 is known, an offset is determined. The position of point 912 is thus the coordinates of robot cleaner 902 determined by the LiDAR, plus the offset, plus the axis distance 910. It should be noted that other localization mechanisms can be used interchangeably, such as VSLAM, indoor GPS, wheel odometer, or any combination thereof. The location calculation would be adjusted accordingly. For VSLAM, for example, the offset would be the distance between camera 904 and the VSLAM camera. If the same camera 904 is used for both functions, the offset can be eliminated. Alternative methods can also be used to determine the planar position of a pixel in the camera image. For example, objects such as walls or devices can be detected in the image, and their position can be known from a height sensor on the robotic cleaner through prior contact. From this known position and the known position of the robotic cleaner, the position of the pixels in the image can be calculated. Any tilt of the robot is detected and used to correct the image point or pixel position, or simply to discard the image. The tilt can be detected by a tilt sensor, accelerometer, or by analyzing the image data. For example, the vertical movement of a known point in an image that would exceed the robot's translational movement can be used as an indication of the tilt. To ensure a sufficient field of view, a fisheye lens or other lens can be placed in front of the camera, such as a lens or objective 644, as shown in Fig. 6. In one embodiment, a field of view between 100 and 180 degrees, such as 160 degrees, is provided. Alternatively, two cameras can be used, as shown in Fig. 8, thereby eliminating the need for a lens or reducing the amount of magnified field of view required by the lens. In one embodiment, the camera is located several centimeters from the ground, for example, between 5 and 20 centimeters, or about 7 centimeters in another embodiment. The camera's field of view can begin several centimeters in front of the robot, between 5 and 25 centimeters in front of it, or 10 centimeters in another embodiment.The field of view captured by the camera is larger than the minimum resolution of the potential planar map and can range from 4x4 mm to 40x60 cm. In one embodiment, the field of view comprises a rectangular area extending 20 cm in the direction of the camera's view and 30 cm perpendicular to the camera's view. The captured images can be cropped, and the cropped images contain one or more tiles or segments that comprise the map. In one embodiment, the segments or tiles are approximately 4x4 cm in size. Alternatively, the segments can be between 4x4 mm and 40x60 cm in size. In one embodiment, the camera has a resolution at the lower end of the HD range. Since the images are used to fill a floor plan visible on a smartphone, a higher resolution is not required, saving costs and processing power. For example, a 720p HD camera (1280 x 720 pixels) would be more than sufficient resolution. The number of frames per second does not need to be high and can be adjusted to the required localization accuracy. For example, if a LiDAR system is used that captures 5 frames per second, a frame rate of 5 frames per second is more than sufficient for the camera. In one embodiment, the image capture time can be synchronized with the localization time to ensure that the image accurately reflects the location. Fig. 10 shows an image from a robot's camera, distorted by a fisheye lens according to one embodiment. As can be seen, the field of view includes a floor 1002 and a wall 1004 with obvious distortion. Fig. 11 shows an image from a robot's camera after a transformation to correct distortion using a fisheye lens according to one embodiment. The image of the floor 1102 was corrected, as was the wall 1104. In one embodiment, the distortion is corrected using the Open CV (Computer Vision) Warp program. Fig. 12 shows a camera image from a robot depicting the cutting of a portion of the floor according to one embodiment. An area 1202 of the floor surface 1204 is cut for processing. The arching can be performed before or after cutting. The cut image can then be converted into a planar map coordinate. Fig. 13 shows a representation of an image from a robot's camera that has been transformed into a planar shape and depicts a captured segment of the image according to one embodiment. As can be seen, the transformed image is a trapezoid 1304, the shape of which changes when mapped from an oblique view onto a planar surface. A segment / tile 1302 is then captured to create an image that can be joined with other segments / tiles to generate the planar plan. In one embodiment, a series of segments / tiles are extracted from each camera image. For example, a series of segments / tiles 1302 can be captured. The segments can overlap, so the joining process can adjust the orientation slightly as needed by shifting the sub-segments until the overlapping sections align.A second row can be taken from a subsequent image after the robot cleaner has progressed to the point where the same row in the image is the next row on the flat surface, overlapping in the forward direction. Several other embodiments can be used. A grid of segments can be captured from a single image, and a series of overlapping segments can be obtained from subsequent images. In one embodiment, the primary series of captured segments is located far enough from the camera to allow for highly accurate location calculation, but not so far as to compromise image clarity. When multiple cameras are used, their fields of view overlap, and this overlap is incorporated into the alignment of the segments. Multiple cameras can also provide redundancy in case one camera's lens becomes dirty or the images are otherwise blurred; in such cases, higher-quality overlapping or subsequent images from the other camera can be weighted and used instead. Image quality can be weighted, with a higher-quality image replacing a lower-quality one, regardless of whether the images were taken by the same camera or multiple cameras. Weighting factors include, for example, the robot's speed when capturing the image, vibrations, angle (tilt), alignment with overlapping parts of other segments, the segment's proximity to the camera, and lighting. An image captured at a slower robot speed can have a higher weight than one captured at a higher speed. Similarly, an image taken closer to the robot can have a higher weight than one taken farther away.An image captured during a tilt or vibration event may be weighted less than an image captured without or with less tilt or vibration. Images captured with higher lighting conditions may be weighted more heavily, up to a point where excessive lighting degrades image quality. Fig. 14 is a representation of a user device display showing joined segments to form a planar image with objects according to one embodiment. A display 1406 shows a planar map 1402 of a floor plan mapped by a cleaning robot. The map shows two carpets 1404 and 1406. The maps are sufficiently detailed to show the fringes of the carpets, thus reminding the user that the carpets have fringes. In one embodiment, a rectilinear grid of the floor plan is first created from localization or position data. The various rectangles in the grid are then filled with camera images. The position markers assigned to the images are used to indicate the appropriate grid position for the image. Adjustments are made using the overlapping areas. The overlapping portion of one of the image segments is then removed. Fig. 15 shows a representation of a user device display that displays joined segments to form a planar image with no-go areas marked on the display by a user according to one embodiment. The planar floor plan 1402 shows carpets 1404 and 1408 as in Fig. 14. The user has interacted with the display to draw no-go areas 1502 and 1504 around the edge of carpet 1404 so that the robot cleaner's brush does not get caught on the edge or fringes. The no-go areas 1506 and 1508 are marked on carpet 1408. Instead of specifying a no-go area, other modifications to the robot cleaner can also be displayed. For example, the robot vacuum cleaner can continue over these areas, but the brush can be switched off to prevent entanglement or tangling. The image is just one example of the objects that can be recognized.Many other objects can be presented, such as whether the floor is tiled or carpeted, small objects or unassigned areas that indicate furniture, etc. Fig. 16 is a flowchart of the method for generating a planar floor plan according to one embodiment. The camera first captures a series of images of the floor (step 1602). The robot's location or position is determined for each position (1604), e.g., by LiDAR, VSLAM, GPS, or other means. A marker indicating the robot's position at the time of image capture (1606) is provided for each image in the series. The images are cropped to identify at least one segment in each image (1608). A transformation of the segment from a robot camera view to a planar view is performed (1610). The floor plan is populated with each of the segments at the robot's position in the planar view, with an offset corresponding to a difference between the robot's position and the segment's position on the floor (1612). The floor plan is displayed to a user (1614).A user input is accepted to change an operating property of the robot (1616). Several other configurations are possible. The camera can capture visible light, infrared light, ultraviolet light, 3D point clouds, or other types of images. The camera images can be stored and processed locally on the cleaning robot, in a docking station, or remotely on a computer connected to the robot via Wi-Fi, Bluetooth, or another wireless connection, or connected to the docking station either wirelessly or via a wired connection. A remote server connected via the internet or another network can store and / or process images. Uploading images to a docking station processor, a remote computer, or a server can be asynchronous, occurring after the cleaning robot has completed a run, or in real time. Processing can be distributed across multiple processors.For example, one or more operations, such as initial cropping, lens distortion correction, and conversion to a planar format (bird's-eye view), can be performed in the cleaning robot, with piecing and other operations being carried out remotely. Although the invention has been described with respect to certain embodiments, a person skilled in the art will recognize that numerous modifications are possible. Embodiments of the invention can be implemented using a variety of computer systems and communication technologies, including, but not limited to, the specific examples described herein. Embodiments of the present invention can be implemented with any combination of specific components and / or programmable processors and / or other programmable devices. The various methods described herein can be implemented on the same processor or on different processors in any combination. If components are described as being configured to perform certain operations, this configuration can be achieved, for example, by designing electronic circuits to perform the operation, by programming programmable electronic circuits (such as microprocessors) to perform the operation, or any combination thereof.Although the embodiments described above may refer to specific hardware and software components, it is known to those skilled in the art that various combinations of hardware and / or software components can also be used and that certain processes described as being implemented in hardware can also be implemented in software or vice versa. Computer programs incorporating various features of the present invention can be encoded and stored on various computer-readable storage media; suitable media include magnetic disks or tapes, optical storage media such as compact discs (CDs) or DVDs (digital versatile disks), flash memory, and other non-volatile media. Computer-readable media encoded with the program code can be packaged with a compatible electronic device, or the program code can be provided separately from electronic devices (e.g., via internet download or as a separately packaged, computer-readable storage medium). Although the invention has been described with regard to certain embodiments, it should therefore be noted that the invention is intended to cover all modifications and equivalents within the scope of protection of the following claims.

Claims

A method for providing a floor plan from images captured by a floor-moving robot, comprising: providing a camera mounted high enough in a robot housing to allow an inclined downward view of a floor; capturing a series of images of the floor with the camera; determining a position of the robot for each of the images; labeling each image in the series of images with a position of the robot at the time of image capture; cropping each of the images to identify at least one segment in each of the images; transforming the segment from a robot camera view into a planar view; filling the floor plan with each of the segments at the robot position in the planar view, with an offset corresponding to a difference between the robot position and the position of the segment on the floor; displaying the floor plan to a user;and accepting user input on the floor plan to change an operating property of the robot.; The method of claim 1, further comprising: providing a lens in front of the robot camera; applying a distortion algorithm to the series of images to correct the image distortion caused by the lens. The method of claim 1, wherein filling the floor plan with each of the segments at the robot position in the planar view further comprises: using the known manufactured height of the camera from the ground and the known downward angle of the camera to correlate a pixel position with a position of the planar ground in the floor plan. Method according to claim 3, further comprising: detecting a tilt of the robot beyond a first threshold value; and correcting the pixel position or discarding the image. The method according to claim 1, further comprising: capturing segments for an area equal to or greater than a minimum resolution of the floor plan. The method of claim 1, further comprising: detecting overlapping segments; and adjusting the alignment setting by aligning an overlapping section of one segment with an overlapping section of an adjacent segment. The method of claim 1, further comprising: multiple acquisition of the same segment to provide multiple versions of the same segment; determining a weighted quality of each version of the same segment; replacing a lower-quality version with a higher-quality version. Method according to claim 7, wherein the weighted quality is determined using at least one of the robot's speed when capturing the image, a vibration, a tilt of the robot, a match with overlapping sections of other segments, a proximity of the segment to the camera and an illumination. The method of claim 1, further comprising: providing a light source on the robot for the camera's field of view; storing an initial value of image brightness for the light source; detecting deviations in the illumination of different segments from the initial value; and making necessary corrections to the brightness of the segments due to illumination deviations. A method for providing a floor plan from images taken by a floor-moving robot, comprising: providing a camera mounted high enough in a robot housing to allow an inclined downward view of a floor; capturing a series of images of the floor with the camera; providing a lens in front of the robot camera; applying a distortion algorithm to the series of images to correct image distortion caused by the lens; determining a position of the robot for each of the images; labeling each image in the series of images with a position of the robot at the time of image capture; detecting a tilt of the robot above a first threshold; correcting the pixel position or discarding the image; cropping each of the images to identify at least one segment in each of the images;Capturing segments for an area equal to or greater than a minimum planar resolution; transforming the segment from a robot camera view to a planar view; populating the planar view with each of the segments at the robot position in the planar view by using the known manufacturing height of the camera from the ground and the known downward angle of the camera to correlate a pixel position with a position of the planar ground in the planar view, with an offset corresponding to a difference between the robot position and the position of the segment on the ground; capturing overlapping segments; adjusting the orientation setting by matching an overlapping section of one segment with an overlapping section of an adjacent segment; displaying the planar view to a user; and accepting user input on the planar view to change an operating property of the robot. System for providing a floor plan from images, comprising: a floor-moving robot; a robot housing for the floor-moving robot; a motor mounted in the housing for moving the floor-moving robot; a battery connected to provide electrical power to the motor; a camera mounted high enough in the robot housing to provide an inclined downward view of a floor; a processor mounted in the robot; memory in the robot; a localization module in the robot; non-volatile, computer-readable code in memory with instructions for capturing a series of images of the floor with the camera, determining a position of the robot for each of the images from data provided by the localization module, and tagging each image in the series of images with a position of the robot at the time of image capture;a non-volatile, machine-readable code containing instructions for cropping each of the images to identify at least one segment in each of the images, transforming the segment from a robot camera view into a planar view, filling the floor plan with each of the segments at the robot position in the planar view, with an offset corresponding to a difference between the robot position and the position of the segment on the floor, displaying the floor plan to a user, and accepting user input on the floor plan to change an operational property of the robot. System according to claim 11, further comprising: a lens mounted in front of the robot camera; and non-volatile, computer-readable code with instructions for applying a distortion algorithm to the image series to correct the image distortion caused by the lens. System according to claim 11, further comprising: a tilt sensor mounted in the robot; and a non-volatile, computer-readable code embedded in a computer-readable data carrier containing instructions for detecting a tilt of the robot beyond a first threshold; and correcting the pixel position or discarding the image. System according to claim 11, further comprising: a second camera mounted sufficiently high in the robot housing to provide an inclined downward view of a floor with a field of view overlapping with the first-mentioned camera. System according to claim 11, further comprising: a light mounted in the robot to provide illumination of the camera's field of view; a non-volatile, computer-readable code embedded in a computer-readable data carrier containing instructions for storing an initial brightness value for the light source, detecting deviations in the illumination of different segments from the initial value; and making necessary corrections to the brightness of the segments due to illumination deviations. System according to claim 11, further comprising: a LIDAR module mounted and configured in the robot to determine a position of the robot using a SLAM algorithm.

Images