User terminal and method for operating the same

Abstract

The present disclosure provides a user terminal and a method for operating the user terminal. The user terminal includes: a receiver; a user interface; a processor coupled to the receiver and the user interface; and a memory coupled to the processor, the memory including a non-transitory computer-readable storage medium having computer-readable program code stored therein, the computer-readable program code being executable by the processor to perform operations including: receiving, via the receiver of the user terminal, map information based on a correlation between positioning data and wireless communication signals, wherein: in response to navigation of at least one mobile robot in its operating environment, the positioning data is detected in the operating environment by a positioning sensor of the at least one mobile robot, and the wireless communication signals are acquired in the operating environment by a wireless receiver of the at least one mobile robot; and displaying, via the user interface of the user terminal, one or more visual representations of the map information.

Description

[0001] This application is a divisional application of the invention patent application with the application date of April 6, 2018, application number 201880040719.4 (international application number PCT / US2018 / 026477), and invention name “Method, system and device for drawing wireless communication signals for guiding mobile robots”. Technical Field

[0002] The present disclosure relates to a mobile robot and a system and method including the mobile robot, as well as a user terminal and a method of operating the user terminal. Background Art

[0003] Connectivity for home appliances, including wireless connections to the internet and remote clients, has been a concern for some time. The term "Internet of Things" (IoT) represents the idea that various household items can be connected to the public internet. Once connected, these items can report various data to servers and client devices. For example, a "smart" light bulb can connect to a home WLAN (wireless local area network). Each bulb may have a microprocessor, memory, some means of detecting or interpreting conditions, power, and a wireless connection. Using these components, the bulb can report its status, be polled, and more.

[0004] The concept of IoT is often considered distinct from home connectivity (e.g., connected computers, cable boxes, media devices, etc.) because IoT devices typically do not include sufficient computing resources or communications to meaningfully connect to the public internet. A regular refrigerator cannot connect to the internet; this same device as an IoT device would include computing, sensor, and communication hardware, as well as sufficient software to become a remotely and locally addressable entity; this internet-connected refrigerator would be expected to report its various states (power consumption, etc.) and respond to remote commands (increase or decrease the internal temperature).

[0005] Home mobile robots can also become IoT devices. In some ways, home mobile robots can be considered a unique species in this set. In particular, the autonomy of home mobile robots can distinguish them from other appliances, which do not operate in unpredictable and changing environmental conditions and cannot make autonomous decisions based on dozens or hundreds of sensor inputs in order to achieve task completion. For example, a dishwasher (even an IoT dishwasher) knows nothing about its contents and runs the equivalent of a simple script controlling the motor and pump, which may be interrupted by a simple blockage or other sensor. In contrast, a vacuuming robot (such as A mobile robot can detect its own state in multiple ways while performing tasks, flexibly escape challenging situations in the home, and engage in anticipatory and planned activities. As such, integrating the autonomous behavior of mobile robots in the home with IoT connectivity can be challenging. Summary of the Invention

[0006] According to some embodiments of the present disclosure, a method of operating a computing device includes executing operations by a processor of the computing device. The operations include: receiving occupancy data for an operating environment of a mobile robot based on positioning data detected by at least one positioning sensor of the mobile robot in response to the mobile robot's navigation in the operating environment; and receiving signal coverage data for the operating environment based on wireless communication signals acquired by at least one wireless receiver of the mobile robot in response to the mobile robot's navigation in the operating environment. The wireless communication signals are transmitted by at least one electronic device local to the operating environment. The operations also include generating a map indicating coverage of wireless communication signals at corresponding locations in the operating environment by correlating the occupancy data and the signal coverage data.

[0007] In some embodiments, the coverage pattern may indicate respective signal strengths of wireless communication signals at respective locations in the operating environment.The operations may also include determining a location of the at least one electronic device in the operating environment based on the coverage pattern.

[0008] In some embodiments, the operations may further include identifying a type, manufacturer, and / or model of the at least one electronic device based on addressing and / or network service information indicated by the signal coverage data.

[0009] In some embodiments, the at least one positioning sensor may be a visual positioning sensor. The operations may also include: determining the placement of the at least one mobile device at its location based on its type, manufacturer, and / or model; and correlating the placement and location with an image of the at least one mobile device in the positioning data.

[0010] In some embodiments, the positioning data may include visual positioning data detected by at least one camera at different times of the day and / or under different lighting conditions, and generating the map may include using the location of at least one electronic device as an anchor point to merge corresponding occupancy maps generated from the visual positioning data detected at different times of the day and / or under different lighting conditions.

[0011] In some embodiments, wireless communication signals may be acquired over a predetermined duration such that the map includes a persistent data structure representing a set of respective signal strengths at respective locations in the operating environment.

[0012] In some embodiments, wireless communication signals may be acquired by at least one wireless receiver of the mobile robot in response to monitoring one or more communication channels and / or one or more communication protocols.

[0013] In some embodiments, the at least one electronic device may be a first electronic device, and the operations may further include transmitting the map to a second electronic device in the operating environment.

[0014] In some embodiments, the mobile robot may be a first mobile robot in the operating environment, the second electronic device may be a second mobile robot in the operating environment, and the second mobile robot may lack a visual positioning sensor.

[0015] In some embodiments, the operation may also include: storing the map as one of a plurality of maps in a non-volatile memory of the computing device; and identifying the map as corresponding to corresponding floors at different heights of the operating environment based on the asymmetry in the corresponding coverage pattern indicated thereby.

[0016] According to some embodiments, a method of operating a mobile robot includes performing operations by a processor of the mobile robot. The operations include: operating an actuator to navigate the mobile robot in an operating environment; and in response to operating the actuator to navigate the mobile robot in the operating environment, detecting positioning data via at least one positioning sensor of the mobile robot. The operations also include: in response to operating the actuator to navigate the mobile robot in the operating environment, acquiring wireless communication signals via at least one wireless receiver of the mobile robot, wherein the wireless communication signals are transmitted by at least one electronic device local to the operating environment, and determining a position of the mobile robot in the operating environment based on a correlation between the positioning data detected by the at least one positioning sensor and the wireless communication signals acquired by the at least one wireless receiver.

[0017] In some embodiments, the operation may further include detecting a repositioning of the mobile robot within the operating environment. The repositioning may be independent of and subsequent to operating the actuator. Determining the position of the mobile robot may include comparing a corresponding signal strength of a wireless communication signal acquired in response to the repositioning with a corresponding signal strength of the wireless communication signal acquired in response to operating the actuator.

[0018] In some embodiments, the operation may also include: determining the position of at least one electronic device in the operating environment based on the corresponding signal strength of the wireless communication signal obtained by at least one wireless receiver at the corresponding position in the operating environment; and determining the position of the mobile robot may be relative to the position of the at least one electronic device in the operating environment.

[0019] In some embodiments, the at least one positioning sensor may be a camera, and the positioning data may be visual positioning data detected by the camera at different times of the day and / or under different lighting conditions. The operations may also include: using the location of the at least one electronic device as an anchor point, merging corresponding occupancy maps generated from the visual positioning data detected at different times of the day and / or under different lighting conditions.

[0020] In some embodiments, the operation may also include operating a driver to navigate the mobile robot to a location of the at least one electronic device in the operating environment in response to activation of a sensor associated with the at least one electronic device, capturing audio and / or visual information at the location of the at least one electronic device via at least one positioning sensor in response to operating the driver to navigate the mobile robot toward the at least one electronic device, and transmitting the audio and / or visual information to a remote device via a wireless transmitter of the mobile robot.

[0021] In some embodiments, the at least one electronic device may be a first device and a second device. The operation may further include: recognizing that the first device and the second device are out of wireless communication range of each other based on respective signal strengths of wireless communication signals acquired from the first device and the second device via the at least one wireless receiver in response to the operation driver, and transmitting detected information about one of the first device and the second device to the other of the first device and the second device via the wireless transmitter of the mobile robot.

[0022] In some embodiments, the information may indicate interfering communications between the first and second devices based on wireless communication signals acquired from the first and second devices. The operations may also include transmitting an instruction to at least one of the first and second devices via a wireless transmitter of the mobile robot to change its communication channel.

[0023] In some embodiments, at least one wireless receiver may be mounted on or coupled to a telescopic mast or other elevated vantage point of the mobile robot, and acquiring wireless communication signals may be responsive to operating the telescopic mast or other elevated vantage point to change the height of the at least one wireless receiver at a corresponding location in the operating environment.

[0024] In some embodiments, the at least one electronic device may be a first device and a second device. The operations may further include: monitoring multiple communication protocols via a wireless receiver; and identifying that the first device and the second device are configured to communicate using different communication protocols based on wireless communication signals acquired from the first device and the second device via the at least one wireless receiver in response to the monitoring. The operations may further include: transmitting, via a wireless transmitter of the mobile robot, detected information about one of the first device and the second device to the other of the first device and the second device via a corresponding communication protocol among the different communication protocols.

[0025] According to some embodiments, a mobile robot includes a drive configured to navigate the mobile robot within an operating environment, at least one positioning sensor, at least one wireless receiver, a processor coupled to the drive, the at least one positioning sensor, the at least one wireless receiver, and a memory coupled to the processor. The memory includes a non-transitory computer-readable storage medium storing computer-readable program code executable by the processor to perform operations, including operating the drive to navigate the mobile robot within the operating environment and, in response to operating the drive, detecting positioning data of the operating environment via at least one positioning sensor of the mobile robot. The operations also include acquiring wireless communication signals via at least one wireless receiver of the mobile robot in response to operating the drive, wherein the wireless communication signals are transmitted by at least one electronic device local to the operating environment; and determining a position of the mobile robot within the operating environment based on a correlation between the positioning data detected by the at least one positioning sensor and the wireless communication signals acquired by the at least one wireless receiver.

[0026] According to some embodiments, a method of operating a user terminal includes executing operations by a processor of the user terminal. The operations include receiving, via a receiver of the user terminal, map information based on a correlation between positioning data detected by a positioning sensor of at least one mobile robot in its operating environment in response to navigation of the at least one mobile robot within the operating environment and wireless communication signals acquired by a wireless receiver of the at least one mobile robot. The wireless communication signals are transmitted by at least one electronic device local to the operating environment. The operations also include displaying, via a user interface of the user terminal, one or more visual representations of the map information.

[0027] In some embodiments, one or more visual representations may include an occupancy map indicating physical characteristics of the operating environment, including obstacles detected in the operating environment by a positioning sensor of at least one mobile robot, and / or a signal coverage map indicating electrical characteristics of the operating environment, including corresponding signal strengths of wireless communication signals acquired by a wireless receiver of at least one mobile robot at corresponding locations in the operating environment.

[0028] In some embodiments, the signal coverage map can be a 2-dimensional representation that indicates the respective signal strengths of the wireless communication signals by changing colors and / or brightness levels at respective locations in the operating environment, and / or the signal coverage map can be a 3-dimensional topological representation that indicates the respective signal strengths of the wireless communication signals at respective locations in the operating environment relative to an axis and the slope changes of the respective signal strengths between the respective locations.

[0029] In some embodiments, the occupancy map may also indicate a location of the at least one electronic device within the operating environment.

[0030] In some embodiments, the occupancy map may also indicate the location of the at least one electronic device using a graphical representation of signals transmitted from the at least one electronic device.

[0031] In some embodiments, the operations may further include receiving an indication of a current location of the at least one mobile robot in the operating environment. The graphical representation of the signal may vary based on a proximity of the current location of the mobile robot to a location of the at least one electronic device.

[0032] In some embodiments, the graphical representation may vary in color, brightness level, and / or distance between displayed bands or rings.

[0033] In some embodiments, the occupancy map may also indicate an identity, including the type, manufacturer, and / or model of at least one electronic device.

[0034] In some embodiments, the occupancy map may also identify at least one electronic device as being mobile or stationary.

[0035] In some embodiments, the operation may also include: displaying, via a user interface of a user terminal, indications of one or more suggested locations for repositioning of at least one electronic device in the operating environment based on wireless communication signals acquired by a wireless receiver of at least one mobile robot.

[0036] In some embodiments, the operation may also include: based on signal interference indicated by wireless communication signals obtained by a wireless receiver of at least one mobile robot, displaying an indication of one or more suggested frequencies or channels for communication by at least one electronic device via a user interface of a user terminal.

[0037] In some embodiments, the location of at least one electronic device in the operating environment may be the latest location at which wireless communication signals are received from the at least one electronic device, and the operation may also include providing an audio and / or visual indication of the latest location via a user interface of the user terminal.

[0038] In some embodiments, one or more visual representations of the map information may indicate areas of the operating environment where signal coverage for wireless communication signals is weak or non-existent, and the operations may further include displaying, via the user interface, suggestions for improving signal coverage.

[0039] In some embodiments, the at least one electronic device may be a wireless router device, and the recommendation may be an incentive to purchase a new wireless router device.

[0040] According to some embodiments, a user terminal includes a receiver, a user interface, a processor coupled to the receiver and the user interface, and a memory coupled to the processor. The memory includes a non-transitory computer-readable storage medium storing computer-readable program code therein, the program code being executable by the processor to perform operations, the operations including receiving, via the receiver of the user terminal, map information based on a correlation between positioning data detected by a positioning sensor of at least one mobile robot in its operating environment in response to navigation of the at least one mobile robot in the operating environment and wireless communication signals acquired by a wireless receiver of the at least one mobile robot in the operating environment. The operations also include displaying, via a user interface of the user terminal, one or more visual representations of the map information.

[0041] According to some embodiments, a method of operating a user terminal includes executing operations by a processor of the user terminal. The operations include: receiving, via the receiver of the user terminal, information indicating that there is no wireless router signal coverage in a corresponding location of the mobile robot's operating environment, based on wireless communication signals acquired by a wireless receiver of the mobile robot in response to navigation in the operating environment. The operations also include: displaying, via a user interface of the user terminal, an incentive to purchase a wireless router device based on the information indicating the lack of wireless router signal coverage.

[0042] Those skilled in the art will recognize further features, advantages, and details of the present disclosure upon review of the drawings and following detailed description of the embodiments, including any and all combinations of the above embodiments, which description is merely illustrative of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1A is a schematic diagram illustrating a system according to an embodiment of the present disclosure.

[0044] Figure 1B The present invention is a diagram showing an embodiment of the present invention including a Figure 1A Schematic diagram of the operating environment of the system.

[0045] Figure 2A and 2B are top and bottom perspective views, respectively, of a mobile robot according to an embodiment of the present disclosure, which is configured to Figure 1Aoperating in the system.

[0046] Figure 2C is a schematic diagram showing a mobile robot according to an embodiment of the present disclosure, which is configured to Figure 1A operating in the system.

[0047] Figure 3 is a schematic diagram showing a remote server according to an embodiment of the present disclosure, which is configured to Figure 1A operating in the system.

[0048] Figure 4 is a schematic diagram showing a user terminal according to an embodiment of the present disclosure, which is configured to Figure 1A operating in the system.

[0049] Figure 5A is a diagram illustrating coverage patterns of wireless communication signals acquired at various locations in an operating environment by a mobile robot according to an embodiment of the present disclosure.

[0050] Figure 5B and 5C is a topology diagram illustrating coverage patterns of wireless communication signals acquired by a mobile robot according to an embodiment of the present disclosure at various locations in an operating environment.

[0051] Figure 6A 、 6B 6C are diagrams illustrating the correlation of signal coverage data and occupancy data obtained by the mobile robot according to an embodiment of the present disclosure in an operating environment.

[0052] Figure 7 is a populated plan view illustrating respective locations and identifications of networked devices based on correlation of signal coverage data and occupancy data obtained by a mobile robot according to an embodiment of the present disclosure.

[0053] Figure 8A and 8B 2-dimensional and 3-dimensional coverage patterns according to an embodiment of the present disclosure are illustrated respectively, where the coverage patterns are based on the correlation of signal coverage data and occupancy data obtained by a mobile robot.

[0054] Figure 9-11 is a flowchart illustrating operations that may be performed by at least one processor of a computing device according to an embodiment of the present disclosure.

[0055] Figure 12A and 12B are diagrams illustrating systems and related operations of devices, respectively, according to embodiments of the present disclosure.

[0056] Figure 13-17 is a plan view illustrating a user interface and related operations of a user terminal according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0057] Embodiments of the present disclosure may arise from the recognition that the autonomous functionality of a mobile robot may present unique advantages for integration with IoT connectivity based on its independent positioning capabilities. In particular, while navigating an operating environment (e.g., while performing cleaning tasks or patrol tasks in a home operating environment), the mobile robot may be configured to simultaneously, intermittently, or subsequently acquire wireless communication signal coverage data over one or more areas of the operating environment. For example, the mobile robot may be configured to acquire multiple samples or readings of wireless communication signals transmitted by multiple electronic devices (such as IoT-enabled devices) and perform temporal and geographic averaging (e.g., based on received signal strength) on the acquired wireless communication signals in order to construct a 2-dimensional or 3-dimensional representation of the wireless communication signals (such as an RSSI field map, which may be represented as a topological map) for each individual electronic device from which the wireless communication signal is received in the entire operating environment. Such a signal coverage map may avoid distortions that may affect magnetic-based navigation.

[0058] In some embodiments, wireless communication signal readings acquired at a plurality of 2-dimensional locations along the floor surface of the operating environment can be used to populate a 2-dimensional map of wireless communication signal strength at corresponding locations within the home. The 2-dimensional map can be stored in a non-transitory computer-readable memory accessible to a network entity and can be displayed on a human-machine interface device that communicates with the mobile robot via the network. Such human-machine interface devices include, but are not limited to, mobile handsets, smartphones, personal computers, smart watches, and mobile tablets having a display and / or touch screen, a wireless network interface, a processor, and memory.

[0059] In some embodiments, wireless communication signal readings obtained at multiple 2-dimensional locations within a home can be used to populate a 3-dimensional map of wireless communication signal strength at corresponding locations within the home. For example, the 3-dimensional map can be a topological map, where the x and y axes correspond to a plan view of the operating environment, and the z axis corresponds to the signal strength, and the slope change in the z direction (including the rate of change and the direction of change) can provide a unique "signature" for the corresponding position in the x and y directions. The wireless communication signal readings can be used for a variety of purposes. For example, the wireless communication signal readings can be used internally by the mobile robot (e.g., to improve positioning), can be shared with one or more other electronic devices (e.g., to improve the functionality of the corresponding device), and / or can be shared with the user terminal (e.g., to improve network functionality).

[0060] A "mobile robot" may refer to any device that includes a processor, memory, and a drive system for navigating variable environmental conditions and making autonomous decisions based on multiple sensor inputs. Mobile robots as described herein may include robotic vacuum cleaners (such as and / or BRAAVA Jet TM Vacuum cleaners), and autonomous patrol robots. Some such autonomous patrol robots may include a telescopic mast on which one or more sensor elements are mounted or otherwise operably associated with the telescopic mast.

[0061] "Wireless communication signals" may refer to any signal transmitted by a network-enabled electronic device (also referred to as a "networked device"). Such electronic devices may include a processor, memory, and a wireless transmitter and / or receiver for communicating via wireless personal, local area networks, and / or wide area networks, and may include, but are not limited to, IoT devices. Wireless communication signals may include radio frequency signals, including, but not limited to, Wi-Fi signals, Bluetooth signals, ZigBee signals, and / or Z-wave signals, and / or optical signals. Such electronic devices may or may not include sufficient computing resources or communications to meaningfully connect to the public Internet.

[0062] refer to Figure 1A 、 1B 2-4, a system 100 according to an embodiment of the present disclosure is installed in an associated residential structure 10. Structure 10 can be a house or residence, including but not limited to a single-family home, a multi-family residence (e.g., a unit in a duplex, apartment, condominium, etc.), a mobile home, or a commercial living space (e.g., an office or studio). Structure 10 can define a living space 20, or interior space, which can be subdivided (physically, spatially, and / or functionally) into one or more defined zones (illustrated as zones AC). In some embodiments, these zones can correspond to rooms within residential structure 10, such as zone A being the kitchen, zone B being the living room, and zone C being the bedroom. Defined zones AC can be separated by walls or can be open-concept areas that blend together without walls. Structure 10 has windows 30, doors 32, light fixtures 34 (including combustible lights 34A), a television (TV) 36 (or other electronic equipment), and a heating, ventilation, and air conditioning (HVAC) system 40. A person P can occupy living space 20.

[0063] refer to Figure 1A and 1BThe system 100 includes nodes including a network-enabled mobile robot 200, one or more wireless access points (WAPs) 164, a gateway and / or hub 110 that interconnects different networking methods to form a local private network 160 that interconnects network-enabled or "networked" electronic devices (including IoT devices) 120, 122, 124, network-enabled automation controller devices 126, 127, 128, a robot dock 140 that can also be a network-enabled automation controller device, and products that can combine multiple such functions. The private network 160 includes one or more wireless access points (WAPs) 164, and the gateway or hub 110 with a combined wireless range sufficient to cover all or most of the living space 20 defined by the living structure 10 or its surroundings. Networked devices connected to the private network 160 can communicate with the remote management service 150 through the WAN interface 170A and its associated WAN connection 170B through the router / firewall 162 to the public network 170. For example, the remote management service 150 can be a cloud computing device, the public network 170 can be the Internet, the WAN interface 170A can be DSL, DOCSIS, or a cellular modem, and its associated WAN connection 170B can be provided by an Internet service provider (ISP). The router 162, WAP 164, and / or modem 170A can be integrated into a single device in various configurations. The local user terminal 142, 400 can be connected (wired or wirelessly) to the private network 160. For example, the local user terminal 142, 400 can be a personal computer (PC), a smart phone, or a tablet computer. The remote user terminal 144, 400 can be connected to the remote server 150 and / or the private network 160 via the public network 170. The hub 110, robot 200, local user terminals 142, 400, and remote user terminals 144, 400 can all be accessed through a public network service implemented in the target device (e.g., a web server that presents a UI over a local network via a web browser on a client device) or via a dedicated client (e.g., a downloadable or pre-installed application software app), thereby enabling communication and control between nodes 110, 200, 140, 142, 144, 400, and 150, as described herein. A network entity as discussed herein is a machine and / or controller registered on a network that is assigned a unique address for sending and receiving communication signals and is available to other network entity machines and / or controllers on the same network or a connected network.

[0064] In some embodiments, "the same network" may refer to a collection of private addresses on a private IP (Internet Protocol) subnet behind a routing network entity 162 that provides network address translation (NAT) between the public Internet and the private network. Each network entity connected to the private network can infer the network addresses of other active network entities by observing the network communications of other active network entities and / or scanning possible IP subnets of the private network for responses. Some gateways / hubs provide network services that can enumerate devices associated with the gateway / hub and / or reachable through the gateway / hub. These technologies generate one or both of the IP address and / or MAC (Media Access Control) address of each active device. The Address Resolution Protocol (ARP) network service can map one type of address to another type of address. In some embodiments, a routine running on a processor of a network entity (such as, mobile robot 200) can collect the network address of another network entity (such as, networked devices 120, 122, 124, 126, 127, 128) and identify the type, manufacturer and / or model of that network entity, which type, manufacturer and / or model can be related to a physical location in the living structure 10, as discussed in more detail herein.

[0065] The robotic dock 140 may include a power source or be connected to a power source and include a charger operable to charge the battery of the mobile robot 200 when the mobile robot 200 is effectively docked on the robotic dock 140. The dock 140 may be an evacuation station including a motorized container that is actuatable to empty debris from the robot 200. In some embodiments, the dock 140 is connected (wired or wirelessly) to the private network 160 to enable or facilitate data transmission from the robot 200 to the private network 160 and / or from the private network 160 to the mobile robot 200. The robotic dock 140 may therefore be considered an automation controller device. In some embodiments, the robotic dock 140 communicates directly with the mobile robot 200 via wireless means including, but not limited to, Bluetooth, near field induction, IR, and / or radio communication signals. Each networked device 120, 122, 124, 126, 127, 128, 140 may include a wireless transceiver, such as a Wi-Fi transceiver, to communicate with the hub 110 and / or the private network 160 via the WAP 164. Although specific networked devices 120, 122, 124, 126, 127, 128, 140 are shown, more or fewer networked devices may be included in the operating environment 10 and may communicate with the private network 160.

[0066] The mobile robot 200 may be any suitable robot and associated computing device(s), and it should be appreciated that not all components, features, and functions described herein are required in a mobile robot according to embodiments of the present disclosure. Figures 2A-2C , the example mobile robot 200 includes a frame 210, a controller 220, a memory 222, a battery 224, a battery charger 226, a human machine interface (HMI) 228, a drive system 230, a mapping / navigation system 240, a service operating system 242 (also referred to herein as a "cleaning system" and a "cleaning head"), a wireless communication system 250, an IR transmitter 260 and environmental sensors 270A-H, a trash bin 242A (to store debris collected by the cleaning operation), a bin level sensor 242B, a dirt extraction sensor 242C (to detect a characteristic density of debris collected by the cleaning operation), an indicator light 274A, an audio transducer 274B and a cleaning mode selection switch or button 274C.

[0067] The environmental sensors 270A-270H may include a camera 270B mounted on the top surface of the mobile robot 200, such as Figure 2A . The camera 270B can be used to navigate the robot 200 and acquire images for other operational purposes. In some embodiments, the camera 270B is a visual simultaneous localization and mapping (VSLAM) camera and is used to detect features and landmarks in the operating environment and build an occupancy map based thereon.

[0068] like Figure 2B As shown in the bottom perspective view of FIG, the mobile robot 200 may also include a bumper 104, cliff sensors 195A-195D, and edge brushes 111 mounted or otherwise positioned on the periphery of the mobile robot housing 106. The housing 106 is Figures 2A-2B 104 is shown as having a square front portion on which the bumper 104 is mounted; however, in other embodiments, the housing may have a circular or annular shape. Casters 196 may be provided on the underside of the mobile robot 200. In some embodiments, the casters 196 may be positioned at the end of the mobile robot 200 opposite the cleaning head 242 with drive rollers / tracks 232A, 232B therebetween so that the cleaning head 242 is a cantilever arrangement. The mobile robot 200 may also include a camera 197 facing downward or toward the ground. In some embodiments, the mobile robot 200 may generally be positioned The mobile robot 200 may be configured in the manner of or include features from a floor cleaning robot and / or a robot as described in U.S. Patent No. 7,024,278 and U.S. Published Application No. 2007 / 0250212, the disclosures of which are incorporated herein by reference as appropriate. In other embodiments, the mobile robot 200 may be generally configured as an autonomous patrol robot that includes a telescopic mast on which one or more elements associated with the sensor elements 270A-H and / or the wireless communication circuitry or system 250 are mounted or otherwise operably associated.

[0069] The controller 220 may include one or more processors of any suitable configuration. The (one or more) processors may include one or more data processing circuits, such as general-purpose and / or special-purpose processors (such as microprocessors and / or digital signal processors) that can be co-located or distributed in one or more networks. The processor is configured to execute program code stored in the memory 222 described below as a computer-readable storage medium to perform some or all of the operations and methods described above for one or more embodiments. The memory 222 represents one or more storage devices according to some embodiments of the present disclosure, which contain software and data for facilitating the operation of the robot. The memory 222 may include, but is not limited to, the following types of devices: cache, ROM, PROM, EPROM, EEPROM, flash memory, SRAM, and DRAM. Therefore, the processor communicates with the controller 200, the memory 222, the cleaning system 242, and the drive system 230.

[0070] The drive system 230 may include any suitable mechanism or system for actively and controllably moving the robot 200 through the living space 20. According to some embodiments, the drive system 230 includes one or more rollers, one or more tracks 232A, 232B, and one or more onboard (i.e., carried by the mobile robot 200) electric motors 234 (collectively referred to herein as "drives" or "drive system") operable by the controller 220 to transport the robot 200 across the floor of the operating environment 10.

[0071] The service operation system 242 may be optional in some embodiments and is operable to perform service operations in the living space 20. According to some embodiments, the service operation system 242 includes a floor cleaning system that cleans the floor surface of the living space 20 as the robot 200 passes through the space 20. In some embodiments, the service operation system 242 includes a suction head and an onboard vacuum generator to vacuum clean the floor. In some embodiments, the service operation system 242 includes an end effector such as (but not limited to) a sweeping or mopping mechanism, one or more rotating brushes, a roller, a wet or dry fixed or oscillating and / or vibrating cloth, or a multi-layer pad assembly.

[0072] The wireless communication system 250 includes a wireless communication transceiver or module 252 and an associated antenna 254 to enable wireless communication between the robot 200 and various other networked devices 120, 122, 124, 126, 127, 128 in the operating environment 10, as well as network segments served by the WAPs, gateways, and hubs that comprise the private network 160 of which the mobile robot 200 constitutes a node. For example, the wireless communication transceiver or module 252 may be a Wi-Fi module.

[0073] In some embodiments, the robot 200 can communicate wirelessly directly with the dock 140 using narrowband or broadband RF communication. For example, if the robot 200 is not equipped with a transmitter compatible with the WAP 164, the robot 200 can communicate with the dock 140, which can in turn relay the data from the robot 200 onto the private network 160 and forward it to the desired network entity (such as the remote management server 150). In some embodiments, the dock 140 includes a bridge device that receives and converts RF signals from the robot 200 and relays them to the router 162 in a format supported by the router for delivery to the remote management server 150 or another device in the private network 160. In some embodiments, the dock 140 includes a low-power mesh data network that employs a mesh topology, in which RF communication signals are relayed from node to node between the mobile robot 200 and the dock 140. In this case, the networked devices 120, 122, 124, 126, 127, 128, 140 and the range extender module (if any; not shown) can serve as mesh nodes. Likewise, the mobile robot 200 may act as a node that relays signals between the dock 140 and other nodes, such as network-enabled sensor devices 120 , 122 , 124 , 126 , 127 , 128 , 140 and range extenders.

[0074] The remote management server 150 may be any suitable computing device(s), computer server(s), or cloud service(s), or a combination thereof, and it should be recognized that not all components, features, and functions described herein are required in the remote server(s) according to embodiments of the present disclosure. Figure 3 In an example of a remote management server 150, the remote management server 150 includes one or more network interfaces 330, a processor circuit system ("processor") 310, and a memory 320 containing program code 322. The network interface 330 can be configured to communicate with any associated (one or more) available resource servers and / or data repositories via one or more wired and / or wireless networks. The processor 310 may include one or more data processing circuits, such as general and / or special-purpose processors (such as microprocessors and / or digital signal processors) that can be co-located or distributed in one or more networks. The processor 310 is configured to execute program code 322 in the memory 320, which is described below as a computer-readable storage medium, to perform some or all of the operations and methods described above for one or more embodiments. The memory 320 represents one or more memory devices containing software and data for facilitating operations for managing the robot according to some embodiments of the present disclosure. The memory 320 may include, but is not limited to, the following types of devices: cache, ROM, PROM, EPROM, EEPROM, flash memory, SRAM, and DRAM. Therefore, Figure 3 The remote server 150 illustrates a hardware / software architecture that can be used to manage and / or distribute information obtained by a mobile robot in accordance with some embodiments described herein. However, it will be understood that embodiments of the present disclosure are not limited to this configuration, but are intended to encompass any configuration capable of performing the operations described herein. Thus, the remote management server 150 can more generally represent any device or device network that provides remote management services and has the ability to receive, process, enhance, store, and transmit data to or from one or more robots 200 in service using any appropriate combination of relevant methods and techniques. The remote management service 150 also provides additional processing to support other clients with different requirements than the robot 200, such as visualization and other rich UI interactions for presentation on the user terminals 142, 144. In some embodiments, the functions assigned to the remote management service 150 herein may also be performed in whole or in part locally on the robot, or in collaboration / coordination with a group of communicating robots and / or servers.

[0075] The user terminals 142, 144 may be any suitable fixed or mobile computing device(s), including desktop computers, laptop computers, and "smart phones," and it will be appreciated that not all components, features, and functions described herein are required in a user terminal according to embodiments of the present disclosure. Figure 4 In some embodiments, the example user terminal 400 is configured to transmit and / or receive communication signals through multiple wired and / or wireless communication interfaces. For example, the transceiver 440 of the user terminal 400 according to some embodiments may include a cellular communication module, an infrared (IR) communication module, a global positioning system (GPS) module, a WLAN module, a wireless personal area network (WPAN) module (such as a Bluetooth, Wi-Fi, ZigBee and / or Z-wave module), and / or other types of communication modules.

[0076] The user interface 410 of the user terminal 400 includes a display 408, such as a liquid crystal display (LCD) and / or an organic light emitting diode (OLED) display. The user interface 410 may optionally include a keypad 402 or other user input mechanism on the housing of the user terminal 400. In some embodiments, the display 408 may have a touch screen function to replace and / or supplement the keypad 402. The user interface 410 may also include a microphone 406 and an earphone / speaker 404. When the earphone / speaker 404 is placed against the user's head, the housing may be designed to form an acoustic seal against the user's ears.

[0077] The keypad 402, display 408, microphone 406, speaker 404, and camera 424 may be coupled to a processor 427, such as a microprocessor or microcontroller that may be configured to control the operation of the user terminal 400. The user terminal 400 may also include a transceiver 440 and a memory 428 coupled to the processor 427. Other electronic circuitry, such as a WLAN communication interface, a Bluetooth interface, a GPS interface, a digital signal processor, etc., may also be included in the electronic circuitry of the user terminal 400.

[0078] The memory 428 may be a general purpose memory used to store program instructions 422 for the processor 427 and data such as audio data, video data, configuration data, and / or other data that may be accessed and / or used by the processor 427. The memory 428 may include non-volatile read / write memory, read-only memory, and / or volatile read / write memory. Specifically, the memory 428 may include read-only memory in which basic operating system instructions are stored, non-volatile read / write memory in which reusable data (such as configuration information, directory information, and other information) may be stored, and volatile read / write memory in which short-term instructions and / or temporary data may be stored.

[0079] The transceiver 440 includes a transmitter circuit 442, a receiver circuit 444, and a modem 446 that cooperate to transmit and receive radio frequency signals to and from a remote transceiver via antenna arrays 450A and 450B. The radio frequency signals transmitted between the user terminal 400 and the remote transceiver may include both traffic and control signals (e.g., paging signals / messages for incoming calls) used to establish and maintain communication with another party or another destination. More particularly, in cooperation with the processor 427, the transceiver 440 may be configured to communicate according to a variety of radio access and / or wireless networking technologies, including (but not limited to) cellular, WLAN (including 802.11), WiMAX (Worldwide Interoperability for Microwave Access), Wi-Fi, Bluetooth, ZigBee, and / or Z-wave. Other radio access technologies and / or frequency bands may also be used in embodiments according to the present disclosure.

[0080] Reference again Figure 2C The mobile robot 200 may include localization circuitry that constructs a metric and distance-based map of walls and obstacles (e.g., using laser rangefinders, sonar, radar, triangulation, flight, or phase difference calculations) and / or an occupancy map of free space (e.g., traversable ground space or ground space not occupied by objects or fixtures), and may localize the robot on the map(s) using techniques such as scan matching, ICP (Iterative Closest Point), and / or RANSAC (Random Sample Consensus). Additionally or alternatively, the mobile robot 200 may include localization circuitry that constructs a feature-based constellation or topological map of fingerprints of features, landmarks, fiducials, and / or beacons within the occupancy map of free space (e.g., using a camera or point cloud generation 3D scanner and feature transformations for identifying, storing, and distinguishing natural or artificial keypoints, features, and / or landmarks), and may localize the robot on this occupancy map using techniques such as VSLAM (vision-based / vision-simultaneous localization and mapping). In either case, an end user can associate a unique identity linked to a unique room or area (shown as Zone 1, Zone 2, Zone 3) with a household room type or unique room tag ("Living Room") via a user interface of any network entity. The occupancy map (also referred to herein as a "floorplan" of the operating environment) can include data defining a plurality of surface locations of the environment 10 (e.g., by pixels), each surface location having a value corresponding to whether the pixel location corresponds to a surface location occupied, traversed, or unexplored by the mobile robot 200. The positioning circuitry can be defined by input from one or more sensors 270A-270H of the mobile robot 200, which can be used by the controller 220 to perform positioning in the operating environment 10.

[0081] More specifically, one or more of the positioning sensors 270A-270H are configured to detect sensor readings from objects located in the operating environment 10, and the controller 220 is configured to determine the current pose ("pose" includes absolute or relative position and optionally absolute or relative orientation) of the mobile robot 200 based on the positioning data detected by the positioning sensors 270A-270H with reference to the observed object ("object" includes not only physical objects including observable features, but also surface "objects" formed by optically or other detectable surface features, such as corners, lines, patterns). The pose of the object can also be determined. The mobile robot 200 can also be configured to associate the robot's pose (or position) with a room identifier, which is specifically associated with the observed object or its fixed pose in the room or a pose observable on a component of the room (walls, ceiling, lighting, doorways, furniture, etc.), as indicated by an occupancy map.

[0082] In addition, reference Figure 2C, the mobile robot may include wireless signal mapping circuitry or capabilities that collects data as input to a software-defined process that constructs and refines a signal coverage map, including data defining a coverage pattern and signal strengths of wireless communication signals acquired or otherwise detected at corresponding locations on a surface. The signal coverage map may also be combined with an occupancy map obtained via a technique such as VSLAM to localize the robot 200. More particularly, during navigation of the operating environment 10 (e.g., while performing a cleaning task), the controller 220 of the mobile robot 200 is configured to acquire wireless communication signals at corresponding locations in the operating environment via the wireless communication circuitry 250. These wireless communication signals may be transmitted by one or more other electronic devices local to the environment 10 (such as networked devices 120, 122, 124, 126, 127, 128). The coverage patterns of wireless communication signals sensed within the operating environment can be associated with corresponding locations and stored in memory 222 to provide a "terrain" of wireless signals covering the operating environment, which is correlated with a floor plan sensed by other means, including but not limited to visual simultaneous localization and mapping (VSLAM) techniques. This "terrain" can represent signal characteristics, including received signal strength, signal-to-noise ratio (SNR), data throughput, and / or multipath distortion at corresponding locations on the floor plan, and can be visually represented in a 3-dimensional topological format. Received wireless communications and / or their signal strength / SNR / data throughput / distortion combinations can identify unique locations (or a small set if unsure) within the operating environment and / or can be used to discard or eliminate potential location candidates that were otherwise inferred to be unlikely given the sensed RF "terrain." Thus, the controller 220 is configured to determine the position of the mobile robot 200 within the operating environment 10 based not only on occupancy data obtained via the positioning sensors 270A-270H, but also on signal coverage data obtained via the wireless communication circuitry 250, so that positioning and repositioning can be performed in a less time-consuming manner.

[0083] For example, if the mobile robot 200 is picked up and moved to a "new" location in the operating environment, the controller 220 can determine the new location of the mobile robot 200 in the operating environment based on the received signal strength, the slope of the received signal strength, and / or the direction of change in the slope at the new location, as indicated by the "terrain" map. The controller 220 can also, alone or in combination with vision positioning technology, determine and / or verify the current pose of the mobile robot 200 based on the direction in which the slope of the signal strength changes as it moves to the new location.

[0084] In some embodiments, the controller 220 of the mobile robot 200 can be configured to determine the position (including the current pose) of the mobile robot 200 based on the correlation of the occupancy data and the signal coverage data, e.g., relative to the position of one or more of the devices 120, 122, 124, 126, 127, 128 in the operating environment 10. In some embodiments, the correlation of the occupancy data and the signal coverage data can be performed by the controller 220 of the mobile robot 200 itself. In other embodiments, the occupancy data and the signal coverage data obtained by the mobile robot 200 can be transmitted to the remote server(s) 150, which can perform the correlation of the occupancy data and the signal coverage data and can transmit the data to and from the mobile robot 200 for positioning.

[0085] refer to Figure 5A 、 5B and 5C, operations are performed by the mobile robot 200 and / or the remote server 150 to perform operations such as Figure 1B Mapping the Wi-Fi fields of one or more electronic devices in an operating environment such as the illustrated environment 10 to determine the respective locations of the electronic devices in the operating environment can be performed while the mobile robot 200 is navigating the operating environment for a primary task (e.g., cleaning and / or patrolling tasks). Figures 5A-5C , for acquiring Wi-Fi signals. Specifically, during navigation of an operating environment, the wireless communication circuitry 250 of the mobile terminal may, for example, use single-channel or multi-channel scanning to acquire Wi-Fi signals indicating signal coverage at multiple locations within the operating environment. A controller of the mobile robot 200 (and / or a remote server 150 communicating therewith) may determine corresponding signal characteristics (including, but not limited to, signal strength, signal-to-noise ratio, data throughput, and / or multipath distortion) of the wireless communication signals at corresponding locations within the operating environment based on the coverage pattern, and may determine corresponding locations of electronic devices local to the operating environment based on the corresponding signal characteristics. Furthermore, after analyzing the data collected during the initial navigation of the environment (referred to herein as the "primary mission"), additional specialized missions (or modifications to subsequent primary missions) may be scheduled to capture more data to refine the derived information (e.g., to capture a higher density of information in areas where the initially collected data was determined to be ambiguous or of particular interest). The determined signal characteristics of the Wi-Fi field may be visually represented in various forms (e.g., for display via the user terminal 142, 144, 400). In some embodiments, the determined signal characteristics of each detected device's Wi-Fi field may be visually represented individually (e.g., to facilitate location determination), while in other embodiments, the determined signal characteristics may be visually represented collectively (e.g., to detect interfering obstacles / structures).

[0086] For example, Figure 5A is a 2-dimensional graph illustrating a coverage pattern 500a of Wi-Fi signals acquired by the wireless communication circuit 250 of the mobile robot 200 at corresponding locations in the operating environment (represented by X and Y axes), wherein the signal strength of the received Wi-Fi signals (also referred to herein as received signal strength) is represented by colors and / or brightness levels that vary according to the received signal strength at the corresponding locations in the operating environment (also referred to herein as a "heat map"). Figure 5B and 5C is a 3D topology diagram, which illustrates the coverage patterns 500b and 500c obtained by the wireless communication circuit 250 of the mobile robot 200 at various positions (represented by the X and Y axes) in the same operating environment, and the Z axis represents the signal strength of the Wi-Fi signal received at the corresponding position in the operating environment. Figures 5A-5C The graph of FIG represents the signal strength of a single electronic device from which a Wi-Fi signal is transmitted as a function of location. As such, the location with the highest signal strength (denoted by Figure 5A The highest brightness / intensity or Figure 5B and 5C The highest peak in the topology in the figure (shown as the highest peak in the topology) indicates the location of the corresponding electronic device from which the Wi-Fi signal was transmitted. In other words, the peak corresponds to the emission point of the signal. Similar graphs can be generated for other electronic devices in the operating environment. In some embodiments, multiple graphs can be combined or overlaid so that corresponding peaks in the overlaid topology graph indicate corresponding locations of multiple electronic devices.

[0087] also, Figure 5B and 5C The changes in slope and slope direction shown in the topologies of coverage patterns 500b and 500c indicate changes in signal strength at relative locations in the operating environment. Figure 5B and 5C The slope angles (and / or rates of change of the slopes) shown in the topologies of coverage patterns 500b and 500c can also help resolve the pose of mobile robot 200. For example, alone or in combination with vision positioning techniques (e.g., using camera-based recognition), the direction and rate at which the slope of the signal strength changes relative to the peak signal strength (which indicates the location of the electronic device transmitting the signal) can allow mobile robot 200 to more quickly determine its pose.

[0088] By extending the surrounding slopes in the collected data to predict where the peak should be, Figure 5B and 5CThe received signal strength peaks and changes in their slope and slope direction shown in the topology of coverage patterns 500b and 500c can also be used to identify the location of a network-enabled device even if it is outside the plotted area. This can address situations where the collected data does not indicate a clear "peak" for a network-enabled device within the plotted area. Such changes in signal strength can also be used to identify locations of the operating environment with weak or no signal coverage ("dead spots"), and / or structures in the operating environment that may contribute to or cause variations in signal strength ("shadows"). For example, in Figure 5B and 5C A rapid change in slope between the operating environment locations shown in topology maps 500b and 500c may indicate the presence of one or more structures at or near those locations that may interfere with signal transmission. In other embodiments, described in more detail below, one or more suggestions for improving signal coverage (such as moving the Wi-Fi router to a different location and / or placement / or orientation) may be provided based on an analysis of the signal strength changes indicated by topology maps 500b and 500c across the surface of the operating environment.

[0089] Operations may be performed at multiple frequencies / channels and / or technologies to generate corresponding coverage patterns for each device from which wireless communication signals are detected in the operating environment. For example, Wi-Fi signal data acquired by the mobile robot 200 may also be used to resolve persistence maps by using Wi-Fi device locations indicated by detected signal strengths as anchor points to merge corresponding occupancy maps generated from VSLAM data acquired at different times and / or under different lighting conditions.

[0090] refer to Figure 6A 、 6B and 6C, performed by the mobile robot 200 and / or the remote server 150 to convert the signal coverage data 500a (which is based on the mobile robot 200 during navigation in the operating environment (such as Figure 1B The wireless communication fields detected in the environment 10 shown are correlated with occupancy data 600 (which is based on data detected by positioning sensors 270A-270H in the operating environment during navigation of the mobile robot 200) to generate a plurality of graphs 605 indicating coverage patterns of wireless communication signals of each network-enabled device at corresponding locations in the operating environment. More specifically, Figure 6C Such an example of Figure 605 illustrates the Figure 6B The occupancy or floor plan 600 is directly related to and overlaid upon Figure 6A The signal strength of the wireless communication signal coverage pattern 500a.

[0091] Figure 6AThe wireless communication coverage pattern 500a corresponds to a specific device or device type in the operating environment. For example, the wireless communication circuitry 250 of the mobile robot 200 can be configured to detect the presence of one or more Wi-Fi routers located and / or transmitting in the operating environment 10 based on single-channel or multi-channel scanning. The mobile robot 200 can also utilize the wireless communication coverage pattern 500a to identify areas of the operating environment 10 where signal coverage is weak or non-existent (i.e., "dead spots") and, in some embodiments described in more detail below, provide recommendations for improving coverage.

[0092] Multi-channel scanning can be used to build a list of available wireless network access points in the operating environment 10. Figure 1B In some embodiments, each multi-channel scan may take approximately 3 seconds to complete. Specifically, since some access points may transmit a beacon approximately every 104 milliseconds (ms) to 110 ms, the wireless communication circuit 250 should perform a scan of each channel for more than 104 ms. The wireless communication circuit 250 may also take approximately 100 ms to 120 ms to change Wi-Fi channels, and 2.4 GHz z The frequency band has 11 channels in the United States (13 in Europe and 14 in Japan), so it takes about 2990ms (13 channels x (110ms + 120ms)). During the movement of the mobile robot 200 in the environment 10, the samples from the multi-channel scan may be too sparse to define Figure 6A 5. The mobile robot 200 may monitor only one channel during a mission, but may rotate which channel is being monitored on each mission, thereby collecting information over an extended period of time, rather than extending the time it takes to complete a single mission.

[0093] A single-channel scan can be performed in a manner similar to a multi-channel scan, but can be performed only for a specific channel of the WAP 164. For example, a single-channel scan can be locked to a specific channel or performed only for a specific channel. Thus, a single-channel scan can report all access points on the same channel as the WAP 164. The samples from the single-channel scan can be of sufficient density to define a coverage pattern 500a when the mobile robot 200 is moved in the environment 10, so that the mobile robot 200 may not need to be paused during the single-channel scan.

[0094] The wireless communication circuitry 250 of the mobile robot 200 may also include one or more chipsets configured to operate in a "monitor mode" to detect a plurality of networked devices in the operating environment (such as in Figure 1B In some embodiments, monitor mode can be implemented using a tool such as Wireshark. TM , or a PCAP-compatible network sniffing tool such as , to inspect wirelessly transmitted packets. For example, the wireless communication circuitry 250 can be configured to passively inspect the header information of all received packets, including data packets not addressed to this device and link-level management packets that are not typically passed to a computer, in a manner similar to the "promiscuous mode" available on some Ethernet adapters. In this way, all devices transmitting wireless communication signals within range of the wireless communication circuitry 250 (including devices that transmit data packets relatively infrequently) can be visible to the mobile robot 200. Through this intensive sampling of a relatively large number of packets, monitor mode can allow for very intensive data collection even when the mobile robot 200 is moving through the operating environment 10. Furthermore, link-level packets can contain a wealth of information (which can help identify the type / model of the transmitting device). In some embodiments, the information streams can be merged and stored in a common file, for example using a packet capture (PCAP) format, in which the robot position is inserted as a forged network packet, and can be stored in the mobile robot's memory 222 for use by the controller 220 and / or transmission to the remote server(s) 150. In other embodiments, the captured information may be stored in separate files, with each entry timestamped at the time it was received from a common time source, so that the separate streams can be correlated as needed during later processing.

[0095] In response to a coverage pattern detected by scanning the operating environment via the wireless communication circuitry 250, the mobile robot 200 (alone or in cooperation with the remote server 150) can also be configured to identify one or more of the networked devices 120, 122, 124, 126, 127, 128 by type, manufacturer, and / or model of the device, for example, based on addressing and / or network service information determined by passively listening or by actively probing the device. For example, the mobile robot 200 can be configured to passively listen to transmitted packets via the wireless communication circuitry 250 to determine a list of media access control (MAC) addresses in the operating environment. Such a hardware (MAC) address can indicate the manufacturer of the device, as conventionally the first 24 to 36 bits (of 48 bits) of a MAC address are assigned to a specific manufacturer by a central authority. The Address Resolution Protocol (ARP) can be used to translate the MAC address into an IP address for the device on the private network 160 (or vice versa). Devices can also be identified as mobile or stationary, for example, by identifying the type / model of the device (e.g., a wall-mounted thermostat, security camera, or smoke detector) implicitly or by heuristically monitoring changes (or lack thereof) in the signal strength / coverage pattern derived from wireless communication signals. For example, when a mobile robot is stationary, changes in received signal strength can indicate motion of the device. That is, stationary devices can be distinguished from mobile devices based on the consistency of the wireless signal strength / coverage pattern received from the stationary device.

[0096] Additionally, the mobile robot 200 can be configured to actively probe the network via the wireless communication circuitry 250 to discover and identify active devices, both wired and wireless. The network probe service can be used to narrow down possible candidates for a model / type to specific models based on the manufacturer indicated by the MAC address. For example, a Nest TM Branded device specifically identified as a Nest Thermostat TM Nest Protect TM or Nest Camera TM . Web services provided by networked devices may also reveal "friendly" (i.e., user-understandable) names assigned by the user, which may indicate information about the placement or intended use of the device from the user's perspective. For example, a lighting device assigned the name "kitchen" by the user may indicate that the lighting device is located in the kitchen. In some embodiments, these network services may be provided by the gateway / hub itself on behalf of the networked devices, allowing the devices themselves to be simpler. For example, one hub may be responsible for many light bulbs and / or wall switches implemented using a wireless protocol intended for IoT devices, such as Zigbee or Z-Wave.

[0097] In addition, the mobile robot 200 can be configured to create (or enhance) a "fingerprint" (which can refer to a unique combination of detected characteristics) of the detected device by actively probing the services it provides to other network entities via the wireless communication circuitry 250. Specific embodiments may use various techniques to distinguish between manufacturers, their products, and versions of products. These may include (but are not limited to) port scanning to identify exposed services, pattern matching with received responses (such as HTML content returned by an embedded web server, the initial logon banner from a terminal connection, etc.), and interacting with certain discovered services to extract more structured information (e.g., properties advertised by the device via a network "discovery" protocol such as UPnP or mDNS). Discovery can also be performed by requesting a gateway / hub (such as Figure 1B Devices can be identified and / or enumerated indirectly by using information stored by the hub 110 about the devices to which they are coupled. For example, by simply asking a SmartThings TM A hub that identifies devices coupled to it as Philips Hue TM Light bulb.

[0098] refer to Figure 7 Based on the above and / or other device identification techniques, the mobile robot 200 and / or the remote server 150 can be configured to automatically (i.e., without human intervention) determine the location of each identified device based on the coverage data 500a. Specifically, the mobile robot 200 and / or the remote server 150 can use the location on the occupancy map 600 corresponding to the corresponding highest received signal strength of the coverage data 500a for each identified device to automatically generate a persistence map 705 to indicate the location of the networked device 120 (illustrated as the Nest Thermostat) TM Devices), 122 (Arlo Q TM Network Camera), 124 (Roku TM Set-top box devices), 126 (Amazon Echo shown TM devices), 128 (Google Home TM Devices), 142 (Apple MacBook shown) TM ), 144 (Apple iPhone TM ) and 162 (Netgear TMFurthermore, based on the identification of the device and its location in the operating environment, the mobile robot 200 and / or the remote server 150 may also be configured to determine the likely placement and / or orientation of the device. For example, a thermostat device (such as a Nest TM Ecobee TM or Honeywell TM Thermostat devices) can be identified as being on the wall, at approximately eye level, at their determined location; smart plugs (such as Wemo TM TP-Link TM 、D-Link TM Orvibo TM Smart plugs) can be recognized as being at the socket level at their determined location; and light bulbs (such as Lifx TM 、TCP TM or Flux TM A light bulb) can be identified as hanging from the ceiling at a determined location.

[0099] Based on the determination of the possible placement of the identified devices at the corresponding locations, the mobile robot 200 and / or the remote server 150 can be configured to associate the placement and location with an image of the corresponding device, for example, as captured in positioning data detected by the mobile robot 200 during environment navigation. For example, the mobile robot 200 can capture visual positioning data (such as an image or a 360° panoramic image) at the corresponding location in the operating environment 10 via the camera 270B and can store the visual positioning data in the memory 222. The determined locations of the networked devices 120, 122, 124, 126, 128, 142, 144, and 162 in the operating environment 10 and their possible placement at the determined locations can allow the captured images at those corresponding locations to be matched with the identification of the networked devices 120, 122, 124, 126, 128, 142, 144, and 162. Therefore, according to embodiments of the present disclosure, identifying each device and its approximate location can improve the accuracy and / or utility of machine-based vision.

[0100] Figure 8A and 8B FIGURE 1 illustrates a specific example of mapping the Wi-Fi field transmitted by networked electronic devices in a "smart" home according to an embodiment of the present disclosure. Figure 8A , Figures 800a1 to 800a6 illustrate a coarse sampling of Wi-Fi signals in a mapped portion of a single-story residence, where the Wi-Fi signal strength is mapped to a floor plan or occupancy map obtained from positioning data collected during house navigation by a mobile robot.

[0101] Six separate graphs 800a1 through 800a6 indicate the signal patterns detected from six different wireless access points located inside or near a house. Figure 8A and 8B In the example shown, the width of the house is approximately 60 feet horizontally. For these specific plots 800a1-800a6, the mobile robot uses its one or more positioning sensors to perform an initial mapping of various parts of the house. It then uses its wireless communication circuitry to perform a multi-channel network scan at the corresponding locations within the house, approximately once every 10 square feet. Each multi-channel scan takes several seconds to complete, and a subset of the locations are plotted on the plot. The coverage patterns of the six different Wi-Fi access points shown in plots 800a1 through 800a6 are different, and the relative position of each access point is visually represented in plots 800a1 through 800a6 by its relative brightness. For example, plot 800a4 shows a cellular hotspot located in the rightmost room of plot 800a4. The bright spots and unevenness in the signal are artifacts of the sparse sampling method. In contrast, plot 800a6 shows the Wi-Fi signal from the primary access point, the home Wi-Fi router. The signal is generally strong, and the access point is located in the center of the house, matching the brightest area in plot 800a6.

[0102] In addition to running Figure 8A In addition to the complete multi-channel network scan shown, Figure 8B An alternative process for measuring Wi-Fi signal strength is shown in , which is by performing a single channel network scan to continuously record packets for a single channel and frequency. Combined with the "Monitor Mode" described previously, the single channel scan can provide significantly more measurements (10x to 100x more than the multi-channel approach) because the mobile robot can continuously listen while it is moving and mapping the house. Figure 8B An example of such a continuous signal scan for the main access points of a house is shown in the full-surface topology map 800b of FIG. The increased sampling provided by the continuous signal scan is shown by the smooth terrain in FIG. 800b. FIG. 800b indicates the location of the access points by following the change in the slope of the topology until the highest point on the graph. Similarly, the change in the direction and height of the slope of the topology shown in FIG. 800b can be used to identify operating environments with weak or no signal coverage and / or structures in the operating environment (e.g., walls) that may cause reduced or lost signal coverage. Moreover, in some embodiments, by continuously scanning and capturing signals from multiple locations, the noise component of the signal can be reduced or otherwise evened out.

[0103] Figure 9-11 is a flowchart illustrating operations that may be performed by at least one processor of a computing device according to the wireless communication signal mapping technology described in embodiments of the present disclosure. Figure 9-11 The operations may represent executable routines or instructions stored in a non-transitory computer-readable medium (such as storage devices 222, 320, 428 described herein) and may be executed by one or more processors (such as processors 220, 310 and / or 427 described herein) to implement the specified routines or instructions.

[0104] In particular, Figure 9 The operation can be performed by Figure 1B The server 150, the mobile robot 200 and / or one or more processors of the user terminal 142, 144 are executed. Figure 9 In response to the mobile robot navigating the operating environment, occupancy data for the operating environment is received by the mobile robot at block 905. The occupancy data is based on positioning data detected by at least one positioning sensor of the mobile robot during navigation of the operating environment, for example, using VSLAM technology. In some embodiments, the occupancy data may be based on positioning data obtained by multiple mobile robots in the operating environment.

[0105] Likewise, in response to the mobile robot navigating within the operating environment, signal coverage data for the operating environment is received at block 910. The signal coverage data is based on wireless communication signals acquired by at least one wireless receiver of the mobile robot, for example, by scanning one or more communication frequencies at corresponding locations within the operating environment. The wireless communication signals are acquired by one or more network-enabled electronic devices (such as Figure 1B One or more of the networked devices 120, 122, 124, 126, 128, 142, 144 and 162) transmit.

[0106] By correlating the occupancy data (received at block 905) with the signal coverage data (received at block 910), a graph is generated at block 915 that indicates coverage patterns of wireless communication signals at respective locations in the operating environment. The coverage patterns indicate respective signal characteristics (including, but not limited to, signal strength, signal-to-noise ratio, data throughput, and / or multipath distortion) of the wireless communication signals at respective locations in the operating environment. Thus, the respective locations of the electronic devices in the operating environment can be determined based on the coverage patterns, for example, based on peaks in signal strength and / or changes in slope / direction of signal strength at the respective locations in the operating environment. The graph can be updated in response to occupancy data and / or signal coverage data obtained by the mobile robot during one or more subsequent navigations through the environment, such that the graph defines a persistent graph that represents a set of received signal strengths for one or more networked devices over a predetermined period of time as the mobile robot navigates the operating environment.

[0107] Figure 10is a flow chart illustrating operations that may be performed by, for example, at least one processor of a mobile robot, such as the processor 220 of the mobile robot 200 described herein. As discussed above, the processor is coupled to a drive configured to navigate the mobile robot in an operating environment, a positioning circuit comprising at least one positioning sensor, a wireless communication circuit, and a memory comprising a computer-readable storage medium storing computer-readable program code defining executable routines or instructions. The wireless communication circuit may be communicatively coupled to one or more remote servers and / or user terminals via a local area network or a wide area network. The processor is configured to execute routines stored in the memory to perform Figure 10 operation.

[0108] In particular, refer to Figure 10 At block 1005, the driver operates to navigate the mobile robot to access locations within an operating environment. For example, the operating environment may include a house or a room within a house. The driver may operate based on a navigation routine that includes one or more of: a coverage pattern (such as rows connected back and forth), a perimeter and area discovery pattern (such as wall and / or obstacle following or skirting patterns), system error protection measures (such as random bounces and other randomizations), and / or a point-to-point or area-to-area trajectory sequence.

[0109] At block 1010, in response to operating an actuator to navigate the mobile robot within an operating environment, positioning data is detected via at least one positioning sensor of the mobile robot. The positioning data may include distance measurements from range sensors, occupancy and obstacle determinations from obstacle and proximity sensors, and feature and landmark determinations from cameras and other pattern / image / reference viewing sensors. The positioning data may be detected in a surface mapping routine, for example, using VSLAM techniques to record 2D occupancy data representing the location of the operating environment and / or the pose of the mobile robot therewith.

[0110] In response to operating the actuator to navigate the mobile robot within the operating environment, wireless communication signals are acquired via at least one wireless receiver of the mobile robot at block 1015. The wireless communication signals may be transmitted by one or more network-enabled electronic devices local to the operating environment. The wireless communication signals may be acquired in a signal mapping routine to record signal coverage data (including coverage patterns and corresponding signal characteristics) for the wireless communication signals at corresponding locations within the operating environment. The wireless communication signals may be acquired periodically, aperiodically, or continuously at corresponding locations within the operating environment using single-channel scanning, multi-channel scanning, and / or network monitoring.

[0111] Based on a correlation between the positioning data detected by the positioning sensor (at block 1010) and the wireless communication signals acquired by the wireless receiver (at block 1015), the position of the mobile robot within the operating environment is determined at block 1020. For example, the identity and location of the mobile robot can be determined based on wireless communication signals received from one or more network-enabled electronic devices in the operating environment, and the position of the mobile robot can be determined relative to the location(s) of the electronic device(s). Furthermore, changes in the signal strength of the wireless communication signals received at corresponding locations in the operating environment (e.g., as indicated by a rate of change in the topology of a 3-dimensional graph in which the received signal strength is correlated with occupancy information from the positioning data) can be further used to determine the position of the mobile robot. That is, the positioning of the mobile robot is performed not only based on the positioning data acquired via its positioning sensor, but also based on the wireless communication acquired via its wireless receiver.

[0112] The mobile robots described herein can be uniquely configured to map wireless signal coverage (such as Wi-Fi coverage) in a house or other operating environment because the mobile robots already include positioning circuitry that operates independently to navigate the mobile robot through the environment. As such, intermittently or concurrently with executing the surface mapping routine, the mobile robot(s) can collect multiple samples of received signal characteristics, such as received signal strength indications (RSSIs), via corresponding wireless communication circuitry in many different locations of the operating environment, which can be used along with temporal and geographic averaging of the data to construct an accurate wireless signal field for each device in the operating environment.

[0113] Figure 11 4 is a flow chart illustrating operations that may be performed by at least one processor of a user terminal, such as the processor 427 of the user terminal 400 described herein. As discussed above, the processor is coupled to a receiver, a user interface, and a memory comprising a computer-readable storage medium storing computer-readable program code defining executable routines or instructions. The receiver may be communicatively coupled to the mobile robot and / or a remote server in communication therewith via a local area network or a wide area network. The processor is configured to execute routines stored in the memory to perform Figure 11 operation.

[0114] In particular, refer to Figure 11At block 1105, map information is received via a receiver of a user terminal, the map information being based on a correlation of positioning data detected by positioning sensors of one or more mobile robots in response to navigation of the mobile robots in an operating environment with wireless communication signals acquired by wireless receivers of the mobile robots. The wireless communication signals may be transmitted by one or more network-enabled electronic devices local to the operating environment. The map information may be received directly from the mobile robots or from a remote server in communication with the mobile robots.

[0115] At block 1110, one or more visual representations of the graph information are displayed via a user interface of the user terminal. In particular, the graph information may be presented in a format that may be more meaningful to a human user or operator. For example, the visual representation displayed at block 1110 may be similar to the one shown above with reference to FIG. Figure 5A 、 5B , 5C, 6A, 6B, 6C, 7, 8A, and / or 8B. In certain embodiments, the visual representation may include an occupancy map indicating physical characteristics of the operating environment (including obstacles detected in the operating environment by the positioning sensor(s) of the mobile robot(s)) and / or a signal coverage map indicating electrical characteristics of the operating environment (including respective signal strengths of wireless communication signals acquired by the wireless receiver(s) of the mobile robot(s) at respective locations in the operating environment), which may be displayed as a 2-dimensional or 3-dimensional representation. A corresponding occupancy map and / or signal coverage map may be generated and displayed for each detected electronic device in the operating environment from which wireless communication signals are received.

[0116] For example, a signal coverage map can be displayed as a 2-dimensional representation (or "heat map") that indicates the respective signal strengths of wireless communication signals by displaying different colors, brightness levels, and / or distances between bands or rings at respective locations in the operating environment. The signal coverage map can also be displayed as a 3-dimensional topological representation, where the X and Y axes correspond to locations defined by an occupancy grid, and the Z axis indicates the wireless communication signal strength of each device at the respective location of the occupancy grid, as well as the change in slope / direction of the signal strength between the respective locations. In some embodiments, different colors, brightness levels, and / or distances between bands / rings can be used in conjunction with the 3-dimensional topological representation of the respective signal strengths.

[0117] At block 1110, a combination of a 2-D occupancy grid and a 3-D signal strength topology or heat map may also be displayed in relation or side-by-side. Topological wireless communication signal strength data from multiple electronic devices in the operating environment may also be combined into a single map, thereby indicating the locations of multiple devices on the same map. In this manner, location information for multiple network-enabled devices in the operating environment may be received by a local or remote user terminal and displayed thereon in the above or other user-friendly formats.

[0118] More generally, by combining occupancy data with signal coverage data obtained by navigating a mobile robot in an operating environment, a map can be generated that indicates physical characteristics at a corresponding location of the operating environment (as indicated by positioning data detected by the mobile robot at the corresponding location) and electrical characteristics (including signal strength, lack of coverage, and / or signal interference) at the corresponding location of the operating environment of the mobile robot (as indicated by the signal strength of wireless communication signals acquired by the mobile robot at the corresponding location). As described herein, the combination of occupancy data and signal coverage data for the operating environment can be used to improve the operation or functionality of mobile robots and / or other devices in the operating environment, as well as to present information about such mobile robots and / or other devices to a user. Such improvements may include, but are not limited to, identifying weak spots and gaps in signal strength, determining the location / identification of items in a room, faster positioning / repositioning of a mobile robot, displaying annotated maps to a user, sharing maps with another robot / device, and / or system diagnostics based on packets received from other devices.

[0119] The following describes example uses of the combined occupancy and signal coverage data by a mobile robot, a server, and / or a user terminal. While some examples may refer to specific wireless communication signals (such as Wi-Fi signals) transmitted by specific electronic devices (such as wireless router devices), it should be understood that these examples are not limited to any specific device or communication standard / protocol.

[0120] In some embodiments, operations for mapping device locations based on wireless signal coverage as described herein may allow for correlation or merging of occupancy maps generated from positioning data obtained by the same mobile robot or multiple robots from multiple navigations of the same operating environment. For example, the positioning data obtained by one or more mobile robots may include visual positioning data detected by vision-based positioning sensors at different times of the day and / or under different lighting conditions, and a map may be generated by merging corresponding occupancy maps generated from visual positioning data detected at different times of the day and / or under different lighting conditions using a location determined for at least one electronic device as an anchor point to resolve ambiguities caused by different lighting conditions in the visual positioning data. Sufficient overlap in the topologically mapped fields based on wireless signal strength may also allow different maps from adjacent / geographically collocated operating environments to be combined or "stitched" together.

[0121] In some embodiments, the correlation between the occupancy grid and the map of wireless communication signals can be transmitted to or shared with one or more other devices within the operating environment. For example, such other devices may include mobile robots that lack visual localization capabilities. These less advanced robots can use the occupancy grid determined by the more advanced mobile robots for self-localization. For example, an autonomous vacuum cleaner robot that may include a camera or other imaging sensor device can navigate the operating environment and generate occupancy grid data for the operating environment based on visual simultaneous localization and mapping techniques. The vacuum cleaner robot can thereby transmit the determined occupancy grid data to a cloud computing device and / or directly to another mobile robot with lesser capabilities (such as a floor-mopping robot that may be constrained to a floor position). Furthermore, an autonomous patrol robot with a camera mounted at a higher vantage point may also include a wireless signal sensor at or coupled to the higher vantage point, thereby allowing wireless communication signals to be received at different altitudes and generating mapping data indicating the signal strength received at different altitudes. In some embodiments, the higher vantage point may be provided by a mechanically actuated element (such as a telescopic or otherwise extendable mast). In this manner, such a patrol robot can generate mapping data that is unavailable to mobile robots constrained to collect their corresponding wireless sensor signals at lower altitudes.

[0122] Asymmetry in corresponding coverage patterns can be used to identify and / or distinguish maps to correspond to corresponding floors at different heights in the operating environment (e.g., multiple floors in the same house and / or different houses). For example, the use of directional antennas and "beam steering" techniques (such as in Multi-User Multiple Input Multiple Output (MU-MIMO) implementations) can increase the asymmetry of the acquired Wi-Fi signals, thereby reducing location ambiguity. Devices with omnidirectional (hemispherical) radiation patterns can be sensed on the floor above the vertical extension of the hemisphere and associated between maps of the floor by reference to the unique network address (e.g., MAC address) they transmit. Adding devices with intentionally non-omnidirectional radiation patterns or patterns that change predictably over time can ensure that a "fingerprint" represented by a combination of measured RSSI strengths produces a unique location. Similar techniques can be used by a mobile robot or located in its dock, including at least one directional antenna to associate floors (e.g., by transmitting a vertical beam upward from the lowest point towards the floor(s) above). The robot may also determine the relative height or horizontal position of the electronic device being detected (e.g., based on two antenna patterns, one sensitive to height and one sensitive to horizontal).

[0123] Additionally, some devices may have inherently asymmetric RF radiation patterns, particularly when viewed from floor level. For example, a thermostat device that includes a metal ring on its face may transmit with low signal strength directly below and in front of the device, while the signal strength on the opposite side of the wall to which it is mounted may be significantly higher (because its internal construction and mounting bracket may be plastic). In response to identifying a specific manufacturer and model of a device as described herein, the signal coverage data may be corrected for the RF radiation pattern of that specific model and, therefore, may be used to determine the location of the device with greater accuracy. For example, the location of a thermostat device on a specific side of a wall may be determined.

[0124] In some embodiments, correlating the signal coverage of wireless communication signals with an occupancy grid independently determined by the mobile robot using VSLAM techniques can improve the speed and / or accuracy of the mobile robot's localization. For example, if a mobile robot is "kidnapped" (i.e., picked up and relocated or moved to a new location within an operating environment), the mobile robot can detect this relocation and determine its new location within the operating environment based on a comparison of the wireless communication signals detected at its new location with the wireless communication signals of a previously acquired wireless communication signal map. Specifically, based on the received signal strength at the new location, the slope of the received signal strength, and / or the direction of change of the slope of the signal strength at the new location (e.g., as indicated by the rate of change in the topology of a 3-dimensional map of signal coverage relative to the occupancy grid), the mobile robot can more quickly determine its new location and / or its current pose at the new location. The angle, direction, and / or rate of change of the slope of the signal strength relative to the peak signal strength can also help determine the current pose of the mobile robot. When combined with visual localization techniques (such as camera-based recognition of the environment) for visual localization, the speed and / or accuracy of such localization can be enhanced. That is, in a topological representation of received signal strengths relative to an occupancy grid of an operating environment, specific signal strengths and their slope rates of change in one or more directions can be used together with the pose of the mobile robot to determine its new position in the operating environment. A relative improvement in the speed and / or accuracy of relocalization can be demonstrated by selectively disabling the mobile robot's positioning sensor(s) or wireless signal reception. Furthermore, the mobile robot can be configured to resolve uncertainty regarding the occupancy grid and / or its position therein based on a unique wireless signature of a corresponding fixed-location device as indicated by one or more wireless communication signal graphs at the device's corresponding location.

[0125] In some embodiments, the actuator may be operable to navigate the mobile robot directly to the location of an electronic device whose location in the operating environment was previously determined, for example, in response to activation of a sensor and / or alarm at the location of the electronic device as a "first responder." Upon arrival at the location of the electronic device, real-time audio and / or visual information may be recorded by the mobile robot and / or transmitted to a remote device such as a Figure 1B user terminals 142, 144).

[0126] In some embodiments, information about the operating environment determined from the coverage pattern of acquired wireless communication signals can be transmitted to one or more other devices operating within the environment, thereby allowing for shareable diagnostics. For example, the transmission coverage and location of each detected device in the operating environment may indicate a network configuration in which two networked devices may each be able to communicate with a particular access point, but may not be able to "hear" (i.e., may not be aware of) each other's transmissions. In some cases, two devices may attempt to transmit simultaneously on the same channel or frequency, making it impossible for the access point to receive valid communications from either device (referred to as the "hidden node" problem). Some embodiments described herein can use relevant data to detect which devices cannot "hear" other devices (even if they may be within range of the same access point (due to external antennas, which typically have higher transmit power and more sensitive reception)) and can send instructions to one or more devices to change or assign different channels or communication frequencies to avoid potential or actual interference. In addition, one or more detected devices can be notified of the presence of one or more other detected devices outside of each other's communication range, for example, devices with wireless connections to different routers in the operating environment. Thus, operation of a mobile robot according to embodiments of the present disclosure may bridge the gap between devices in an operating environment that may otherwise be unaware of each other.

[0127] In some embodiments, information about the operating environment determined from the coverage pattern of acquired wireless communication signals can be stored and provided to devices subsequently added to the operating environment. For example, a newly added device can communicate its presence to one of the access points within the environment, and previously stored information about the operating environment can be downloaded or shared with the newly added device to assist with setup operations for the newly added device. In other words, the mobile robot's prior navigation of the operating environment can allow a subsequently added device to perform automated setup operations within the operating environment.

[0128] In some embodiments, the mobile robot's wireless receiver can monitor multiple communication protocols, for example using multi-protocol scanning, to detect multiple electronic devices in the environment that can be configured to communicate using different communication protocols. Information about the operating environment and / or the multiple electronic devices operating therein can thus be formatted and shared with one or more detected devices via their corresponding communication standards. In this way, devices that cannot communicate with each other can be made aware of each other's presence and / or operation within the operating environment.

[0129] In some embodiments, the wireless communication signal map may indicate signal strength information (and a corresponding location within the operating environment) for a particular device, such as a lost user device. For example, a user who misplaces a smartphone in the operating environment may generate a request for the last known location of the lost user device via another user terminal, such as a laptop or desktop computer or tablet computer. In response, the user terminal may retrieve and display an image of the location in the operating environment corresponding to the peak signal strength of the wireless communication signal most recently recorded in the lost user device, a floor plan of the operating environment with the location marked, and / or an audible description of the location. In this way, the mapping of the wireless communication signal may be used to locate the lost device even if the lost device is out of power.

[0130] In some embodiments, based on the acquired wireless communication signals, the device manufacturer can be identified from the MAC address, and the model / type of the device can be identified from multi-protocol scanning or other network detection / monitoring. This can enable identification and differentiation between fixed devices with fixed locations (such as thermostats or wireless access points) and mobile devices (such as smartphones or tablets). Additionally or alternatively, the wireless signals broadcast from these devices can include customer definitions indicated by registered addresses. The mobile robot and / or the cloud computing device associated therewith can transmit or share such identification information with the user terminal for display via its user interface.

[0131] In some embodiments, Wi-Fi coverage data for a home, as indicated by Wi-Fi signal strengths obtained at corresponding locations, can be used to inform a user of improved positioning or placement of corresponding electronic devices in the home. In particular, for Wi-Fi access points, the Wi-Fi signal strength data can indicate which areas of the home have weak or no Wi-Fi signal coverage from the access point. As such, based on the signal strength data indicated by the graph, one or more recommendations regarding placement of Wi-Fi access points can be transmitted to a remote user device to improve Wi-Fi signal coverage. Similarly, recommendations (including recommendations regarding placement of (one or more) repeaters) can also be transmitted to the remote user device to cover areas with weak or no Wi-Fi signal coverage to enhance Wi-Fi signal coverage in those areas. In addition, in response to detecting signal interference between multiple devices and an operating environment, the mobile robot and / or a cloud computing device communicating therewith can generate and transmit directions or recommendations related to the spatial orientation or placement of the devices that can reduce and / or resolve signal interference.

[0132] Figure 12Ais a diagram illustrating a system of devices for combining sensor-based positioning and wireless signal mapping according to some embodiments of the present disclosure. System 1200 may include a mobile robot 200 that can obtain signal coverage data 500 and occupancy data 600 for an operating environment as described herein. The signal coverage data 500 and occupancy data 600 may be transmitted to a cloud computing device and / or system 1270. In some embodiments, the cloud computing device and / or system 1270 may include the remote management server(s) 150 described herein. The cloud computing device and / or system 1270 may include a processor 1210 and a user data storage device 1220. The user data storage device 1220 may store the signal coverage data 500, the occupancy data 600, and / or other data associated therewith (e.g., acquired wireless communication signals and / or detected positioning data). While some embodiments provide for the mobile robot 200 to perform the correlation of the signal coverage data 500 and the occupancy data 600, in some embodiments, the correlation may be performed by the cloud computing device and / or system 1270.

[0133] In some embodiments, the cloud computing device and / or system 1270 may provide a communication channel between the mobile robot 200 and the user device 142, 144, 400, such that the user device 142, 144, 400 may be operable to receive messages and / or data from the robot 200 and / or the cloud computing device and / or system 1270. For example, the user device 142, 144, 400 may display one or more visual representations 605 of signal coverage data 500, occupancy data 600, and / or data based on correlations thereof, and may receive user input allowing the user to edit or modify the displayed data.

[0134] Figure 12B FIG is a diagram illustrating in more detail the cooperative operation of devices for combining sensor-based positioning and wireless signal mapping according to some embodiments of the present disclosure. The blocks shown with dashed lines indicate that the Figure 12B Operations optionally performed in an embodiment.

[0135] like Figure 12B As shown in FIG, at block 1201, the mobile robot 200 acquires wireless communication signals during navigation of the operating environment by using the wireless receiver 250 via RF packet capture. At block 1202, the mobile robot 200 also determines its location or position in the operating environment based on the occupancy data 600, and in some embodiments, based on interpolation (e.g., if the position is not continuously reported) at block 1204. The occupancy data 600 can be based on positioning data obtained concurrently or previously by the mobile robot 200 during navigation of the operating environment, for example, via one or more positioning sensors.

[0136] Based on the RF packet capture at block 1201, the mobile robot 200 performs packet header decoding / extraction at block 1203 to determine, for example, a unique network address and / or received signal strength corresponding to one or more senders of the received packet. For example, the sender of the received packet can be referenced to the networked device from which the packet was transmitted. Additional information can also be determined from the received packet, including but not limited to signal-to-noise ratio, data throughput, and / or multipath distortion. The information determined from the received packet may be collectively referred to herein as signal coverage data.

[0137] The mobile robot 200 thus combines the signal coverage data determined at blocks 1201 and 1203 with the occupancy data and location determined at block 1202 (and optionally 1204) into a log at block 1205. Optionally, time data associated with the packet capture and / or location determination may also be determined at block 1206 and recorded in the log at block 1205 (e.g., as a timestamp), e.g., to provide additional data points for filtering and / or other related operations.

[0138] At block 1207, the signal coverage data, occupancy data, location, and (in some embodiments) associated timestamp data are filtered, averaged, and / or compressed and converted to a spatial representation (shown as graph 605) at block 1208. At block 1207, the spatial representation 605 may indicate signal characteristics at corresponding locations in the operating environment based on correlation of the signal coverage data and occupancy data. The sender's device type (e.g., fixed / mobile, manufacturer, model, etc.) may be determined at block 1209a and stored in database 1220a. The sender's corresponding location in the operating environment may also be determined at block 1209b and represented in graph 705. Persistent and temporal averaging may also be performed at block 1211 based on the determined sender type(s) and / or location(s), the results of which may be stored in database 1220b. In some embodiments, the related operations 1207, 1208, 1209a, 1209b, and / or 1211 may be performed by a remote server 150 that is communicatively coupled to the mobile robot to receive signal coverage, occupancy, and positioning data therefrom. However, it will be understood that one or more of the related operations 1207, 1208, 1209a, 1209b, and / or 1211 may be performed by the mobile robot 200 itself and / or one or more other computing devices that may be communicatively coupled to the mobile robot 200 (including the remote server 150 and / or the user terminal 142, 444, 400).

[0139] The user terminal 142, 444, 400 receives data derived from the correlation of signal coverage data and occupancy data at blocks 1207, 1208, 1209a, 1209b, and / or 1211 and displays one or more visual representations of the data at block 1212. Figure 13-17 Examples of data that may be displayed via the user terminals 142 , 144 , 400 are shown in the user interface of .

[0140] refer to Figure 13 13. User interface 1310 of FIG. 13 shows a 2-D occupancy grid 600 alongside a 3-D topological representation of signal coverage data 500. Indications of locations within the operating environment where signal coverage is strong, weak, or absent are indicated by arrows ("strong coverage," "weak coverage," or "no coverage") on occupancy grid 600 and are also visible from changes in the slope of the 3D topological representation of signal coverage data 500.

[0141] refer to Figure 14 14 shows a user interface 1410 that illustrates a 2-D occupancy grid 600 juxtaposed with a photograph 1440 of a location "X" in the operating environment. The location or area "X" and the corresponding photograph 1440 indicate a suggested location ("Move device here") for repositioning a wireless router in the operating environment based on wireless communication signals acquired by a mobile robot in the operating environment, e.g., to improve signal coverage.

[0142] refer to Figure 15 15. User interface 1510 of FIG. 15 shows a 2-D occupancy grid 600 juxtaposed with a photograph 1440 of a location or area “X” indicating a suggested location in the operating environment for repositioning a wireless router to improve signal coverage based on wireless communication signals acquired by the mobile robot in the operating environment, and an indication 1550 showing a suggested height for placing the wireless router (“Position the device at this height”), for example, to avoid interference with one or more other networked devices in the operating environment.

[0143] refer to Figure 16 1, and a user interface 1610 of FIG. 1 displays an incentive 1640 for purchasing a new wireless router or range extender on the 2-D occupancy grid 600. The incentive 1640 can be displayed based on the poor signal coverage characteristics of the existing wireless router, as indicated by wireless communication signals acquired by the mobile robot in the operating environment.

[0144] refer to Figure 17User interface 1710 depicts 2-D occupancy grid 600 alongside a photograph 1740 of location "X" in the operating environment. Location "X" and corresponding photograph 1740 indicate the most recent location of the mobile electronic device in the operating environment that received the wireless communication signal from the mobile electronic device ("last known device location") based on the wireless communication signal acquired by the mobile robot. The view shown in photograph 1740 points in the direction of the last peak emission of the signal from the mobile electronic device. Additionally or alternatively, an audible description of location "X" represented by photograph 1740 may be provided via user interface 1710 (e.g., "Your phone was last detected in the dining room").

[0145] In the above description of various embodiments of the present disclosure, aspects of the present disclosure may be shown and described in any of a number of patentable classes or contexts, including any new and useful process, machine, manufacture or combination of materials, or any new and useful improvement. Thus, aspects of the present disclosure may be implemented entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or a combination of software and hardware implementations, which may be generally referred to herein as "circuits," "modules," "components," or "systems." Additionally, aspects of the present disclosure may take the form of a computer program product comprising one or more computer-readable media having computer-readable program code implemented thereon.

[0146] Any combination of one or more computer-readable media may be used. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media would include the following: a hard drive, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a suitable optical fiber with a repeater, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any non-transitory medium that can contain or store a program used by or in conjunction with an instruction execution system, apparatus, or device.

[0147] A computer-readable signal medium may include a propagated data signal (e.g., in baseband or as part of a carrier wave) in which computer-readable information is embodied. Such a propagated signal may take any of a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium that is not a computer-readable storage medium and that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Computer-readable information embodied on a computer-readable medium may be sent using any suitable medium (e.g., as program code), including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0148] Computer program code for performing the operations of various aspects of the present disclosure can be written in any combination of one or more programming languages. The program code can be executed entirely on the user terminal, mobile robot, or remote server described herein, or partially on one or more of each. In the latter case, the remote server can be connected to the user terminal and / or mobile robot via any type of network, including a local area network (LAN) or a wide area network (WAN), or be connected to an external computer (e.g., via the Internet using an Internet service provider) and / or in a cloud computing environment, or provided as a service (such as software as a service (SaaS)).

[0149] Various aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It will be understood that each box in the flowchart illustration and / or block diagram and the combination of boxes in the flowchart illustration and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that instructions executed by the processor of the computer or other programmable instruction execution device create a mechanism for implementing the function / action specified in one or more boxes of the flowchart and / or block diagram.

[0150] These computer program instructions may also be stored in a computer-readable medium, which, when executed, may direct a computer, other programmable data processing apparatus, or other device to operate in a specific manner, such that when the instructions are stored in the computer-readable medium, an article of manufacture including the instructions is produced, wherein when the instructions are executed, the computer is caused to implement the functions / actions specified in one or more blocks of the flowchart and / or block diagram. The computer program instructions may also be loaded onto a computer, other programmable instruction execution apparatus, or other device to cause a series of operating steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, such that the instructions executed on the computer or other programmable apparatus provide a process for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0151] It should be understood that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meanings as those generally understood by those of ordinary skill in the art to which the present disclosure belongs. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the same meaning as their meaning in the context of this specification and related art, and will not be interpreted in an idealized or overly formalized sense unless explicitly defined as such herein.

[0152] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various aspects of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a fragment or part of a code comprising one or more executable instructions for realizing the specified (one or more) logical functions. It should also be noted that in some alternative implementations, the functions mentioned in the box may not occur in the order shown in the figure. For example, depending on the functions involved, the two boxes shown in succession can actually be performed substantially simultaneously, or the boxes can sometimes be performed in the opposite order. It will also be noted that each box illustrated in the block diagram and / or flowchart and the combination of the boxes illustrated in the block diagram and / or flowchart can be implemented by a system based on dedicated hardware or a combination of dedicated hardware and computer instructions that performs the specified function or action.

[0153] The foregoing is an illustration of the embodiments of the present disclosure and should not be construed as limiting thereof. Although some example embodiments have been described, it will be readily appreciated by those skilled in the art that many modifications may be made to the example embodiments without departing substantially from the teachings and advantages of the present invention. Thus, all such modifications are intended to be included within the scope of the present invention. Therefore, it should be understood that the foregoing is an illustration of the present invention and should not be construed as being limited to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included within the scope of the present invention.

Claims

1. A user terminal, comprising: Receiver; user interface; a processor coupled to the receiver and the user interface; as well as a memory coupled to the processor, the memory comprising a non-transitory computer-readable storage medium having computer-readable program code stored therein, the computer-readable program code being executable by the processor to perform operations including: receiving, via a receiver of the user terminal, map information based on a correlation of positioning data with wireless communication signals, wherein: in response to navigation of at least one mobile robot in its operating environment, the positioning data is detected in the operating environment by a positioning sensor of the at least one mobile robot, and the wireless communication signals are acquired in the operating environment by a wireless receiver of the at least one mobile robot, wherein the correlation is a correlation between placement and position of the at least one electronic device in the operating environment identified based on the wireless communication signals and an image of the at least one electronic device captured in the positioning data; and One or more visual representations of the map information are displayed via a user interface of the user terminal.

2. The user terminal according to claim 1, wherein the one or more visual representations include: an occupancy map indicating physical characteristics of the operating environment, the physical characteristics including obstacles detected in the operating environment by positioning sensors of the at least one mobile robot; or A signal coverage map indicates electrical characteristics of the operating environment, the electrical characteristics including respective signal strengths of the wireless communication signals acquired by the wireless receiver of the at least one mobile robot at respective locations in the operating environment.

3. A user terminal according to claim 2, wherein the signal coverage map includes a two-dimensional representation, which indicates the corresponding signal strength of the wireless communication signal by changing the color and / or brightness level at the corresponding positions in the operating environment; and / or the signal coverage map includes a three-dimensional topological representation, which indicates the corresponding signal strength of the wireless communication signal at the corresponding positions in the operating environment relative to the axis and the slope change of the corresponding signal strength between the corresponding positions. 4 . The user terminal of claim 2 , wherein the occupancy map further indicates a location of the at least one electronic device in the operating environment. 5 . The user terminal of claim 4 , wherein the occupancy map indicates the location of the at least one electronic device using a graphical representation of signals transmitted from the at least one electronic device.

6. The user terminal according to claim 5, wherein the operation further comprises: receiving an indication of a current position of the at least one mobile robot in the operating environment, Wherein the graphical representation of the signal changes based on the proximity of the current location of the mobile robot to the location of the at least one electronic device.

7. The user terminal of claim 5, wherein the graphical representation varies in color, brightness level and / or distance between displayed bands or rings. 8 . The user terminal according to claim 2 , wherein the occupancy map further indicates an identity, the identity comprising a type, a manufacturer and / or a model of the at least one electronic device.

9. The user terminal of claim 2, wherein the occupancy map further identifies the at least one electronic device as mobile or stationary.

10. The user terminal according to claim 1, wherein the operation further comprises: Based on the wireless communication signals acquired by the wireless receiver of the at least one mobile robot, an indication of one or more suggested locations in the operating environment for repositioning of the at least one electronic device is displayed via a user interface of the user terminal.

11. The user terminal according to claim 1, wherein the operation further comprises: Based on the signal interference indicated by the wireless communication signal acquired by the wireless receiver of the at least one mobile robot, an indication of one or more suggested frequencies or channels for communication by the at least one electronic device is displayed via the user interface of the user terminal.

12. The user terminal according to claim 4, wherein the position of the at least one electronic device in the operating environment is a latest position at which the wireless communication signal is received from the at least one electronic device, and wherein the operation further comprises: An audio and / or visual indication of the latest location is provided via a user interface of the user terminal.

13. The user terminal of claim 1 , wherein the one or more visual representations of the map information indicate areas in the operating environment where signal coverage of the wireless communication signal is weak or non-existent, and wherein the operations further comprise: Suggestions for improvements to the signal coverage are displayed via the user interface.

14. A method for operating a user terminal, comprising: The processor of the user terminal performs an operation, the operation including: receiving, via a receiver of the user terminal, map information based on a correlation of positioning data with wireless communication signals, wherein: in response to navigation of at least one mobile robot in its operating environment, the positioning data is detected in the operating environment by a positioning sensor of the at least one mobile robot, and the wireless communication signals are acquired in the operating environment by a wireless receiver of the at least one mobile robot, wherein the correlation is a correlation between placement and position of the at least one electronic device in the operating environment identified based on the wireless communication signals and an image of the at least one electronic device captured in the positioning data; and One or more visual representations of the map information are displayed via a user interface of the user terminal.

15. The method of claim 14, wherein the one or more visual representations comprise: an occupancy map indicating physical characteristics of the operating environment, the physical characteristics including obstacles detected in the operating environment by positioning sensors of the at least one mobile robot; or A signal coverage map indicates electrical characteristics of the operating environment, the electrical characteristics including respective signal strengths of the wireless communication signals acquired by the wireless receiver of the at least one mobile robot at respective locations in the operating environment.

16. A method according to claim 15, wherein the signal coverage map includes a two-dimensional representation, which indicates the corresponding signal strength of the wireless communication signal by changing the color and / or brightness level at the corresponding positions in the operating environment; and / or the signal coverage map includes a three-dimensional topological representation, which indicates the corresponding signal strength of the wireless communication signal at the corresponding positions in the operating environment relative to the axis and the slope change of the corresponding signal strength between the corresponding positions.

17. The method of claim 14, wherein the operations further comprise: Based on the wireless communication signals acquired by the wireless receiver of the at least one mobile robot, an indication of one or more suggested locations in the operating environment for repositioning of the at least one electronic device is displayed via a user interface of the user terminal.

18. The method of claim 14, wherein the operations further comprise: Based on the signal interference indicated by the wireless communication signal acquired by the wireless receiver of the at least one mobile robot, an indication of one or more suggested frequencies or channels for communication by the at least one electronic device is displayed via the user interface of the user terminal.

19. The method of claim 14, wherein the one or more visual representations of the map information indicate areas in the operating environment where signal coverage of the wireless communication signal is weak or non-existent, and wherein the operations further comprise: A suggestion to reposition the wireless router to improve signal coverage or an incentive to purchase a wireless router is displayed via the user interface.

Images