Method, system and device for mapping, controlling and displaying device status

Abstract

A method of operating a user terminal includes receiving occupancy data for an operating environment in response to navigation of the operating environment by a mobile robot, and displaying a visual representation of the operating environment based on the occupancy data. The method also includes receiving information identifying a plurality of electronic devices local to the operating environment and their respective operating states, and populating the visual representation of the operating environment with visual indications of respective spatial locations of the electronic devices in the operating environment and status indications of respective operating states of the electronic devices. Related methods for controlling electronic devices based on respective spatial locations of the electronic devices and relative spatial context of the operating environment are also discussed.

Description

[0001] This application is a divisional application of the invention patent application with the application date of May 9, 2018, application number 201880041922.3, and invention name "Method, system and device for drawing, controlling and displaying device status". Technical Field

[0002] The present disclosure relates to connected devices and systems and methods for interoperation thereof. Background Art

[0003] Connectivity for home appliances, including wireless connections to the Internet and remote clients, has been considered for some time. The term "Internet of Things" (IoT) represents the idea that various household items can be connected to the public Internet. Once connected, such items can report various data to servers and client devices. For example, a "smart" light bulb can be connected to a home WLAN (wireless local area network). Each light bulb can have a microprocessor, memory, some means of detecting or interpreting conditions, power, and wireless connectivity. Using these components, the light bulb can report its condition, be polled, etc.

[0004] The concept of IoT is often considered distinct from household connectivity (e.g., connected computers, cable boxes, media devices, etc.) because IoT devices typically do not include sufficient computing resources or communications to meaningfully connect to the public Internet. A regular refrigerator cannot connect to the Internet; this same device as an IoT device would include computing, sensor, and communication hardware and sufficient software to be a remotely and locally addressable entity; it is expected that this Internet refrigerator can report its various states (power consumption, etc.) and respond to remote commands (increase or decrease the internal temperature).

[0005] Home mobile robots can also become IoT devices. In some ways, home mobile robots can be considered a unique species in this set. In particular, the autonomy of home mobile robots can distinguish them from other appliances, which do not operate in unpredictable and changing environmental conditions, nor can they make autonomous decisions based on dozens or hundreds of sensor inputs in order to achieve task completion. For example, a dishwasher (even an IoT dishwasher) knows nothing about its contents and runs the equivalent of a simple script controlling the motor and pump, which may be interrupted by a simple jam or other sensor. In contrast, a vacuuming robot can detect its own state in a variety of ways while performing its tasks, and can flexibly escape from challenging situations in the home and engage in predictive and planned activities. As such, integrating the autonomous behavior of home mobile robots with IoT device functionality can be challenging. Summary of the invention

[0006] According to some embodiments of the present disclosure, a method of operating a user terminal includes: receiving occupancy data for the operating environment in response to navigation of the operating environment by a mobile robot; and displaying a visual representation of the operating environment based on the occupancy data. The method also includes: receiving information identifying multiple electronic devices local to the operating environment and their corresponding operating states; and populating the visual representation of the operating environment with visual indications of the corresponding spatial locations of the electronic devices in the operating environment and status indications of the corresponding operating states of the electronic devices. The method can be executed by instructions from at least one processor of the user terminal or in response to the instructions.

[0007] In some embodiments, the method may further include receiving an input specifying a time via the user interface, and the corresponding operating state displayed in the visual representation may be associated with time data corresponding to the time specified by the input. The specified time may be one minute, one hour, one day, one week, etc.

[0008] In some embodiments, the method may further include displaying a user interface element visually representing a time range via a user interface, and receiving input specifying a time in response to manipulation of the user interface element. The time range may include past, present, and future times, and may also be specified in response to user input.

[0009] In some embodiments, the user interface element may be a bar element representing the time range and a slider element displayed at a corresponding position along the bar element in response to manipulation.

[0010] In some embodiments, the time data may include past, present or future time data. The corresponding operating state displayed in the visual representation may be an actual operating state associated with the past or present time data, or may be an expected operating state associated with the future time data.

[0011] In some embodiments, the status indication may identify a conflict between intended operating states of two or more of the electronic devices based on their respective spatial locations.

[0012] In some embodiments, the input specifying a time may be a time range, and a representation of the actual and / or expected operating status of one or more of the electronic devices within the time range specified by the user input may be displayed via a user interface.

[0013] In some embodiments, the information for identifying multiple electronic devices may be positioning data detected by one or more positioning sensors of the mobile robot, wireless communication signal coverage data acquired by one or more wireless receivers of the mobile robot, network data detected by one or more network elements, and / or user selection data received via a user interface of a user terminal.

[0014] In some embodiments, one or more of the electronic devices may be non-connected devices that lack a network interface.

[0015] In some embodiments, populating the visual representation of the operating environment may include determining a corresponding spatial position of the electronic device in the operating environment. The corresponding spatial position of the electronic device may be determined based on positioning data detected by at least one positioning sensor of the mobile robot in response to navigation of the electronic device in the operating environment and / or wireless communication signals acquired by at least one wireless receiver of the mobile robot.

[0016] In some embodiments, populating the visual representation of the operating environment may include, in response to displaying the visual representation of the operating environment, receiving, via the user interface, a respective input specifying a respective spatial location of the electronic device.

[0017] In some embodiments, conditional instructions for controlling the operation of one or more of the electronic devices may be automatically generated based on the corresponding spatial positions of the electronic devices. For example, one or more of the electronic devices may be lighting devices, and the conditional instructions may specify activation of the lighting devices based on the proximity of their corresponding spatial positions to natural light sources in the operating environment and the expected or detected amount of light provided by the natural light sources. As another example, one or more of the electronic devices may be motion-based alarm sensors, and the conditional instructions may specify deactivation of the motion-based alarm sensors based on a predetermined operating time of the mobile robot at the corresponding spatial positions of the motion-based alarm sensors.

[0018] In some embodiments, the visual indication may be a respective icon representing the electronic device, and the status indication may be a graphical feature of the respective icon.

[0019] According to some embodiments of the present disclosure, a user terminal includes at least one receiver, a user interface, a processor coupled to the at least one receiver and the user interface, and a memory coupled to the processor. The memory includes a non-transitory computer-readable storage medium storing a computer-readable program code therein, which is executable by the processor to perform operations. The operations include: receiving occupancy data for the operating environment via at least one receiver in response to navigation of the operating environment by a mobile robot; and displaying a visual representation of the operating environment via the user interface based on the occupancy data. The operations also include: receiving information identifying multiple electronic devices and their corresponding operating states local to the operating environment via at least one receiver; and filling the visual representation of the operating environment with visual indications of the corresponding spatial positions of the electronic devices in the operating environment and status indications of the corresponding operating states of the electronic devices.

[0020] According to some embodiments of the present disclosure, a method of operating a computing device includes: receiving occupancy data for the operating environment in response to navigation of the operating environment by a mobile robot; and associating an electronic device local to the operating environment with a corresponding spatial position in the operating environment based on the occupancy data. The method also includes transmitting a control signal to one or more electronic devices in the electronic device to control its operation based on the corresponding spatial position associated therewith. The method can be executed by an instruction from at least one processor of the computing device or in response to the instruction.

[0021] In some embodiments, the occupancy data may indicate a room in the operating environment, and the association may further include identifying a relative spatial context of the room in the operating environment based on the occupancy data. Transmitting control signals to one or more of the electronic devices may also be based on the relative spatial context of the room corresponding to the respective spatial location associated therewith.

[0022] In some embodiments, the associating may further include grouping the electronic devices into respective subsets based on respective spatial locations associated with the electronic devices corresponding to the same room in the room and / or consecutive rooms in the room. Transmitting a control signal to one or more of the electronic devices may be based on the grouping thereof.

[0023] In some embodiments, transmitting the control signal to one or more of the electronic devices may also be based on the respective type of room corresponding to the respective spatial location associated with the electronic device.

[0024] In some embodiments, the method further includes receiving, via at least one receiver of the computing device, data indicating operation of at least one other electronic device in the electronic devices. The corresponding spatial position of the at least one other electronic device in the electronic devices may be adjacent to the corresponding spatial position of one or more electronic devices in the electronic devices, and the transmission of the control signal to the one or more electronic devices in the electronic devices may also be based on the operation of the at least one other electronic device in the electronic devices.

[0025] In some embodiments, at least one other electronic device among the electronic devices may be a motion sensor in one of the rooms adjacent to the corresponding spatial locations of one or more of the electronic devices in the relative spatial context. The method may also include: in response to the operation of the motion sensor in one of the rooms adjacent thereto in the relative spatial context, predicting the presence of a user in one of the rooms corresponding to the corresponding spatial locations of the one or more electronic devices, and the transmission of the control signal may be responsive to the prediction.

[0026] In some embodiments, at least one other electronic device among the electronic devices may be an environmental sensor in one of the rooms adjacent to the corresponding spatial location of one or more of the electronic devices in the relative spatial context. The method may also include: in response to the operation of the environmental sensor, determining an environmental condition in one of the rooms, and transmitting a control signal to one or more of the electronic devices may be based on the environmental condition in one of the rooms adjacent to its corresponding spatial location.

[0027] In some embodiments, the method may also include automatically generating conditional instructions for controlling the operation of one or more electronic devices in the electronic devices in response to the association of the electronic device with the corresponding spatial location, and transmitting a control signal to one or more electronic devices in the electronic devices may also be based on the occurrence of one or more conditions specified by the conditional instructions.

[0028] In some embodiments, the method may further include receiving, via at least one receiver of the computing device, data indicating a corresponding operating state of the electronic device and time data associated with the corresponding operating state. The time data may include past, present, or future time data, and the corresponding operating state may be an actual operating state associated with the past or present time data or an expected operating state associated with the future time data. The corresponding operating state of the electronic device and the time data associated with the corresponding operating state may be stored in a database.

[0029] According to some embodiments of the present disclosure, a computing device includes at least one transmitter, a processor coupled to the at least one transmitter, and a memory coupled to the processor. The memory includes a non-transitory computer-readable storage medium, which stores therein a computer-readable program code that can be executed by the processor to perform the following operations: receiving occupancy data for the operating environment in response to the navigation of the operating environment by a mobile robot, and associating an electronic device local to the operating environment with a corresponding spatial position in the operating environment based on the occupancy data. The operation also includes transmitting a control signal to one or more electronic devices in the electronic device via the at least one transmitter to control its operation based on the corresponding spatial position associated with it.

[0030] In some embodiments, the computing device may be a mobile robot or a component thereof, and the occupancy data may be detected by at least one positioning sensor of the mobile robot.

[0031] A person of ordinary skill in the art will recognize further features, advantages and details of the present disclosure by reading the drawings and the following detailed description of the embodiments, including any and all combinations of the above embodiments, which description is merely illustrative of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1Ais a schematic diagram showing a system according to an embodiment of the present disclosure.

[0033] Figure 1B The present invention is a diagram showing an embodiment of the present invention including a Figure 1A Schematic diagram of the operating environment of the system.

[0034] Figure 2A and 2B are top and bottom perspective views of a mobile robot according to an embodiment of the present disclosure, which is configured to Figure 1A operating in the system.

[0035] Figure 2C is a schematic diagram showing a mobile robot according to an embodiment of the present disclosure, which is configured to Figure 1A operating in the system.

[0036] Figure 3 is a schematic diagram showing a computing device according to an embodiment of the present disclosure, which is configured to Figure 1A operating in the system.

[0037] Figure 4A is a schematic diagram showing a user terminal according to an embodiment of the present disclosure, which is configured to Figure 1A operating in the system.

[0038] Figure 4B is a plan view of a user terminal including a user interface configured to display a visual representation of an operating environment and to control connected devices therein according to an embodiment of the present disclosure.

[0039] Figure 5 is a plan view illustrating an example operation of displaying a visual representation of an operating environment indicating relative spatial positions of devices and their operating states according to an embodiment of the present disclosure.

[0040] Figure 6 is a plan view illustrating an example operation for controlling connected devices in a continuous living space according to an embodiment of the present disclosure.

[0041] Figure 7 is a plan view illustrating an example operation for predictively controlling a connection device according to an embodiment of the present disclosure.

[0042] Figure 8 is a plan view illustrating an example operation for controlling a connection device based on an operation state of an electronic device in an adjacent space according to an embodiment of the present disclosure.

[0043] Fig. 9 is a flow chart illustrating operations that may be performed by at least one processor of a user terminal or in response to instructions from at least one processor of a user terminal according to an embodiment of the present disclosure.

[0044] Fig.10 is a flow chart illustrating operations that may be performed by at least one processor of a computing device according to an embodiment of the present disclosure.

[0045] Figures 11A-11E is a plan view illustrating an example operation for displaying a visual representation of an operating environment indicating the relative spatial locations of devices and their operating states in accordance with an embodiment of the present disclosure. DETAILED DESCRIPTION

[0046] Embodiments of the present disclosure may be derived from the following recognition: the autonomous functions of the mobile robot can present unique advantages of integration with IoT and / or other local device functions based on its independent positioning capabilities. In particular, the occupancy data collected by the mobile robot when navigating the operating environment (e.g., when performing cleaning tasks or patrol tasks in a home operating environment) can be used to intuitively indicate the corresponding spatial positions and operating states of other electronic devices within the relative spatial range of the operating environment. Electronic devices may include network-connected devices and non-connected devices, and may be produced by different manufacturers and / or operated according to different standards including different communication protocols. The visual representation of the operating environment may be a floor plan filled with icons indicating the corresponding positions and operating states of the electronic devices, thereby providing a "whole-home" snapshot of the device positions and operations. The historical and predetermined operating state information of the electronic device may also be stored and presented in response to a user request to allow viewing of the actual past or present operating state and / or the expected future operating state. The understanding of the spatial position and operating status of the device as provided by the embodiments of the present disclosure may also allow the control of these electronic devices based on the corresponding spatial positions of the connected electronic devices, the relative spatial context of the operating environment area where they are located, and / or the operating conditions in the adjacent areas of the operating environment.

[0047] A mobile robot may refer to any device including a processor, memory, and drive system for navigating variable environmental conditions and making autonomous decisions based on multiple sensor inputs. The mobile robots described herein may include robotic cleaners and autonomous patrol robots. Some such autonomous patrol robots may include a telescopic mast on which one or more sensor elements are mounted or otherwise operably associated with the telescopic mast.

[0048] An electronic device with network functionality (also referred to as a "connected device") may refer to any device that includes or is coupled to a network interface for transmitting and / or receiving wired or wireless communication signals via a wired or wireless personal, local and / or wide area network. Such connected devices may include, but are not limited to, network controllers, sensors, terminals and / or IoT devices. Other examples of such connected devices may include, but are not limited to, lamps / bulbs, door / window sensors, door locks, speakers, thermostats, environmental sensors (temperature, humidity, air quality, lighting levels), blinds, voice interface devices, surveillance cameras, etc., which have an integrated or other connection to a network interface. Wireless communication signals may include radio frequency signals, including but not limited to Wi-Fi signals, Bluetooth signals, ZigBee signals and / or Z-wave signals and / or optical signals. Such electronic devices may or may not include sufficient computing resources or communications to meaningfully connect to the public Internet. Other electronic devices described with reference to the operating environment described herein may lack a network interface and may be referred to as "non-connected devices."

[0049] refer to Figure 1A , 1B 2-4A, the system 100 according to an embodiment of the present disclosure is installed in an associated living structure 10. The structure 10 can be a house or residence, including but not limited to a single-family home, a multi-family home (e.g., a unit of a duplex, an apartment, a condominium, etc.), a mobile home, or a commercial living space (e.g., an office or studio). The structure 10 can define a living space or interior space 20, also referred to herein as an operating environment. The living space 20 can be subdivided (physically, spatially, and / or functionally) into one or more established zones (illustrated as zones AC). In some embodiments, the zones can correspond to areas of rooms in the living structure 10, such as zone A is a kitchen, zone B is a living room, and zone C is a bedroom. The established zones AC can be separated by walls, or as shown, can be open concept areas with multiple adjacent rooms, representing a wider continuous living space without the need for wall separation. The structure 10 has windows 30 (with blinds 30A connected to a controller 126), doors 32, light fixtures 34 and combustible lights 34A (coupled to a controller 128), a TV 36 (or other electronic equipment), and a heating, ventilation, and air conditioning system (HVAC) 40 (coupled to a controller 127). A user P may occupy the living space 20.

[0050] refer to Figure 1A and 1B, the system 100 includes nodes including a network-enabled mobile robot 200, one or more wireless access points (WAPs) 164, a gateway and / or hub 110 that interconnects different networking methods to form a local private network 160, which interconnects network-enabled or "connected" electronic devices (including IoT devices) 120, 122, 124, network-enabled automation controller devices 126, 127, 128, a robot dock 140 that can also be a network-enabled automation controller device, and products that can combine multiple such functions. The private network 160 includes one or more wireless access points (WAPs) 164, a gateway or hub 110 with a combined wireless range to adequately cover all or most of the living space 20 or surroundings defined by the living structure 10. In some embodiments, one or more of the network nodes 110, 126, 127, 128, 200, 140, 142, 144 and 150 can define connected computing devices (such as Figure 3 A computing device 300) is configured to control the operation of one or more other connected electronic devices 30A, 34 / 34A, 36, 40, 120, 122, 124, 126, 127, 128, 200, 140, 142, 144 and / or electronic devices connected thereto based on their respective spatial locations in the operating environment 20.

[0051] The networked device connected to the private network 160 can reach the public network 170 through the WAN interface 170A and its associated WAN connection 170B, through the router / firewall 162 to communicate with the remote management service 150. For example, the remote management service 150 can be a cloud computing device, the public network 170 can be the Internet, the WAN interface 170A can be a DSL, DOCSIS or cellular modem, and its associated WAN connection 170B can be provided by an Internet service provider (ISP). The router 162, WAP 164 and / or modem 170A can be integrated into a single device in various configurations. The local user terminal 142 can be connected to the private network 160 (wired or wirelessly). The remote user terminal 144 can be connected to the remote server 150 and / or the private network 160 via the public network 170. For example, the local user terminals 142, 144 can be PCs, smart phones or tablet computers. The hub 110, robot 200, controllers 126, 127, 128, and user terminals 142, 144 can all be accessed through a public network service implemented in the target device (e.g., a web server that presents a UI on a local network through a web browser on a client device) or via a dedicated client (e.g., a downloadable or pre-installed application software app), thereby enabling communication and control between nodes 110, 126, 127, 128, 200, 140, 142, 144, and 150, as described herein. A network entity as discussed herein is a machine and / or controller registered on a network, assigned a unique address for sending and receiving communication signals, and available to other network entity machines and / or controllers on the same network or a connected network.

[0052] In some embodiments, "the same network" may refer to a collection of private addresses on a private IP (Internet Protocol) subnet behind a routing network entity 162 that provides network address translation (NAT) between the public Internet and the private network. Each network entity connected to the private network can infer their network addresses by observing the network communications of other activated network entities and / or scanning the possible IP subnets of the private network to find responses. Some gateways / hubs provide network services that can enumerate devices associated with the gateway / hub and / or reachable through the gateway / hub. These technologies generate one or both of the IP address and / or MAC (media access control) address of each activated device. The address resolution protocol (ARP) network service can map one type of address to another type of address. In some embodiments, a routine running on a processor of a network entity (such as a mobile robot 200) can collect the network addresses of other network entities (such as connected devices 120, 122, 124, 126, 127, 128) and identify the type, manufacturer and / or model of these network entities, as well as their relative spatial locations in the living structure 10.

[0053] The robot dock 140 may include a power source or be connected to a power source and include a charger operable to charge the battery of the mobile robot 200 when the mobile robot 200 is effectively docked on the robot dock 140. The dock 140 may be an evacuation station including a powered container that can be actuated to empty debris from the robot 200. In some embodiments, the dock 140 is connected (wired or wirelessly) to the private network 160 to enable or facilitate data transmission from the robot 200 to the private network 160 and / or from the private network 160 to the mobile robot 200. The robot dock 140 may therefore be considered an automation controller device. In some embodiments, the robot dock 140 communicates directly with the mobile robot 200 via wireless means including, but not limited to, Bluetooth, near field induction, IR, and / or radio communication signals. Each connection device 120, 122, 124, 126, 127, 128, 140, 142, 144 may include a wireless transceiver, such as a Wi-Fi transceiver, to communicate with the hub 110 and / or the private network 160 via the WAP 164. Although specific connection devices 30A, 34 / 34A, 36, 40, 110, 120, 122, 124, 126, 127, 128, 140, 142, 144, 150, 200 are shown by way of example, fewer or more connection devices may be included in the operating environment 10 and may communicate with the private network 160.

[0054] The mobile robot 200 may be any suitable robot and associated computing device(s), and it should be appreciated that not all components, features, and functions described herein are required in a mobile robot according to an embodiment of the present disclosure. Figure 2A-2C , the example mobile robot 200 may include a frame 210, a controller 220, a memory 222, a battery 224, a battery charger 226, a human-machine interface (HMI) 228, a drive system 230, a mapping / navigation system 240, a service operating system 242 (also referred to herein as a "cleaning system" and a "cleaning head"), a wireless communication system 250, an IR transmitter 260 and environmental sensors 270A-H, a trash bin 242A (to store debris collected by the cleaning operation), a bin level sensor 242B, a dirt extraction sensor 242C (to detect a characteristic density of debris collected by the cleaning operation), an indicator light 274A, an audio transducer 274B and a cleaning mode selection switch or button 274C.

[0055] The environmental sensors 270A-270H may include a camera 270B mounted on the top surface of the mobile robot 200, such as Figure 2A 200 and acquire images for other operational purposes. In some embodiments, the camera 270B is a visual simultaneous localization and mapping (VSLAM) camera and is used to detect features and landmarks in the operating environment and build an occupancy map based thereon.

[0056] like Figure 2B As shown in the bottom perspective view of FIG. 1 , the mobile robot 200 may also include a bumper 104, cliff sensors 195A-195D, and edge brushes 111 mounted or otherwise positioned on the periphery of the mobile robot housing 106. The housing 106 is Figure 2A-2B104 is shown as having a square front portion on which the bumper 104 is mounted; however, in other embodiments, the housing may have a circular or annular shape. Casters 196 may be provided on the underside of the mobile robot 200. In some embodiments, the casters 196 may be positioned at the end of the mobile robot 200 opposite the cleaning head 242, with the drive rollers / tracks 232A, 232B between them, so that the cleaning head 242 is a cantilever arrangement. The mobile robot 200 may also include a camera 197 facing downward or facing the ground. In some embodiments, the mobile robot 200 may generally be configured in the manner of a floor cleaning robot and / or a robot as described in U.S. Patent No. 7,024,278 and U.S. Published Application No. 2007 / 0250212 or include features from these robots, the disclosures of which are incorporated herein by reference with appropriate modifications. In other embodiments, the mobile robot 200 can generally be configured as an autonomous patrol robot, which includes a telescopic mast on which one or more elements associated with the sensor elements 270A-H and / or the wireless communication circuit or system 250 are mounted or the one or more elements are otherwise operably associated with the telescopic mast.

[0057] The controller 220 may include one or more processors of any suitable configuration.(One or more) processors may include one or more data processing circuits, such as general and / or special processors (such as microprocessors and / or digital signal processors) that may be collocated or distributed in one or more networks. The processor is configured to execute program code stored in the memory 222 described below as a computer-readable storage medium to perform some or all of the operations and methods described above for one or more embodiments. The memory 222 represents one or more storage devices according to some embodiments of the present disclosure, which contain software and data for facilitating robot operation. The memory 222 may include, but is not limited to, the following types of devices: cache, ROM, PROM, EPROM, EEPROM, flash memory, SRAM, and DRAM. Therefore, the processor communicates with the controller 200, the memory 222, the cleaning system 242, and the drive system 230.

[0058] The drive system 230 may include any suitable mechanism or system for actively and controllably moving the robot 200 through the living space 20. According to some embodiments, the drive system 230 includes one or more rollers, one or more tracks 232A, 232B, and one or more onboard (i.e., carried by the mobile robot 200) electric motors 234 (collectively referred to herein as "drives" or "drive systems") operable by the controller 220 to transport the robot 200 across the floor of the operating environment 10.

[0059] The service operation system 242 may be optional in some embodiments and is operable to perform service operations in the living space 20. According to some embodiments, the service operation system 242 includes a floor cleaning system that cleans the floor surface of the living space 20 as the robot 200 passes through the space 20. In some embodiments, the service operation system 242 includes a suction head and an onboard vacuum generator to vacuum clean the floor. In some embodiments, the service operation system 242 includes an end effector such as (but not limited to) a sweeping or mopping mechanism, one or more rotating brushes, rollers, wet or dry fixed or oscillating and / or vibrating cloths, or a multi-layer pad assembly.

[0060] The wireless communication system 250 includes a wireless communication transceiver or module 252 and an associated antenna 254 to enable wireless communication between the robot 200 and various other networked devices 120, 122, 124, 126, 127, 128 in the operating environment 10 and network segments served by WAPs, gateways, and hubs that constitute the private network 160 of which the mobile robot 200 constitutes a node. For example, the wireless communication transceiver or module 252 can be a Wi-Fi module.

[0061] In some embodiments, the robot 200 can communicate wirelessly directly with the dock 140 using narrowband or broadband RF communications. For example, if the robot 200 is not equipped with a transmitter compatible with the WAP 164, the robot 200 can communicate with the dock 140, which can in turn relay data from the robot 200 onto the private network 160 and forward it to the intended network entity (such as the remote management server 150). In some embodiments, the dock 140 includes a bridge device that receives and converts RF signals from the robot 200 and relays them to the router 162 in a format supported by the router for delivery to the remote management server 150 or another device in the private network 160. In some embodiments, the dock 140 includes a low-power mesh data network that employs a mesh topology, in which RF communication signals are relayed from node to node between the mobile robot 200 and the dock 140. In this case, the networked devices 120, 122, 124, 126, 127, 128, 140 and the range extender module (if any; not shown) can act as mesh nodes. Likewise, the mobile robot 200 may serve as a node that relays signals between the dock 140 and other nodes, such as network-enabled sensor devices 120, 122, 124, 126, 127, 128, 140 and range extenders.

[0062] The connected computing device 300 may be any suitable computing device(s), computer server(s) or cloud service(s) or combination thereof, and it will be appreciated that not all components, features, and functions described herein are required in the computing device(s) according to embodiments of the present disclosure. In some embodiments, the computing device 300 may be a mobile robot such as the mobile robot 200 or a component of a mobile robot. Figure 3 In the example of , the computing device 300 includes one or more network interfaces 330, a processor circuit ("processor") 310, and a memory 320 containing program code 322. The network interface 330 may include one or more network transmitters / receivers / transceivers configured to communicate with any associated (one or more) available servers and / or data repositories via one or more wired and / or wireless networks. The processor 310 may include one or more data processing circuits, such as general and / or special processors (such as microprocessors and / or digital signal processors) that may be collocated or distributed on one or more networks. The processor 310 is configured to execute program code 322 in the memory 320 (hereinafter described as a computer-readable storage medium) to perform some or all of the operations and methods described above for one or more embodiments. The memory 320 represents one or more storage devices containing software and data for facilitating operations according to some embodiments of the present disclosure. The memory 320 may include, but is not limited to, the following types of devices: cache, ROM, PROM, EPROM, EEPROM, flash memory, SRAM, and DRAM.

[0063] therefore, Figure 3The computing device 300 illustrates a hardware / software architecture according to some embodiments described herein that can be used to generate visual status indicators for connected devices and / or control connected devices in an operating environment. However, it will be understood that embodiments of the present disclosure are not limited to such configurations, but are intended to cover any configuration capable of performing the operations described herein. Therefore, using any appropriate combination of related methods and techniques, the computing device 300 can more generally represent any device or device network having the ability to receive, process, enhance, store and transmit data or commands to one or more connected devices 30A, 34 / 34A, 36, 40, 120, 122, 124, 126, 127, 128, 140, 142, 144, 200 in the service and from these devices. The computing device 300 can also provide additional processing to support other clients with different needs than the robot 200, such as visualization and other rich UI interactions to be presented on the user terminals 142, 144. In some embodiments, the functionality assigned herein to computing device 300 may also be performed, in whole or in part, locally on a robot, or in coordination / coordination with a group of robots and / or servers or other computing devices in communication.

[0064] The user terminals 142, 144 may be any suitable fixed or mobile computing device(s) (including desktop computers, laptop computers and "smart phones"), and it should be appreciated that not all components, features and functions described herein are required in a user terminal according to embodiments of the present disclosure and may more generally be implemented by a user terminal. Figure 4A The user terminal 400 in FIG. Figure 4A In some embodiments, the user terminal 400 is configured to transmit and / or receive communication signals through multiple wired and / or wireless communication interfaces. For example, the transceiver 440 of the user terminal 400 according to some embodiments may include a cellular communication module, an infrared (IR) communication module, a global positioning system (GPS) module, a WLAN module, a wireless personal area network (WPAN) module (such as Bluetooth, Wi-Fi, ZigBee and / or Z-wave module), and / or other types of communication modules.

[0065] The user interface 410 of the user terminal 400 includes a display 408, such as a liquid crystal display (LCD) and / or an organic light emitting diode (OLED) display. The user interface 410 may optionally include a keypad 402 or other user input mechanism on the housing of the user terminal 400. In some embodiments, the display 408 may be provided with a touch screen capability to replace and / or supplement the keypad 402. The user interface 410 may also include a microphone 406 and an earphone / speaker 404. When the earphone / speaker 404 is placed against the user's head, the housing may be designed to form an acoustic seal against the user's ears.

[0066] The keypad 402, display 408, microphone 406, speaker 404, and camera 424 may be coupled to a processor 427, such as a microprocessor or microcontroller that may be configured to control the operation of the user terminal 400. The user terminal 400 may also include a transceiver 440 and a memory 428 coupled to the processor 427. Other electronic circuit systems (such as a WLAN communication interface, a Bluetooth interface, a GPS interface, a digital signal processor, etc.) may also be included in the electronic circuit system of the user terminal 400.

[0067] The memory 428 may be a general purpose memory used to store program instructions 422 for the processor 427 and data such as audio data, video data, configuration data, and / or other data that may be accessed and / or used by the processor 427. The memory 428 may include non-volatile read / write memory, read-only memory, and / or volatile read / write memory. In particular, the memory 428 may include a read-only memory in which basic operating system instructions are stored, a non-volatile read / write memory in which reusable data (such as configuration information, directory information, and other information) may be stored, and a volatile read / write memory in which short-term instructions and / or temporary data may be stored.

[0068] The transceiver 440 includes a transmitter circuit 442, a receiver circuit 444, and a modem 446, which cooperate to transmit and receive radio frequency signals to a remote transceiver via antenna arrays 450A, 450B. The radio frequency signals transmitted between the user terminal 400 and the remote transceiver may include both traffic and control signals (e.g., paging signals / messages for incoming calls) that are used to establish and maintain communications with another party or destination. More particularly, in cooperation with the processor 427, the transceiver 440 may be configured to communicate according to a variety of radio access and / or wireless networking technologies, including (but not limited to) cellular, WLAN (including 802.11), WiMAX (Worldwide Interoperability for Microwave Access), Wi-Fi, Bluetooth, ZigBee, and / or Z-wave. In embodiments according to the present disclosure, other radio access technologies and / or frequency bands may also be used.

[0069] Reference again Figure 2C, the mobile robot 200 may include localization circuitry that builds a metric and distance-based map of walls and obstacles (e.g., using laser rangefinders, sonar, radar, triangulation, flight or phase difference calculations) and / or an occupancy map of free space (e.g., traversable ground space or ground space not occupied by objects or fixtures). The mobile robot 200 may use techniques such as scan matching, ICP (iterative closest point) and / or RANSAC (random sample consensus) to locate the robot on the (one or more) maps. Additionally or alternatively, the mobile robot 200 may include localization circuitry that builds a feature-based constellation or topology map of fingerprints of features, landmarks, fiducials and / or beacons within an occupancy map of free space (e.g., using a camera or point cloud-generating 3D scanner, and feature transformations for identifying, storing and recognizing natural or artificial keypoints, features and / or landmarks), and may use techniques such as VSLAM (vision-based / vision-simultaneous localization and mapping) to locate the robot on this occupancy map. In this document, data collected by the positioning circuitry of the mobile robot 200 based on navigation of the environment is generally referred to as occupancy data for the environment. A unique identifier for a unique room or area (shown as Zone 1, Zone 2, Zone 3) can be associated with a house room type or unique room label ("Living Room") by an end user via a user interface of any network entity, or can be automatically determined by identifying landmarks or equipment therein. The occupancy data (which can be visually represented as a "floor plan" of the operating environment) can include data defining multiple surface locations of the environment 10 (e.g., by pixels), each surface location having a value corresponding to whether the pixel location corresponds to a surface location occupied, traversed, or unexplored by the mobile robot 200.

[0070] The positioning circuitry may be defined by input from one or more sensors 270A-270H of the mobile robot 200, which may be used by the controller 220 to perform positioning in the operating environment 10. More specifically, one or more of the positioning sensors 270A-270H are configured to detect sensor readings from objects located in the operating environment 10, and the controller 220 is configured to determine the current pose ("pose" includes absolute or relative position and optionally absolute or relative orientation) of the mobile robot 200 based on the positioning data detected by the positioning sensors 270A-270H with reference to the observed object ("object" includes not only physical objects containing observable features, but also surface "objects" formed by optically or otherwise detectable surface features, such as corners, lines, patterns). The pose of the object may also be determined. The mobile robot 200 can also be configured to associate the robot pose (or position) with a room identifier, which is specifically associated with an observed object or its pose fixed in the room or observable on components of the room (walls, ceiling, lighting, doorways, furniture, etc.), as indicated by an occupancy map.

[0071] According to an embodiment of the present disclosure, a visual representation of occupancy data collected by the positioning circuit of the mobile robot 200 in response to navigation of the operating environment can be displayed on a user terminal to enhance awareness and / or control of the operating status of one or more electronic devices local to the operating environment (such as the connected devices 30A, 34 / 34A, 36, 40, 110, 120, 122, 124, 126, 127, 128, 140, 200 described herein by way of example). Figure 4B is a plan view of a user terminal 400 including an example user interface 410 configured to indicate operating conditions and / or control connected devices therein according to an embodiment of the present disclosure.

[0072] refer to Figure 4B, the user interface 410 of the user terminal 400 is configured to display a visual representation 50 of an occupancy map, which is also referred to herein as a floor plan. The floor plan 50 indicates the relative position or spatial context of one area or room in the operating environment relative to another area or room (shown as a cabin 503, a study 502, a corridor 501, a dining room 504, a kitchen 506, and a living room 507), for example, based on whether there is a physical boundary between the areas indicated by the occupancy data. The room name / type / label can be manually typed by the user, or can be automatically determined based on the identification of objects or connected devices therein. The floor plan 50 is filled with icons 30a, 34a-34f, and 40a, which represent electronic devices within the boundaries of the operating environment or locally. The placement of icons 30a, 34a-34f, and 40a on the floor plan 50 indicates the corresponding spatial position of the electronic devices relative to the room / area, and the graphic characteristics of icons 30a, 34a-34f, and 40a indicate the corresponding operating states of these electronic devices. The user interface 410 may also be configured to display a list 420 of devices local to the operating environment and a user input element 550 configured to receive a user-specified time for desired operating status information within a user-specified time range, as described below with reference to Figure 5 Described in more detail.

[0073] Figure 5-8 is a schematic plan view illustrating operations for displaying and controlling the operation of electronic devices based on the relative spatial position of the electronic devices in an operating environment according to some embodiments of the present disclosure. By way of example, reference is mainly made to displaying and controlling lighting-related devices (such as Figure 1B The operation of the lamp 34, the combustible lamp 34A and the shutter 30A) is described and illustrated Figure 5-8 However, it will be understood that embodiments of the present disclosure are not limited thereto, but may include displaying and / or controlling the operation of other electronic devices local to the operating environment, including but not limited to lights / bulbs, door / window sensors, door locks, speakers, thermostats, environmental sensors (including but not limited to temperature, humidity, air quality and lighting level sensors), blinds, voice interface devices, surveillance cameras, etc.

[0074] Figure 5 The plan view of FIG. 1 shows an example operation for displaying a visual representation of an operating environment according to an embodiment of the present disclosure, the visual representation visually indicating the relative spatial position of the device and its operating state. More specifically, Figure 5 An example user interface 510 of a user terminal (such as user terminals 142, 144, 400) is illustrated, which displays an operating environment (such as Figure 1A-1BThe visual representation 50 of the living space 20 of the embodiment of the present invention is illustrated as a plan view 50 populated with icons 30a, 34a-34f, 40a indicating the corresponding spatial locations and operating states of the electronic devices in the operating environment. Figure 5 In the example of FIG. 1 , the electronic devices represented by icons 34a-34f are connected light bulbs / light fixtures (such as Figure 1B 34 and / or combustible lamp 34A), the device represented by icon 30a is a connected blind (such as Figure 1B 30A), and the electronic device represented by icon 40a is a connected HVAC vent (such as Figure 1B The populated floor plan 50 provides a visual "snapshot" of the operating environment, with the icons 30a, 34a-34f, 40a indicating the location and status of detected or user-specified devices. The user interface 510 can thus visually indicate the whole-home operating mode and confirm that the devices represented by the icons 30a, 34a-34f, 40a are operating and / or cooperating according to the user's expectations.

[0075] The floor plan 50 may be populated with icons 30a, 34a-34f, 40a automatically (i.e., without human intervention) and / or in response to user input via the user interface 510. For example, a user terminal executing an application providing the user interface 510 may receive information identifying electronic devices (including both connected and non-connected devices) in the operating environment and / or their respective operating states from one or more network sources, including, but not limited to, the mobile robot 200, the hub 110, the controller 120, or other connected devices. A list 420 identifying local devices and / or operating states may thereby be generated. Additionally or alternatively, information identifying devices and / or their operating states may be received via the user interface (e.g., via manual entry in the device list 420). Similarly, the respective spatial locations of the devices represented by the icons 30a, 34a-34f, 40a may be determined automatically (e.g., based on occupancy data and / or wireless signal coverage data collected by the mobile robot) and / or in response to receiving user input (e.g., from a Figure 4B The device list 420 is determined by drag and drop input).

[0076] Figure 5 The user interface 510 also includes a user input element 550 configured to receive a user specification of a time during which operational status information of the device represented by the icon 30a, 34a-34f, and / or 40a is desired. Figure 5In the example of, user input element 550 is shown as a time slider bar, which is configured to receive a user input that manipulates slider element 551 to a corresponding position along the length of bar element 555. Bar element 555 visually represents a time range, and the operating state information of the device represented by icon 30a, 34a-34f and / or 40a has been stored or can be accessed in other ways within this time range. A mark indicating corresponding time 552, 553, 554 (respectively past, present and future time) is provided along the length of bar element 555. User interface 510 is configured to detect the position of slider element 551 along the length of bar element 550 in response to the manipulation of slider element or other user input, and the user terminal is configured to determine the time specified by user input from this position based on the corresponding relationship of the position of slider element 551 along the time range represented by bar element 555. User interface 510 is thus configured to display a status indication (also referred to herein as a time status indication) that represents the operational state of the device represented by icon 30a, 34a-34f and / or 40a and is associated with time data corresponding to the time specified by the user input.

[0077] In some embodiments, the time range visually represented by user input element 550 may also be specified by the user. Figure 5 The bar element 555 shown represents a period of time in a day, with the mark 552 at one end of the bar element 555 corresponding to "12am" and the mark 554 at the other end of the bar element 555 corresponding to "11:59pm", whereby, depending on the relative position of the slider element 551 along the length of the bar element 555, manipulation of the slider element 551 results in the display of the time within the day represented by the bar element 555 (via the mark 553). Figures 11A-11E Illustrated is example information regarding the operating status and operating status interoperation of electronic devices in operating environment 50 that may be displayed in response to real-time manipulation of slider element 551 along the length of bar element 555 selected to represent a time range of one day.

[0078] In another example, bar element 555 may be selected to represent a time range of one week, with indicia 552 at one end of bar element 555 corresponding to "Sunday" and indicia 554 at the other end of bar element 555 corresponding to "Saturday," whereby manipulation of slider element 551 results in display of the day of the week (e.g., Monday, Tuesday, Wednesday, Thursday, Friday; via indicia 553) represented by bar element 555, depending on the relative position of slider element 551 along the length of bar element 555. Positioning slider element 551 between consecutive days (e.g., between Monday and Tuesday) may display the time of day between the consecutive days (e.g., 12 p.m.). Sub-portions of a day, week, month, etc. may similarly be represented by the relative positions of bar element 555 and slider element 551 along their lengths. More generally, the time range represented by user input element 550 may be user selectable, and manipulation of slider element 551 along bar element 555 may display different ranges and time measurement units (hours, days, weeks, etc.) based on the selected time range.

[0079] As such, user interface 510 is configured to display a temporal status indication of the device represented by icons 30a, 34a-34f, and / or 40a, corresponding to a time selected or otherwise specified by a user via user input element 550. Such a temporal status indication may include actual device operating status information (based on actual past or present operation of the device, as previously recorded or currently reported) and / or expected device operating status information (based on a predetermined future operation of the device, as specified by the user), depending on the position of slider element 551. The actual and expected / predicted operating states (and the past, present, or future time data associated therewith) may be continuously recorded and updated by the devices 30A, 34 / 34A, 36, 40 themselves and / or by connected computing devices coupled thereto (such as one or more of the network nodes 110, 120, 122, 124, 126, 127, 128, 200, 140, 142, 144, and 150), and may be stored in a database accessible by one or more of the connected devices via the networks 160 and / or 170. For example, in some embodiments, a computing device (such as computing device 300) may receive data indicating the actual operating state of the electronic device (and associated past or present time data) and / or data indicating the expected operating state (and associated future time data) from one or more different sources (including, but not limited to, the hub 110, the controller 120, the mobile robot 200, or other connected devices), and may store the corresponding operating state of the electronic device and the time data associated therewith in a database. Data indicative of operating status and associated time data may be requested (or "pulled") by the computing device, or may be reported (or "pushed") by an electronic device and / or network source in communication therewith.

[0080] In some embodiments, the past time status indication may include, but is not limited to, the presentation of the following information: when / how long the light was turned on / off in the past, or whether the light was always on (e.g., Fig.11A 54A, which changes based on the relative position of slider element 551 within the display period represented by the length of bar element 555); where / when motion is detected; whether the device performs as intended by the user; how the temperature changes throughout the day (e.g., as indicated by the operation of an environmental sensor or as set by a thermostat, such as Fig. 11B50 ); the time(s) that a door or window is opened / closed; the time a door is locked and unlocked; and / or a graphical display of location-specific temperature, humidity, and / or air quality readings from sensors on the mobile robot 200 and / or fixed environmental sensors positioned throughout the environment. In addition, the user interface 510 can be configured to display recorded images / video from a surveillance camera as an overlay 1120 on FIG. 50 , which changes correspondingly to the time selected via the input element 550, such as Fig. 11C In an embodiment, the time selected via input element 550 is associated with a timestamp in a video feed collected and stored by mobile robot 200, and the image / video corresponds to an image captured by a surveillance camera mounted to mobile robot 200 at the selected time. Moreover, based on information received from a connected energy monitor device, user interface 510 can be configured to visually represent the energy usage of a device (or total amount for a room) instantaneously or over a given time period, such as Fig.11D 54A, which varies based on the relative position of slider element 551 within the displayed time period represented by the length of bar element 555. More generally, past temporal status indications can provide a visual history of instantaneous operation and operating patterns of electronic devices in the overall operating environment at a user-specified time or within a user-specified time range.

[0081] In some embodiments, the future time status indication may include, but is not limited to, the presentation of the following information: a device schedule for the rest of the day; the time(s) at which lights in the living space are illuminated (e.g., automatically illuminated at a predetermined time); information regarding whether the blinds will close near sunset; information regarding whether all devices will close at a predetermined time in the evening; whether the thermostat is set to the expected degree for night mode operation; whether the door locks are scheduled to lock at night; whether the alarm system sensors are set to arm tonight; and / or the next scheduled time that the mobile robot 200 (e.g., a robotic vacuum cleaner) will start and begin a task. The future time status indication can therefore provide a visual schedule of the operating modes of electronic devices throughout the operating environment within a user-specified time range. Conflicts such as conflicts between two or more scheduled device operations, such as location-based conflicts, may also be visually indicated. As used herein, a conflict between two or more devices may refer to a situation where the operation of one device may interfere with the operation of another device or otherwise be problematic with respect to the operation of another device. For example, if Fig.11EAs shown, if the alarm system's motion sensor 1107 is armed for a period of time during which operation of the robotic vacuum cleaner 1200 is predetermined in an area of ​​the operating environment 50 within the field of view of the motion sensor 1107, then the user interface 510 can indicate a conflict via a combination of icons and / or text in response to selection of a time via the relative position of the slider element 551 within the display period represented by the length of the bar element 555.

[0082] Icons 30a, 34a-34f, 40a may visually represent or indicate operating status in a variety of ways, including through shading, color, symbols, punctuation, and / or text. Figure 5 As shown, the icon 34f representing the light bulbs / light fixtures in the living room 507 has a solid pattern to indicate that these light bulbs / light fixtures are turned on, while the icon 34a representing the light bulbs / light fixtures in the cabin 503 is shaded to indicate that the light bulbs are off. The icons 34c, 34d and 34e representing the light bulbs / light fixtures in the dining room 504, the corridor 501 and the kitchen 506 (which are all spatially adjacent to the living room 507) have spots to indicate that these light bulbs / light fixtures are dimmed. At the same time, the icon 34b representing the light bulb / light fixture in the study 502 displays an exclamation point (!) to indicate that there is a problem with its operation. These visual indications are provided by way of example only, and it will be understood that other visual indications and / or text can be displayed by the user interface 510 in response to the operating time and desired time range selected by the user via the slider bar 550 to indicate the actual or expected / predetermined operating state of the device represented by the icons 34a-34f. More generally, in response to user input specifying a desired time, user interface 510 may visually indicate actual or expected activity patterns of the devices represented by icons 34a - 34f in the overall operating environment.

[0083] The mobile robot can be positioned based on sensors (such as Figure 2A-2C The floor plan(s) described herein may be generated based on occupancy data detected by sensors 270A-H of the mobile robot 200. In some embodiments, the floor plan(s) described herein may be generated based on occupancy data and / or based on a wireless transceiver (such as a wireless transceiver) of the mobile robot. Figure 2A-2C The relative spatial location of the local electronic device is determined based on the signal coverage data obtained by the wireless transceiver 252 of the mobile robot 200. The operations for generating a map and determining the spatial location based on the occupancy data and / or signal coverage data obtained by the mobile robot are further described in the commonly owned U.S. Patent Application No. 15 / 588,117 filed on May 5, 2017, the disclosure of which is incorporated herein by reference in its entirety.

[0084] The list of electronic devices 420 local to the operating environment may be generated based on information identifying the devices and / or their operating states received via the network 160 or 170. For example, the list of electronic devices 420 local to the operating environment may be generated by requesting a gateway / hub such as Figure 1B In certain examples, the hub 110 may include a user interface for merging the device (e.g., SmartThings) with the device to which they are coupled to identify the device and / or operating status. TM or Wink Hub) to connect to and control IoT devices and is coupled to SmartThings TM Devices connected to a network or Wink hub can be identified by querying the hub. In another example, a networked home energy monitoring device (such as a Sense home energy monitor) can automatically detect and identify devices and operating data (whether network-enabled or not) based on their power consumption. That is, the list 420 (and / or the populated floor plan 50) can visually represent electronic devices produced by different manufacturers and / or operating according to different protocols. Other operations for obtaining device identification may include (but are not limited to) performing port scans to identify exposed services, performing pattern matching on received responses, and / or interacting with discovered services to extract more structured information (e.g., properties advertised by devices via network "discovery" protocols (such as UPnP or mDNS)). Additionally or alternatively, information identifying a device, its operating state, and / or its spatial location can be received via a user interface (e.g., by manual entry in the device list 420). For example, the icons 30a, 34a-34f, 40a shown in the list 420 can be dragged and dropped to corresponding locations on Figure 50.

[0085] The connected computing device(s) as described herein may also be configured to transmit control signals to control the operation of one or more electronic devices based on their respective spatial locations in the operating environment and the relative spatial context of the area they are located in. In some embodiments, the floor plan may indicate whether there are walls and / or obstacles that separate one area / room of the operating environment (and devices located in that area / room) from another area / room. Figure 6 is a plan view illustrating an example user interface 610 of a user terminal, such as user terminals 142, 144, 400, for controlling connected devices in a continuous living space extending between multiple adjacent rooms without walls or obstacles therebetween, according to an embodiment of the present disclosure.

[0086] like Figure 6As shown, based on occupancy data collected by one or more positioning sensors of a mobile robot (such as mobile robot 200) in response to navigation of an operating environment (such as living space 20), a user interface 610 displays a visual representation 60 of the operating environment. The visual representation is shown as a floor plan 60 populated with icons 30a, 34a / e / f, 40a, which indicate the corresponding spatial locations and operating states of detected or user-specified electronic devices local to the operating environment. Figure 6 In the example of FIG. 1 , the electronic devices represented by icons 30 a , 40 a and 34 a / e / f are blinds, HVAC vents and lamps / lights, such as Figure 1B Specifically, icons 30a, 34a / e / f, 40a in the filled floor plan 60 indicate the location of the blinds 30A and lights 34A in the kitchen 606, the location of the HVAC vents 40 and lights in the cabin 603, and the location of the lights 34A in the living room 607. The wavy lines shown in icon 40a indicate the operating status of the HVAC vents 40 as "on" or "open", the solid line pattern of icon 34e indicates the operating status of the lights in the kitchen 606 as "on", the spotted patterns of icons 34a, 34f indicate the operating status of the lights in the cabin 603 and the living room 607 as "dim", and the spacing between the triangular elements of icon 30a indicates the operating status of the blinds as "open".

[0087] The floor plan 60 also indicates that there are no walls or obstacles (represented by dotted area 61) between the three rooms (living room 607, kitchen 606, and cabin 603), thereby defining a continuous space 650. As described above, the absence of walls / obstacles can be indicated by occupancy data detected by the positioning sensor in response to navigation of the operating environment and / or signal coverage data acquired by the wireless transceiver of the mobile robot. That is, although the living space (and / or the grouping of devices therein) can be defined as separate rooms (shown as living room 607, kitchen 606, and cabin 603) through user input, Figure 6 The actual floor plan 60 of the living space shown in includes a plurality of adjacent rooms defining a wider continuous space 650.

[0088] Based on the presence or absence of walls / obstacles or other physical boundaries in the plan view, embodiments of the present disclosure may also identify the relative spatial context of an area / room in the operating environment, and may segment or group the electronic devices in the operating environment into subsets corresponding to the relative spatial context of the area / room. Figure 6In the example of , the blinds, HVAC vents, and light fixtures represented by icons 30a, 34a / e / f, and 40a can be grouped into a common subset in response to their respective spatial locations being identified as corresponding to continuous space 650, and further grouped into respective subsets based on the rooms in which they are located (living room 607, kitchen 606, and cabin 603). Thus, embodiments of the present disclosure can coordinate separate control of electronic devices based on the relative spatial context of the rooms, in conjunction with the groups or subsets with which the electronic devices can be associated, rather than solely based on artificial boundaries (defined by the user or otherwise) between the rooms in which the electronic devices are located.

[0089] For example, when providing motion activated lighting in response to detecting the presence of a user in the kitchen 606, it may also be desirable to illuminate the living room 607 and the cabin 603 at the same time, because these three rooms are included in the continuous space 650. However, a user-defined control scheme based on the location of the devices represented by the icons 30a, 34a / e / f, 40a may be logistically challenging because the user-based definition of the space 650 of a large room reduces the granularity of connected devices that can be controlled, and the user-based association of these devices with multiple, respectively named rooms may not be logically coordinated based on the room names. However, by identifying the relative spatial context of the rooms and the corresponding spatial locations of the devices, embodiments of the present disclosure can individually control the devices represented by the icons 30a, 34a / e / f, 40a based on the continuous space 650 in which these devices are located, while also allowing control based on the individual rooms (living room 607, kitchen 606, cabin 603) in which they are located.

[0090] In particular, in response to detecting the presence of a user in kitchen 606 and identifying living room 607 and cabin 603 as part of the same continuous space 650, a control signal may be transmitted to turn on the lights in kitchen 606 (represented by the solid pattern of icon 34e) and turn on the lights in living room 607 and cabin 603 at a lower level (represented by the spotted patterns of icons 34a and 34e). Conversely, turning on only the lights in kitchen 606 in response to detecting the presence of a user in kitchen 606 would cause the user to gaze at the dark living room 607 and cabin 603 portions of continuous space 650. Such individual control of electronic devices based not only on their association with the continuous space but also on the spatial context of the room in which the devices are located can provide a more pleasing automatic lighting experience.

[0091] In another example, if a window 30 in the kitchen 606 is detected to be in an "open" state (indicated by the separation between the triangular shutter elements in icon 30a), and if efficient HVAC operation is specified, a control signal may be transmitted to the connected HVAC vent 40 in the cabin 603 to switch its operational state from "open" (indicated by the wavy line in icon 40a) to "closed" based on the location of the connected HVAC vent 40 in the cabin 603 in spatial proximity to the open window 30, even if the HVAC vent 40 is located in a different room (in the cabin 603, not in the kitchen 606 where the window 30 is located). That is, control based on identifying that the HVAC vent 40 and the open window 30 are in a continuous space 650 may be more advantageous than control based solely on the spatial location of the HVAC vent 40 and the window 30 in separate rooms 603 and 606 of the operating environment.

[0092] The connected computing device(s) as described herein may also be configured to transmit control signals to predictively control the operation of one or more connected electronic devices based on device operation and / or environmental conditions in adjacent areas of the operating environment. For example, in controlling motion-activated events such as lighting, it may be desirable to control the operation of the connected electronic device before (or when) a user enters the space, rather than controlling it after detecting the user's presence in the space. Figure 7 is a plan view illustrating example operations for predictive control of a connected electronic device in response to detecting a user's presence and / or trajectory in an area of ​​an operating environment proximate to a location of the connected electronic device in accordance with an embodiment of the present disclosure.

[0093] like Figure 7 As shown, based on occupancy data collected by one or more positioning sensors of a mobile robot (such as mobile robot 200) in response to navigation of an operating environment (such as living space 20), the relative spatial context of rooms 701, 702 in the operating environment 70 can be identified. For example, the data collected by the mobile robot can be put together to construct or generate a plan view of the interior space. In particular, Figure 7 The relative spatial context of a corridor 701 to a bedroom 702 in the operating environment 70 is illustrated. In addition, based on occupancy data and / or signal coverage data collected by the mobile robot, information received from a user via a user interface of a user terminal (e.g., user terminal 142, 144, 400, etc.), and / or from one or more network nodes (such as Figure 1ABased on the information received by the nodes 110, 126, 127, 128, 200, 140 and 150 of the operating environment 70, the electronic devices 34A1, 34A2, 120A1, 120A2 are associated with corresponding spatial locations in the operating environment 70. In particular, the light 34A1 and the motion sensor 120A1 are associated with corresponding locations in the corridor 701, while the light 34A2 and the motion sensor 120A2 are associated with corresponding locations in the bedroom 702.

[0094] exist Figure 7 In the example of , the operation of lights 34A1 and 34A2 can be controlled respectively based on the respective spatial positions of lights 34A1 and 34A2 and the relative spatial context of rooms 701 and 702 in which they are located. For example, in addition to turning on light 34A1 in response to detecting the presence of user P in corridor 701 within field of view 130A1 of motion sensor 120A1, the embodiments described herein can also predict which room user P is likely to enter and take proactive actions based on the adjacency of spaces 701 and 702 indicated by the relative spatial context. In particular, since corridor 701 has been identified as being adjacent to bedroom 702, a control signal can be sent to turn on light 34A2 in bedroom 702 in response to detecting the presence of user P in corridor 701 before user P enters bedroom 702 and is detected by motion sensor 120A2 in the bedroom. That is, based on the relative spatial position of the motion sensor 120A1 to the light 34A2 and the relative spatial context of the rooms 701, 702 indicated by the mobile robot occupancy data, the presence of the user P in the bedroom 702 is predicted (and the connected device 34A2 in the bedroom 702 is controlled accordingly) according to the presence of the user P detected in the hallway 701. In contrast, if the light 34A2 in the bedroom 702 is not activated before the motion sensor 120A2 in the bedroom is triggered, the user P may have to take many steps in the dark, especially when the field of view 130A2 of the motion sensor 120A2 does not cover the door 702d to the bedroom 702.

[0095] In some embodiments, the motion sensor 120A1 in the corridor 701 may also be configured to detect the trajectory of the user P (toward or away from the door 702d to the bedroom 702), and the light 34A2 in the bedroom may also be controlled based on the detected trajectory rather than just based on the detected presence. That is, if the motion sensor 120A1 detects the presence of the user P in the corridor 701 and detects the trajectory of the user P toward the bedroom door 702d, then a control signal may be transmitted to activate the light 34A2 in the bedroom 702. Conversely, if the motion sensor 120A1 detects the presence of the user P in the corridor 701 but detects that the user P is not moving toward the bedroom door 702d, then a control signal to activate the light 34A2 in the bedroom 702 may not be transmitted.

[0096] Furthermore, electronic devices may be controlled differently in response to detecting the same event based on the relative spatial locations of the electronic devices and / or the relative spatial context of the room in which they are located. Figure 7 When the lighting of the adjacent spaces 701 and 702 in the hallway 701 is detected by the motion sensor 120A1, the light 34A1 in the hallway 701 (where the motion sensor 120A1 directly detects the presence of the user P) may be activated to a higher brightness than the light 34A2 in the bedroom 702 where the user P is predicted to be present. That is, when the user P walks along the hallway 701 toward the bedroom 702, the light 34A1 may be activated to provide brighter task lighting, while the light 34A2 in the adjacent bedroom 702 may be activated to provide a lower level of accent lighting. Thereafter, the light 34A2 may be activated to provide brighter task lighting in response to the motion sensor 120A2 detecting the actual presence in the bedroom 702 (or the predicted presence in the bedroom 702 based on the proximity to the motion sensor 120A1), and the light 34A1 may be dimmed or turned off. However, in some embodiments, in response to motion sensor 120A2 previously detecting the presence of another person (e.g., a spouse or significant other) in bedroom 702, light 34A2 in bedroom 702 may not be activated to a brighter state, even if no further motion is detected (e.g., if the spouse / significant other is sleeping). That is, the prior detection by motion sensor 120A2 may override subsequent control operations of light 34A2 in response to subsequent detection of user P by motion sensor 34A2. In response to detecting that user P leaves bedroom 702, this control scheme may be reversed. Such different control schemes for connected devices based on the relative spatial locations of the connected devices and / or the relative spatial context of the rooms in which they are located may be displayed via user interfaces described herein (such as a user interface described herein). Figure 5 's user interface 510) is more intuitively specified.

[0097] The connected computing device(s) as described herein may also be configured to transmit control signals to control the operation of one or more connected electronic devices based on the operating state of the electronic devices in the adjacent space and / or environmental conditions in the adjacent space. Figure 8 is a plan view illustrating an example operation for controlling a connection device based on a condition in an adjacent space according to an embodiment of the present disclosure.

[0098] like Figure 8 As shown, the relative spatial context of rooms 701, 702 in the operating environment 70' can be identified based on occupancy data collected by one or more positioning sensors of a mobile robot (such as mobile robot 200) in response to navigation of the operating environment (such as living space 20). Specifically, Figure 8The relative spatial context of a corridor 701 to a bedroom 702 in an operating environment 70' is illustrated. In addition, based on occupancy data and / or signal coverage data collected by the mobile robot, information received from a user via a user interface of a user terminal (such as user terminals 142, 144, 400), and / or from one or more network nodes (e.g., Figure 1A Based on the information received by the nodes 110, 126, 127, 128, 200, 140 and 150 of the operating environment 70, the electronic devices 34A1, 34A2, 120A1, 120A2, 120B are associated with the corresponding spatial positions in the operating environment 70'. In particular, the lamp 34A1 and the motion sensor 120A1 are associated with the corresponding position in the corridor 701, while the lamp 34A2, the motion sensor 120A2 and the brightness sensor 120B are associated with the corresponding position in the bedroom 702.

[0099] exist Figure 8 In the example, the operation of lights 34A1 and 34A2 can be controlled respectively based on the respective spatial positions of lights 34A1 and 34A2 in rooms 701 and 702, the relative spatial contexts of rooms 701 and 702 in which they are located, and their respective operating states. For example, in response to detecting the movement of user P in bedroom 702 and / or the trajectory of user P toward door 702d through motion sensor 120A2, embodiments of the present disclosure can predict the presence of user P in corridor 701, and in response thereto, can transmit a control signal to activate light 34A1 in corridor 701. In addition, embodiments of the present disclosure can also control light 34A1 in corridor 701 based on the operating state of light 34A2 in bedroom 702. Specifically, as Figure 8 As shown, in response to detecting low or dim light output from the lamp 34A2 in the bedroom 702 (as reported by the lamp 34A2 itself or the luminance sensor 120B), a control signal may be transmitted to the lamp 34A1 in the hallway to provide a similar low level or dim light output so that when the user P arrives in the hallway 701, the user P will encounter consistent lighting to avoid blinding the user P. These operations may occur before the motion sensor 120A1 detects the presence of the user P in the hallway 701. That is, when the user P leaves the dark bedroom 702 and enters the hallway 701, the hallway light 34A1 may be controlled based on the operating state of the lamp 34A2 at the location where the user P arrives. Conversely, when the user P leaves a brightly lit area, the lighting in the adjacent space may be controlled to be relatively bright.

[0100] In addition, because the luminance sensor 120B in the bedroom 720 can detect both artificial and natural lighting, the lamp 34A1 in the corridor 701 can be controlled based on the light level detected by the operating state of the luminance sensor 120B in the adjacent space 702, separately from or in addition to the operating state of the lamp 34A2. That is, the luminance sensor 120B may detect a high brightness level in the bedroom 702 due to sunlight or other ambient lighting, and the lamp 34A1 in the windowless corridor 701 can be controlled to provide a similar lighting level based on the brightness level detected by the luminance sensor 120B. More generally, a connected device located in one area of ​​the operating environment can be controlled based on one or more detected conditions (such as those affected by the operation of the electronic device and / or the environment) from another area of ​​the operating environment where the user is located. Thus, understanding the layout of a home or other operating environment and the locations of connected sensors, devices, and personnel can provide a "smarter" smart home experience.

[0101] Further operations and advantages according to embodiments of the present disclosure may include automatically generating and / or customizing automatic rules for controlling connected devices in response to associating the device with a corresponding spatial location in an operating environment. For example, in response to receiving a user input of dragging and dropping a device at a location on a displayed floor plan, one or more conditional instructions for device operation may be automatically generated based on the type of device, the type of room corresponding to or adjacent to the location, and / or the spatial context of the room relative to other rooms and / or devices in the operating environment. In one example, in response to detecting a user input associating a motion sensor type device with a room on a diagram, a conditional instruction may be automatically generated to turn on the lights in the room when motion is detected. Similarly, in response to detecting a user input associating a motion-based alarm sensor with a spatial location, a conditional instruction may be automatically generated to deactivate the motion-based alarm sensor based on a predetermined operating time of the mobile robot at the same spatial location.

[0102] In another example, in response to determining that the spatial location of the lighting device is close to a natural light source (e.g., a window), conditional instructions can be automatically generated to specify the operation of the lighting device based on expected or detected lighting conditions. For example, conditional instructions can be automatically generated to specify that the lighting device should be turned off during daytime hours (based on expected natural light during the daytime hours) or turned on based on insufficient light from the natural light source (detected by a brightness detector). Conditions can also include conditions related to seasons and / or weather (e.g., the daytime hours corresponding to expected natural light can vary based on seasons and / or weather).

[0103] In yet another example, in response to detecting that the lighting device in the bedroom is off, a control signal for turning on the lighting device in the bedroom may not be transmitted in response to detecting motion in an adjacent hallway. The room type may be determined by receiving user input of the room name, or the room type may be automatically detected based on visual recognition of objects in the room and / or identification of electronic devices in the room. That is, conditional instructions for operating electronic devices may be automatically created based on the association of the device with a spatial location, and the device may be operated in response to the occurrence of one or more conditions indicated by the conditional instructions (including conditions based on time, season, temperature, and / or weather).

[0104] Conditional instructions for operating electronic devices can also be automatically adjusted based on the detection of the presence of one or more users or mobile robots in the operating environment and / or based on their positions in the relative spatial context. For example, in the above example of a lighting device adjacent to a natural light source, the threshold for sufficient light from the natural light source (thus, the occurrence of the condition that triggers the activation of the lighting device) can be changed based on the presence of a user or a camera-based mobile robot, which may require greater lighting requirements. In this example, a schedule associated with the mobile robot can also be used to determine the presence and adjust the conditional instructions; for example, if the mobile robot is scheduled to clean the spatial location of the lighting device adjacent to the natural light source at a specific time of the day, and if the season and / or weather indicates that the natural lighting provided by the natural light source at the scheduled time of the day is insufficient for the camera-based imaging of the mobile robot, then the lighting device can be activated. More generally, conditional instructions for operating electronic devices can be automatically generated not only based on the association of the device with the spatial location, but also based on any combination of conditions that may be detected or expected to occur at the spatial location.

[0105] In addition, embodiments of the present disclosure may also control the operation of electronic devices in the operating environment to verify the consistency and / or accuracy of occupancy data. For example, during navigation of the operating environment, a mobile robot (such as mobile robot 200) may activate connected lights to their "on" and "off" states to visually identify the spatial location of the connected lights in the environment and add the spatial location of the lights to the occupancy data. This updated occupancy data may be compared with previously collected occupancy data to resolve ambiguities and / or update changes in the environment corresponding to the floor plan. In addition, the mobile robot may selectively activate connected lighting devices and establish a "light map" for each lighting device to determine the corresponding lighting impact on the space of each lighting device. Such a light map may allow selective control of specific lighting devices to achieve desired lighting levels and coverage. Such operations for selectively activating connected devices and / or comparing activations to previous results may also be performed by one or more other network elements in the operating environment.

[0106] Moreover, embodiments of the present disclosure are not limited to representing the spatial location and status information of connected devices in an operating environment. For example, some embodiments may provide temporal status information based on device energy monitoring, even for non-connected devices local to the operating environment. In particular, a networked energy monitoring device may be used to determine the energy usage of non-connected devices, such as a sensing home energy monitor, which automatically detects and identifies non-connected devices based on their power consumption curves. For example, air conditioning units and refrigerators in the operating environment may be uniquely identified based on their corresponding energy usage and power consumption curves. The operating status information and corresponding spatial location of such non-connected devices may also be determined from the energy usage and corresponding location of power outlets. During navigation of the operating environment, the spatial location of such non-connected devices may also be visually identified by a mobile robot. Therefore, non-connected devices may be identified, added to a floor plan to indicate their corresponding spatial location, and associated with their energy usage information to provide a visual indication of the location and status in the operating environment as described herein.

[0107] Figure 9-10 is a flow chart illustrating operations that may be performed by or in response to instructions from at least one processor of a computing device according to embodiments of the present disclosure. Figure 9-10 The operations may represent executable routines or instructions stored in a non-transitory computer-readable medium (such as the memory devices 222, 320, 428 described herein) and executable by one or more processors (such as the processors 220, 310 and / or 427 described herein) to implement the specified routines or instructions.

[0108] Fig. 9 is a flowchart illustrating operations that can be performed by a user terminal according to an embodiment of the present disclosure. Specifically, Fig. 9 The operations of may be performed by instructions from one or more processors of a user terminal (eg, the user terminals 142, 144, 400 described herein) or in response to such instructions. Fig. 9, at block 905, occupancy data for the operating environment is received in response to navigation of the operating environment by the mobile robot. The occupancy data may be generated based on positioning data detected by one or more positioning sensors of the mobile robot in response to navigation in the operating environment, and the occupancy data may be received via one or more receivers of the user terminal. The positioning data may include distance measurements from range sensors, occupancy and obstacle determinations from obstacle and proximity sensors, and feature and landmark determinations from cameras and other pattern / image / reference observation sensors. The positioning data may be detected in a surface mapping routine, for example, using VSLAM technology to record 2-dimensional occupancy data representing the location of the operating environment and / or the posture of the mobile robot on it. At block 910, a visual representation of the operating environment is generated based on the occupancy data, and the visual representation is displayed via a user interface of the user terminal. More specifically, in response to receiving the occupancy data, one or more processors of the user terminal may generate and transmit instructions to operate the user interface to display a plan view of the operating environment (such as the plan view 50 of Figures 4-5).

[0109] At block 915, information identifying an electronic device in the operating environment and a corresponding operating state of the electronic device is received. Information identifying the electronic device and / or information identifying its corresponding operating state may be received (or otherwise determined from data provided by) one or more network elements in the operating environment (e.g., based on information collected by the hub 110, the sensor 120, and / or the controllers 126, 127, 128), a mobile robot (e.g., based on occupancy data or wireless signal coverage data collected thereby during navigation of the operating environment), and / or a user interface of a user terminal (e.g., based on input to add icons 30a, 34a-34f, 40a to the list 420).

[0110] At box 920, the visual representation of the operating environment is filled with visual indications of the corresponding spatial positions of the electronic device and status indications of its corresponding operating state. For example, based on information identifying the electronic device and its corresponding operating state, the displayed floor plan of the operating environment (such as the floor plan 50 of Figures 4-5) can be filled with icons (such as icons 30a, 34a-34f, 40a) that graphically represent the corresponding electronic device. The icon can be displayed on the floor plan at a spatial position corresponding to the actual position of the electronic device in the operating environment. The spatial position can be determined from occupancy data and / or wireless signal coverage data collected by the mobile robot in response to navigation of the operating environment and / or from user input received via a user interface (e.g., dragging icons 30a, 34a-34f, 40a shown in list 420 into Figure 50).

[0111] In some embodiments, the icon itself can also graphically indicate the corresponding operating state of the electronic device represented by the icon. Figure 5 As shown in the user interface 510 of FIG. 5 , the floor plan 50 is populated with light bulb icons 34a-34e representing connected lighting devices in the operating environment, vent icons 40a representing connected HVAC vents in the operating environment, and window icons 30a representing connected blinds in the operating environment. The symbols and / or shading of the light bulb icons 34a-34e, the wavy line of the vent icon 40a, and the extension of the blinds element across the window icon 30a indicate the respective operating states of the devices represented by the icons 34a-34e, 40a, and 30a.

[0112] Fig.10 is a flowchart illustrating operations that can be performed by a computing device according to an embodiment of the present disclosure. Specifically, Fig.10 The operations may be performed by or in response to instructions from one or more processors of a network-enabled computing device, such as (but not limited to) hub 110, controllers 126, 127, 128, mobile robot 200, dock 140, user terminal 142, terminals 144, 400, server 150, and / or computing device 300. Reference Fig.10 At block 1005, occupancy data for the operating environment is received in response to the mobile robot navigating the operating environment. Fig. 9 As described in block 905 of , occupancy data may be generated based on positioning data detected by one or more positioning sensors of the mobile robot, for example using VSLAM technology. In some embodiments, occupancy data may be received via one or more network receivers of a computing device.

[0113] At box 1010, based on the occupancy data, the electronic devices in the operating environment are associated with the corresponding spatial positions of the operating environment. As described above, the spatial position of the electronic device can be determined based on the occupancy data and / or wireless signal coverage data collected by the mobile robot in response to the navigation of the operating environment and / or based on the user input received via the user interface. In addition, the occupancy data can indicate the room in the operating environment, for example, based on the presence or absence of physical boundaries between areas of the operating environment encountered by the mobile robot during navigation. The relative spatial context of the room in the operating environment can be further identified based on the occupancy data, and the electronic devices can be segmented or grouped into corresponding subsets based on their corresponding spatial positions according to the relative spatial context of the room in the operating environment. For example, electronic devices in / near the same room or within / near the boundaries of consecutive rooms can be grouped into the same subset, while electronic devices outside the boundaries of a room or consecutive rooms can be grouped into different subsets.

[0114] At block 1015, control signals are transmitted to one or more electronic devices to control their operation based on their respective spatial locations (and / or based on the relative spatial context of the room corresponding to the respective spatial locations). For example, control signals may be transmitted to the electronic devices based on their respective groupings or subsets in a common room or continuous space. In one example, in response to receiving a user input selection (e.g., by clicking or tapping a portion of a displayed plan view of a room) or otherwise indicating a particular room, a corresponding control signal may be generated and transmitted to activate (e.g., dim) lighting devices located in a particular room (based on their common grouping). In another example, lighting devices in a continuous area spatially located in an operating environment may be similarly controlled based on their grouping in a common subset, even though they are physically located in different rooms. Moreover, control signals may be transmitted to the electronic devices based on the type of room corresponding to the respective spatial locations of the one or more electronic devices, and / or device activity and / or environmental conditions in an area of ​​the operating environment adjacent to their respective spatial locations.

[0115] That is, as described herein, electronic devices may be controlled differently based on their respective spatial positions in an operating environment, the relative spatial context of a room corresponding to their respective spatial positions, and / or operating conditions in adjacent areas of the operating environment (including operating states and / or environmental conditions of other electronic devices). Data indicating actual operation of the electronic device may also be recorded and stored in a database together with data indicating expected or predetermined operation of the electronic device for use in generating and displaying temporal status information, thereby presenting the operating state of the electronic device at a time point selected by a user. Thus, based on occupancy data collected by a mobile robot during navigation of an operating environment, a user interface may be generated that provides a visual representation of the device's location and past / present / future operating states of the device, and also allows control of the current or future operating state of the electronic device.

[0116] In the above description of various embodiments of the present disclosure, various aspects of the present disclosure may be shown and described in any of a number of patentable classes or contexts, including any new and useful process, machine, manufacture or combination of substances, or any new and useful improvement. Thus, various aspects of the present disclosure may be implemented entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or a combination of software and hardware implementations, which may all be generally referred to herein as "circuits," "modules," "components," or "systems." In addition, various aspects of the present disclosure may take the form of a computer program product comprising one or more computer-readable media having computer-readable program code implemented thereon.

[0117] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic or semiconductor system, device or apparatus, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer-readable storage media would include the following: a hard drive, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a suitable optical fiber with a repeater, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any non-transitory medium that can contain or store a program used by or in conjunction with an instruction execution system, device or device.

[0118] A computer-readable signal medium may include a propagated data signal (e.g., in baseband or as part of a carrier wave) in which computer-readable information is embodied. Such propagated signals may take any of a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium that is not a computer-readable storage medium and that can transmit, propagate, or transport a program used by or in conjunction with an instruction execution system, apparatus, or device. Computer-readable information embodied on a computer-readable medium may be sent using any suitable medium (e.g., as program code), including, but not limited to, wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0119] Computer program code for performing the operations of various aspects of the present disclosure may be written in any combination of one or more programming languages. The program code may be executed entirely on a user terminal, a mobile robot, or a remote server as described herein, or partially on one or more of each. In the latter case, the remote server may be connected to the user terminal and / or the mobile robot via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing (e.g., via the Internet using an Internet service provider) and / or in a cloud computing environment, or provided as a service (such as software as a service (SaaS)).

[0120] Various aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It will be understood that each box in the flowchart illustration and / or block diagram and the combination of boxes in the flowchart illustration and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that instructions executed by a processor of a computer or other programmable instruction execution device create a mechanism for implementing the function / action specified in one or more boxes of a flowchart and / or block diagram.

[0121] These computer program instructions may also be stored in a computer-readable medium, which, when executed, may direct a computer, other programmable data processing device, or other device to operate in a specific manner, so that when the instructions are stored in a computer-readable medium, an article of manufacture including the instructions is generated, wherein when the instructions are executed, the computer implements the functions / actions specified in one or more blocks of the flowchart and / or block diagram. The computer program instructions may also be loaded onto a computer, other programmable instruction execution device, or other device, so that a series of operating steps are performed on the computer, other programmable device, or other device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide a process for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0122] It should be understood that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meanings as those of ordinary skill in the art to which the present disclosure belongs. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having the same meanings as those in the context of this specification and related art, and will not be interpreted in an idealized or overly formal sense, unless explicitly defined in this article.

[0123] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various aspects of the present disclosure. In this regard, each square block in the flowchart or block diagram can represent a module, a fragment or a part of the code including one or more executable instructions for implementing the specified (one or more) logical functions. It should also be noted that in some alternative implementations, the functions mentioned in the square blocks may not occur in the order shown in the figure. For example, depending on the functions involved, the two square blocks shown in succession can actually be performed substantially simultaneously, or the square blocks can sometimes be performed in the opposite order. It will also be noted that each square block illustrated in the block diagram and / or the flowchart and the combination of the square blocks illustrated in the block diagram and / or the flowchart can be implemented by a system based on special-purpose hardware or a combination of special-purpose hardware and computer instructions that performs the specified function or action.

[0124] The foregoing is an explanation of the embodiments of the present disclosure and should not be construed as limiting thereof. Although some example embodiments have been described, it will be readily appreciated by those skilled in the art that many modifications may be made to the example embodiments without departing substantially from the teachings and advantages of the present invention. Thus, all such modifications are intended to be included within the scope of the present invention. Therefore, it should be understood that the foregoing is an explanation of the present invention and should not be construed as being limited to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included within the scope of the present invention.

Claims

1. A method for operating a user terminal, comprising: By instructions from at least one processor of the user terminal or in response to the instructions, performing operations including the following: In response to navigation of the operating environment by the mobile robot, receiving, via at least one receiver of the user terminal, occupancy data for the operating environment, the occupancy data comprising continuity information corresponding to areas of the operating environment that are not physically separated but are artificially separated; displaying, via a user interface of the user terminal, a visual representation of the operating environment based on the occupancy data; Receiving, via the at least one receiver of the user terminal, information identifying a plurality of electronic devices, including a lighting-related device, local to the operating environment and corresponding spatial locations and operating states of the plurality of electronic devices; automatically grouping, via a processor, two or more of the non-physically separated but artificially separated areas associated with at least one of the electronic devices into a continuous space in a subset of the operating environment based on the received continuity information and the corresponding spatial locations of the electronic devices; populating the visual representation of the operating environment with a visual indication of the corresponding spatial location of the electronic device in the operating environment and a status indication of a corresponding operating state of the electronic device; Based on the corresponding spatial positions of the lighting-related devices in the grouped continuous space, automatically generate conditional instructions to control the operation of the lighting-related devices in the grouped continuous space; as well as A control signal is provided to the mobile robot to clean the grouped continuous spaces according to the controlled operation of the lighting-related equipment.

Images