System and method for initializing an autonomous robot for location-specific actions

Abstract

A method is provided for autonomous robot guidance. The method includes ascertaining a position of a robot. The method further includes detecting an object. The method includes associating the object with the position. The method also includes, in accordance with a determination that the object is associated with a set of autonomous operation instructions, at least partially autonomously operating the robot in accordance with the instructions. The method also includes, in accordance with a determination that the object is not associated with the set of autonomous operation instructions, receiving position information along a route that the robot is moved along by a user. The method also includes generating a set of autonomous operation instructions based at least in part on the position information, associating the set of autonomous operation instructions with the object, and storing the set of autonomous operation instructions that are associated with the object.

Description

System and Method for Initializing an Autonomous Robot for Location-Specific ActionsTECHNICAL FIELD

[0001] The disclosed implementations relate generally to robotics, and more specifically to system and method for initializing an autonomous robot to conduct locationspecific actions.BACKGROUND

[0002] Autonomous systems or robots often require user training and expertise to operate, interact with, and troubleshoot. However, end users of floor cleaners, who are typically lower-skilled and non-technical staff, may not have the necessary knowledge to effectively control and command robots for achieving desired outcomes. In long-term usage with high utilization, it is crucial for the robot to be able to determine the intended action based on an accurate determination of the robot’s physical location. For example, a user may need to understand basic robot operations to bring the robot to a specific location in a building and trigger the appropriate cleaning task. However, floor plans and room layouts may look similar, while requiring different cleaning and care plans based on the floor materials.SUMMARY

[0003] Accordingly, there is a need for systems and methods that reduce the steps for a user to initialize a robot to a macro location. Some implementations allow a robot to autonomously poll a database for an instruction set for a specific location without further user intervention. Some implementations enable users to start a cleaning activity with minimal input, simply by bringing the robot to a position near a detectable object. Subsequently, the robot autonomously accesses and executes location-specific instructions from a cloud-hosted database. The techniques described herein help streamline user experience and ensure efficient and reliable robot behavior execution in complex environments, even for non-technical users with limited expertise in robot operation.

[0004] Systems, methods, devices, and non-transitory computer readable storage media for robotic guidance are disclosed.

[0005] (Al) In some implementations, a method of robotic guidance is provided. The method includes ascertaining, using one or more first sensors, a position of a robot. The method also includes detecting an object via a camera. The method also includes associating the object with the position. The method also includes, in accordance with a determination that the object is associated with a previously stored set of autonomous operation instructions: retrieving the autonomous operation instructions; and at least partially autonomously operating the robot in accordance with the set of autonomous operation instructions. The method also includes, in accordance with a determination that the object is not associated with a previously stored set of autonomous operation instructions: receiving, from one or more second sensors, position information along a route that the robot is moved along by a user; generating a set of autonomous operation instructions based at least in part on the position information; associating the set of autonomous operation instructions with the object; and storing the set of autonomous operation instructions that are associated with the object.

[0006] (A2) In some implementations of (Al), at least partially autonomously operating the robot is performed further in accordance with a determination that the position is associated with the previously stored set of autonomous operation instructions.

[0007] (A3) In some implementations of (Al), the determination that the object is not associated with a previously stored set of autonomous operation instructions also includes a determination that the position is not associated with a previously stored set of autonomous operation instructions.

[0008] (A4) In some implementations of (Al), the method further includes prior to receiving position information, instructing the user to move the robot around an area.

[0009] (A5) In some implementations of (Al), receiving position information along a route that the robot is moved along by a user further comprises receiving, from the one or more second sensors, location and / or orientation information along the route that the robot is moved along by a user. Generating the set of autonomous operation instructions is based on the location and / or orientation information.

[0010] (A6) In some implementations of (Al), the method further includes, after detecting the object via the camera and ascertaining the position of the robot, autonomously moving the robot to a new location, initializing the robot’s location to the new location, and either generating the set of autonomous operation instructions or autonomously operating the robot in relation to the new location. In some implementations, the method further includes,after the robot is in the new location, rotating and scanning surroundings, using the one or more second sensors, to orient the robot and ensure no obstacles before initializing.

[0011] (A7) In some implementations of (Al), associating the object with the position includes associating a position on a map with a current position and orientation of the robot.

[0012] (A8) In some implementations of (Al), the ascertaining and detecting are not performed in any particular order.

[0013] (A9) In some implementations of (Al), the position is proximity to the one or more first sensors.

[0014] (A 10) In some implementations of (Al), the one or more first sensors includes a nearfield communication device.

[0015] (Al l) In some implementations of (Al), the one or more first sensors includes a Bluetooth sensor.

[0016] (A12) In some implementations of (Al), the object is a QR-code, a barcode, or a fiducial marker.

[0017] (A13) In some implementations of (Al), the autonomous operation instructions are retrieved from a cloud-hosted database, using the object as an identifier.

[0018] (A14) In some implementations of (Al), generating the set of autonomous operation instructions includes generating cleaning instructions for the robot, and wherein subsequent at least partial autonomous operation of the robot includes executing the cleaning instructions.

[0019] (A 14) In some implementations of (Al), the camera is a front-facing camera of the robot.

[0020] (A15) In some implementations of (Al), the camera provides a live view of an environment on its screen, allowing the user to align the detected object with a view of the camera to ensure accurate positioning.

[0021] (A16) In some implementations of (Al), the method further includes, after the object is centered in the camera view, autonomously accessing the set of autonomous operation instructions associated with that object's identifier.

[0022] (A17) In some implementations of (Al), the set of autonomous operation instructions includes information related to an initial location, orientation, cleaning behaviors,environment requirements, scheduling requirements, and / or other relevant instructions for a cleaning robot or autonomous system.

[0023] (A18) In some implementations of (Al), the set of autonomous operation instructions includes information related to coordinates, markers, and / or other identifying information that defines boundaries of each location or area where the robot is expected to operate.

[0024] (A19) In some implementations of (Al), the set of autonomous operation instructions includes information related to specific cleaning behaviors including vacuuming, mopping, sweeping, scrubbing, burnishing, and / or their associated parameters.

[0025] (A20) In some implementations of (Al), the method further includes updating the set of autonomous operation instructions to accommodate changes in an environment or user preferences.

[0026] (A21) In some implementations of (Al), autonomously operating the robot comprises autonomously driving and cleaning.

[0027] (A22) In some implementations of (Al), the set of autonomous operation instructions includes information related to actions that the robot needs to perform based on floor type.

[0028] (A23) In some implementations of (Al), the set of autonomous operation instructions includes information related to scheduling requirements, including specific times or frequencies for cleaning operations, to ensure the robot operates according to desired cleaning schedule or preferences set by the user.

[0029] (A24) In some implementations of (Al), the set of autonomous operation instructions includes information related to safety protocols or guidelines, such as avoiding certain areas or hazards during cleaning operations.

[0030] (A25) In some implementations of (Al), the set of autonomous operation instructions includes information related to user preferences or settings, such as preferred cleaning modes, water usage levels, tool settings, or other customization options, to provide a personalized cleaning experience based on the user's preferences.

[0031] (A26) In some implementations of (Al), generating the set of autonomous operation instructions based on the position information comprises using sensor data to generate a point cloud through SLAM techniques of the robot’s surroundings.

[0032] (A27) In some implementations of (Al), the set of autonomous operation instructions includes information related to vacuuming, including different settings for water usage, floor tool pressure, and / or brush type, based on floor material and debris type.

[0033] (A28) In some implementations of (Al), the set of autonomous operation instructions includes information related to mopping with options for different settings such as water usage, floor tool pressure, and cleaning solution type based on floor material and desired level of cleaning.

[0034] (A29) In some implementations of (Al), the set of autonomous operation instructions includes information related to sweeping for different settings, such as brush type, brush speed, and brush pressure based on the floor material and debris type.

[0035] (A30) In some implementations of (Al), the set of autonomous operation instructions includes information related to scrubbing for different settings such as brush type, brush speed, cleaning solution type, and water usage based on the floor material and level of cleaning required.

[0036] (A31) In some implementations of (Al), the set of autonomous operation instructions includes information related to burnishing for different settings such as pad type, pad speed, and pad pressure based on the floor material and desired level of polishing.

[0037] (A32) In some implementations of (Al), the set of autonomous operation instructions includes information related to user preferences, such as cleaning frequency, time of day, and specific instructions for unique situations or areas.

[0038] (A33) In some implementations of (Al), the method further includes reporting and / or tracking, including generating reports or logs of cleaning activities, including information such as date, time, location, cleaning actions performed, and any issues or exceptions encountered.

[0039] (A34) In some implementations of (Al), the one or more first sensors and the one or more second sensors are different sensors.

[0040] (A35) In some implementations of (Al), the one or more first sensors and the one or more second sensors are same sensors.

[0041] (B 1) In another aspect, a method is provided for initializing a robot for locationspecific actions. The method includes, in response to detecting an object via a camera, in response to receiving a first input, localizing and mapping a region, based on signals received from one or more sensors, to obtain a plurality of poses. In response to receiving a second input, the method includes ceasing to localize and map the region, generating and displaying (i) a representation of the region based on the plurality of poses, and (ii) a plurality of actions for cleaning or maintaining the region. In response to detecting (i) a third input to approve the region and (ii) a fourth input to select one or more actions from the plurality of actions, the method includes encoding the plurality of poses and the one or more actions as an instruction set; and associating the instruction set with the object.

[0042] (B2) In some implementations of (Bl), the method further includes, prior to detecting the object via the camera: detecting a presence of the object using a short-range wireless technology; and in response to detecting the presence of the object, displaying a prompt to position the camera towards the object.

[0043] In another aspect, a computer system includes one or more processors, memory, and one or more programs stored in the memory. The programs are configured for execution by the one or more processors. The programs include instructions for performing any of the methods described herein.

[0044] In another aspect, a non-transitory computer readable storage medium stores one or more programs configured for execution by one or more processors of a computer system. The programs include instructions for performing any of the methods described herein.BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a block diagram of a system for autonomous robotic guidance in accordance with some implementations.

[0046] Figure 2A shows a block diagram of an example autonomous robot in accordance with some implementations.

[0047] Figure 2B is a block diagram illustrating an example server in accordance with some implementations.

[0048] Figure 3 shows a diagram of an example process for setting up (or initializing) an autonomous robot, according to some implementations.

[0049] Figure 4 shows a diagram of an example process 400 for using an autonomous robot 102, according to some implementations.

[0050] Figures 5A-5Z10 show a flowchart for an example method for autonomous robot guidance, in accordance with some implementations.

[0051] Figure 6 show a flowchart for an example method for initializing a robot for location-specific actions, in accordance with some implementations.

[0052] Like reference numerals refer to corresponding parts throughout the drawings.DESCRIPTION OF IMPLEMENTATIONS

[0053] Reference will now be made to various implementations, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention and the described implementations. However, the invention may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the implementations .

[0054] Disclosed implementations enable autonomous robotic guidance. Systems and devices implementing the techniques in accordance with some implementations are illustrated in Figures 1-6.

[0055] Figure 1 is a block diagram of a system 100 for autonomous robotic guidance in accordance with some implementations. In some implementations, the system 100 includes autonomous robots 102-1, 102-2, ... , controllable by users 104-1, 104-2, and operating in areas 108-1, 108-2, . . communication network(s) 110, a server 112, and a database 114. Each autonomous may operate independently and / or concurrently in a corresponding area, based on a corresponding set of autonomous operation instructions. The autonomous operation instructions may each correspond to a respective area or a zone of a building (e.g., an airport, a hotel) or several buildings. For example, the operation instructions 108-1 corresponds to an area 106-1 and the operation instructions 108-2 corresponds to an area 106-2. Each autonomous robot may access a different set of operation instructions and / or operate in a different area. Theautonomous robots may have access to operation instructions of other autonomous robots. The autonomous robots may have access to different operation instructions for a given area. A user 104 may control an autonomous robot, e.g., for training the robot and / or to initialize the robot with operation instructions.

[0056] The user 104 includes a person, a machine, and / or other means of interacting with the autonomous robot 102. In some implementations, the user 104 is not part of the system 100 (e.g., when the autonomous robots are pre-trained robots), but interacts with the system 100 via the autonomous robot 102 or another means. For instance, the user 104 provides input (e.g., touch screen input or alphanumeric input) to the autonomous robot 102 and the input is communicated to the server 112 via the one or more communication networks 110. In this instance, the system 100, in response to receiving the input from the user 104, communicates information to the autonomous robot 102 via the one or more communication networks 110 to be presented to the user 104. In this way, the user 104 can interact with the system 100.

[0057] The autonomous robot 102 communicates with the server 112 through the one or more communication networks 110. The autonomous robot 102 provides autonomous driving and / or cleaning capabilities based on a set of operation instructions stored locally and / or in a database 114 and / or retrieved via the server 112. The server 112 provides serverside functionality (e.g., processing data corresponding to operation instructions for the autonomous robots, updating the database 114, storing operation instructions, schedules and handling requests from users 104, e.g., for visualizing operation instructions, schedules, etc.) for any number of autonomous robots 102 and / or users 104.

[0058] The system 100 shown in Figure 1 may be viewed as having both a client-side portion (e.g., modules in the autonomous robots 102) and a server-side portion (e.g., modules in the server 112). In some implementations, data preprocessing is implemented as a standalone application installed on the autonomous robot 108. In addition, the division of functionality between the client and server portions can vary in different implementations. In some implementations, the autonomous robot 102 is a thin client that provides only driving and / or cleaning operations, and delegates all other data processing functionality to the server 112.

[0059] The communication network(s) 110 can be any wired or wireless local area network (LAN) and / or wide area network (WAN), such as an intranet, an extranet, or the Internet. It is sufficient that the one or more communication networks 110 provide communication capability between the server 112 and the autonomous robots 102. Examplesof the communication network(s) 110 include local area networks (LAN) and wide area networks (WAN) such as the Internet. The communication network(s) 110 are, optionally, implemented using any known network protocol, including various wired or wireless protocols, such as Ethernet, Universal Serial Bus (USB), FIREWIRE, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), code division multiple access (CDMA), time division multiple access (TDMA), Bluetooth, Wi-Fi, voice over Internet Protocol (VoIP), Wi-MAX, or any other suitable communication protocol. In some implementations, the communication network(s) 110 include an ad hoc network, an intranet, an extranet, a Virtual Private Network (VPN), a Local Area Network (LAN), a wireless LAN (WLAN), a WAN, a wireless WAN (WWAN), a Metropolitan Area Network (MAN), a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a cellular telephone network, a wireless network, a Wireless Fidelity (Wi-Fi®) network, a Worldwide Interoperability for Microwave Access (WiMax) network, and / or any suitable combination thereof.

[0060] Figure 2 A shows a block diagram of an example autonomous robot 102 in accordance with some implementations. In some implementations, the autonomous robot 102 includes one or more processors 202, user interface components 206, one or more sensors and / or one or more actuators 232, one or more cameras 234, one or more network interfaces 204 (sometimes referred to as communication interfaces) that facilitate the processing of input and output associated with the autonomous robot 102 and / or the server 112, and / or one or more cleaning mechanism(s) 266. The one or more sensors and / or one or more actuators 232 is sometimes referred to as the sensors or the one or more sensors when the sensors are described. The one or more sensors and / or one or more actuators 232 is sometimes referred to as the actuators or the one or more actuators when the actuators are described.

[0061] The one or more processors 202 (e.g., a Central Processing Unit (CPU), a Reduced Instruction Set Computing (RISC) processor, a Complex Instruction Set Computing (CISC) processor, a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an ASIC, a Radio-Frequency Integrated Circuit (RFIC), another processor, or any suitable combination thereof) execute instructions. The term “processor” is intended to include multicore processors that may comprise two or more independent processors (sometimes referred to as “cores”) that may execute instructions contemporaneously.

[0062] The one or more cameras 234 may include two-dimensional (2D) imaging cameras, three-dimensional (3D) sensing cameras, ultrasonic cameras, and / or infrared cameras. Additionally, the one or more cameras 234 may include one or more front-facing cameras and / or one or more rear-facing cameras.

[0063] The user interface 206 includes a display 208, input device mechanism 210, and / or any other input / output components, according to some implementations. This includes components to receive input, provide output, produce output, transmit information, exchange information, capture measurements, and so on. The specific components that are included may depend on the type of the autonomous robot. The display 208 may include visual components (e.g., a display such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)). The user interface 206 may include acoustic components (e.g., speakers), haptic components (e.g., a vibratory motor, resistance mechanisms), and / or other signal generators. The input device mechanism 210 may include alphanumeric input components (e.g., a keyboard, a touch screen, or other similar input components), point-based input devices (e.g., a mouse, a touchpad, a trackball, a joystick, or other similar devices), tactile input components (e.g., a physical button, a touch screen), and / or audio input components (e.g., a microphone).

[0064] The one or more sensors and / or actuators 232 may include acceleration sensor components (e.g., accelerometer), gravitation sensor components, and / or rotation sensor components (e.g., gyroscope). The one or more sensors and / or actuators 232 may include illumination sensor components (e.g., photometer), temperature sensor components (e.g., thermometers), humidity sensor components, pressure sensor components (e.g., barometer), acoustic sensor components (e.g., microphone(s) for detecting background noise), proximity sensor components (e.g., infrared sensors that detect nearby objects), and / or gas sensors (e.g., sensors to detect concentrations of gases or pollutants). The one or more sensors and / or actuators 232 may provide indications, measurements, or signals corresponding to a physical environment. The one or more sensors and / or actuators 232 may include location sensor components (e.g., a GPS receiver component), altitude sensor components (e.g., altimeters or barometers that detect air pressure from which altitude may be derived), and / or orientation sensor components (e.g., magnetometers). The one or more sensors and / or actuators 232 may include actuators to produce rotary and / or linear motion, motors, and / or servos, for driving the autonomous robot 102 and / or cleaning using the autonomous robot 102.

[0065] The communication interfaces 204 may include communication components operable to couple the autonomous robot 102 to the one or more communication networks 110. The communication interfaces 204 may include wired communication components, wireless communication components, cellular communication components, Near Field Communication (NFC) components, Bluetooth® components (e.g., Bluetooth® Low Energy), Wi-Fi® components, and other communication components to provide communication via other modalities. The communication interfaces 204 may be a separate device (e.g., a peripheral device coupled via a Universal Serial Bus). The communication interfaces 204 may detect identifiers or include components operable to detect identifiers. For example, the communication interfaces 204 may include Radio Frequency Identification (RFID) tag reader components, NFC smart tag detection components, optical reader components (e.g., an optical sensor to detect one-dimensional bar codes such as Universal Product Code (UPC) bar code, multi-dimensional bar codes such as Quick Response (QR) code, Aztec code, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2D bar code, and other optical codes), or acoustic detection components (e.g., microphones to identify tagged audio signals). Additionally, information may be derived via the communication interfaces 204, such as location via Internet Protocol (IP) geolocation, location via Wi-Fi® signal triangulation, location via detecting an NFC beacon signal that may indicate a particular location.

[0066] The one or more cleaning mechanisms 266 may include brushes, vacuum generators, valves, cleaning solutions, storage areas or tanks, and similar tools and / or hardware.

[0067] The memory 214 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid state memory devices; and, optionally, includes non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other nonvolatile solid state storage devices. The memory 214, optionally, includes one or more storage devices remotely located from the one or more processors 202. The memory 214, or alternatively the non-volatile memory within the memory 214, includes a non-transitory computer readable storage medium. In some implementations, the memory 214, or the non- transitory computer readable storage medium of the memory 214, stores the following programs, modules, and data structures, or a subset or superset thereof:• an operating system 216 including procedures for handling various basic system services and for performing hardware dependent tasks;a communications module 218 for connecting the autonomous robot 102 to other autonomous robots and / or one or more servers via one or more communication interfaces 204 (wired or wireless); an object detection module 220 for detecting objects, such as furniture, humans, other robots and / or movable or moving objects. The object detection module 220 may different viewpoints, different sized images, and / or illumination conditions. Some implementations use a neural network (e.g., convolutional neural network (CNN)) that has been trained to detect and classify objects. Some implementations determine orientation of objects, for interaction and / or navigation, e.g., using homography algorithms such as linear least square solver, random sampling and consensus, and least median of squares, to compute points between frames of 2D imagery. Some implementations assign metadata, such as an identifier, bounding box for object detection and navigation; a signal processing module 222 that processes signals from the one or more sensors and / or actuators 232 to track objects and / or generate maps for navigation. Some implementations use centroid tracking for objects that only move a certain distance each frame (e.g., each frame of image captured by the one or more cameras 234). Some implementations use a Kalman filter to predict the location of an object. Some implementations use the mean shift algorithm to account for changes in scale and / or orientation for object tracking, such as scale, orientation, etc., and to ultimately track where the object is. For navigation (e.g., for generating maps), some implementations use Simultaneous Localization and Mapping (SLAM) to estimate a position and / or location of the autonomous robot 102 to derive a map of the environment. SLAM may be implemented by fusing data from the one or more sensors; a visualization module 224 to generate and / or display (e.g., displaying using the display 208) visualizations of maps, routes, statistics, and / or history; and / or an autonomous operation module 226 for operating the autonomous robot 102 based on autonomous operation instructions 228 generated based on output of the object detection module 220, the signal processing module 222, and / or external input from the user interface 206 and / or from the server 112. The autonomous operation module 226 may also download (or retrieve) autonomous operation instructions from the database 114 by communicating with the server 112 via the one or more communicationnetworks 110. The autonomous operation module 226 may include a signal generation module 230 to control the one or more actuators to operate the autonomous robot 102. This may include control signals for driving and / or cleaning, examples of which are described below. The autonomous operation module 226 may include a cleaning module 248 for generating cleaning -related signals and / or instructions (e.g., for operating and / or controlling the one or more cleaning mechanisms 266).

[0068] Each of the above identified elements may be stored in one or more of the previously mentioned memory devices, and corresponds to a set of instructions for performing a function described above. The above identified modules or programs (i.e., sets of instructions) need not be implemented as separate software programs, procedures, or modules, and thus various subsets of these modules may be combined or otherwise re-arranged in various implementations. In some implementations, the memory 214, optionally, stores a subset of the modules and data structures identified above. Furthermore, the memory 214, optionally, stores additional modules and data structures not described above.

[0069] Figure 2B is a block diagram illustrating an example server 112 in accordance with some implementations. The server 112, typically, includes one or more processors 250 (e.g., CPUs and / or GPUs), one or more network interfaces 254, a power supply 252 for powering the components, memory 236, and / or one or more communication buses 266 for interconnecting these components (sometimes called a chipset). Some implementations include a display 256 to view, and / or update history, statistics, operational characteristics, such as control parameters, data and / or information related to operation of the autonomous robots 102.

[0070] The memory 236 may include high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid state memory devices; and, optionally, includes non-volatile memory, such as one or more magnetic disk storage devices, one or more optical disk storage devices, one or more flash memory devices, or one or more other non-volatile solid state storage devices. The memory 236, optionally, includes one or more storage devices remotely located from the one or more processors 250. The memory 236, or alternatively the non-volatile memory within the memory 236, includes a non-transitory computer readable storage medium. In some implementations, the memory 236, or the non- transitory computer readable storage medium of the memory 236, stores the following programs, modules, and data structures, or a subset or superset thereof:an operating system 238 including procedures for handling various basic system services and for performing hardware dependent tasks; a communication module 258 for connecting the server 112 to the autonomous robots 102 via the one or more communication networks 110 using the one or more network interfaces 254 (wired or wireless); an optional object detection module 260 for detecting objects, such as furniture, humans, docking stations / charging stations / refueling stations, other robots and / or movable or moving objects. The object detection module 260 may detect objects using detection techniques such as but not limited to shortrange wireless communication or remote sensing techniques or like combination in addition to or instead of the object detection module 220. The object detection module 260 may have different viewpoints, different sized images, and / or illumination conditions. Some implementations use a neural network (e.g., convolutional neural network (CNN)) that has been trained to detect and classify objects. Some implementations determine orientation of objects, for interaction and / or navigation, e.g., using homography algorithms such as linear least square solver, random sampling and consensus, and least median of squares, to compute points between frames of 2D imagery. Some implementations assign metadata, such as an identifier, bounding box for object detection and navigation; an optional signal processing module 262 that processes signals from the one or more sensors and / or actuators 232 to track objects and / or generate maps for navigation. The signal processing module 262 may process signals in addition to or instead of the signal processing module 222. Some implementations use centroid tracking for objects that only move a certain distance each frame (e.g., each frame of image captured by the one or more cameras 234). Some implementations use a Kalman filter to predict the location of an object. Some implementations use the mean shift algorithm to account for changes in scale and / or orientation for object tracking, such as scale, orientation, etc., and to ultimately track where the object is. For navigation (e.g., for generating maps), some implementations use Simultaneous Localization and Mapping (SLAM) to estimate a position and / or location of the autonomous robot 102 to derive a map of the environment. SLAM may be implemented by fusing data from the one or more sensors; an optional visualization module 264 to generate and / or display (e.g., displaying using the display 208) visualizations of maps, routes, statistics, and / or history; and / or• an optional autonomous instructions management module 240 for obtaining autonomous operation instructions 242 that may be stored in the database 114. The autonomous instructions management module 240 may include an optional aggregation module 244 to obtain, collect, and / or aggregate signals and / or autonomous operation instructions from the autonomous robots 102. For example, a user may update an instruction using a graphical user interface on a display of an autonomous robot. Such update may be gathered by the aggregation module 244, and / or supplied to other autonomous robots in the system 100. Some implementations include an optimization module 246 to optimize operation, driving and / or cleaning using the autonomous robots 102. For example, an autonomous robot determines a condition of a floor and an optimal navigation and / or cleaning option for a floor, and communicates the same to the server 112. The optimization module 246 may then subsequently optimize future operations, driving, and / or cleaning, for robots, using the findings. These updates may be real-time and / or staged or performed in batches. These modules may help determine schedules and / or optimal schedules for operating the autonomous robots 102. Some implementations include database management modules to interface with the database 114 to manage storing and / or retrieving autonomous operation instructions from the database 114. In some implementations, the database management module manages multiple repositories, providing methods to access and modify data that can be stored in local folders, and / or cloud-based storage systems. In some implementations, the database management module can search offline repositories. In some implementations, offline requests are handled asynchronously, with large delays or hours or even days if the remote machine is not enabled. In some implementations, a catalog module manages permissions and secure access for a wide range of databases and / or operation instructions.

[0071] Each of the above identified elements may be stored in one or more of the previously mentioned memory devices, and corresponds to a set of instructions for performing a function described above. The above identified modules or programs (i.e., sets of instructions) need not be implemented as separate software programs, procedures, or modules, and thus various subsets of these modules may be combined or otherwise re-arranged in various implementations. In some implementations, the memory 236, optionally, stores a subset of the modules and data structures identified above. Furthermore, the memory 236, optionally, stores additional modules and data structures not described above.Example Implementations

[0072] Some implementations use a robot with a front-facing camera and wireless communication capabilities to detect visible objects, such as QR-codes, barcodes, or fiducial markers, which serve as identifiers for accessing location-specific instructions from a cloud- hosted database. The robot utilizes the detected object to determine its location and orientation in the environment, and downloads the appropriate instruction set from the cloud-hosted database. Some implementations use machine readable code formats on the object. Some implementations do not encode any identifier information in the object (e.g., no visual information). No additional infrastructure may need to be fitted onto walls or buildings. Some implementations use objects that are inconspicuous (e.g., black in color). In some implementations, because no visible information is used, in some implementations, lack of cleanliness or visibility of the object due to lighting or damage or abuse to the physical object are of less concern.

[0073] In some implementations, the robot's front-facing camera provides a live view of the environment on its screen, allowing the user to align the detected object with the camera view to ensure accurate positioning. After the object is in the camera view, the robot autonomously accesses the instruction set associated with that object's identifier from the cloud-hosted database. The instruction set contains information about the specific cleaning requirements or other behaviors that need to be executed at that location, such as using the right cleaning techniques for different floor materials. The robot then autonomously performs the appropriate behaviors based on the downloaded instruction set, without further user intervention. This reduces the need for users to have in-depth knowledge of robot operations or to manually input instructions, making it easy for low-skilled and non-technical staff to initiate cleaning activities with the robot.

[0074] By using wireless object detection, computer vision, and cloud-hosted databases, a robot can access and execute location-specific instructions, reducing the steps and user intervention required to initiate robot behaviors, and providing a seamless and efficient user experience in complex environments.Example Instruction Set Database

[0075] In some implementations, the instruction set database is a repository of data that contains information related to the location, orientation, cleaning behaviors, environment requirements, scheduling requirements, and / or other relevant instructions for a cleaning robot or autonomous system. The database may store coordinates, markers, or other identifying information that defines the boundaries of each location or area where the robot is expected to operate. The database may also contain instructions on specific cleaning behaviors such as vacuuming, mopping, sweeping, scrubbing, burnishing, and their associated parameters such as water usage, pressure, or tool settings. The database can be updated or modified to accommodate changes in the environment or user preferences. The robot can access this information from the instruction set database based on the detected object or fiducial marker in the environment, and use it to autonomously determine the appropriate cleaning actions, settings, and navigation behaviors for each location, with the goal of minimizing user intervention and ensuring efficient and effective cleaning operations.Example Instructions and Behaviors

[0076] In some implementations, the instruction set database (sometimes referred to as autonomous operation instructions) includes one or more of the following types of instructions or behaviors:• Cleaning behaviors: This includes instructions on various cleaning actions such as vacuuming, mopping, sweeping, scrubbing, burnishing, or other relevant cleaning actions that the robot needs to perform based on the location and floor type.• Environment requirements: The database may contain information on specific environmental requirements for each location, such as the type of flooring material (hardwood, carpet, tile, etc.), or other environmental factors that may impact the cleaning process.• Scheduling requirements: Instructions related to scheduling requirements, such as specific times or frequencies for cleaning operations, can also be included in the database to ensure the robot operates according to the desired cleaning schedule or preferences set by the user.Safety considerations: While safety considerations are typically implemented through the onboard autonomy framework of the robot, the instruction set database may includeinstructions related to safety protocols or guidelines, such as avoiding certain areas or hazards during cleaning operations.• User preferences: The instruction set database may also store user preferences or settings, such as preferred cleaning modes, water usage levels, tool settings, or other customization options, to provide a personalized cleaning experience based on the user's preferences.• Technical setup behaviors: Start the Mapping: robot can be triggered to enter a mapping / configuration mode, where sensor data is used to generate a point cloud through SLAM techniques of its surroundings, and save the objects position for recall or robot can also be triggered to determine a location and a pose, wherein the location may be considered as a start location for setup on a first run, and for recall on subsequent runs in response to detecting an object based on the sensor data.• Updates or modifications: The database may also allow for updates or modifications to accommodate changes in the environment, user requirements, or other factors that may affect the cleaning process, ensuring that the robot can adapt to changing conditions and perform optimally.

[0077] Some implementations include the following types of instructions in the instruction set database to guide the cleaning robot in autonomously determining the appropriate actions, settings, and behaviors for each location, minimizing the need for user intervention and enabling efficient and effective cleaning operations.• Vacuuming: With options for different settings such as water usage, floor tool pressure, and brush type based on the floor material and debris type. For example, lower water usage and lower floor tool pressure for hardwood floors, and no water usage with light pressure for high pile carpets.• Mopping: With options for different settings such as water usage, floor tool pressure, and cleaning solution type based on the floor material and desired level of cleaning. For example, higher water usage and lower floor tool pressure for tile floors, and no water usage for hardwood floors.• Sweeping: With options for different settings such as brush type, brush speed, and brush pressure based on the floor material and debris type. For example, soft brush for high pile carpets, and stiff brush for tile floors.• Scrubbing: With options for different settings such as brush type, brush speed, cleaning solution type, and water usage based on the floor material and level of cleaning required. For example, soft brush, low brush speed, and higher water usage for deep cleaning of tile floors.• Burnishing: With options for different settings such as pad type, pad speed, and pad pressure based on the floor material and desired level of polishing. For example, soft pad with high speed and low pressure for burnishing hardwood floors.• Customized settings: Providing options for users to specify their own preferences, such as cleaning frequency, time of day, and specific instructions for unique situations or areas.• Reporting and tracking: Generating reports or logs of cleaning activities, including information such as date, time, location, pose, cleaning actions performed, and any issues or exceptions encountered.

[0078] The techniques described herein provide several advantages over conventional methods including the following:• Reduced user training requirements: By incorporating a comprehensive instruction set database, a robot developer can minimize the need for extensive user training. The robot can autonomously access and follow the instructions from the database, reducing the complexity and expertise required from end users to operate and interact with the robot effectively.• Enhanced user experience: The instruction set database can enable a seamless and user- friendly experience for end users, as they can initiate cleaning operations with minimal clicks or user intervention. This can result in a more pleasant and convenient experience for users, leading to increased user satisfaction and loyalty.• Improved cleaning efficiency and effectiveness: The instructions in the database can guide the robot to perform the appropriate cleaning behaviors, settings, and actions based on the specific location requirements. This can result in improved cleaning efficiency and effectiveness, as the robot can adapt to different floor types, environmental conditions, and scheduling requirements, leading to better cleaning outcomes.• Flexibility and scalability: The instruction set database can be updated or modified to accommodate changes in the environment, user preferences, or other factors, providing flexibility and scalability to the robot system. This allows for easy customization and adaptation to different customer requirements, making the robot system more versatile and adaptable in different settings and scenarios.• Competitive advantage: Implementing an instruction set database in the robot system can provide a competitive advantage for the robot developer, as it can differentiate their product from other similar robots in the market. A robot system that requires minimal user training, provides a superior user experience, and offers efficient and effective cleaning operations can attract more customers and lead to increased market share.

[0079] Additionally, the techniques described herein provide several efficacy improvements, examples of which are described next.

[0080] Accurate cleaning behavior selection: The instruction set database can guide the robot in selecting the appropriate cleaning behavior, such as vacuuming, mopping, sweeping, scrubbing, or burnishing, based on the location and floor type. This can ensure that the robot uses the most effective cleaning behavior for the specific environment, leading to improved efficacy in cleaning performance.

[0081] Optimal settings and actions: The instruction set database can provide the robot with optimal settings and actions to perform during the cleaning process, such as water usage, floor tool pressure, scheduling requirements, and other environmental considerations. This can ensure that the robot operates at the optimal parameters for efficient and effective cleaning, resulting in improved efficacy.

[0082] Adaptability to different locations: The instruction set database can contain information on different floor types, environmental conditions, and user preferences, allowing the robot to adapt its cleaning behavior and settings accordingly. This can ensure that the robot performs effectively in diverse locations, leading to improved efficacy in different cleaning scenarios.

[0083] Real-time updates and customization: The instruction set database can be updated or customized in real-time to accommodate changes in the environment, user requirements, or other factors. This allows the robot to access the most up-to-date informationand instructions, ensuring that it performs at its best efficacy level based on the latest data available.

[0084] Some implementations do not use odometry unit to track the position of the robot relative to the object. In some implementations, robot only detects the availability of an object; the robot position estimation is observed only from SLAM feedback.

[0085] In some implementations, for object detection, the robot's front-facing camera captures images or video of the environment, and the object detection algorithm analyzes the visual input to identify the visible objects, such as QR-codes, barcodes, or fiducial markers, that serve as location identifiers.

[0086] In some implementations, for object recognition, the detected object is recognized by the robot's software using image processing techniques, computer vision algorithms, or machine learning algorithms to identify the type of object, such as a QR-code, barcode, or fiducial marker.

[0087] In some implementations, for database lookup, the robot uses the recognized object as a key to access the cloud-hosted database that contains the instruction set for locationspecific behaviors. The robot retrieves the relevant instruction set associated with the detected object from the database.

[0088] In some implementations, for location estimation, the instruction set may include information, such as the location and orientation of the robot in the environment. The robot uses this information, along with its sensor data, to estimate its current position and orientation relative to the detected object.

[0089] Some implementations do not user absolute or relative position association between the object and robot, when training a robot to run a route by demonstration. The object is detectable within a radius, within which the robot may enter a camera assisted manual positioning behavior.

[0090] In some implementations, the object and the instruction set database serve to provide ease of deployment. Each object may correspond to one cleaning instruction set. In the event that one object has more than one desired option, some implementations automatically trigger a first (preferred) plan after a countdown, without user selection.

[0091] Some implementations do not redetect the object from a second position after the first detection. Some implementations do not require detecting the object to terminate or end an instruction set.

[0092] Some implementations use a centralized instruction set database. The instruction set database can be hosted on a centralized server or cloud-based platform, accessible by multiple robots. Each robot can communicate with the database to retrieve relevant instructions based on its location, floor type, environmental conditions, and other parameters. This centralization allows for consistent and synchronized instructions across multiple robots, ensuring uniformity and efficacy in their cleaning behaviors.

[0093] Some implementations use a distributed instruction set database. The instruction set database can be distributed across multiple robots, where each robot stores a local copy of the database onboard. The robots can exchange information with each other to update and synchronize their databases. This approach can be useful in scenarios where connectivity to a central server is limited or not feasible, such as in remote or offline environments.

[0094] Some implementations use customized instruction sets for individual robots. The instruction set database can be customized for each individual robot based on its specific capabilities, preferences, and usage patterns. This customization can be done during the initial setup or through ongoing updates, tailoring the instructions to each robot's unique requirements. This allows for personalized and optimized cleaning behaviors for each robot, leading to improved efficacy.

[0095] Some implementations provide scalability for multiple robot types. The instruction set database can be designed to be scalable and adaptable to different types of robots, such as vacuuming robots, mopping robots, sweeping robots, scrubbing robots, or burnishing robots. The instructions can be categorized and organized based on the type of cleaning behavior or floor type, allowing for easy integration with various robot models or brands. This enables the invention to be used across a fleet of diverse robots, enhancing their efficacy in different cleaning tasks.

[0096] Figure 3 shows a diagram of an example process 300 for setting up (or initializing) an autonomous robot 102, according to some implementations. As shown by label 302, a user sets up an object at a suitable point (or location) in a room or a building. As shownby labels 304 and 306, a user brings the autonomous robot 102 towards an object. The object, for example, may be a stationary object associated with a charging station of the autonomous robot 102 or other fixed / movable / moving objects. The autonomous robot 102 detects a presence of the object using detection techniques, such as, but not limited to shortrange wireless or remote sensing or like combinations. According to some implementation, a user interface of the autonomous robot 102 prompts the user to face camera toward the object for computer vision recognition. When the object is visually recognized, the autonomous robot 102 determines a location and / or a pose as a start location for setup on a first run, and for recall on subsequent runs. According to some implementation, when object presence is detected using the detection techniques, the autonomous robot 102 determines a location and / or a pose as a start location for setup on a first run, and for recall on subsequent runs. The object is only used to trigger the robot mapping. When the object is visually recognized, mapping may start, according to some implementations. When the object is not visually recognized, mapping may start using only the detection techniques, according to some implementations. As shown by label 308, the user can choose different teaching methods to configure the map for the robot. For example, the user can use the interface to create a boundary; the autonomous robot 102 will then clean within the boundary. The user can also provide a path that the robot will then drive and / or clean along. As shown by label 310, suppose the user brings the autonomous robot 102 to a new position to initiate a teaching session. After the user initiates the teaching session, the autonomous robot 102 starts encoding the autonomous operation instructions until the user terminates the teaching process. Location and orientation information has no relation to the object, according to some implantations. The autonomous robot 102 localizes and maps using sensors (e.g., LIDAR, odometry, using one or more mapping techniques). As shown by label 312, the user can review the area created and / or configure cleaning settings for the area. The user can also add a new area which leads them back to step indicated by labels 304 and 306. As shown by label 314, after the user is satisfied, the user saves the configuration (sometimes referred to as autonomous operation instructions) which is associated with the object.

[0097] Figure 4 shows a diagram of an example process 400 for using an autonomous robot 102, according to some implementations. Following the training process (sometimes referred to as an initialization process or a set up process) described above in reference to Figure 3, the autonomous robot 102 may be used for driving and / or cleaning an area. The autonomous robot 102 may also independently and / or under the supervision of a user, download or retrieve a previously stored autonomous operation instructions from the database 114, via the server11, and operate autonomously. For the sake of illustration, suppose the autonomous robot 1202 was trained using the example process described above in reference to Figure 3. As shown by label 402, suppose a user brings the autonomous robot 102 to a predetermined location, at which an object (e.g., an object used during a training process) had been placed. Similar to the training process, as shown by labels 404 and 406, further suppose that the user brings the autonomous robot 102 towards the object. The autonomous robot 102 detects (e.g., using shortrange wireless, such as Bluetooth or remote sensing, such as LiDAR or their combination) a presence of the object. According to some implementations, a user interface of the autonomous robot 102 prompts the user to face the camera towards the object for computer vision recognition (object detection). When the object is visually recognized, the autonomous robot 102 determines a location and / or a pose as the start location for recalling autonomous operation instructions. According to some implementations, the presence of the object is detected using the remote sensing followed by establishing the shortrange wireless communication to determine a location and / or a pose of the autonomous robot 102 as the start location for recalling autonomous operation instructions. As shown by label 408, upon successful detection of this object, the autonomous robot 102 polls a database (e.g., the database 114, prefetched autonomous operation instructions 228) for the associated instruction set. These instructions are unique to the object. As shown by label 410, the user is provided the flexibility to interrupt and / or modify the preferred settings before the autonomous robot 102 starts operating (e.g., driving and / or cleaning). As shown by label 412, after this point, the autonomous robot 102 can start operating using the preferred settings or autonomous operation instructions, without further user input.

[0098] In some implementations, the autonomous robot 102 includes a Bluetooth radio to detect when the robot has come into the vicinity of an object (e.g., an object emitting Bluetooth signals). In some implementations, the autonomous robot 102 includes a LiDAR to detect when the robot has come into the vicinity of an object (e.g., an object emitting Bluetooth signals). In some implementations, upon detection of the object, a camera of the autonomous robot 102 observes its frontal view to guide the user to position the object in the camera’s view. In some implementations, the autonomous robot 102 includes odometry units, laser scanner, touch screen interface for user interaction, and / or a handle bar for manual positioning by the user before autonomous function is triggered. In some implementations, the autonomous robot 102 includes a set of sensors for the purpose of detecting visible objects (e.g., a camera). Some implementations include a Bluetooth, radio frequency beacon, or similar device to detectnearfield signals emitted by the object. In some implementations, the object is a visible object (e.g., a 3D object) in the environment, or a 2D image positioned in a convenient location in the environment. For non-physical objects that are detected via nearfield wireless communications, the objects can be implemented as Bluetooth or RFID beacons. When the object comes into a robot nearfield, the robot may trigger a transfer of information that informs the robot regarding the object detected. The autonomous robot 102 then uses the object (or an identifier of the object) as a pointer to the database for its related instruction set. In some implementations, the object broadcasts a unique UUID via wireless communications, so non-physical objects may also be used. Some implementations include a database on which objects are registered, and / or correlated with a corresponding instruction set. An example instruction set may include start the mapping process (e.g., a 2D or 3D Map), save a starting point (cartesian coordinates and heading), a planned path for traversal, and an associated endpoint triggered by detection of an object. Based on user input, various cleaning paths may be generated and / or stored as one cleaning plan.

[0099] An example use of the autonomous robot 102 is described herein for illustration, according to some implementations. An autonomous cleaning robot is brought to a specific room in a building. At this point, the robot may be unaware of where it is within this building. A human operator knows that in each desired location there is an affixed object. The operator approximately faces the robot to the object. The robot detects the object, via the camera if the object is a visible object, or via the nearfield / farfield sensors if the object is a wireless communication object. The object broadcasts a unique ID. The autonomous robot polls the database for instructions. Based on the unique ID from the object, the robot may pull an instruction set from the database. The instruction set may be executed immediately if there are no options for paths and / or cleaning; otherwise, the autonomous robot may poll the human operator for its desired behavior. In this way, the autonomous robot automatically loads the map into the memory, initializes the robot location and pose, loads preset operation plan, and / or navigates autonomously for its operation.

[0100] Figures 5A-5Z10 show a flowchart for an example method 500 for autonomous robot guidance, in accordance with some implementations. The method may be performed by the autonomous robot 102 and / or the server 112.

[0101] Referring to Figure 5A, the method includes ascertaining (502), using one or more first sensors, a position of a robot. For example, the signal processing module 222 mayprocess one or more signals from the one or more sensors 232 to detect the position of the robot in relation to an object. The one or more first sensors, for example, may be LiDAR or Bluetooth sensors. The method also includes detecting (504) the object via a camera (e.g., the one or more cameras 234). The method also includes associating (506) the object with the position. For example, the object detection module 220 may detect the object and / or associate the object with the position. The association between the object and the position may be stored locally in the autonomous robot 102, the server 112, and / or the database 114. The method also includes, in accordance with a determination (508) that the object is associated with a previously stored set of autonomous operation instructions: retrieving (510) the autonomous operation instructions (e.g., the autonomous operation instructions 228); and at least partially autonomously operating (512) the robot in accordance with the set of autonomous operation instructions. For example, an identifier for the object (e.g., an identifier broadcasted by the object) may be used to index into a database (e.g., the database 114) to obtain the instructions. Such instructions may be prefetched and stored in the autonomous operation instructions 228. The method also includes, in accordance with a determination (514) that the object is not associated with a previously stored set of autonomous operation instructions: referring next to Figure 5B, receiving (516), from one or more second sensors (e.g., the one or more sensors 232 or a subset thereof), position information along a route that the robot is moved along by a user; generating (518) a set of autonomous operation instructions (e.g., using one or more modules of the autonomous operation module 226) based at least in part on the position information; associating (520) the set of autonomous operation instructions with the object; and storing (522) the set of autonomous operation instructions that are associated with the object. The information from the sensors (e.g., Bluetooth sensors) may be used to confirm proximity to the object. Object detection may serve as a homing mechanism. For example, a user may be prompted to move the robot until the object is in the field of view of the camera (e.g., the object appears near the center of the view finder).

[0102] Referring next to Figure 5C, in some implementations, at least partially autonomously operating the robot is performed (524) further in accordance with a determination that the position is associated with the previously stored set of autonomous operation instructions.

[0103] Referring next to Figure 5D, in some implementations, the determination that the object is not associated with a previously stored set of autonomous operation instructionsalso includes a determination (526) that the position is not associated with a previously stored set of autonomous operation instructions.

[0104] Referring next to Figure 5E, in some implementations, the method further includes prior to receiving position information, instructing (528) the user to move the robot around an area. For example, the visualization module 224 may generate and display (e.g., using the display 208) a visualization and / or instructions to move the autonomous robot 102.

[0105] Referring next to Figure 5F, in some implementations, receiving position information along a route that the robot is moved along by a user further comprises receiving (530), from the one or more second sensors, location and / or orientation information along the route that the robot is moved along by a user. Generating the set of autonomous operation instructions is based on the location and / or orientation information.

[0106] Referring next to Figure 5G, in some implementations, the method further includes, after detecting the object via the camera and ascertaining the position of the robot, autonomously moving (532) the robot to a new location (e.g., 10 cm - 360 cm forward, backwards, or to a side; preferably 30 cm forwards), initializing the robot’s location to the new location, and either generating the set of autonomous operation instructions or autonomously operating the robot in relation to the new location. In some implementations, the method further includes, after the robot is in the new location, rotating and / or scanning (534) surroundings, using the one or more second sensors, to orient the robot and to ensure there are no obstacles before initializing the autonomous robot and / or resuming operations.

[0107] Referring next to Figure 5H, in some implementations, the ascertaining and detecting steps are not performed (536) in any particular order.

[0108] Referring next to Figure 51, in some implementations, the position is (538) proximity to the one or more first sensors.

[0109] Referring next to Figure 5J, in some implementation, the one or more first sensors includes (540) a nearfield communication device.

[0110] Referring next to Figure 5K, in some implementation, the one or more first sensors includes (542) a LiDAR or Bluetooth sensor.

[0111] Referring next to Figure 5L, in some implementation, the object is (544) a QR- code, a barcode, or a fiducial marker.

[0112] Referring next to Figure 5M, in some implementation, the autonomous operation instructions are retrieved (546) from a cloud-hosted database, using the object as an identifier.

[0113] Referring next to Figure 5N, in some implementation, generating the set of autonomous operation instructions includes generating (548) cleaning instructions for the robot, and subsequent at least partial autonomous operation of the robot includes executing the cleaning instructions. For example, the cleaning module generates instructions and / or controls and / or operates the cleaning mechanisms 266.

[0114] Referring next to Figure 50, in some implementation, the camera is (550) a front-facing camera of the robot.

[0115] Referring next to Figure 5P, in some implementation, the camera provides (552) a live view of an environment on its screen, allowing the user to align the detected object with a view of the camera to ensure accurate positioning. In some implementation, the method further includes, after the object is centered in the camera view, autonomously accessing (554) the set of autonomous operation instructions associated with that object's identifier.

[0116] Referring next to Figure 5Q, in some implementation, the set of autonomous operation instructions includes (556) information related to a location, orientation, cleaning behaviors, environment requirements, scheduling requirements, and / or other relevant instructions for a cleaning robot or autonomous system.

[0117] Referring next to Figure 5R, in some implementation, the set of autonomous operation instructions includes (558) information related to coordinates, markers, and / or other identifying information that defines boundaries of each location or area where the robot is expected to operate.

[0118] Referring next to Figure 5S, in some implementation, the set of autonomous operation instructions includes (560) information related to specific cleaning behaviors including vacuuming, mopping, sweeping, scrubbing, burnishing, and / or their associated parameters, that may be used to control and / or operate the cleaning mechanisms 266.

[0119] Referring next to Figure 5T, in some implementation, the method further includes updating (562) the set of autonomous operation instructions to accommodate changes in an environment or user preferences. For example, the one or more sensors 232 may indicate changes (e.g., a change in temperature, noise level), and the autonomous operations module226 may then update the autonomous operation instructions 228 automatically based on signals processed by the signal processing module 222.

[0120] Referring next to Figure 5U, in some implementation, autonomously operating the robot comprises autonomously driving and cleaning (564). For example, the signal generation module 230 may generate signals to operate and / or control the actuators 232 and / or the cleaning mechanisms 266.

[0121] Referring next to Figure 5V, in some implementation, the set of autonomous operation instructions includes (566) information related to actions that the robot needs to perform based on floor type. For example, the one or more sensors 232 may provide signals processed by the signal processing module 222 to automatically infer a floor type. For example, a roughness parameter may indicate that the floor is not made of tiles. Further, such parameters may be set by the user, predetermined by the autonomous robot 102 and / or the server 112, and / or such parameters may be exchanged between the autonomous robots, and / or retrieved from the server 112 and / or the database 114.

[0122] Referring next to Figure 5W, in some implementation, the set of autonomous operation instructions includes (568) information related to scheduling requirements, including specific times or frequencies for cleaning operations, to ensure the robot operates according to desired cleaning schedule or preferences set by the user. Further, such schedules may be computed and / or stored locally in the autonomous robot 102 and / or stored remotely on the server 112 and / or the database 114.

[0123] Referring next to Figure 5X, in some implementation, the set of autonomous operation instructions includes (570) information related to safety protocols or guidelines, such as avoiding certain areas or hazards during cleaning operations. For example, the one or more sensors 232 may provide signals processed by the signal processing module 222 that may then infer the presence of hazards. In some implementations, the object detection module 220 may detect hazards and / or objects, for navigation, control and / or cleaning.

[0124] Referring next to Figure 5Y, in some implementation, the set of autonomous operation instructions includes (572) information related to user preferences or settings, such as preferred cleaning modes, water usage levels, tool settings, or other customization options, to provide a personalized cleaning experience based on the user's preferences.

[0125] Referring next to Figure 5Z, in some implementation, generating the set of autonomous operation instructions based on the position information includes using (574) sensor data (e.g., sensor data form the one or more sensors 232) to generate a point cloud through SLAM techniques of the robot’s surroundings.

[0126] Referring next to Figure 5Z1, in some implementation, the set of autonomous operation instructions includes (576) information related to vacuuming, including different settings for water usage, floor tool pressure, and / or brush type, based on floor material and debris type.

[0127] Referring next to Figure 5Z2, in some implementation, the set of autonomous operation instructions includes (578) information related to mopping with options for different settings (e.g., water usage, floor tool pressure, and cleaning solution type based on floor material and desired level of cleaning).

[0128] Referring next to Figure 5Z3, in some implementation, the set of autonomous operation instructions includes (580) information related to sweeping for different settings (e.g., brush type, brush speed, and brush pressure based on the floor material and debris type).

[0129] Referring next to Figure 5Z4, in some implementation, the set of autonomous operation instructions includes (582) information related to scrubbing for different settings such as brush type, brush speed, cleaning solution type, and water usage based on the floor material and level of cleaning required.

[0130] Referring next to Figure 5Z5, in some implementation, the set of autonomous operation instructions includes (584) information related to burnishing for different settings such as pad type, pad speed, and pad pressure based on the floor material and desired level of polishing.

[0131] Referring next to Figure 5Z6, in some implementation, the set of autonomous operation instructions includes (586) information related to user preferences, such as cleaning frequency, time of day, and specific instructions for unique situations or areas.

[0132] Referring next to Figure 5Z7, in some implementation, the method further includes reporting and / or tracking (588), including generating reports or logs of cleaning activities, including information such as date, time, location, cleaning actions performed, and any issues or exceptions encountered.

[0133] Referring next to Figure 5Z8, in some implementation, the one or more first sensors and the one or more second sensors are (590) different sensors.

[0134] Referring next to Figure 5Z9, in some implementation, the one or more first sensors and the one or more second sensors are (592) same sensors.

[0135] Referring next to Figure 5Z10, associating the object with the position includes associating (594) a position on a map with a current position and / or orientation of the robot. For example, during the setting up of a robot and / or the object at a new position in a customer’s location (e.g., during the flow described above in reference to Figure 3), suppose the robot is brought to a desired position, and the object has not yet been placed. The object is detected with a unique identifier. The user may bring the robot to a stationary position, at a desired initial position, with a camera view shown live on the display. The user may fix the object on wall, such that it is near the center of the display. A position on a map may then be associated with the current position and / or orientation of the robot.

[0136] Figure 6 shows a flowchart for an example method 600 for initializing a robot for location- specific actions, in accordance with some implementations. The method may be performed by the autonomous robot 102 and / or the server 112. The method includes, in response to detecting (602) an object via a camera, in response to receiving a first input, localizing and mapping (604) a region, based on signals received from one or more sensors, to obtain a plurality of poses. In response to receiving (606) a second input, the method includes ceasing (608) to localize and map the region, generating and displaying (610) (i) a representation of the region based on the plurality of poses, and (ii) a plurality of actions for cleaning or maintaining the region. In response to detecting (612) (i) a third input to approve the region and (ii) a fourth input to select one or more actions from the plurality of actions, the method includes encoding (614) the plurality of poses and the one or more actions as an instruction set; and associating (616) the instruction set with the object.

[0137] In some implementations, the method further includes, prior to detecting the object via the camera: detecting a presence of the object using a short-range wireless technology or remote sensing technology or their combination; and in response to detecting the presence of the object, displaying a prompt to position the camera towards the object.

[0138] The foregoing description, for purpose of explanation, has been described with reference to specific implementations. However, the illustrative discussions above are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Manymodifications and variations are possible in view of the above teachings. The implementations were chosen and described in order to best explain the principles of the invention and its practical applications, to thereby enable others skilled in the art to best utilize the invention and various implementations with various modifications as are suited to the particular use contemplated.

Claims

What is claimed is:

1. An autonomous robot guidance method, comprising: ascertaining, using one or more first sensors, a position of a robot; detecting an object via a camera; associating the object with the position; in accordance with a determination that the object is associated with a previously stored set of autonomous operation instructions: retrieving the autonomous operation instructions; and at least partially autonomously operating the robot in accordance with the set of autonomous operation instructions; and in accordance with a determination that the object is not associated with a previously stored set of autonomous operation instructions: receiving, from one or more second sensors, position information along a route that the robot is moved along by a user; generating a set of autonomous operation instructions based at least in part on the position information; associating the set of autonomous operation instructions with the object; and storing the set of autonomous operation instructions that are associated with the object.

2. The autonomous robot guidance method of claim 1, wherein at least partially autonomously operating the robot is performed further in accordance with a determination that the position is associated with the previously stored set of autonomous operation instructions.

3. The autonomous robot guidance method of claim 1, wherein the determination that the object is not associated with a previously stored set of autonomous operation instructions also includes a determination that the position is not associated with a previously stored set of autonomous operation instructions.

4. The autonomous robot guidance method of claim 1, further comprising: prior to receiving position information, instructing the user to move the robot around an area.

5. The autonomous robot guidance method of claim 1, wherein receiving position information along a route that the robot is moved along by a user further comprises receiving, from the one or more second sensors, location and / or orientation information along the route that the robot is moved along by a user, wherein generating the set of autonomous operation instructions is based on the location and / or orientation information.

6. The autonomous robot guidance method of claim 1, further comprising: after detecting the object via the camera and ascertaining the position of the robot, autonomously moving the robot to a new location, initializing the robot’s location to the new location, and either generating the set of autonomous operation instructions or autonomously operating the robot in relation to the new location.

7. The autonomous robot guidance of method of claim 6, further comprising: after the robot is in the new location, rotating and scanning surroundings, using the one or more second sensors, to orient the robot and ensure no obstacles before initializing.

8. The autonomous robot guidance method of claim 1, wherein associating the object with the position comprises associating a position on a map with a current position and / or orientation of the robot.

9. The autonomous robot guidance method of claim 1, wherein the ascertaining and detecting are not performed in any particular order.

10. The autonomous robot guidance method of claim 1, wherein the position is proximity to the one or more first sensors.

11. The autonomous robot guidance method of claim 1, wherein the one or more first sensors includes a nearfield communication device.

12. The autonomous robot guidance method of claim 1, wherein the one or more first sensors includes a LiDAR or Bluetooth sensor.

13. The autonomous robot guidance method of claim 1, wherein the object is a QR-code, a barcode, or a fiducial marker.

14. The autonomous robot guidance method of claim 1, wherein the autonomous operation instructions are retrieved from a cloud-hosted database, using the object as an identifier.

15. The autonomous robot guidance method of claim 1, wherein generating the set of autonomous operation instructions includes generating cleaning instructions for the robot, and wherein subsequent at least partial autonomous operation of the robot includes executing the cleaning instructions.

16. The autonomous robot guidance method of claim 1, wherein the camera is a frontfacing camera of the robot.

17. The autonomous robot guidance method of claim 1, wherein the camera provides a live view of an environment on its screen, allowing the user to align the detected object with a view of the camera to ensure accurate positioning.

18. The autonomous robot guidance method of claim 17, further comprising: after the object is centered in the view of the camera, autonomously accessing the set of autonomous operation instructions associated with that object's identifier.

19. The autonomous robot guidance method of claim 1, wherein the set of autonomous operation instructions includes information related to an initial location, pose, orientation, cleaning behaviors, environment requirements, scheduling requirements, and / or other relevant instructions for a cleaning robot or autonomous system.

20. The autonomous robot guidance method of claim 1, wherein the set of autonomous operation instructions includes information related to coordinates, markers, and / or other identifying information that defines boundaries of each location or area where the robot is expected to operate.

21. The autonomous robot guidance method of claim 1, wherein the set of autonomous operation instructions includes information related to specific cleaning behaviors including vacuuming, mopping, sweeping, scrubbing, burnishing, and / or their associated parameters.

22. The autonomous robot guidance method of claim 1, further comprising updating the set of autonomous operation instructions to accommodate changes in an environment or user preferences.

23. The autonomous robot guidance method of claim 1, wherein autonomously operating the robot comprises autonomously driving and cleaning.

24. The autonomous robot guidance method of claim 1, wherein the set of autonomous operation instructions includes information related to actions that the robot needs to perform based on floor type.

25. The autonomous robot guidance method of claim 1, wherein the set of autonomous operation instructions includes information related to scheduling requirements, including specific times or frequencies for cleaning operations, to ensure the robot operates according to desired cleaning schedule or preferences set by the user.

26. The autonomous robot guidance method of claim 1, wherein the set of autonomous operation instructions includes information related to safety protocols or guidelines, for avoiding certain areas or hazards during cleaning operations.

27. The autonomous robot guidance method of claim 1, wherein the set of autonomous operation instructions includes information related to user preferences or settings, including preferred cleaning modes, water usage levels, tool settings, and / or other customization options, to provide a personalized cleaning experience based on the user's preferences.

28. The autonomous robot guidance method of claim 1, wherein generating the set of autonomous operation instructions based on the position information comprises using sensor data to generate a point cloud through SLAM techniques of the robot’s surroundings.

29. The autonomous robot guidance method of claim 1, wherein the set of autonomous operation instructions includes information related to vacuuming, including different settings for water usage, floor tool pressure, and / or brush type, based on floor material and debris type.

30. The autonomous robot guidance method of claim 1, wherein the set of autonomous operation instructions includes information related to mopping with options for differentsettings, including water usage, floor tool pressure, and / or cleaning solution type, based on floor material and desired level of cleaning.

31. The autonomous robot guidance method of claim 1, wherein the set of autonomous operation instructions includes information related to sweeping for different settings, including brush type, brush speed, and / or brush pressure, based on floor material and / or debris type.

32. The autonomous robot guidance method of claim 1, wherein the set of autonomous operation instructions includes information related to scrubbing for different settings, including brush type, brush speed, cleaning solution type, and / or water usage, based on floor material and level of cleaning required.

33. The autonomous robot guidance method of claim 1, wherein the set of autonomous operation instructions includes information related to burnishing for different settings, including pad type, pad speed, and / or pad pressure, based on the floor material and desired level of polishing.

34. The autonomous robot guidance method of claim 1, wherein the set of autonomous operation instructions includes information related to user preferences, including cleaning frequency, time of day, and / or specific instructions for unique situations or areas.

35. The autonomous robot guidance method of claim 1, further comprising: reporting and / or tracking, including generating reports or logs of cleaning activities, including date, time, location, cleaning actions performed, and / or any issues or exceptions encountered.

36. The autonomous robot guidance method of claim 1, wherein the one or more first sensors and the one or more second sensors are different sensors.

37. The autonomous robot guidance method of claim 1, wherein the one or more first sensors and the one or more second sensors are same sensors.

38. A method of initializing a robot for location-specific actions, comprising: in response to detecting an object via a camera:in response to receiving a first input, localizing and mapping a region, based on signals received from one or more sensors, to obtain a plurality of poses; and in response to receiving a second input: ceasing to localize and map the region; generating and displaying (i) a representation of the region based on the plurality of poses, and (ii) a plurality of actions for cleaning or maintaining the region; and in response to detecting (i) a third input to approve the region and (ii) a fourth input to select one or more actions from the plurality of actions: encoding the plurality of poses and the one or more actions as an instruction set; and associating the instruction set with the object.

39. The method of claim 38, further comprising: prior to detecting the object via the camera: detecting a presence of the object using one of a short-range wireless, remote sensing technology or their combination, and wherein the object is capable of emitting bluetooth signals; and in response to detecting the presence of the object, displaying a prompt to position the camera towards the object.

40. A robot comprising: a camera configured to detect one or more objects; an object detection module configured to detect a presence of the one or more objects; a graphical user interface; one or more sensors configured to obtain signals related to poses of the robot; a memory configured to store (i) one or more instruction sets, each instruction set associated with an object of the one or more objects and (ii) a plurality of actions for cleaning or maintaining; and a processor configured to: detect a presence of an object using the short-range wireless module; in response to detecting the presence of the object, display a prompt on the graphical user interface to position the camera towards the object; and in response to detecting the object via the camera:in response to receiving a first input, via the graphical user interface, localize and map a region, based on signals received from the one or more sensors, to obtain a plurality of poses; and in response to receiving a second input via the graphical user interface: cease to localize and map the region; generate and display, via the graphical user interface, (i) a representation of the region based on the plurality of poses, and (ii) the plurality of actions for cleaning or maintaining the region; and in response to detecting, via the graphical user interface, (i) a third input to approve the region and (ii) a fourth input to select one or more actions from the plurality of actions: encode the plurality of poses and the one or more actions as an instruction set; and associate the instruction set with the object.

Images