SYSTEMS AND METHODS FOR DYNAMIC OUTPATIENT COMPUTINGS TO THE CLOUD

Abstract

A method for operating a mobile robot to dynamically offload computational tasks to a cloud system is described. Advantageously, the method allows the mobile robot to switch between local execution by the robot itself and remote execution by the cloud system at any time, depending on time, energy, or other requirements. This enables the mobile robot to be robust against varying network conditions while simultaneously leveraging the advantages of external computing resources whenever possible. In at least some embodiments, middleware, such as a robot operating system, is used for communication between the mobile robot and the cloud system.

Description

AREA

[0001] The devices and methods disclosed in this document relate to mobile robot systems and, in particular, the dynamic outsourcing of robot calculations to the cloud. BACKGROUND

[0002] Unless otherwise stated herein, the materials described in this section shall not be deemed to represent prior art simply by virtue of their inclusion in this section.

[0003] Mobile robots, which move around an environment to perform a task, have become increasingly popular in recent years due to their ease of use and effectiveness in autonomously executing tasks. To perform their tasks effectively, many mobile robots employ computationally intensive techniques to understand their environment and execute their tasks accurately and robustly. For example, many mobile robots need to perform image processing, simultaneous localization and mapping (SLAM), and motion planning to carry out their tasks.

[0004] However, many mobile robots have limited integrated or vehicle-side computing resources. This is especially true for battery-powered mobile robots with limited payload capacity, such as robotic vacuum cleaners and unmanned aerial vehicles (UAVs). As a result, such mobile robots may only be able to operate for limited periods or perform tasks requiring rapid calculations to a limited extent.

[0005] Therefore, a method for operating a mobile robot is needed that reduces the computational load of the mobile robot while still allowing the mobile robot to benefit from computationally intensive techniques for understanding its environment and for accurately and robustly performing its tasks. SUMMARY

[0006] A method for operating a mobile robot is disclosed. The method includes receiving input data, with which a computational task is to be performed, using a local processing system of the mobile robot. The method further includes determining whether the computational task should be outsourced to a remote processing system. In response to a determination that the computational task should be outsourced to the remote processing system, the method further includes (i) transmitting a message to the remote processing system using a transceiver of the mobile robot, the message comprising the input data, and (ii) receiving an output of the computational task from the remote processing system using the transceiver.The method further specifies, in response to a determination that the computation task should not be outsourced to the remote processing system, a determination of the output of the computation task using the local processing system. The method further specifies operating the mobile robot with the local processing system based on the output of the computation task.

[0007] A method for remotely executing calculations for a mobile robot is also disclosed. The method includes storing second program instructions, comprising a copy of first program instructions used by the mobile robot to perform a computational task, in a memory of a remote computer. The method further includes receiving a message from a mobile robot with a remote processing system of the remote computer, the message comprising input data and specifying the computational task to be performed with respect to the input data. The method further includes determining an output of the computational task with the remote processing system by executing the second program instructions. The method further includes transmitting the output of the computational task to the mobile robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above-mentioned designs and other features of the systems and processes are explained in the following description in conjunction with the attached drawings. Fig. 1 summarizes the components and operating procedures of a mobile robot system. Fig. Figure 2A shows an example embodiment of one of the mobile robots. Fig. Figure 2B shows an example implementation of the cloud system. Fig. Figure 3 shows a flowchart for a procedure for operating a mobile robot to dynamically outsource computational tasks to a cloud system. Fig. Figure 4 shows an example of dynamic offloading with a local robot node, where the offloading decision is made by a scheduler of the robot node. The Fig. 5A and Fig. Figure 5B shows exemplary graphical user interfaces that are displayed on a screen to represent profile data captured in the mobile robot system. Fig. Figure 6 shows an embodiment of the mobile robot system with an integrated gateway. Fig. Figure 7 shows an embodiment of the mobile robot system with integrated cloud-based services. DETAILED DESCRIPTION

[0009] For a better understanding of the principles of the disclosure, reference is now made to the embodiments illustrated in the drawings and described in the following patent specification. It is understood that this is not intended to limit the scope of the disclosure. Furthermore, it is understood that the present disclosure includes all changes and modifications of the illustrated embodiments and further applications of the principles of the disclosure that would normally occur to a person skilled in the art in the field to which this disclosure belongs. Overview

[0010] With reference to Fig. Figure 1 summarizes the components and operational processes of a mobile robot system 10. The mobile robot system 10 comprises one or more mobile robots 120, each configured to perform a task in a specific environment. The mobile robots 120 advantageously utilize cloud computing or fog computing to dynamically offload computational tasks. For this purpose, the mobile robot system 10 also includes a cloud system 150. The cloud system 150 can comprise any computing device that is not physically located on the robot.

[0011] It should be noted that cloud computing refers to accessing and using remote computing and data storage devices. Similarly, fog computing refers to accessing and using cloud and closer computing devices, such as network gateways, servers in the same facility as the robot, desktops on a local network, etc. Since the 120 mobile robots often have limited built-in computing resources, cloud or fog computing is used to extend these capabilities.

[0012] In at least some embodiments, the operating procedures of the mobile robots 120 are implemented using robotics middleware. The robotics middleware provides a set of software libraries to enable communication between mobile robots and other computing devices. Robots and computing devices communicating via the robotics middleware do not need to have the same hardware, as the robotics middleware aggregates or abstracts data from a hardware-specific format into robotics middleware messages. Using the robotics middleware, the operating procedures of the mobile robots are modularly organized into a multitude of discrete processes, referred to here as robot nodes 20. Each robot node 20 performs a specific function of the mobile robot 120, which executes computational tasks on request and returns a result.

[0013] The robot nodes 20 communicate with each other by sending middleware messages in a predefined format (e.g., in a format defined by the robotics middleware and defined prior to sending or receiving). Using such messages, the various mobile robots 120 can also communicate with each other, with the cloud system 150, or with any other computing device. In some embodiments, each robot node 20 also acts as one or both of (i) a publisher, which publishes data to topics, and (ii) a subscriber, which receives data from topics. As used here, “topics” refer to communication channels for sending or receiving messages to or from a device that has published or subscribed to the topic.

[0014] To facilitate the execution of outsourced computational tasks by Cloud System 150, Cloud System 150 implements copies of the Robot Nodes 20, referred to here as Robot Cloud Nodes 30, which are configured to perform the same computational tasks but have more extensive computing resources. The Robot Cloud Nodes 30 are implemented with the same robotics middleware as the Robot Nodes 20. Cloud System 150 executes computational tasks that are dynamically outsourced from a given Robot Node 20 using the corresponding Robot Cloud Node 30, which is a copy of that Robot Node 20. The Mobile Robots 120 and Cloud System 150 use a middleware-compatible messaging system to transmit middleware messages over a network (e.g., the internet) with possible intermediate conversions.

[0015] In at least one embodiment, the robotics middleware implemented by the mobile robot 120 and the cloud system 150 is the Robot Operating System (ROS). In such embodiments, the robot nodes 20 and the robot cloud nodes 30 are ROS nodes, and the middleware messages comprise ROS messages or FogROS messages. It should be noted that FogROS is a software library for connecting local ROS nodes with cloud-based ROS nodes. ROS and FogROS enable the mobile robots 120 to communicate with each other, with the cloud system 150, and with other local computing devices or cloud systems. ROS messages are transmitted between the mobile robots 120 using the standard ROS message transmission mechanisms. ROS messages are forwarded to the cloud via FogROS.It should be noted that any version of ROS and any version of FogROS can be used as robotics middleware for the Mobile Robot System 10. Furthermore, it should be noted that other robotics middleware can also be used, and the disclosure is not to be understood as being limited to ROS or ROS-related middleware solutions.

[0016] With continued reference to Fig. Each mobile robot 120 includes a robot profiler 40. The robot profilers 40 are executed by the respective mobile robots 120 to collect profile data, including CPU utilization, network latency, and the like for each mobile robot 120. Similarly, the cloud system 150 includes a profiler 50, which is executed to collect additional profile data, including, for example, CPU utilization, GPU utilization, network utilization, outsourced computing costs per hour, and the like. The profilers 40 and 50 transmit profile messages to share the collected profile data with other devices in the mobile robot system 150. In some embodiments, the profile messages take the form of middleware messages as described above.

[0017] Based on the profile data collected by profilers 40 and 50, the robot nodes 20 of the mobile robots dynamically determine whether computational tasks should be performed locally or outsourced to the cloud system 150. The robot nodes 20 of each mobile robot 120 make this determination based on several criteria, such as round-trip latency, frame rate, battery life, CPU utilization, cloud costs, etc., which are evaluated based on the profile data collected by profilers 40 and 50. If the processing resources of the mobile robot 120 are almost fully utilized, its battery power is low, and a good network connection to the cloud system 150 exists, the computational task can be scheduled in the cloud.Similarly, the computation task can be planned in such a way that it is performed locally if the network connection becomes poor, the cloud system takes too long to respond, or cloud costs exceed a budget.

[0018] In this way, the mobile robots 120 can offload computational tasks to the cloud system 150 or to other offboard computing devices. Furthermore, it should be noted that this offloading is dynamic; that is, the execution of computational tasks can be switched at any time between local execution by the mobile robot 120 and remote execution by the cloud system 150, depending on time constraints, energy requirements, or other needs. Thus, dynamic offloading enables the mobile robots 120 to be robust against changing network conditions while simultaneously leveraging the advantages of external or offboard computing resources when available.

[0019] In some embodiments, the Cloud System 150 is configured to dynamically allocate computing resources, such as GPUs, CPUs, virtual machines, and elastic compute instances. Specifically, computing resources are automatically allocated and released based on one or more of the following factors: the Quality of Service (QoS) requested by the mobile robots 120 connected to the Cloud System 150, the number of mobile robots 120 currently connected to the Cloud System 150, the processing volume requested by the mobile robots 120, and a desired budget for cloud computing costs.

[0020] In some embodiments, the mobile robot system 10 further comprises a display screen 60. The display screen 60 is equipped with a network-connected computing device (not shown) or can be directly integrated into one or more of the mobile robots 120. In each case, the computing device associated with the display screen 60 receives profile messages from the profilers 40, 50 and operates the display screen 60 to display profile data acquired by the profilers 40, 50. In some embodiments, the computing device associated with the display screen 60 implements one or more middleware nodes (e.g., ROS nodes), as similarly described with respect to the robot nodes 20. Mobile robot

[0021] Fig. Figure 2A shows an embodiment of one of the mobile robots 120. In the illustrated embodiment, the mobile robot 120 has, for example, a processor 122, a memory 124, one or more sensors 126, one or more actuators 128, and at least one network communication module 130. It is understood that the illustrated embodiment of the mobile robot 120 is only an embodiment and is merely representative of various types or configurations or arrangements of mobile robots that autonomously navigate or move within an environment to perform a task.

[0022] The processor 122 is configured to execute instructions to operate the mobile robot 120 in order to enable the features, functionalities, properties, and / or the like described herein. For this purpose, the processor 122 is functionally connected to the memory 124, the one or more sensors 126, and the one or more actuators 128. The processor 122 generally comprises one or more processors that can operate in parallel or otherwise coordinated with one another. It is known to those skilled in the art that a "processor" comprises any hardware system, hardware mechanism, or hardware component that processes data, signals, or other information. Accordingly, the processor 122 may comprise a system with a central processing unit, graphics processing units, multiple processing units, dedicated circuits for achieving functionality, programmable logic, or other processing systems.

[0023] Memory 124 is configured to store data and program instructions which, when executed by processor 122, enable the mobile robot 120 to perform various operating procedures or operations described herein. Memory 124 can be any type of device capable of storing information accessible to processor 122, such as a memory card, ROM, RAM, hard disks, floppy disks, flash memory, or any other computer-readable medium serving as a data storage device, as is known to those skilled in the art. As explained in further detail below, processor 122 is configured to execute program instructions of an operating procedure 132 stored in memory 124 to navigate its environment and perform a task.In at least one embodiment, the operating method 132 is implemented using middleware in the form of one or more robot nodes 20. Furthermore, the processor 122 is configured to execute program instructions from the robot profiler 40 to acquire profile data, including CPU utilization, network latency, and the like, for the mobile robot 120.

[0024] The one or more sensors 126 can comprise a variety of different sensors. In some embodiments, the sensors 126 include sensors configured to measure one or more accelerations, rotational speeds, and / or orientations of the mobile robot 120. In one embodiment, the sensors 126 include one or more accelerometers configured to measure linear accelerations of the mobile robot 120 along one or more axes (e.g., roll, pitch, and yaw axes), or one or more gyroscopes configured to measure rotational speeds of the mobile robot 120 along one or more axes (e.g., roll, pitch, and yaw axes), and / or an inertial measurement unit configured to measure all of the above quantities.

[0025] In at least some embodiments, the sensors 126 comprise a light sensor (e.g., LiDAR or another time-of-flight or structured light sensor) configured to emit measurement light (e.g., a laser) and receive the measurement light after it has been reflected in the environment. In time-of-flight-based embodiments, the processor 122 is configured to calculate travel times and / or return times for the measurement light. Based on the calculated travel times and / or return times, the processor 122 can, for example, generate map data, such as a point cloud. In structured light-based embodiments, the processor 122 applies an algorithm to extract a 3D profile of surfaces onto which the structured light is projected (e.g., based on a stripe pattern generated on a surface).

[0026] In some embodiments, the sensors 126 include, as an alternative to or in addition to the light sensor, one or more cameras configured to capture a multitude of images of the environment as the mobile robot 120 navigates through it. The camera(s) generate image frames of the environment, each of which has a two-dimensional arrangement of pixels. Each pixel has corresponding photometric information (color, intensity, and / or brightness). In some embodiments, the camera(s) is / are configured to generate RGB-D images in which each pixel has corresponding photometric and geometric information (depth and / or distance).In such embodiments, the camera(s) can take the form of an RGB camera operating in conjunction with a LiDAR or IR sensor, in particular a LiDAR or IR camera configured to provide both photometric and geometric information. The LiDAR or IR camera can be separate from the RGB camera or directly integrated into it. Alternatively or additionally, the camera can have two RGB cameras configured to capture stereoscopic images from which depth and / or distance information can be derived. Based on RGB-D images captured while the mobile robot 120 navigates its environment, the mobile robot 120 can implement visual and / or visual inertial odometry methods such as simultaneous localization and mapping (SLAM) techniques.

[0027] The one or more actuators 128 comprise at least motors of a locomotion system that, for example, drive a set of wheels to cause the mobile robot 120 to move through the environment to perform the task. In addition, in some embodiments, the one or more actuators 128 comprise a vacuum suction system configured to vacuum a floor surface while the mobile robot 120 navigates the environment. Mobile robots 120 that perform other tasks in the environment may, of course, comprise different types of actuators 128 suitable for other tasks.

[0028] The Network Communications Module 130 may include one or more transceivers, modems, processors, memory, oscillators, antennas, or other hardware typically contained in a communications module to enable communication with various other devices, including at least the Cloud System 150 and / or the other Mobile Robots 120. Specifically, the Network Communications Module 130 generally includes a Wi-Fi module configured to enable communication with a Wi-Fi network and / or a Wi-Fi router (not shown). Furthermore, the Network Communications Module 130 may include a Bluetooth® module (not shown) configured to enable communication with a mobile device (not shown). Finally, the Network Communications Module 130 may include one or more cellular modems configured to communicate with wireless telephone networks.

[0029] The mobile robot 120 may also include a battery or other power source (not shown) configured to power the various components within the mobile robot 120. In one embodiment, the battery of the mobile robot 120 is a rechargeable battery configured to be recharged when the mobile robot 120 is connected to a base station configured for use with the mobile robot 120. Cloud system

[0030] As mentioned above, the mobile robots 120 communicate with a cloud system 150. In particular, the cloud system 150 is set up to perform computational tasks that have been dynamically outsourced by the mobile robots 120.

[0031] Fig. Figure 2B shows an embodiment of the cloud system 150. The cloud system 150 comprises one or more cloud servers 152. The cloud servers 152 can include servers configured to perform a variety of functions for the cloud system 150, including web servers or application servers, depending on the features provided by the cloud system 150. However, they include at least one or more cloud servers 152 for dynamically offloading computational tasks from the mobile robots 120. Each cloud server 152 includes, for example, a processor 154, a memory 156, a user interface 158, and a network communication module 160. It is understood that the embodiment of the cloud servers 152 shown is only one embodiment of a cloud server 152 and is merely representative of different types or configurations.Arrangements of a personal computer, server or other data processing system that operates in the manner described herein.

[0032] The processor 154 is configured to execute instructions to operate the cloud server 152 in order to enable the features, functionalities, properties, and / or the like described herein. For this purpose, the processor 154 is functionally connected to the memory 156, the user interface 158, and the network communication module 160. The processor 154 generally comprises one or more processors that can operate in parallel or otherwise coordinated with one another. It is known to those skilled in the art that a "processor" includes any hardware system, hardware mechanism, or hardware component that processes data, signals, or other information. Accordingly, the processor 154 may comprise a system with a central processing unit, graphics processing units, multiple processing units, dedicated circuits for achieving functionality, programmable logic, or other processing systems.

[0033] Memory 156 is configured to store program instructions which, when executed by Processor 154, enable Cloud Server 152 to perform various operations described herein. Memory 156 can be any type of device or combination of devices capable of storing information accessible to Processor 154, such as memory cards, ROM, RAM, hard disks, floppy disks, flash memory, or any other computer-readable medium known to the average person. As further explained below, Processor 154 is configured to execute program instructions stored in Memory 156 to perform computational tasks dynamically offloaded by Mobile Robots 120.Specifically, memory 156 stores program instructions corresponding to the robot cloud nodes 30, which are implemented using the middleware as explained above. Processor 154 performs computational tasks that are dynamically outsourced from each robot node 20 using the corresponding robot cloud node 30, which is a copy of the respective robot node 20. Furthermore, processor 122 is configured to execute program instructions from profiler 50 to collect additional profile data, such as CPU utilization, GPU utilization, network utilization, outsourced computing costs per hour, and the like for cloud system 150.

[0034] The cloud server 152 can be operated locally or remotely by an administrator. To facilitate local operation, the cloud server 152 can include the user interface 158. In at least one embodiment, the user interface 158 can suitably include an LCD display screen or the like, a mouse or other pointing device, a keyboard or other keypad or control panel, a speaker, and a microphone, as is known to a person skilled in the art. Alternatively, in some embodiments, an administrator can operate the cloud server 152 remotely from another computing device that communicates with it via the network communication module 160 and has an analog user interface.

[0035] The network communication module 160 provides an interface that enables communication with any devices, including at least the mobile robots 120. Specifically, the network communication module 160 can include a LAN (Local Area Network) port, enabling communication with any local computers located in the same or a nearby facility. Generally, the cloud server 152 communicates with remote computers over the internet via a separate modem and / or router on the local network. Alternatively, the network communication module 160 can also include a WAN (Wide Area Network) port, enabling communication over the internet. In one embodiment, the network communication module 160 is equipped with a Wi-Fi transceiver or other wireless communication device.Therefore, it follows that communication with the Cloud Server 152 can take place via wired or wireless communication. This communication can be carried out using any of the various known communication protocols. Method for dynamically outsourcing computational tasks from a mobile robot

[0036] The following describes various methods and processes for dynamically offloading computational tasks from a mobile robot. In these descriptions, statements that a method, processor, and / or system performs a task or function refer to a controller or processor (e.g., processor 154 of cloud server 152 or processor 122 of mobile robot 120) that executes programmed instructions stored in non-volatile, computer-readable storage media (e.g., memory 156 of cloud server 152 or memory 124 of mobile robot 120) that are functionally connected to the controller or processor to manipulate data or operate one or more components in cloud server 152 or mobile robot 120 to perform the task or function.Furthermore, the steps of the procedures can be carried out in any possible chronological order, regardless of the order shown in the figures or the order in which the steps are described.

[0037] Fig. Figure 3 shows a flowchart for a method 200 for operating a mobile robot to dynamically offload computational tasks to a cloud system. Method 200 advantageously enables a mobile robot 120 to switch between local execution by the mobile robot 120 and remote execution by the cloud system 150 at any time, depending on time, energy, or other requirements. Thus, method 200 allows the mobile robot 120 to be robust against varying network conditions while simultaneously utilizing external computing resources where possible.

[0038] Method 200 begins by receiving input data for processing (block 210). Specifically, the processor 122 of the mobile robot 120 receives input data to be processed. In at least some embodiments, the input data is sensor data acquired by one or more of the sensors 126 of the mobile robot 120. In one example, the sensor data includes an image captured by a camera of the sensors 126 of the mobile robot 120. In some embodiments, the processor 122 executes program instructions from corresponding robot nodes 20 to operate the one or more sensors 126 to acquire sensor data. Next, the processor 122 executes program instructions from the corresponding robot nodes 20 to publish the sensor data in an appropriate topic, for example, by sending an appropriately formatted middleware message from the corresponding robot nodes 20.

[0039] In further embodiments, the input data is user input that identifies a task to be performed by the mobile robot. In particular, in some embodiments, the mobile robot 120 includes a user interface or communicates with a mobile device (not shown) that has a mobile robot application functioning as a user interface. Via the user interface, the processor 122 of the mobile robot 120 receives user input that, at least in some cases, specifies a task to be performed by the mobile robot 120, which includes one or more computational tasks.

[0040] The procedure 200 continues with a determination of whether a computational task to be performed with respect to the input data should be offloaded (block 220). Specifically, the processor 122 determines, based on the received input data or as a result of receiving the input data, which computational tasks should be performed. Next, the processor 122 or the processor 154 of the cloud system 150 determines whether or not the computational task should be offloaded to the cloud system 150. In some embodiments, the processor 122 or the processor 154 divides the computational task into small time segments or has a mechanism for interrupting processing so that the computational task can be switched between local and remote execution at any time.

[0041] As mentioned previously, in some embodiments the input data is sensor data published by the corresponding robot node(s) 20 in a corresponding topic, e.g., by sending a suitably formatted middleware message. One or more of the other robot nodes 20 of the mobile robot 120 can subscribe to the middleware topic and / or receive the sensor data or other input data in another way. In any case, the program instructions of these other robot nodes 20 are configured to perform a computational task with respect to the sensor data or other input data.

[0042] In one embodiment, the robot nodes 20 of the mobile robot 120 include a robot node 20 configured to receive an image of the environment captured by a camera of the mobile robot 120 and to detect an object located in the image, e.g., to determine a boundary field around the object and / or to classify the object. Such a robot node 20 of the mobile robot 120 can subscribe to a middleware topic to which images from another of the robot nodes 20 are published. Thus, the processor 122 can recognize that an object recognition task needs to be performed with respect to the image.

[0043] In one embodiment, the robot nodes 20 of the mobile robot 120 include a robot node 20 configured to receive sensor data, such as image or LiDAR data, acquired in the environment and to determine the position or orientation of the mobile robot 120 within that environment. Such a robot node 20 of the mobile robot 120 can subscribe to a middleware topic to which such sensor data is published by one or more other robot nodes 20. Thus, the processor 122 can recognize that a robot localization task needs to be performed with respect to the sensor data.

[0044] In one embodiment, the robot nodes 20 of the mobile robot 120 include a robot node 20 configured to receive sensor data or map data regarding the environment and / or input data identifying a task to be performed in the environment, and to determine a trajectory along which the mobile robot 120 is to navigate through the environment. Such a robot node 20 of the mobile robot 120 can subscribe to a middleware topic in which such sensor data or input data from one or more other robot nodes 20 are published. Thus, the processor 122 can recognize that a robot motion planning task is to be performed with respect to the sensor data or other input data.

[0045] Once the computational task(s) to be performed on the input data have been identified, System 10 determines whether each computational task should be offloaded to Cloud System 150. This determination can be made locally by Processor 122 of Mobile Robot 120 or by Processor 154 in the cloud by Cloud System 150. If Cloud System 150 determines that a computational task should be offloaded, it transmits a message indicating whether the computational task should be offloaded, for example, with a binary flag in the message, and Mobile Robot 120 receives this message.

[0046] To enable an informed decision as to whether or not to outsource the computational task to the cloud system 150, the profilers 40 and 50 of the mobile robot system 10 collect a variety of profile data. Specifically, the processor 122 of the mobile robot 120 executes program instructions from the robot profiler 40 to collect profile data that characterizes the processor 122's potential performance of the computational task. Such profile data collected using the robot profiler 40 includes, for example, network latencies, latency across middleware topics or messages, processing times, frame rates across middleware topics or messages, computational resource consumption such as CPU utilization, GPU utilization, system memory utilization, and video memory utilization, and computational costs such as battery / power consumption.

[0047] Similarly, the processor 154 of the cloud system 150 executes program instructions from the profiler 50 to collect profile data that characterizes the processor 154's potential performance of the computational task. Such profile data collected using the profiler 50 includes, for example, network latency, network utilization, latency across middleware topics or middleware messages, processing times, frame rates across middleware topics or middleware messages, computational resource consumption such as CPU utilization, GPU utilization, system memory utilization, and video memory utilization, and computational costs such as energy consumption and financial costs (e.g., dollar cost per hour).

[0048] With continued reference to Fig. 3. Based on the acquired profile data, the processor 122 of the mobile robot 120 or the processor 154 of the cloud system 150 determines whether the computation task should be outsourced to the cloud system 150 or not. In at least some embodiments, the processor 122 or the processor 154 determines whether the computation task should be outsourced based on at least one optimization criterion. Such optimization criteria may include, for example, minimizing the total round-trip computation time of the computation task, minimizing the computation costs of the computation task, and maximizing the battery life of the mobile robot 120. The processor 122 or the processor 154 determines an optimal decision, taking the optimization criteria into account, using a suitable optimization method for the specific set of selected criteria, e.g., integer programming.

[0049] In one example, processor 122 receives a stream of images, for instance, for obstacle detection. Each time an image is captured by the camera, processor 122 or processor 154 determines whether the obstacle detection task should be offloaded. For example, if the average round-trip processing time in the cloud has historically been shorter than that of the robot, the scheduler will decide to offload the task. Once the processing of one image is complete, a decision is made as to whether the processing of the second image should be offloaded, and so on. To ensure an accurate estimate of the processing time for both the cloud system 150 and the mobile robot 120, processor 122 or processor 154 might, in some embodiments, periodically make a suboptimal decision to keep the processing time estimates up to date.In other words, the calculation tasks are performed at least periodically on the mobile robot 120 and at least periodically on the cloud system 150, so that the processing times or other profile data are kept up-to-date and accurate.

[0050] Fig. Figure 4 shows an example of dynamic offloading with a local robot node 20, where the decision to offload is made by a scheduler of the robot node 20. In particular, in at least some embodiments, the mobile robot 120 and / or each robot node implemented on it includes program instructions corresponding to a scheduler 300. The processor 122 executes the scheduler 300 to determine whether or not the computation task should be offloaded to the cloud system 150. The scheduler 300 receives input, often in the form of sensor data, e.g., camera images, indicating that a computation task has been requested. The scheduler 300 optimizes a number of criteria to decide whether to offload. The scheduler 300's decision can be based on profile data acquired by the profilers 40 and 50.The Scheduler 300 can divide the task into small time segments or has a mechanism to interrupt local and remote processing. Once the computation task is complete, the result is output, often in the form of robot actuator commands.

[0051] In some embodiments, processor 122 or processor 154 determines whether the computational task should be partially offloaded, based on whether the computational task involves performing a task in the environment that requires coordination between the mobile robot 120 and at least one other mobile robot 120. Particularly for multi-robot coordination tasks, it is often necessary to know the state or status of other mobile robots 120 when certain computational tasks, such as motion planning, are performed. However, it should be noted that the cloud system 150 can have a more accurate and up-to-date understanding of the actual state of each individual mobile robot 120 involved in the multi-robot coordination task.Conversely, the individual mobile robots 120 involved in the multi-robot coordination task may have a limited and estimated understanding of the actual state of each other mobile robot 120. In this way, the cloud system 150 can perform certain types of computational tasks related to a multi-robot coordination task more effectively than the individual mobile robots 120.

[0052] Accordingly, in some embodiments, the processor 122 or the processor 154 determines whether the computational task should be partially outsourced, depending on a desired or required degree of coordination between other mobile robots 120. In one example, the processor 122 or the processor 154 determines that the computational task should be outsourced in response to the desired / required degree of coordination between other mobile robots 120 exceeding a threshold. In another embodiment, the processor 122 or the processor 154 determines that the computational task should be outsourced in response to a number of mobile robots located close to each other (e.g., within a predefined radius) exceeding a predefined threshold, as a high degree of coordination would be required.In the event that local execution by the mobile robot 120 is nevertheless chosen, the cloud system 150 in some embodiments transmits the latest states of the other mobile robots 120 to each mobile robot 120 involved in the multi-robot coordination task.

[0053] In at least some embodiments, the acquired profile data is displayed on the display screen 60. It is particularly important to note that the profilers 40 and 50 are distributed across the mobile robots 120 and the cloud system. In at least one embodiment, a network-connected computing device associated with the display screen 60 receives profile messages from the profilers 40 and 50 and operates the display screen 60 to display profile data acquired by the profilers 40 and 50. In some embodiments, the computing device associated with the display screen 60 implements one or more middleware nodes (e.g., ROS nodes), as was similarly described with respect to the robot nodes 20.

[0054] The Fig. 5A and Fig. Figure 5B shows exemplary graphical user interfaces, or user interfaces, displayed on the display screen 60 to present profile data acquired in the mobile robot system 10. In one example, the computing device associated with the display screen 60 implements Foxglove Studio to provide a graphical user interface for viewing and interacting with the profile data. Top left in Fig. Figure 5A shows an image 500, which was captured by a mobile robot 120 of the mobile robot system 10 and with respect to which object recognition is to be performed. Top right in Fig. Figure 5A shows a data plot 520 comparing an object recognition frame rate 522 of the mobile robot 120 with an object recognition frame rate 524 of the cloud system 150. (Lower left in) Fig. Figure 5A shows image 500 with boundary fields 530 that identify 150 objects captured by the cloud system. (Lower right in) Fig. Figure 5A shows image 500 with boundary fields 540 that identify objects detected by the mobile robot 120. (Above in) Fig. Figure 5B shows a data graph 550 comparing the CPU utilization 552 of mobile robot 120, the CPU utilization 554 of cloud system 150, and the GPU utilization 556 of cloud system 150. Below in Fig. Figure 5B shows a data drawing 560 that compares a number of megabytes of data 562 sent by the mobile robot 120 with a number of megabytes of data 564 received by the mobile robot 120.

[0055] Alternatively or additionally to displaying the profile data on the display screen 60, the system 10 is configured in some embodiments to store the profile data in one or more log files, e.g., in memory 156 of the cloud system 150 or in memory 124 of the mobile robots 120. In some embodiments, the system 10 is configured to upload the profile data to another remote server and / or to transfer the profile data to a web server for viewing from a remote device.

[0056] Returning to Fig. 3. If the system determines not to outsource the computation task, the procedure 200 proceeds with performing the computation task locally with the mobile robot (block 230). In particular, in response to a determination that the computation task should not be outsourced, the processor 122 determines the output of the computation task. In some embodiments, the processor 122 executes program instructions from corresponding robot nodes 20 to determine the output of the computation task. In one example, the computation task involves capturing an object in an image. Thus, the output of the computation task may, for example, include a bounding box around the object in the image and / or a classification of the object in the image. In another example, the computation task involves determining a position or orientation of the mobile robot 120 in the environment.In another example, the computational task involves determining a trajectory for the mobile robot to navigate through its environment. In yet another example, the computational task involves generating output commands for the actuators 128 to instruct the mobile robot 120 to perform a task within its environment.

[0057] Otherwise, if the system determines to outsource the computation task, the procedure 200 continues with the execution of the computation task remotely with the cloud system (block 240). Specifically, the processor 122 operates the network communication module 130 to transmit an outsourcing message to the cloud system 150, the outsourcing message comprising the input data and a specification of the computation task to be performed with respect to the input data. In at least some embodiments, the processor 122 executes program instructions from corresponding robot nodes 20 to transmit the outsourcing message in the form of a middleware message (e.g., a ROS message). The processor 154 of the cloud system 150 operates the network communication module 160 to receive the message from the mobile robot 120. Next, the processor 154 determines the output of the computation task.In some embodiments, the processor 154 executes program instructions from a corresponding robot cloud node 30 or from corresponding robot cloud nodes 30 to determine the output of the computation task. Finally, the processor 154 operates the network communication module 160 to transmit a computation output message to the mobile robot 120. In some embodiments, the processor 154 executes program instructions from a corresponding robot cloud node 30 or from corresponding robot cloud nodes 30 to transmit the computation output message in the form of a middleware message (e.g., a ROS message).

[0058] In some embodiments, the cloud system 150 includes a gateway for communication with mobile robots 120 that can only communicate via manufacturer-specific message formats instead of using middleware messages (e.g. ROS messages). Fig. Figure 6 shows an embodiment of the mobile robot system 10, which includes a gateway 500. The gateway 500 acts as a hardware abstraction layer, translating messages between vendor-specific (or robot-type-specific) message formats from at least some of the mobile robots 120 and the middleware message format used by the cloud system 150. Specifically, the mobile robots 120 send messages to the cloud system 150, for example, to offload a computational task, as explained above. The messages are sent in a vendor-specific or robot-type-specific message format, not in the middleware message format (e.g., ROS). The cloud system 150 receives the messages from the mobile robots 120, and the processor 154 executes program instructions from the gateway 500 to convert the received messages into middleware messages using the middleware message format.Conversely, the processor 154 executes program instructions from the robot cloud node 30 to generate middleware messages with the middleware message format. Next, the processor 154 executes program instructions from the gateway 500 to convert the middleware messages into messages with the manufacturer-specific or robot-type-specific message format. In this way, mobile robots 120 from different manufacturers can offload computations without requiring updates to the executable files 510 on the mobile robots 120 to specifically adapt to the middleware architecture used by the rest of the mobile robot system 10.

[0059] In at least some embodiments, the cloud system 150 is configured to perform the computational task on behalf of the mobile robot 120 using a virtual robot (or "digital twin") that resides in the cloud system 150 and communicates with the real mobile robot 120. The virtual robot is a digital representation of the hardware, memory, and / or software of the mobile robot 120. In the example of Fig. 1. The robot cloud nodes 30 can be considered virtual robots because they contain a copy of part of the software of the real mobile robot 120. The processor 154 executes a copy of the program instructions of the real mobile robot 120 to determine the output of a computation task.

[0060] Furthermore, the virtual robots can go beyond mere copies of a portion of the software of the real mobile robot 120 and can also include virtual hardware models and real-time state information that characterizes the current state of the real mobile robot 120. Accordingly, in at least some embodiments, the processor 154 determines the output of a computational task based on the input data, the current state of the mobile robot 120, and / or a virtual hardware model of the mobile robot 120. In particular, in some embodiments, the processor 154 of the cloud system 150 simulates the real mobile robot 120 using a virtual hardware model. The hardware of the virtual robot can be updated in response to real-world changes to the robot's hardware.In some embodiments, the processor 154 manipulates the virtual hardware models in a physics-based simulation to predict the outcome of hypothetical actions performed by the real robot or its environment. For example, the processor 154 can use the physics-based simulation in a motion planning task to determine a suitable trajectory for the mobile robot 120. The simulation enables the safe exploration of various actions to achieve a desired result.

[0061] In some embodiments, in addition to simply outsourcing computational tasks that could otherwise be performed by the mobile robot 120, the cloud system 150 is set up to provide services that go beyond the functionality of the mobile robot 120 alone. Fig.Figure 7 shows an embodiment of the mobile robot system 10 that includes cloud-based services 600. In particular, the cloud system 150 includes a service abstraction layer that hosts a variety of cloud-based services 600 that can be used by the mobile robots 120. Such cloud-based services 600 can include, for example, human-in-the-loop remote control (which allows a human operator to connect to the cloud to manually operate the mobile robot), object recognition, 2D / 3D mapping, global motion planning, grasp planning and detection, simultaneous localization and mapping (SLAM), or any other service.

[0062] It should be noted that, at least in some embodiments, the virtual robots 610 and / or the robot cloud nodes 30 perform computational tasks in a substantially similar manner to how the mobile robot 120 would have performed the computational task locally, i.e., because they include copies of the same program instructions used by the mobile robot 120 to perform these computational tasks. In contrast, however, the cloud-based services 600 can perform equivalent computational tasks using different techniques that may provide higher quality or different results. In this way, the cloud-based services 600 enable the addition and continuous improvement of services available to the mobile robots 120 without requiring software updates on the mobile robots 120 themselves.

[0063] As explained above, in some embodiments, the cloud system 150 receives a service request message from the mobile robot 120, which includes input data and specifies a requested service. The processor 154 receives the service request message from the mobile robot 120 and converts it into a format suitable for the requested cloud-based service 600. Finally, the processor 154 operates the network communication module 160 to transmit a service output message to the mobile robot 120, which includes an output of the requested cloud-based service 600.

[0064] Embodiments within the scope of the disclosure may also include non-volatile computer-readable storage media or machine-readable media on which computer-executable instructions (also referred to as program instructions) or data structures are stored or are stored. Such non-volatile computer-readable storage media or machine-readable media may be any available media accessible to a general-purpose or specialized computer. By way of example, and not as a limitation, such non-volatile computer-readable storage media or machine-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to receive, transmit, or store desired program code means in the form of computer-executable instructions or data structures.Combinations of the above-mentioned elements should also fall within the scope of non-volatile computer-readable storage media or machine-readable media.

[0065] Computer-executable instructions include, for example, instructions and data that cause a general-purpose computer, a specialized computer, or a specialized processing device to perform a particular function or group of functions. Computer-executable instructions also include program modules that are executed by computers in standalone or networked environments. In general, program modules include routines, programs, objects, components, and data structures, etc., that perform specific tasks or implement or realize certain abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of the program code means for performing steps of the procedures disclosed herein.The specific sequence of such executable instructions or associated data structures represents examples of corresponding actions for implementing / realizing the functions described in such steps.

[0066] Although the disclosure has been illustrated and described in detail in the drawings and the preceding description, it should nevertheless be regarded as illustrative and not limiting. It is understood that only preferred embodiments have been presented and that all changes, modifications, and further applications that are within the scope of the disclosure are to be protected.

Claims

[1] Method for operating a mobile robot comprising the method: Receiving input data with which a computational task is to be performed, with a local processing system of the mobile robot; Determine whether the computation task should be outsourced to a remote processing system; in response to a request to outsource the computation task to the remote processing system, (i) transmitting a message to the remote processing system using a transceiver of the mobile robot, the message comprising the input data, and (ii) receiving an output of the computation task from the remote processing system using the transceiver; In response to a determination that the computation task should not be outsourced to the remote processing system, determining the output of the computation task using the local processing system; and Operating the mobile robot with the local processing system based on the output of the computational task. [2] Method according to claim 1, wherein the input data is a user input that identifies a task to be performed by the mobile robot. [3] Method according to claim 1, wherein the input data are sensor data measured by a sensor of the mobile robot. [4] Method according to claim 3, wherein the sensor data is an image of an environment captured by a camera of the mobile robot. [5] Method according to claim 4, wherein the computational task is to detect an object in the image. [6] Method according to claim 3, wherein the computational task is to locate at least one of a position and orientation of the mobile robot in the environment. [7] Method according to claim 1, wherein the computational task is to determine a trajectory by which the mobile robot is to navigate through the environment. [8] The method of claim 1, wherein determining whether the computation task should be outsourced further comprises: Acquisition of profile data that characterizes (i) the execution of the computation task by the local processing system and (ii) the execution of the computation task by the remote processing system; and Determine whether the calculation task should be outsourced, based on the profile data. [9] The method of claim 8, wherein the profile data includes at least one of the latency times, frame rates, computational resource consumption and computational costs. [10] The method of claim 8, wherein determining whether the computation task should be outsourced further comprises: Determine whether the computational task should be outsourced, based on at least one optimization criterion. [11] Method according to claim 10, wherein the optimization criteria include at least one of minimizing the computation time, minimizing the computation costs and maximizing the battery life of the mobile robot. [12] Method according to claim 8, further comprising: Displaying profile data on a display screen. [13] The method of claim 1, wherein determining whether the computation task should be outsourced further comprises: Determine whether the computation task should be outsourced, based on whether the computation task involves performing a task that requires coordination between the mobile robot and at least one other mobile robot. [14] Method according to claim 1, wherein: The mobile robot has an initial memory that stores initial program instructions which are executed by the local processing system to perform a variety of processes; the remote computer has a second memory that stores second program instructions which are executed by the remote processing system to carry out the multitude of processes on behalf of the mobile robot, wherein the second program instructions comprise a copy of at least part of the first program instructions; and The remote processing system determines the output of the calculation task by executing the second program instructions based on the input data. [15] Method according to claim 14, wherein both the first program instructions and the second program instructions implement the plurality of processes using a middleware software library. [16] Method according to claim 1, wherein: the second program instructions further comprise additional program instructions corresponding to second services that differ from first services implemented by the mobile robot's first program instructions; and The remote processing system determines the output of the computation task partly by executing the additional program instructions that correspond to the second services. [17] Method according to claim 1, further comprising, in response to a determination that the computation task should be outsourced to the remote processing system: Determining the output of the computational task partially with the remote processing system by simulating the hardware of the mobile robot using a virtual robot model. [18] Method according to claim 1, further comprising, in response to a determination that the computation task should be outsourced to the remote processing system: Determining the output of the computation task with the remote processing system based on a state of the mobile robot. [19] Method according to claim 1, in response to a determination that the computation task should be outsourced to the remote processing system: Converting the format of the message received by the mobile robot with the remote processing system into a format that corresponds to a middleware software library implemented by the remote computer. [20] Method for remotely performing calculations for a mobile robot, comprising the method: Storing secondary program instructions, which comprise a copy of primary program instructions used by the mobile robot to perform a computational task, in the memory of a remote computer; Receiving a message from a mobile robot with a remote processing system of the remote computer, wherein the message includes input data and specifies the computational task to be performed with respect to the input data; Determining an output of the computation task with the remote processing system by executing the second program instructions; and Transferring the output of the calculation task to the mobile robot.

Images