Service robot recovery from a blocked path

Abstract

A computer-implemented method and apparatus for restoring movement of a service robot, the method including detecting a first obstacle on a first path of the service robot to a destination, performing an orientation task to align a camera of the service robot with the first obstacle, capturing an image of the first obstacle, processing the image to identify an obstacle type of the first obstacle, and generating a help-request communication based on the first obstacle being identified as a human obstacle type.

Description

[Technical Field]

[0001] This application claims benefit of the filing date of U.S. Patent Application No. 63 / 385,856, filed December 2, 2022, the disclosure of which is incorporated herein by reference in its entirety.

[0002] The disclosed subject matter relates generally to the technical field of mobile robots and delivery systems, and in one specific example, to a solution for recovering movement of a mobile robot from a blocked path. [Background technology]

[0003] Mobile service robots operate in many environments, from commercial and hospitality facilities (e.g., shops, restaurants, and hotels) to healthcare facilities, warehouses, and conference centers. Service robots perform a wide range of tasks, including food and beverage delivery, sanitation tasks, and customer service functions such as concierge service and automated valet parking. To perform these tasks in a timely manner, mobile service robots must efficiently navigate potentially complex environments, where static obstacles such as walls and stairs, as well as dynamic obstacles such as people, tables, or chairs, are all present. Such navigation solutions can be fully autonomous or involve communication between the robot and humans. [Brief explanation of the drawings]

[0004] To easily identify the discussion of any particular element or operation, the most significant digit or digits of a reference number indicate the figure number in which that element is first introduced.

[0005] [Figure 1] FIG. 1 is a perspective view of a service robot that may be deployed in place according to some examples.

[0006] [Figure 2] FIG. 1 is a block diagram illustrating perspectives of components and modules of a service robot according to some examples.

[0007] [Figure 3] FIG. 1 is a block diagram illustrating perspectives of components and modules of a service robot according to some examples.

[0008] [Figure 4] FIG. 1 is a block diagram illustrating perspectives of components and modules of a service robot according to some examples.

[0009] [Figure 5] FIG. 1 is a block diagram illustrating perspectives of components and modules of a service robot according to some examples.

[0010] [Figure 6] 1 is a flowchart illustrating aspects of a mobile recovery method performed by components and modules of a service robot, according to some examples.

[0011] [Figure 7] 1 is a schematic diagram of a portion of a mobile recovery of a service robot, according to some examples.

[0012] [Figure 8] FIG. 1 is a schematic diagram of a portion of an example mobile recovery of a service robot, according to some examples.

[0013] [Figure 9] 1 is a schematic diagram of a portion of a mobile recovery of a service robot, according to some examples.

[0014] [Figure 10] 1 is a flowchart illustrating aspects of a robot orientation method performed by components and modules of a service robot, according to some examples.

[0015] [Figure 11]1 is a flowchart illustrating aspects of a robot orientation method performed by components and modules of a service robot, according to some examples.

[0016] [Figure 12] 1 is a flowchart illustrating aspects of an obstacle location method performed by components and modules of a service robot, according to some examples.

[0017] [Figure 13] 1 is a flowchart illustrating aspects of an obstacle location method performed by components and modules of a service robot, according to some examples.

[0018] [Figure 14] 1 is an illustration of an obstacle location method performed by components and modules of a service robot, according to some examples.

[0019] [Figure 15] FIG. 1 is a schematic diagram of an example mobile recovery module behavior of a service robot, according to some examples.

[0020] [Figure 16] 16 is a flowchart illustrating aspects of some example methods such as those in FIG. 15.

[0021] [Figure 17] 16 is a flowchart illustrating method aspects for some examples such as in FIG. 15.

[0022] [Figure 18] 10 is a flowchart illustrating additional methods implemented by the mobility recovery module, according to some examples.

[0023] [Figure 19]FIG. 1 is a block diagram illustrating a model system according to some examples that operates to generate and maintain image localization models deployed on various service robots at one or more locations.

[0024] [Figure 20] FIG. 1 is a block diagram illustrating a software architecture in which, by some examples, the present disclosure may be embodied.

[0025] [Figure 21] FIG. 1 is a schematic diagram of a machine in the form of a computer system upon which a set of instructions may be executed to cause the machine to perform any one or more of the methodologies discussed herein, by way of some examples.

[0026] [Figure 22] 1 is a schematic diagram of some example processing environments.

[0027] [Figure 23] FIG. 1 is a schematic diagram of an environment in which multiple service robots are deployed in individual locations or environments, such as a cafeteria, a hospital, or a nursing home. DETAILED DESCRIPTION OF THE INVENTION

[0028] Mobile service robots operate in many environments, from commercial and hospitality facilities to healthcare facilities, warehouses, and conference centers. Service robots perform tasks such as food and beverage and goods delivery, sanitation tasks, and customer service tasks such as concierge service and automated valet parking. To complete these tasks in a timely manner, mobile service robots must efficiently navigate potentially complex environments, where static obstacles such as walls and stairs, as well as dynamic obstacles such as people, tables, or chairs, are all present. Such navigation solutions can be fully autonomous or involve communication between the robot and humans.

[0029] Examples disclosed herein relate to methods and apparatus for restoring the movement of a service robot. An exemplary method detects a first obstacle on the service robot's path to a destination. The method performs an orientation operation to align the service robot's camera with the first obstacle and captures an image of the first obstacle. The image is processed, for example, using CPU calculations, to identify the obstacle type of the first obstacle. If the first obstacle is determined to be a PERSON obstacle, the method generates a call-for-assistance communication. This communication may be a voice request that the first obstacle be removed from the service robot's path of movement. Processing the image captured by the service robot camera may include performing a depth estimation calculation to identify the detected obstacle as a person. Obstacle detection by the service robot may be performed using a service robot LiDAR. Aligning the service robot's camera with the obstacle may be performed by rotating the service robot to an obstacle-aligned position using a yaw angle orientation based on the service robot's current position.

[0030] In some examples, after detecting an obstructed first path of travel, the service robot can begin moving along a second path of travel. The first path of travel can be calculated using a global cost map. The second path of travel can be calculated using a local cost map. Once the second path of travel is identified as obstructed, the service robot camera can be aligned with the first obstacle as follows: the global cost map is subtracted from the local cost map to generate a subtracted cost map, and the first obstacle is identified as the closest point from the subtracted cost map.

[0031] In some examples, the service robot may detect multiple obstacles in its path to the destination and / or perform an orientation task to align its camera with the multiple obstacles. The service robot may capture images containing multiple obstacles and process the captured images to identify one or more types of obstacles. If at least one obstacle is determined to be anthropomorphic, the service robot may generate a communication requesting assistance in removing the obstacle from its path to the destination.

[0032] 1 is a diagram of a service robot 104 that may be deployed within a location 2302, such as a cafeteria or a nursing home, according to some examples. The service robot 104 has a housing 102 that houses various components and modules, including a wheeled transport system that allows the service robot 104 to propel itself autonomously within the service location 2302. Navigation and perception systems are also housed within the housing 102. The housing 102 supports a number of trays 106 that can support and carry plates and other utensils to be delivered to a kitchen within the location 2302 and from the kitchen within the location 2302 to tables.

[0033] The service robot 104 includes multiple sensors, including external sensors for capturing information about the environment and location in which the service robot 104 operates, and self-sensing sensors for capturing information about the service robot 104 itself. Examples of external sensors include vision sensors (e.g., two-dimensional (2D), three-dimensional (3D), depth, and RGB cameras), light sensors, acoustic sensors (e.g., microphones or ultrasonic sensors), proximity sensors (e.g., infrared (IR) transceivers, ultrasonic sensors, photoresistors), tactile sensors, temperature sensors, and navigation and orientation sensors (e.g., GPS sensors). Visual odometry and visual simultaneous localization and mapping (SLAM) can assist the service robot 104 in navigating both indoor and outdoor environments where lighting conditions can be maintained appropriately. 3D cameras, depth, and stereo vision cameras provide pose (e.g., position and orientation) information.

[0034] The service robot 104 may have a limited number of cameras (e.g., 1-4), including a single forward-facing camera (e.g., robot camera 108). This configuration allows for a manageable amount of image data to be captured and processed, and allows image processing to be performed efficiently or even entirely locally (e.g., using CPU computation). In some examples, the service robot may have a high-resolution forward-facing camera and a smaller number of lower-resolution rear-facing cameras. When activated, the forward-facing camera can capture "upward" images, which correspond to images taken while the camera is pointing upward toward an obstacle. The camera can capture "downward" images, which correspond to images taken while the camera is pointing downward toward an obstacle. The camera can capture "intermediate" images, which are images taken while the camera is level with the bottom. A service robot using forward-facing cameras can recognize and classify objects in its environment when the forward-facing camera is oriented so that it is directly aligned with the target object or set of objects.

[0035] Alignment of the front camera with a target object can be achieved by rotating the body of the service robot 104 (e.g., by controlling the motors of the wheels of the service robot 104) to align the robotic camera 108. The robotic camera 108 itself can be moved in multiple directions within or relative to the body of the service robot 104 to achieve or assist in alignment with a target object or set of objects. For example, the robotic camera 108 can be rotatably mounted in a socket or housing fixed to the body of the service robot 104 and controlled by an electromechanical mechanism to rotate in multiple directions.

[0036] Examples of self-sensing sensors include inertial sensors (e.g., tilt and acceleration), accelerometers, gyroscopes, magnetometers, compasses, wheel encoders, and temperature sensors. An inertial measurement unit (IMU) within the service robot 104 can include multiple accelerometers and gyroscopes as well as magnetometers and barometers. The instantaneous attitude (e.g., position and orientation), velocity (linear velocity, angular velocity), acceleration (linear acceleration, angular acceleration), and other parameters of the service robot 104 can be obtained through the IMU.

[0037] 2 is a block diagram illustrating a component and module perspective of some example service robots 104. The service robots 104 include a robotics open platform 202, a perception stack 204, and a robotics controller 230.

[0038] Robotics Open Platform 202 is ●Device API206 ● Diagnostic API208 Robotics API210 ●Data API212 and ●Flock API214 It provides a number of application program interfaces (APIs), including:

[0039] The recognition stack 204 includes: ●Recognition 216 ●P2P operation 218 ●Semantic Operation 320 ●Sensor correction 222 ●Sensor processing 224 ● Obstacle Avoidance 226 It includes components that support:

[0040] The ROS navigation stack 228 also forms part of the recognition stack 204 .

[0041] The robotics controller 230 ●Power management 232 ●Wireless charging 234 ●Device Interface 236 ●Motor Control 238 It includes components that support:

[0042] 3 is a block diagram illustrating another view of the components and modules of a service robot 104 according to some examples. The service robot 104 includes a robotics stack 302 and an application stack 306. The robotics stack 302 in turn includes a navigation stack 304 and a perception stack 204. The application stack 306 provides telemetry 308 and login 310 services for the service robot 104.

[0043] 4 is a block diagram illustrating yet another view of the components and modules of the service robot 104, according to some examples. The perception stack 204 is shown as including an object detector 402, an object detector 404, and an object detector 406. The navigation stack 304 is shown as including a motion recovery stack 408, which is shown as including example modules such as a motion recovery module 410, a motion recovery module 412, and a motion recovery module 414. In some examples, the navigation stack 304 can include a ROS navigation stack.

[0044] Robot Movement Recovery: An Example

[0045] 5 is a block diagram illustrating another perspective on the components and modules of the service robot 104 (see FIG. 6 for an associated flowchart, and FIGS. 7, 8, and 9 for additional illustrations). The service robot 104 uses an object detector 506 (e.g., part of the perception stack 204) to detect objects in the robot's path toward its destination. When a stuck robot condition is detected, the service robot 104 uses a motion recovery module 508 (e.g., part of the navigation stack 304) to recover its path toward its destination.

[0046] The mobility recovery module 508 includes: a robot orientation module 504, which may include an obstacle location module 510; and ●Communication module 512 Includes:

[0047] The movement recovery module 508 operates to perform movement recovery of the service robot 104, for example, in situations where movement of the service robot 104 from a starting point to a destination becomes impeded or difficult. To this end, the movement recovery module 508 includes an obstacle location module 510 for determining the location of an obstacle blocking the path of the service robot 104 (see FIGS. 12 and 13 for exemplary methods). The robot orientation module 504 orients a selected camera (e.g., the robot camera 108) of the service robot 104 toward an individual obstacle (or multiple obstacles). The robot camera may be a forward-facing camera. The service robot 104 captures images of the individual obstacle (or multiple obstacles), which are processed by the object detector 506 to identify the individual obstacle type(s). If the object detector 506 deems the obstacle anthropomorphic (e.g., identifies the obstacle as a human), the communication module 512 generates a call-for-assistance communication (see at least FIGS. 7, 8, and 9).

[0048] Object and object type detection

[0049] The service robot 104 uses an object detector 506 to identify the service robot 104 and detect objects on its current path of travel. The object detector 506 can use vision sensors (e.g., two-dimensional (2D), three-dimensional (3D), depth, and RGB cameras), optical sensors, acoustic sensors (e.g., microphones or ultrasonic sensors), proximity sensors (e.g., infrared (IR) transceivers, ultrasonic sensors, photoresistors), tactile sensors, temperature sensors, navigation and orientation sensors (e.g., GPS sensors), visual odometry, and visual simultaneous localization and mapping (SLAM).

[0050] The object detector 506 processes images captured by the robotic cameras 108, e.g., forward-facing cameras. For service robots 104 using forward-facing cameras with limited viewing angles, aligning all of the robotic cameras 108 to face potential obstacles and positioning the robotic cameras 108 according to their distance from the potential obstacles improves object and object type detection performance. The object detector 506 can better identify objects and object types when the robotic cameras 108 are substantially aligned with the potential obstacles, rather than being partially aligned with them or facing away from them. Accordingly, the robotic cameras 108 may be aligned to the extent necessary to face the potential obstacles sufficiently to capture images of the potential obstacles with sufficient range and clarity to be usable for object recognition purposes.

[0051] The performance of the object detector 506 improves when processing images captured by the robotic camera 108 that is not too far or too close to an obstacle (e.g., minimum distance 0.5 m and maximum distance 1.0 m). The captured images can be "up," "forward," or "downward" images, with "upward" images being useful for detecting anthropomorphic obstacles, while "downward" and "forward" images can be useful for detecting other obstacle types.

[0052] The object detector 506 uses a pre-trained object recognition and object type identification model (e.g., a TensorFlow model identical to one in the TensorFlow 2 Detection Model Zoo, an OpenCV model, or a Detectron2 model) to label and identify objects in the processed images. The object detector 506 executes the pre-trained model using CPU computations and processes data locally, which offers several advantages. First, local processing using CPU computations avoids the potential time-consuming process of transmitting data to a server, enabling faster object detection and more robust obstacle avoidance, which is important in exemplary service environments where dynamic obstacles exist (e.g., cafeterias) and care environments (e.g., hospitals) where collisions are costly. Second, local processing using CPU computations allows the service robot 104 to operate even when the network is overloaded (e.g., in a crowded conference or event center) or when a network connection is absent. Third, processing data locally enhances the protection of personal information, which is important in care environments (e.g., hospitals) and select service environments (e.g., hotels).

[0053] The object detector 506 analyzes the detected bounding boxes around the detected objects with the most likely identified object types. Depending on the model used by the object detector 506, each bounding box and each identified object type is accompanied by a calculated score that indicates the likelihood that the detected object is an object of that type.

[0054] The object detector 506 can use an adjustable threshold to filter out or isolate objects that have a low score for their identified type (eg, when the object detector 506 is unsure of the object type).

[0055] The object detector 506 performs depth estimation while processing the camera images to determine if the path is actually blocked. The service robot 104 uses a depth camera to capture depth camera images that are processed by the object detector 506.

[0056] According to some examples, the object detector 506 uses the identified bounding box dimensions for the detected obstacle to calculate a distance metric for the service robot and the potential obstacle. The distance metric corresponds to an estimate of the distance between the service robot 104 and the potential obstacle. The distance metric may be a binary value corresponding to whether the potential obstacle is close enough to the robot to block the robot's path. Depth estimates from depth camera images may be used to calculate the distance metric. In some examples, a standard size for each object type may be used to calculate the distance metric. A focal length measurement of the robot camera may be used to calculate the distance metric. The distance metric calculation may be implemented using a rule, a rule set, and / or a machine learning model.

[0057] Obstacle-oriented robot orientation

[0058] After the service robot 104 detects an object on its path to the destination, it uses the motion recovery module 508 to recover the path to the destination. To capture better images of the obstacle and ensure improved performance of the image-based object and object type detector, the robot orientation module 504 orients the robot so that the robotic camera 108 is aligned with the obstacle blocking the path to the destination. The robot orientation module 504 employs an obstacle location module 510 to determine the location of the primary (e.g., closest) obstacle blocking the path. In some examples, the robot orientation module 504 orients the robotic camera 108 based on the location of the discovered primary (e.g., closest) blocking obstacle (or obstacles) relative to the service robot 104. In some examples, the module orients the robotic camera 108 based on the current position of the service robot 104. FIGS. 12 and 13 illustrate a method for obstacle location, and FIGS. 10 and 11 describe additional operations of the robot orientation module 504.

[0059] Mobility restoration through communication between robots and humans

[0060] Once the service robot's 104 camera is aligned with the main obstacle (or multiple obstacles) on the path to the destination, the service robot 104 captures an obstacle image, and the object detector 506 processes the image to determine the corresponding object type.

[0061] When the mobile recovery module detects at least one humanoid obstacle, the communication module 512 calls for assistance. The assistance-related communication may be an audio communication and / or a visual communication (e.g., a visual signal, an LED signal, etc.) generated through the speaker service of the service robot.

[0062] An exemplary communication is a request to move an obstacle from the path of the service robot 104. The obstacle to be moved can be an inanimate object such as a backpack, bag, purse, laptop, suitcase, chair, table, cart, trolley, plant, ladder, etc. The obstacle to be moved can also be another service robot, a person (e.g., a child), or an animal (e.g., a companion animal such as a dog or cat).

[0063] The communication may be a request to a first obstacle (e.g., a person) requesting the first obstacle to move out of the path of the service robot. For example, the communication may be to an obstacle whose identified type is a person. The communication may involve an implicit request to have the obstacle move out of the path of the service robot (e.g., the communication may refer to "Excuse me," "May I pass?", "Can you let me through?", etc.). The communication may also be an explicit request to have the obstacle move out of the path of the robot (e.g., "Can you help me move / remove this <obstacle>?", "Can you help me with this <obstacle>?", where the obstacle corresponds to one of the obstacle types listed above).

[0064] FIG. 6 is a flow chart illustrating another aspect of some example methods 600 performed by parts and components of a service robot such as those described in FIG.

[0065] At operation 602, the object detector 506 detects a first obstacle on a first path to a destination of the service robot 104. At operation 604, the robot orientation module 504 performs an orientation operation to align the robot camera 108 of the service robot 104 with the first obstacle. At operation 606, the mobile recovery module 508 captures an image of the first obstacle. At operation 608, the object detector 506 processes the image to identify an obstacle type of the first obstacle. At operation 610, if the first obstacle is identified as a human obstacle type, the communication module 512 generates a help-call communication.

[0066] 7, 8, and 9 are schematic diagrams of a portion of an example navigational recovery for the service robot 104. In this example, the robot detects an obstacle blocking its path to a destination point, and the service robot 104 classifies the detected obstacle as a person and sends an "excuse me" communication to the person, after which the person unblocks the path and the robot can continue to the destination point. FIG. 7 depicts a portion of an example navigational recovery for the service robot 104 illustrating the service robot 104 detecting an obstacle blocking its path to destination point D. FIG. 8 depicts a portion of an example navigational recovery illustrating the results of the following operations: The robotic camera 108 of the service robot 104, which was initially not aligned with the obstacle 706, is reoriented and aligned with the obstacle 706. The captured image of the obstacle 706 is processed to classify the obstacle as a "person" type. Upon detecting a "person" type obstacle, the service robot 104 requests assistance from the person blocking the path in the form of an "excuse me" communication. 9 depicts a portion of an exemplary movement recovery illustrating an exemplary result of the service robot's communication with the person. The person clears the way, allowing the service robot to follow the path to destination point D.

[0067] 10 is a flowchart illustrating a method 1000 performed by the robot orientation module 504, according to some examples. In operation 1002, the robot orientation module determines that the service robot is in a first orientation in which its camera is not aligned with a first obstacle. In operation 1004, the robot orientation module performs an orientation operation to reorient the service robot from the first orientation to a second orientation in which the service robot's camera is aligned with the first obstacle.

[0068] 11 is a flowchart illustrating a method 1100 performed by the robot orientation module 504, according to some examples. In operation 1102, the robot orientation module calculates a yaw angle orientation based on at least the current position of the service robot. In operation 1104, the robot orientation module 504 rotates the service robot to a position where a camera of the first robot is aligned with a first obstacle. According to some examples, the yaw angle orientation calculation uses the position of the first obstacle. According to some examples, the operations in method 1100 are performed as part of performing the orientation operation in operation 1004.

[0069] Obstacle Location: Methods

[0070] 12 and 13 are flowcharts illustrating obstacle location methods used by the example obstacle location module 510 of FIG. 5. FIG. 12 illustrates an example "Point-on-Path" method, and FIG. 13 illustrates an example "Closest Point in the Subtracted Cost Map" method. In some examples, the obstacle location module 510 uses a combination of these two methods. In some examples, the obstacle location module 510 receives location information (e.g., example potential obstacles send coordinates). Both obstacle detection methods can use a global cost map, a current local cost map, and / or a global movement plan.

[0071] Cost Map

[0072] In some examples, the cost map is a grid-based map that stores information about obstacles in a given environment (e.g., a service environment). Each cell in the grid contains a cost value that indicates whether the cell is occupied by an obstacle and to what extent (and thus the degree to which the robot can or cannot pass through the cell on an example path to a given destination). This value may be a categorical score or a numeric score. A minimum cost threshold may be applied to the numeric cost value to filter noise generated during construction of the cost map, ensuring that only important or highly reliable obstacles are reflected in the cost map.

[0073] Global Cost Map

[0074] According to some examples, a global cost map (see, for example, FIG. 14) is a pre-computed cost map that stores information about static obstacles (e.g., walls, stairs, pillars, fixed equipment) that already exist in the environment. The global cost map for a given environment may be pre-computed based on at least recognition information about the environment and provided by a third party. The global cost map for a given environment may be pre-computed by starting with a pre-computed cost map for an environment with a similar layout and performing a series of update operations to accurately reflect the target environment.

[0075] Local Cost Map

[0076] According to some examples, a local cost map (e.g., see FIG. 14) is a cost map that is dynamically calculated at the time of operation and reflects obstacles in the service environment at that time. An exemplary local cost map reflects obstacles located in a variable-sized area around the robot. According to some examples, an advantage of a local cost map is that it provides partial information about changes in environmental obstacles that have occurred since the global cost map was generated. For example, a local cost map can capture dynamic obstacles (e.g., people who are standing or moving, chairs or tables that have been moved, accessories such as bags, carriers, or purses, movable equipment such as trolleys or carts, etc.).

[0077] Subtracted Cost Map

[0078] In some examples, the subtracted cost map is a grid-based cost map, where the subtracted cost value for a given cell is calculated by subtracting the value of the corresponding cell in the global cost map from the value of the corresponding cell in the local cost map. The obstacle location module 510 can calculate the subtracted cost map to distinguish between dynamic obstacles (captured only by the local cost map) and static obstacles (reflected in both the local and global cost maps).

[0079] Route planning

[0080] According to some examples, a global travel plan for the service robot 104 may be calculated by a travel planner taking into account the origin, destination, and global map. According to some examples, a localized travel plan may be calculated by a travel planner (e.g., teb_local_planner in the ROS navigation stack 228) taking into account the origin, destination, current local cost map, and the global travel plan.

[0081] Detecting obstacle locations on the route

[0082] FIG. 12 is a flowchart illustrating a "point-on-path" obstacle location method according to some examples.

[0083] In operation 1202, the obstacle localization module 510 of the service robot 104 begins operation at a predetermined location along a first movement path toward the robot's destination. The first movement path may be a global movement plan calculated by a global planner module of the navigation stack 304 based on a global cost map. The global movement plan may be calculated by the ROS navigation stack (e.g., using one of the global planner classes in the ROS navigation library, such as global_planner, navfn, or carot_planner). The global cost map and global movement plan are provided to the obstacle localization module 510 by the movement recovery module 508. The obstacle localization module 510 may have access to the current local cost map of the service robot 104.

[0084] In operation 1204, the obstacle location module 510 searches the first movement path for a point closest to the predetermined starting location such that the cost value of the corresponding grid cell in the local cost map is greater than a predetermined minimum-cost threshold. In some examples, the obstacle location module 510 first applies a clustering operation, the output of which is a set of point clusters, to attempt to identify small obstacles and noise. The obstacle location module 510 can identify a set of "spurious" clusters based on a predetermined metric, such as a minimum number of cluster points. The obstacle location module 510 can identify a set of N most salient clusters, where N is an adjustable parameter. The number of points in each cluster is used as a saliency metric. In some examples, the obstacle location module 510 searches for a point closest to the predetermined starting location, which is part of a non-spurious cluster, and which is within a predetermined maximum distance from the first movement path. For example, the obstacle location module 510 searches for a point closest to a predetermined start location, where the point is part of the top N significant clusters and is within a predetermined maximum distance from the first movement path. The predetermined start location may be the current location of the service robot. In some examples, the closest point is selected to be within a predetermined (e.g., minimum, maximum) distance range from the predetermined start location. In some examples, the minimum distance is 0.5 m and the maximum distance is 1.0 m.

[0085] In operation 1206, the obstacle location module 510 returns the closest identified point to the service robot 104 as the closest obstacle.

[0086] "Closest point in subtraction map" obstacle detection

[0087] FIG. 13 is a flowchart illustrating a "closest point in subtracted cost map" obstacle location method according to some examples.

[0088] In operation 1302, the obstacle location module 510 subtracts each global cost map cost value from the corresponding local cost map cost value to generate a subtraction cost map. In operation 1304, the obstacle location module 510 identifies the point in the subtraction cost map that is closest to the current position of the service robot and returns this as the closest obstacle to the service robot (see, for example, FIG. 14).

[0089] FIG. 14 illustrates an example of a nearest point in a subtraction map method for obstacle location, according to some examples. The three panels illustrate a global cost map (applying a minimum cost threshold of 50 in this example), a local cost map, and individual subtraction cost maps. The subtraction cost map is calculated by subtracting each global cost map value from the corresponding local cost map value. A minimum cost threshold of 250 is also applied in this example. The black dot "R" indicates the service robot 104. "D" indicates the robot's destination, and the dotted line indicates the robot's path. The large, patterned circle in the local cost map panel indicates the location of the "main" dynamic obstacle blocking the robot's path. As can be seen from the subtraction map panel and the corresponding small, patterned circle, the location of this obstacle is accurately recovered as the point in the subtraction map that is closest to the robot and has a cost value higher than the minimum cost threshold.

[0090] Robot Movement Recovery: Additional Examples

[0091] Mobile restoration for multiple blocked paths

[0092] FIG. 15 illustrates additional exemplary behaviors of the mobile recovery module 508, according to some example implementations. In the top left panel, the service robot 104 detects a first obstacle 1506 on a first path toward destination "D." In response to detecting this first obstacle, the mobile recovery module 508 initiates a second path toward the destination (see the top right panel). The service robot 104 then detects a second obstacle 1502 on the second path toward the destination (see the bottom left panel). For example, as the service robot 104 moves along the second path that bypasses the detected obstacle, it may encounter another obstacle (e.g., a static object such as a wall or door) while attempting to pass through a narrow space. In response to detecting the second obstacle, the robot's mobile recovery module 508 uses the obstacle location module 510 to determine the location of the first obstacle 1506, and the mobile recovery module 508 then uses the robot orientation module to orient the robot's camera so that the robot's camera is aligned with the first obstacle 1506 (see the bottom right panel). The service robot 104 then captures an image of the first obstacle 1506, and in response to the object detector identifying it as a humanoid obstacle, the mobile recovery module generates an assistance-call communication using the communication module (e.g., transmitting a message to the person such as "Excuse me," "May I pass?", etc.). In some examples, if the object detector identifies the first obstacle as another service robot, the mobile recovery module 508 generates an assistance-call communication using the communication module.

[0093] Figure 16 is a flowchart illustrating a method 1600 according to some examples such as that in Figure 15. At operation 1602, in response to detecting a first obstacle in a first path of the service robot 104, the service robot 104 begins movement on a second path of movement toward a destination. At operation 1604, the service robot 104 detects a second obstacle in the second path of movement of the service robot 104. At operation 1606, in response to detecting the second obstacle, the service robot 104 performs an orientation operation to align the service robot's camera with the first obstacle.

[0094] Figure 17 is a flowchart illustrating an additional method 1700 related to some examples such as that in Figure 15. At operation 1702, a first movement path is calculated based on a global cost map. At step 1704, a second movement path is calculated based on the local cost maps. At operation 1706, a subtracted cost map is calculated by subtracting the global cost map from the lowest cost map. At step 1708, a first obstacle is identified as the closest point from the subtracted cost map.

[0095] Mobile recovery for multiple obstacles by route

[0096] 18 is a flowchart illustrating an additional method 1800 embodied by a mobile recovery module, according to some examples. In operation 1802, the mobile recovery module of the service robot 104, upon activation, detects multiple obstacles in a first path of travel of the service robot to a destination. In operation 1804, an orientation operation is performed to align the camera of the service robot with the multiple obstacles.

[0097] The method includes capturing an image including a plurality of obstacles using a camera of the service robot in operation 1806. The method includes processing the image to identify an individual obstacle type for each of the plurality of obstacles based on identifying at least one of the plurality of obstacles as a human obstacle type in operation 1808.

[0098] In operation 1810, the service robot generates a communication requesting assistance in removing at least one of a number of obstacles from a first path of movement. The obstacle to be removed may be an inanimate object such as a backpack, bag, wallet, laptop, suitcase, chair, table, cart, trolley, plant, ladder, etc. The obstacle to be removed may be another service robot. The obstacle to be removed may be a person (e.g., a child) or an animal (e.g., a companion animal such as a dog or cat). This communication is a request to the obstacle, requesting that the obstacle move out of the path of the service robot. This communication is targeted to an obstacle whose identified type is a person.

[0099] The communication may involve an implicit request to have an obstacle moved out of the service robot's path (e.g., the communication may refer to "Excuse me," "May I pass?", "Can you let me through?", etc.). The communication may also be an explicit request to have an obstacle moved out of the robot's path (e.g., "Can you help me move / remove this <obstacle>?", "Can you help me with this <obstacle>?", where the obstacle corresponds to one of the obstacle types listed above).

[0100] Additional Mobile Recovery Examples

[0101] In some examples, the service robot 104 uses multiple mobile recovery modules. In some examples, different mobile recovery modules can be used in a sequential (sequential) manner. In some examples, the mobile recovery behaviors can overlap.

[0102] In some examples, the service robot 104 uses a "cost map cleanup" recovery module (not shown) or behavior (e.g., the "clear_costmap_recovery" behavior in the ROS navigation stack 228). This module can be useful when a dynamic obstacle erroneously remains in the local cost map after the service robot has passed through it. Cleaning up the local cost map can include replacing values ​​of cells in the robot's local cost map with values ​​of corresponding cells in a pre-computed global cost map. In some examples, the cells whose values ​​are replaced are located outside a regular rectangle having sides of predetermined lengths and centered at the service robot's location.

[0103] In some examples, the service robot 104 uses a "panning recovery" module (not shown) or behavior. One goal of the panning recovery behavior is to determine whether the robot can actually follow the path to the destination. The local cost map may be noisy, and the robot may attempt to avoid obstacles that are not present or that do not block the path. In some examples, the service robot 104 first cleans up the local cost map using a "cost map cleanup" recovery module. In some examples, the "panning recovery" behavior involves the service robot turning left and / or right. In some examples, this behavior involves recalculating or obtaining a subset of the cell values ​​in the current local cost map to reflect the current potential obstacle. In some examples, the service robot considers the local cost map to determine whether the path to the destination is actually blocked.

[0104] Service Robots: An Overview

[0105] 19 is a block diagram illustrating a model system 1902 that, by way of some examples, operates to generate and maintain image position models 1904 that are deployed on various service robots 104 at one or more locations 2302. The model system 1902 can include the following components or modules: ● Data collection and preparation module 1906; ● Model training and evaluation module 1908; ● Model placement module 1910; and ●Model Refresh Module 1912

[0106] Further details regarding the operation of such an exemplary module are provided below.

[0107] 20 is a block diagram 2000 illustrating a software architecture 2004 that may be implemented in any one or more of the devices described herein. The software architecture 2004 is supported by hardware, such as a machine 2002 that includes a processor 2020, memory 2026, and / or I / O components 2038. In this example, the software architecture 2004 may be conceptualized as a stack of layers, with each layer providing a specific function in some examples. The software architecture 2004 includes layers such as an operating system 2012, libraries 2010, frameworks 2008, and applications 2006. Operationally, the applications 2006 invoke API calls 2050 through the software stack and receive messages 2052 in response to the API calls 2050.

[0108] The operating system 2012 manages hardware resources and provides common services. The operating system 2012 includes, for example, a kernel 2014, services 2016, and drivers 2022. The kernel 2014 acts as an abstraction layer between the hardware and different software layers. For example, the kernel 2014 provides memory management, processor management (e.g., scheduling), component management, networking, and security configuration, among other functions. Services 2016 may provide other common services for other software layers. Drivers 2022 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 2022 may be a display driver, a camera driver, a BLUETOOTH® or BLUETOOTH® Low These may include an Energy driver, a flash memory driver, a serial communication driver (e.g., a Universal Serial Bus (USB) driver), a WI-FI driver, an audio driver, and a power management driver.

[0109] Libraries 2010 provide low-level common infrastructure used by applications 2006. Libraries 2010 may include system libraries 2018 (e.g., the C standard library) that provide functionality such as memory allocation functions, string manipulation functions, mathematical functions, etc. The library 2010 may also include API libraries 2024 such as a media library (e.g., a library for supporting the display and manipulation of various media formats such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (AVC) or H.264, Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG) or JPG, or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework used to render graphical content on a display in two dimensions (2D) and three dimensions (3D)), a database library (e.g., SQLite for providing various relational database functions), a web library (e.g., Web Kit for providing web browsing functions), etc. The library 2010 may also include various other libraries 2028 for providing various other APIs to the application 2006.

[0110] The framework 2008 provides a high-level common infrastructure used by the applications 2006. For example, the framework 2008 provides a variety of graphical user interface (GUI) features, high-level resource management, and high-level location services. The framework 2008 may, in some instances, provide a wide range of other APIs that may be used by the applications 2006, some of which may be specific to a particular operating system or platform.

[0111] The applications 2006 may include a wide variety of other applications, such as a home application 2036, a contacts application 2030, a browser application 2032, a book reader application 2034, a location application 2042, a media application 2044, a messaging application 2046, a game application 2048, and third party applications 2040. The applications 1406 are programs that perform programmatically defined functions. In some instances, various programming languages ​​structured in various ways, such as object-oriented programming languages ​​(e.g., Objective-C, Java, or C++) or procedural programming languages ​​(e.g., C or assembly language), may be used to generate one or more of the applications 2006. In certain instances, third party applications 2040 (e.g., applications developed for Android by entities other than the vendor of a particular platform) may be used. TM or IOS TM Applications developed using the Software Development Kit (SDK) are compatible with iOS TM , ANDROID TM , WINDOWS Phone, or other mobile operating system. In this example, the third party application 2040 can invoke API calls 2050 provided by the operating system 2012 to implement the functionality described herein.

[0112] FIG. 21 is a schematic diagram of a machine 2100 on which instructions 2110 (e.g., software, programs, applications, applets, apps, or other executable code) may be executed to cause the machine 2100 to perform any one or more of the methodologies discussed herein. For example, the instructions 2110 may cause the machine 2100 to perform any one or more of the methods described herein. The instructions 2110 transform a general, unprogrammed machine 2100 into a specific machine 2100 that is programmed to perform the functions described and illustrated in the manner described. The machine 2100 may operate as a stand-alone device or may be coupled to other machines (e.g., networked). In a networked deployment, the machine 2100 may operate in the capacity of a server machine or a client machine in a server-client network environment or as a peer machine in a peer-to-peer (or distributed) network environment. Machine 2100 may include (but is not limited to) a server computer, client computer, personal computer (PC), tablet computer, laptop, netbook, set-top box (STB), entertainment media system, cellular phone, smartphone, mobile device, wearable device (e.g., smart watch), smart home device (e.g., smart appliance), other smart device, web appliance, network router, network switch, network bridge, or any machine capable of sequentially or differently executing instructions 2110 that specify operations to be performed by machine 2100. Also, although a single machine 2100 is illustrated, the term "machine" can include a collection of machines that individually or collectively execute instructions 2110 to perform any one or more of the methodologies discussed herein.

[0113] Machine 2100 may include a processor 2104, memory 2106, and I / O components 2102, which may be configured to communicate via a bus 2140. In some examples, processor 2104 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a radio-frequency integrated circuit (RFIC), other processor, or any suitable combination thereof) may include, for example, processor 2108 and processor 2112 that execute instructions 2110. The term "processor" is intended to include multi-core processors that may include two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously. Although FIG. 21 illustrates multiple processors 2104, machine 2100 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0114] The memory 2106 includes a main memory 2114, a static memory 2116, and a storage unit 2118, all accessible to the processor 2104 via the bus 2140. The main memory 2106, the static memory 2116, and the storage unit 2118 store instructions 2110 that embody any one or more of the methodologies or functions described herein. The instructions 2110 may also reside, wholly or partially, within the main memory 2114, within the static memory 2116, within the storage unit 2118, within the machine-readable medium 2120, within the processor 2104 (e.g., within the processor's cache memory), or any suitable combination thereof during execution by the machine 2100.

[0115] The I / O components 2102 may include various components for receiving input, providing output, generating output, transmitting information, exchanging information, or capturing measurements. The specific I / O components 2102 included in a particular machine vary depending on the type of machine. For example, a portable machine such as a cell phone may include a touch input device or other such input mechanism, while a headless server machine likely does not include such a touch input device. The I / O components 2102 may include many other components not shown in FIG. 21 . In various examples, the I / O components 2102 may include an output component 2126 and an input component 2128. The output component 2126 may include a visual component (e.g., a display such as a plasma display panel (PDP), a light-emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), an audio component (e.g., a speaker), a tactile component (e.g., a vibration motor, a resistance mechanism), or other signal generator. The input components 2128 may include alphanumeric input components (e.g., a keyboard, a touchscreen configured to receive alphanumeric input, a photo-optical keyboard, or other alphanumeric input component), point-based input components (e.g., a mouse, touchpad, trackball, joystick, motion sensor, or other pointing mechanism), tactile input components (e.g., physical buttons, a touchscreen that provides the position and / or force of a touch or touch gesture, or other tactile input component), audio input components (e.g., a microphone), etc.

[0116] In further examples, the I / O component 2102 can include a biometric component 2130, a motion component 2132, an environmental component 2134, or a position component 2136, among various other components. For example, the biometric component 2130 includes components that detect facial expressions (e.g., hand expressions, facial expressions, vocal expressions, body movements, or eye tracking), measure biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), or identify people (e.g., voice identification, retinal identification, facial identification, fingerprint identification, or brainwave-based identification). The motion component 2132 includes an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, and a rotation sensor component (e.g., a gyroscope). The environmental components 2134 include, for example, one or more cameras, illuminance sensor components (e.g., a photometer), temperature sensor components (e.g., one or more thermometers for detecting ambient temperature), humidity sensor components, pressure sensor components (e.g., a barometer), acoustic sensor components (e.g., one or more microphones for detecting background noise), proximity sensor components (e.g., an infrared sensor for detecting surrounding objects), gas sensors (e.g., a gas detection sensor for detecting concentrations of harmful gases or measuring airborne pollutants for safety purposes), or other components capable of providing indicators, measurements, or signals corresponding to the surrounding physical environment. The position components 2136 include location sensor components (e.g., a GPS receiver component), altitude sensor components (e.g., an altimeter or barometer for detecting air pressure from which altitude can be derived), orientation sensor components (e.g., a magnetometer), etc.

[0117] Communications may be implemented using a variety of technologies. The I / O component 2102 further includes a communications component 2138 operable to couple the machine 2100 to the network 2122 or the device 2124 through a respective coupling or connection. For example, the communications component 2138 may include a network interface component or other suitable device for interfacing with the network 2122. In additional examples, the communications component 2138 may include a wired communications component, a wireless communications component, a cellular communications component, a Near Field Communication (NFC) component, a BLUETOOTH® component (e.g., BLUETOOTH® Low Energy), a flash memory driver, a serial communications driver (e.g., a Universal Serial Bus (USB) driver), a Wi-Fi® component, and other communications components for providing communications through other modalities. The device 2124 may be another machine or any of a variety of peripheral devices (e.g., a peripheral device coupled via USB).

[0118] The communication component 2138 can also include a component operable to detect or detect an identifier. For example, the communication component 2138 can include a Radio Frequency Identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional barcodes such as Universal Product Code (UPC) barcodes, multidimensional barcodes such as Quick Response (QR) code, Aztec code, Data Matrix, Data glyph, Maxi Code, PDF417, Ultra Code, UCC RSS-2D barcodes, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying tagged audio signals). A variety of information can also be derived via the communication component 2138, such as location through Internet Protocol (IP) geolocation, location through Wi-Fi signal triangulation, or location through detection of NFC beacon signals that can indicate a specific location.

[0119] Various memories (e.g., main memory 2114, static memory 2116, and / or memory of processor 2104) and / or storage unit 2118 can store one or more sets of instructions and data structures (e.g., software) implemented or used in any one or more of the methodologies or functions described herein. When executed by processor 2104, such instructions (e.g., instructions 2110) cause various operations to be performed to implement the disclosed embodiments.

[0120] The command 2110 may be transmitted or received over the network 2122 using a transmission medium through a network interface device (e.g., a network interface component included in the communications component 2138) and using any one of a variety of well-known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, the command 2110 may be transmitted or received over the network 2122 using a transmission medium through a connection (e.g., a peer-to-peer connection) to the device 2124.

[0121] Referring to FIG. 22, a schematic diagram of a processing environment 2200 is shown including a processor 2202, a processor 2206, and a processor 2208 (eg, a GPU, a CPU, or a combination thereof).

[0122] The processor 2202 is coupled to a power supply 2204 and is shown as including the following modules (permanently configured or temporarily instantiated): a data collection and preparation module 1906, a model training and evaluation module 1908, a model deployment module 1910, and a model refresh module 1912.

[0123] 23 is a schematic diagram of an environment in which multiple service robots 104 (e.g., a swarm of service robots) are deployed in individual locations 2302 or environments, such as a cafeteria, a hospital, or a nursing home. Depending on the location, the service robots 104 may perform any of a variety of functions within the location 2302. For example, if such a location 2302 is a service location, such as a cafeteria, the service robots 104 may be operated to deliver items from the kitchen to tables within a particular cafeteria and assist in transporting dishes, trash, etc. from the tables back to the kitchen.

[0124] Each service robot 104 is communicatively coupled by a network 2304 or multiple networks 2304 to a cloud service 2308 residing on one or more server systems 2306 .

[0125] Example

[0126] Example 1 is a computer-implemented method for restoring the movement of a service robot, including the steps of detecting a first obstacle on a first movement path of the service robot to a destination; performing an orientation task to align a camera of the service robot with the first obstacle; capturing an image of the first obstacle; processing the image to identify an obstacle type of the first obstacle; and generating an assistance request communication based on the first obstacle being identified as a human obstacle type.

[0127] In Example 2, in Example 1, the communication is a request that a first obstacle be removed from a first travel path.

[0128] In Example 3, in Example 2, the communication is a request to the first obstacle requesting the first obstacle to move from the first movement path.

[0129] In Example 4, in Examples 2 and 3, the communication is an audio communication generated by a speaker system of the service robot.

[0130] In Example 5, in Examples 1 to 4, the method further includes a step of determining that the service robot is in a first orientation in which the camera of the service robot is not aligned with the first obstacle, and a step of performing an orientation operation in response to determining that the service robot is in the first orientation, to reorient the service robot from the first orientation to a second orientation in which the camera of the service robot is aligned with the first obstacle.

[0131] In Example 6, in Examples 1 to 5, the step of performing the orientation task includes the steps of calculating a yaw angle orientation based on the current position of the service robot; and rotating the service robot to a position where the camera of the service robot is aligned with the first obstacle.

[0132] In Example 7, in Examples 1 to 6, the method further includes the steps of: initiating movement on a second movement path to a destination in response to detecting a first obstacle on the first movement path of the service robot; detecting a second obstacle on the second movement path of the service robot; and performing an orientation task to align a camera of the service robot with the first obstacle in response to detecting the second obstacle.

[0133] In Example 8, in Example 7, the first movement path is calculated using a global cost map and the second movement path is calculated using a local cost map, and performing the orientation task includes subtracting the global cost map from the local cost map to generate a subtraction cost map; and identifying the first obstacle as the closest point in the subtraction cost map.

[0134] In Example 9, the steps of processing the image in Examples 1 to 8 are performed using CPU calculations.

[0135] In Example 10, in Examples 1 to 9, processing the image includes performing depth estimation to identify the first obstacle as a person.

[0136] In Example 11, in Examples 1 to 10, the step of detecting the first obstacle is performed using a lidar of the service robot.

[0137] In Example 12, in Examples 1 to 11, the method further includes detecting a plurality of obstacles on a first movement path of the service robot to a destination; performing an orientation task to align a camera of the service robot with the plurality of obstacles; capturing an image using the camera of the service robot to include the plurality of obstacles; processing the image to identify individual obstacle types for each of the plurality of obstacles based on identifying at least one of the plurality of obstacles as a human obstacle type; and generating an assistance request communication for removing at least one of the plurality of obstacles from the first movement path.

[0138] In Example 13, in Examples 5 to 12, performing the orientation task further includes subtracting the global cost map from the local cost map to generate a subtraction cost map.

[0139] In Example 14, in Example 13, the method further includes identifying the first obstacle as the closest point in the subtracted cost map.

[0140] In Example 15, in Examples 5 to 14, the step of performing the orientation task further includes the step of identifying the first obstacle as the nearest obstacle along the first moving path.

[0141] In Example 16, in Example 15, the step of identifying the nearest obstacle along the first movement path includes the steps of starting from a predetermined position along the first movement path, searching for a nearest point along the first movement path whose individual cost in the local cost map is greater than a predetermined minimum cost threshold value; and returning the nearest point as the nearest obstacle.

[0142] In example 17, in example 16, the predetermined position according to the first movement path is the current position of the service robot.

[0143] In example 18, in examples 16 and 17, the closest point along the first travel path is further selected to be within a predetermined distance from the predetermined location.

[0144] Example 19 is at least one machine-readable medium containing instructions that, when executed by a processing circuit, cause the processing circuit to perform operations to implement any of Examples 1 through 18.

[0145] Example 20 is a device including means for realizing any one of Examples 1 to 18.

[0146] Example 21 is a system that embodies any one of Examples 1 to 18.

[0147] Glossary

[0148] "Carrier signal" refers to any intangible medium capable of storing, encoding, or transmitting instructions for execution by a machine, including digital or analog communication signals or other intangible media for facilitating communication of such instructions. Instructions may be transmitted or received over a network using a transmission medium through a network interface device.

[0149] A "communications network" refers to one or more portions of a network, which may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular network, a wireless network, a Wi-Fi network, or other type of network, or a combination of two or more such networks. For example, a network or portion of a network may include a wireless or cellular network, and the connection may be a Code Division Multiple Access (CDMA) connection, a Global System for Mobile communications (GSM) connection, or other type of cellular or wireless connection.In this example, the combination may embody any of various types of data transmission technologies, such as 1xRTT (Single Carrier Radio Transmission Technology), EVDO (Evolution-Data Optimized) technology, GPRS (General Packet Radio Service) technology, EDGE (Enhanced Data rates for GSM Evolution) technology, 3GPP (third Generation Partnership Project) including 3G, fourth generation (4G) wireless networks, UMTS (Universal Mobile Telecommunications System), HSPA (High-Speed ​​Packet Access), WiMAX (Worldwide Interoperability for Microwave Access), LTE (Long Term Evolution) standards, other technologies defined by various standards-setting organizations, other long-distance protocols, or other data transmission technologies.

[0150] A "component" refers to a device, physical entity, or logic with boundaries defined by functions or subroutine calls, branch points, APIs, or other techniques that provide division or modularization of specific processing or control functions. A component can be coupled to other components through interfaces to perform machine processes. A component may be a packaged functional hardware unit designed for use with other components, typically a part of a program that performs a specific function of related functionality. A component may be a software component (e.g., code embodied on a machine-readable medium) or a hardware component. A "hardware component" is a tangible unit capable of performing a specific task, and may be configured or arranged in a specific physical manner. For example, one or more computer systems (e.g., a stand-alone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or group of processors) may be configured as hardware components that operate in conjunction with software (e.g., an application or application portion) to perform specific tasks described herein. A hardware component may also be embodied mechanically, electronically, or any suitable combination thereof. For example, a hardware component may include dedicated circuitry or logic permanently configured to perform a specific task. A hardware component may be a special-purpose processor such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). A hardware component may also include programmable logic or circuitry temporarily configured by software to perform a specific task. For example, a hardware component may include software running on a general-purpose processor or other programmable processor.When configured by such software, a hardware component becomes a specific machine (or a specific component of a machine) customized to perform a configured function, and is no longer a general-purpose processor. Whether a hardware component is embodied mechanically, with permanently configured dedicated circuitry, or with temporarily configured circuitry (e.g., configured by software) can be determined by cost and time considerations. Thus, the phrase "hardware component" (or "hardware-embodied component") should be understood to include a tangible entity that is physically configured, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a particular manner or perform a particular task described herein. When considering examples in which a hardware component is temporarily configured (e.g., programmed), the hardware component need not be configured or instantiated at any one time. For example, if a hardware component includes a general-purpose processor configured by software to be a special-purpose processor, the general-purpose processor may be configured at a different time as a different special-purpose processor (e.g., including different hardware components). Accordingly, software may configure a particular processor or processors to, for example, be particular hardware components in one instance and other hardware components in another instance. Hardware components may provide information to and receive information from other hardware components. Accordingly, the described hardware components may be considered to be communicatively coupled. Where multiple hardware components are simultaneously present, communication may occur through signal transmission between two or more of the hardware components (e.g., through appropriate circuits and buses). In instances where multiple hardware components are configured or instantiated at different times, communication between such hardware components may occur, for example, through storing and retrieving information in memory structures to which the multiple hardware components have access.For example, a hardware component may perform a task and store the output of that task in a memory device to which it is communicatively coupled. Additional hardware components may subsequently access the memory device to retrieve and process the stored output. A hardware component may initiate communication with input or output devices or operate on resources (e.g., collect information). Various tasks of the exemplary methods described herein may be performed, at least in part, by one or more processors that are temporarily or permanently configured (e.g., by software) to perform the associated tasks. Whether temporarily or permanently configured, such processors may constitute processor implementations that operate to perform one or more tasks or functions described herein. As used herein, the term "processor implementation" refers to a hardware component implemented using one or more processors. Similarly, the methods described herein may be implemented, at least in part, by a processor, and a particular processor or processor implementations may be an example of hardware. For example, at least a portion of the tasks of the methods described herein may be performed by one or more processors or processor implementations. Additionally, one or more processors may operate to facilitate performance of related tasks in a "cloud computing" environment or "Software as a Service (SaaS)." For example, at least a portion of the tasks may be performed by a group of computers (examples of which include the processors), and such tasks may be accessed through a network (e.g., the Internet) and one or more appropriate interfaces (e.g., APIs). Performance of particular tasks may be distributed among processors, and may reside within a single machine or may be located across multiple machines. In some examples, the processor, or an embodiment of the processor, may be located in a single geographic location (e.g., in a home environment, an office environment, or a server farm). In some examples, the processor, or an embodiment of the processor, may be distributed across multiple geographic locations.

[0151] "Computer-readable medium" refers to all machine storage media and transmission media. Thus, the term includes all storage devices / media and carrier / modulated data signals. The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" mean the same thing and may be used interchangeably in this disclosure.

[0152] A "machine storage medium" refers to a single or multiple storage devices and / or media (e.g., centralized or distributed databases and / or associated caches and servers) that store executable instructions, routines, and / or data. This term includes solid-state memory, including memory internal or external to a processor, as well as optical and magnetic media. Specific examples of machine storage media, computer storage media, and / or device storage media include, for example, semiconductor memory devices (e.g., erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGA, and flash memory devices), magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and non-volatile memory including CD-ROM and DVD-ROM disks. The terms "machine storage medium," "device storage medium," and "computer storage medium" have the same meaning and may be used interchangeably in this disclosure. The terms "machine storage medium," "computer storage medium," and "device storage medium" specifically exclude carrier waves, modulated data signals, and other such media, some of which are encompassed by the term "signal media."

[0153] A "module" refers to logic with boundaries defined by functions or subroutine calls, branch points, application program interfaces (APIs), or other techniques that provide division or modularization of specific processing or control functions. Modules are typically coupled to other modules through interfaces to perform machine processes. A module is a packaged functional hardware unit designed for use with other components and may be a portion of a program that generally performs specific functions of related functionality. A module may comprise a software module (e.g., code embodied on a machine-readable medium) or a hardware module. A "hardware module" is a tangible unit capable of performing specific tasks, and may be configured or arranged in a specific physical manner. In various examples, one or more computer systems (e.g., a stand-alone computer system, a client computer system, or a server computer system) or one or more hardware modules (e.g., a processor or group of processors) of a computer system may be configured as a hardware module that operates in response to software (e.g., an application or application portion) to perform specific tasks described herein. In some examples, a hardware module may be embodied mechanically, electronically, or any suitable combination thereof. For example, a hardware module may include dedicated circuitry or logic permanently configured to perform a particular task. For example, a hardware module may be a special-purpose processor such as an FPGA or ASIC. A hardware module may also include programmable logic or circuitry temporarily configured by software to perform a particular task. For example, a hardware module may include software executed by a general-purpose processor or other programmable processor. When configured by such software, the hardware module becomes a particular machine (or a particular component of a machine) uniquely customized to perform the function for which it is configured, and is no longer a general-purpose processor.It will be appreciated that whether a hardware module is implemented mechanically, by permanently configured dedicated circuitry, or by temporarily configured circuitry (e.g., configured by software) is determined by cost and time considerations. Therefore, the phrase "hardware module" (or "hardware implementation module") should be understood to include a tangible entity that is physically configured, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a particular manner or perform a particular task described herein. Considering an example in which hardware modules are temporarily configured (e.g., programmed), each hardware module need not be configured or instantiated at any one time. For example, if a hardware module includes a general-purpose processor configured by software to be a special-purpose processor, the general-purpose processor may be configured as different special-purpose processors at different times (e.g., configured with different hardware modules). Accordingly, software configures a particular processor or processors to, for example, be a particular hardware module at one instance and another hardware module at another instance. Hardware modules can provide information to and receive information from other hardware modules. Accordingly, the described hardware modules may be considered to be communicatively coupled. When multiple hardware modules are simultaneously present, communication may occur through signal transmission between two or more of the hardware modules (e.g., through appropriate circuits and buses). In instances where multiple hardware modules are configured or instantiated at different times, communication between such hardware modules may occur, for example, through storing and retrieving information in memory structures to which the multiple hardware modules have access. For example, one hardware module may perform an operation and store the output of that operation in a memory device to which it is communicatively coupled.Additional hardware modules can subsequently access the memory device to retrieve and process the stored output. Hardware modules may initiate communication with input or output devices or operate on resources (e.g., collecting information). Various operations of the exemplary methods and routines described herein may be performed, at least in part, by one or more processors that are temporarily or permanently configured (e.g., by software) to perform the associated operations. Whether temporarily or permanently configured, such processors may constitute processor implementation modules that operate to perform one or more operations or functions described herein. As used herein, a "processor implementation module" refers to a hardware module implemented using one or more processors. Similarly, methods described herein may be implemented, at least in part, by a processor, with a particular processor or processor being an example of hardware. For example, at least a portion of the operations of a method may be performed by one or more processors or processor implementation modules. Additionally, one or more processors may operate to facilitate performance of the associated operations in a "cloud computing" environment or as a "software as a service (SaaS)." For example, at least a portion of the operations may be performed by a group of computers (e.g., machines including processors), and such operations may be accessed through a network (e.g., the Internet) and one or more appropriate interfaces (e.g., APIs). Performance of particular operations may be distributed among processors, and may reside within a single machine or may be located across multiple machines. In some examples, the processor or processor implementation may be located in a single geographic location (e.g., in a home environment, an office environment, or a server farm). In other examples, the processor or processor implementation may be distributed across multiple geographic locations.

[0154] A "processor" refers to any circuit or virtual circuit (i.e., a physical circuit emulated by logic running on an actual processor) that manipulates data values ​​in response to control signals (e.g., "instructions," "op codes," "machine code," etc.) and generates corresponding output signals that are applied to machine operation. A processor may be, for example, a Central Processing Unit (CPU), a Reduced Instruction Set Computing (RISC) processor, a Complex Instruction Set Computing (CISC) processor, a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), a Radio-Frequency Integrated Circuit (RFIC), or any combination thereof. A processor may also be a multi-core processor having two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously.

[0155] "Signal medium" refers to any intangible medium capable of storing, encoding, or transmitting machine-executable instructions, including digital or analog communication signals or other intangible media that facilitate the communication of software or data. The term "signal medium" can include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a way as to encode information in the signal. The terms "transmission medium" and "signal medium" have the same meaning and may be used interchangeably in this disclosure.

Claims

1. 1. A computer-implemented method for restoring movement of a service robot, comprising: detecting a first obstacle on a first movement path of the service robot to a destination; performing an orientation operation to align a camera of the service robot with the first obstacle; capturing an image of the first obstacle; processing the image to identify an obstacle type of the first obstacle; and generating a call-for-assistance communication based on the first obstacle being identified as a human obstacle type.

2. The method of claim 1 , wherein the communication is a request to have the first obstacle removed from the first travel path.

3. The method of claim 2 , wherein the communication is a request to the first obstacle requesting the first obstacle to move from the first travel path.

4. The method of claim 2 , wherein the communication is an audio communication generated by a speaker system of the service robot.

5. determining that the service robot has a camera in a first orientation that is not aligned with the first obstacle; and 2. The method of claim 1, further comprising: in response to determining that the service robot is in the first orientation, performing an orientation task to reorient the service robot from the first orientation to a second orientation in which a camera of the service robot is aligned with the first obstacle.

6. The step of performing the orientation operation includes: Calculating a yaw angle orientation based on the current position of the service robot; and The method of claim 1 , comprising rotating the service robot to a position where a camera of the service robot is aligned with the first obstacle.

7. Initiating movement along a second path of movement to the destination in response to detecting the first obstacle along the first path of movement of the service robot; detecting a second obstacle in the second movement path of the service robot; and The method of claim 1 , further comprising, in response to detecting the second obstacle, performing an orientation task to align a camera of the service robot with the first obstacle.

8. The first movement path is calculated using a global cost map, and the second movement path is calculated using a local cost map; The step of performing the orientation operation includes: subtracting the global cost map from the local cost map to generate a subtracted cost map; and The method of claim 7 further comprising identifying the first obstacle as the closest point in the subtracted cost map.

9. The method of claim 1 , wherein the step of processing the image is performed using CPU computation.

10. The method of claim 1 , wherein processing the image includes performing depth estimation to identify the first obstacle as a person.

11. The method of claim 1 , wherein detecting the first obstacle is performed using a LiDAR (LiDAR) on the service robot.

12. detecting a plurality of obstacles along the first movement path of the service robot to a destination; performing an orientation task to align a camera of the service robot with the plurality of obstacles; capturing the image using a camera of the service robot to include the plurality of obstacles; processing the images to identify an individual obstacle type for each of the plurality of obstacles based on identifying at least one of the plurality of obstacles as a human obstacle type; and The method of claim 1 , further comprising generating a communication requesting assistance in removing at least one of the plurality of obstacles from the first path of travel.

13. The method of claim 5 , wherein performing the orientation task further comprises subtracting the global cost map from the local cost map to generate a subtracted cost map.

14. The method of claim 13 further comprising identifying the first obstacle as the closest point in the subtracted cost map.

15. 6. The method of claim 5, wherein the step of performing the orientation task further comprises the step of identifying the first obstacle as a nearest obstacle along the first path of travel.

16. The step of identifying the nearest obstacle along the first movement path includes: Starting from a predetermined location along the first movement path, searching for a closest point along the first movement path whose individual cost in a local cost map is greater than a predetermined minimum cost threshold; and The method of claim 15 further comprising returning the closest point as the closest obstacle.

17. The method of claim 16 , wherein the predetermined location along the first path of travel is the current location of the service robot.

18. 17. The method of claim 16, wherein the closest point along the first travel path is further selected to be within a predetermined distance from the predetermined location.

19. 1. A computing device comprising: a processor; and Includes memory, The memory includes: When executed by the processor, the apparatus: Detecting a first obstacle on a first movement path of the service robot to a destination; performing an orientation task to align a camera of the service robot with the first obstacle; capturing an image of the first obstacle; processing the image to identify an obstacle type of the first obstacle; A computing device storing instructions configured to generate a help-call communication based on the first obstacle being identified as a human obstacle type.

20. A non-transitory computer-readable storage medium, comprising: When executed by a computer, the computer: Detecting a first obstacle on a first movement path of the service robot to a destination; performing an orientation task to align a camera of the service robot with the first obstacle; capturing an image of the first obstacle; processing the image to identify an obstacle type of the first obstacle; A non-transitory computer-readable storage medium comprising instructions for generating a call-for-help communication based on the first obstacle being identified as a human obstacle type.

Images