Autonomous map traversal with waypoint matching

Abstract

A robot (100) includes a drive system configured to manipulate the robot around an environment (10) and data processing hardware (112) in communication with memory hardware (114) and the drive system. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving image data (124) of the robot maneuvering in the environment and executing at least one waypoint heuristic (212). The at least one waypoint heuristic is configured to trigger waypoint placement (200) on a waypoint map. In response to triggering waypoint placement, the operations include recording a waypoint (210) on the waypoint map, wherein the waypoint is associated with at least one waypoint edge (220) and includes sensor data obtained by the robot. The at least one waypoint edge includes a pose transformation that expresses how to move between two waypoints.

Description

Technical Field

[0001] The present disclosure relates to autonomous map traversal with waypoint matching. Background Art

[0002] A robot is generally defined as a reprogrammable and multifunctional manipulator designed to move materials, parts, tools, or specialized equipment through variable programmed movements to perform tasks. A robot can be a physically anchored manipulator (e.g., an industrial robotic arm), a mobile robot that moves throughout an environment (e.g., using legs, wheels, or traction-based mechanisms), or some combination of a manipulator and a mobile robot. Robots are used in a variety of industries, such as manufacturing, transportation, hazardous environments, exploration, and healthcare. As such, the ability to operate autonomously can enhance the functionality of a robot and provide additional benefits to such industries. Summary of the invention

[0003] One aspect of the present disclosure provides a method for autonomous map traversal. The method includes receiving sensor data of an environment surrounding a robot manipulating in the environment at data processing hardware. The sensor data includes image data. The method also includes executing, by the data processing hardware, at least one waypoint heuristic based on the image data. The at least one waypoint heuristic is configured to trigger waypoint placement on a waypoint map. In response to at least one waypoint heuristic triggering waypoint placement, the method includes recording, by the data processing hardware, a waypoint on the waypoint map. The waypoint is associated with at least one waypoint edge and includes at least some of the sensor data obtained by the robot. The at least one waypoint edge includes a pose transformation that expresses how to move between two waypoints.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the image data includes three-dimensional point cloud data captured by a three-dimensional stereo image sensor. Here, the three-dimensional stereo image sensor may include one or more of a stereo camera, a scanning light detection and ranging (LIDAR) sensor, or a scanning laser detection and ranging (LADAR) sensor. The waypoint heuristic may be configured to detect a threshold feature detection within the image data at the current position of the robot.

[0005] In some examples, the sensor data for the waypoint includes three-dimensional point cloud data and posture data captured by a three-dimensional stereo image sensor, the posture data including inertial measurement data measured by an inertial measurement unit (IMU) of the robot. For a given waypoint, the method may include identifying neighbor waypoints to the given waypoint, and identifying three-dimensional points of the three-dimensional point cloud data of the given waypoint and the neighbor waypoint as a three-dimensional point set. For a given waypoint, the method may also include determining whether a three-dimensional point in the three-dimensional point set corresponds to a visual edge of an image represented by the three-dimensional point cloud data, and associating a three-dimensional point in the three-dimensional point set that corresponds to a visual edge of the image represented by the three-dimensional point cloud data with the given waypoint.

[0006] In some configurations, the waypoint includes a robot pose constraint configured to enable the robot to achieve an oriented pose at the waypoint. The waypoint map can be locally consistent with neighbor waypoints. The waypoint edges can include annotations indicating spatial features of the environment. Recording the waypoint on the waypoint map includes recording a basin area around the waypoint, the basin area specifying an area adjacent to the waypoint in which the robot is configured to use an iterative closest point (ICP) algorithm.

[0007] Another aspect of the present disclosure provides a robot. The robot includes a drive system configured to manipulate the robot around an environment, and data processing hardware in communication with the drive system. The robot also includes memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving sensor data of the robot manipulating the environment. The sensor data includes image data. The operations also include executing at least one waypoint heuristic based on the image data. The at least one waypoint heuristic is configured to trigger waypoint placement on a waypoint map. In response to at least one waypoint heuristic triggering waypoint placement, the operations include recording waypoints on the waypoint map. The waypoints are associated with at least one waypoint edge and include sensor data obtained by the robot. At least one waypoint edge includes a posture transformation that expresses how to move between two waypoints.

[0008] Implementations of the present disclosure may include one or more of the following optional features. In some examples, the image data includes three-dimensional point cloud data captured by a three-dimensional volumetric image sensor. Here, the image sensor includes one or more of a stereo camera, a scanning light detection and ranging (LIDAR) sensor, or a scanning laser detection and ranging (LADAR) sensor. The waypoint heuristic can be configured to detect a threshold feature detection within the image data at the current position of the robot.

[0009] In some embodiments, the sensor data of the waypoint includes three-dimensional point cloud data and posture data captured by a three-dimensional stereo image sensor, the posture data including inertial measurement data measured by an inertial measurement unit (IMU) of the robot. For a given waypoint, the operation may include identifying neighbor waypoints to the given waypoint, and identifying three-dimensional points of the three-dimensional point cloud data of the given waypoint and the neighbor waypoint as a three-dimensional point set. For a given waypoint, the operation may also include determining whether a three-dimensional point in the three-dimensional point set corresponds to a visual edge of an image represented by the three-dimensional point cloud data, and associating a three-dimensional point in the three-dimensional point set corresponding to a visual edge of the image represented by the three-dimensional point cloud data with the given waypoint.

[0010] In some configurations, the waypoint includes a robot pose constraint configured to cause the robot to achieve an oriented pose at the waypoint. The waypoint map can be locally consistent with neighbor waypoints. Additionally or alternatively, the waypoint edges can include annotations indicating spatial features of the environment. Recording the waypoint on the waypoint map includes recording a basin area around the waypoint, the basin area specifying an area adjacent to the waypoint in which the robot is configured to use an iterative closest point (ICP) algorithm.

[0011] Another aspect of the present invention provides a method for autonomous map traversal. The method includes receiving image data of an environment surrounding a robot manipulating in the environment at data processing hardware. The method also includes executing an iterative closest point (ICP) algorithm by the data processing hardware, the iterative closest point (ICP) algorithm being configured to locate a first waypoint of a waypoint map based on the received image data. The method also includes executing a traversal path from the first waypoint of the waypoint map to a second waypoint of the waypoint map by the data processing hardware, and updating the location to the second waypoint of the waypoint map by the data processing hardware through the ICP algorithm based on the received image data.

[0012] Implementations of this aspect of the disclosure may include one or more of the following optional features. In some configurations, the image data includes three-dimensional point cloud data captured by a three-dimensional volumetric image sensor. The image sensor may include one or more of a stereo camera, a scanning light detection and ranging (LIDAR) sensor, or a scanning laser detection and ranging (LADAR) sensor.

[0013] In some examples, updating the position fix to the second waypoint includes comparing the received image data with stored image data of the waypoint map, the stored image data associated with the second waypoint. Additionally or alternatively, updating the position fix to the second waypoint may include determining that a location associated with the position fix satisfies a position fix distance threshold, and performing, by the data processing hardware, a position fix at the location based on the kinematic data and the inertial measurement data. Here, the position fix distance threshold indicates a distance from the corresponding waypoint.

[0014] Another aspect of the present disclosure provides a robot. The robot includes a body, legs coupled to the body and configured to manipulate the robot around an environment, and data processing hardware in communication with the legs. The robot also includes memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving image data of an environment surrounding the robot manipulating the environment. The operations also include executing an iterative closest point (ICP) algorithm that is configured to locate a first path point of a waypoint map based on the received image data, and to perform a traversal path from the first path point of the waypoint map to a second path point of the waypoint map. The operations also include updating the location to the second path point of the waypoint map by the ICP algorithm based on the received image data.

[0015] Implementations of this aspect of the disclosure may include one or more of the following optional features. In some implementations, the image data includes three-dimensional point cloud data captured by a three-dimensional volumetric image sensor. Here, the image sensor includes one or more of a stereo camera, a scanning light detection and ranging (LIDAR) sensor, or a scanning laser detection and ranging (LADAR) sensor.

[0016] In some examples, updating the position fix to the second waypoint includes comparing the received image data with stored image data of the waypoint map, the stored image data being associated with the second waypoint. Optionally, updating the position fix to the second waypoint may also include determining that a location associated with the position fix satisfies a position fix distance threshold, and performing, by the data processing hardware, a position fix at the location based on the kinematic data and the inertial measurement data. Here, the position fix distance threshold indicates a distance from the corresponding waypoint.

[0017] The details of one or more embodiments of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a schematic diagram of an example of a robot in a robotic environment.

[0019] Figure 2A is a schematic diagram of an example of a map for a robot.

[0020] Figure 2B is a schematic diagram of example waypoints for a map with a basin.

[0021] Figure 2C is a diagram of an example map with map constraints.

[0022] Figure 2D is a diagram of an example map for the robot.

[0023] Figure 3 and Figure 4 is a flow diagram of an example arrangement for implementing robot environment traversal by a robot.

[0024] Figure 5 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0025] Like reference numbers in the various drawings indicate like elements. DETAILED DESCRIPTION

[0026] Figure 1 is an example of a robot environment 10. The robot environment 10 generally refers to a spatial area associated with some type of terrain that can be traversed by the robot 100 according to the map 200. For example, the terrain is urban terrain (such as buildings, streets, sidewalks, parks, alleys, stairs, ramps, tunnels, etc.), rural terrain (such as roads, fields, hills, mountains, caves, etc.), underground terrain (such as caves, tunnels, etc.), or any combination thereof. The robot 100 includes motion-based structures, such as legs and / or wheels attached to the body 150, which enable the robot 100 to move around the robot environment 10. Although Figure 1 A quadruped robot having four legs 103 , 104 , 106 , 108 is depicted, but the robot 100 may include any number of legs or motion-based structures (eg, a bipedal or humanoid robot having two legs) that provide a means to traverse the robotic environment 10 .

[0027] In some implementations, the robot 100 includes computing hardware 110 and at least one sensor system 120. The computing hardware 110 generally includes data processing hardware 112 and memory hardware 114. The computing hardware 110 is configured to control a robot traversal system 116. The robot traversal system 116 operates a behavior system 102 of the robot 100 to move the robot 100 around the robot environment 10. The behavior system 102 is generally responsible for controlling (i.e., executing) the behavior of the robot 100. For example, the behavior system 102 controls different footstep patterns, leg patterns, body movement patterns, or vision system sensing patterns. The robot traversal system 116 operates the behavior system 102 based on at least one map 200 provided to the robot traversal system 116.

[0028] The traversal system 116 is configured to communicate with the memory hardware 114 of the robot 100 to provide the map 200 to operate the behavior system 102 using the data processing hardware 112 of the robot 100. In some examples, the memory hardware 114 stores the map 200 locally on the robot 100. In other examples, the map 200 is stored and / or accessed remotely by the traversal system 116. For example, the traversal system 116 communicates with a remote system 140 via a network 130. The remote system 140 can be a server or a cloud-based environment that includes remote resources 142 such as remote data processing hardware 144 and remote memory hardware 146. In some implementations, the map 200 is stored and / or processed on the remote system 140 using the remote resources 142 and communicated to the traversal system 116 of the robot 100 via the network 130. In other examples, different portions of the map 200 are processed and / or stored remotely (e.g., via the remote system 140) and locally (e.g., via the computing hardware 110).

[0029] The sensor system 120 includes one or more sensors 122, 122a-n. The sensors 122 may include visual / image sensors, inertial sensors (e.g., inertial measurement units (IMUs)), and / or kinematic sensors. Some examples of sensors 122 include cameras, such as stereo cameras, scanning light detection and ranging (LIDAR) sensors, or scanning laser detection and ranging (LADAR) sensors. In some examples, the sensors 122 have (one or more) corresponding fields of view F V , which defines a sensing range or area corresponding to the sensor 122. For example, Figure 1 The field of view F of the robot 100 is depicted V Each sensor 122 may be pivotable and / or rotatable so that the sensor 122 may change the field of view F, for example, about one or more axes (e.g., an x-axis, a y-axis, or a z-axis relative to a ground plane G). VIn some embodiments, the body 150 of the robot 100 includes a sensor system 120 having a plurality of sensors 122 surrounding the body 150 to collect sensor data 124 in all directions around the robot 100. For example, Figure 1 The quadruped robot depicted in FIG. 1 includes sensors 122 on its chest 122 a, sensors 122 on each side 122 n (only one side is depicted), and sensors 122 on its retractable hands 122 b. The robot 100 may include any number of sensors 122 as part of a sensor system 120 to generate sensor data 124 of the robot environment 10 surrounding the robot 100.

[0030] When the sensor 122 is used to survey the field of view F V When the sensor system 120 generates V Corresponding sensor data 124. In some examples, the sensor data 124 is image data corresponding to the three-dimensional point cloud generated by the three-dimensional stereo image sensor 122. Additionally or alternatively, when the robot 100 is maneuvering around the robot environment 10, the sensor system 120 collects posture data of the robot 100, the posture data including inertial measurement data (e.g., measured by an IMU). In some examples, the posture data includes kinematic data and / or orientation data about the robot 100.

[0031] The sensor data 124 collected by the sensor system 120 (such as image data, posture data, inertial data, kinematic data, etc. related to the robot environment 10) can be communicated to the computing hardware 110 (e.g., the data processing hardware 112 and the memory hardware 114) of the robot 100. In some examples, the sensor system 120 collects and stores the sensor data 124 (e.g., in the memory hardware 114 or the memory hardware 146 of the remote resource 142). In other examples, the sensor system 120 collects the sensor data 124 and processes the sensor data 124 in real time without storing the raw (i.e., unprocessed) sensor data 124. In other examples, the computing hardware 110 and / or the remote resource 142 store both the processed sensor data 124 and the raw sensor data 124.

[0032] Figures 2A-2D 2 are examples of maps 200, 200a-d generated by the robot 100 and / or the remote system 140. Each map 200 is a representation of an environment (e.g., the robot environment 10) for autonomously guiding the robot 100. Figure 1As shown in , as the robot 100 maneuvers around the robot environment 10, the robot 100 (or the remote system 140) generates a map 200 by receiving image data 124 of the robot environment 10 captured by one or more visual sensors 122, 122a-c of the sensor system 120. In some examples, the robot 100 generates the map 200 with the computing hardware 110, while in other examples, the remote system 140 generates the map 200 with the remote resources 142. Once the computing hardware 110 or the remote system 140 generates the map 200, the map 200 is communicated to the traversal system 116 for execution by the behavior system 102 of the robot 100. Here, the behavior system 102 controls the robot 100 to traverse within the robot environment 10 according to the map 200. The map 200 can be generated by the same robot 100 or a different robot that traverses the robot environment 10.

[0033] Some of the advantages of the map 200 can be immediately, repeatedly and / or transferably used with another robot having a similar sensor system 120. Here, the map 200 does not require final optimization once all sensor data 124 has been collected. In addition, the robot 100 can use the map 200 to autonomously navigate the robot environment 10 without global positioning information or other navigation data from beacons. Each map 200 can be automatically processed by the system of the robot 100 (or remote system 140) or manually processed (e.g., editing waypoints 210, edges 220, basins 230 ( Figure 2B )、Constraint 240( Figure 2C ) etc.). This enables the map 200 to flexibly include additional data (e.g., annotations 222) or provide adjusted robot behavior (e.g., suppressing ICP positioning in feature-less areas). Moreover, modifying the waypoints 210 or edges 220 of the map 200 (e.g., by adding, editing, or removing) should have no effect on the unmodified parts of the map 200, and thus simplifies quality control.

[0034] The map 200 includes waypoints 210 and edges 220 (also referred to as waypoint edges) that form connections between the waypoints 210. The waypoints 210 are representations of what the robot 100 senses (e.g., according to its sensor system 120) at a particular location within the robot environment 10. The robot 100 and / or the remote system 140 generates the waypoints 210 based on image data 124 collected by the robot's sensor system 120. Because the map 200 is generated from the waypoints 210, the map 200 can be locally consistent (e.g., spatially consistent within a region due to neighbor waypoints), but need not be globally accurate and / or consistent.

[0035] Using the image data 124, the robot 100 and / or the remote system 140 executes at least one waypoint heuristic 212 ( Figure 1 ) (e.g., a waypoint algorithm). At least one waypoint heuristic 212 is configured to trigger waypoint placement on the waypoint map 200. For example, the waypoint heuristic 212 is configured to detect a threshold feature detection within the image data 124 at the location of the robot 100 (e.g., when the map 200 is generated or updated). The waypoint heuristic 212 can identify features within the robot environment 10 that serve as reliable visual sensor features that provide repeatability for the robot 100 to maneuver around the robot environment 10. For example, the waypoint heuristic 212 is preprogrammed for feature recognition (e.g., programmed with stored features), or is programmed to identify features of spatial clusters in which the volumetric image data 124 appears (e.g., corners of a room or edges of a wall). In response to the at least one waypoint heuristic 212 triggering the waypoint placement, the robot 100 and / or the remote system 140 records the waypoint 210 on the map 200. This waypoint identification process can be repeated when the robot 100 is manually driven through an area (e.g., the robot environment 10). In some examples, the robot 100 and / or the remote system 140 stores a plurality of maps 200 generated for areas that the robot 100 has previously sensed (via its sensor system 120 ).

[0036] Each waypoint 210 is generally associated with a waypoint edge 220. More specifically, an edge 220 is configured to indicate how one waypoint 210 (e.g., a first waypoint 210a) is related to another waypoint 210 (e.g., a second waypoint 210b). For example, an edge 220 represents a positional relationship between waypoints 210 (e.g., adjacent waypoints 210). In other words, an edge 220 is a connection between two waypoints 210 (e.g., Figure 2A The edge 220a shown in is the connection between the first waypoint 210a and the second waypoint 210b. For example, the edge 220 is the path between the first waypoint 210a and the second waypoint 210b (e.g., the movement path of the robot 100). In some examples, each edge 220 includes a pose transformation that expresses how the robot 100 moves along the edge 220 between the two waypoints 210. The pose transformation can describe the position and / or orientation of one coordinate system relative to another coordinate system within the environment. In some embodiments, the edge 220 includes a complete three-dimensional transformation (e.g., six numerical values). Some of these numerical values ​​include various estimates, such as, for example, dead reckoning pose estimates, vision-based estimates, or other estimates based on kinematic and / or inertial measurements of the robot 100.

[0037] In some examples, the edge 220 includes an annotation 222 associated with the edge 220 that provides further indication / description of the robot environment 10. Some examples of the annotation 222 include a description or indication that the edge 220 is located on a staircase or passes through a doorway. These annotations 222 can assist the robot 100 during manipulation, particularly when visual information is lost or absent (e.g., a gap such as a doorway). In some configurations, the annotation 222 includes a directional constraint (which may also be referred to as a pose constraint). The directional constraint of the annotation 222 can specify an alignment and / or orientation (e.g., a pose) of the robot 100 at a particular environmental feature. For example, the annotation 222 specifies a particular alignment or pose for the robot 100 before the robot 100 travels down a staircase or down a narrow hallway (which may restrict the robot 100 from turning).

[0038] In some implementations, when the robot 100 senses the robot environment forming the map 200, each waypoint 210 of the map 200 also includes sensor data 124 corresponding to data collected by the sensor system 120 of the robot 100. Here, the sensor data 124 at the waypoint 210 enables the robot 100 to locate by comparing the real-time sensor data 124 collected as the robot 100 traverses the robot environment 10 according to the map 200 with the sensor data 124 stored for the waypoint 210 of the map 200. In some configurations, after the robot 100 moves along the edge 220 (e.g., intending to be at the target waypoint 210), the robot 100 is configured to locate by directly comparing the real-time sensor data 124 with the map 200 (e.g., the sensor data 124 associated with the intended target waypoint 210 of the map 200). Here, by storing the raw sensor data 124 or approximating the raw sensor data 124 with minimal processing of the waypoints 210 of the map 200, the robot 100 can use the real-time sensor data 124 to efficiently locate itself as the robot 100 maneuvers within the mapped robotic environment 10. In some examples, an iterative closest point (ICP) algorithm locates the robot 100 relative to the waypoints 210.

[0039] The sensor data 124 may also allow initialization of the robot 100 to autonomously traverse the robot environment 10 using the map 200. In some examples, the robot 100 initially receives a prompt that defines an initial pose P of the robot 100 relative to the waypoints 210, P I (For example, Figure 2A ). The robot 100 may be configured to receive this prompt automatically or from an operator of the robot 100. Here, the posture P refers to the kinematic position and / or orientation of the robot 100. Using the initial posture P of the robot 100 relative to the corresponding waypoint 210I , the robot 100 traverses the map 200 according to commands from the traversal system 116 instructing the robot 100 to move along the edge 220. For example, Figure 2A The map 200a shows the robot 100 moving between waypoints 210a-f along edges 220a-f.

[0040] In some configurations, the map 200 includes an unprocessed map 202 and / or a processed map 204. The unprocessed map 202 is a map that includes all raw or nearly raw sensor data 124 collected by the sensor system 120 of the robot 100 during the generation of the map 200 (e.g., Figure 2A In contrast, the processed map 204 processes the sensor data 124 of the waypoint 210 together with the immediate neighbors of the waypoint 210. In other words, the processed map 204 may include data for a greater range, such as data for multiple waypoints 210 within the robot environment 10. In some examples, the processed map 204 reduces noise near the waypoints 210 and edges 220, thereby increasing the effectiveness of autonomous traversal. For example, referring to Figure 2A The processed map 200a is processed by processing the sensor data 124 of the waypoint 210a together with all the sensor data 124 of the neighbor waypoints 210b, 210d. In some examples, the processed map 204 depicts points (e.g., volumetric points) corresponding to the edge 220 as a visual image of the processed map 204. As an example, for a given waypoint 210a, the robot 100 or remote system 140 (i.e., processing system) generating the processed map 204 identifies the neighbor waypoints 210b, 210d of the given waypoint 210a. Using the identified neighbor waypoints 210b, 210d, the traversal system 116 determines whether the three-dimensional point corresponding to the waypoints 210a, 210b, 210d in the three-dimensional point cloud corresponds to the visual edge of the image represented by the three-dimensional point cloud data. Based on this determination, the traversal system 116 associates the given waypoint 210a with the three-dimensional point corresponding to the visual edge of the image represented by the three-dimensional point cloud data. Additionally or alternatively, processing of processed map 204 removes any sensor data 124 along edges 220 that are not observed by sensor system 120 from multiple vantage points.

[0041] Figure 2B 2 is an example of a map 200b using a basin area surrounding one or more waypoints 210. Here, the waypoint 210 is configured with a corresponding basin 230. The basin 230 refers to a designated area surrounding the waypoint 210 (e.g., Figure 2BThe robot 100 is configured to use kinematics and / or inertial measurements to determine the movement of the robot 100 when the robot 100 determines that the position L of the robot 100 is outside of the basin 230 corresponding to the waypoint 210. In the example shown, when the robot 100 is at the first position L 1 , the robot 100 uses the ICP algorithm to determine the movement of the robot 100. However, when the robot 100 is in the second position L 2 When the robot 100 is moving, it determines the movement of the robot 100 based on kinematics and / or inertial measurements.

[0042] In some examples, the area within the waypoint 210 is referred to as a target zone. Once the robot 100 is within the target zone, the robot 100 has successfully navigated to the waypoint 210. After successfully navigating to the waypoint 210 (i.e., entering the target zone), the robot 100 may continue to move along the path of the robot 100 toward subsequent waypoints 210.

[0043] In some configurations, when the localization quality is poor, the traversal system 116 may stop autonomously guiding the robot 100 according to the map 200. Such stopping may occur based on the basin 230 or based on, for example, a localization threshold (e.g., a location L from the waypoint 210 at which an operator of the robot 100 would consider the robot 100 lost). For example, the localization threshold is a distance from the waypoint 210 that is determined to be outside the range of the waypoint 210 for performing localization based on the ICP algorithm.

[0044] Additionally or alternatively, the robot 100 can utilize global positioning (e.g., a global positioning system (GPS) receiver) while navigating according to kinematics, inertial measurements, or an ICP algorithm. In some examples, the robot 100 first uses the map 200 to determine an initial heading of the robot 100 (e.g., along the edge 220 between the waypoints 210), but then supplements navigation (e.g., supplements kinematics, inertial measurements, or an ICP algorithm) with GPS measurements received from a GPS receiver at the robot 100. In some configurations, the computing hardware 110 is configured to receive GPS measurements (e.g., the computing hardware 110 is configured with a GPS receiver). Supplementing the GPS system can be particularly useful when the robot 100 is navigating in an outdoor robotic environment 10. The robot 100 can be configured such that when the robot 100 senses a particular robotic environment 10 (e.g., when the robot 100 senses an outdoor environment with fewer adjacent features), the robot 100 always uses the GPS system or utilizes the GPS system to provide navigation input. For example, when the positioning quality is poor, the robot 100 activates and / or uses the GPS system to provide navigation input. In some configurations, the GPS system tracks the path of the robot 100. In these configurations, the GPS path can be compared to the edges 220 on the map 200 to provide feedback to the traversal system 116 (e.g., using the traversal system 116 to fine-tune autonomous navigation).

[0045] Reference Figure 2C In some embodiments, the map 200c includes map constraints 240, 240a-b. The map constraints 240 refer to virtual obstacles that prevent the robot 100 from deviating from the area designated or blocked by the virtual obstacles. In other words, the map constraints 240 can help prevent the robot 100 from significantly deviating from the movement path corresponding to one or more edges 220. The map constraints 240 can be placed relative to the recorded path, manually added by the operator, or pre-calculated based on the scene geometry. For example, in Figure 2C , the edges 220a-c may correspond to corridors. When the robot 100 records the map 200 with waypoints 210a-c and the edges 220a-c corresponding to corridors, the robot 100 may never be driven into the rooms and / or doorways on either side of the corridor. It may be advantageous to block these rooms at their corresponding doorways by the first map constraint 240a and the second map constraint 240b. Thus, using the first map constraint 240a and the second map constraint 240b, the robot 100 should be prevented from roaming into the room while following the edges 220a-c, thereby preventing the robot 100 from getting lost or yawing.

[0046] Figure 2DA map 200d (e.g., the processed map 204) is shown with a time-indexed robot trajectory 250 that the traversal system 116 configures the robot 100 to follow. The time-indexed robot trajectory 250 is a movement trajectory of the robot 100 between the waypoints 210. The time-indexed robot trajectory 250 may also be referred to as a traversal path 250. Here, Figure 2D The time-indexed robot trajectory 250 is depicted by an arrow pointing from a first waypoint 210a (shown as an "X" within waypoint 210a) where the robot 100 is currently located to a third waypoint 210c. To navigate the time-indexed robot trajectory 250, the robot 100 will traverse the edges 220a, 220b.

[0047] In some examples, the traversal system 116 generates a time-indexed robot trajectory 250 across upcoming waypoints 210 for movement of the robot 100. In some implementations, for the time-indexed robot trajectory 250, the traversal system 116 specifies that the orientation of the robot 100 at each waypoint 210 is halfway between the current edge 220 and the next edge 220 of the route that passes through the corresponding waypoint 210. Here, the time t may be based on the expected time t to reach each waypoint 210 due to the trajectory. e As the robot 100 travels along the trajectory 250, the time t of the trajectory 250 may be updated based on the updated estimate of when the robot 100 should reach each waypoint 210 (e.g., the time t to travel from the second waypoint 210b to the third waypoint 210c). 2 ). In this configuration, when the time t corresponding to the waypoint 210 approaches the current time, the time of the waypoint is updated. In some examples, after the current time has passed the time associated with the waypoint 210, the traversal system 116 considers the waypoint 210 reached and then removes it from the trajectory.

[0048] Additionally or alternatively, such as Figure 2A and 2C As shown in the maps 200a, 200c of FIG. 1 , the annotation 222 associated with the edge 220 can modify the orientation of the robot 100 at the waypoint 210 associated with the time-indexed robot trajectory so that the robot 100 complies with the directional constraints of the annotation 222. Based on these directional constraints, the orientation of the robot 100 can also be estimated for the waypoint 210 within the trajectory. When the orientation estimation determines that the orientation will be incorrect according to the directional constraints, an orientation segment (e.g., a turn-in-place segment) can be added to the time-indexed robot trajectory to ensure that the robot 100 complies with the directional constraints.

[0049] Return to reference Figure 1, the behavior system 102 of the robot 100 may optionally receive a path description 206 from the traversal system 116, the path description 206 including target conditions and constraints for reaching one or more waypoints 210. Based on these target conditions and constraints, the robot 100 follows the path description 206 with the ability to adjust its route as long as the adjusted route meets the target and constraints. Some examples of conditions include specifying a position or orientation of the robot 100 at or near a target waypoint 210, or specifying a position, orientation, and / or speed for traveling to a target waypoint 210.

[0050] Figure 3 is an example flow chart of a method 300 for configuring a map 200. At operation 302, the method 300 receives image data 124 of a robot environment 10 surrounding a robot 100 manipulating in the robot environment 10. At operation 304, the method 300 executes at least one waypoint heuristic 212, wherein the at least one waypoint heuristic 212 is configured to trigger waypoint placement on the waypoint map 200. In response to the at least one waypoint heuristic 212 triggering the waypoint placement, at operation 306, the method 300 records the waypoint 210 on the waypoint map 200. Here, the waypoint 210 is associated with at least one waypoint edge 220 and includes sensor data 124 obtained by a sensor system 120 of the robot 100. At least one waypoint edge 220 may include a pose transformation that expresses how to move between two waypoints 210.

[0051] Figure 4 is an example flow chart of a method 400 for localizing a robot 100 relative to a waypoint map 200. At operation 402, the method 400 receives image data 124 of a robot environment 10 surrounding a robot 100 maneuvering in the robot environment 10. At operation 404, the method 400 performs an iterative closest point (ICP) algorithm based on the received image data 124, the algorithm being configured to localize to a first waypoint 210a of the waypoint map 200. At operation 406, the method 400 performs a traversal path 250 from the first waypoint 210a of the waypoint map 200 to a second waypoint 210b of the waypoint map 200. At operation 404, the method 400 updates the localization to the second waypoint 210b of the waypoint map 200 through the ICP algorithm based on the received image data.

[0052] Figure 51 is a schematic diagram of an example computing device 500 that can be used to implement the systems and methods described in this document (e.g., behavior system 102, computing hardware 110, traversal system 116, or remote resources 142). Computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are exemplary only and are not meant to limit the implementation of the inventions described and / or claimed in this document.

[0053] The computing device 500 includes a processor 510, a memory 520, a storage device 530, a high-speed interface / controller 540 and a high-speed expansion port 550 connected to the memory 520, and a low-speed interface / controller 560 connected to a low-speed bus 570 and the storage device 530. Each of the components 510, 520, 530, 540, 550, and 560 is interconnected using various buses and can be mounted on a common motherboard or in other appropriate ways. The processor 510 can process instructions for execution within the computing device 500, including instructions stored in the memory 520 or on the storage device 530 for displaying graphical information of a graphical user interface (GUI) on an external input / output device (such as a display 580 coupled to the high-speed interface 540). In other embodiments, multiple processors and / or multiple buses, as well as multiple memories and memory types, can be used appropriately. Moreover, multiple computing devices 500 can be connected, each computing device providing a portion of the necessary operations (for example, as a server bank, a group of blade servers, or a multi-processor system).

[0054] The memory 520 stores information non-temporarily within the computing device 500. The memory 520 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). The non-transitory memory 520 may be a physical device for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) for use by the computing device 700. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0055] Storage device 530 can provide mass storage for computing device 500. In some embodiments, storage device 530 is a computer readable medium. In various embodiments, storage device 530 can be a floppy disk device, a hard disk device, an optical disk device or a tape device, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configurations. In other embodiments, a computer program product is tangibly implemented in an information carrier. The computer program product includes instructions for executing one or more methods (e.g., the above method) when executed. The information carrier is a computer or machine readable medium, such as a memory 520, a storage device 530, or a memory on a processor 510.

[0056] The high-speed controller 540 manages bandwidth-intensive operations of the computing device 500, while the low-speed controller 560 manages less bandwidth-intensive operations. This division of responsibilities is exemplary only. In some embodiments, the high-speed controller 540 is coupled to the memory 520, the display 580, and the high-speed expansion port 550 that can accept various expansion cards (not shown) (e.g., through a graphics processor or accelerator). In some embodiments, the low-speed controller 560 is coupled to the storage device 530 and the low-speed expansion port 590. The low-speed expansion port 590, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device (such as a switch or router), for example, through a network adapter.

[0057] Computing device 500 can be implemented in many different forms, as shown. For example, it can be implemented as a standard server 500a or multiple implementations in a group of such servers 500a, implemented as a laptop computer 500b or implemented as part of a rack server system 500c.

[0058] Various implementations of the systems and techniques described herein can be implemented in digital electronic devices and / or optical circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be special purpose or general purpose, coupled to receive data and instructions from and send data and instructions to a storage system, at least one input device, and at least one output device.

[0059] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0060] The processes and logic flows described in this specification can be performed by one or more programmable processors that execute one or more computer programs to perform functions by operating on input data and generating outputs. The processes and logic flows can also be performed by dedicated logic circuit systems, such as FPGAs (field programmable gate arrays) or ASICs (application-specific integrated circuits). As an example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, and any one or more processors in any kind of digital computer. Generally speaking, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally speaking, a computer will also include or be operably coupled to receive data from it or transfer data to it or both One or more large-capacity storage devices for storing data, such as disks, magneto-optical disks, or optical disks. However, a computer does not have to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, as examples, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0061] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, and optionally a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0062] Many embodiments have been described. However, it will be appreciated that various modifications may be made without departing from the spirit and scope of the present disclosure. Therefore, other embodiments are within the scope of the present disclosure.

Claims

1. A method for a robot, comprising: receiving, at data processing hardware, sensor data of a robot manipulating in an environment; executing, by the data processing hardware, at least one waypoint heuristic based on the sensor data, the at least one waypoint heuristic configured to trigger waypoint placement on a waypoint map; Based on the execution of the at least one waypoint heuristic, the data processing hardware: recording the waypoint on the waypoint map, the waypoint being associated with at least one waypoint edge, and Recording a basin area corresponding to the waypoint, the basin area being a designated area surrounding the waypoint, wherein different algorithms are used to determine movement of the robot relative to the waypoint based on the position of the robot relative to the basin area; determining, by the data processing hardware, a position of the robot relative to the basin region; and Based on determining the position of the robot relative to the basin region, an algorithm for determining movement of the robot is determined by the data processing hardware from a plurality of algorithms. 2 . The method of claim 1 , wherein the sensor data comprises three-dimensional point cloud data captured by a three-dimensional volumetric image sensor.

3. The method of claim 2, wherein the three-dimensional volume image sensor comprises one or more of a stereo camera, a scanning light detection and ranging (LIDAR) sensor, or a scanning laser detection and ranging (LADAR) sensor.

4. The method of any of claims 1-3, wherein executing the at least one waypoint heuristic results in a threshold feature detection within the sensor data at a current position of the robot.

5. The method of any one of claims 1 to 3, wherein the sensor data comprises: 3D point cloud data captured by a 3D volumetric image sensor; as well as The posture data includes inertial measurement data measured by an inertial measurement unit (IMU) of the robot.

6. The method of claim 5, further comprising: identifying neighbor waypoints to the waypoint; identifying three-dimensional points of the waypoint and the neighbor waypoints from the three-dimensional point cloud data; determining that the three-dimensional point corresponds to a visual edge of an image represented by the three-dimensional point cloud data; and The three-dimensional point is associated with the waypoint.

7. The method of any one of claims 1-3, wherein the waypoint comprises a robot posture constraint configured to enable the robot to achieve an oriented posture at the waypoint.

8. A method as claimed in any one of claims 1-3, wherein the waypoint map is locally consistent with the neighbor waypoints of the waypoint.

9. The method of any one of claims 1-3, wherein the at least one waypoint edge includes annotations indicating spatial characteristics of the environment.

10. The method according to any one of claims 1 to 3, wherein the robot is configured to: Navigating to the waypoint according to different algorithms; and Based on navigating to the waypoint, navigating to a second waypoint according to a different algorithm, wherein a different algorithm is used to determine movement of the robot relative to the second waypoint based on the position of the robot relative to a second basin area corresponding to the second waypoint.

11. A robot comprising: a drive system configured to maneuver the robot in an environment; data processing hardware in communication with the drive system; as well as Memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations comprising: receiving sensor data of the robot manipulating the environment; executing at least one waypoint heuristic based on the sensor data, the at least one waypoint heuristic configured to trigger waypoint placement on a waypoint map; Based on the execution of the at least one waypoint heuristic: recording the waypoint on the waypoint map, the waypoint being associated with at least one waypoint edge, and Recording a basin area corresponding to the waypoint, the basin area being a designated area surrounding the waypoint, wherein different algorithms are used to determine movement of the robot relative to the waypoint based on the position of the robot relative to the basin area; determining a position of the robot relative to the basin region; and Based on determining the position of the robot relative to the basin area, an algorithm for determining movement of the robot is determined from a plurality of algorithms.

12. The robot of claim 11, wherein the sensor data comprises three-dimensional point cloud data captured by a three-dimensional volumetric image sensor.

13. The robot of claim 12, wherein the three-dimensional volume image sensor comprises one or more of a stereo camera, a scanning light detection and ranging LIDAR sensor, or a scanning laser detection and ranging LADAR sensor.

14. The robot of any of claims 11-13, wherein executing the at least one waypoint heuristic results in a threshold feature detection within the sensor data at a current position of the robot.

15. The robot of any one of claims 11-13, wherein the sensor data comprises: 3D point cloud data captured by a 3D volumetric image sensor; as well as The posture data includes inertial measurement data measured by an inertial measurement unit of the robot.

16. The robot of claim 15, wherein the operations further comprise: identifying neighbor waypoints to the waypoint; identifying three-dimensional points of the waypoint and the neighbor waypoints from the three-dimensional point cloud data; determining that the three-dimensional point corresponds to a visual edge of an image represented by the three-dimensional point cloud data; as well as The three-dimensional point is associated with the waypoint.

17. The robot of any one of claims 11-13, wherein the waypoint comprises a robot pose constraint configured to cause the robot to achieve an oriented pose at the waypoint.

18. The robot of any one of claims 11-13, wherein the waypoint map is locally consistent with neighbor waypoints of the waypoint.

19. The robot of any one of claims 11-13, wherein the at least one waypoint edge comprises an annotation indicating a spatial feature of the environment.

20. The robot of any one of claims 11-13, wherein the operations further comprise: Navigating to the waypoint according to different algorithms; as well as Based on navigating to the waypoint, navigating to a second waypoint according to a different algorithm, wherein a different algorithm is used to determine movement of the robot relative to the second waypoint based on the position of the robot relative to a second basin area corresponding to the second waypoint.

21. A method for a robot, comprising: receiving image data of a robot manipulating an environment at data processing hardware, wherein based on a position of the robot relative to a particular waypoint of a waypoint map, different algorithms are used to determine movement of the robot relative to the particular waypoint; determining, by the data processing hardware, that a first position of the robot is within a first basin area corresponding to a first waypoint of the waypoint map, wherein the first basin area refers to a designated area surrounding the first waypoint; Based on determining that the first position of the robot is inside the first basin region, determining, by the data processing hardware, a first movement of the robot relative to the first waypoint using the image data and an iterative closest point (ICP) algorithm; executing, by the data processing hardware, a traversal path from the first waypoint to a second waypoint of the waypoint map to move the robot along an edge between the first waypoint and the second waypoint; determining, by the data processing hardware, that a second position of the robot is outside a second basin region corresponding to the second waypoint, wherein the second basin region refers to a designated area surrounding the second waypoint; as well as Based on determining that the second position of the robot is outside of the second basin region, a second movement of the robot relative to the second waypoint is determined by the data processing hardware using the image data and a kinematic algorithm.

22. The method of claim 21, wherein the image data comprises three-dimensional point cloud data captured by a three-dimensional volumetric image sensor.

23. The method of claim 22, wherein the three-dimensional volumetric image sensor comprises one or more of a stereo camera, a scanning light detection and ranging (LIDAR) sensor, or a scanning laser detection and ranging (LADAR) sensor.

24. The method according to any one of claims 21 to 23, wherein: Determining a second movement of the robot relative to the second waypoint includes comparing the image data to stored image data of the waypoint map, the stored image data associated with the second waypoint.

25. A robot comprising: main body; legs coupled to the body and configured to maneuver the robot around an environment; data processing hardware in communication with the legs; as well as Memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations comprising: receiving image data of the robot maneuvering in an environment, wherein based on the position of the robot relative to a particular waypoint of a waypoint map, different algorithms are used to determine movement of the robot relative to the particular waypoint; Determining that a first position of the robot is within a first basin area corresponding to a first waypoint of the waypoint map, wherein the first basin area refers to a designated area surrounding the first waypoint; Based on determining that the first position of the robot is inside the first basin region, determining a first movement of the robot relative to the first waypoint using the image data and an iterative closest point (ICP) algorithm; executing a traversal path from the first waypoint to a second waypoint of the waypoint map to move the robot along an edge between the first waypoint and the second waypoint; determining that a second position of the robot is outside a second basin area corresponding to the second waypoint, wherein the second basin area refers to a designated area surrounding the second waypoint; and Based on determining that the second position of the robot is outside of the second basin region, a second movement of the robot relative to the second waypoint is determined using the image data and a kinematic algorithm.

26. The robot of claim 25, wherein the image data comprises three-dimensional point cloud data captured by a three-dimensional volumetric image sensor.

27. The robot of claim 26, wherein the image sensor comprises a stereo camera, a scanning light detection and ranging LIDAR sensor, or a scanning laser detection and ranging LADAR sensor.

28. The robot of any one of claims 25-27, wherein determining a second movement of the robot relative to the second waypoint comprises comparing the image data with stored image data of the waypoint map, the stored image data being associated with the second waypoint.

Images