Floor debris avoidance for mobile robot

Abstract

Methods and apparatus are provided for navigating a robot through an environment along a route associated with a task. The method includes identifying a set of potential obstacles in the environment based on sensor data received by one or more sensors of the robot, determining that at least one of the potential obstacles in the set is an obstacle based at least in part on stored data indicative of the set of fall positions of the robot during a previous run of the task, and determining that at least one of the potential obstacles in the set is an obstacle. And navigating the robot to avoid stepping onto the obstacle.

Description

Background Art

[0001] A robot is typically a reprogrammable and versatile manipulator, usually designed to move materials, parts, tools, or specialized equipment through variable programmed motions to perform tasks. A robot can be a physically anchored manipulator (e.g., an industrial robot arm), a mobile robot that moves throughout an environment (e.g., using legs, wheels, or traction-based mechanisms), or some combination of manipulators and mobile robots. Robots are used in a variety of industries, including, for example, manufacturing, warehouse logistics, transportation, hazardous environments, exploration, and healthcare. Summary of the invention

[0002] In some embodiments, a method of navigating a robot through an environment along a route is provided, the route being associated with a mission. The method includes identifying a set of potential obstacles in the environment based on sensor data received by one or more sensors of the robot, determining that at least one of the potential obstacles in the set is an obstacle based at least in part on stored data indicating a set of step locations of the robot during previous performance of the mission, and navigating the robot to avoid stepping onto the obstacle.

[0003] In one aspect, the sensor data comprises image data. In one aspect, the sensor data comprises point cloud data representing distances to objects in the environment. In one aspect, the method further comprises: processing the sensor data to generate a topographic map of the environment, and identifying the set of potential obstacles in the environment is based at least in part on the topographic map. In one aspect, the method further comprises segmenting the topographic map to generate a segmentation map, and identifying the set of potential obstacles in the environment is based at least in part on the segmentation map. In one aspect, segmenting the topographic map comprises detecting at least one boundary in the topographic map, and detecting at least one surface in the topographic map, wherein identifying the set of potential obstacles in the environment is based at least in part on at least one boundary and at least one surface. In one aspect, identifying the set of potential obstacles in the environment is based at least in part on the detected at least one boundary and the detected at least one surface, comprising identifying a set of segments in the segmentation map, the segments comprising at least one boundary connected to at least one detected surface, and including in the set of potential obstacles objects associated with at least some of the segments in the set of segments. In one aspect, the method further comprises discarding segments having a size below a threshold size from the set of segments, and including in the set of potential obstacles objects associated with segments having a size above a threshold size.

[0004] In one aspect, based at least in part on stored data indicating a set of landing positions of the robot during previous execution of the task, determining that at least one of the potential obstacles in the set is an obstacle includes: for each of the potential obstacles in the set, determining whether the robot stepped onto the potential obstacle during the previous execution of the task based on the set of landing positions, and determining that the potential obstacle is an obstacle when it is determined that the robot did not step onto the potential obstacle during the previous execution of the task. In one aspect, the set of landing positions includes upcoming landing positions along a portion of a route that the robot has not yet traversed, and based on the set of landing positions, determining whether the robot stepped onto the potential obstacle during the previous execution of the task includes determining whether the robot stepped onto the potential obstacle based on one or more upcoming landing positions of the route that the robot has not yet traversed.

[0005] In one aspect, the method further includes identifying a set of one or more large obstacles in the robot's environment based on the sensor data, adding the obstacle to the set of one or more large obstacles, and navigating the robot to avoid stepping onto all obstacles in the set of one or more large obstacles. In one aspect, the height of the obstacles is less than 30 cm. In one aspect, the previous execution of the task corresponds to the execution of the task when the task was recorded during operation of the robot by the operator.

[0006] In some embodiments, a legged robot is provided. The legged robot includes a perception system and at least one computer processor, the perception system including one or more sensors configured to sense sensor data. The at least one computer processor is configured to identify a group of potential obstacles in the environment based on the sensor data, and determine that at least one of the potential obstacles in the group is an obstacle based at least in part on stored data indicating a group of step locations of the robot during previous execution of a task. The legged robot also includes a navigation system configured to navigate the robot to avoid stepping onto an obstacle.

[0007] In one aspect, the sensor data comprises image data. In one aspect, the sensor data comprises point cloud data representing distances to objects in the environment. In one aspect, the at least one computer processor is further configured to process the sensor data to generate a topographic map of the environment, and identifying the set of potential obstacles in the environment is based at least in part on the topographic map. In one aspect, the at least one computer processor is further configured to segment the topographic map to generate a segmentation map, and identifying the set of potential obstacles in the environment is based at least in part on the segmentation map. In one aspect, segmenting the topographic map comprises detecting at least one boundary in the topographic map, and detecting at least one surface in the topographic map, wherein identifying the set of potential obstacles in the environment is based at least in part on at least one boundary and at least one surface. In one aspect, identifying the set of potential obstacles in the environment is based at least in part on the at least one detected boundary and at least one detected surface, comprising identifying a set of segments in the segment map, the segments comprising at least one boundary connected to at least one detected surface; and including in the set of potential obstacles objects associated with at least some of the segments in the set of segments. In one aspect, the at least one computer processor is further configured to discard segments having a size below a threshold size from the set of segments and include in the set of potential obstacles objects associated with segments having a size above the threshold size.

[0008] In one aspect, based at least in part on stored data indicating a set of landing positions of the robot during previous execution of the task, determining that at least one of the potential obstacles in the set is an obstacle includes: for each of the potential obstacles in the set, determining whether the robot stepped onto the potential obstacle during the previous execution of the task based on the set of landing positions, and determining that the potential obstacle is an obstacle when it is determined that the robot did not step onto the potential obstacle during the previous execution of the task. In one aspect, the set of landing positions includes upcoming landing positions along a portion of a route that the robot has not yet traversed, and determining whether the robot stepped onto the potential obstacle during the previous execution of the task based on the set of landing positions includes determining whether the robot stepped onto the potential obstacle based on one or more upcoming landing positions of the route that the robot has not yet traversed.

[0009] In one aspect, the at least one computer processor is further programmed to identify a set of one or more large obstacles in the robot's environment based on the sensor data, add the obstacle to the set of one or more large obstacles, and navigate the robot to avoid stepping onto all obstacles in the set of one or more large obstacles. In one aspect, the height of the obstacles is less than 30 cm. In one aspect, the previous execution of the task corresponds to the execution of the task when the task was recorded during operation of the robot by the operator.

[0010] In some embodiments, a non-transitory computer-readable medium is provided. The non-transitory computer-readable medium is encoded with a plurality of instructions that, when executed by at least one computer processor, perform a method of navigating a robot through an environment along a route, the route being associated with a task. The method includes identifying a group of potential obstacles in the environment based on sensor data received by one or more sensors of the robot, determining that at least one of the potential obstacles in the group is an obstacle based at least in part on stored data indicating a group of step locations of the robot during previous execution of the task, and navigating the robot to avoid stepping onto the obstacle.

[0011] In one aspect, the sensor data comprises image data. In one aspect, the sensor data comprises point cloud data representing distances to objects in the environment. In one aspect, the method further comprises: processing the sensor data to generate a topographic map of the environment, and identifying the set of potential obstacles in the environment is based at least in part on the topographic map. In one aspect, the method further comprises segmenting the topographic map to generate a segmentation map, and identifying the set of potential obstacles in the environment is based at least in part on the segmentation map. In one aspect, segmenting the topographic map comprises detecting at least one boundary in the topographic map, and detecting at least one surface in the topographic map, wherein identifying the set of potential obstacles in the environment is based at least in part on at least one boundary and at least one surface. In one aspect, identifying the set of potential obstacles in the environment is based at least in part on at least one detected boundary and at least one detected surface, comprising identifying a set of segments in the segmentation map, the segments comprising at least one boundary connected to at least one detected surface, and including in the set of potential obstacles objects associated with at least some of the segments in the set of segments. In one aspect, the method further comprises discarding segments having a size below a threshold size from the set of segments, and including in the set of potential obstacles objects associated with segments having a size above a threshold size.

[0012] In one aspect, based at least in part on stored data indicating a set of landing positions of the robot during previous execution of the task, determining that at least one of the potential obstacles in the set is an obstacle includes: for each of the potential obstacles in the set, determining whether the robot stepped onto the potential obstacle during the previous execution of the task based on the set of landing positions, and determining that the potential obstacle is an obstacle when it is determined that the robot did not step onto the potential obstacle during the previous execution of the task. In one aspect, the set of landing positions includes upcoming landing positions along a portion of a route that the robot has not yet traversed, and based on the set of landing positions, determining whether the robot stepped onto the potential obstacle during the previous execution of the task includes determining whether the robot stepped onto the potential obstacle based on one or more upcoming landing positions of the route that the robot has not yet traversed.

[0013] In one aspect, the method further comprises identifying a set of one or more large obstacles in the robot's environment based on the sensor data, adding the obstacle to the set of one or more large obstacles, and navigating the robot to avoid stepping onto all obstacles in the set of one or more large obstacles. In one aspect, the height of the obstacles is less than 30 cm. In one aspect, the previous execution of the task corresponds to the execution of the task when the task was recorded during operation of the robot by the operator.

[0014] The foregoing device and method embodiments can be implemented with any suitable combination of the aspects, features and actions described above or further described in detail below. These and other aspects, embodiments and features of the present teachings can be more fully understood from the following description in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Various aspects and embodiments will be described with reference to the following drawings. It should be understood that the drawings are not necessarily drawn to scale. In the drawings, each identical or nearly identical component shown in the various figures is represented by the same reference numeral. For clarity, not every component may be labeled in every drawing.

[0016] Figure 1A illustrates an example of a legged robot configured to navigate in an environment along a route in accordance with some embodiments;

[0017] Figure 1B is a block diagram of the robot's components, such as Figure 1A The robot shown;

[0018] Figure 2A A method for navigating a robot (such as a Figure 1A Components of the navigation system of a robot;

[0019] Figure 2B-2F An example technique for navigating a robot to avoid one or more obstacles is shown in accordance with some embodiments;

[0020] Figure 3 is a block diagram of a system for detecting ground clutter obstacles in a robot's environment according to some embodiments;

[0021] Figure 4 is a flow chart of a process for identifying ground clutter obstacles during execution of a mission based on sensor data and stored data indicating landing locations of a robot during previous executions of the mission, according to some embodiments;

[0022] Figure 5 is a flow chart of a process for segmenting a terrain map to detect potential ground clutter obstacles in an environment according to some embodiments;

[0023] Figure 6 is a flow chart of a process for classifying potential ground clutter obstacles as obstacles based on step-down information according to some embodiments;

[0024] Figure 7 schematically illustrates detection of boundaries in a topographic map as part of a segmentation process according to some embodiments;

[0025] Figure 8 schematically illustrates detection of surfaces in a topographic map as part of a segmentation process according to some embodiments;

[0026] Fig. 9 schematically illustrates the superposition of step information on a segmented terrain map for identifying ground debris obstacles according to some embodiments; and

[0027] Fig.10 is a block diagram of components of a robot upon which some embodiments may be implemented. DETAILED DESCRIPTION

[0028] Some robots are used to navigate an environment to perform various tasks or functions. These robots are often operated to perform a "task" by navigating the robot through the environment. Tasks are sometimes recorded so that the robot can perform the task again at a later time. In some tasks, the robot both navigates through the environment and interacts with the environment. The interaction sometimes takes the form of collecting data using one or more sensors.

[0029] Robots can be configured to run tasks to accomplish specific goals, such as performing surveillance, collecting sensor data, etc. Figure 1A - Figure B depicts an example of a robot 100 capable of performing such a task. In order for the robot 100 to be able to perform a task, the robot 100 may undergo an initial mapping process during which the robot 100 moves around the environment 10 (typically in response to commands input by a user to a tablet or other controller) to collect data about the environment 10 (e.g., via one or more sensors), and may generate a topology map 204 (an example of which is shown in FIG. Figure 2A ), the topological graph 204 defines waypoints 212 along the path of the robot 100 and edges 214 representing the paths between corresponding pairs of waypoints 212. Each waypoint 212 can be associated with sensor data, reference points and / or robot pose information at a specific time and place, for example, and each edge 214 can topologically connect the waypoints 212.

[0030] In some existing systems, a given “mission record” may identify a sequence of actions to occur at a particular waypoint 212 included on the topology map 204. For example, the mission record may indicate that the robot 100 is to go to a first waypoint 212 and perform a first action, then go to a second waypoint 212 and perform a second action, etc. In some implementations, such a mission record need not specify all of the waypoints 212 that the robot 100 will actually traverse when executing the mission, and may instead specify only those waypoints 212 at which a particular action is to be performed. As explained in detail below, such a mission record may be executed by the mission execution system 184 of the robot 100 (in Figure 1B The task execution system 184 can communicate with other systems of the robot 100 as needed to successfully execute the task. For example, in some implementations, the task execution system 184 can communicate with the navigation system 200 (also in FIG. Figure 1B 204 and the mission record, requesting the navigation system 200 to determine a navigation route 202 using the topology map 204 and the mission record, the navigation route 202 including the various waypoints 212 of the topology map 204 identified in the mission record, and any number of additional waypoints 212 of the topology map 204 located between the waypoints 212 identified in the mission record. The determined navigation route 202 may also include edges 214 located between corresponding pairs of such waypoints 212. Enabling the robot to follow the navigation route 202 including all waypoints 212 identified in the mission record may enable the mission execution system 184 to perform corresponding actions in the mission record when the robot 100 reaches those waypoints 212.

[0031] As reference below Figure 2A As described, the navigation system 200 may include a navigation generator 210 and a route executor 220, wherein the navigation generator 210 may generate a navigation route 202 including specified waypoints 212 (e.g., waypoints identified in a mission recording), and the route executor 220 is configured to control the robot 100 to move along the identified navigation route 202, possibly changing the route of the robot along an alternative path 206, for example, if it is necessary to avoid obstacles 20 that may not have existed when the mission was recorded.

[0032] In some existing systems, the route runner 220 may determine to reroute the robot to bypass all obstacles along the route that the robot cannot pass through (e.g., obstacle 20). When making a decision to reroute, such systems may not take into account the presence of smaller objects along the route that the robot may step onto during the execution of the task while traveling along the route. Some existing systems simply follow the route specified by the waypoints and edges in the task record, rather than navigating around such smaller objects to avoid them by taking an alternative path different from the path specified in the task record, which may cause the robot to step onto such smaller objects or otherwise pass through such smaller objects. The inventors have recognized and understood that failure to reroute around at least some smaller objects in the robot's environment may result in damage to fragile objects (e.g., light bulbs, tools) and / or may cause the robot to trip and fall when stepping onto or into certain objects (such as open barrels or boxes), thereby potentially causing damage to the robot.

[0033] Some embodiments of the present disclosure relate to techniques for detecting and / or classifying "ground clutter" objects in a robot's environment, thereby enabling the robot to avoid stepping onto such objects, thereby improving task robustness and reducing potential damage to the robot and / or objects in the environment. Also referred to herein as "ground clutter avoidance," some embodiments relate to techniques for identifying objects in the environment that the robot should not step onto or pass through (e.g., walk) to reduce the likelihood of damage to the robot and / or the environment. By modeling smaller objects in the local environment around the robot, the robot's obstacle avoidance capabilities (e.g., previously focused on avoiding larger objects) are improved, thereby causing less damage to the robot and objects in its environment during the operation of the task. For example, an obstacle with a flat surface may otherwise appear to be a suitable location for the robot to step, but doing so may result in damage to the obstacle and / or the robot. As described in more detail below, in some embodiments, during execution of a mission, objects in the environment (e.g., objects located on the ground) are detected, and information describing how the robot traversed (e.g., stepped) in the environment during previous execution of the mission (e.g., during recording of the mission) is used at least in part to determine which objects it is acceptable to traverse (e.g., step onto) and which objects the robot should reroute around.

[0034] refer to Figure 1A, the robot 100 may include a body 110 having a motion-based structure, such as legs 120a-120d coupled to the body 110, which enables the robot 100 to move around the environment 10. In some implementations, each leg 120 may be an articulated structure such that one or more joints J allow a member 122 of the leg 120 to move. For example, each leg 120 may include a hip joint J that couples an upper member 122, 122u of the leg 120 to the body 110. H , and coupling the upper member 122u of the leg 120 to the lower member 122 of the leg 120 L Knee joint J K For impact detection, the hip joint J H Can be further decomposed into hip joint J H The abduction-adduction rotation occurs in the frontal plane of the robot 100 (ie, in the x-axis A x and z-axis A Z The XZ plane extends in the direction of the hip joint J H flexion-extension rotation, which occurs in the sagittal plane of the robot 100 (i.e., in the y-axis A Y and z-axis A Z The YZ plane extends in the direction of Figure 1A A quadruped robot is depicted having four legs 120a - 120d , but it should be understood that the robot 100 may include any number of legs or motion-based structure (eg, a biped or humanoid robot having two legs) that provides a means for traversing terrain within the environment 10 .

[0035] To traverse terrain, each leg 120 may have a distal end 124 that contacts a surface 14 of the terrain (i.e., a traction surface). In other words, the distal end 124 of the leg 120 is the end of the leg 120 used by the robot 100 to pivot, securely position, or generally provide traction during movement of the robot 100. For example, the distal end 124 of the leg 120 may correspond to a "foot" of the robot 100. In some examples, although not shown, the distal end 124 of the leg 120 may include an ankle joint so that the distal end 124 is relative to the lower member 122 of the leg 120. L Articulated.

[0036] In the illustrated example, the robot 100 includes an arm 126 that functions as a robotic manipulator. The arm 126 can be configured to move about multiple degrees of freedom in order to engage elements of the environment 10 (e.g., objects within the environment 10). In some implementations, the arm 126 can include one or more members 128, where the members 128 are coupled by joints J such that the arm 126 can pivot or rotate about the joints J. For example, with more than one member 128, the arm 126 can be configured to extend or retract. To illustrate the example, Figure 1A An arm 126 having three members 128 is depicted, wherein the three members 128 correspond to the lower member 128 L 、Upper member 128 U and hand component 128 H (For example, hand member 128 H Also called end effector 128 H Here, the lower member 128 L The arm joint J may be located around one or more arm joints J located near the body 110. A Rotate or pivot (e.g., where the arm 126 is connected to the body 110 of the robot 100). For example, Figure 1A Depicts the ability to move around the first arm joint J A1 The arm 126 can be rotated by a yaw arm joint. The arm 126 can rotate about the vertical gravity axis (eg, shown as A) of the robot 100 by using the yaw arm joint. z ) Axial rotation of "360" degrees (or some fraction thereof). Lower member 128 L Can be around the second arm joint J A2 Pivot (e.g., when rotating). For example, the second arm joint J A2 (shown near the body 110 of the robot 100) allows the arm 126 to pitch to a certain angle (eg, raise or lower one or more members 128 of the arm 126). L At the third arm joint J A3 is coupled to the upper member 128 U , and the upper member 128 U At the fourth arm joint J A4 In some examples, such as Figure 1A , hand component 128 H or end effector 128 H The end effector 128 may be a mechanical gripper including one or more movable jaws configured to perform different types of grasping of elements within the environment 10. In the example shown, the end effector 128 HThe invention comprises a fixed first jaw and a movable second jaw, wherein the second jaw grasps an object by clamping the object between the jaws. The movable jaw can be configured to move relative to the fixed jaw so as to move between an open position of the gripper and a closed position of the gripper (e.g., closed around the object).

[0037] In some implementations, the arm 126 may include an additional joint J A , such as the fifth arm joint J A5 and / or sixth arm joint J A6 . Fifth joint J A5 Can be located on the upper member 128 U With hand component 128 H near the coupling and can be used to allow the hand member 128 H With respect to the lower member 128 U In other words, the fifth arm joint J A4 Can be used as a fourth arm joint similar to J A4 The torsion joint or hand member 128 H The wrist joint of the adjacent arm 126. For example, as a torsion joint, one member coupled at the joint J can move or rotate relative to another member coupled at the joint J (for example, a first member portion coupled at the torsion joint is fixed, while a second member portion coupled at the torsion joint rotates). Here, the fifth joint J A5 The arm 126 can also be made to rotate the hand member 128 H The hand member 128 is rotated in such a way that H It is possible to yaw instead of pitch. For example, the fifth joint J A5 The arm 126 may be allowed to twist within a "180" degree range of motion so that the hand member 128 H The associated jaws can pitch, yaw, or some combination of the two. This is useful for hooking portions of the arm 126 around an object or improving the grip of the hand member 128. H How to grasp the object may be advantageous. A6 Can be connected with the fifth arm joint J A5 Functions similarly (e.g., as a torsion joint). For example, the sixth arm joint J A6 It is also possible to allow a portion of the arm member 128 (eg, the upper arm member 128 U ) rotates or twists within a "180" degree range of motion (eg, relative to another portion of the arm member 128 or another arm member 128). Here, the fifth arm joint J A5 and the sixth arm joint J A6The combination of the range of motion of can achieve a "360" degree rotation. In some implementations, the arm 126 can be connected to the robot 100 at a socket on the body 110 of the robot 100. In some configurations, the socket can be configured as a connector so that the arm 126 can be attached or detached from the robot 100 depending on whether the arm 126 is needed to operate. In some examples, the first arm joint J A1 and the second arm joint J A2 It may be located at, adjacent to, or a portion of a socket that connects the arm 126 to the body 110 .

[0038] The robot 100 may have a vertical gravity axis (eg, shown as a Z-axis A) along the direction of gravity. Z ) and the center of mass CM, which is the point where the sum of the weighted relative positioning of the distributed mass of the robot 100 is zero. The robot 100 may also have a relative position based on CM relative to the vertical gravity axis A. Z The posture P of the robot 100 (i.e., a fixed reference frame relative to gravity) is defined to define a particular posture or stance taken by the robot 100. The posture of the robot 100 may be defined by the orientation or angular position of the robot 100 in space. Movement of the legs 120 relative to the body 110 may change the posture P of the robot 100 (i.e., a combination of the positioning of the robot's CM and the posture or orientation of the robot 100). Here, height (i.e., vertical distance) generally refers to the height along (e.g., parallel to) the z-direction (i.e., z-axis A). Z ) The sagittal plane of the robot 100 corresponds to the axis A in the y direction. Y and z-axis A Z In other words, the sagittal plane divides the robot 100 into left and right sides. Approximately perpendicular to the sagittal plane, the ground plane (also called the transverse plane) passes through the x-axis A X and the y-axis A Y The ground plane refers to a support surface 14 where the distal ends 124 of the legs 120 of the robot 100 can generate traction to help the robot 100 move around the environment 10. Another anatomical plane of the robot 100 is a frontal plane extending across the body 110 of the robot 100 (e.g., from the left side of the robot 100 with the first leg 120a to the right side of the robot 100 with the second leg 120b). The frontal plane passes through the x-axis Ax and the z-axis Ax. Z The XZ plane extends in the direction of the axis.

[0039] As a legged robot moves around an environment 10, the robot's legs 120 may undergo a gait cycle. Generally, a gait cycle begins when the leg 120 lands or contacts the support surface 14 and ends when the same leg 120 contacts the ground surface 14 again. The touchdown of the leg 120 may also be referred to as a "footfall," which defines the point or location at which the distal end 124 of the motion-based structure 120 lands in contact with the support surface 14. The gait cycle may be primarily divided into two phases, a swing phase and a stance phase. During the swing phase, the leg 120 may undergo (i) lifting off from the support surface 14 (sometimes also referred to as toe-off and the transition between the stance phase and the swing phase), (ii) flexion at the knee joint JK of the leg 120, (iii) flexion of the knee joint JK of the leg 120, K and (iv) descending (or stepping) back to the support surface 14. Here, the leg 120 in the swing phase is referred to as the swing leg 120. SW When swinging legs 120 SW Perform swing phase 120 SW , the other leg 120 performs a stance phase. The stance phase refers to the period of time when the distal end 124 (e.g., foot) of the leg 120 is on the support surface 14. During the stance phase, the leg 120 may experience (i) initial support surface contact, which triggers the transition from the swing phase to the stance phase, (ii) a load response, in which the leg 120 inhibits support surface contact, (iii) an intermediate stance support, which is used when the contralateral leg (i.e., the swing leg 120) SW ) when lifted off and swung to a balanced position (approximately halfway through the swing phase), and (iv) final stance support, which starts from the time when the robot's CM is above the leg 120 until the contralateral leg 120 lands on the support surface 14. The leg 120 in the stance phase is referred to herein as the stance leg 120. ST .

[0040] In order to maneuver around the environment 10 or perform tasks using the arm 126, the robot 100 may include a sensor system 130 having one or more sensors 132, 132a-132n. For example, Figure 1A 1 shows a first sensor 132, 132a mounted at the head of the robot 100, a second sensor 132, 132b mounted near the hip of the second leg 120b of the robot 100, a third sensor 132, 132c mounted on one side of the body 110 of the robot 100 corresponding to one of the sensors 132, a fourth sensor 132, 132d mounted near the hip of the fourth leg 120d of the robot 100, and an end effector 128 mounted on the arm 126 of the robot 100. HA fifth sensor 132, 132e is provided at or near the sensor 132. The sensor 132 may include a visual / image sensor, an inertial sensor (e.g., an inertial measurement unit (IMU)), a force sensor, and / or a motion sensor. Some examples of the sensor 132 include a camera, such as a stereo camera, a time-of-flight (TOF) sensor, a scanning light detection and ranging (LIDAR) sensor, or a scanning laser detection and ranging (LADAR) sensor. In some implementations, the respective sensor 132 may have a corresponding field of view F defining a sensing range or area corresponding to the sensor 132. v .For example, Figure 1A The field of view F of the robot 100 is depicted v Each sensor 132 may be pivotable and / or rotatable so that the sensor 132 may change its field of view F, for example, about one or more axes (eg, an x-axis, a y-axis, or a z-axis relative to a ground plane). V .

[0041] In some implementations, the sensor system 130 may include sensors 132 coupled to the joints J. In some implementations, these sensors 132 may be coupled to motors (e.g., sensors 132, 132a-b) that operate the joints J of the robot 100. Here, these sensors 132 may be based on sensor data 134 (e.g., Figure 1B The joint dynamics collected as joint-based sensor data 134 may include joint angles (e.g., the upper member 122u relative to the lower member 122 L ), joint velocity (e.g., joint angular velocity or joint angular acceleration), and / or joint torque (also referred to as joint force) experienced at joint J. Here, the joint-based sensor data 134 generated by one or more sensors 132 can be raw sensor data, data that is further processed to form different types of joint dynamics, or some combination of the two. For example, the sensor 132 can measure the joint positioning (or the positioning of the member 122 coupled at the joint J), and the system of the robot 100 can perform further processing to derive the velocity and / or acceleration from the positioning data. In other examples, one or more sensors 132 can be configured to directly measure velocity and / or acceleration.

[0042] When the sensor 132 is used to survey the field of view F V When the sensor system 130 can also generate a corresponding field of view F V The sensor system 130 may generate the field of view F using sensors 132 (eg, sensors 132a, 132b) mounted on or near the body 110 of the robot 100. vThe sensor system may additionally and / or alternatively utilize an end effector 128 mounted on the arm 126 H A sensor 132 (eg, sensor 132c) at or near the sensor 132 may be used to generate a field of view F v .

[0043] The one or more sensors 132 may capture sensor data 134 that defines a three-dimensional point cloud for an area within the environment 10 surrounding the robot 100. In some examples, the sensor data 134 may be image data corresponding to the three-dimensional volumetric point cloud generated by the three-dimensional volumetric image sensor 132.

[0044] Additionally or alternatively, the sensor system 130 may collect posture data of the robot 100, including inertial measurement data (e.g., inertial measurement data measured by an IMU), as the robot 100 maneuvers around the environment 10. In some examples, the posture data may include kinematic data and / or orientation data about the robot 100, such as kinematic data and / or orientation data about a joint J or other portion of a leg 120 or arm 126 of the robot 100. Using the sensor data 134, various systems of the robot 100 may use the sensor data 134 to define a current state of the robot 100 (e.g., a current state of the kinematics of the robot 100) and / or a current state of the environment 10 surrounding the robot 100.

[0045] As the sensor system 130 collects sensor data 134, the computing system 140 may store, process, and / or transmit the sensor data 134 to various systems of the robot 100 (e.g., the computing system 140, the control system 170, the perception system 180, and / or the navigation system 200). To perform computing tasks related to the sensor data 134, the computing system 140 of the robot 100 may include data processing hardware 142 and memory hardware 144. The data processing hardware 142 may be configured to execute instructions stored in the memory hardware 144 to perform computing tasks related to the activities (e.g., movement and / or movement-based activities) of the robot 100. In general, the computing system 140 refers to one or more instances of the data processing hardware 142 and / or the memory hardware 144.

[0046] Continue to refer Figure 1A and Figure 1BIn some implementations, the computing system 140 may be a local system located on the robot 100. When located on the robot 100, the computing system 140 may be centralized (i.e., in a single location / region on the robot 100, e.g., the body 110 of the robot 100), decentralized (i.e., located at various locations around the robot 100), or a hybrid combination of the two (e.g., with a majority of centralized hardware and a small amount of decentralized hardware). A decentralized computing system 140 may, for example, allow processing to occur at active locations (e.g., at the motors of the joints of the moving legs 120), while a centralized computing system 140 may, for example, allow a central processing hub to communicate with systems located at various locations on the robot 100 (e.g., communicating with the motors of the joints of the moving legs 120).

[0047] Additionally or alternatively, the computing system 140 may include computing resources located remotely from the robot 100. For example, the computing system 140 may communicate with a remote system 160 (e.g., a remote computer / server or a cloud-based environment) via a network 150. Much like the computing system 140, the remote system 160 may include remote computing resources, such as remote data processing hardware 162 and remote memory hardware 164. Here, sensor data 134 or other processed data (e.g., data processed locally by the computing system 140) may be stored in the remote system 160 and accessible by the computing system 140. In some embodiments, the computing system 140 may be configured to utilize the remote resources 162, 164 as an extension of the computing resources 142, 144, such that resources of the computing system 140 may reside on resources of the remote system 160.

[0048] In some implementations, such as Figure 1A and Figure 1B As shown, the robot 100 may include a control system 170 and a perception system 180. The perception system 180 may be configured to receive sensor data 134 from the sensor system 130 and process the sensor data 134 to generate one or more perception maps 182. The perception system 180 may transmit such perception maps 182 to the control system 170 in order to perform controlled actions for the robot 100, such as moving the robot 100 around the environment 10. In some implementations, by having the perception system 180 separate from the control system 170 but in communication with the control system 170, the processing of the control system 170 may be focused on controlling the robot 100, while the processing of the perception system 180 may be focused on interpreting the sensor data 134 collected by the sensor system 130. For example, these systems 170, 180 may run their processing in parallel to ensure accurate, smooth movement of the robot 100 in the environment 10.

[0049] In some implementations, the control system 170 may include one or more controllers 172, a path generator 174, a step locator 176, and a body planner 178. The control system 170 may be configured to communicate with at least one sensor system 130 and any other systems of the robot 100 (e.g., the perception system 180 and / or the navigation system 200). The control system 170 may use the hardware 140 to perform operations and other functions. The controller 172 may be configured to control the movement of the robot 100 to traverse the environment 10 based on input or feedback from the systems of the robot 100 (e.g., the control system 170, the perception system 180, and / or the navigation system 200). This may include movement between postures and / or behaviors of the robot 100. For example, the controller 172 may control different stepping patterns, leg patterns, body movement patterns, or vision system sensing patterns.

[0050] In some implementations, the controller 172 may include a plurality of controllers 172, wherein each controller 172 may be configured to operate the robot 100 at a fixed cadence. A fixed cadence refers to a fixed timing of the step or swing phase of the legs 120. For example, a separate controller 172 may instruct the robot 100 to move the legs 120 (e.g., take a step) at a particular frequency (e.g., step every 250 milliseconds, 350 milliseconds, etc.). With a plurality of controllers 172, wherein each controller 172 is configured to operate the robot 100 at a fixed cadence, the robot 100 may experience variable timing by switching between different controllers 172. In some implementations, the robot 100 may continuously switch / select a fixed cadence controller 172 (e.g., reselecting a controller 170 every three milliseconds) as the robot 100 traverses the environment 10.

[0051] In some implementations, the control system 170 may additionally or alternatively include one or more dedicated controllers 172 dedicated to a particular control purpose. For example, the control system 170 may include one or more stair controllers dedicated to planning and coordinating the movement of the robot to traverse a set of stairs. For example, the stair controller may ensure that the swing leg 120 SW The walking path of the embodiment of the present invention maintains the swing height to cross the riser and / or edge of the stairs. Other special controllers 172 may include a path generator 174, a step locator 176 and / or a body planner 178.

[0052] refer to Figure 1B , the path generator 174 can be configured to determine the horizontal motion of the robot 100. As used herein, the term "horizontal motion" refers to the translation (i.e., movement in the XY plane) and / or yaw (i.e., movement around the Z-axis A) of the robot 100. zThe path generator 174 may determine obstacles within the environment 10 surrounding the robot 100 based on the sensor data 134. The path generator 174 may determine the trajectory of the robot's body 110 for some future period (e.g., the next second). However, such determination of the trajectory of the body 110 by the path generator 174 may occur more frequently, such as hundreds of times per second. In this manner, in some implementations, the path generator 174 may determine a new trajectory for the body 110 every few milliseconds, with each new trajectory being planned for a period of about one second in the future.

[0053] The path generator 174 can transmit information about the currently planned trajectory and identified obstacles to the step locator 176 so that the step locator 176 can identify the foot drop points of the legs 120 of the robot 100 (e.g., the locations where the distal ends 124 of the legs 120 of the robot 100 are placed). The step locator 176 can generate the foot drop points (i.e., the locations where the robot 100 should step) using input from the perception system 180 (e.g., the perception map 182). The body planner 178 is very similar to the step locator 176 and can receive input from the perception system 180 (e.g., the perception map 182). In general, the body planner 178 can be configured to adjust the dynamics of the body 110 of the robot 100 (e.g., rotations, such as pitch or yaw and / or the height of the CM) to successfully move around the environment 10.

[0054] The perception system 180 can enable the robot 100 to move more accurately in terrain with various obstacles. When the sensors 132 collect sensor data 134 of the space around the robot 100 (i.e., the robot's environment 10), the perception system 180 can use the sensor data 134 to form one or more perception maps 182 of the environment 10. In some implementations, the perception system 180 can also be configured to modify an existing perception map 182 (e.g., by projecting the sensor data 134 onto a pre-existing perception map) and / or remove information from the perception map 182.

[0055] In some implementations, one or more perception maps 182 generated by the perception system 180 may include a ground height map 182, 182a, a no-step map 182, 182b, and a body obstacle map 182, 182c. The ground height map 182a refers to the perception map 182 generated by the perception system 180 based on voxels from the voxel map. In some implementations, the ground height map 182a may function such that at each XY position within a grid of the perception map 182 (e.g., designated as a cell of the ground height map 182a), the ground height map 182a specifies a height. In other words, the ground height map 182a may convey that at a particular XY position in the horizontal plane, the robot 100 should step at a certain height.

[0056] The no-step map 182b generally refers to the perception map 182, which defines the area where the robot 100 is not allowed to step, so as to inform the robot 100 when the robot 100 can step at a specific horizontal position (i.e., a position in the XY plane). In some implementations, much like the body obstacle map 182c and the ground height map 182a, the no-step map 182b can be divided into a grid of cells, where each cell represents a specific area in the environment 10 of the robot 100. For example, each cell can correspond to three square centimeters in the XY plane within the environment 10. When the perception system 180 generates the no-step map 182b, the perception system 180 can generate a Boolean map, where the Boolean map identifies no-step regions and step regions. The no-step region refers to the region of one or more cells where obstacles exist, and the step region refers to the region where one or more cells where obstacles are not perceived to exist. The perception system 180 can further process the Boolean map so that the no-step map 182b includes a signed distance field. Here, the signed distance field for no-step map 182 b may include a distance to a boundary of an obstacle (eg, a distance to a boundary of no-step region 244 ) and a vector “v” to the boundary of the obstacle (eg, defining the closest direction to the boundary of no-step region 244 ).

[0057] The body obstacle map 182c may be used to determine whether the body 110 of the robot 100 overlaps with a location in the XY plane relative to the robot 100. In other words, the body obstacle map 182c may identify obstacles of the robot 100 to indicate whether the robot 100 is at risk of collision or potential damage with an obstacle near or at the same location by overlapping at the location in the environment 10. As an obstacle map of the body 110 of the robot 100, the system of the robot 100 (e.g., the control system 170) may use the body obstacle map 182c to identify the boundaries adjacent to or closest to the robot 100 and to identify the direction (e.g., the best direction) in which the robot 100 may be moved to avoid the obstacle. In some implementations, much like other perception maps 182, the perception system 180 may generate the body obstacle map 182c based on a grid of cells (e.g., a grid of the XY plane). Here, each cell within the body obstacle map 182c may include a distance to the obstacle and a vector pointing to the nearest cell that is the obstacle (i.e., the boundary of the obstacle).

[0058] Further references Figure 1B, the robot 100 may also include a navigation system 200 and a task execution system 184. The navigation system 200 may be a system of the robot 100 that navigates the robot 100 along a path, referred to as a navigation route 202, in order to traverse the environment 100. The navigation system 200 may be configured to receive the navigation route 202 as an input or to generate the navigation route 202 (e.g., to generate the route in its entirety or some portion thereof). In order to generate the navigation route 202 and / or to guide the robot 100 along the navigation route 202, the navigation system 200 may be configured to operate in conjunction with the control system 170 and / or the perception system 180. For example, the navigation system 200 may receive the perception map 182, which may inform decisions performed by the navigation system 200 or otherwise influence some form of mapping performed by the navigation system 200 itself. The navigation system 200 may operate in conjunction with the control system 170 such that one or more controllers 172 and / or dedicated controllers 174 , 176 , 178 may control movement of components of the robot 100 (eg, legs 120 and / or arms 126 ) to navigate along the navigation route 202 .

[0059] The task execution system 184 described in further detail below may be a system of the robot 100 that is responsible for executing the recorded tasks. The recorded tasks may, for example, specify that the robot 100 is to perform tasks in the topology map 204 (e.g., Figure 2A A sequence of one or more actions performed at corresponding waypoints 212 defined on the path (as shown).

[0060] In addition, Figure 1B As shown in , in some implementations, the robot controller 188 can communicate wirelessly (or wired) with the robot 100 (via the network 150 or otherwise) and can allow an operator to control the robot 100. In some implementations, the robot controller 188 can be a tablet computer with "soft" UI controls for the robot 100 presented via the tablet computer's touch screen. In other implementations, the robot controller 188 can take the form of a traditional video game controller, but may include a display screen, and may include various physical buttons and / or soft buttons that can be pressed or otherwise manipulated to control the robot 100.

[0061] In some implementations, an operator may use the robot controller 188 to initiate a mission recording process. During such a process, the operator may (e.g., via the robot controller 188) direct the movement of the robot 100 and instruct the robot 100 to take various “mission actions” (e.g., acquire sensor readings, monitor video, etc.) along the desired path of the mission. As the mission is recorded, the robot 100 may generate a topology map 204 (e.g., Figure 2A), which includes waypoints 212 at various locations along its path, and edges 214 between such waypoints 212. In some implementations, for each task action that the operator instructs the robot to perform, a new waypoint 212 can be added to the topology map 204 being generated on the robot 100. In addition, for each such task action, data can be stored in the topology map 204 and / or the task record to associate the task action identified in the task record with the waypoint 212 of the topology map 204 in which the task action is performed. In some implementations, at the end of the task recording process, the topology map 204 generated during the task recording can be transmitted to the robot controller 188 and stored in association with the task record. In some implementations, other information associated with the robot's movement in the environment during the task recording process can also be stored. For example, a group of landing positions of the robot can be stored in the topology map 204 and / or the task record. As discussed in more detail below, the stored landing position group can be used in conjunction with sensor data to facilitate deciding whether to step onto an object in the robot's environment during subsequent execution of the task. After the task recording process, the task record and the associated topology map 204 (if not already present on the robot 100 ) may then be provided to the robot 100 , and the robot 100 may be instructed (eg, autonomously) to run the recorded task.

[0062] Now refer to Figure 2A-2F A detailed description of the route runner 220 of the navigation system 200 is provided. As described above, the navigation route 202 executed by the route runner 220 may include a sequence of instructions for moving the robot 100 along a path, the path corresponding to a sequence of waypoints 212 defined on the topological map 204 (e.g., Figure 2A As the route runner 220 guides the robot 100 through movement following the navigation route 202, the route runner 220 may determine whether the navigation route 202 becomes blocked by an object. As described above, in some implementations, the navigation route 202 may include one or more features of the topology map 204. For example, as previously described, such a topology map 204 may include waypoints 212 and edges 214, and the navigation route 202 may indicate that the robot 100 is to travel along a path that includes a particular sequence of those waypoints 212. In some implementations, the navigation route 202 may also include movement instructions that specify how the robot 100 moves from one waypoint 212 to another waypoint. Such movement instructions may, for example, take objects or other obstacles into account when recording the waypoints 212 and edges 214 to the topology map 204.

[0063] Because the environment 10 can change dynamically from the time the waypoint 212 is recorded to the topology map 204, the route runner 220 can be configured to determine whether the navigation route 202 becomes blocked by an object that was not previously identified when the waypoint 212 on the topology map 204 used by the navigation route 202 was recorded. Such an object can be considered an "unforeseen obstacle" in the navigation route 202 because the initial map drawing process that informed the navigation route 202 did not identify the object in the location of the blocked object. This may occur, for example, when an object is moved or introduced into the drawing environment.

[0064] like Figure 2A As shown, when an unforeseen obstacle blocks the navigation route 202, the route runner 220 may attempt to generate an alternative path 206 to another feature on the topology map 204 that avoids the unforeseen obstacle. The alternative path 206 may temporarily deviate from the navigation route 202, but then resume the navigation route 202 after the deviation. Unlike other methods of generating obstacle avoidance paths, the route runner 220 attempts to only temporarily deviate from the navigation route 202 to avoid the unforeseen obstacle so that the robot 100 can return to using road features of the navigation route 202 (e.g., similar to the topology features from the topology map 204). In this sense, successful obstacle avoidance by the route runner 220 occurs when the obstacle avoidance path (i) avoids the unforeseen obstacle and (ii) enables the robot 100 to resume a portion of the navigation route 202. This technique of merging with the navigation route 202 after obstacle avoidance may be advantageous because the navigation route 202 may be important to the work or task performance of the robot 100 (or the operator of the robot 100). For example, the operator of the robot 100 may have assigned the robot 100 to perform inspection work at waypoint 212 of the navigation route 202. The navigation system 200 is intended to facilitate the success of the robot 100's work or mission by generating an obstacle avoidance route that continues on the navigation route 202 after obstacle avoidance.

[0065] To illustrate, Figure 1AA robot 100 is depicted traveling along a navigation route 202 that includes three waypoints 212a-212c. While moving along a first portion of the navigation route 202 (e.g., shown as a first side 214a) from a first waypoint 212a to a second waypoint 212b, the robot 100 encounters an unforeseen obstacle 20 depicted as a partial pallet of boxes. The unforeseen obstacle 20 blocks the robot 100 from completing the first portion of the navigation route 202 to the second waypoint 212b. Here, the "X" above the second waypoint 212b indicates that the robot 100 cannot successfully travel to the second waypoint 212b given the pallet of boxes. As depicted, the navigation route 202 will typically have a second portion (e.g., shown as a second side 214b) extending from the second waypoint 212b to the third waypoint 212c. However, due to the unforeseen object 20, the route runner 220 generates an alternate path 206 that directs the robot 100 to move to avoid the unforeseen obstacle 20 and travel to a third waypoint 212c of the navigation route 202 (e.g., from a point along the first portion of the navigation route 202). In this regard, the robot 100 may not be able to successfully navigate to one or more waypoints 212, such as the second waypoint 212b, but may resume a portion of the navigation route 202 after avoiding the obstacle 20. For example, the navigation route 202 may include additional waypoints 212 after the third waypoint 212c, and the alternate path 206 may enable the robot 100 to continue to those additional waypoints 212 after the navigation system 200 directs the robot 100 to the third waypoint 212c via the alternate path 206.

[0066] like Figure 2AAs shown, and as briefly noted above, the navigation system 200 may include a navigation generator 210 operating in conjunction with a route generator 220. The navigation generator 210 (also referred to as the generator 210) may be configured to construct a topology map 204 (e.g., during a mission recording process) and to generate a navigation route 202 based on the topology map 204. To generate the topology map 204, the navigation system 200, and more specifically, the generator 210, may record sensor data corresponding to locations within the environment 10 that the robot 100 has traversed or is traversing as waypoints 212. As described above, the waypoints 212 may include representations of what the robot 100 sensed at a particular location within the environment 10 (e.g., according to its sensor system 120). The generator 210 may generate the waypoints 212, for example, based on image data 134 collected by the sensor system 130 of the robot 100. For example, the robot 100 may perform an initial mapping process in which the robot 100 moves through the environment 10. While moving through the environment 10, systems of the robot 100 (such as the sensor system 130) may collect data (e.g., sensor data 134) as a means of understanding the environment 10. By gaining an understanding of the environment 10 in this manner, the robot 100 may later use the information gathered from the initial mapping process, or derivatives thereof, to move around the environment 10 (e.g., autonomously, semi-autonomously, or with the assistance of a user).

[0067] In some implementations, the navigation generator 210 may construct the topology map 204 by running at least one waypoint heuristic (e.g., a waypoint search algorithm) that triggers the navigation generator 210 to record a waypoint drop at a particular location in the topology map 204. For example, such a waypoint heuristic may be configured to detect a threshold feature detection within the image data 134 at the location of the robot 100 (e.g., when generating or updating the topology map 204). The navigation generator 210 (e.g., using the waypoint heuristic) may identify features within the environment 10 that serve as reliable visual sensor features that provide repeatability for the robot 100 to maneuver around the environment 10. For example, the waypoint heuristic of the generator 210 may be pre-programmed for feature recognition (e.g., programmed with stored features), or programmed to identify features of a spatial cluster of occurrences of the volumetric image data 134 (e.g., a corner of a room or an edge of a wall). In response to at least one waypoint heuristic triggering a waypoint drop, the navigation generator 210 may record the waypoint 212 on the topology map 204. The navigation generator 210 can repeat the waypoint identification process as the robot 100 drives through the area (e.g., the robot environment 10). For example, an operator of the robot 100 can manually drive the robot 100 through the area used for the initial mapping process that established the waypoints 212 for the topological map 204.

[0068] As each waypoint 212 is recorded, the generator 210 may associate waypoint edges 214 (also referred to as edges 214) with consecutive pairs of corresponding waypoints 212, such that the topological graph 204 generated by the generator 210 includes waypoints 212 and edges 214 between pairs of those waypoints 212. An edge 214 may indicate how one waypoint 212 (e.g., a first waypoint 212a) is related to another waypoint 212 (e.g., a second waypoint 212b). For example, an edge 214 may represent a positioning relationship between a pair of adjacent waypoints 212. In other words, an edge 214 may represent a connection between two waypoints 212 or specify a path (e.g., Figure 2A The edge 214a shown in FIG. 2 may represent a connection between the first waypoint 212a and the second waypoint 212b).

[0069] In some implementations, each edge 214 can therefore represent a path between a pair of waypoints 212 that it interconnects (e.g., a movement path of the robot 100). In addition, in some implementations, each edge 214 can also reflect additional useful information. In particular, the route runner 220 of the navigation system 200 can be configured to recognize specific annotations on the edges 214 and control other systems of the robot 100 to take actions indicated by these annotations. For example, one or more edges 214 can be annotated to include movement instructions that inform the robot 100 how to move or navigate between their interconnected waypoints 212. Such movement instructions can, for example, identify a pose transformation of the robot 100 before the robot 100 moves along the edge 214 between the two waypoints 212. Therefore, the pose transformation can describe one or more positions and / or orientations that the robot 100 assumes to successfully navigate along the edge 214 between the two waypoints 212. In some implementations, the edge 214 can be annotated to specify a complete three-dimensional pose transformation (e.g., six numbers). Some of these numbers represent estimates, such as dead reckoning pose estimates, vision-based estimates, or other estimates based on kinematic and / or inertial measurements of the robot 100 .

[0070] In some implementations, one or more edges 214 may additionally or alternatively include annotations that further provide an indication / description of the environment 10. Some examples of annotations include descriptions or indications that the edge 214 is associated with or located on a certain feature of the environment 10. For example, an annotation of an edge 214 may specify that the edge 214 is located on a staircase or through a doorway. Such annotations may assist the robot 100 during maneuvers, particularly when visual information is lost or lacking (e.g., due to the presence of a doorway). In some configurations, the edge annotations may additionally or alternatively identify one or more directional constraints (which may also be referred to as "pose constraints"). Such directional constraints may, for example, specify an alignment and / or orientation (e.g., pose) of the robot 100 to enable it to navigate over or through a particular environmental feature. For example, such an annotation may specify a particular alignment or pose that the robot 100 is to assume before going up or down a staircase or down a narrow hallway, which may restrict the robot 100 from turning.

[0071] In some implementations, the sensor data 134 may be associated with each waypoint 212 of the topology map 204. The sensor system 130 of the robot 100 may have collected such sensor data 134 when the generator 210 recorded the corresponding waypoint 212 to the topology map 204. The sensor data 134 stored for each waypoint 212 may enable the robot 100 to locate by comparing the real-time sensor data 134 collected while the robot 100 traverses the environment 10 according to the topology map 204 (e.g., via the route 202) with the sensor data 134 stored for the waypoint 212 of the topology map 204. In some configurations, after the robot 100 moves along the edge 214 (e.g., with the goal of reaching the target waypoint 212), the robot 100 may locate by directly comparing the real-time sensor data 134 with the sensor data 134 associated with the expected target waypoint 212 of the topology map 204. In some implementations, by storing raw or near-raw sensor data 134 (i.e., data with minimal processing) for waypoints 212 of the topological map 204, the robot 100 can use the real-time sensor data 134 to effectively locate itself as the robot 100 maneuvers within the mapped environment 10. In some examples, an iterative closest point (ICP) algorithm can be used to locate the robot 100 relative to a given waypoint 212.

[0072] By using the waypoints 212 and the edges 214 to generate the topology map 204, the topology map 204 can be locally consistent (e.g., spatially consistent within the region due to adjacent waypoints), but does not need to be globally accurate and / or consistent. That is, as long as the geometric relationship between the adjacent waypoints 212 (e.g., the edges 214) is roughly accurate, the topology map 204 does not require accurate global metric positioning of the robot 100 and any sensed objects within the environment 10. Therefore, the navigation route 202 derived or constructed using the topology map 202 does not require accurate global metric information. In addition, because the topology map 204 can be constructed based on the waypoints 212 and the relationship between the waypoints (e.g., the edges 214), the topology map 204 can be considered an abstract or high-level map, rather than a metric map. That is, in some implementations, the topology map 204 may lack other metric data about the mapping environment 10 that is not related to the waypoints 212 or their corresponding edges 214. For example, in some implementations, the drawing process (e.g., performed by the generator 210) that creates the topology map 204 may not store or record other metric data, and / or the drawing process may remove the recorded metric data to form the topology map 204 of the waypoints 212 and the edges 214. In either case, navigating using the topology map 204 can simplify the hardware required for navigation and / or the computing resources used during navigation. That is, when compared to navigation using metric positioning that typically requires expensive LIDAR sensors and / or expensive IMU sensors, topology-based navigation can operate using low-cost vision and / or low-cost inertial measurement unit (IMU) sensors. Metric-based navigation tends to require more computing resources than topology-based navigation because metric-based navigation typically performs positioning (e.g., using waypoints 212) at a much higher frequency than topology navigation. For example, a common navigation method of simultaneous localization and mapping (SLAM) using a global occupancy grid continuously performs robot positioning.

[0073] refer to Figure 2A, the navigation generator 210 may record a plurality of waypoints 212, 212a-n on the topological map 204. From the plurality of recorded waypoints 212, the navigation generator 210 may select a number of the recorded waypoints 212 as a sequence of waypoints 212 that form a navigation route 202 for the robot 100. In some implementations, an operator of the robot 100 may use the navigation generator 210 to select or construct a sequence of waypoints 212 to form the navigation route 202. In some implementations, the navigation generator 210 may generate the navigation route 202 based on receiving a destination location and a starting location of the robot 100. For example, the navigation generator 210 may match the starting location with the nearest waypoint 212, and similarly match the destination location with the nearest waypoint 212. The navigation generator 210 may then select a number of waypoints 212 between these nearest waypoints 212 to generate the navigation route 202.

[0074] In some configurations, the navigation generator 210 may receive as input a mission record and possibly an associated topology map 204, such as from a mission execution system 184, and in response, may generate a navigation route 202 that includes various waypoints 212 included in the mission record, as well as intermediate waypoints 212 and edges between pairs of waypoints 212. For example, for a mission to inspect different locations on a pipeline, the navigation generator 210 may receive a mission record identifying waypoints 212 at which inspections are to occur and a topology map 204 generated during the recording process, and may generate a navigation route 202 that includes waypoints 212 consistent with the identified inspection locations. Figure 2A In the example shown, the navigation generator 210 has generated the navigation route 202 having a sequence of waypoints 212 including nine waypoints 212a - 212i and their corresponding edges 214a - 214h . Figure 2A Each waypoint 212 of the navigation route 202 is shown with a double circle, while recorded waypoints 212 that are not part of the navigation route 202 have only a single circle. The navigation generator 210 can then transmit the navigation route 202 to the route runner 220 as shown.

[0075] The route runner 220 may be configured to receive and execute the navigation route 202. To execute the navigation route 202, the route runner 220 may coordinate with other systems of the robot 100 to control the motion-based structures (e.g., legs) of the robot 100 to drive the robot 100 through the sequence of waypoints 212 included in the navigation route 202. For example, the route runner 220 may transmit movement instructions to the control system 170 that are associated with an edge 214 connecting the waypoints 212 in the sequence of waypoints 212 of the navigation route 202. The control system 170 may then use such movement instructions to position the robot 100 (e.g., in an orientation) according to one or more posture transformations to successfully move the robot 100 along the edge 214 of the navigation route 202.

[0076] As the robot 100 travels along the navigation route 202, the route runner 220 may also determine whether the robot 100 is unable to execute a particular move instruction for a particular edge 214. For example, the robot 100 may be unable to execute the move instruction for the edge 214 because the robot 100 encounters an unforeseen obstacle 20 while moving to the waypoint 212 along the edge 214. Here, the route runner 220 may identify that the unforeseen obstacle 20 is blocking the path of the robot 100 (e.g., using real-time or near real-time sensor data 134), and may be configured to determine whether an alternate path 206 exists for the robot 100 to the sequence of untraveled waypoints 212, 212U in the navigation route 202. An untraveled waypoint 212U refers to a waypoint 212 of the navigation route 202 to which the robot 100 has not yet successfully traveled. For example, if the robot 100 has traveled to three waypoints 212a-212c of the nine waypoints 212a-212i of the navigation route 202, the route runner 220 may attempt to find an alternative path 206 to one or the remaining six waypoints 212d-i, if possible. In this sense, the alternative path 206 may be an obstacle avoidance path that avoids unforeseen obstacles 20, and also a path that allows the robot 100 to resume the navigation route 202 (e.g., toward a specific goal or work). This means that after the robot 100 travels along the alternative path 206 to the destination of the untraveled waypoint 212U, the route runner 220 may continue to run the navigation route 202 from the destination of the alternative path 206. Such an approach may enable the robot 100 to return to navigation using the sparse topology map 204.

[0077] For example, refer to Figure 2A, if the unforeseen obstacle 20 blocks a portion of the third side 214c (e.g., blocks some portions of the third side 214c and the fourth waypoint 212d), the robot 100 has already traveled to three waypoints 212a-212c. In this case, the route runner 220 can generate an alternative path 206 to the fifth waypoint 212e that avoids the unforeseen obstacle 20, and the fifth waypoint 212e is the untraveled waypoint 212U. The robot 100 can then continue to traverse the sequence of waypoints 212 of the navigation route 202 from the fifth waypoint 212e. This means that the robot 100 will then follow the sequence of waypoints 212 of the navigation route 202 to travel to the untraveled portion (e.g., by using the movement instructions of the side 214 of the untraveled portion). In the example shown, the robot 100 will therefore travel from the fifth waypoint 212e to the sixth, seventh, eighth, and finally ninth waypoints 212, 212f-212i unless some other unforeseen object 20 is detected. This means that, despite the presence of the unforeseen object 20 along the third side 214c, the robot 100 has only missed a single waypoint during its movement path while executing the navigation route 202, namely the fourth waypoint 212d.

[0078] In some implementations, when the route runner 220 determines that an unforeseen obstacle 20 blocks an edge 214, the route runner 220 may determine that the topology map 204 fails to provide an alternative route 206 that avoids the unforeseen obstacle 20. This is often the case because the topology map 204 includes waypoints 212 and edges 214 that were recorded during the drawing process (e.g., by the navigation generator 210). Since the unforeseen obstacle 20 did not exist at the time of drawing, the topology map 204 may not be able to generate the alternative path 206 on its own. In other words, the navigation generator 210 does not anticipate the need for an alternative route 206 similar to the one described above. Figure 2A 204, i.e., from the third waypoint 212c to the fifth waypoint 212e. This also means that the alternative path 206 may be a path that does not correspond to an existing edge 214 in the topology map 204. In other words, the alternative path 206 results in a path between two waypoints 212 that were not previously connected (e.g., by the edge 214) in the navigation route 202. In other implementations, the route operator 220 may assume that the presence of an unforeseen obstacle 20 requires the route operator 220 to use other information besides the topology map 204 to generate the alternative path 206.

[0079] In some configurations, when the edge 214 is blocked by an unforeseen object 20, the route runner 220 may resort to other maps available from the system of the robot 100. In some implementations, for example, the route runner 220 may use or generate a local obstacle map 222 (e.g., Figure 2B 212U). Such a local obstacle map 222 may, for example, be a more detailed map of the environment 10 than the topological map 204, but only for a local area around the robot 100 (e.g., a three-meter by three-meter square area). In some configurations, the local obstacle map 222 may include an occupancy grid, where each cell within the grid specifies whether an obstacle exists in that cell. The route runner 220 may then use the unoccupied cells of the occupancy grid in conjunction with the locations of the untraveled waypoints 212U to generate an alternative path 206. In some implementations, the local obstacle map 222 may be formed in whole or in part using one or more of the perception maps 182 (e.g., a ground height map 182a, a no-step map 182b, and / or a body obstacle map 182c) from the perception system 180 for a local area around the robot 100.

[0080] Using a local obstacle map 222 of limited size, the route operator 220 can determine which untraveled waypoint 212U should be the destination of the alternative path 206 by determining which untraveled waypoints 212U exist within the boundaries of the local obstacle map 222. Figure 2B As shown, the navigation route 202 may include a sequence of five waypoints 212a-212e. As can be seen in this example, the robot 100 has traveled to the second waypoint 212b, only to discover an unforeseen obstacle 20 blocking the second side 214b connecting the second waypoint 212b and the third waypoint 212c. In order to determine the alternative path 206 (e.g., the destination of the alternative path 206), the route operator 220 can determine which untraveled waypoints 212U exist within the boundaries of the local obstacle map 222. In the example shown, the untraveled waypoints 212U include the third, fourth, and fifth waypoints 212c-212e, and only the fourth waypoint 212d is within the boundaries of the local obstacle map 222. Figure 2BThe local obstacle map 222 is shown as a grid. Although the third waypoint 212c and the fifth waypoint 212e are also untraveled waypoints 212U in addition to the fourth waypoint 212d, none of those waypoints 212c,e are located within the boundary of the local obstacle map 222. As such, the local obstacle map 222 may lack sufficient information to enable the route runner 220 to generate an alternative path 206 to either the third waypoint or the fifth waypoint 212c,e. However, since the fourth waypoint 212d is within the boundary of the local obstacle map 222, the route runner 220 can generate an alternative path 206 that avoids the obstacle 20 to reach the fourth waypoint 212d. The robot 100 can then proceed along the fourth side 214d following the navigation route 202 from the fourth waypoint 212d to the fifth waypoint 212e.

[0081] In some examples, the route operator 220 can function in an organized manner such that, for each untraveled waypoint 212U, the route operator 220 identifies whether the corresponding untraveled waypoint 212U exists within the local obstacle map 222. For example, the route operator 220 can perform this identification for each untraveled waypoint 212U that sequentially follows the sequence of waypoints of the navigation route 202. Figure 2B For example, this would mean that route operator 220 would first determine whether third waypoint 212c is within local obstacle map 222, which it is not. Route operator 220 would then determine whether fourth waypoint 212d is within local obstacle map 222, which it is. Route operator 220 would then determine whether fifth waypoint 212e is within local obstacle map 222. Figure 2B As shown, the fifth waypoint 212e is not within the boundary of the local obstacle map 222, but is within Figure 2C , the fifth waypoint 212e is located within the boundary of the local obstacle map 222. Figure 2C In the case where more than one untraveled waypoint 212U is within the boundary of the local obstacle map 222, the route runner 220 ensures that the robot 100 travels to as many waypoints 212 of the navigation route 202 as possible. Using this criterion, the route runner 220 can generate an alternative path 206 to the fourth waypoint 212d instead of the fifth waypoint 212e to ensure that the robot 100 travels to the most likely waypoint 212 of the navigation route 202 during its movement path.

[0082] Figure 2D2 shows a scenario that may occur where there are no untraveled waypoints 212U within the boundaries of the local obstacle map 222. In this case, the route runner 220 can be configured to generate an exploration path 224 in the direction of the next waypoint 212 in the sequence of waypoints 212 of the navigation route 202 that avoids the unforeseen obstacles 20. Figure 2D In the example of , an unforeseen obstacle 20 blocks the second side 214b between the second waypoint 212b and the third waypoint 212c. Because the third waypoint 212c is the next untraveled waypoint 212U in the sequence of waypoints 212 of the navigation route 202, the route runner 220 can generate an exploration path 224, wherein the exploration path 224 is an obstacle avoidance path toward the third waypoint 212c. As the robot 100 moves along the exploration path 224, the local obstacle map 222 can continue to cross its limited area. This means that an untraveled waypoint 212U that was previously not within the boundary of the local obstacle map 222 may become within the boundary of the local obstacle map 222. In this regard, the robot 100 can explore along the exploration path 224 until it is found that the untraveled waypoint 212U exists within the boundary of the local obstacle map 222. Once the untraveled waypoint 212U exists within the boundaries of the local obstacle map 222, the route runner 220 can generate an alternative path 206 to the destination with the untraveled waypoint 212U. By traveling along the exploration path 224 toward the next waypoint 212 in the sequence of waypoints 212 of the navigation route 202, the route runner 220 can minimize the number of waypoints 212 that the robot 100 will not travel to (or maximize the number of waypoints 212 that the robot 100 achieves).

[0083] Figure 2F Similar to Figure 2C , because the boundary of the local obstacle map 222 includes more than one untraveled waypoint 212U. However, except for this case, Figure 2FThe example also depicts how the route runner 220 can identify multiple unforeseen obstacles 20, 20a-20b. Here, the first unforeseen obstacle 20, 20a blocks the second side 214b and the fifth side 214e (e.g., completely blocks a room or hallway). At the same time, the second obstacle 20, 20b does not block the side 214, but forces the alternative path 206 to consider that its presence at least partially blocks the shortest path between the second waypoint 212b and the sixth waypoint 212f. The example also shows that the alternative path 206 can result in skipping multiple waypoints 212 (e.g., the third, fourth, and fifth waypoints 212u, 212c-212e). Because the route runner 220 generates the alternative path 206 from the second waypoint 212b to the untraveled sixth waypoint 212u, 212f, the robot 100 can continue along the navigation route 202 from the sixth waypoint 212f along the sixth side 214f ​​to the seventh waypoint 212g.

[0084] In the above description, during the execution of the mission, the route runner 220 may recognize that the obstacle 20 blocks the path of the robot 100 (e.g., using real-time or near real-time sensor data 134), and the robot 100 may change the route to bypass the obstacle 20 to continue along the route specified by the mission record. The obstacle 20 may be a relatively large object (e.g., greater than 30 cm in height) that the robot may not be able to pass through (e.g., step onto it), thereby preventing the robot from advancing along the route specified in the mission record. In some embodiments, the object detected as an obstacle 20 may be of the order of magnitude of the height of the robot's body.

[0085] The inventors have recognized and appreciated that, in addition to larger obstacles, there may be one or more smaller objects (e.g., less than 30 cm in height) in the environment that do not necessarily impede the robot 100 from following the route recorded in the mission (e.g., because the robot 100 may be able to traverse rough terrain and / or may otherwise be able to step over some objects in its path), but may still be objects that the robot 100 is expected to navigate around (e.g., using one or more of the rerouting techniques described herein), even if the robot 100 is able to step onto such objects. Non-limiting examples of such objects include, but are not limited to, pallets, hard hats, buckets, sections of building materials, and forklift tines. Such objects may pose a hazard to the robot, potentially preventing the robot from completing its mission. Some objects in the environment may be fragile equipment that may be damaged if the robot steps onto them. The inventors have recognized that in many environments, it is not practical or easy to always keep the robot's path away from objects, especially when the robot is autonomously navigating to run a previously recorded mission. To this end, some embodiments of the present disclosure describe techniques for distinguishing objects in an environment that the robot is allowed to step onto (e.g., stairs, above pipes) from objects that the robot should not step onto and should instead go around.

[0086] like Figure 3 As shown, some embodiments of the present disclosure include a ground clutter module 300 that uses information from the perception system 180 (e.g., one or more perception maps generated based on sensor data received from the sensor system 130) and stored data indicating a set of stepping locations of the robot from a previously run mission (e.g., when the mission was recorded) to identify objects in the environment that can be stepped on by the robot 100, but these objects should be considered obstacles that the robot should change its route around. As described in more detail below, the ground clutter module 300 generates a map of "ground clutter" obstacles in the local area around the robot 100. One or more perception maps of the terrain around the robot (e.g., the ground height map 182a) can be used together with information about where the robot stepped during a previous run of the mission (e.g., during the recording of the mission) to identify ground clutter obstacles that should be added to the set of obstacles (along with larger objects blocking the robot's path) that the robot should change its route around during the run of the mission.

[0087] Figure 4 A process 400 for identifying ground clutter obstacles during the execution of a mission according to some embodiments is shown. In some embodiments, the process 400 is performed by Figure 3. In action 410, a perception map of the robot 100's environment (also referred to herein as a "terrain map") is received from the perception module 180 and segmented into regions to identify groups of potential ground clutter obstacles in the robot's local environment. The terrain map received from the perception module 180 may include one or more of the perception maps 182 described herein (e.g., the ground height map 182a), or may include other suitable perception maps that can be used to characterize environmental features. The terrain map described in more detail below Figure 5 One implementation for performing segmentation of a terrain map according to some embodiments is shown. Process 400 then proceeds to act 420, where segments of the terrain map associated with potential ground clutter obstacles identified in act 410 are classified based at least in part on stored data indicating the robot's step locations recorded during at least one previous run of the task (e.g., during recording of the task). Figure 6 One implementation for performing classification of the identified segments according to some embodiments is shown. For example, during segment classification, it may be assumed that if the robot stepped onto an object (e.g., a staircase) during a previous run of a task (e.g., when an operator was controlling the robot to record the task), then it should be acceptable to step onto the same object during the current run of the task.

[0088] The one or more ground clutter obstacles identified in act 420 may be combined with any larger obstacles identified by perception system 180 as obstacle information for the robot's local environment. For example, the obstacle information may be represented in an obstacle map that is provided to navigation system 200 in act 430 of process 400. Navigation system 200 may then use, for example, the above-described combined Figure 2B to Figure 2F One or more of the described obstacle avoidance techniques enables the robot to reroute around obstacles identified in the obstacle information.

[0089] Figure 5An implementation of process 410 according to some embodiments is shown for segmenting a terrain map of a local environment of a robot to identify sections of the terrain map that include potential ground debris obstacles. In action 510, the terrain map is processed to detect boundaries between separate objects in the map. For example, the terrain map may include information about the height of objects in the robot's environment, and boundaries may be detected in the terrain map as local height changes that exceed a threshold. In some embodiments, the threshold is in the range of 3-10 cm. In some embodiments, the threshold is in the range of 4-8 cm. In some embodiments, the threshold is 6 cm. The inventors have recognized that the terrain map may include incomplete information describing the robot's environment. In order to at least partially account for such incomplete information, in some embodiments, connected components of portions identified as boundaries in the terrain map may be used to detect boundaries in action 510. Figure 7 A portion of a topographic map is shown from which different boundaries have been detected. Figure 7 A boundary associated with a first object 710 located on the ground in front of the robot and a boundary associated with a second object 720 located on the ground next to the robot are detected in the terrain map.

[0090] Process 410 then proceeds to act 520 where one or more surfaces in the topographic map are detected. For example, the floor on which the robot travels may be a first detection surface, the top of a pallet in the environment may be a second detection surface, and the top of a box resting on the floor may be a third detection surface. In some embodiments, connected components identified as portions of a surface in the topographic map may be used to detect boundaries in act 510 to at least partially account for incomplete information in the topographic map. Figure 8 A portion of a topographic map is shown from which a surface has been detected. For example, Figure 8 The ground surface 810 and the pallet top surface 820 are detected in the topographic map.

[0091] Although actions 510 and 520 are shown as being performed serially, it should be understood that actions 510 and 520 may be performed in full or in part in parallel and in any order. For example, a group of surfaces in the terrain (e.g., including a floor / ground and one or more surfaces above the ground) and boundaries connecting the surfaces may be detected.

[0092] Then, process 410 proceeds to action 530, where a group of segments of the terrain map including potential clutter obstacles is identified based on the detected boundaries and surfaces. The connected components between the detected boundaries and the detected surfaces can be identified separately to account for the fact that in some instances, only the boundary between two objects can be observed (e.g., when the top surface of the object is occluded or unclear), while in other instances, only the surface, but not the boundary between them can be observed (e.g., when there is an unobserved area between the two surfaces due to, for example, occlusion). In some embodiments, segments separated by a small occlusion distance can be combined. For example, in some embodiments, segments separated by a distance of 3cm-10cm, a distance of 4cm-8cm, or a distance of 6cm can be combined. Then, process 410 proceeds to action 540, where segments with low confidence (e.g., segments that are too small and / or segments that do not have too many observed boundaries) can be discarded from the group of segments. For example, in some embodiments, segments less than 500cm can be discarded from the group of segments. 2 Sections less than 300cm 2 Section or less than 200cm 2 In addition, in some embodiments, segments with observed boundaries less than 40 cm, less than 30 cm, or less than 20 cm may be discarded from the set of segments. It will be appreciated that additional and / or alternative factors may also be used to determine whether a segment has low confidence and should be discarded. For example, some objects (e.g., people or other robots) moving around the environment may be modeled, and it may be determined that segments corresponding to such objects have low confidence and may be discarded. Each segment in the set of segments may then be classified based at least in part on the stored step information to identify one or more ground clutter obstacles, such as reference Figure 6 described.

[0093] Figure 6 One implementation of process 420 for classifying a segment as a ground clutter obstacle is illustrated in accordance with some embodiments. In act 610, the segment is classified as a ground clutter obstacle as described above with reference to Figure 5The group of segments detected from the topographic map is provided as input to the classification process 420. In some embodiments, the group of segments can be represented as a segmented map (e.g., a segmented version of the topographic map). Each segment in the group represents a potential ground clutter obstacle in the group of potential ground clutter obstacles, and is classified based at least in part on stored data indicating the landing position of the robot during one or more previous runs of the task, as further described below. Then, the process 420 proceeds to action 620, where it is determined whether there are more segments in the group to be classified. If it is determined that there are more segments to be classified, the process 420 proceeds to action 630, where it is determined whether the robot has previously stepped to a position in the environment corresponding to the position where the segment is located, based at least in part on stored data indicating the landing position of the robot during previous runs of the task. Fig. 9 An example of a segmentation map 910 is illustrated, which includes an overlay of a group of step locations 920 indicating locations where a robot stepped during a previous run of a task (e.g., during task recording). As can be observed in the segmentation map 910, several surfaces have been identified including a ground surface 930 and a stair surface 940. In one implementation, each upcoming pair of step locations 920a, 920b (also referred to as a "stance" of the robot) that has not yet been traversed by the robot along the route in the task recording is compared to the segmentation map to determine whether the robot previously stepped onto the segment during a previous run of the task (e.g., during task recording).

[0094] In some embodiments, a comparison between the robot's position based on the upcoming step information and the segmented terrain map is used to check for large height differences (e.g., distances too large to be positioning errors for the step information and / or segment position). In some embodiments, a comparison between the robot's position based on the upcoming step information and the segmented terrain map is used to check for large height change differences (e.g., new climbs in previously flat terrain). In either case (i.e., when a large height difference is detected or a large height change difference is detected), it can be determined that the potential ground debris obstacle did not exist during the previous run of the task, and therefore, it was not necessarily stepped on by the robot during the previous run of the task. In some embodiments, the height change can be the absolute difference (e.g., 20 cm, 30 cm, 40 cm) between the upcoming step information and the segmented terrain map. In other embodiments, the height change can be the relative height change between the step pairs represented in the upcoming step information and the segmented terrain map. For example, the estimated z positioning of the robot can drift as it walks. For example, based on the step information stored at the time of recording, the robot may believe that it walked down a very gentle slope, and the segmented terrain map may indicate that the environment has a very gentle uphill slope. In this case, there may be a relatively large height difference between the step information and the segmented terrain map, which may not indicate an obstacle. However, if based on the step information stored at the time of recording, the robot believes that it climbed 6 cm, but the segmented terrain map shows relatively flat terrain, this may indicate that an obstacle has been removed since the time of recording (e.g., in this case, an object has been removed from the path).

[0095] In some embodiments, in addition to the step information, stored data other than the step information may be used to classify potential ground clutter obstacles. For example, one or more perceptual maps of the environment recorded during a previous run of the mission (e.g., during recording of the mission) may be used to classify one or more of the potential ground clutter obstacles as obstacles.

[0096] If it is determined in action 630 that the segment was stepped on by the robot during the previous operation of the task, the process 420 proceeds to action 640, where it is determined that the segment is not a ground debris obstacle. In this way, potential obstacles in the environment that have been stepped on by the robot during the previous operation of the task (for example, stepped on by the robot during the task recording when the operator navigates the robot) can be excluded as obstacles that should cause the robot to change the route. Using the stored data indicating the landing position of the robot during the previous operation of the task for the classification of ground debris obstacles, the classification is simplified to a binary problem by not needing to identify the object to be classified (for example, it is a box, a pallet or an expensive precision tool). On the contrary, it is assumed that when the robot is advancing along the route during the current operation of the task, it is acceptable for the robot to step on the object that was stepped on by it in the environment during the previous operation of the task. Then, the process 420 returns to action 620, where it is determined whether there are more segments to be classified.

[0097] If it is determined in action 630 that the segment has not been stepped onto during the previous run of the task, process 420 proceeds to action 650, where it is determined that the segment is a ground debris obstacle that should not be stepped onto by the robot during the current run of the task. In this case, the potential ground debris obstacles corresponding to the segment are classified as ground debris obstacles. Then, process 420 returns to action 620, where it is determined whether there are more segments to be classified. The classification process in actions 620-650 is repeated until it is determined that there are no longer any segments to be classified, after which the group of ground debris obstacles is output to the navigation system, as described above. For example, the group of ground debris obstacles can be combined with a group of larger obstacles within the local obstacle map so that the robot can change the route around the obstacles in the local obstacle map instead of stepping onto them, as described above.

[0098] Fig.10 An example configuration of a robotic device (or "robot") 1000 is shown in accordance with some embodiments. The robotic device 1000 may correspond, for example, to the robot 100 described above. The robotic device 1000 represents an illustrative robotic device configured to perform any of the techniques described herein. The robotic device 1000 may be configured to operate autonomously, semi-autonomously, and / or using directions provided by a user, and may exist in various forms, such as a humanoid robot, a bipedal robot, a quadrupedal robot, or other movable robots, among other examples. Additionally, the robotic device 1000 may also be referred to as a robotic system, a movable robot, or a robot, among other names.

[0099] like Fig.10As shown, the robotic device 1000 may include one or more processors 1002, a data storage device 1004, program instructions 1006, a controller 1008, one or more sensors 1010, one or more power supplies 1012, a mechanical component 1014, and an electrical component 1016. The robotic device 1000 is shown for illustrative purposes and may include more or fewer components without departing from the scope of the present disclosure. The various components of the robotic device 1000 may be connected in any manner, including via electronic communication means, such as wired or wireless connections. In addition, in some examples, the components of the robotic device 1000 may be placed on multiple different physical entities rather than on a single physical entity.

[0100] The processor 1002 may operate as one or more general purpose processors or special purpose processors (e.g., digital signal processors, application specific integrated circuits, etc.). The processor 1002 may correspond, for example, to the data processing hardware 142 of the robot 100 described above. The processor 1002 may be configured to execute computer readable program instructions 1006, which are stored in the data storage 1004 and are executable to provide the operation of the robotic device 1000 described herein. For example, the program instructions 1006 may be executable to provide the operation of the controller 1008, wherein the controller 1008 may be configured to cause the activation and / or deactivation of the mechanical components 1014 and the electrical components 1016. The one or more processors 1002 may operate the robotic device 1000 and enable the robotic device 1000 to perform various functions, including the functions described herein.

[0101] The data storage 1004 may exist as various types of storage media, such as memory. The data storage 1004 may correspond, for example, to the memory hardware 144 of the robot 100 described above. The data storage 1004 may include or take the form of one or more non-transitory computer-readable storage media that can be read or accessed by the processor 1002. The one or more computer-readable storage media may include volatile and / or non-volatile storage components, such as optical, magnetic, organic or other memory or disk storage, which may be integrated with the processor 1002 in whole or in part. In some implementations, the data storage 1004 may be implemented using a single physical device (e.g., one optical, magnetic, organic or other memory or disk storage unit), while in other implementations, the data storage 1004 may be implemented using two or more physical devices that may be in electronic communication (e.g., via wired or wireless communication). Furthermore, in addition to the computer-readable program instructions 1006, the data storage 1004 may include additional data, such as diagnostic data, and other possibilities.

[0102] The robotic device 1000 may include at least one controller 1008, which may be docked with the robotic device 1000, and may be integrated with the robotic device, or separate from the robotic device 1000. The controller 1008 may serve as a link between parts of the robotic device 1000, such as a link between mechanical components 1014 and / or electrical components 1016. In some cases, the controller 1008 may serve as an interface between the robotic device 1000 and another computing device. In addition, the controller 1008 may serve as an interface between the robotic system 1000 and a user. The controller 1008 may include various components for communicating with the robotic device 1000, including one or more joysticks or buttons and other features. The controller 1008 may also perform other operations of the robotic device 1000. Other examples of controllers may also exist.

[0103] In addition, the robotic device 1000 may include one or more sensors 1010, such as image sensors, force sensors, proximity sensors, motion sensors, load sensors, position sensors, touch sensors, depth sensors, ultrasonic range sensors, and / or infrared sensors, or combinations thereof, etc. The sensors 1010 may correspond, for example, to the sensors 132 of the robot 100 described above. The sensors 1010 may provide sensor data to the processor 1002 to allow for appropriate interaction of the robotic system 1000 with the environment and to monitor the operation of the system of the robotic device 1000. The sensor data may be used to evaluate various factors for activating and deactivating the mechanical components 1014 and the electrical components 1016 by the controller 1008 and / or the computing system of the robotic device 1000.

[0104] The sensor 1010 may provide information indicative of the environment of the robotic device to the controller 1008 and / or computing system for use in determining the operation of the robotic device 1000. For example, the sensor 1010 may capture data corresponding to the topography of the environment or the location of nearby objects, which may assist in environmental recognition and navigation, etc. In an example configuration, the robotic device 1000 may include a sensor system that may include a camera, radar, LIDAR, time-of-flight camera, global positioning system (GPS) transceiver, and / or other sensors for capturing information of the environment of the robotic device 1000. The sensor 1010 may monitor the environment in real time and detect obstacles, terrain elements, weather conditions, temperature, and / or other parameters of the environment of the robotic device 1000.

[0105] Additionally, the robotic device 1000 may include other sensors 1010 configured to receive information indicative of a state of the robotic device 1000, including sensors 1010 that may monitor the state of various components of the robotic device 1000. The sensors 1010 may measure the activity of the system of the robotic device 1000 and receive information based on the operation of various features of the robotic device 1000, such as the operation of extendable legs, arms, or other mechanical and / or electrical features of the robotic device 1000. The sensor data provided by the sensors may enable a computing system of the robotic device 1000 to determine errors in operation and to monitor the overall functionality of components of the robotic device 1000.

[0106] For example, the computing system can use sensor data to determine the stability of the robotic device 1000 during operation and measurements related to power levels, communication activities, parts that need repair, and other information. As an example configuration, the robotic device 1000 may include a gyroscope, an accelerometer, and / or other possible sensors to provide sensor data related to the operating state of the robotic device. In addition, the sensor 1010 can also monitor the current state of the functions that the robotic system 1000 can currently operate, such as gait. In addition, the sensor 1010 can measure the distance between a given robotic leg of the robotic device and the center of mass of the robotic device. Other example uses of the sensor 1010 may also exist.

[0107] In addition, the robotic device 1000 may also include one or more power sources 1012 configured to provide power to various components of the robotic device 1000. Among possible power systems, the robotic device 1000 may include a hydraulic system, an electrical system, a battery, and / or other types of power systems. As an example, the robotic device 1000 may include one or more batteries configured to provide power to the components via a wired and / or wireless connection. In the example, the components of the mechanical components 1014 and the electrical components 1016 may each be connected to a different power source or may be powered by the same power source. The components of the robotic system 1000 may also be connected to multiple power sources.

[0108] Within the example configuration, any suitable type of power source can be used to power the robotic device 1000, such as gasoline and / or an electric motor. In addition, the (one or more) power sources 1012 can be charged using various types of charging, such as a wired connection to an external power source, wireless charging, combustion, or other examples. Other configurations are also possible. In addition, the robotic device 1000 may include a hydraulic system that is configured to use fluid power to provide power to the mechanical components 1014. For example, the components of the robotic device 1000 can be operated based on hydraulic fluid transmitted to various hydraulic motors and hydraulic cylinders throughout the hydraulic system. The hydraulic system of the robotic device 1000 can transmit a large amount of power through small tubes, flexible hoses, or other connecting rods between the components of the robotic device 1000. Other power sources may be included in the robotic device 1000.

[0109] The mechanical components 1014 may represent the hardware of the robotic system 1000 that enables the robotic device 1000 to operate and perform physical functions. As a few examples, the robotic device 1000 may include actuators, extendable legs ("legs"), arms, wheels, one or more structural bodies for housing a computing system or other components, and / or other mechanical components. The mechanical components 1014 may depend on the design of the robotic device 1000, and may also be based on the functions and / or tasks that the robotic device 1000 may be configured to perform. In this way, depending on the operation and function of the robotic device 1000, different mechanical components 1014 may be available for the robotic device 1000 to utilize. In some examples, the robotic device 1000 may be configured to add and / or remove mechanical components 1014, which may involve assistance from a user and / or other robotic devices. For example, the robotic device 1000 may be initially configured with four legs, but may be changed by a user or the robotic device 1000 to remove two of the four legs to operate as a bipedal robot. Other examples of mechanical components 1014 may be included.

[0110] For example, the electrical components 1016 may include various components capable of processing, transmitting, providing an electric charge or an electrical signal. In possible examples, the electrical components 1016 may include wires, circuits, and / or wireless communication transmitters and receivers to enable the operation of the robotic device 1000. The electrical components 1016 may interact with the mechanical components 1014 to enable the robotic device 1000 to perform various operations. For example, the electrical components 1016 may be configured to provide power from one or more power sources 1012 to various mechanical components 1014. In addition, the robotic device 1000 may include an electric motor. Other examples of electrical components 1016 may also exist.

[0111] In some implementations, the robotic device 1000 may also include a communication link 1018 configured to send and / or receive information. The communication link 1018 may transmit data indicative of the status of various components of the robotic device 1000. For example, information read in by the sensor 1010 may be transmitted to a separate device via the communication link 1018. Other diagnostic information indicative of the integrity or health of the power source 1012, mechanical components 1014, electrical components 1018, processor 1002, data storage 1004, and / or controller 1008 may be transmitted to an external communication device via the communication link 1018.

[0112] In some implementations, the robotic device 1000 may receive information processed by the processor 1002 at the communication link 1018. For example, the received information may indicate data accessible by the processor 1002 during execution of the program instructions 1006. Additionally, the received information may change aspects of the controller 1008 that may affect the behavior of the mechanical component 1014 or the electrical component 1016. In some cases, the received information indicates a query requesting a specific piece of information (e.g., an operational status of one or more components of the robotic device 1000), and the processor 1002 may then send the specific piece of information back to the communication link 1018.

[0113] In some cases, the communication link 1018 comprises a wired connection. The robotic device 1000 may include one or more ports to connect the communication link 1018 to an external device. In addition to or as an alternative to a wired connection, the communication link 1018 may include a wireless connection. Some example wireless connections may utilize a cellular connection, such as CDMA, EVDO, GSM / GPRS, or 4G telecommunications, such as WiMAX or LTE. Alternatively or in addition, the wireless connection may utilize a Wi-Fi connection to transfer data to a wireless local area network (WLAN). In some implementations, the wireless connection may also communicate via an infrared link, radio, Bluetooth, or a near field communication (NFC) device.

[0114] The above embodiments may be implemented in any of a variety of ways. For example, embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code may be run on any suitable processor or set of processors, whether arranged in a single computer or distributed between multiple computers. It should be understood that any component or set of components that perform the above functions may generally be considered to be one or more controllers that control the above functions. One or more controllers may be implemented in a variety of ways, such as using dedicated hardware or using one or more processors that are programmed using microcode or software to perform the above functions.

[0115] Various aspects of the technology may be used alone, in combination, or in various arrangements not specifically described in the previously described embodiments, and therefore its application is not limited to the details and arrangements of components set forth in the foregoing description or shown in the accompanying drawings. For example, aspects described in one implementation may be combined in any manner with aspects described in other implementations.

[0116] In addition, some embodiments may be implemented as one or more methods for which examples have been provided. The actions performed as part of the method(s) may be ordered in any suitable manner. Thus, implementations may be constructed in which the actions are performed in an order different from the order shown, which may include performing some actions simultaneously, even though shown as sequential actions in an illustrative implementation.

[0117] The use of ordinal terms such as "first," "second," "third," etc. in the claims to modify claim elements does not by itself imply any priority, precedence, or order of one claim element relative to another claim element, or a temporal order of performing the acts of a method. Such terms are merely used as labels to distinguish one claim element with a particular name from another element with the same name (but using an ordinal term).

[0118] The phraseology and terminology used herein are for the purpose of description and should not be regarded as limiting. The use of "including," "comprising," "having," "containing," "involving," and variations thereof, is intended to encompass the items listed thereafter and additional items.

[0119] Having described several embodiments in detail, it will be easy for those skilled in the art to think of various modifications and improvements. Such modifications and improvements are intended to fall within the spirit and scope of the present technology. Therefore, the foregoing description is only exemplary and not restrictive.

Claims

1. A method of navigating a robot through an environment along a route, the route being associated with a task, the method comprising: identifying a set of potential obstacles in the environment based on sensor data received by one or more sensors of the robot; determining that at least one of the potential obstacles in the set is an obstacle based at least in part on stored data indicating a set of step locations of the robot during previous performance of the task; and The robot is navigated to avoid stepping onto the obstacle.

2. The method according to claim 1, wherein: The sensor data includes image data.

3. The method according to claim 1, wherein: The sensor data includes point cloud data representing distances to objects in the environment.

4. The method according to claim 1, further comprising: processing the sensor data to generate a topographic map of the environment, Wherein identifying the set of potential obstacles in the environment is based at least in part on the terrain map.

5. The method according to claim 4, further comprising: segmenting the topographic map to generate a segmentation map, Wherein identifying the set of potential obstacles in the environment is based at least in part on the segmentation map.

6. The method according to claim 5, wherein: Segmenting the topographic map comprises: detecting at least one boundary in the topographic map; and detecting at least one surface in the topographic map, Wherein identifying the set of potential obstacles in the environment is based at least in part on the at least one boundary and the at least one surface.

7. The method according to claim 6, wherein: Identifying the set of potential obstacles in the environment based at least in part on the at least one detected boundary and the at least one detected surface comprises: identifying a group of segments in the segmentation map, the segments comprising at least one boundary connected to at least one detected surface; and Included in the group of potential obstacles are objects associated with at least some of the segments in the group of segments.

8. The method according to claim 7, further comprising: discarding segments having a size below a threshold size from the group of segments; and Included in the set of potential obstacles are objects associated with segments having a size above the threshold size.

9. The method according to claim 1, wherein: Determining that at least one of the potential obstacles in the group is an obstacle based at least in part on stored data of a group indicating landing positions of the robot during previous performance of the task includes, for each of the potential obstacles in the group: determining, based on the set of step locations, whether the robot stepped onto the potential obstacle during the previous performance of the task; and The potential obstacle is determined to be an obstacle when it is determined that the robot did not step onto the potential obstacle during the previous execution of the task.

10. The method according to claim 9, wherein: The set of landing locations includes upcoming landing locations along a portion of the route that the robot has not yet traversed, and Based on the group of step positions, determining whether the robot has stepped onto the potential obstacle during the previous run of the task includes: determining whether the robot has stepped onto the potential obstacle based on one or more of the upcoming step positions along the route that the robot has not yet traversed.

11. The method according to claim 1, further comprising: identifying, based on the sensor data, a group of one or more large obstacles in the environment of the robot; adding the obstacle to the set of one or more large obstacles; and The robot is navigated to avoid stepping onto all of the obstacles in the group of one or more large obstacles.

12. The method according to claim 1, wherein: The height of the obstacle is less than 30 cm.

13. The method according to claim 1, wherein: The previous execution of the task corresponds to execution of the task when the task was recorded during operation of the robot by an operator.

14. A legged robot comprising: a perception system including one or more sensors configured to sense sensor data; At least one computer processor configured to: identifying a set of potential obstacles in the environment based on the sensor data; and determining that at least one of the potential obstacles in the set is an obstacle based at least in part on stored data indicating a set of step locations of the robot during previous performance of the task; and A navigation system is configured to navigate the robot to avoid stepping on the obstacle.

15. The legged robot according to claim 14, wherein: The sensor data includes image data.

16. The legged robot according to claim 14, wherein: The sensor data includes point cloud data representing distances to objects in the environment.

17. The legged robot according to claim 14, wherein: The at least one computer processor is further configured to: processing the sensor data to generate a topographic map of the environment, Wherein identifying the set of potential obstacles in the environment is based at least in part on the terrain map.

18. The legged robot according to claim 17, wherein: The at least one computer processor is further configured to: segmenting the topographic map to generate a segmentation map, Wherein identifying the set of potential obstacles in the environment is based at least in part on the segmentation map.

19. The legged robot according to claim 18, wherein: Segmenting the topographic map comprises: detecting at least one boundary in the topographic map; and detecting at least one surface in the topographic map, Wherein identifying the set of potential obstacles in the environment is based at least in part on the at least one boundary and the at least one surface.

20. The legged robot according to claim 19, wherein: Identifying the set of potential obstacles in the environment based at least in part on the at least one detected boundary and the at least one detected surface comprises: identifying a group of segments in the segmentation map, the segments comprising at least one boundary connected to at least one detected surface; and Included in the group of potential obstacles are objects associated with at least some of the segments in the group of segments.

21. The legged robot according to claim 20, wherein: The at least one computer processor is further configured to: discarding segments having a size below a threshold size from the group of segments; and Included in the set of potential obstacles are objects associated with segments having a size above the threshold size.

22. The legged robot according to claim 14, wherein: Determining that at least one of the potential obstacles in the group is an obstacle based at least in part on stored data of a group indicating landing positions of the robot during previous performance of the task includes, for each of the potential obstacles in the group: determining, based on the set of step locations, whether the robot stepped onto the potential obstacle during the previous performance of the task; and The potential obstacle is determined to be an obstacle when it is determined that the robot did not step onto the potential obstacle during the previous execution of the task.

23. The legged robot according to claim 22, wherein: The set of landing locations includes upcoming landing locations along a portion of a route that the robot has not yet traversed, and Determining whether the robot stepped onto the potential obstacle during the previous run of the task based on the group of step positions includes: determining whether the robot stepped onto the potential obstacle based on one or more of the upcoming step positions along the route that the robot has not yet traversed.

24. The legged robot according to claim 14, wherein: The at least one computer processor is further programmed to: identifying a group of one or more large obstacles in the environment of the robot based on the sensor data; adding the obstacle to the set of one or more large obstacles; and The robot is navigated to avoid stepping onto all of the obstacles in the group of one or more large obstacles.

25. The legged robot according to claim 14, wherein: The height of the obstacle is less than 30 cm.

26. The legged robot according to claim 14, wherein: The previous execution of the task corresponds to execution of the task when the task was recorded during operation of the robot by an operator.

27. A non-transitory computer readable medium encoded with a plurality of instructions that, when executed by at least one computer processor, perform a method of navigating a robot along a route through an environment, the route being associated with a task, the method comprising: identifying a set of potential obstacles in the environment based on sensor data received by one or more sensors of the robot; determining that at least one of the potential obstacles in the set is an obstacle based at least in part on stored data indicating a set of step locations of the robot during previous performance of the task; and The robot is navigated to avoid stepping onto the obstacle.

28. The non-transitory computer readable medium of claim 27, wherein: The sensor data includes image data.

29. The non-transitory computer readable medium of claim 27, wherein: The sensor data includes point cloud data representing distances to objects in the environment.

30. The non-transitory computer readable medium of claim 27, wherein: The method further comprises: processing the sensor data to generate a topographic map of the environment, Wherein identifying the set of potential obstacles in the environment is based at least in part on the terrain map.

31. The non-transitory computer readable medium of claim 30, wherein: The method further comprises: segmenting the topographic map to generate a segmentation map, Wherein identifying the set of potential obstacles in the environment is based at least in part on the segmentation map.

32. The non-transitory computer readable medium of claim 31 , wherein: Segmenting the topographic map comprises: detecting at least one boundary in the topographic map; and detecting at least one surface in the topographic map, Wherein identifying the set of potential obstacles in the environment is based at least in part on the at least one boundary and the at least one surface.

33. The non-transitory computer readable medium of claim 32, wherein: Identifying the set of potential obstacles in the environment based at least in part on the at least one detected boundary and the at least one detected surface comprises: identifying a group of segments in the segmentation map, the segments comprising at least one boundary connected to at least one detected surface; and Included in the group of potential obstacles are objects associated with at least some of the segments in the group of segments.

34. The non-transitory computer readable medium of claim 33, wherein: The method further comprises: discarding segments having a size below a threshold size from the group of segments; and Included in the set of potential obstacles are objects associated with segments having a size above the threshold size.

35. The non-transitory computer readable medium of claim 27, wherein: Determining that at least one of the potential obstacles in the group is an obstacle based at least in part on stored data of a group indicating landing positions of the robot during previous performance of the task includes, for each of the potential obstacles in the group: determining, based on the set of step locations, whether the robot stepped onto the potential obstacle during the previous performance of the task; and The potential obstacle is determined to be an obstacle when it is determined that the robot did not step onto the potential obstacle during the previous execution of the task.

36. The non-transitory computer readable medium of claim 35, wherein: The set of landing locations includes upcoming landing locations along a portion of the route that the robot has not yet traversed, and Based on the group of step positions, determining whether the robot has stepped onto the potential obstacle during the previous run of the task includes: determining whether the robot has stepped onto the potential obstacle based on one or more of the upcoming step positions along the route that the robot has not yet traversed.

37. The non-transitory computer readable medium of claim 27, wherein: The method further comprises: identifying a group of one or more large obstacles in the environment of the robot based on the sensor data; adding the obstacle to the set of one or more large obstacles; and The robot is navigated to avoid stepping onto all of the obstacles in the group of the one or more large obstacles.

38. The non-transitory computer readable medium of claim 27, wherein: The height of the obstacle is less than 30 cm.

39. The non-transitory computer readable medium of claim 27, wherein: The previous execution of the task corresponds to the execution of the task when the task was recorded during operation of the robot by an operator.

Images