Restricted Mobility Mapping

Abstract

A method for constrained mobility mapping includes receiving from at least one sensor (132) of a robot (100) a set of at least one original data (134) and a current set of data about an environment (10) of the robot (100). The method also includes generating a voxel map (210) including a plurality of voxels (212) based on the original set of at least one sensor data. The plurality of voxels include at least one ground voxel and at least one obstacle voxel. The method also includes generating a spherical depth map (218) based on the current set of sensor data and determining that an obstacle represented by the voxel map has changed based on a comparison between the voxel map and the spherical depth map. The method further includes updating the voxel map to reflect the change in the obstacle.

Description

Technical Field

[0001] The present disclosure relates to restricted mobility mapping. Background Art

[0002] Robotic devices are increasingly being used in confined or otherwise cluttered environments to perform a variety of tasks or functions. These robotic devices may need to navigate through these confined environments without stepping on or crashing into obstacles. As these robotic devices become more common, real-time navigation and step planning are required to avoid contact with obstacles while maintaining balance and speed. Summary of the invention

[0003] One aspect of the present disclosure provides a method for constrained mobility mapping. The method includes receiving at least one original set of sensor data and a current set of sensor data from at least one sensor of a robot at data processing hardware. Here, each of the original set of at least one sensor data and the current set of sensor data corresponds to an environment about the robot, wherein the robot includes a body. The method also includes generating a voxel map including a plurality of voxels based on the original set of at least one sensor data by the data processing hardware. The plurality of voxels include at least one ground voxel and at least one obstacle voxel. The method also includes generating a spherical depth map based on the current set of sensor data by the data processing hardware, and determining by the data processing hardware that an obstacle represented by the voxel map has changed based on a comparison between the voxel map and the spherical depth map. The method additionally includes updating the voxel map to reflect changes in obstacles within the environment by the data processing hardware.

[0004] Embodiments of the present disclosure may include one or more of the following optional features. In some embodiments, the robot includes four legs defining a quadruped. In some examples, generating a voxel map includes determining whether a three-dimensional cell of a space about the robot is occupied, and for each three-dimensional cell that is occupied, classifying the corresponding cell as one of the ground, an obstacle, or neither the ground nor an obstacle. In some configurations, the spherical depth map includes a spherical representation of a current set of sensor data, wherein the spherical representation includes a rectangular structure defined by points of the sensor data at a distance and height from at least one sensor that captured the current set of sensor data. In some embodiments, updating the voxel map to reflect changes in obstacles within the environment includes removing one or more voxels corresponding to obstacles associated with the change from the voxel map. Here, removing one or more voxels may include using a heuristic method to identify nearby voxels associated with changes in objects within the environment and removing the identified nearby voxels.

[0005] In some examples, the voxel map includes a three-dimensional (3D) grid, and the method further includes, for each cell of the 3D grid of the voxel map, merging, by data processing hardware, corresponding vertical columns of consecutive voxels to form a segment. Here, the segment includes a height and a point weight, wherein the point weight indicates a degree of certainty that one or more voxels forming the segment are occupied based on an original set of at least one sensor data. In these examples, the method may also include reducing, by data processing hardware, the point weight of the corresponding segment when a current set of sensor data does not include sensor data defining the corresponding segment. Additionally or alternatively, in these examples, the method may also include, by data processing hardware, comparing the height of the segment at a location in the voxel map to a height range from a column at a corresponding location in the spherical depth map, wherein the location of the segment and the corresponding location of the column correspond to the same location relative to the robot. In these examples, updating the voxel map to reflect a change in an obstacle within the environment includes pruning the segment corresponding to the obstacle associated with the change.

[0006] Other aspects of the present disclosure also provide a method for constrained mobility mapping. The method includes receiving sensor data corresponding to an environment about the robot from at least one sensor of the robot at data processing hardware, wherein the robot includes a body. The method also includes generating a voxel map including a plurality of voxels based on the sensor data by the data processing hardware. Here, the plurality of voxels include at least one ground voxel and at least one obstacle voxel. The method also includes, based on the voxel map, generating a body obstacle map by the data processing hardware, the body obstacle map being configured to indicate a location in the environment where the robot's body can move without being disturbed by obstacles in the environment. The body obstacle map is divided into cells, wherein the plurality of cells include an indication of a closest obstacle boundary, wherein the closest obstacle boundary is derived from at least one obstacle voxel of the voxel map. The method also includes transmitting the body obstacle map to a control system configured to move the robot about the environment.

[0007] This aspect may include one or more of the following optional features. In some embodiments, the indication includes an assessment of the distance to the closest obstacle boundary and the direction to the closest obstacle boundary. Here, generating the body obstacle map may include generating a vector field including a plurality of vectors, wherein each vector of the plurality of vectors indicates a direction of obstacle avoidance, and wherein each vector includes a vector direction opposite to the direction to the closest obstacle boundary. In some examples, the control system is configured to use the body obstacle map to control horizontal movement of the robot's body and yaw rotation of the robot's body. Multiple cells may not correspond to boundaries of obstacles.

[0008] In some configurations, the method may also include filtering, by data processing hardware, a plurality of voxels of the voxel map based on a point weight associated with each voxel of the plurality of voxels. Here, the point weight indicates a degree of certainty that the corresponding voxel is occupied based on the sensor data. In these configurations, generating a body obstacle map based on the voxel map includes converting the filtered plurality of voxels that satisfy a point weight threshold and correspond to obstacle voxels into the body obstacle map.

[0009] The third aspect of the present disclosure also provides a method for constrained mobility mapping. The method includes receiving sensor data corresponding to an environment about the robot from at least one sensor of the robot at data processing hardware, wherein the robot includes a body and legs, and each leg includes an end. The method also includes generating a voxel map including a plurality of fragments based on the sensor data by the data processing hardware, wherein each fragment of the plurality of fragments corresponds to a vertical column defined by one or more voxels. Here, the plurality of fragments include at least one ground fragment and at least one obstacle fragment. Based on the voxel map, the method also includes generating a ground height map by the data processing hardware, the ground height map being configured to indicate the height at which the end of the corresponding leg of the robot is placed when the robot moves about the environment. The ground height map is divided into cells, wherein at least one cell corresponds to a corresponding ground fragment and includes a corresponding height based on the corresponding ground fragment. The method also includes transmitting the ground height map to a control system by the data processing hardware, and the control system is configured to move the end of the corresponding leg to a placement position in the environment based on the ground height map.

[0010] This aspect may include one or more of the following optional features. In some embodiments, generating a ground height map includes determining that point weights of one or more voxels of a corresponding ground segment satisfy a height accuracy threshold, wherein the point weight indicates a degree of certainty that the corresponding voxel is occupied based on sensor data. Here, the height accuracy threshold indicates a level of accuracy of the height of a given object represented by the corresponding ground segment. In these embodiments, determining that the point weights of one or more voxels of a corresponding ground segment satisfy the height accuracy threshold includes passing through one or more voxels defining the corresponding ground segment from a maximum height of the corresponding ground segment to a minimum height of the corresponding ground segment.

[0011] In some examples, the method further includes the following: identifying, by data processing hardware, one or more cells of the ground height map corresponding to missing terrain; determining, by data processing hardware, whether the missing terrain corresponds to an occlusion of the sensor data; and replacing, by data processing hardware, the missing terrain with flat terrain when the missing terrain corresponds to the occlusion of the sensor data. When the missing terrain fails to correspond to the occlusion of the sensor data, the method may further include replacing, by data processing hardware, the missing terrain with smooth terrain. Here, with the smooth terrain, the method may not maintain the smooth terrain of the ground height map during subsequent iterations of the ground height map. In some configurations, the flat terrain is maintained within the ground height map until new sensor data identifies actual terrain corresponding to the flat terrain.

[0012] The fourth aspect of the present disclosure also provides a method for constrained mobility mapping. The method includes receiving sensor data corresponding to an environment about the robot from at least one sensor of the robot at data processing hardware, wherein the robot includes a body and legs, each leg including an end. The method also includes generating a voxel map including a plurality of segments based on the sensor data by the data processing hardware, wherein each segment of the plurality of segments corresponds to a vertical column defined by one or more voxels. Here, the plurality of segments include at least one ground segment and at least one obstacle segment. Based on the voxel map, the method also includes generating a ground height map by the data processing hardware, the ground height map being configured to indicate the height at which the end of the corresponding leg of the robot is placed when the robot moves about the environment. Based on the ground height map, the method also includes generating a no-step map including one or more no-step regions by the data processing hardware, wherein each no-step region is configured to indicate an area where the end of the corresponding leg of the robot is not placed when the robot moves about the environment. Here, the no-step map is divided into cells, wherein each cell includes a distance value and a direction vector. The distance value indicates the distance to the boundary of the closest obstacle of the cell. The direction vector indicates the direction of the boundary of the closest obstacle to the cell. The method further includes transmitting, by the data processing hardware, the no-step map to a control system configured to move a distal end of a corresponding leg to a placement location in the environment based on the no-step map.

[0013] This aspect may include one or more of the following optional features. The distance to the boundary of the nearest obstacle may include a symbol identifying whether the cell is inside the nearest obstacle or outside the nearest obstacle. At least one no-step region of the one or more step regions may identify an area that is inaccessible to the robot based on a current posture of the robot, and the area is accessible to the robot in an alternative posture different from the current posture. In some examples, generating a no-step map also includes generating a no-step map for a particular leg of the robot. In some embodiments, the method may also include determining, by data processing hardware, a nearest obstacle to a corresponding cell based on at least one obstacle segment of the voxel map.

[0014] In some configurations, the method further includes determining, by data processing hardware, a first no-stride region corresponding to a potential shin impact by: determining a minimum slope of the leg to achieve the commanded speed; identifying a shin impact height based on the minimum slope; and for each cell of the no-stride map, comparing the shin impact height to a ground height of the corresponding cell, the ground height of the corresponding cell received from the ground height map. In these configurations, the method may also include determining, by data processing hardware, that a difference between the shin impact height and the ground height of the corresponding cell satisfies a shin impact threshold.

[0015] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1A is a schematic diagram of an example robot within the environment.

[0017] Figure 1B yes Figure 1A Schematic diagram of an example system of a robot.

[0018] Figure 2A and Figure 2B yes Figure 1A An example projection of a voxel map of the robot.

[0019] Figure 2C yes Figure 1A A perspective view of an example voxel map of a robot.

[0020] Figure 2D and Figure 2E is a schematic diagram of an example of voxel classification.

[0021] Figure 2F and Figure 2G is based on Figure 1A A perspective view of an example of voxel classification of the robot's position.

[0022] Figure 2H is a perspective view of an example voxel map with negative fragments.

[0023] Figure 2I-2L is a perspective view of an example of ray tracing of a voxel map.

[0024] Figure 3A-Figure 3F Is Figure 1A Schematic diagram of an example physical obstacle map generated by the robot.

[0025] Figure 3G-Figure 3L Is Figure 1A Schematic diagram of an example processing technique for generating a body obstacle map of a robot.

[0026] Figure 4A Is Figure 1A Schematic diagram of an example of ground height map generation for a robot.

[0027] Figure 4B Is Figure 1A A perspective view of an example of a robot-generated ground height map.

[0028] Figure 5A-Figure 5C Is Figure 1A Illustration of an example no-step graph generated by a robot.

[0029] Figure 5D is a perspective illustration of an example no-step diagram based on the risk of shin impact.

[0030] Figure 5E Is Figure 1A Illustration of an example no-step graph generated by a robot.

[0031] Fig. 5F and Figure 5G is based on Figure 1A An example perspective view of one or more feet of a robot without a step diagram.

[0032] Figure 5H and Fig.5I Is Figure 1A Schematic diagram of an example of a no-step graph generated by a robot.

[0033] Figure 6-Figure 9 is an example arrangement of operations for a robot to generate a graph to traverse an environment for the robot.

[0034] Fig.10 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0035] Like reference numbers indicate like elements throughout the various drawings. DETAILED DESCRIPTION

[0036] As legged robotic devices (also referred to as "robots") become more common, there is a growing need for robots to navigate environments that are restricted in a variety of ways. For example, a robot may need to traverse a cluttered room where objects of all sizes are littered on the ground or navigate stairs. Typically, navigating such environments is a slow and arduous process that causes the legged robot to frequently stop, collide with objects, and / or become unbalanced. For example, even if the risk of colliding with an object is avoided, the robot's balance may be disturbed. To address some of these shortcomings, the robot builds maps based on sensors about the robot that guide and / or help manage the robot's movement in environments with obstacles. With these maps, the robot can traverse terrain while taking into account movement restrictions in real time, allowing the legged robotic device to quickly and / or efficiently navigate restricted environments while maintaining smooth movement and balance.

[0037] refer to Figure 1A , the robot 100 includes a body 110 having a locomotion-based structure (such as legs 120a-d coupled to the body 110) to enable the robot 100 to move with respect to the environment 10. In some examples, each leg 120 is an articulated structure such that one or more joints J allow a member 122 of the leg 120 to move. For example, each leg 120 includes an upper member 122, 122 of the leg 120. U Hip joint J connected to body 110 H , and the upper member 122 of the leg 120 U The lower member 122 of the leg 120 L Combined knee joint J K .although Figure 1A A quadruped robot having four legs 120a - d is depicted, but the robot 100 may include any number of legs or locomotion-based structures (eg, a bipedal or humanoid robot having two legs) that provide a means for traversing terrain within the environment 10 .

[0038] To traverse the terrain, each leg 120 has a tip 124 that contacts the surface of the terrain. In other words, the tip 124 of the leg 120 is the end of the leg 120 used by the robot 100 to pivot, rest, or generally provide traction during movement of the robot 100. For example, the tip 124 of the leg 120 corresponds to the foot of the robot 100. In some examples, although not shown, the tip 124 of the leg 120 includes an ankle joint. A , so that the end 124 is about the lower member 122 of the leg 120 L Articulated.

[0039] The robot 100 has a vertical gravity axis (eg, shown as a Z-axis A) along the direction of gravity. Z ) and a center of mass CM, which is the point where the sum of the weighted relative positions of the distributed mass of the robot 100 is zero. The robot 100 also has a relative position based on CM relative to the vertical gravity axis A. Z The robot 100 may be positioned relative to the body 110 to define a particular attitude or stance assumed by the robot 100. The posture of the robot 100 can be defined by the orientation or angular position of the robot 100 in space. Movement of the legs 120 relative to the body 110 changes the posture P of the robot 100 (i.e., a combination of the position of the robot's CM and the posture or orientation of the robot 100). Here, height generally refers to distance along the z-direction. The sagittal plane of the robot 100 corresponds to the y-axis A. Y and the Z-axis A Z The YZ plane extends in the direction of the x-axis. The horizontal plane (also called the transverse plane) is usually perpendicular to the sagittal plane and passes through the x-axis A X and the y-axis A Y The ground plane refers to the ground surface 12 where the ends 124 of the legs 120 of the robot 100 may generate traction to help the robot 100 move about the environment 10.

[0040] To maneuver with respect to the environment 10, the robot 100 includes a sensor system 130 having one or more sensors 132, 132a-n (e.g., shown as first sensors 132, 132a and second sensors 132, 132b). The sensors 132 may include vision / image sensors, inertial sensors (e.g., inertial measurement units (IMUs)), force sensors, and / or kinematic sensors. Some examples of sensors 132 include cameras such as stereo cameras, scanning light detection and ranging (LIDAR) sensors, or scanning laser detection and ranging (LADAR) sensors. In some examples, the sensors 132 have corresponding fields of view F that define a sensing range or area corresponding to the sensor 132. v .for example, Figure 1A Describe the field of view F of the robot 100 V Each sensor 132 may be pivotable and / or rotatable such that the sensor 132 may change its field of view F, for example, about one or more axes (eg, an x-axis, a y-axis, or a z-axis relative to a ground plane). V .

[0041] When using sensor 132 to survey the field of view F V When the sensor system 130 generates a signal corresponding to the field of view F VThe sensor system 130 may also include sensor data 134 (also referred to as image data) of the robot 100. In some examples, the sensor data 134 is data corresponding to a three-dimensional volumetric point cloud generated by the three-dimensional volumetric image sensor 132. Additionally or alternatively, as the robot 100 maneuvers with respect to the environment 10, the sensor system 130 collects posture data of the robot 100, the posture data including inertial measurement data (e.g., measured by an IMU). In some examples, the posture data includes kinematic data and / or orientation data about the robot 100. With the sensor data 134, the perception system 200 of the robot 100 may generate maps 210, 220, 230, 240 of the terrain with respect to the environment 10.

[0042] While the robot 100 is maneuvering about the environment 10, the sensor system 130 collects sensor data 134 related to the topography of the environment 10. For example, Figure 1AA sensor system 130 is depicted collecting sensor data 134 about a room that is an environment 10 of a robot 100. As the sensor system 130 collects the sensor data 134, a computing system 140 is configured to store, process, and / or transmit the sensor data 134 to various systems of the robot 100 (e.g., the perception system 200 or the control system 170). To perform computing tasks related to the sensor data 134, the computing system 140 of the robot 100 includes data processing hardware 142 and memory hardware 144. The data processing hardware 142 is configured to execute instructions stored in the memory hardware 144 to perform computing tasks related to the activities of the robot 100 (e.g., movement and / or activity-based movement). Generally speaking, the computing system 140 refers to one or more locations of the data processing hardware 142 and / or the memory hardware 144. In some examples, the computing system 140 is a local system located at the robot 100. When located at the robot 100, the computing system 140 may be centralized (i.e., at a single location / area on the robot 100, e.g., within the body 110 of the robot 100), decentralized (i.e., located at various locations about the robot 100), or a hybrid combination of the two (e.g., with a majority of centralized hardware and a minority of decentralized hardware). To illustrate some of the differences, a decentralized computing system 140 may allow processing to occur at active locations (e.g., at the motors of the joints of the mobile legs 120), while a centralized computing system 140 may allow a central processing hub to communicate with systems located at various locations on the robot 100 (e.g., communicating with the motors of the joints of the mobile legs 120). Additionally or alternatively, the computing system 140 includes computing resources that are remote from the robot 100. For example, the computing system 140 communicates with a remote system 160 (e.g., a remote server or cloud-based environment) via a network 150. Similar to the computing system 140, the remote system 160 includes remote computing resources such as remote data processing hardware 162 and remote memory hardware 164. Here, sensor data 134 or other processed data (e.g., data processed locally by computing system 140) may be stored on remote system 160 and accessible to computing system 140. In some examples, computing system 140 is configured to utilize remote resources 162, 164 as an extension of computing resources 142, 144, such that resources of computing system 140 may reside on resources of remote system 160.

[0043] In some embodiments, Figure 1A and Figure 1BAs shown, the robot 100 includes a control system 170 and a perception system 200. The perception system 200 is configured to receive sensor data 134 from the sensor system 130 and process the sensor data 134 into maps 210, 220, 230, 240. With the maps 210, 220, 230, 240 generated by the perception system 200, the perception system 200 may transmit the maps 210, 220, 230, 240 to the control system 170 to perform controlled actions of the robot 100, such as moving the robot 100 with respect to the environment 10. In some examples, with the perception system 200 separate from the control system 170 but in communication with it, the processing of the control system 170 may be focused on controlling the robot 100, while the processing of the perception system 200 is focused on interpreting the sensor data 134 collected by the sensor system 130. For example, these systems 200, 170 perform their processing in parallel to ensure accurate and smooth movement of the robot 100 in the environment 10.

[0044] In some examples, the control system 170 includes at least one controller 172, a path generator 174, a step locator 176, and a body planner 178. The control system 170 is configured to communicate with at least one sensor system 130 and a perception system 200. The control system 170 uses the hardware 140 to perform operations and other functions. The controller 172 is configured to control the movement of the robot 100 to traverse the environment 10 based on input or feedback from the system of the robot 100 (e.g., the control system 170 and / or the perception system 200). This may include movement between postures and / or behaviors of the robot 100. For example, the controller 172 controls different footstep patterns, leg patterns, body movement patterns, or vision system sensing patterns.

[0045] In some examples, the controller 172 includes a plurality of controllers 172, each controller 172 having a fixed cadence. A fixed cadence refers to a fixed timing of a step or swing phase of the leg 120. For example, the controller 172 instructs the robot 100 to move the leg 120 (e.g., step) at a particular frequency (e.g., step every 250 milliseconds, 350 milliseconds, etc.). With multiple controllers 172, each controller 172 having a fixed cadence, the robot 100 is able to experience variable timing by switching between controllers 172. In some embodiments, as the robot 100 traverses the environment 10, the robot 100 continuously switches / selects a fixed cadence controller 172 (e.g., reselects a controller 170 every 3 milliseconds).

[0046] refer to Figure 1B , the path generator 174 is configured to determine the horizontal motion of the robot 100. For example, the horizontal motion refers to the translation (i.e., movement in the XY plane) and / or yaw (i.e., movement around the Z-axis A) of the robot 100. ZRotation). The path generator 174 determines obstacles about the robot 100 within the environment 10 based on the sensor data 134. The path generator 174 provides a nominal collision-free path to the step locator 176 as a starting point for its optimization. The step locator 176 also receives information about obstacles so that the step locator 176 may identify foot placement locations for the legs 120 of the robot 100 (e.g., where the ends 124 of the legs 120 of the robot 100 are placed). The step locator 176 uses input from the perception system 200 (e.g., Figures 210, 220, 230, 240) to generate the foot placement locations (i.e., where the robot 100 should step). The body planner 178 is similar to the step locator 176 and receives input from the perception system 200 (e.g., Figures 210, 220, 230, 240). Generally speaking, the body planner 178 is configured to adjust the dynamics (eg, rotations such as pitch or yaw and / or height of the COM) of the body 110 of the robot 100 to successfully move about the environment 10 .

[0047] The perception system 200 is a system of the robot 100 that helps the robot move more accurately in terrain with various obstacles. When the sensor 132 collects sensor data 134 about the space of the robot 100 (i.e., the robot's environment 10), the perception system 200 uses the sensor data 134 to form one or more graphs 210, 220, 230, 240 for the environment 10. Once the perception system 200 generates the graphs 210, 220, 230, 240, the perception system 200 is also configured to append information to the graphs 210, 220, 230, 240 (e.g., by projecting the sensor data 134 onto a pre-existing graph) and / or remove information from the graphs 210, 220, 230, 240 (e.g., by ray tracing the pre-existing graph based on the current sensor data 134). Although the graphs 210, 220, 230, 240 are described separately herein, the perception system 200 may generate any number of graphs to convey the information and features described for each graph.

[0048] refer to Figure 2A-2LIn some embodiments, the perception system 200 generates a voxel map 210. The perception system 200 generates the voxel map 210 based on a combination of a world reference frame of the robot 100 and a local reference frame about the robot 100. Here, the perception system 200 receives odometry information of the robot 100, the odometry information defining the position of the robot 100 (e.g., by the position and / or velocity of the body 110 of the robot 100) to represent the world reference frame, and receives sensor data 134, the sensor data 134 defining an area within the range of the sensor 132 as an area proximate to the robot 100, the area representing the local reference frame. With the odometry information and the sensor data 134, the perception system 200 generates the voxel map 210 to represent the three-dimensional space about the robot 100. In some embodiments, the system of the robot 100 may track the relative motion of the robot over time (e.g., using simultaneous localization and mapping (SLAM)) to maintain current odometry information of the robot 100. In some examples, voxel map 210 is a data structure representing a historical collection of sensor data 134 by perception system 200 , such that voxel map 210 includes multiple sets of sensor data 134 over a period of time.

[0049] Voxel map 210 generally represents a three-dimensional space as voxels 212 (i.e., a graphical unit corresponding to a three-dimensional representation of a pixel). Figure 2A Describing voxels 212, 212 1-i In some examples, each voxel 212 of the voxel map 210 represents an area of ​​three cubic centimeters. In some configurations, the voxel map 210 represents the voxel 212 as segments 214, 214 1-i (For example Figure 2B As shown in FIG. 2A , a segment 214 refers to merging voxels 212 into vertical columns. In other words, the perception system 200 combines voxels 212 in the same vertical column of the 3D grid to form at least one segment 214. For example, Figure 2C A 3D grid of cells is illustrated having first segments 214, 214a and second segments 214, 214b. By representing voxels 212 as segments 214, the perception system 200 may simplify the classification of various obstacles or objects within the environment 10 by the robot 100. In other words, due to vertical merging, the perception system 200 processes a voxel map 210 having hundreds of segments 214 instead of thousands of voxels 212.

[0050] In some embodiments, the perception system 200 is configured with a gap threshold when forming the segments 214. In other words, a gap Gp or a non-continuous vertical column of voxels 212 may cause the perception system 200 to terminate the representation of the first segment 214 as a continuous portion of the vertical column of voxels 212 before the gap Gp, and to represent the second continuous portion of the vertical column of voxels 212 after the gap Gp as a second segment 214. For example, although Figure 2C The second segment 214b is illustrated as a single segment 214 (e.g., indicated by the same gray shading), if Figure 2C The gap Gp shown is large enough to satisfy the gap threshold, and the perception system 200 divides the second segment 214b into other segments 214. Therefore, a vertical column of voxels 212 may include multiple segments 214, depending on whether the size of the gap within the column satisfies (e.g., exceeds) the gap threshold. On the other hand, when the size of the gap Gp fails to satisfy the threshold (e.g., exceeds Figure 2C ), the perception system 200 is configured to ignore the gap Gp and interpret the entire vertical column of voxels 212 with the gap Gp as a single segment 214. In some examples, the gap threshold is thirty centimeters, so that any vertical gap exceeding thirty centimeters terminates the segment 214 on one side of the gap Gp and causes the formation of a new segment 214 on the other side of the gap Gp. By separating the segments 214 at the gap Gp, the perception system 200 may be configured to infer that all voxels in the same segment 214 correspond to the same potential object.

[0051] Continue to refer Figure 2C , the perception system 200 is configured to classify the voxel map 210 (e.g., classify the segments 214) to identify portions corresponding to ground (i.e., geometric areas that the perception system 200 interprets the robot 100 to be able to step on), obstacles (i.e., geometric areas that the perception system 200 interprets as possibly interfering with the movement of the robot 100), or portions that are neither ground nor obstacles (e.g., something above the robot 100 that can be ignored). In some configurations, the voxel map 210 includes a dense two-dimensional grid of columns, where a column is a numerical representation of the number of segments 214 within each particular region (i.e., cell) of the two-dimensional grid. Additionally, each column may include a sparse list of voxels 212, such that a column includes a count of the number of voxels 212 present in the column. Because a column may correspond to a vertical segment 214 at a cell of the two-dimensional grid, each cell may have zero or more segments 214. When the perception system 200 groups the voxels 212 into one or more segments 214, the perception system 200 is configured to classify each segment 214 (or voxel 212) into a corresponding classification, such as ground 214, 214 G , underground 214, 214 UG , obstacles 214, 214 OB, or head 214, 214 OH By classifying the fragment 214 as ground 214 G , the perception system 200 indicates that the robot 100 may step on the top of the segment 214. When the perception system 200 classifies the segment 214 as underground 214 U When the underground classification indicates that the segment 214 may be ignored for further processing by the perception system 200 or other systems of the robot 100. OB The segment 214 is an object that the robot 100 may collide with and cannot step on. Here, it is classified as the overhead 214 OH The segment 214 refers to the segment 214 that the perception system 200 recognizes the robot 100 can pass through from below.

[0052] Generally speaking, the language herein sometimes refers to ground surface 12 (or ground plane) and also to "ground". Ground surface 12 refers to a feature of world environment 10. In contrast, ground surface G refers to a designation by perception system 200 of an area (e.g., voxel 212 or segment 214) where robot 100 may take a step. Similarly, object 14 is a physical structure or feature in world environment 10 and "obstacle O" is a designation by perception system 200 of object 14 (e.g., occupied voxel 212 or obstacle segment 214). OB ). In other words, the sensor system 13 collects sensor data 134 about an object 14 that is close to the robot 100 in the environment 10 and is interpreted (i.e., perceived) by the perception system 200 as an obstacle O because the object 14 is an area that may hinder or prevent the movement of the robot 100.

[0053] In some embodiments, the perception system 200 is configured to perform classification based on a convexity assumption. The convexity assumption assumes that the robot 100 generally moves outward from the center without changing direction. With respect to the perception system 200, the convexity assumption commands the perception system 200 to begin its classification process closest to the robot 100 and classify outward. During classification by the perception system 200 based on the convexity assumption, the perception system 200 may classify cells (or segments 214) in an associative manner. In other words, the classification of cells is based on cells that the perception system 200 has seen between the robot 100 and the cell.

[0054] The perception system 200 may encounter various problems when classifying objects perceived by the robot 100. For example, if the perception system 200 uses 1.5 dimensional (1.5D) analysis for classification (i.e., a 1D line with a height function for each point on the 1D line), the perception system 200 runs the risk of encountering problems identifying whether the robot 100 has traversed upward multiple times in succession and should probably not continue its upward traversal for a period of time. In other words, the robot 100 may be climbing terrain and not necessarily traversing relatively along the lowest true surface of the environment 10. Other potential problems with 1.5D analysis are that the overall slope of a sequence of cells (e.g., adjacent cells) may be difficult to quantify, causing the robot 100 to attempt to traverse cells with too steep a slope.

[0055] A potential approach to addressing these shortcomings is for the perception system 200 to use an allowed height process. In the allowed height approach, the perception system 200 defines a region of space that is close to (e.g., adjacent to) each cell that the robot 100 cannot step on. By classifying all cells or some cell clusters sensed by the perception system 200 where the robot 100 cannot step on the spatial region, the perception system 200 classifies the areas where the robot 100 can step on (i.e., ground classification) as the intersection of the spatial regions that are not marked as areas where the robot 100 cannot step on. Although this approach may address some of the shortcomings of the 1.5D classification approach, depending on the environment 10, this approach may become too strict, causing the perception system 200 to not classify enough cells as ground where the robot 100 may step on.

[0056] In some embodiments, Figure 2D , in order to make the allowed height processing more robust so that the perception system 200 may effectively and / or accurately classify the segment 214, the perception system 200 begins its classification processing for cells that have a trajectory of allowed heights forward from the particular cell. Here, a trajectory refers to an allowed range of heights that the robot 100 may step from one cell to an adjacent cell (e.g., when considering the convexity assumption). For example, Figure 2D and 2E Trajectory 216 and corresponding starting cell locations (shown in gray) are depicted referring to five segments 214, 214a-e. As perception system 200 traverses cells during classification, trajectory 216 shifts (e.g., from Figure 2D Towards Figure 2E ) and continues to append the allowed range of altitudes at the ends of the track. After the transfer, perception system 200 processes the current allowed range of altitudes for the track; removes small disturbances (e.g., altitude disturbances) and shapes the allowed altitude range to be monotonic. In some examples, the post-transfer processing causes both addition and subtraction to occur to segments 214 during classification. Although Figure 2D and 2EThe trajectory classification process is described with respect to a one-dimensional approach, but the perception system 200 may perform similar processing in other dimensions (eg, in two or three dimensions).

[0057] In some examples, the classification of perception system 200 depends on the context. Figure 2F and Figure 2G As shown, when the robot 100 is in the first posture P, P relative to the obstacle 1 When the robot 100 is in a posture P (e.g., a staircase), the object 14 (e.g., a staircase) may be an obstacle to the robot 100. Figure 2G The second posture P 2 , the object 14 is not an obstacle to the robot 100, but should be considered as a ground that the robot 100 can pass through. Therefore, when classifying the segment 214, the perception system 200 describes the position and / or posture P of the robot 100 with respect to the object 14.

[0058] In some configurations, the voxel map 210 corresponds to a visual certainty of each voxel 212 within the voxel map 210, rather than to a strict map of voxel occupancy. For example, the perception system 200 includes a point weight W for each voxel 212 in the voxel map 210. p (For example Figure 2C As shown), the point weight W p The representation system 200 has sensed (i.e., received / processed) the occupancy of a particular voxel 212 based on the sensor data 134. More specifically, the perception system 200 receives the sensor data 134 at a particular frequency. In these examples, when the perception system 200 receives the sensor data 134 for a voxel 212 that was previously identified by the perception system 200 as occupied, the perception system 200 adjusts the point weight W p To convey a higher level of confidence in the occupancy of the previously identified voxel 212. In some examples, the point weight W p It also includes factors that identify the type of sensor 132 that the voxel occupies. For example, a LIDAR sensor 132 has a higher accuracy than a stereo camera sensor 132. Here, the perception system 200 adjusts the point weight W p To represent the accuracy of the sensor 132 that collected the sensor data 134 (e.g., the LIDAR sensor 132 is more accurate than the stereo camera sensor 132). In other examples, the point weight W p The type of sensor 132 is accounted for based on the distance of the identified voxel 212. For example, a stereo camera is less accurate (e.g., will receive a lower point weight) when further away from the object 14. Conversely, a LIDAR sensor 132 is accurate at greater distances, but is less accurate when the object 14 is closer to the LIDAR sensor 132 due to the increased point cloud density. Therefore, the point weight W of the voxel 212 isp One or more factors that affect the accuracy of voxel identification may be considered (eg, previous identification, distance, type of sensor 132, or any combination).

[0059] In some examples, the point weight W of a voxel is p The occupancy threshold exists (i.e., is assigned by the perception system 200) based on an occupancy threshold. The occupancy threshold indicates to the perception system 200 a certain confidence that the voxel 212 is occupied based on the sensor data 134. For example, the occupancy threshold is set to a count of the number of times the voxel 212 has been perceived as occupied based on the sensor data 134. In other words, if the occupancy threshold is set to a value of ten, when the perception system 200 encounters sensor data 134 indicating occupancy of the voxel 212 ten times, the voxel 212 is given a point weight W specifying its existence. p In some embodiments, the perception system 200 reduces the point weight W of the presence of a given voxel 212 based on characteristics of the sensor data 134 (eg, distance, type of sensor 132, etc.). p .

[0060] Refer to Figure 2C In some embodiments, the voxel map 210 includes a voxel height 212h (e.g., the voxel 212 of the third segment 214c is shown in dark gray shading at the voxel height 212h). The voxel height 212h refers to the average height of the points within the voxel 212 where the perception system 200 identifies the presence of an object based on the sensor data 134. By including the voxel height 212h of each voxel 212, the voxel map 210 of the perception system 200 includes a higher level of accuracy than assuming that the object occupies the entire voxel 212. For example, when the voxel 212 is three cubic centimeters, the voxel map 210 identifies the height of the object within the voxel 212 with a resolution of more than three cubic centimeters. This allows the perception system 200 to reflect the true value of the height of the object (e.g., ground height) rather than being discretized by the size of the voxel 212. In some examples, when the voxel map 210 includes the voxel height 212h, the perception system 200 tracks the changes in the height 212h of each voxel 212 over time.

[0061] Although the voxel height 212h of the voxel 212 and the point weight W pOften discussed separately, the perception system 200 may generate a voxel map 210 that includes one or some combination of these characteristics. Moreover, regardless of the characteristics of the voxel map 210, the perception system 200 may be configured to disqualify sensor data 134 based on certain criteria. Some examples of criteria include sensor data 134 being too bright, too dark, too close to an object perceived from the sensor 132, too far from an object perceived from the sensor 132, or too close to a structure of the robot 100 (e.g., an arm or leg 120). For example, a stereo camera sensor 132 may have limited accuracy when the condition of the sensor 132 meets the criteria (e.g., too bright, too dark, too close, or too far). By disqualifying sensor data 134 that tends to be inaccurate, the perception system 200 ensures an accurate voxel map 210 that the robot 100 may use by the control system 170 to move about the environment 10 and perform activities within the environment 10. Without such accuracy, the robot 100 may be at risk of collisions, other types of interference, or unnecessary avoidance during its maneuvers in the environment 10.

[0062] The perception system 200 may accumulate the voxel map 210 over time such that the voxel map 210 spans some or all portions of the environment 10 captured by the sensors 132 of the robot 100. Because the voxel map 210 may be quite large, an area directly centered on the robot 100 may have a higher accuracy than an area previously perceived by the sensor system 130 and perceived by the perception system 200. This may be particularly true when the robot 100 has been away from a particular area of ​​the voxel map 210 for a long period of time.

[0063] In some embodiments, the point weight W of a voxel 212 in the voxel map 210 is p Gradual decay over time. Gradual decay allows objects (i.e., occupied voxels 212) to have a temporal component, so that recently seen objects have greater importance to the voxel map 210 than objects seen a long time ago. For example, the perception system 200 reduces the point weight W based on the gradually decaying frequency of voxels 212 that do not appear or do not appear accurately (e.g., not disqualified) in the current sensor data 134. p (That is, the point weight W P ) (e.g., every three seconds reduce the point weight W by some factor p (eg, some percentage). The gradual decay may be configured so that the point weight W of the occupied voxel 212 p cannot be reduced to less than a certain threshold. Here, the point weight threshold may be another form of an occupancy threshold or its own independent threshold. By using the point weight threshold, the perception system 200 is aware that the space corresponding to the voxel 212 is occupied, but has not appeared in the sensor data 134 recently (i.e., during a specified time period).

[0064] In some examples, portions of the voxel map 210 are stored in the computing system 140 of the robot 100 and / or in a remote system 160 that is in communication with the computing system 140. For example, the perception system 200 transfers a portion of the voxel map 210 to storage having a particular point weight W. p (e.g., based on a point weight storage threshold) to reduce potential processing by the perception system 200. In other examples, the perception system 200 removes or eliminates portions of the voxel map 210 that satisfy a particular point weight W. p , such as the portion of the point weight removal threshold (e.g., below the point weight removal threshold). For example, once the perception system 200 sets the point weight W of the voxel 212 p To decrease to near zero (or substantially zero), perception system 200 eliminates voxel 212 from voxel map 210 .

[0065] With the point weight W of each voxel 212 p , the perception system 200 may be based on the point weight W p To generate the fragment 214. In other words, in some configurations, the perception system 200 includes a fragment generation threshold, which indicates that during the fragment generation, points with a weight W less than the fragment generation threshold are ignored. p Therefore, the perception system 200 does not generate a segment at a point weight W that is less than the segment generation threshold. p A fragment 214 is generated at the voxel 212 of FIG.

[0066] refer to Figure 2H In some examples, the perception system 200 is configured to generate negative segments 214, 214 N (For example, the first negative segment 214 N a and second negative segment 214 N b) Negative fragments 214, 214 N is a representation of the regions of the voxel map 210 known as empty space. N This allows differentiation between regions in the voxel map 210 that have been verified to be absent and regions that are unknown. In other words, the negative segments 214 N This enables perception system 200 to distinguish between spaces that robot 100 has seen and spaces that robot 100 has not seen (e.g., with the aid of sensor system 130). In some examples, negative segment 214 N The processing resources of the perception system 200 are reserved because the negative fragment 214 N is not further divided (i.e., processed) into voxels 212. This prevents the perception system 200 from performing further processing specifically on known empty spaces. In some embodiments, when the robot 100 moves away from the negative segment 214 N The negative segment 214 generated by the perception system 200 moves with the associated position.N Contraction (i.e., similar to attenuation). Here, the perception system 200 contracts the negative segment 214 N The ratio may be based on the estimated odometry drift of the robot 100 (ie, the change in position as evidenced by the odometry information).

[0067] Negative fragment 214 N It is possible to assist perception system 200 in classifying segments as ground versus obstacles by providing an estimate of where ground may be in the unperceived space. For example, when perception system 2 does not identify ground (e.g., classifies ground), but perception system 200 identifies that negative segment 214 exists in a particular range of voxel map 210 N , even if sensor system 130 does not see (i.e., does not perceive) an area below the negative segment range, perception system 200 may assume that the ground is below the negative segment range. N It is possible to place an upper limit on the invisible region of the voxel map 210 because the perception system 200 may generate negative fragments 214 that are higher than the upper limit of the invisible region. N (i.e., known empty space). For example, if the perception system 200 perceives the first negative segment 214 N a and second negative segment 214 N b Both, but not less than each negative segment 214 N ab ground clip 214 G Then, perception system 200 may assume that ground segment 214 G Negative fragments of perception 214 N ab. Additionally or alternatively, negative segment 214 N Allows perception system 200 to infer the height of invisible terrain approaching map 220 generated by perception system 200.

[0068] In some examples, the perception system 200 utilizes the concept of ray tracing to remove data from the voxel map 210. Traditionally, ray tracing refers to a technique of tracing lines between sensor data 134 (e.g., a point cloud) and the sensor 132 that generated the sensor data 132. Based on this technique, when the sensor 132 perceives an object at a certain distance, it may be assumed that the line between the object and the sensor 132 is unobstructed. Therefore, by tracing the line between the object and the sensor 132, the ray tracing technique checks whether there is something on the line. Ray tracing may be advantageous because objects may physically move around in the environment 10 of the robot 100. With physically moving objects, the perception system 200 may generate a voxel map 210 with space occupied by the moving objects that is not currently occupied by the moving objects; therefore, potentially introducing false obstacles to the robot 100. By using ray tracing-based techniques, the perception system 200 generally applies the following processing strategy: if the sensor system 130 is currently able to see through (e.g., the point cloud now extends beyond the range of the original point cloud in the specified space) a portion of the environment 10 (e.g., one or more voxels 212) of an object that was previously perceived by the perception system 200, the original portion of the voxel map 210 corresponding to the previously perceived object should be removed or at least partially modified. In other words, the current set of sensor data 134 (e.g., image data) indicates that the object perceived from the previous set of sensor data 134 (e.g., original sensor data 134) is no longer accurately depicted by the voxel map 210. Additionally or alternatively, the ray tracing-based techniques may also provide assistance when there is odometry drift or when false objects appear in the voxel map 210 due to sensor noise.

[0069] refer to Figure 2I-2L In some examples, the perception system 200 is configured to perform a modified method of ray tracing. Instead of tracing a line between the sensor 132 and the sensor data 134, the perception system 200 constructs a spherical depth map 218 for the sensor data 134 (e.g., a set of the most recent sensor data 134 received by the perception system 200). With the spherical depth map 218 for the sensor data 134, the perception system 200 compares the spherical depth map 218 to the voxel map 210 generated so far by the perception system 200. In some examples, the perception system 200 performs the comparison at a fragment level, such that the perception system 200 compares an existing fragment 214 of the voxel map 210 to the spherical depth map 218. By comparing the spherical depth map 218 to the voxel map 210, the processing of the perception system 200 is computationally more efficient than the traditional ray tracing method. The following equations (1) and (2) illustrate the computational cost of the traditional ray tracing technique of equation (1) and the modified ray tracing method of equation (2).

[0070] Cost trad =O(N p*R) (1)

[0071] Cost mod =O(N p +N S ) (2)

[0072] where O(f(N)) is the set of N objects in the environment 10. Here, as shown in equation (1), the computational cost of traditional ray tracing is the point N scaled by R p (i.e., points corresponding to sensor data 134), where R represents how many voxels 212 the ray (i.e., the traced line) passes through on average. In contrast, as shown in equation (2), the computational cost of the modified ray tracing method is the number of points involved in the comparison, N. p and the number N of segments 214 s Since the computational cost of traditional ray tracing scales by R, rather than by the number of fragments N involved, S Scaling by the sum of , conventional ray tracing is typically computationally more expensive than modified ray tracing methods by some factor.

[0073] In some implementations, the perception system 200 informs that a comparison is performed between columns, comparing the existing voxel map 210 to the spherical depth map 218. In other words, each column (i.e., vertical plane or z-plane) of the voxel map 210 corresponds to a column of the spherical depth map 218. For each fragment 214 in the column, the perception system 200 checks the height range of the corresponding column of the spherical depth map 218 to determine whether the sensor data 134 forming the spherical depth map 218 is perceived farther than the fragment 214. In other words, when the perception system 200 encounters a fragment 214 from a column of the voxel map 210 that matches the height range of a column at the same location in the spherical depth map 218, the perception system 200 does not update the voxel map 210 by removing the fragment 214 (i.e., the sensor data 134 forming the spherical depth map 218 verifies the presence of the fragment 214 in the voxel map 210). On the other hand, when the perception system 200 encounters a fragment 214 from a column of the voxel map 210 that does not match the height range of the column at the same location in the spherical depth map 218, the perception system 200 updates the voxel map 210 by removing the fragment 214 (i.e., the sensor data 134 forming the spherical depth map 218 verifies that the fragment 214 in the voxel map 210 no longer exists). In some examples, when the height range changes (e.g., when the potential object moves slightly), the perception system 200 modifies the corresponding fragment 214 of the voxel map 210 instead of removing it completely. From a voxel perspective, the comparison process uses the sensor data 134 forming the spherical depth map 218 to confirm that the voxel 212 no longer occupies the location where the perception system 200 removed or modified the fragment 214. Here, the sensor data 134 forming the spherical depth map 218 includes a set of current sensor data 134 (e.g., image data) obtained after the original sensor data 134 forming the voxel map 210.

[0074] like Fig.2I and 2J As shown, in some configurations, the spherical depth map 218 is a spherical representation of the sensor data 134. As a spherical representation, the perception system 200 may construct the spherical depth map 218 by forming a rectangular structure with a certain distance (e.g., in the xy plane) and height (e.g., in the z plane) from each sensor 132 that generates the sensor data 134 for the robot 100. For example, Fig.2I A rectangular structure defined by points of sensor data 134 is illustrated. Figure 2J Depict overlay on sensor data 134 (eg Fig.2I ) and the current segment 214 of the voxel map 210 1-i Here, the object 14 (e.g., the lower part of a person) is shown as an obstacle segment 214 approaching the robot 100. OBIn other words, the robot 100 is located at the center of a sphere (or three-dimensional shape) that extends radially based on the range of the sensor 132. In some examples, the robot 100 (e.g., the perception system 200) divides the sphere into wedge-shaped or pyramid-shaped sections, where the vertices of the sections correspond to the robot 100. The size of the sections may vary depending on the configuration of the perception system 200. In some examples, with this partial approach, the bases of the pyramid-shaped sections form a rectangular structure of the spherical depth map 218.

[0075] refer to Figure 2K and 2L , when the robot 100 and the object 14 (e.g., Fig.2I and 2J When the person shown in FIG. 1 is moving in the environment 10, there is a field of view F that changes with the change of V For example, when the robot 100 is at a certain distance from the person, the sensor system 130 may see the entire person at a certain position. Figure 2K The segment 214 in corresponds to a person. However, when the robot 100 Figure 2L As shown in the figure, when approaching the person, the sensor system 130 does not perceive the entire person (e.g., the sensor data 134 captures the person from the knees to the hips). In these scenarios, when the person moves away from this location, the perception system 200 perceives that the person from the knees to the hips no longer exists based on the modified ray tracing, but may not be able to associate that other segments 214 corresponding to the person below the knees and above the hips (e.g., shown as the second segment 214b and the third segment 214c) no longer exist. In other words, the robot 100 lacks a method to associate the segments 214 during removal or modification by the perception system 200; resulting in artifacts of the person being inaccurately present in the voxel map 210. To address this issue, when the perception system 200 removes voxels 212 and / or segments 214 through the modified ray tracing method, the perception system 200 uses a heuristic method to identify and remove nearby ambiguous voxels 212 that may be part of the same potential object. am .

[0076] In some examples, the voxel map 210 includes a color visualization of the voxels 212 and / or segments 214. For example, the perception system 200 may transmit the voxel map 210 with the color visualization to a debugger of the robot 100 to allow an operator to visually understand a terrain problem of the robot 100. In other examples, the perception system 200 transmits the voxel map 210 with the visualization to an operator of the robot 100, who controls the movement of the robot 100, to enable the operator to understand the surroundings of the robot 100. A manual operator may prefer a visualization, particularly when the robot 100 is at a distance from the operator, where the operator visualizes some or all of the surroundings of the robot 100.

[0077] refer to Figure 3A-Figure 3L Based on the voxel map 210, the perception system 200 is configured to generate one or more body obstacle maps 220. The body obstacle map 220 generally determines whether the body 110 of the robot 100 may overlap with a position in the XY plane with respect to the robot 100. In other words, the body obstacle map 220 identifies obstacles of the robot 100 to indicate whether the robot 100 is at risk of colliding with or potentially damaging an obstacle at a nearby or same position by overlapping at a position in the environment 10. As a map of obstacles of the body 110 of the robot 100, the system of the robot 100 (e.g., the control system 170) may use the body obstacle map 220 to identify boundaries adjacent to or closest to the robot 100 and to identify directions (e.g., optimal directions) for moving the robot 100 to avoid obstacles. In some examples, similar to the other maps 210, 230, 240, the perception system 200 generates the body obstacle map 220 based on a grid of cells 222 (e.g., a grid of the XY plane). Here, each cell 222 within the body obstacle map 220 includes a distance d to the obstacle and a vector v pointing to the nearest cell 222 that is the obstacle (i.e., the boundary of the obstacle). For example, although the entire body obstacle map 220 may be divided into cells 222 (e.g., a 128x128 grid of three centimeter cells), Figure 3A Six cells 222, 222a-f are illustrated, each comprising a vector v and a distance d to the closest boundary of an obstacle.

[0078] refer to Figure 3A and Figure 3B In some examples, the perception system 200 derives two body obstacle maps 220a, 220b from the voxel map 210, a first body obstacle map 220, 220a and a second body obstacle map 220, 220b. Figure 3A As shown, the first body obstacle map 220a (also referred to as the standard obstacle map 220a) generated by the perception system 200 allows the step locator 176 of the control system 170 to generate a step plan that identifies the foot placement position of the robot 100. Figure 3B , a second body obstacle map 220b (also referred to as an extended obstacle map 220b) generated by the perception system 200 allows the body path generator 174 of the control system 170 to define a rough trajectory of the body 110 of the robot 100 (e.g., horizontal movements of the body 110, such as translation and yaw rotation). Generally speaking, the standard obstacle map 220 is a less processed map 220 than the extended obstacle map 220b and may therefore be considered a more realistic representation of real physical obstacles within the environment 10. As a more processed map 220, the extended obstacle map 220b includes a potential field representation of obstacles within the environment 10.

[0079] Continue to refer Figure 3A and Figure 3B , each map 220 includes a body obstacle region 224 (e.g., shown in black) and a no-body obstacle region 226 (e.g., shown as a diagonal cross-hatched pattern). The body obstacle region 224 refers to a region (e.g., one or more cells 222) of the body obstacle map 220 where the perception system 200 identifies an obstacle based on the voxel map 210. For example, the body obstacle region 224 corresponds to cells 222 located at the boundary of the object (i.e., cells that spatially represent the boundary of the object) and cells 222 that the perception system 200 marks as obstacles during voxel / fragment classification. In contrast, the no-body obstacle region 226 refers to a region (e.g., one or more cells 222) of the body obstacle map 220 where the perception system 200 does not identify an obstacle based on the voxel map 210. In some embodiments, these regions 224, 226 may be further processed by the perception system 200 (e.g., modifying the regions 224, 226). For example, the perception system 200 T The body obstacle area 224 on the Figure 3A When compared with the standard body obstacle diagram 220a Figure 3B The extended body obstacle diagram 220b is narrower.

[0080] In some configurations, initially, the perception system 200 generates both body obstacle maps 220a-b in a similar processing manner. Because the voxel map 210 includes a classification of whether an obstacle is present or absent at a particular location (e.g., a cell of the voxel map 210) of the voxel map 210, the perception system 200 converts the obstacle / no obstacle designation to each corresponding location of the body obstacle map 220. Once the obstacle or lack of obstacles is represented within the body obstacle map 220 (e.g., as regions 224, 226), the perception system 200 filters each body obstacle map 220 to remove small areas with low weights (e.g., poor sensor data 134). In some examples, the filtering process performed by the perception system 200 modifies the information converted from the voxel map 210 by dilation, elimination of low weight areas, and / or attenuation. Here, the low weight area refers to the area of ​​some combination of the height of the segment 214 identified by the voxel map 210 and the point weight of the segment 214. In other words, during filtering, the perception system 200 may include one or more thresholds for the height of the segment 214 and / or the point weight of the segment 214 to indicate when to remove the segment 214 from the area of ​​the body obstacle map 220a-b. This removal is intended to eliminate sensor noise (i.e., poor sensor data) while retaining the representation of real physical objects. Additionally or alternatively, when forming the body obstacle map 220, the perception system 200 marks the area below the robot's current pose P and prevents new obstacles from being marked in the area.

[0081] refer to Figure 3C and Figure 3D In some configurations, the perception system 200 further processes the standard body map 220a to include obstacle shadows. Obstacle shadows are extensions of obstacles (e.g., body obstacle regions 224) that enable the step locator 176 to perform constraint extraction more efficiently. Without obstacle shadows, the step locator 176 may have difficulty performing constraint extraction on thin obstacles (e.g., walls) represented by the body obstacle map 220. For example, Figure 3C The sensing system 200 is shown expanding the body obstacle region 224 to include Figure 3D The obstacle shadow illustrated with thickened features is a thin feature (e.g., a thin wall) that precedes a body obstacle region 224 within the body obstacle map 220 (i.e., a larger body obstacle region 224). Here, the thickened features occur by the perception system 200 designating one or more adjacent cells as cells of the body obstacle region 224 as part of the same body obstacle region 224. In some examples, the obstacle shadow fills the area behind the obstacle all the way to the edge of the map. The method generates shadows directionally so that the wall has a true distance to the robot 100 on its front side (i.e., facing the robot 100), but has a thicker obstacle shadow only on its back side (i.e., the side not facing the robot 100).

[0082] refer to Figure 3E-3L The expanded physical obstacle map 220b includes one or more obstacle-based features 228, such as a user-defined area 228. UR ( Figure 3E-3G ), gap filling region 228 GF ( Figure 3H ) and / or vector field 228 VF ( Figure 3I-3L ). In some examples, the operator of the robot 100 may define the user defined area 228 UR Included in the expanded body obstacle map 220b. User defined area 228 UR Refers to an operator inserting an extended body obstacle map 220b to generate an obstacle shape (e.g., a polygon) (i.e., a virtual obstacle that forms and / or modifies the body obstacle region 224). For example, the operator limits how far the robot 100 can deviate from the route. The operator may interact with an interface (e.g., an application program interface (API)) to draw or select a shape to insert into the extended body obstacle map 220b. For example, the interface is a controller (e.g., a remote control) or some type of terminal (e.g., a computer display). In some embodiments, based on the user-defined region 228 generated by the operator UR , the sensing system 200 defines the user defined area 228 URThe user-defined areas 228 are converted into body obstacle areas 224 on the expanded body obstacle map 220b. UR As part of the expanded physical obstacle map 220b, further restrictions on the path of travel of the robot 100 may result. For example, Figure 3E Delineating two user defined regions 228 UR , 228 UR ab, which is based on the first user defined area 228 UR a's square shape to modify Figure 3F The body obstacle area 224 is shown. Here, the user defines the area 228 UR a is configured to indicate a virtual obstacle such that the perception system 200 will intersect the user-defined area 228 with the physical obstacle area 224 UR The portion of a is integrated into the body obstacle area 224. User defined area 228 UR The user-defined area 228 may be marked as a body obstacle (e.g., integrated with one or more body obstacle regions 224) or not an obstacle (e.g., integrated with one or more no body obstacle regions 226). UR The user-defined area 228 may affect the body trajectory of the robot 100 UR is not input into the step locator 176 to prevent the step locator 176 from reacting to false obstacles (ie, virtual objects).

[0083] refer to Figure 3G and Figure 3H In some configurations, the expanded body obstacle map 220b includes forming a gap filling region 228 GF Gap filling is a processing technology of gap filling of the sensing system 200, which forms a gap filling area 228 by filling in a narrow gap. GF For example, the perception system 200 forms the gap filling region 228 by filling in a narrow gap that is equal to or nearly equal to the width of the robot 100. GF With the gap filler, the sensing system 200 is configured to form a gap filling region 228 GF To clearly distinguish what size passages the robot 100 can fit through. In some examples, the gap filler forms the gap filling region 228 on a cell-by-cell basis according to the distance d and vector v to the closest obstacle boundary included in each cell 222. GF For a given cell 222, the gap filler identifies the closest obstacle boundary of the cell and the closest obstacle boundaries of at least two adjacent cells. When two of the three identified obstacle boundaries are separated by a distance that satisfies a distance filling threshold (e.g., a predetermined distance range), the gap filler fills the gaps to form a gap filling region 228.GF By gap filling, the gap filler ensures that the robot 100 avoids forcing itself into narrow and potentially impassable passages. This seeks to prevent the robot 100 from getting stuck or blocked in narrow passages, especially when the voxel map 210 and the extended body obstacle map 220 may change or update based on the sensor system 130 (e.g., the continuous receipt of sensor data 134). UR Similarly, the sensing system 200 does not fill the gap in the region 228. GF to the step locator 176. The step locator 176 may need to utilize the gap filling area 228 GF to maintain balance (e.g., from slipping or tripping).

[0084] For example, Figure 3G Two body obstacle regions 224a-b are depicted with a gap between each region 224s. As shown, the perception system 200 identifies cells 222 (labeled A, B, and C) and determines the closest obstacle boundary (e.g., shown as vector v) for each cell 222. 1-3 Here, the perception system 200 determines (e.g., via the gap filler) that cell A and cell B identify different closest obstacle boundaries and the distance between these identified closest obstacle boundaries is less than the distance filling threshold. Based on this determination, the gap filler fills the gaps such as Figure 3H The gap-filled region 224 shown spanning the cross-hatched cells GF The cross-hatched cell 222 is shown. Figure 3H This process bridges the two body obstacle regions 224a, 224b ( Figure 3G ) to form a gap-filling region 228 GF A single body obstacle area 224.

[0085] refer to Figure 3I-Figure 3L In some examples, the extended physical obstacle map 220b includes a vector field 228 VF As obstacle-based features 228. With the help of vector fields 228 VF , the extended body obstacle map 220b may allow for potential field-based obstacle avoidance. In other words, the control system 170 (e.g., during body trajectory generation) may avoid obstacles by following the vectors v, v that form the potential field. 1-i Here, the vector field 228 VF The direction of the vector v is defined for each cell 222 by reversing the direction to the closest obstacle boundary of each cell 222 included in the body obstacle map 220. Unfortunately, if there is no vector field 228 VFWith further processing of the field direction (i.e., acquisition direction) of , the direction of vector v may change suddenly from cell 222 to cell 222 (e.g., adjacent cells) because the original field direction derived from the nearest boundary vector is not smooth and often causes the control system to suffer from orbital tremors. If left in this state, the field direction may disrupt potential field-based obstacle avoidance (e.g., through sudden control maneuvers). For example, Fig. 3I Depicting a raw vector field with sudden changes in direction 228 VF Leading to the overall vector field 228 VF Sawtoothing occurs and results in an overall disturbed potential field.

[0086] like Figure 3J As shown, in order to prevent disturbances to the potential field-based obstacle avoidance, the perception system 200 performs further processing to smooth the raw vector field 228 VF Making Meaningful Vector Fields 228 VF can be used for the body trajectory of the robot 100. In some examples, the perception system 200 smoothes the raw vector field 228 by a smoothing kernel VF For example, a smoothing kernel is an average filter over a square area of ​​the cell 222. A smoothing kernel is an image processing technique that may normalize the vector v in its original vector form. For the extended body obstacle map 220b, smoothing may be particularly important when multiple obstacles have overlapping influence areas; meaning that nearby obstacles may affect each other's vector field 228. VF . refer to Figure 3K and 3L , two nearby L-shaped obstacles are represented by the first body obstacle region 224a and the second body obstacle region 224b. Each body obstacle region 224a, 224b has an effect on the original vector field 228. VF Contributes to the original vector field 228 VF The first vector field 228 associated with the first body obstacle region 224a VF a and a second vector field 228 associated with a second body obstacle region 224b VF b depicts. Due to the proximity of these obstacles, Figure 3K The original potential field describing the obstacle is included in the original vector field 228 VF If the control system 170 of the robot 100 is in the original potential field 228 VF If the robot 100 attempts to operate the potential field-based obstacle avoidance in a sharp valley, it will encounter many back-and-forth oscillations (e.g., bounces back and forth) in the direction and magnitude of the vector v based on the original potential field. Here, the smoothing processing techniques performed by the perception system 200 are as follows: Figure 3L As shown, the magnitude of the vector v is related to the vector field 228 VFAdjusted to the path P that the robot 100 can traverse T In other words, smooth identification exists between the two obstacles, where the potential field 228 of each obstacle exists. VF The traversable path P formed by the overlapping cancellation effect T .

[0087] In some examples, the smoothing technique performed by the perception system 200 causes the direction of the vector v to change to the closest boundary. To illustrate, for a narrow gap, the smoothing technique may form a vector field 228 having a potential field VF , the potential field prevents the robot 100 from entering the narrow gap or squeezing the robot 100 out of the end of the narrow gap. To correct the change in direction of the vector v to be closest to the boundary, the perception system 200 readjusts the distance between the vectors v after smoothing. To readjust the distance between the vectors v, the perception system identifies locations where the vector direction is significantly changed by the smoothing technique (e.g., cell 222). For example, the perception system 200 stores the vector field 228 before the smoothing technique VF (eg, the original field vectors based on the vectors to the closest boundary) and convert these vector fields 228 VF and the vector field formed by the smoothing technique 228 VF For comparison, especially with respect to vector fields 228 VF Based on the comparison, perception system 200 re-estimates the distance to the obstacle based on the new direction from the smoothing technique and adjusts the magnitude of vector v in the new direction according to the re-estimated distance. For example, when there is no obstacle along the new direction from the smoothing technique, perception system 200 scales the magnitude of vector v to zero. In some examples, vector field 228 before and after the smoothing technique VF The comparison between identifies vectors v that meet a certain direction change threshold to reduce the number of vectors v that the perception system 200 re-evaluates.

[0088] refer to Figure 4A and Figure 4B , the perception system 200 generates a ground height map 230 based on the voxel map 210. In some embodiments, the ground height map 230 functions such that at each XY location (e.g., a cell labeled as a ground height map 230), the ground height map 230 specifies a height 232. In other words, the ground height map 230 communicates that at a specific XY location on a horizontal plane, the robot 100 should step at a specific height. For example, the ground height map 230 is a 2.5-dimensional (2.5D) map. For practical purposes, if a portion of a table and a portion of the ground exist at an XY location, the ground height map 230 communicates the height 232 of the ground (i.e., where the robot 100 should step) and ignores the table.

[0089] In some examples, perception system 200 can identify segments 214 that are classified as ground G in voxel map 220 (i.e., ground segments 214 G ) is converted into the ground height map 230 to form the ground height map 230. For example, the ground height map 230 and the voxel map 210 use the same grid system, so that the ground classification at a specific location in the voxel map 210 may be directly transferred to the same location in the ground height map 230. In some embodiments, in cells where the voxel map 210 does not include a segment classified as ground, the perception system 200 generates a segment classified as an obstacle (i.e., the obstacle segment 214) in the ground height map 230. OB ).

[0090] When converting the height of each segment 214 of the voxel map 210 classified as ground into the ground height map 230, the perception system 200 attempts to deliver an accurate representation of the height of the segment 214 to ensure ground accuracy. To ensure accuracy, in some examples, for each segment 214 classified as ground, the perception system 200 analyzes the segment 214 starting from the top of the segment 214 (i.e., the highest z-point on the segment 214) and working down along the segment 214 (e.g., along the voxels 212 corresponding to the segment 214). For example Figure 4A Arrows are used to depict ground segments 214 G , the arrow indicates the ground segment 214 G The analysis of each voxel 212a-n at the highest z height of . During this analysis, the perception system 200 determines the specific point weight W of the point P Does the value meet the high accuracy threshold 232 TH Here, the height accuracy threshold is 232 TH indicates the level of accuracy on the height 232 of the object. P is configured so that the point weight W P The larger the value, the higher the confidence level of the representation of the voxels and / or segments regarding the presence of a potential object. The higher the accuracy threshold 232 TH The confidence level of ground height map 230 that helps ensure that voxel map 210 converts accurate height 232 into ground height map 230 for segments 214 that are classified as ground. In some examples, when perception system 200 determines that sufficient points have point weights W P Value meets high accuracy threshold 232 TH (e.g., exceeds a height accuracy threshold), the perception system 200 uses the average height of the point as the classified ground segment 214 G The height 232 is transmitted to the ground height map 230. By using the height accuracy threshold 232 THWith this top-down approach, height 232 is generally accurate regardless of whether segment 214 includes a lot of data or sparse data 134. In some implementations, although not as accurate as techniques using point weights, perception system 200 transmits the height corresponding to the top of segment 214 that is classified as ground to ground height map 230. This approach may be used in situations where less accuracy is tolerated for ground height map 230.

[0091] In some examples, when a segment 214 from the voxel map 210 is not classified as ground (ie, the ground segment 214 G ) or obstacles (ie, obstacle segment 214 OB ), the perception system 200 is configured to generate inferences 234 for missing terrain within the ground height map 230 (e.g., by filling in the gaps). Generally, the perception system 200 uses two main strategies to generate inferences 234, an occlusion-based approach and / or a smoothing-based approach. As the perception system 200 generates the ground height map 230, the perception system 200 identifies discontinuities in the sensor data 134 (e.g., depth sensor data 134). A discontinuity refers to when the sensor data 134 indicates a near object that is adjacent to a far object. When the perception system 200 encounters a discontinuity, the perception system 200 assumes that the near-far contrast is due to an occlusion 234 of the sensor system 130 within the environment 10. O and occurs Figure 4B ). When the perception system 200 assumes that the occlusion 234 O When this occurs, the perception system 200 maps these gaps as flat terrain 234 FT , filling the gaps in the sensor data 134. For example, Figure 4B The robot 100 is depicted traversing a staircase having occlusions at the risers of the staircase based on the sensor data 134. Based on these occlusions, Figure 4B Explain the shielding between the tread parts of the steps 234 O Filled with flat terrain 234 by the perception system 200 FT , the flat terrain 234 FT Bridge the back of the tread of the first step with the front of the tread of the second step above the first step. Here, the bridge is shown as a line segment connecting two points that are adjacent in image space but far apart in the three-dimensional world.

[0092] In contrast, when sensor data 134 does not indicate near-far contrast (i.e., occlusion), perception system 200 assumes that the missing sensor data 134 is due to poor vision of sensor system 130 and maps the missing sensor data 134 to smooth terrain 234. STIn other words, when sensor data 134 is missing and sensor data 134 does not indicate near-far contrast, perception system 200 uses a smoothing technique. In some examples, the smoothing technique used by perception system 200 is an iterative average flood fill algorithm. Here, the algorithm may be configured to extract the actual sensor data 134 and the image from occlusion 234. O In some embodiments, the perception system 200 performs the consideration of negative segments 214. N The smoothing technique makes the negative fragment 214 N The perception system 200 provides a boundary to the inference 234 (i.e., the inference 234 results in a smooth terrain 234 ST In some configurations, since the perception system 200 is concerned with filling gaps, the perception system 200 removes the extrapolated portion even though it performs both interpolation and extrapolation.

[0093] Depending on the inference 234 made by the perception system 200, the perception system 200 maintains (ie, retains) or removes the inferred terrain (eg, flat terrain 234). FT or smooth terrain 234 ST ). For example, perception system 200 is configured to interpret sensor data 134 at a specific frequency (e.g., the frequency at which sensor 132 generates sensor data 134 or certain intervals of frequency). In other words, perception system 200 may perform iterative processing based on received sensor data 134 at set time intervals. Figure 4A When the perception system 200 receives new sensor data 134, the perception system 200 determines whether to maintain the inference 234. When the previous processing iteration performed by the perception system 200 interpreted the sensor data 134 as an occlusion 234 O (i.e., resulting in flat terrain 234 FT ), subsequent processing iterations of the sensor data 134 by the perception system 200 retain the previous flat terrain inference 234 FT Unless sensor system 130 actually provides sensor data 134 to perception system 200 for the occluded area. Instead, when a previous processing iteration performed by perception system 200 inferred the missing sensor data 134 as smooth terrain 234 ST , the perception system 200 clears (i.e., removes) the inferences 234 during subsequent processing iterations of the sensor data 134. More specifically, the perception system 200 is configured to recalculate these types of inferences 234 (i.e., no occlusion) in each iteration of the perception system 200. Thus, whether the inference 234 remains is based on the context of the robot 100. For example, because the occlusion 234 O may be visible only in certain postures of the robot 100 and not in others, if the occlusion 234 ONo maintenance, obstruction 234 O Therefore, the perception system 200 maintains the occlusion 234 O In contrast, smooth terrain 234 ST It does not depend on the state of the robot 100, but on the mapping state. In other words, remove the smooth terrain 234 ST This does not pose any risk to the robot 100 because the additional state information is not included in the smooth terrain 234. ST Thus, the perception system 200 can simply reconstruct the smooth terrain 234 in the following iteration ST .

[0094] In some embodiments, during subsequent iterations, perception system 200 evaluates occlusion-based inferences 234 O Is it still adjacent to the actual sensor data 134 (e.g., the actual point of the sensor data 134 is adjacent to the flat terrain 234 FT Here, even though the general rule of the perception system 200 is to retain the occlusion-based inference 234 O , when occlusion-based inference 234 O The evaluation identified occlusion-based inference 234 O No longer adjacent to any current sensor data 134, the perception system 200 removes these unattached occlusion-based inferences 234 O .

[0095] During creation of the ground height map 230, the perception system 200 may be configured to fill in narrow or small pits 236 that occur within the sensor data 134. Figure 4B ). Generally, pits 236 refer to gaps (e.g., holes or cavities) in the sensor data 134. When these pits 236 occur, these pits 236 are likely related to bad sensor data 134 rather than actual terrain irregularities. In some configurations, the perception system 200 is configured to fill pits 236 that are smaller than the ends 124 of the legs 120 of the robot 100 (e.g., feet). In other words, the perception system 200 may include a pit filling threshold that is set to a value that is less than the ground contact surface area of ​​the feet of the robot 100. When the robot 100 is multi-legged and the ground contact surface area of ​​the feet varies from leg 120 to leg 120, the pit filling threshold may be set to the minimum ground contact surface area of ​​all the feet of the robot 100. In some examples, the perception system 200 uses morphological dilation followed by morphological erosion to fill the pits 236, wherein the perception system 200 uses real values ​​instead of Boolean values.

[0096] In some embodiments, the perception system 200 generates a ground height map 230 with further processing to widen the obstacle O (e.g., increase the size of the obstacle). By widening the obstacle O, the ground height map 230 helps the robot 100 (e.g., as the robot 100 swings its legs 120 while maneuvering about the environment 10) avoid the terrain. In other words, by widening the obstacle O, the perception system 200 allows the ground height map 230 to have a buffer between the location of the obstacle O on the map 230 and the actual location of the obstacle O. This buffer allows components of the robot 100, such as the knees, feet, or other joints J of the robot 100, to be further constrained so that there is a margin for movement error before the portion of the robot 100 collides with the actual obstacle O. For example, a wall within the ground height map 230 may be widened (e.g., offset) to a space of approximately six centimeters adjacent to the wall to provide a buffer for object avoidance.

[0097] In some examples, for each cell of the map 230, the ground height map 230 includes a ground height estimate 232 est and ground height accuracy assessment 232 Acc Here, the perception system 200 generates a ground height accuracy estimate 232 Acc To indicate the accuracy of the ground height 232. In some embodiments, the ground height accuracy 232 Acc The sensor data 134 indicates how well the perception system 200 perceives the ground surface and the odometry drift of the robot 100. For example, when the perception system 200 has not received sensor data 134 visualizing the ground surface G for a particular cell for approximately three seconds and the odometry of the robot 100 is drifting by approximately one centimeter per second, the perception system 200 associates (e.g., modifies) the pre-existing ground height accuracy 232 Acc or additional ground height 232 to include) and ground height accuracy 232 of ground height 232 + / - three centimeters Acc The method may be used to determine under what circumstances the robot's control system 170 may trust the ground height estimate 232 est (For example, according to the specified ground height assessment 232 est to operate).

[0098] refer to Figure 5A-Figure 5I In some embodiments, the perception system 200 is configured to generate a no-step map 240. The no-step map 240 generally refers to a map that defines areas where the robot 100 is not allowed to step in order to suggest that the robot 100 may step when the robot 100 is at a specific horizontal position (i.e., a position in the XY plane). In some examples, similar to the body obstacle map 220 and the ground height map 230, the no-step map 240 is divided into cells 242, 242 1-, where each cell 242 represents a specific area of ​​the robot 100 in the environment 10. For example, each cell 242 is three square centimeters. For ease of explanation, each cell 242 exists in the XY plane of the environment 10. When the perception system 200 generates the no-step map 240, the perception system 200 may generate a Boolean map, such as Figure 5A As shown, the Boolean value map identifies a no-stepping area 244 (e.g., shown as a gray cell) and a stepping area 246 (e.g., shown as a white cell). The no-stepping area 244 refers to an area of ​​one or more cells 242 where an obstacle O exists, and the stepping area 246 refers to an area of ​​one or more cells 242 where the obstacle O is not perceived.

[0099] In some configurations, such as Figure 5B and 5C , the perception system 200 further processes the Boolean map so that the no-step map 240 includes a signed distance field. Here, the signed distance field of the no-step map 240 includes a distance d to the boundary of the obstacle O (e.g., the distance d to the boundary of the no-step region 244) and a vector v to the boundary of the obstacle O (e.g., defining the closest direction to the boundary of the no-step region 244). Although the signed distance field may be combined so that the cell 242 includes both the distance d to the boundary of the obstacle O and the vector v, for ease of explanation, Figure 5B and 5C Represent each element individually (i.e. Figure 5B The distance d is shown. Figure 5C, vector v is shown). In a signed distance field, the sign of the value indicates whether the cell 242 is within the boundary of the obstacle O (e.g., the sign of the distance is negative) or outside the boundary of the obstacle O (e.g., the sign of the distance is positive). In other words, the farther the cell 242 is from the boundary, the larger the distance value of the cell 242 (e.g., whether it is within the boundary or outside the boundary). In some examples, the no-step graph 240 represents the distance d of a particular cell 242 to the boundary of the obstacle O as a count of the number of cells 242 between the particular cell 242 and the boundary of the obstacle O. In other examples, the no-step graph 240 represents the distance d to the boundary of the obstacle O as the actual distance, so that the no-step graph 240 provides greater granularity (e.g., than the cell count) and therefore greater accuracy in maneuvering the robot 100 with respect to the object 15 of the environment 10. For example, when the cell 242 is on the boundary of the obstacle O, the perception system 200 represents the distance as zero without a sign. With the distance d to the boundary of obstacle O and the vector v to the closest boundary of obstacle O, the no-step graph 240 can communicate (e.g., to the control system 170 of the robot 100) not only where to step and where not to step, but also the potential space to step into (e.g., within the stepping region 246). To illustrate, a normal gait pattern of the robot 100 may indicate that the foot of the robot 100 will be placed in the no-step region 244. Because the control system 170 utilizes the no-step graph 240, the control system 170 can recognize in this example that the control system 170 should not place the foot in the no-step region 244 as originally planned, but instead place the foot within the stepping region 246 (e.g., the foot placement of the group of cells 242 that are closest to the originally planned). In some examples, by interpreting the vector v and distance of the cells 242 within the no-step graph 240, the control system 170 can minimize the disturbance to the gait pattern of the robot 100 (e.g., to help step without slipping during gait adjustment to maintain balance).

[0100] When generating the no-step map 240, the perception system 200 may identify the no-step region 244 and / or the step region 246 for a number of different reasons. Some of the reasons may include a slope within the region, a potential risk of shin collision within the region, the presence of a pit within the region, the presence of a no-swing shadow within the region, and / or the possibility of the robot 100 self-colliding within the region. In some examples, the perception system 200 calculates the slope within the region by applying two different scale filters (e.g., Sobel filters) to the sensor data 134. With the two different scales, the first filter may detect small-scale slopes and the second filter detects large-scale slopes. The perception system 200 marks the region as a no-step region 244 when the small-scale slope is high (i.e., a first condition) and the small-scale slope is greater than the large-scale slope (i.e., a second condition). For example, the perception system 200 is configured with a slope threshold such that when the value of the small-scale slope meets the slope threshold (e.g., is greater than the slope threshold), the perception system 200 marks the small-scale slope as high (i.e., meets the first condition). The same slope threshold or other slope thresholds may be configured to indicate a threshold difference between the value of the small-scale slope and the value of the large-scale slope. Here, when the difference between the value of the small-scale slope and the value of the large-scale slope meets the threshold difference (e.g., exceeds the threshold difference), the perception system 200 identifies the small-scale slope as being greater than the large-scale slope (i.e., the second condition is met). When both the first condition and the second condition are met for a specified area, the perception system 200 marks the area as a no-step area 244. In other words, a mountain may be navigated by the robot 100 (e.g., because both the small-scale slope and the large-scale slope are large) while the edge of the step is not navigable (e.g., because the small-scale slope is high and the large-scale slope is less than the small-scale slope). More generally, the perception system 200 attempts to identify areas where the slope is steeper than the surrounding area and is also steep enough (e.g., steep enough to be problematic for the robot 100 to maintain balance during movement).

[0101] refer to Figure 5D As previously described, the perception system 200 may identify the area as a no-stepping area 244 to avoid shin collision. Generally, based on the structure of the leg 120 of the robot 100, the shin 122 of the leg 120 L The ground surface 12 has a specific slope. In some embodiments, one or more legs 120 of the robot 100 are configured so that the joint (i.e., hip joint J) connecting the leg 120 to the body 110 is H ) is located at the knee joint J in the sagittal plane of the body 110 of the robot 100 K Front. Here, the knee joint J K It refers to the upper members 122, 122 of the legs 120 U The lower member 122, 122 is connected to the leg 120 including the end 124 LBased on this configuration, when the robot 100 moves faster, the end 124 moves away from the hip joint J. H stride further (ie, further forward in the sagittal plane); causing the lower member 122 of the leg 120 to L (i.e., tibia) is more horizontal with respect to the ground surface 12 at a faster speed (or stationary) than at a slower speed. Here, when the robot 100 executes a movement command (e.g., from the control system 170) for a specific speed, the commanded speed requires the lower member 122 to move horizontally. L Due to this structural anatomy of the robot 100, the perception system 200 determines whether the ground height 232 of the ground height map 230 will be consistent with the lower member 122 when the robot 100 moves through the environment 10 at the desired speed. L When the perception system 200 determines that the ground height 232 of a cell 242 may cause a collision, the perception system 200 designates the cell 242 as a no-stepping region 244. For example, based on the anatomy of the leg 120 (i.e., the known dimensions of the leg 120) and the minimum tibia slope during the movement of the robot 100, the perception system 200 determines that the collision height h of the leg 120 of the robot 100 is C And the collision height h C is compared with the ground height 232 of the ground height map 230. In some configurations, when the collision height h C When compared to the ground height 232, the perception system 200 will have a collision height greater than or equal to h C Any cell with a ground height 232 less than 230° is identified as a no-step region 244 of no-step map 240 .

[0102] In some examples, for each cell 242, the sensing system 200 detects a location along the lower member 122. L In other words, the perception system 200 samples the nearby or adjacent cells 242 based on the cell 242 (herein referred to as the footprint cell) where the foot of the robot 100 is located or theoretically located (for example, based on the fixed yaw of the robot 100), and identifies the foot that will be located on the lower member 122. L With the help of the identified cell 242, the sensing system 200 determines the collision height hc as the lower member 122 in the path of the stride of the leg 120. L The sensing system 200 determines the minimum expected height of the footsteps (i.e., minimum tibial slope) and identifies any of those cells 242 that would interfere with the minimum tibial slope (i.e., cells that would be under the leg 120). When any cell 242 would cause interference, the sensing system 200 identifies the footprint cell as an illegal stepping space (i.e., no stepping cell / region).

[0103] Because the minimum tibia slope may change as the speed of the robot 100 changes, the perception system 200 may adjust the no-step map 240 whenever the control system 170 implements or modifies the speed of the robot 100 (e.g., the legs 120 of the robot 100) or at some frequency intervals after the speed is input by the control system 170. In some examples, when determining whether a collision will occur to the legs 120 of the robot 100, the perception system 200 additionally considers the current yaw (i.e., rotation about the z-axis) of the body 110 of the robot 100 and / or the direction of movement of the robot 100. In other words, when the robot 100 moves parallel to the wall, the wall on one side of the robot 100 does not pose a risk of collision with the robot 100, but when the robot 100 moves perpendicular to the wall, the wall poses a risk of collision.

[0104] For specific terrain, such as steps (e.g. Figure 5H and Fig.5I As shown in FIG. 2 ), when the perception system 200 combines shin collision avoidance and slope avoidance to generate a no-step map 240, the perception system 200 is at risk of over-inclusion when specifying a no-step area 244 of the no-step map 240. In order to prevent the perception system 200 from being over-inclusive and thereby preventing the robot 100 from crossing steps, in some embodiments, the perception system 200 is configured to ensure that there is a stepping area 246 (e.g., a stepping area 246 of at least one cell width) on each step of the detected stairs. In some examples, the perception system 200 forms a stepping area 246 on each step by determining the least dangerous no-step condition. In some embodiments, for steps, the least dangerous no-step condition is a shin collision. In these embodiments, the perception system 200 ignores potential shin collisions within the step area of ​​the ground height map 230. For example, the perception system 200 ignores potential shin collisions that fail to meet a shin collision threshold. Here, the shin collision threshold divides the shin collision height h C The difference between the height 232 of the ground and the ground should indicate a high risk of shin impact (e.g., a wall) and a low risk of shin impact (e.g., a riser of a stair). To ensure at least one cell-wide stepping area 246 per stair, the shin impact threshold may be configured based on the building standard dimensions of the stair. In some examples, shin impact is the least dangerous no-stepping impact because the shin impact is based on an assumed angle, but the control system 170 may adjust the angle of the shin by changing the body 110 of the robot 100 relative to the legs 120 of the robot 100.

[0105] Optionally, the perception system 200 is configured to designate areas of the sensor data 134 indicating narrow pits 236 or trenches as no-stepping areas 244 when generating the no-stepping map 240. Generally speaking, when generating the ground height map 230, the perception system 200 should remove or fill the narrow pits 236. In some examples, since the perception system 200 generates the no-stepping map 240 based on the ground height map 230, the remaining narrow pits 236 may prove to be a problem for the robot 100. In these examples, when generating the no-stepping map 240 by designating the narrow pits 236 as no-stepping areas 244, the perception system 200 avoids the narrow pits 236 from being perceived. Although the perception system 200 is configured to fill the narrow pits 236 (i.e., remove these pits 236 through processing techniques) during the generation of the ground height map 230, by designating the narrow pits 236 as no-step areas 244, the no-step map 240 ensures that potentially bad data areas do not cause problems for the robot 100 when the robot 100 moves with respect to the environment 10.

[0106] In some examples, such as Figure 5E , the perception system 200 includes a no-sway region 248 within the no-step graph 240 R and no swing shadow 248 S . No swing area 248 R Refers to an area with obstacles that the robot 100 cannot travel through (i.e., a physical obstacle). For example, there is an object 14 (e.g., a log - although not shown) on the ground surface 12 within the environment 10, but the robot 100 cannot transfer one or more legs 120 over and / or around the object (e.g., lift its legs 120 high enough to pass through the object 14). Even if there is flat ground on the other side of the object 14, the robot 100 cannot step to the other side based on the current position of the robot 100. Here, the perception system 200 identifies the object 14 as a no-swing area 248 R and identifies the other side of the object 14 as a non-swinging shadow 248 S , because the robot 100 cannot enter the area on the other side of the object 14. In other words, there is no swing shadow 248 S Indicates areas that the robot 100 cannot physically enter based on the current position and / or rotation (e.g., yaw) due to obstacles (i.e., areas that the robot 100 cannot access based on the current posture P of the robot 100, but are accessible to the robot 100 in a different posture P). For example, Figure 5E Indicates that the area adjacent to the rear legs 120c-d of the robot 100 includes the no-sway area 248 R and no swing shadow 248 S area, because the robot 100 must move one leg 120 across the other legs 120. Here, Figure 5EThe color pattern of the image indicates the following: the stepping area 246 is white or cross-hatched white (e.g., on the steps); the non-stepping area 244 (i.e., the area without stepping for reasons other than no swing or no swing shading) is light gray; the non-swinging area 248 is light gray. R Dark gray; no swing shadow 248 S In some embodiments, the perception system 200 uses a flood fill algorithm issued at the convex hull of the end 124 of the leg 120 of the robot 100 to form a non-swaying shadow 248 S .

[0107] refer to Fig. 5F and Figure 5G Additionally or alternatively, the perception system 200 generates a no-step map 240 for one or more individual legs 120 of the robot 100 at the current stepping position of the robot 100. Typically, the no-step map 240 is generally valid for all legs 120 of the robot 100. For example, Fig. 5F A no-swing graph 240 for all legs 120 is depicted with cross-hatched white areas illustrating three swing regions 246, 246a-c. More specifically, the first swing region 246, 246a indicates a large area on the ground in front of a step that a leg 120 may swing into. However, there may be some configurations where the control system 170 is concerned about collisions (also referred to as self-collisions) between legs 120. Compare Figure 5F to Figure 5G , Figure 5G A no-swing map 240 specifically for the right front leg 120 of the robot 100 is illustrated. In this particular no-swing map 240, the perception system 200 generates no-swing regions 244 (shown in black) to additionally identify areas where movement of the right front leg 120 to the area would result in a collision between the legs 120 if the right front leg 120 of the robot 100 were to step into the area. For example, the perception system 200 generates a first no-swing region 244, 244a near the left front leg 120 and to the left of the left front leg 120. In some embodiments, the perception system 200 uses the no-swing shadow 248 to identify the area. S Similar techniques are used to generate these no-stepping areas 244. In some configurations, the perception system 200 uses the ratio with the no-swinging shadows 248 s Similar techniques more quickly compute predefined shapes to generate these no-step regions 244 .

[0108] refer to Figure 5H and 5IIn some embodiments, the perception system 200 further processes the no-step map 240 to indicate a distinction between locations where the robot 100 may move and locations where the robot 100 may not move. More specifically, the no-step map 240 indicates no-step regions 244 that the robot 100 may have to move through in order to move to a step region 246 of the map 240. For example, Figure 5H The robot 100 is depicted traversing a portion of stairs having no-step regions 244 (e.g., shown as white areas) on each rung such that the legs 120 of the robot 100 step over these no-step regions 244 to place the ends 124 of the legs 120 into step regions 246 (e.g., shown as black and white diagonal patterned areas). Even though these no-step regions 244 may be used by the control system 170 for some functions (e.g., movement planning), other aspects of the robot 100 and / or the control system 170 may not require such granularity. In other words, one or more controllers of the control system 170 may prefer to know whether portions of the more general map 240 are being navigated by the robot 100. Thus, in some examples, the perception system 200 generates large area no-step maps 240, 240 B , which indicates whether an area of ​​the map 240 is navigable, even though some portions of the area may not have a no-step region 244. B , the perception system 200 first performs morphological weakening on the no-step graph 240, followed by morphological dilation. Generally speaking, the weakening technique peels off the boundaries of the region (e.g., removing small no-step regions 244) and then the dilation technique expands the more dominant step regions 246; resulting in a more general distinction between whether the region is navigable by the robot 100 (e.g., such as Fig.5I In other words, the technique attempts to remove small areas without changing large areas. In some configurations, the robot 100 uses the large area no step map 240 B The other graphs 210 , 220 , 230 are combined to convey different information to portions of the control system 170 .

[0109] Figure 66 is an example of a method 600 for generating a voxel map 210. At operation 602, the method 600 receives at least one set of raw sensor data 134 and a current set of sensor data 134. Each of the at least one set of raw sensor data 134 and the current set of sensor data 134 corresponds to an environment 10 about a robot 100 from at least one sensor 132. Here, the robot 100 includes a body 110 and legs 120. At operation 604, the method 600 includes generating a voxel map 210 including a plurality of voxels 212 based on the at least one set of raw sensor data 134. The plurality of voxels 212 include at least one ground voxel 212 and at least one obstacle voxel 212. At operation 606, the method 600 generates a spherical depth map based on the current set of sensor data 134. At operation 608, the method 600 includes determining that an obstacle represented by the voxel map 210 has changed based on a comparison between the voxel map 210 and the spherical depth map. At operation 610 , method 600 includes updating voxel map 210 to reflect changes to obstacles in environment 10 .

[0110] Figure 7 is an example of a method 700 for generating a body obstacle map 220. At operation 702, the method 700 receives sensor data 134 corresponding to an environment 10 about a robot 100 from at least one sensor 132. Here, the robot 100 includes a body 110 and legs 120. At operation 704, the method 700 generates a voxel map 210 including a plurality of voxels 212 based on the sensor data 134, wherein the plurality of voxels 212 includes at least one ground voxel 212. G and at least one obstacle pixel 212 OB At operation 706, based on the voxel map 210, the method 700 generates a body obstacle map 220 configured to indicate locations in the environment 10 where the body 110 of the robot 100 can move without being disturbed by obstacles in the environment 10. The body obstacle map 220 is divided into cells 222, wherein a plurality of cells 222 include indications of the closest obstacles O. Here, the closest obstacle O is derived from at least one obstacle O of the voxel map 210. At operation 708, the method 700 transmits the body obstacle map 220 to the control system 170 of the robot 100. The control system 170 is configured to move the robot 100 with respect to the environment 10.

[0111] Figure 8is an example of a method 800 for generating a ground height map 230. At operation 802, the method 800 receives sensor data 134 corresponding to an environment 10 about a robot 100 from at least one sensor 132. Here, the robot 100 includes a body 110 and legs 120, wherein each leg 120 includes a tip 124. At operation 804, the method 800 generates a voxel map 210 including a plurality of segments 214 based on the sensor data 134, wherein the plurality of segments 214 include at least one ground segment 214. G and at least one obstacle segment 214 OB Each segment 214 of the plurality of segments 214 corresponds to a vertical column defined by one or more voxels 212. At operation 806, based on the voxel map 210, the method 800 generates a ground height map 230 configured to indicate the height at which the tip 124 of the corresponding leg 120 of the robot 100 is placed when the robot 100 moves with respect to the environment 10. Here, the ground height map 230 is divided into cells, wherein at least one cell corresponds to a corresponding ground segment 214. G and includes a corresponding ground segment 214 G At operation 808 , the method 800 transmits the ground height map 230 to the control system 170 of the robot 100 , wherein the control system 170 is configured to move the tip 124 of the corresponding leg 120 to a placement position in the environment 10 based on the ground height map 230 .

[0112] Fig. 9 is an example of a method 900 for generating a step-free map 240. At operation 902, the method 900 receives sensor data 134 corresponding to an environment 10 about a robot 100 from at least one sensor 132. The robot includes a body 110 and legs 120, wherein each leg 120 includes a tip 124. At operation 904, the method 900 generates a voxel map 210 including a plurality of segments 214 based on the sensor data 134, wherein the plurality of segments 214 include at least one ground segment 214. G and at least one obstacle segment 214 OBEach segment 214 of the plurality of segments 214 corresponds to a vertical column defined by one or more voxels 212. At operation 906, based on the voxel map 210, the method 900 generates a ground height map 230 configured to indicate a height at which the tip 124 of the corresponding leg 120 of the robot 100 is placed when the robot 100 moves with respect to the environment 10. At operation 908, based on the ground height map 230, the method 900 generates a no-step map 240 including one or more no-step regions 244. Each no-step region 244 is configured to indicate an area where the tip 124 of the corresponding leg 120 of the robot 100 is not placed when the robot 100 moves with respect to the environment 10. Here, the no-step map is divided into cells 242, wherein each cell 242 includes a distance value and a direction vector v. The distance value indicates a distance to a boundary of a closest obstacle of the cell 242, and the direction vector v indicates a direction to a boundary of the closest obstacle of the cell 242. At operation 910 , the method 900 communicates the no-step map 240 to the control system 170 , wherein the control system 170 is configured to move the distal end 124 of the corresponding leg 120 to a placement position in the environment 10 based on the no-step map 240 .

[0113] Fig.10 1 is a schematic diagram of an example computing device 1000 that may be used to implement the systems (e.g., sensor system 130, control system 170, perception system 200, etc.) and methods (e.g., methods 600, 700, 800, 900) described herein. Computing device 1000 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are merely exemplary and are not meant to limit the implementation of the inventions described and / or claimed herein.

[0114] The computing device 1000 includes a processor 1010 (e.g., data processing hardware 142, 162), a memory 1020 (e.g., memory hardware 144, 164), a storage device 1030, a high-speed interface / controller 1040 connected to the memory 1020 and a high-speed expansion port 1050, and a low-speed interface / controller 1060 connected to a low-speed bus 1070 and the storage device 1030. Each component 1010, 1020, 1030, 1040, 1050, and 1060 is interconnected using various buses and may be mounted on a common motherboard or in other appropriate ways. The processor 1010 is capable of processing instructions executed within the computing device 1000, including instructions stored in the memory 1020 or on the storage device 1030 to display graphical information of a graphical user interface (GUI) on an external input / output device, such as a display 1080 combined with the high-speed interface 1040. In other embodiments, multiple processors and / or multiple buses may be used as appropriate with multiple memories and memory types. In addition, multiple computing devices 1000 may be connected, with each device providing portions of the desired operations (e.g., as a server array, a group of blade servers, or a multi-processor system).

[0115] Memory 1020 stores information non-temporarily within computing device 1000. Memory 1020 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. Non-temporary memory 1020 may be a physical device used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by computing device 1000. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0116] Storage device 1030 can provide mass storage for computing device 1000. In some embodiments, storage device 1030 is a computer readable medium. In various embodiments, storage device 1030 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices, including devices in a storage area network or other configuration. In additional embodiments, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, as described above. The information carrier is a computer or machine readable medium, such as memory 1020, storage device 1030, or memory on processor 1010.

[0117] The high-speed controller 1040 manages bandwidth-intensive operations of the computing device 1000, while the low-speed controller 1060 manages less bandwidth-intensive operations. Such a division of responsibilities is exemplary only. In some embodiments, the high-speed controller 1040 is combined with the memory 1020, the display 1080 (e.g., through a graphics processor or accelerator), and with the high-speed expansion port 1050 that may accept various expansion cards (not shown). In some embodiments, the low-speed controller 1060 is combined with the storage device 1030 and the low-speed expansion port 1090. The low-speed expansion port 1090, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be combined with one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a network device such as a switch or router, for example, through a network adapter.

[0118] The computing device 1000 may be implemented in many different forms, as shown. For example, it may be implemented as a standard server 1000a or multiple implementations in such a server 1000a, as a laptop computer 1000b, as part of a rack server system 500c, or as part of a robot 100.

[0119] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs executable and / or interpretable in a programmable system including at least one programmable processor, which may be special-purpose or general-purpose, in combination to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to a storage system, at least one input device, and at least one output device.

[0120] These computer programs (also referred to as programs, software, software applications or code) include machine instructions for a programmable processor and can be implemented in high-level processing and / or object-oriented programming languages ​​and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receives machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0121] The processing and logic flow described in this specification can be performed by one or more programmable processors, which execute one or more computer programs to perform functions by operating input data and generating outputs. Processing and logic flow can also be performed by a dedicated logic circuit (for example, FPGA (field programmable gate array) or ASIC (application-specific integrated circuit)). Processors suitable for executing computer programs include, for example, general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Usually, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor that executes instructions and one or more memory devices that store instructions and data. Usually, a computer will also include or be operably combined to receive data from one or more large-capacity storage devices (for example, magnetic, permanent magnet optical disks, or optical disks) for storing data, or to send data to it, or both. However, a computer does not need to have such a device. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; permanent magnet optical disks; and CD ROM and DVD-ROM disks. The processor and memory can be supplemented or combined with special purpose logic circuits.

[0122] To provide interaction with a user, one or more aspects of the disclosure can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen for displaying information to the user, and an optional keyboard and pointing device, e.g., a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, voice, or tactile input. In addition, the computer can interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0123] A number of implementations have been described, however, it will be appreciated that various modifications may be made without departing from the spirit and scope of the disclosure.

Claims

1. A method comprising: receiving, at data processing hardware, a first set of sensor data and a second set of sensor data from at least one sensor of a robot, each of the first set of sensor data and the second set of sensor data corresponding to an environment of the robot; generating, by the data processing hardware, a voxel map comprising a plurality of voxels based on the first set of sensor data, the plurality of voxels comprising at least one ground voxel and at least one obstacle voxel; generating, by the data processing hardware, a spherical depth map based on a second set of the sensor data; comparing, by the data processing hardware, the voxel map and the spherical depth map; determining, by the data processing hardware, that an object represented as an obstacle by the voxel map has changed based on comparing the voxel map and the spherical depth map; as well as The voxel map is updated, via the data processing hardware, to reflect the changes to the object.

2. The method of claim 1, wherein: The robot includes four legs defining a quadruped.

3. The method of claim 1, wherein: Generating the voxel map comprises: determining that one or more three-dimensional cells of space about the robot are occupied; and For each of the one or more three-dimensional cells, the three-dimensional cell is classified as corresponding to ground, an obstacle, or neither ground nor an obstacle.

4. The method of claim 1, wherein: The voxel map comprises a three-dimensional grid of cells, and wherein the method further comprises, for each cell of the three-dimensional grid of cells, merging two or more consecutive voxels of corresponding columns to form a fragment, the fragment being associated with the height and the point weight W P Associated, the point weight W P Indicating a degree of certainty that one or more voxels forming the segment are occupied based on the first set of sensor data.

5. The method of claim 4, further comprising: Based on the second set of sensor data not including sensor data defining the corresponding segment, reducing the point weight W of the segment P .

6. The method of claim 4, further comprising: comparing the height to a range of heights from a column at a corresponding position in the spherical depth map, the position of the segment and the corresponding position of the column corresponding to the same position relative to the robot; and It is determined that the altitude does not match the altitude range.

7. The method of claim 4, wherein: The fragment corresponds to the object, wherein updating the voxel map includes pruning the fragment.

8. The method according to any one of claims 1 to 7, wherein: The spherical depth map includes a spherical representation of the second set of sensor data, the spherical representation including one or more rectangular structures defined by one or more points of the second set of sensor data at a distance and height from the at least one sensor.

9. The method according to any one of claims 1 to 7, wherein: Updating the voxel map includes removing one or more voxels corresponding to the object from the voxel map.

10. The method of claim 9, wherein: Removing the one or more voxels includes: using a heuristic method to identify the one or more voxels associated with the change in the object; and The one or more voxels are removed.

11. A method comprising: receiving, at data processing hardware, sensor data corresponding to an environment of a robot from at least one sensor of the robot, the robot comprising a body; generating, by the data processing hardware, a voxel map comprising a plurality of voxels based on the sensor data, the plurality of voxels comprising at least one ground voxel and at least one obstacle voxel; generating, by the data processing hardware, a spherical depth map based on the sensor data; comparing, by the data processing hardware, the voxel map and the spherical depth map; determining, by the data processing hardware, that an object represented as an obstacle by the voxel map has changed based on comparing the voxel map and the spherical depth map; updating, by the data processing hardware, the voxel map to reflect the change to the object; generating, by the data processing hardware, a body obstacle map based on the voxel map or the updated voxel map, the body obstacle map indicating one or more locations of movement of the body in the environment without interfering with one or more obstacles within the environment, the body obstacle map being divided into a plurality of cells, wherein the plurality of cells include an indication of a closest obstacle boundary, the closest obstacle boundary being derived from the at least one obstacle voxel; as well as The body obstacle map is transmitted, via the data processing hardware, to a control system configured to cause the robot to move around the environment.

12. The method of claim 11, wherein: The indication includes an estimate of a distance to the closest obstacle boundary and a direction to the closest obstacle boundary.

13. The method of claim 12, wherein: Generating the body obstacle map includes generating a vector field, the vector field including a plurality of vectors, each of the plurality of vectors indicating a corresponding direction of obstacle avoidance, wherein each of the plurality of vectors includes a vector direction opposite to the direction to the closest obstacle boundary.

14. The method according to any one of claims 11 to 13, wherein: The control system is configured to control horizontal movement of the body and yaw rotation of the body using the body obstacle map.

15. The method according to any one of claims 11 to 13, further comprising: filtering, by the data processing hardware, the plurality of voxels based on a point weight associated with each voxel of the plurality of voxels, the point weight indicating a degree of certainty that the corresponding voxel is occupied based on the sensor data, Wherein, generating the body obstacle map includes converting a plurality of filtered voxels into the body obstacle map, wherein the plurality of filtered voxels satisfy a point weight threshold and correspond to obstacle voxels.

16. A method comprising: receiving, at data processing hardware, sensor data corresponding to an environment of the robot from at least one sensor of the robot, the robot comprising a body and two or more legs, each of the two or more legs comprising a respective tip; generating, by the data processing hardware, a voxel map comprising a plurality of segments based on the sensor data, each segment of the plurality of segments corresponding to a column defined by one or more voxels, the plurality of segments comprising at least one ground segment and at least one obstacle segment; generating, by the data processing hardware, a spherical depth map based on the sensor data; comparing, by the data processing hardware, the voxel map and the spherical depth map; determining, by the data processing hardware, that an object represented as an obstacle by the voxel map has changed based on comparing the voxel map and the spherical depth map; updating, by the data processing hardware, the voxel map to reflect the change to the object; Based on the voxel map or the updated voxel map, generating, by the data processing hardware, a ground height map, the ground height map indicating one or more heights at which the ends of the two or more legs are placed, the ground height map being divided into a plurality of cells, wherein at least one cell of the plurality of cells corresponds to a ground segment, wherein the at least one cell includes a height based on the ground segment; and The ground height map is transmitted to a control system via the data processing hardware, and the control system is configured to move the terminal to a placement position in the environment based on the ground height map.

17. The method of claim 16, wherein: Generating the ground height map includes determining a point weight W of one or more voxels of the ground segment. P Satisfying the high accuracy threshold, the point weight W P Indicating a degree of certainty that a voxel is occupied based on the sensor data, the height accuracy threshold indicates a level of accuracy of the height of a given object represented by the ground segment.

18. The method of claim 17, wherein: Determine the point weight W P Satisfying the height accuracy threshold includes traversing the one or more voxels from a highest height of the ground segment to a lowest height of the ground segment.

19. The method of any one of claims 16 to 18, further comprising: identifying one or more cells of the plurality of cells corresponding to missing terrain; determining whether the missing terrain corresponds to an occlusion of the sensor data; as well as When the missing terrain corresponds to the occlusion, the missing terrain is replaced with a flat terrain.

20. The method of claim 19, further comprising, when the missing terrain does not correspond to the occlusion, replacing the missing terrain with smooth terrain.

21. The method of claim 19, wherein: The flat terrain remains within the ground height map until the sensor data indicates terrain corresponding to the flat terrain.

22. The method of claim 20, wherein: During subsequent iterations of the ground height map, the smooth terrain is not maintained within the ground height map.

23. A method comprising: receiving, at data processing hardware, sensor data corresponding to an environment of the robot from at least one sensor of the robot, the robot comprising a body and two or more legs, each of the two or more legs comprising a respective tip; generating, by the data processing hardware, a voxel map comprising a plurality of segments based on the sensor data, each segment of the plurality of segments corresponding to a column defined by one or more voxels, the plurality of segments comprising at least one ground segment and at least one obstacle segment; generating, by the data processing hardware, a spherical depth map based on the sensor data; comparing, by the data processing hardware, the voxel map and the spherical depth map; determining, by the data processing hardware, that an object represented as an obstacle by the voxel map has changed based on comparing the voxel map and the spherical depth map; updating, by the data processing hardware, the voxel map to reflect the change to the object; generating, by the data processing hardware, a ground height map based on the voxel map or the updated voxel map, the ground height map indicating one or more heights at which the tips of the legs of the two or more legs are placed; Based on the ground height map, generating a no-step map by the data processing hardware, the no-step map comprising one or more no-step areas, each of the one or more no-step areas indicating a corresponding area where the end is not placed, the no-step map being divided into a plurality of cells, wherein each of the plurality of cells comprises a corresponding distance value and a corresponding direction vector, each of the corresponding distance values ​​indicating a corresponding distance from a boundary of a corresponding closest obstacle of the corresponding cell, and each of the corresponding direction vectors indicating a corresponding direction from the boundary of the corresponding closest obstacle of the cell; as well as The no-step map is transmitted to a control system via the data processing hardware, and the control system is configured to move the extremity to a placement location in the environment based on the no-step map.

24. The method of claim 23, wherein: Each of the respective distances includes a respective symbol identifying whether the respective cell is within the respective nearest obstacle or outside the respective nearest obstacle.

25. The method of claim 23, further comprising determining a first no-swing zone corresponding to a potential shin impact by: determining a minimum slope of the leg to achieve a commanded speed; identifying a tibia impact height based on the minimum slope; and For each cell of the plurality of cells, the tibia impact height is compared to a corresponding height of a corresponding cell, the corresponding height of the corresponding cell being received from the ground height map.

26. The method of claim 25, further comprising determining that a difference between the shin impact height and the corresponding height of the corresponding cell satisfies a shin impact threshold.

27. The method of any one of claims 23 to 26, wherein: At least one of the one or more no-swing regions indicates an area that is inaccessible to the robot based on a first posture of the robot, the area being accessible to the robot having a second posture different from the first posture.

28. The method of any one of claims 23 to 26, wherein: Generating the no-step map includes generating the no-step map of the leg.

Images