Using a depth sensor at the bottom of a legged robot for stair climbing

Abstract

The present invention relates to a system and method that uses depth sensors on the front, back, and bottom of a legged robot for stair climbing. The method uses real-time depth information to assist the legged robot in navigating various leveled terrains. In addition to generating a compound field of view that extends from the front to the back of the legged robot, a sensing method is employed. A downward-facing depth camera positioned at a specific angle allows the system to provide a persistent view of the environment, allowing the legged robot to guide itself through the environment as it navigates. Other tools such as height map interpolation, gradient map calculation, and strategic footing selection are also implemented.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Application No. 63 / 396,319, filed August 9, 2022, all of which are incorporated herein by reference. [Background technology]

[0002] The use of depth information and real-time image information is essential for robot navigation. Legged robots require these sensors to represent the terrain around the robot with sufficient accuracy and density to locate footholds. Furthermore, the time window within which a safe foothold can be determined is typically short, so the representation must be updated without delay as the robot moves rapidly through this environment.

[0003] One method of creating this representation, which has been popular in the past, is to accumulate measurements over time to create a "map" of the terrain around the robot. This method relies on state estimation or odometry to maintain an understanding of the robot's movement relative to the terrain, and then uses that knowledge to create a composite terrain map. These odometry methods may combine visual, inertial, and encoder measurements. The advantage of this common method is that it requires fewer depth sensors, since it assumes that data accumulates over time. However, frequent footsteps in a legged robot appear as noise in the inertial data, and toe-slips can cause significant errors in the integration of encoder, inertial, and visual data. If the state estimation results are not sufficiently accurate for the aforementioned reasons, there is no possibility of recovery, as depth sensors may not immediately view the terrain near the robot's feet. Furthermore, approaches that combine multiple measurements over time often assume that the environment is relatively unchanging and static, which is not necessarily true. Summary of the Invention [Problem to be solved by the invention]

[0004] To avoid this estimation error and ultimately achieve the desired accuracy when operating vision-guided legged locomotion on stairs, a compound field of view extending from directly in front to directly behind the robot must be continuously maintained and updated at a rate appropriate for legged locomotion. The addition of "downward-facing" depth and visual sensors creates a comprehensive compound field of view that covers the area the legged robot may walk. [Means for solving the problem]

[0005] The present invention places multiple depth cameras at various locations on a legged robot, specifically under the front, back, and center of the robot's chassis. Positioning the depth cameras at specific angles provides a composite field of view that extends along the front, center, and back of the legged robot using the depth cameras and depth and visual sensors, producing more reliable results for vision-based legged locomotion. This approach provides a way to create a more accurate and safer walking path for a legged robot on stairs.

[0006] The present invention utilizes multiple depth cameras located at the base of the robot chassis and on the front and back of the legged robot. The cameras provide a comprehensive view of the robot's surroundings and the terrain below it. These cameras capture depth information in the form of a point cloud, which is used to assist the robot in climbing and descending stairs. This point cloud data is processed by removing occlusions from parts of the robot's body and used to create a height map. Each element in the height map holds terrain height information, and a stair model fitting is performed to determine the stair height and run dimensions (distance of the run path, and number of steps). This model interpolates missing areas in the height map, allowing the legged robot to navigate stairs and elevated terrain.

[0007] A gradient map is then calculated on the height map, which is essential for the foothold selection process. Combining these techniques allows a legged robot to climb stairs while leveraging depth information from multiple cameras mounted on the robot's body, improving perception and decision-making during the locomotion process.

[0008] The depth camera positioning of the present invention provides a comprehensive field of view. Legged robots with cameras only on the front and back cannot adequately observe the terrain beneath them and are unable to estimate the terrain height. Furthermore, estimation is difficult due to the need for accurate foot placement despite the presence of measurement noise, and reinitialization is impossible if inaccuracies in the estimate accumulate. To alleviate this estimation problem, the present invention discloses a system design equipped with multiple cameras that cover the entire field of view, including the front, back, and underfoot.

[0009] The present invention uses multiple depth cameras to acquire visual data, using a combination of cameras that can accurately acquire depth images over a wide field of view and generate depth data at a sufficiently high speed. The acquired images are converted into point cloud information about the height of the surrounding environment, including the height below the robot. The cameras are strategically placed on the front of the robot, tilted downward at a 25-degree angle, including but not limited to: a second camera on the rear, also facing downward, but at a 15-degree angle; a second camera is installed on the belly of the robot, tilted 10 degrees relative to the horizon and facing directly downward; and when the robot's height exceeds 330 cm, these cameras effectively cover the entire field of view below the robot. This configuration ensures comprehensive visual coverage and facilitates robust data collection for robotic navigation and perception tasks.

[0010] The present invention then generates a height map from the point cloud. Multiple cameras provide a wide field of view, providing depth information about the areas below, in front of, and behind the robot. However, during climbing and lowering movements, the robot's legs may enter the camera's field of view, disrupting the depth information of the environment. To address this issue, a slicing strategy is implemented to reduce the impact of the legs on the depth point cloud. This height map processing is typically relayed through a computing box located inside the legged robot, equipped with a microprocessor and an inertial memory unit.

[0011] The kinematics of the robot's legs are used to determine the width of the point cloud slice. The toe positions determined by the kinematics are used to establish the y-extent of the point cloud slice, forming a height map. Specifically, the minimum y-position of the left toes and the maximum y-position of the right toes are used to define this extent.

[0012] Obtaining point cloud slices without toe occlusion produces height map information. A height map consists of several elements that store the height of the terrain. Height maps accumulate spatially coherent point cloud data into a more concise, spatially ordered structure, facilitating operations such as gradients and reducing computation time for algorithms that rely on them.

[0013] The present invention also adjusts staircase model fitting. When a legged robot traverses stairs, the distance between the robot itself and the stairs may be less than 330 cm, which may result in incomplete camera footage. To address this issue, a staircase fitting algorithm is employed. The staircase fitting algorithm is executed by a processor installed in a computing box attached to the chassis of the legged robot and operates over a wireless network.

[0014] The algorithm begins by assuming that the steps of a staircase are uniform and models the staircase using two parameters: height and run. It calculates the fitting error for each combination of height and run, then changes these parameters incrementally by 1 cm. This process continues until all fitting errors have been calculated for all possible combinations of height and run. The algorithm then selects the best result with the smallest fitting error.

[0015] The foothold selection process utilizes a multi-objective optimization search (Equation 1). The first two terms are the cost of deviating from the nominal foothold location (Jnom) and the gradient at the current position (Jgrad). Jnom is proportional to the distance between the current position and the nominal foothold location. The nominal foothold is the toe position based on the robot's dynamics. Jgrad is calculated based on the gradient map. This combination allows us to consider both the proximity to the desired foothold location and the slope of the terrain.

[0016] To improve stability and prevent excessive scaffold position movement even in the presence of noisy height maps, we introduce a damping term, Jdamp. The damping term penalizes discrepancies between the current scaffold position and previous positions. This results in a more robust scaffold selection process, allowing for smoother and more controlled scaffold placement even in difficult, uncertain, and / or unstructured terrain conditions. The objective function is Equation 1.

[0017]

number

[0018] Our staircase model fitting method theoretically allows the robot to walk anywhere. However, using a gradient map, the robot will preferentially walk on flat areas over uneven areas. The staircase model generates an optimal height and stair path (run, run path, number of steps), and missing areas in the height map are interpolated by the employed algorithm. This improves terrain perception when navigating stairs, allowing the robot to make more strategic and informed decisions when navigating stairs.

[0019] The gradient map calculation reveals how suitable a location in the map is for a legged robot to place its feet. This method uses a convolution operation rather than a 3D signed distance field calculated from a terrain map, a feature designed to aid in sensor data processing and anomaly detection. Taking all features into consideration, the result is an advanced method for efficient and accurate stair-climbing maneuvers using depth sensors placed on a legged robot.

[0020] Other features and aspects of the present invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, which illustrate, by way of example, features in accordance with embodiments of the invention. This summary is not intended to limit the scope of the invention, which is defined solely by the claims appended hereto. [Brief explanation of the drawings]

[0021] [Figure 1] Figure 1 shows a comparison of the current "state of the art" for legged robots on the left (not facing down), where the robot uses front and rear depth sensors. [Figure 2] Figure 2 shows the robot walking up stairs. [Figure 3] Figure 3 shows the design of the depth camera position. [Figure 4] Figure 4 shows the position of the depth camera. [Figure 5] Figure 5 is a slice of the point cloud used to form a height map based on the toe position. [Figure 6] Figure 6 shows the process of generating a height map from a point cloud. [Figure 7] Figure 7 shows the height map and gradient map of the stairs.

[0022]

[0013] Having generally described the invention, reference will now be made to the accompanying drawings, which are not necessarily drawn to scale, in which various embodiments are illustrated by way of example, and not by way of limitation, and in which: DETAILED DESCRIPTION OF THE INVENTION

[0023] Figure 1 shows a comparison of the current "state of the art" for legged robots (without downward-facing vision and depth sensors) on the left. In this case, the robot (a) uses front and rear cameras and is able to create a model of the environment from its fields of view (FOVs) (b). On the right, instead, the addition of downward-facing depth sensors and their FOVs (c) provides a persistent and complete view of the environment being navigated, without the need for estimation techniques. Such an approach is more robust to system and sensor noise.

[0024] Figure 2 depicts a legged robot climbing stairs, utilizing the manipulation disclosed above and cameras located on the front, back, and center chassis of the legged robot.

[0025] Figure 3 shows the process of leveraging sensory information to enable stair climbing and descending. The depth camera uses the robot computer to perform a series of operations. Point cloud data helps design a height map used for stair model fitting. Next, a gradient map calculation using a 1D convolution operation is created to help describe the suitability of locations in the map for placing the legs of the legged robot. Finally, the foothold selection process involves a multi-objective optimization search using Equation (1) below.

number

[0026] Figure 4 shows the design of the depth camera positions. One camera on the front is set at a 25-degree angle downward, the other on the back is set at a 15-degree angle downward, and the other two cameras are set on the abdomen of the robot, facing downward at a 10-degree angle to the horizon.

[0027] Figure 5 shows a slice of the point cloud used to form a height map based on the toe position. To address the issue of the toe and bottom links causing confusion as a result of the environment's depth information, a slicing strategy is implemented that reduces the influence of the legs on the depth point cloud.

[0028] Figure 6 illustrates the process of creating a height map from a point cloud. A height map is essentially a 1D vector, with each element storing the height of the terrain. All points at the same Y position in the point cloud are grouped together. The corresponding element of the height map is set to the average value of all points in the group.

[0029] Figure 7 shows the height map (dashed and solid lines) and the gradient map (dotted line). Missing areas in the height map are interpolated using a fitting model that assumes the stair steps are uniform, models the staircase using parameters for height and stair path (run, run, and number of steps), and calculates the fitting error for each combination of height and stair path (run, run, and number of steps). This allows the algorithm to enhance terrain perception when navigating stairs, allowing the robot to make more informed decisions and navigate stairs more effectively and accurately. Once the height map information is obtained, a gradient map is calculated using a one-dimensional convolution operation. This gradient map indicates the suitability of foot placement within the map.

[0030] While various embodiments of the disclosed technology have been described above, it should be understood that they are presented by way of example only, and not limitation. Similarly, various figures may depict example architectural or other configurations of the disclosed technology, this is done to aid in understanding the features and functionality that may be included in the disclosed technology. The disclosed technology is not limited to the example architectures or configurations shown, and the desired functions may be implemented using a variety of alternative architectures and configurations. Indeed, it will be apparent to those skilled in the art how alternative functional, logical, or physical divisions and configurations can be implemented to implement the desired features of the technology disclosed herein. Also, various component module names may be applied to various divisions other than those shown herein. Furthermore, with respect to flow diagrams, operational descriptions, and method claims, the order of steps presented herein does not require various embodiments to perform the described functions in the same order, unless the context dictates otherwise.

[0031] While the disclosed technology has been described above in terms of various exemplary embodiments and implementations, it should be understood that various features, aspects, and functions described in one or more of the individual embodiments are not limited in application to the particular embodiment described, but may also be applicable to one or more other embodiments of the disclosed technology, either alone or in various combinations, regardless of whether such embodiment is described and whether such features are presented as part of the described embodiment. Thus, the breadth and scope of the technology disclosed herein should not be limited by any of the exemplary embodiments described above.

[0032] Terms and phrases used in this document, and variations thereof, unless expressly stated otherwise, should be construed as open-ended and not limiting. For the foregoing examples, the term "including" should be construed as meaning "including, without limitation," and the term "examples" is used to provide illustrative instances of the items under discussion, not as an exhaustive or limiting list thereof. The terms "a" or "an" should be construed as meaning "at least one," "one or more," and the like. Adjectives such as "conventional," "traditional," "usually," "standard," "known," and similar terms should not be construed as limiting the items being described to items available during a particular period or at a particular time, but should be construed as embracing conventional, traditional, usual, or standard technology that may be available or known at any time now or in the future. Similarly, when this document refers to technology that would be apparent or known to one of ordinary skill in the art, such technology includes technology that would be apparent or known to one of ordinary skill in the art at any time now or in the future.

Claims

1. A system using a depth sensor at the bottom of a legged robot for climbing stairs, comprising: Multiple depth cameras located on the front, back, and bottom of the central chassis of the legged robot to provide a comprehensive field of view; a processor located in the computing box for storing depth data obtained from the depth camera; a point cloud generated from the depth data to utilize data related to stair climbing and descending by the legged robot; a height map created by the point cloud and the depth data, the height map including terrain height information for performing staircase model fitting to estimate the stair height and stair run dimensions; a gradient map calculated using a one-dimensional convolution operation and the height map to utilize depth data from the plurality of depth cameras to enhance perception and decision making and assist in the foothold selection process and to determine suitable locations for the legged robot to place its feet; a foothold selection using a multi-objective optimization search formula to determine a distance between a current position of the legged robot and a nominal foothold position based on the dynamics of the legged robot, the foothold selection increasing the stability of the legged robot in the process of the foothold selection; Including, the system.

2. The system of claim 1 , wherein at least one of the depth cameras located on the front of the legged robot is tilted downward at approximately 25 degrees.

3. The system of claim 1 , wherein at least one of the depth cameras located on the back of the legged robot is tilted downward by 15 degrees.

4. The system of claim 1 , wherein one or more of the depth cameras is centrally located on a central chassis of the legged robot and oriented at a 10 degree angle.

5. The system of claim 1 , wherein the gradient map causes the legged robot to prefer walking on flat terrain over uneven terrain.

6. The multi-objective optimization search formula is the following formula (1): [Equation 1] The system of claim 1 , wherein:

7. The system of claim 6 , wherein the scaffold selection and the multi-objective optimization search formula utilize the point cloud and depth data in their calculations.

8. 1. A method of using a depth sensor at the bottom of a stair-climbing legged robot, comprising: adding downward-facing depth and vision sensors to the front, back, and bottom of a central chassis of the legged robot to provide a complete compound field of view; positioning the downward-facing depth and vision sensors at angles that provide an unobstructed view according to their placement on the legged robot; processing depth and vision sensor data about the legged robot's environment to generate a point cloud; generating a height map using the depth and vision sensor data; performing staircase model fitting to estimate the stair height and stair run dimensions and to interpolate missing areas of the height map; calculating a gradient map based on said height map to aid in the scaffold selection process; providing a continuous view of the environment in which the vehicle is moving; Methods including:

9. The method of claim 8 , wherein the depth and vision sensor data is terrain height information.

10. The method of claim 8 , wherein the staircase model fitting is one-dimensional (1D).

11. The method of claim 8 , wherein the data about the legged robot utilizes depth information from the depth and vision sensors to enhance perception.

12. The method of claim 8 , wherein at least one of the vision and depth sensors is located on the front of the legged robot and is tilted downward at approximately 25 degrees.

13. The method of claim 8 , wherein at least one of the vision and depth sensors is located on the back of the legged robot and tilted downward at 15 degrees.

14. The method of claim 8 , wherein at least one of the vision and depth sensors is located in the center of the chassis of the legged robot and oriented at a 10 degree angle.

15. 1. A method of using a depth sensor at the bottom of a stair-climbing legged robot, comprising: disposing a plurality of depth and vision sensors on the front and rear of a central chassis of the legged robot to provide a comprehensive view of the environment of the legged robot; using a microprocessor located within a computing limit of the legged robot to process and store depth data obtained from the depth and vision sensors, wherein the computing unit generates a point cloud according to the depth data; utilizing data relating to the legged robot's stair climbing and descending; creating a height map from the point cloud and depth data; evaluating the terrain height information, performing 1D stair model fitting, estimating the stair height and stair run dimensions, and calculating the fitting error for each combination of said height and stair run by varying its parameters; interpolating missing areas in the height map using a desired staircase mode that results in an optimal height and path to complete the field of view acquired by said depth and vision sensors; calculating a gradient map calculated using a one-dimensional convolution operation, said height map utilizing depth data from said depth and vision sensors to enhance perception and decision making, assisting in the foothold selection process and determining suitable locations for said legged robot to place its feet; Executing a multi-objective optimization search formula to determine the distance between the current position of the legged robot and a nominal foothold position based on the dynamics of the legged robot, thereby increasing the stability of the legged robot in foothold selection; A method comprising:

16. The method of claim 15 , wherein the gradient map causes the legged robot to prefer walking on flat terrain over uneven terrain.

17. 16. The method of claim 15, wherein at least one of the vision and depth sensors is located on the front of the legged robot and tilted downward at approximately 25 degrees.

18. 16. The method of claim 15, wherein at least one of the vision and depth sensors is located on the back of the legged robot and tilted downward at 15 degrees.

19. 16. The method of claim 15, wherein at least one of the vision and depth sensors is located in the center of the chassis of the legged robot and oriented at a 10 degree angle.

20. 16. The method of claim 15, wherein the plurality of vision and depth sensors provide and capture a wide field of view at at least 90 frames per second, generate captures, and convert the captures into a point cloud.

Images