Robotic step timing and sequencing using reinforcement learning

Abstract

Techniques are provided for determining robot step timing and sequencing using reinforcement learning. In one aspect, a method includes receiving a target trajectory for a robot and receiving a state of the robot. The method also includes generating a set of gait timing parameters for the robot based at least in part on the state of the robot and the target trajectory using a neural network, and controlling motion of the robot based on the set of gait timing parameters.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 596,024, filed November 3, 2023, the disclosure of which is incorporated herein by reference in its entirety. Any and all applications with foreign or domestic priority claims identified in the application data sheet filed with this application, pursuant to 37 CFR 1.57, are incorporated herein by reference. Technical Field

[0003] This disclosure relates to step timing and sequencing of robots using reinforcement learning. Background Technology

[0004] A legged robot's gait can be viewed as a cyclical pattern of leg movement, generating locomotion through a series of contacts between the feet and surfaces. The legs provide support for the robot's body while simultaneously propelling it by forces generated from these surface contacts. Gaits can vary in many ways, and different gaits can produce different styles of locomotion. Choosing the appropriate gait (including when the legged robot is in a particular state and / or environment) can be challenging, as some gaits may cause the robot to become unstable or exhibit undesirable movements. Summary of the Invention

[0005] In one aspect, a method is provided, comprising: receiving a target trajectory of a legged robot by a control system of the legged robot; receiving a state of the legged robot by the control system; generating a set of gait timing parameters of the legged robot using a neural network of the control system, based at least in part on the state of the legged robot and the target trajectory; and controlling the movement of the legged robot by the control system based on the set of gait timing parameters.

[0006] In some embodiments, reinforcement learning is used to train the neural network.

[0007] In some embodiments, the state of the legged robot includes at least one or more joint angle measurements of the legged robot, one or more pose estimates of the legged robot, and / or a terrain model of the legged robot.

[0008] In some embodiments, gait timing parameters include a contact sequence.

[0009] In some embodiments, the contact sequence includes at least one target step time.

[0010] In some embodiments, the gait timing parameters further include a velocity scaling factor.

[0011] In some embodiments, the method further includes: using a model predictive controller (MPC) of the control system to generate a set of stepping parameters based on gait timing parameters, wherein controlling the movement of the legged robot is also based on the set of stepping parameters.

[0012] In some embodiments, the set of step parameters includes at least one of step placement or desired center of mass acceleration.

[0013] In some embodiments, the method further includes: initializing a neural network to reproduce the set of step parameters within a threshold difference of a previous set of step parameters; and training the initialized neural network using reinforcement learning, which includes simulating MPC to search the space of possible solutions.

[0014] In some embodiments, the method further includes: receiving perception data indicating the environment of the legged robot; and generating a map of the environment based on the perception data, wherein the neural network also uses the map of the environment as input.

[0015] In some embodiments, the method further includes: rescaling the trajectory such that the movement of the legged robot does not exceed a speed limit while following the trajectory.

[0016] In some embodiments, the method further includes: generating a set of stepping parameters based on gait timing parameters and a rescaled trajectory, wherein controlling the movement of the legged robot is also based on the set of stepping parameters.

[0017] In some embodiments, the neural network also uses a set of input parameters as input, including one or more of the following: terrain height map, no-stepping map, control state, user-specified desired robot behavior, subject path, or perception data.

[0018] In some embodiments, the method further includes: receiving obstacle data; and generating a main path based on the trajectory and obstacle data, wherein the neural network also uses the main path as input.

[0019] On the other hand, a legged robot is provided, comprising: a body; two or more legs coupled to the body; one or more sensors configured to measure the state of the legged robot; and a control system communicating with the body and the two or more legs. The control system includes data processing hardware and memory hardware communicating with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations including: receiving a target trajectory of the legged robot; receiving the state of the legged robot from the one or more sensors; generating a set of gait timing parameters of the legged robot using a neural network of the control system, at least in part based on the state of the legged robot and the target trajectory; and controlling the movement of the legged robot based on the set of gait timing parameters.

[0020] In some embodiments, reinforcement learning is used to train the neural network.

[0021] In some embodiments, the state of the legged robot includes at least one or more joint angle measurements of the legged robot, one or more pose estimates of the legged robot, and / or a terrain model of the legged robot.

[0022] In some embodiments, gait timing parameters include a contact sequence.

[0023] In some embodiments, the contact sequence includes at least one target step time.

[0024] In some embodiments, the gait timing parameters further include a velocity scaling factor.

[0025] In some embodiments, when executed on data processing hardware, the instructions also cause the data processing hardware to: use a model predictive controller (MPC) of the control system to generate a set of stepping parameters based on gait timing parameters, wherein the motion of the legged robot is also controlled based on this set of stepping parameters.

[0026] In some embodiments, the set of step parameters includes at least one of step placement or desired center of mass acceleration.

[0027] In some embodiments, when the instructions are executed on the data processing hardware, the data processing hardware is further instructed to: initialize a neural network to reproduce a set of step parameters within a threshold difference of a previous set of step parameters; and train the initialized neural network using reinforcement learning, which includes simulating MPC to search the space of possible solutions.

[0028] In some embodiments, when executed on data processing hardware, the instructions also cause the data processing hardware to: receive perception data indicative of the environment of the legged robot; and generate a map of the environment based on the perception data, wherein the neural network further uses the map of the environment as input.

[0029] In some embodiments, the instructions, when executed on the data processing hardware, further cause the data processing hardware to: rescale the trajectory such that the legged robot's motion does not exceed a speed limit while following the trajectory.

[0030] In some embodiments, when the instructions are executed on the data processing hardware, the data processing hardware further causes the data processing hardware to generate a set of stepping parameters based on gait timing parameters and a rescaled trajectory, wherein the control of the legged robot's motion is further based on the set of stepping parameters.

[0031] In some embodiments, the neural network also uses a set of input parameters as input, including one or more of the following: terrain height map, no-stepping map, control state, user-specified desired robot behavior, subject path, or perception data.

[0032] In some embodiments, when executed on data processing hardware, the instructions also cause the data processing hardware to: receive obstacle data; and generate a main path based on the trajectory and obstacle data, wherein the neural network also uses the main path as input.

[0033] In another aspect, a non-transitory computer-readable medium is provided, wherein instructions are stored, which, when executed by data processing hardware of a control system, cause the data processing hardware to: receive a target trajectory of a legged robot; receive a state of the legged robot; generate a set of gait timing parameters of the legged robot using a neural network of the control system, based at least in part on the state of the legged robot and the target trajectory; and control the movement of the legged robot based on the set of gait timing parameters.

[0034] In some embodiments, reinforcement learning is used to train the neural network.

[0035] In some embodiments, the state of the legged robot includes at least one or more joint angle measurements of the legged robot, one or more pose estimates of the legged robot, and / or a terrain model of the legged robot.

[0036] In some embodiments, gait timing parameters include a contact sequence.

[0037] In some embodiments, the contact sequence includes at least one target step time.

[0038] In some embodiments, the gait timing parameters further include a velocity scaling factor.

[0039] In some embodiments, when the instructions are executed by the data processing hardware, the data processing hardware also causes the data processing hardware to generate a set of stepping parameters based on gait timing parameters using a model predictive controller (MPC) of the control system, wherein the motion of the legged robot is also controlled based on this set of stepping parameters.

[0040] In some embodiments, the set of step parameters includes at least one of step placement or desired center of mass acceleration.

[0041] In some embodiments, when the instructions are executed on the data processing hardware, the data processing hardware is further instructed to: initialize a neural network to reproduce a set of step parameters within a threshold difference of a previous set of step parameters; and train the initialized neural network using reinforcement learning, which includes simulating MPC to search the space of possible solutions.

[0042] In some embodiments, when executed on data processing hardware, the instructions also cause the data processing hardware to: receive perception data indicative of the environment of the legged robot; and generate a map of the environment based on the perception data, wherein the neural network further uses the map of the environment as input.

[0043] In some embodiments, the instructions, when executed on the data processing hardware, further cause the data processing hardware to: rescale the trajectory such that the legged robot's motion does not exceed a speed limit while following the trajectory.

[0044] In some embodiments, when the instructions are executed on the data processing hardware, the data processing hardware further causes the data processing hardware to generate a set of stepping parameters based on gait timing parameters and a rescaled trajectory, wherein the control of the legged robot's motion is further based on the set of stepping parameters.

[0045] In some embodiments, the neural network also uses a set of input parameters as input, including one or more of the following: terrain height map, no-stepping map, control state, user-specified desired robot behavior, subject path, or perception data.

[0046] In some embodiments, when executed on data processing hardware, the instructions also cause the data processing hardware to: receive obstacle data; and generate a main path based on the trajectory and obstacle data, wherein the neural network also uses the main path as input. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of an example robot used for navigation on a site.

[0048] Figure 2 It is used for navigation Figure 1 A schematic diagram of the robot's navigation system.

[0049] Figure 3 This is a block diagram illustrating an example of a gait control system used to generate step parameters.

[0050] Figure 4 This is a block diagram illustrating an embodiment of a gait control system that uses reinforcement learning to generate step parameters.

[0051] Figure 5 An example visualization of perception data generated by one or more sensors of a robot according to aspects of this disclosure is shown.

[0052] Figure 6 An example visualization of gait timing parameters generated by a learning parameter selector according to aspects of this disclosure is shown.

[0053] Figure 7 A method for training a neural network according to aspects of this disclosure has been explained.

[0054] Figure 8 A method for controlling the motion of a robot using a trained neural network, according to aspects of this disclosure, is shown.

[0055] Figure 9 This is a schematic diagram of an example computing device that can be used to implement the systems and methods described in this paper. Detailed Implementation

[0056] One aspect of ensuring a robot can traverse diverse terrains without becoming unstable or exhibiting undesirable movement is determining the stepping parameters associated with the placement of the robot's feet on the terrain. According to embodiments, stepping parameters may include the contact sequence, timing, and position of the robot's feet. Determining a set of stepping parameters that allows the robot to traverse varied terrain without becoming unstable or causing other undesirable movements can become increasingly complex as the robot traverses it. For example, when traversing multi-terrain conditions, it may be difficult to identify the stepping parameters that enable the robot to safely traverse the terrain. Furthermore, anticipating all combinations of terrain that the robot might traverse may be impractical, and therefore, designing an algorithm that determines robust stepping parameters under all possible conditions can be particularly difficult. As described herein, aspects of this disclosure relate to using a learned parameter selector trained using reinforcement learning as part of a gait control system to improve performance under certain conditions involving multi-factor terrain compared to other embodiments.

[0057] Example robot system

[0058] refer to Figure 1 and Figure 2 In some embodiments, robot 100 includes a body 110 having one or more motor-based structures, such as legs 120a-d coupled to the body 110, which enable robot 100 to move within a field 30 surrounding robot 100. In some examples, all or part of the legs 120 are articulated structures, such that one or more joints J allow movement of components 122 of the legs 120. For example, in the illustrated embodiment, all or part of the legs 120 includes upper components 122, 122... U Hip joint J coupled to body 110 H and the upper component 122 of leg 120 U The lower component 122 is coupled to the leg 120. L Knee joint J K .although Figure 1A quadruped robot with four legs 120a-d is depicted, but robot 100 may include any number of legs or a motor-based structure (e.g., a bipedal or humanoid robot with two legs, or other arrangements of one or more legs) that provides a means of traversing the terrain within site 30.

[0059] To traverse terrain, all or part of the leg 120 may have a distal end 124 that contacts a surface of the terrain (e.g., a traction surface). In other words, the distal end 124 of the leg 120 is the end of the leg 120 used by the robot 100 for pivoting, planting, or generally providing traction during the movement of the robot 100. For example, the distal end 124 of the leg 120 corresponds to the foot of the robot 100. In some examples, although not shown, the distal end of the leg includes an ankle joint, such that the distal end 124 is positioned relative to the lower member 122 of the leg. L It is hinged.

[0060] In the example shown, robot 100 includes an arm 126 used as a robot manipulator. Arm 126 can move about multiple degrees of freedom to engage elements of field 30 (e.g., objects within field 30). In some examples, arm 126 includes one or more members 128, wherein members 128 are coupled via joint J, such that arm 126 can pivot or rotate about joint J. For example, in the case of more than one member 128, arm 126 can extend or retract. To illustrate the example, Figure 1 It depicts a structure with corresponding lower member 128 L 128 upper components U Hand-made components 128 H The arm 126 of the three components 128 (also known as the end-effector). Here, the lower component 128 L It can surround the first arm joint J located near the main body 110 A1 (For example, the position where arm 126 is connected to the body 110 of robot 100) rotates or pivots. Lower component 128 L Second arm joint J A2 The coupling is connected to the upper component 128 U And the upper component 128 U At the third arm joint J A3 Component 128 H In some examples, such as Figure 1 128 hand-made components H It is a mechanical gripper comprising movable and fixed jaws, capable of performing different types of gripping on components within the area 30. In the example shown, hand component 128 HThe gripper includes a fixed first jaw and a movable second jaw that grips an object by clamping it between the jaws. The movable jaw can move relative to the fixed jaw to move between an open position and a closed position (e.g., closed around the object). In some embodiments, arm 126 further includes a fourth joint J. A4 Fourth joint J A4 It can be located in the lower component 128 L With upper component 128 U Near the coupling point, and used to allow the upper component 128 U Compared to the lower component 128 L Twist or rotate. In other words, the fourth joint J A4 It can be used as a torsion joint, similar to hand component 128. H The third joint J of the adjacent arm 126 A3 Or a wrist joint. For example, as a torsion joint, one member coupled at joint J can move or rotate relative to another member coupled at joint J (e.g., the first member coupled at the torsion joint is fixed, while the second member coupled at the torsion joint rotates). In some embodiments, arm 126 is connected to robot 100 at a socket on the body 110 of robot 100. In some configurations, the socket is configured as a connector, such that arm 126 is attached to or detached from robot 100 depending on whether arm 126 is intended for a particular operation.

[0061] Robot 100 has a vertical gravity axis along the direction of gravity (e.g., shown as axis A in the Z direction). Z The robot 100 also has a center of mass CM, which is the position corresponding to the average position of all parts of the robot 100, where the parts are weighted according to their mass (e.g., the point where the weighted relative positions of the distributed mass of the robot 100 sum to zero). The robot 100 also has a center of mass CM based on the vertical gravity axis A. Z The pose P of the CM (e.g., a fixed reference frame relative to gravity) defines a specific posture or stance adopted by the robot 100. The posture of the robot 100 can be defined by its orientation or angular position in space. Movement of the leg 120 relative to the body 110 alters the posture P of the robot 100 (e.g., a combination of the position of the robot's CM and the posture or orientation of the robot 100). Here, height typically refers to the height along the z-direction (e.g., along the z-axis A). Z The distance is ( ). The sagittal plane of robot 100 corresponds to the distance along the Y-axis A. Y and Z-axis A ZThe YZ plane extends in the direction of the sagittal plane. In other words, the sagittal plane divides the robot 100 into left and right sides. Typically perpendicular to the sagittal plane, the ground plane (also called the lateral plane) passes through the x-axis A. X and y-axis A Y The ground plane extends across the XY plane in the direction of the robot 100. The ground plane refers to the surface 14 of the ground where the distal ends 124 of the robot 100's legs 120 can generate traction to aid the robot 100's movement within the field 30. Another anatomical plane of the robot 100 is the frontal plane extending across the body 110 of the robot 100 (e.g., from the right side of the robot 100 with the first leg 120a to the left side of the robot 100 with the second leg 120b). The frontal plane extends through the X-axis A. X and Z-axis A Z It extends in the direction of and crosses the XZ plane.

[0062] In order to manipulate or use the arm 126 to perform tasks within the site 30, the robot 100 includes a sensor system 130 having one or more sensors 132, 132a–n. For example, Figure 1 The first sensors 132 and 132a are shown installed in the head of robot 100 (near the front part of robot 100 adjacent to the front legs 120A-B), and the sensor J is installed in the buttock of the second leg 120b of robot 100. Hb The second sensors 132 and 132b are located nearby; the third sensors 132 and 132c are mounted on one side of the main body 110 of the robot 100; and the sensor J is mounted on the buttock of the fourth leg 120d of the robot 100. Hd The fourth sensors 132 and 132d are located nearby. In some cases, the sensor system may include a hand component 128 mounted on the arm 126 of the robot 100. H A fifth sensor located at or near the location. Sensor 132 may include a vision / image sensor, an inertial sensor (e.g., an inertial measurement unit (IMU)), a force sensor, and / or a motion sensor. For example, sensor 132 may include one or more of a camera (e.g., a stereo camera), a time-of-flight (TOF) sensor, a scanning light detection and ranging (lidar) sensor, or a scanning laser detection and ranging (ladar) sensor. In some examples, sensor 132 has a corresponding field of view F that defines a sensing range or area corresponding to sensor 132. V .For example, Figure 1 The field of view F of the first sensors 132 and 132a of robot 100 is depicted. V Each sensor 132 may be pivotable and / or rotatable, such that the sensor 132 may change its field of view F, for example, about one or more axes (e.g., the x-axis, y-axis, or z-axis relative to the ground plane). VIn some examples, multiple sensors 132 can be clustered together (e.g., similar to the first sensor 132a) to stitch together a larger field of view F than any single sensor 132. V With multiple sensors 132 positioned around the robot 100, the sensor system can have a 360-degree view or near-360-degree view of the robot 100’s surroundings around a vertical and / or horizontal axis.

[0063] When using sensor 132 to survey the field of view F V At that time, the sensor system generates a value corresponding to the field of view F. V Sensor data 134 (e.g., image data) (see example) Figure 2 The sensor system can generate a field of view F using sensors 132 (e.g., sensors 132a, 132c) mounted on or near the body 110 of the robot 100. V The sensor system may additionally and / or alternatively utilize the hand component 128 mounted on the arm 126. H Sensor 132 at or near the location generates a field of view F V One or more sensors 132 capture sensor data 134 that defines a three-dimensional point cloud within the field 30 for robot 100. In some examples, sensor data 134 is image data corresponding to the three-dimensional volumetric point cloud generated by the three-dimensional volumetric image sensor 132. Additionally or alternatively, when robot 100 manipulates within field 30, the sensor system collects pose data of robot 100, which includes inertial measurement data (e.g., measured by an IMU). In some examples, pose data includes kinematic and / or orientation data about robot 100, such as kinematic and / or orientation data about joints J or other parts of robot 100, such as leg 120 or arm 126. Using sensor data 134, various systems of robot 100 can use sensor data 134 to define the current state of robot 100 (e.g., kinematic) and / or the current state of field 30 of robot 100. In other words, the sensor system can transmit sensor data 134 from one or more sensors 132 to any other system of robot 100 to assist the functionality of that system.

[0064] In some embodiments, the sensor system includes sensors 132 coupled to joint J. Furthermore, these sensors 132 may be coupled to motors M (e.g., sensors 132) of joint J of the robot 100. Here, these sensors 132 generate joint dynamics in the form of joint-based sensor data 134. The joint dynamics collected as joint-based sensor data 134 may include joint angles (e.g., upper member 122). U Relative to the lower component 122L Joint-based sensor data generated by one or more sensors 132 can be raw sensor data, data further processed to form different types of joint dynamics, or some combination of both. For example, sensor 132 measures joint position (or the position of components 122 or 128 coupled at joint J), and the robot 100's system performs further processing to derive velocity and / or acceleration from the position data. In other examples, sensor 132 may directly measure velocity and / or acceleration.

[0065] Reference Figure 2 When sensor system 130 collects sensor data 134, computing system 140 stores, processes, and / or transmits the sensor data 134 to various systems of robot 100 (e.g., control system 170, navigation system 101, topology component 103, and / or remote controller 10). To perform computational tasks related to sensor data 134, computing system 140 of robot 100 includes data processing hardware 142 and memory hardware 144. Data processing hardware 142 can execute instructions stored in memory hardware 144 to perform computational tasks related to the activities of robot 100 (e.g., motion and / or motion-based activities). Generally, computing system 140 refers to one or more locations of data processing hardware 142 and / or memory hardware 144.

[0066] In some examples, computing system 140 is a local system located on robot 100. When located on robot 100, computing system 140 can be centralized (e.g., at a single location / area on robot 100, such as the body 110 of robot 100), distributed (e.g., located at various locations around robot 100), or a hybrid combination of both (e.g., comprising most centralized hardware and a few distributed hardware). To illustrate some differences, distributed computing system 140 may allow processing to occur at the active location (e.g., at the motors of the joints of the moving leg 120), while centralized computing system 140 may allow a central processing center to communicate with systems located at various locations on robot 100 (e.g., with the motors of the joints of the moving leg 120).

[0067] Additionally or alternatively, computing system 140 includes computing resources located remotely from robot 100. For example, computing system 140 communicates with remote system 160 (e.g., a remote server or cloud-based environment) via network 180. Much like computing system 140, remote system 160 includes remote computing resources such as remote data processing hardware 162 and remote memory hardware 164. Here, sensor data 134 or other processed data (e.g., data processed locally by computing system 140) may be stored in remote system 160 and accessible by computing system 140. In another example, computing system 140 may utilize remote resources 162, 164 as an extension of computing resources 142, 144, such that resources of computing system 140 reside on resources of remote system 160. In some examples, topology component 103 executes on robot-local data processing hardware 142, while in other examples, topology component 103 executes on data processing hardware 162 remotely from robot 100.

[0068] In some embodiments, such as Figure 1 and Figure 2 As shown, robot 100 includes a control system 170. The control system 170 can communicate with systems of robot 100, such as at least one sensor system 130, navigation system 101, and / or topology component 103. For example, navigation system 101 can provide stepping plans 105 to control system 170. Control system 170 can use hardware such as computing system 140 to perform operations and other functions. Control system 170 includes at least one controller 172 that can control robot 100. For example, controller 172 controls the movement of robot 100 to traverse field 30 based on input or feedback from systems of robot 100 (e.g., sensor system 130 and / or control system 170). In another example, controller 172 controls the movement between postures and / or behaviors of robot 100. At least one controller 172 can be responsible for controlling the movement of arm 126 of robot 100 so that arm 126 uses hand component 128. H Perform various tasks. For example, at least one controller 172 controls the hand component 128. H (For example, a gripper) to manipulate an object or element in the field 30. For example, controller 172 actuates the movable jaws in a direction toward the fixed jaws to close the gripper. In other examples, controller 172 actuates the movable jaws in a direction away from the fixed jaws to close the gripper.

[0069] A given controller 172 of the control system 170 can control the robot 100 by controlling the movement of one or more joints J around the robot 100. In some configurations, the given controller 172 is software or firmware with programmable logic that controls at least one joint J or operates or couples a motor M to a joint J. A software application (software resource) can refer to computer software that instructs a computing device to perform a task. In some examples, a software application may be referred to as an "application," "app," or "program." For example, controller 172 controls the amount of force applied to joint J (e.g., torque at joint J). As a programmable controller 172, the number of joints J controlled by controller 172 is scalable and / or customizable for a specific control purpose. Controller 172 can control the actuation of a single joint J of the robot 100 (e.g., controlling the torque at a single joint J), multiple joints J, or one or more components 128 (e.g., hand component 128). H (Actuation). By controlling one or more joints J, actuators, or motors M, controller 172 can coordinate the movement of all different parts of robot 100 (e.g., body 110, one or more legs 120, arm 126). For example, to perform an action with some motion, controller 172 can control the movement of multiple parts of robot 100, such as two legs 120a-b, four legs 120a-d, or two legs 120a-b combined with arm 126. In some examples, controller 172 is configured as an object-based controller, set to perform specific actions or sets of actions for interacting with interactive objects.

[0070] Continue to refer to Figure 2 Operator 12 (also referred to herein as user or client) can interact with robot 100 via remote controller 10, which communicates with robot 100 to perform actions. For example, operator 12 transmits command 174 to robot 100 via wireless communication network 16 (executed via control system 170). Additionally, robot 100 can communicate with remote controller 10 to display images on user interface 190 (UI 190) of remote controller 10. For example, UI 190 can display a three-dimensional field of view F relative to one or more sensors 132. V The corresponding image. The image displayed on the UI 190 of the remote controller 10 is a two-dimensional image, which corresponds to the three-dimensional point cloud of sensor data 134 of the area within the field 30 of the robot 100 (e.g., field of view F). V In other words, the image displayed on UI 190 can be a three-dimensional field of view F with one or more sensors 132. V The corresponding two-dimensional image representation.

[0071] Techniques for determining stepping timing and sequencing of legged robots

[0072] One technique for determining stepping parameters (including stepping timing and sequencing) of a legged robot involves using a gait controller that includes a model predictive controller (MPC). The MPC can be configured to generate precise specifications of contact sequences, timing, and foot positions, which can be used to control the placement of the robot's feet. For example, the MPC can receive contact sequences and stepping timing as inputs and generate outputs that include precise foot positions and subject control objectives (e.g., acceleration or force).

[0073] Since the stepping parameters that provide stable motion for the robot without exhibiting undesirable motion can vary greatly depending on the specific terrain the robot is traversing, the control system can include multiple different gait controllers, each designed to receive a unique set of stepping parameter inputs (e.g., contact sequence and stepping timing) for a specific terrain type.

[0074] In one example embodiment, each of a plurality of MPC-based gait controllers can be specifically designed to generate step parameters for a corresponding type of terrain. Heuristics can be used to configure each gait controller (e.g., by programming / designing the controller and MPC) to generate step parameters for the appropriate type of terrain. Example terrain types that the gait controllers can be configured to handle include: crossing obstacles, smooth floors, flat surfaces, uneven surfaces, stairs, narrow corridors, etc.

[0075] To select an appropriate gait controller, the robot may include a controller selection system that evaluates the maximum possible cost (MPC) over a discrete set of possible contact sequences and assigns a cost to each possibility. Additionally, the controller selection system may evaluate a cost function to select a specific gait controller for the robot to use for a given terrain.

[0076] While this heuristic-based system may perform well in single-terrain scenarios (e.g., crossing obstacles or walking on slippery floors), it may perform poorly in scenarios involving multiple terrain factors.

[0077] This document discloses apparatus and methods for timing and sequencing the steps of a robot using reinforcement learning. In some embodiments, the legged robot utilizes a neural network trained using reinforcement learning to provide step sequences and velocity limits. For example, the neural network can select gait timing parameters and / or velocity limits for the robot using various input parameters (e.g., joint angle measurements, pose data, and / or terrain models) that indicate the current state of the robot's measurements. Using a learned model can improve the robot's ability to successfully navigate complex, multi-factor terrain compared to hand-coded heuristic systems.

[0078] Figure 3This is a block diagram illustrating an example of a gait control system 300 used to generate step count parameters. Figure 3 As shown, the gait control system 300 includes a main path generator 306, a speed rescaling controller 312, a speed limit controller 310, a gait controller selector 314, multiple gait controllers 318, and one or more downstream controllers 322.

[0079] The main body path generator 306 is configured to receive trajectory 302 (also known as target trajectory) and obstacle data 304 as input, and generate a main body path to be followed by the robot body, which avoids the obstacles represented by obstacle data 304 while following trajectory 302. The body path generated by the body path generator 306 is provided to each of the speed rescaling controller 312 and speed limit controller 310.

[0080] The speed limit controller 310 is configured to receive a main path and one or more maps 308, and generate a speed limit for the robot's movement while following trajectory 302. Map 308 may include, for example, the locations of different features within the environment that may affect the robot's ability to traverse the environment. For example, map 308 may include: a staircase map, an obstacle map, a barrier map, a no-entry zone map, a cliff map, etc. In some embodiments, one or more maps 308 may be generated based on perception data obtained using one or more of the robot's sensors 132. Perception data may include, for example, terrain elevation changes, staircase models, etc., which may be used to construct at least a portion of one or more maps 308. Maps 308 may be included in separate maps, or in other embodiments, one or more maps 308 may be included in the same map.

[0081] Speed ​​limit controller 310 can be configured to generate speed limits designed to ensure the robot remains stable while following trajectory 302. Speed ​​limit controller 310 is also configured to generate speed limits that enable the robot to navigate to features and / or objects identified by map 308. According to embodiments, the speed limit can vary along the trajectory, allowing the robot to move faster along terrain that is relatively easier to traverse and to slow down when traversing more difficult terrain. In some embodiments, speed limit controller 310 can be configured to generate speed limits based on detecting one or more of the following terrain conditions: stair run and rise height, narrow obstacle corridors, hurdle height, changes in terrain height, approach to no-entry zones, and / or approach to cliffs.

[0082] The velocity rescaling controller 312 is configured to receive the main path and velocity limits, and rescale the trajectory as necessary such that the rescaled trajectory does not exceed the velocity limits. For example, the velocity rescaling controller 312 may be configured to modify the velocity of the received main trajectory to conform to the velocity limits received from the velocity limit controller 310. Since the velocity limits are generated by the velocity limit controller 310 to take into account the robot's state and perception state, the velocity rescaling controller 312 can generate rescaled trajectories to ensure that the robot can robustly traverse any mapped features and / or objects in the environment.

[0083] In some embodiments, the main trajectory can be represented as a series of x / y / yaw points and associated time. While the main path generator 306 is configured to generate the main trajectory as a feasible trajectory for the robot (e.g., the robot body can be in all positions along the trajectory without intersecting with obstacles), the velocity rescaling controller 312 can improve the reliability of the robot following the main trajectory by rescaling the velocity of the trajectory as described above.

[0084] Gait controller selector 314 is configured to receive a rescaled trajectory and select one of a plurality of gait controllers 318 for generating step parameters based on the trajectory. Each of the gait controllers 318 can be designed to have a unique, immutable set of parameters for the corresponding gait (e.g., trot and its attributes), and therefore, it is desirable to select the gait controller 318 that best follows the trajectory taking into account terrain conditions. For example, gait controller selector 314 can be configured to evaluate each of the plurality of gait controllers 318 based on generating possible contact sequences for each gait controller 318. Gait controller selector 314 can assign a cost to each contact sequence and select the gait controller 318 associated with the lowest-cost contact sequence.

[0085] In one exemplary embodiment, the gait controller selector 314 may select one of the gait controllers 318 based on the following process: The gait controller selector 314 may determine a step sequence for each gait controller 318, which will instruct the robot to follow a rescaled trajectory. For example, the gait controller selector 314 may determine a simplified MPC output that includes the step sequence of each gait controller 318. Since it may be difficult to determine the full MPC output of each gait controller 318 at the speed of real-time robot control, the simplified MPC output can be used as an initial step to select one of the gait controllers 318 for performing the rescaled trajectory.

[0086] The gait controller selector 314 can apply a set of costs to each simplified MPC output. Each cost can be applied to a corresponding metric that measures one or more aspects of the corresponding MPC output. Example metrics that can be used to apply costs to MPC outputs include: a measurement of the extent to which the MPC output steps into a no-step region, a measurement of the extent to which the robot's legs extend while following the MPC output, a measurement of the distance between the MPC output timing parameters and the nominal timing, etc. Once the costs have been assigned to each of the MPC outputs, the gait controller selector 314 can select the gait controller 318 associated with the step sequence of the lowest-cost simplified MPC output.

[0087] The selected gait controller 318 is configured to generate a set of stepping parameters 320 to cause the robot to follow a rescaled trajectory. Such stepping parameters 320 may include, for example, desired step placement, desired center of mass (CoM) acceleration, desired force at the CoM, desired force at each actuator, desired momentum change, etc. The selected gait controller 318 may receive the rescaled trajectory and robot state as inputs for generating the set of stepping parameters 320. The selected gait controller 318 may provide this set of stepping parameters 320 to one or more downstream controllers 322, which may then control the robot to follow the set of stepping parameters 320. Depending on the implementation, the one or more downstream controllers 322 may include, for example, a body planner, a swing planner, a whole-body controller, one or more leg controllers, etc.

[0088] Although Figure 3 The type of gait control system 300 can perform well when the robot traverses terrain types where there is a specially designed gait controller 318, but the gait control system 300 may not perform well at the same level when traversing multiple terrain types or terrain types without a corresponding gait controller 318.

[0089] Therefore, aspects of this disclosure relate to gait control systems that incorporate neural networks to improve robot performance when traversing these types of multi-factor terrain conditions.

[0090] Figure 4 This is a block diagram illustrating one embodiment of a gait control system 400 for generating step parameters using reinforcement learning. (Reference) Figure 4 The gait control system 400 includes a main path generator 306, a speed limit controller 310, a learning parameter selector 410, a gait controller 414, and one or more downstream controllers 322. Figure 3Compared to the gait control system 300, the gait control system 400 includes a learning parameter selector 410 in place of one or more of the speed limit controller 310 and the gait controller selector 314. Additionally, the gait control system 400 may include a single gait controller 414 replacing a plurality of gait controllers 318 from the gait control system 300. However, aspects of this disclosure are not limited thereto, and in other embodiments, the gait control system 400 may include a plurality of gait controllers 414. Depending on the embodiment, the gait controller 414 may include an MPC controller configured to generate a set of step parameters 320 following a rescaled trajectory based on a set of gait timing parameters 412 received from the learning parameter selector 410.

[0091] The learning parameter selector 410 is configured to receive a main path and one or more other input parameters from the main path generator 306, such as a terrain height map 402, a no-step map 404, a robot state 406, and / or a control state (or stage) 408. In some embodiments, the robot state 406 may include: robot joint angle measurements, pose estimation, terrain model, etc. In addition to the robot's current state, in some embodiments, the robot state 406 may also include a previous state.

[0092] The learning parameter selector 410 is configured to generate gait timing parameters 412 based on the main path and input parameters. In some embodiments, the gait timing parameters 412 may include a gait sequence and velocity limit parameters, which may be used by the gait controller 414 to generate the set of step parameters 320. In some embodiments, the gait timing parameters 412 may include: desired step time, desired step time for N future stances, velocity scaling multiplier, and a series of future contact states.

[0093] In some embodiments, the learning parameter selector 410 includes a neural network 411 trained using reinforcement learning. As described herein, by using the learning parameter selector 410 trained using reinforcement learning, the gait control system 400 is able to improve performance under certain conditions, including multi-factor terrain, compared to other embodiments.

[0094] Depending on the embodiment, the learning parameter selector 410 may receive various input parameters that can be used to generate the gait timing parameters 412. These input parameters may include: a terrain height map 402, a no-step map 404, a robot state 406, a control state 408, user-specified desired robot behavior, a trajectory 302, a main path, and / or perception information. In some embodiments, the user-specified desired robot behavior may include user-defined constraints that limit the types of motion that the robot can perform. Perception information may include, for example, perception data, distances to foot obstacles, distances to main obstacles, etc.

[0095] Figure 5 An example visualization of perception data 500 generated by one or more sensors 132 of a robot according to aspects of this disclosure is shown. Figure 5 As shown, the perception data can be compiled to generate one or more maps representing features and / or objects within the robot's environment.

[0096] As described above, the learning parameter selector 410 is configured to generate gait timing parameters 412 based on the received input parameters. In some examples, the gait timing parameters 412 include modifications to trajectory velocity and / or contact sequence.

[0097] Where gait timing parameter 412 defines a step sequence and / or timing for a specific gait (e.g., trot), modifications may include modifying the timing parameters for that specific gait. In other embodiments, gait timing parameter 412 may generate a more general gait timing parameter 412, including modifications to the sequence and / or contact time for any gait type. In some embodiments, gait timing parameter 412 may also include adjustments to desired body position and / or velocity.

[0098] Because the gait timing parameters 412 generated by the learning parameter selector 410 can define the properties of a specific gait that the robot will perform, the gait controller 414 can be designed to be gait-agnostic. That is, the gait controller 414 does not need to be assigned a set of parameters associated with a unique gait, because the gait timing parameters 412 provided to the gait controller 414 can include enough information to define the gait. Figure 6 An example visualization of gait timing parameters 412 generated by the learning parameter selector 410 according to aspects of this disclosure is shown. Figure 6 As shown, gait timing parameter 412 may include modifications to the timing of the contact sequence and / or trajectory velocity.

[0099] Figure 4The gait controller 414 is configured to generate a set of step parameters 320 based on a rescaled trajectory received from the speed limit controller 310 and gait timing parameters 412 received from the learning parameter selector 410. In some embodiments, because the gait controller 414 receives the gait timing parameters 412 from the learning parameter selector 410, this set of step parameters 320 can be generated using a single gait controller 414. In other words, because the gait controller 414 can generate a set of step parameters 320 for a variety of different terrain types, including multi-terrain conditions, the gait control system 400 can be implemented without a separate gait controller 318 designed to handle different terrain types. Therefore, the gait controller 414 can be considered a continuous gait controller that can generate a set of step parameters 320 for substantially the entire range of possible gaits.

[0100] Neural networks 411 can be trained to improve a robot's ability to traverse terrain conditions that the robot may not have been specifically programmed to do so. For example, Figure 3 The gait control system 300 includes a plurality of gait controllers 318, each of which can be programmed to generate a set of stepping parameters 320 that enable the robot to traverse specific terrain conditions. One limitation of this embodiment is that the robot may have a lower success rate when traversing terrain conditions that do not have corresponding gait controllers 318. In some embodiments, the success rate can be measured by the robot's stability and / or whether the robot violates one or more user-defined constraints.

[0101] In contrast, neural network 411 can be trained to mimic the behavior of gait controller 318, while also being trained to handle other and / or unexpected terrain types. For example, neural network 411 can be trained to generate a set of step parameters 320 that are substantially similar to a set of step parameters 320 generated by multiple gait controllers 318 when neural network 411 and gait controller 318 have the same set of input parameters.

[0102] In some embodiments, reinforcement learning techniques can be used to train the neural network 411. Advantageously, reinforcement learning does not require labeled input / output data; instead, it can be used to search or explore the space of possible solutions to find the weights of the neural network that best match one or more objectives (e.g., stability and / or user-specified desired robot behavior). In some embodiments, reinforcement learning may include a random search or exploration of the space of possible solutions. Due to the number of degrees of freedom the robot has in its environment, the space of possible solutions can be a relatively high-dimensional space.

[0103] Figure 7A method 700 for training a neural network 411 according to aspects of this disclosure is illustrated. One or more blocks of method 700 may be implemented, for example, by data processing hardware configured for training the neural network 411. In some embodiments, the data processing hardware may be a dedicated processor, such as a graphics processing unit (GPU), or other data processing hardware designed for training neural networks. Method 700 begins at block 701.

[0104] like Figure 7 As shown, method 700 involves training a neural network 411 using at least two separate training steps, as shown in blocks 702 and 704. At block 702, data processing hardware may initialize the neural network 411 to generate multiple sets of step parameters 320 that are substantially similar to those generated by multiple sets of step parameters 320 of a plurality of gait controllers 318 of the gait control system 300. For example, the data processing hardware may initialize the neural network 411 such that the multiple sets of step parameters 320 generated by the neural network 411 are within a threshold difference of the multiple sets of step parameters 320 generated by the multiple gait controllers 318 of the model gait control system (e.g., the previously designed gait control system 300). In some embodiments, initializing the neural network 411 may include training the neural network 411 to reproduce the set of step parameters 320 generated by the multiple gait controllers 318 within a threshold deviation level. In some embodiments, the data processing hardware may initialize the neural network 411 by minimizing the difference between the multiple sets of step parameters 320 generated by the neural network 411 and the multiple sets of step parameters 320 generated by the multiple gait controllers 318.

[0105] In some embodiments, initializing the neural network 411 may involve using a cloning method (also known as “behavioral cloning,” “imitation learning,” or “knowledge distillation”) to mimic multiple gait controllers 318. In some embodiments, the cloning method may include the DAgger algorithm (dataset aggregation), although other cloning methods are also possible. In some embodiments, initializing the neural network 411 may include using physical simulations to generate a large number of different datasets that demonstrate the gait parameters of the previously designed gait control system 300 for a given simulated robot state. The weights of the initialized neural network 411 may be adjusted to minimize the output error on the simulated dataset. The initialization of the neural network 411 may be performed to mimic each of the multiple gait controllers 318. While initializing the neural network 411 using a cloning method may have some of the advantages described above, aspects of this disclosure are not limited to the use of cloning techniques. For example, in some embodiments, the data processing hardware may randomly initialize the neural network 411, which may also produce good results after reinforcement learning at block 704.

[0106] At box 704, the data processing hardware can use reinforcement learning to train the initialized neural network 411. In some embodiments, the data processing hardware may also partially randomize the neural network 411 before performing reinforcement learning. The data processing hardware can modify the neural network 411 using same-policy reinforcement learning, for example, by running an MPC controller of the gait controller 414 multiple times in a simulation. In one embodiment, simulating the MPC controller may include randomly exploring the space of possible solutions. Each simulation may generate a set of step parameters 320, and the data processing hardware may evaluate the simulated set of step parameters 320 for one or more costs to determine the performance of the simulation. Costs may include, for example, measurements of robot stability and measurements of whether the step parameters 320 violate one or more constraints associated with the desired robot behavior. For example, one or more constraints associated with the desired robot behavior may be user-defined constraints that limit the types of motion performed by the robot. The data processing hardware may adjust the weights of the neural network 411 based on the simulated set of step parameters 320 and corresponding evaluations of the performance of the simulated step parameters 320.

[0107] Figure 8 A method 800 for controlling the motion of a robot using a trained neural network 411, according to aspects of this disclosure, is illustrated. One or more blocks of method 800 may be, for example, derived from... Figure 1 The data processing hardware 142 of the computing system 140 of the robot 100 or any other data processing hardware of the robot is used for implementation. Method 800 begins at box 801.

[0108] At box 802, data processing hardware 142 receives the trajectory the robot is to follow. The trajectory can correspond to... Figure 4 The trajectory 302. In one example, data processing hardware 142 can generate a body path for the robot's body to follow, which avoids objects in the environment as it traverses the environment according to the trajectory.

[0109] At box 804, data processing hardware 142 receives the robot's status (such as...). Figure 4 (as shown in robot state 406). In some embodiments, data processing hardware 142 executes a learning parameter selector 410, which uses robot state 406 and the main path as input. Figure 4 As shown, the learning parameter selector 410 can also receive additional inputs, such as terrain height map 402, no-step map 404 and / or control status 408, as well as other possible inputs.

[0110] At box 806, data processing hardware 142 uses a neural network (e.g., Figure 4The neural network 411 uses the robot's state and trajectory as input to the neural network to generate multiple gait timing parameters of the robot (e.g., Figure 4 (Gait timing parameters 412). In some embodiments, the neural network 411 may be included as part of the learning parameter selector 410. In some embodiments, it may be included, for example, by combining with Figure 7 The method described is 700 to train the neural network 411.

[0111] At block 808, data processing hardware 142 controls the robot's motion based on gait timing parameters. In some embodiments, data processing hardware 142 may use gait controller 414 to generate a set of step parameters 320 based on gait timing parameters 412. Data processing hardware 142 may also implement one or more downstream controllers 322 configured to control the robot's motion based on this set of step parameters 320. Method 800 ends at block 810.

[0112] Figure 9 This is a schematic diagram of an example computing device 900 that can be used to implement the systems and methods described in this document. The computing device 900 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the inventions described and / or claimed in this document.

[0113] Computing device 900 includes a processor 910, a memory 920, a storage device 930, a high-speed interface / controller 940 connected to the memory 920 and a high-speed expansion port 950, and a low-speed interface / controller 960 connected to a low-speed bus 970 and the storage device 930. Each of components 910, 920, 930, 940, 950, and 960 is interconnected using various buses and may be suitably mounted on a common motherboard or otherwise mounted. Processor 910 can process instructions for execution within computing device 900, including instructions stored in memory 920 or storage device 930, to display graphical information of a graphical user interface (GUI) on an external input / output device, such as a display 980 coupled to high-speed interface 940. In other embodiments, multiple processors and / or multiple buses, as well as multiple memories and various types of memory, may be suitably used. Furthermore, multiple computing devices 900 may be connected, each providing a portion of the necessary operation (e.g., as a server group, a set of blade servers, or a multiprocessor system).

[0114] Memory 920 stores information non-transitorily within computing device 900. Memory 920 may be a computer-readable medium, a volatile memory cell, or a non-volatile memory cell. Non-transitory memory 920 may be a physical device for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) for use by computing device 900. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used in firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0115] Storage device 930 provides mass storage for computing device 900. In some embodiments, storage device 930 is a computer-readable medium. In various embodiments, storage device 930 may be a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory, or other similar solid-state storage device or array of devices, including devices in a storage area network or other configuration. In other embodiments, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 920, storage device 930, or memory on processor 910.

[0116] High-speed controller 940 manages bandwidth-intensive operations of computing device 900, while low-speed controller 960 manages lower bandwidth-intensive operations. This allocation of responsibilities is merely exemplary. In some embodiments, high-speed controller 940 is coupled to memory 920, display 980 (e.g., via a graphics processor or accelerator), and high-speed expansion port 950 which can accept various expansion cards (not shown). In some embodiments, low-speed controller 960 is coupled to storage device 930 and low-speed expansion port 990. Low-speed expansion port 990, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), may be coupled to one or more input / output devices, such as keyboards, pointing devices, scanners, or networking devices such as switches or routers, for example, via a network adapter.

[0117] As shown in the figure, the computing device 900 can be implemented in a variety of different forms. For example, it can be implemented as a standard server 900a or multiple times in a set of such servers 900a, as a laptop computer 900b, or as part of a rack server system 900c.

[0118] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs executable and / or interpretable on a programmable system, which includes at least one programmable processor, which may be dedicated or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, at least one input device, and at least one output device.

[0119] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0120] The processes and logic flows described in this specification can be executed by one or more programmable processors (also known as data processing hardware), executing one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). As an example, processors suitable for executing computer programs include both general-purpose and special-purpose microprocessors, and any one or more processors of any kind of digital computer. A processor can receive instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. A computer may include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or operatively coupled to receive data from or transfer data to, or both. However, a computer does not need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROMs, EEPROMs, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. Processors and memory can be supplemented by dedicated logic circuits or incorporated into dedicated logic circuits.

[0121] To provide interaction with a user, one or more aspects of this disclosure can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touchscreen, and optionally a keyboard and indicating device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending web pages to a web browser on the user's client device in response to a request received from a web browser.

[0122] Many embodiments have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other embodiments are within the scope of the appended claims.

Claims

1. A method comprising: The target trajectory of the legged robot is received by the control system of the legged robot; The control system receives the status of the legged robot; Based at least in part on the legged robot's state and the target trajectory, a set of gait timing parameters for the legged robot is generated using a neural network of the control system; and The control system controls the movement of the legged robot based on the set of gait timing parameters.

2. The method according to claim 1, wherein, The neural network is trained using reinforcement learning.

3. The method according to claim 1, wherein, The gait timing parameters include the contact sequence.

4. The method according to claim 3, wherein, The contact sequence includes at least one target step time.

5. The method according to claim 3, wherein, The gait timing parameters also include a velocity scaling factor.

6. The method according to claim 1, further comprising: The model predictive controller (MPC) of the control system generates a set of stepping parameters based on the gait timing parameters. The movement of the legged robot is also controlled based on the set of stepping parameters.

7. The method according to claim 6, wherein, The set of stepping parameters includes at least one of stepping placement or desired center of mass acceleration.

8. The method according to claim 6, further comprising: The neural network is initialized to reproduce the set of step parameters within a threshold difference of the previous set of step parameters; and The initialized neural network is trained using reinforcement learning, which includes simulating the MPC to search the space of possible solutions.

9. The method according to claim 1, further comprising: Receive perception data indicating the environment of the legged robot; and A map of the environment is generated based on the perceived data. The neural network also uses the map of the environment as input.

10. The method according to claim 1, wherein, The neural network also uses a set of input parameters as input, which include one or more of the following: terrain height map, no-stepping map, control state, user-specified desired robot behavior, main path or perception data.

11. The method according to claim 1, further comprising: Receive obstacle data; and The main path is generated based on the trajectory and obstacle data. The neural network also uses the main path as input.

12. A legged robot, comprising: main body; Two or more legs, said two or more legs being coupled to the body; One or more sensors, the one or more sensors being configured to measure the state of the legged robot; as well as A control system, communicating with the main body and the two or more legs, includes data processing hardware and memory hardware communicating with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to: Receive the target trajectory of the legged robot; The state of the legged robot is received from the one or more sensors; Based at least in part on the legged robot's state and the target trajectory, a set of gait timing parameters for the legged robot are generated using a neural network of the control system; and The movement of the legged robot is controlled based on the set of gait timing parameters.

13. The legged robot according to claim 12, wherein, The neural network is trained using reinforcement learning.

14. The legged robot according to claim 12, wherein, The gait timing parameters include the contact sequence.

15. The legged robot according to claim 12, wherein, When the instruction is executed on the data processing hardware, it further causes the data processing hardware to: The model predictive controller (MPC) of the control system generates a set of stepping parameters based on the gait timing parameters. The movement of the legged robot is also controlled based on the set of stepping parameters.

16. The legged robot according to claim 15, wherein, The set of stepping parameters includes at least one of stepping placement or desired center of mass acceleration.

17. The legged robot according to claim 15, wherein, When the instruction is executed on the data processing hardware, it further causes the data processing hardware to: Initialize the neural network to reproduce the set of step parameters within a threshold difference of a previous set of step parameters; and The initialized neural network is trained using reinforcement learning, which includes simulating the MPC to search the space of possible solutions.

18. The legged robot according to claim 12, wherein, When the instruction is executed on the data processing hardware, it further causes the data processing hardware to: Receive perception data indicating the environment of the legged robot; and A map of the environment is generated based on the perceived data. The neural network also uses the map of the environment as input.

19. A non-transitory computer-readable medium storing instructions that, when executed by data processing hardware of a control system, cause the data processing hardware to: Receive the target trajectory of the legged robot; Receive the status of the legged robot; Based at least in part on the legged robot's state and the target trajectory, a set of gait timing parameters for the legged robot are generated using a neural network of the control system; and The movement of the legged robot is controlled based on the set of gait timing parameters.

20. The non-transitory computer-readable medium according to claim 19, wherein, The neural network is trained using reinforcement learning.

Images