Robot Choreography

Through action scripts and user interface combination actions, robot motion routines are generated, which solves the problem of hard-coded time-consuming and neglecting environmental characteristics, and realizes fast and safe robot motion programming.

CN114466732BActive Publication Date: 2024-11-26BOSTON DYNAMICS INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202080067130.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-10-14
Filing Date
2020-10-12
Publication Date
2024-11-26
Estimated Expiration
2040-10-12

AI Technical Summary

Technical Problem

In the prior art, robot motion programming requires hard code, which is time-consuming and limited to a few coders' operations, making it difficult to quickly create robot behavior, and may ignore the robot environment and physical characteristics, resulting in programming failure or damage.

Method used

Through action scripts and user interfaces, users can combine and parameterize existing actions in an intuitive and near real-time manner, generate robot motion routines without hard coding, and the data processing hardware recognizes multiple actions in action scripts and generates joint commands , controls robot motion, considers the robot environment and physical characteristics.

Benefits of technology

It realizes the rapid generation of robot motion routines, reduces programming complexity and time, improves the ability of most users to program, and ensures motion safety and success rate.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114466732B_ABST
    Figure CN114466732B_ABST
Patent Text Reader

Abstract

A method for generating joint commands (302) includes receiving an action script (202) including a plurality of actions (210) for a legged robot (100) to perform, wherein each action is associated with a cost (336). The method also includes identifying that two or more of the plurality of actions of the action script occur at the same time. The method also includes determining a combined action performed by the legged robot at the time based on the two or more actions and the costs associated with the two or more actions. The method further includes generating a joint command to control movement of the legged robot at a time when the joint command commands a set of joints (J) of the legged robot. Here, the set of joints corresponds to the combined action.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to robotic choreography. Background Art

[0002] A robot is generally defined as a reprogrammable and multifunctional manipulator designed to move materials, parts, tools, or specialized equipment through variable programmed motions for performing tasks. A robot can be a physically anchored manipulator (e.g., an industrial robot arm), a mobile robot that moves throughout an environment (e.g., using legs, wheels, or traction-based mechanisms), or some combination of a manipulator and a mobile robot. Robots are used in a variety of industries, including, for example, manufacturing, transportation, hazardous environments, exploration, and healthcare. Therefore, the ability to program robots for a variety of motion routines in a fast and efficient manner provides additional benefits to these industries. Summary of the invention

[0003] One aspect of the present disclosure provides a method for generating joint commands. The method includes receiving an action script at data processing hardware, and the action script includes multiple actions for a legged robot to perform. Here, each action is associated with a cost. The method also includes identifying by the data processing hardware that two or more actions of the multiple actions of the action script occur at the same time. The method also includes determining by the data processing hardware based on the two or more actions and the costs associated with the two or more actions that the legged robot performs at the time. The method also includes generating joint commands by the data processing hardware to control the movement of the legged robot at the moment when the joint commands command a group of joints of the legged robot. Here, the group of joints corresponds to the combined action.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the legged robot corresponds to a quadruped robot. In some examples, one of the multiple actions includes a prompt, wherein the prompt corresponds to a body movement of the body of the legged robot. In these examples, the method may also include determining, by data processing hardware, whether the prompt is compatible with another action among the multiple actions that occur at the same time as the prompt, and when the prompt is compatible with the other action, modifying the other action by the data processing hardware to combine the body movement of the prompt. In some configurations, each action is configured to use an active controller for the legged robot or to modify an active controller for the legged robot, wherein the action of modifying the active controller for the legged robot defines a prompt. Optionally, the cost associated with each action includes a user-defined cost that indicates the importance of the legged robot performing the action.

[0005] In some examples, at least one of the multiple actions includes a footstep action, wherein the footstep action includes the position and time at which the swinging leg of the legged robot touches the ground. In some implementations, at least one of the multiple actions includes an arm action, wherein the arm action includes the posture of the manipulator of the legged robot. In some configurations, the action script includes a dance script, and each action includes a dance move. Here, the method may also include synchronizing each dance move of the dance script with the beat of the song by the data processing hardware. Additionally or alternatively, the method may also include determining by the data processing hardware that the exit state of the first action of the multiple actions of the action script meets the entry state of the subsequent action.

[0006] Another aspect of the present disclosure provides a robot capable of generating joint commands. The robot includes a main body, two or more legs connected to the main body, and a computing system communicating with the main body and the two or more legs. The computing system includes data processing hardware and memory hardware communicating with the data processing hardware. The memory hardware stores instructions that cause the data processing hardware to perform operations when these instructions are executed on the data processing hardware. These operations include receiving an action script that includes multiple actions for the robot to perform. Here, each action is associated with a cost. The operation also includes identifying that two or more actions of the multiple actions of the action script occur at the same time. The operations also include determining a combined action performed by the robot at the time based on the two or more actions and the costs associated with the two or more actions. The operations also include generating a joint command to control the movement of the robot at the moment when the joint command commands a group of joints of the robot. Here, the group of joints corresponds to the combined action.

[0007] This aspect may include one or more of the following optional features. In some implementations, the legged robot corresponds to a quadruped robot. In some examples, one of the multiple actions includes a prompt, where the prompt corresponds to a main body movement of a body. In these examples, the operation may also include determining whether the prompt is compatible with another action in the multiple actions that occur at the same time as the prompt, and when the prompt is compatible with the other action, modifying the other action to incorporate the main body movement of the prompt. In some configurations, each action is configured to use an active controller of the robot or to modify the active controller of the robot, where the action of modifying the active controller of the robot defines the prompt. Optionally, the cost associated with each action includes a user-defined cost that indicates the importance of the robot performing the action.

[0008] In some examples, at least one of the plurality of actions includes a footstep action, wherein the footstep action includes the position and time at which the swinging leg of the robot touches the ground. In some implementations, at least one of the plurality of actions includes an arm action, wherein the arm action includes the posture of the robot's manipulator. In some configurations, the action script includes a dance script, and each action includes a dance move. Here, the operation may further include synchronizing each dance move of the dance script with the beat of the song. Additionally or alternatively, the operation may also include determining that the exit state of the first action of the plurality of actions of the action script meets the entry state of the subsequent action.

[0009] The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the following description. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1A A perspective view of an example robot in a robotics environment.

[0011] Figure 1B and Figure 1C for Figure 1A Schematic diagram of an example system of a robot.

[0012] Figures 2A-2F For Figure 1A Schematic diagram of an example user interface for robot interaction.

[0013] Figure 3 for Figure 1A Schematic diagram of an example motion system of a robot.

[0014] Figure 4 Example arrangement of operations for a robot to perform actions.

[0015] Figure 5 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0016] Like reference symbols in the various drawings indicate like elements.

[0017] Specific implementation method

[0018] Often, an entity that owns or controls a robot wishes to pre-program the robot's motion. For example, the entity wishes the robot to perform a repeatable motion routine. Unfortunately, the pre-programmed motion may require hard coding to generate the desired motion. Here, hard coding refers to someone, such as a programmer, generating or editing a set of instructions written in a computer programming language. Hard coding is often time-consuming and can result in bottlenecks where only a limited number of people (e.g., programmers / coders) are able to code the robot's motion. Furthermore, modifications to the coded motion or debugging of the coded motion may result in lengthy feedback loops to implement the motion routine. In other words, robots often lack the means to quickly author robot behaviors. Furthermore, the robot's manufacturer may block the ability to hard code new robot motions and / or block the ability to edit code that was previously hard-coded for existing robot motions.

[0019] Another potential issue with pre-programmed motion is that it may or may not take into account the environment surrounding the robot and / or the physical characteristics of the robot itself. In other words, the person coding the pre-programmed motion may not be aware that there may be other constraints placed on the robot during the pre-programmed motion. If these other constraints are not taken into account, the pre-programmed routine may fail or damage the robot or the environment surrounding the robot during execution. For example, a coder for a pre-programmed motion routine may not realize that the robot will be carrying an additional 25 lbs of load during the routine, or that the routine may take place in a dynamic environment with static or dynamic obstacles. This obviously places the onus on the entity or coder to consider all or most of the conditions that the robot will experience during the motion routine in order to achieve a successful motion routine.

[0020] To overcome some of these problems, refer to Figure 1A, a user 10 designs a motion routine for a robot 100 at a user interface (UI) 200. The robot 100 includes a motion system 300 in communication with the UI 200. Here, the UI 200 enables the user 10 of the robot 100 to generate a motion routine in the form of an action script 202. The action script 202 generally refers to a plurality of actions 210, 210a-n (e.g., possibly overlapping in time) that the user 10 designs at the UI 200 for the robot 100 to perform in the robot environment 30. Here, the UI 200 enables the robot 100 to build motions without requiring any new hard coding. In other words, the user 10 uses the UI 200 to obtain existing actions 210 of the robot 100, which can be layered or combined together to generate a motion routine for the robot 100 in an intuitive and near real-time manner. In addition, some actions 210 of the UI 200 are parameterized, so that the user 10 can customize aspects of these actions 210 at the UI 200 to fit his / her desired motion routine. As will become apparent, while the actions 210 obtained and selected by the user 10 for the motion routine may themselves be pre-programmed or hard-coded into the robot (e.g., by the manufacturer of the robot 100), the combination and sequence of actions 210 in the action script 202 enables the robot 100 to execute / perform the motion routine without requiring any hard coding of the motion routine by the user 10.

[0021] In some instances, the user 10 runs the UI 200 on the user device 20. Here, the user device 20 uses computing resources 22, 24 such as data processing hardware 22 and memory hardware 24 in communication with the data processing hardware 22 to execute the UI 200. Some examples of the user device 20 include a mobile device (e.g., a mobile phone, a tablet, a laptop, etc.), a personal computer (PC), a workstation, a terminal, or any other computing device with computing capabilities to execute the UI 200. The UI 200 can be web-based (e.g., a web-based application launched from a web browser), local to the user device 20 (e.g., installed on and / or configured to execute on the computing resources 22, 24 of the user device 20), or some combination of the two. In some implementations, the user device 20 uses the UI 200 to transmit the action script 202 to the robot 100 using a wired or wireless connection. For example, the UI 200 is transmitted over a network (e.g., a computer). Figure 1B The network 150 shown transmits the action script 202 to the robot 100.

[0022] Continue to refer Figure 1A, the robot 100 includes a body 110 having a motion-based structure, such as legs 120a-d connected to the body 110, the legs 120a-d enabling the robot 100 to move around the environment 10. In some examples, each leg 120 is an articulated structure such that one or more joints J allow a member 122 of the leg 120 to move. For example, each leg 120 includes an upper member 122, 122 of the leg 120 U Hip joint J connected to the main body 110 H and the upper member 122 of the leg 120 U The lower member 122 connected to the leg 120 L Knee joint J K .although Figure 1A A quadruped robot having four legs 120a - d is depicted, but the robot 100 may include any number of legs or locomotion-based structures (eg, a biped or humanoid robot having two legs) that provide a means of traversing terrain within the environment 10 .

[0023] To traverse terrain, each leg 120 has a distal end 124 that contacts the terrain surface (i.e., a traction surface). In other words, the distal end 124 of the leg 120 is the end of the leg 120 used by the robot 100 to pivot, position, or generally provide traction during movement of the robot 100. For example, the distal end 124 of the leg 120 corresponds to the foot of the robot 100. In some examples, although not shown, the distal end 124 of the leg 120 includes an ankle joint J. A , so that the distal end 124 can be relative to the lower member 122 of the leg 120 L Articulated.

[0024] In some examples, the robot 100 includes an arm 126 that functions as a robotic manipulator. The arm 126 can be configured to move about multiple degrees of freedom in order to engage elements of the environment 30 (e.g., objects within the environment 30) or to perform a gesture (e.g., an aesthetic gesture). In some examples, the arm 126 includes one or more members 128, wherein the members 128 are connected by joints J so that the arm 126 can pivot or rotate about joint(s) J. For example, the arm 126 can be configured to extend or retract with more than one member 128. To illustrate, Figure 1A An arm 126 is depicted having three members 128 corresponding to the lower member 128 L 、Upper member 128 U and hand component 128 H Here, the lower member 128 L The first arm joint J located near the main body 110 can be A1Rotate or pivot (eg, where the arm 126 is connected to the body 110 of the robot 100). L At the second arm joint J A2 Connected to the upper member 128 U , and the upper member 128 U At the third arm joint J A3 Connected to hand member 128 H In some examples, hand member 128 H Additional components are included to achieve different types of grips. These additional components can range from a simple two-component claw-like hand component 128 H To a more complex hand component 128 that simulates the fingers of a human hand H In some implementations, the arm 126 connects to the robot 100 at a socket on the body 110 of the robot 100. In some configurations, the socket is configured as a connector so that the arm 126 can be attached to or detached from the robot 100 depending on whether the arm 126 is needed for operation.

[0025] The robot 100 has a vertical gravity axis (eg, shown as a Z-axis A) along the direction of gravity. Z ) and a center of mass CM, the position of which corresponds to the average position of all parts of the robot 100, where these parts are weighted according to their mass (i.e., the point where the sum of the weighted relative positions of the distributed mass of the robot 100 is zero). The robot 100 also has a relative gravity axis A based on the vertical gravity axis A. Z The posture P of the CM of the robot 100 (i.e., a fixed reference frame relative to gravity) is defined to define a particular posture or stance assumed by the robot 100. The posture of the robot 100 may be defined by the orientation or angular position of the robot 100 in space. Movement of the legs 120 relative to the body 110 changes the posture P of the robot 100 (i.e., a combination of the position of the CM of the robot and the posture or orientation of the robot 100). Here, height generally refers to distance along the z-direction. The sagittal plane of the robot 100 corresponds to the y-axis A. Y and Z-axis A Z In other words, the sagittal plane divides the robot 100 into left and right sides. Generally perpendicular to the sagittal plane, the ground plane (also called the transverse plane) passes through the X-axis A X and Y-axis A YThe ground plane refers to the ground 12 where the distal ends 124 of the legs 120 of the robot 100 can generate traction to help the robot 100 move around the environment 30. Another anatomical plane of the robot 100 is a front plane extending through the body 110 of the robot 100 (e.g., from the left side of the robot 100 with the first leg 120a to the right side of the robot 100 with the second leg 120b). The front plane passes through the X-axis A in the X direction. X and Z-axis A Z The XZ plane extends in the direction of the axis.

[0026] When the legged robot moves around the environment 30, each leg 120 of the robot 100 may be in contact with the ground 12, or not in contact with the ground 12. When the leg 120 is in contact with the ground 12, the leg 120 is referred to as a standing leg 120. ST When the leg 120 is not in contact with the ground 12, the leg 120 is referred to as a swing leg 120. SW When the leg 120 is lifted off the ground 12 (LO), the leg 120 is moved from the standing leg 120 ST Transition to swing leg 120 SW On the contrary, when the swing leg contacts the ground (TD) 12 after not contacting the ground 12, the swing leg 120 SW You can also transition to standing leg 120 ST Here, when the leg 120 is used as a swing leg 120 SW When the robot 100 is in the state of standing, the other leg 120 can be used as the standing leg 120 ST (For example, to maintain the balance of the robot 100).

[0027] To maneuver around the environment 30, the robot 100 includes a sensor system 130 having one or more sensors 132, 132a-n (e.g., shown as a first sensor 132, 132a and a second sensor 132, 132b). The sensors 132 may include vision / image sensors, inertial sensors (e.g., inertial measurement units (IUMs)), force sensors, and / or kinematic sensors. Some examples of vision / image sensors 132 include cameras (such as stereo cameras), scanning light detection and ranging (LIDAR) sensors, or scanning laser detection and ranging (LADAR) sensors. In some examples, the sensors 132 have corresponding (multiple) fields of view F that define a sensing range or area corresponding to the sensor 132. V .For example, Figure 1A The field of view F of the robot 100 is depicted V Each sensor 132 may be pivotable and / or rotatable such that the sensor 132 may change its field of view F, for example, about one or more axes (e.g., an x-axis, a y-axis, or a z-axis relative to a ground plane).V .

[0028] When using sensor 132 to survey the field of view F V When the sensor system 130 generates a signal corresponding to the field of view F V The sensor data 134 (also referred to as image data) is provided. In some examples, the sensor data 134 is image data corresponding to a three-dimensional volumetric point cloud generated by the three-dimensional volumetric image sensor 132. Additionally or alternatively, the sensor system 130 collects pose data of the robot 100 as the robot 100 maneuvers around the environment 30, the pose data including inertial measurement data (e.g., measured by an IMU). In some examples, the pose data includes kinematic data and / or orientation data about the robot 100, for example, kinematic data and / or orientation data about other parts or joints J of the legs 120 of the robot 100. Using the sensor data 134, various systems of the robot 100 can use the sensor data 134 to define a current state of the robot 100 (e.g., the kinematics of the robot 100) and / or a current state of the environment 30 around the robot 100.

[0029] In some implementations, the sensor system 130 includes (a plurality of) sensors 132 connected to the joint J. In some examples, these sensors 132 (e.g., sensors 132, 132a-b) are connected to the motors M that operate the joint J of the robot 100. Here, these sensors 132 generate joint dynamics in the form of joint-based sensor data 134. The joint dynamics collected as the joint-based sensor data 134 may include joint angles (e.g., upper member 122 U With respect to the lower member 122 L ), joint velocity (e.g., joint angular velocity or joint angular acceleration), and / or force (also referred to as joint force) at the joint J. Here, the joint-based sensor data 134 generated by the one or more sensors 132 may be raw sensor data, further processed to form different types of joint dynamics 134, JD For example, the sensor 132 measures the joint position (or the position of the member(s) 122 connected at the joint), and the system of the robot 100 performs further processing to derive the velocity and / or acceleration from the position data. In other examples, the sensor 132 is configured to measure the velocity and / or acceleration directly.

[0030] As the sensor system 130 collects sensor data 134, the computing system 140 is configured to store, process, and / or transmit the sensor data 134 to various systems of the robot 100 (e.g., the control system 170 and / or the motion system 300). To perform computing tasks related to the sensor data 134, the computing system 140 of the robot 100 includes data processing hardware 142 and memory hardware 144. The data processing hardware 142 is configured to execute instructions stored in the memory hardware 144 to perform computing tasks related to the activities (e.g., motion and / or motion-based activities) of the robot 100. In general, the computing system 140 refers to one or more locations of the data processing hardware 142 and / or the memory hardware 144.

[0031] In some examples, the computing system 140 is a local system located on the robot 100. When located on the robot 100, the computing system 140 can be centralized (i.e., at a single location / area on the robot 100, such as the body 110 of the robot 100), decentralized (i.e., located at various locations on the robot 100), or a hybrid combination of the two (e.g., with a majority of centralized hardware and a minority of decentralized hardware). To illustrate some of the differences, a decentralized computing system 140 can allow processing to occur at active locations (e.g., at the motors of the joints of the mobile legs 120), while a centralized computing system 140 can allow a central processing center to communicate with systems located at different locations on the robot 100 (e.g., communicating with the motors of the joints of the mobile legs 120).

[0032] Additionally or alternatively, the computing system 140 includes computing resources that are remote from the robot 100. For example, the computing system 140 communicates with a remote system 160 (e.g., a remote server or a cloud-based environment) via a network 150. Much like the computing system 140, the remote system 160 includes remote computing resources, such as remote data processing hardware 162 and remote memory hardware 164. Here, sensor data 134 or other processed data (e.g., data processed locally by the computing system 140) can be stored in the remote system 160 and can be accessed by the computing system 140. In some examples, the computing system 140 is configured to utilize the remote resources 162, 164 as an extension of the computing resources 142, 144, such that resources of the computing system 140 can reside on resources of the remote system 160. In some configurations, the remote system 160 and / or the computing system 140 hosts the UI 200, such that the user 10 accesses the UI 200 via the network 150 using a web browser application.

[0033] In some implementations, such as Figures 1A-1CAs shown, the robot 100 includes a control system 170. The control system 170 may be configured to communicate with systems of the robot 100, such as at least one sensor system 130 and / or an action system 300. The control system 170 may use the hardware 140 to perform operations and other functions. The control system 170 includes at least one controller 172 configured to control the robot 100. For example, the controller 172 controls the movement of the robot 100 to traverse around the environment 30 based on input or feedback from systems of the robot 100 (e.g., the sensor system 130, the control system 170, and / or the action system 300). In some implementations, the controller 172 controls the movement between postures and / or behaviors of the robot 100.

[0034] In some examples, the controller 172 controls the robot 100 by controlling the movement of one or more joints J of the robot 100. In some configurations, the controller 172 is software having programming logic to control at least one joint J or a motor M operating or connected to the joint J. For example, the controller 172 controls the amount of force applied to the joint J (e.g., torque at the joint J). As a programmable controller 172, the number of joints J controlled by the controller 172 is scalable and / or customizable for a specific control purpose. The controller 172 can control a single joint J of the robot 100 (e.g., control the torque at a single joint J) or control multiple joints J of the robot 100. When controlling multiple joints J, the controller 172 can apply the same or different torques to each joint J controlled by the controller 172. By controlling multiple joints J, the controller 172 can coordinate the movement of a larger structure (e.g., the body 110, one or more legs 120, arms 126) of the robot 100. In other words, for some actions 210, the controller 172 may be configured to control the movement of multiple parts of the robot 100, such as two legs 120a-b, four legs 120a-d, or a combination of two legs 120a-b and an arm 126. For example, Figure 1C Depicted is an example relationship between the motors M of the robot 100 and the joints J that these motors M control for a given structure of the robot 100. Here, Figure 1C Four controllers 172, 172a-d are shown which are arranged to control portions of the robot 100. A first controller 172, 172a controls a knee joint J associated with a first leg 120, 120a and a second leg 120, 120b. K and hip joint J H The second controller 172b is shown configured to control a knee joint J associated with a single leg (such as the third leg 120, 120c). K and hip joint J HThe third controller 172c is shown configured to control the motor M associated with the third leg 120c, the fourth leg 120, 120d, and the arm joint J A1 The fourth controller 172d is shown configured to control only the motor M associated with the arm 126 of the robot 100. For practical comparison, the third controller 172c can allow the robot 100 to tilt to the side of the robot's body 110 over the third leg 120c and the fourth leg 120d while also grasping something with the arm 180. Compared to the third controller 172c, the robot 100 can perform the tilt and grasp functions with three controllers 172, where two controllers 172 control the third and fourth legs 120c-d, respectively, and the fourth controller 172d controls the arm 126. There may be some advantages or reasons for coordinating multiple controllers 172 (e.g., for fine motor control) or using a single controller 172 (e.g., for simplicity).

[0035] refer to Figures 2A-2F , the UI 200 is configured to allow the user 10 to build a motion routine in the form of an action script 202. In some examples, the action script 202 is a computer-readable format that the user 10 (e.g., via the UI 200) communicates with the robot 100 to enable the robot 100 to perform the motion routine. In some implementations, the action script 202 is a series of movements within a set time period t that the user 10 requests the robot 100 to perform. In order to build the action script 202, the UI 200 provides a plurality of actions 210 an for the user 10 to select. Here, the action 210 refers to a hard-coded movement of the robot 100 coordinated by the controller 172 of the control system 170. The action 210 may correspond to the controller 172 of the robot 100 (e.g., an active controller programmed for the control system 170), a modification to the controller 172 of the robot 100, one or more input parameters 214 of the controller 172 of the robot 100, or any combination thereof. In some examples, an action 210 defines (e.g., by specifying a controller 172 that controls a particular joint J or joints J) a set of joints J necessary to perform the basic motion(s) associated with the action 210. Some different types of actions 210 include the following: footstep actions 210 of a foot 124 of a mobile robot 100, 210 f(or leg motion); arm motions 210, 210 a of the arm 126 of the mobile robot 100; and / or body motions 210, 210 b of the body 110 of the mobile robot 100. In addition, since the motions 210 may correspond to the controller 172, the movement of the motions 210 may correspond to any number of joints J, even joints J from different parts of the robot 100 (e.g., from two legs 120 on different sides of the body 110 of the robot). In other words, the motions 210 may be configured as a mixture of different types of motions. For example, a combination of footstep motions 210 and footstep motions 210 may be configured as a mixture of different types of motions. f and arm movements 210 a In some configurations, the action 210 is parameterized such that the action 210 is hard-coded with parameters 214 that can be modified to generate customizable changes in a set amount of motion. For example, the footstep action 210 f The parameters 214 may include, for example, which foot 124 is designated to step with, the coordinates / position of the step (e.g., LO / TD position), the swing height of the designated stepping foot 124, and / or the swing speed of the designated stepping foot 124. b In the case of, the parameters 214 may include a sway area, a sway position, a sway direction, a sway speed, and / or a sway style.

[0036] In some examples, to sequence one or more actions 210 to form an action script 202, the UI 200 divides both the actions 210 and the action script 202 into sections called strips. S In other words, the time band t from action 210 S Time band t with action script 202 S compatible, so that the action 210 can occupy one or more strips of the action script 202 S By subdividing the action script 202 into units, such as strips t S , the action script 202 can be scaled to different time lengths. In other words, the UI 200 divides the action script 202 into a plurality of discrete actions 210 over time, wherein each action 210 corresponds to a fixed number of strips t S For example, for building in timeline 222 Figure 2A The script builder 222 of the action script 202 in FIG. 1 shows a strip having four actions 210, 210a-d and lasting a total of forty strips. S Here, the first action 210 a Occupies twelve strips S , the second action 210b and the third action 210c each occupy eight stripes t S , and the fourth action 210d occupies twelve strips tS . When the user 10 wishes to execute the forty strips of action script 202 within forty seconds, the robot 100 allocates twelve seconds to each of the first action 210a and the fourth action 210d, and allocates eight seconds to each of the second action 210b and the third action 210c. Conversely, when the user 10 wishes to execute the forty strips of action 202 in twice the time of eighty seconds, the robot 100 scales the strips of action 210 so that the robot 100 allocates twenty-four seconds to each of the first action 210a and the fourth action 210d, and allocates sixteen seconds to each of the second action 210b and the third action 210c. Additionally or alternatively, since each action 210 is also executed within a set time period t having a start time and an end time, M The action script 202 may include unoccupied strips before, between, or after the action 210 as part of the motion routine.

[0037] In some implementations, UI 200 includes a list 212 of actions 210 that user 10 can select from list 212 to build action script 202. When user 10 selects from list 212 (e.g., the selection is in Figure 2B 210), selection of the action 210 may display parameters 214 specific to the action 210 (e.g., shown in simulator 230) that the user 10 may customize. In some examples, the user 10 adds the action 210 to the action script 202 by selecting it or by selecting and dragging the action 210 into the script builder 220. Some examples of selection include double-clicking or selecting the action 210 and clicking a GUI element, such as "Add" (e.g., as shown in FIG. 214). Figure 2B Here, the script builder 220 may be another panel in the same window as the action list 212, or a separate window.

[0038] When adding actions 210 to script builder 220 to build action script 202, actions 210 may have a hard-coded default size (e.g., number of stripes) based on the action 210. For example, by default, a single action 210 corresponding to a simple movement may occupy fewer stripes than a more complex movement (e.g., a handstand). S In either case, the default stripe t of action 210 S The amount may correspond to a minimum amount of time required for the robot 100 to successfully perform the action 210. In some implementations, the action 210 may then be further modified by adding the action 210 to the script builder 220. For example, the user 10 resizes the action 210 to extend the action 210 to a size larger than the default width of the action 210. S Number of strips tS In some examples, the script builder 220 is a script that includes a S ) of a timeline 222. Here, the user 10 imports the action(s) 210 into the timeline 222 at a strip position within the timeline 222 specified by the user 10 (e.g., by selection or a drag-and-drop process).

[0039] In some examples, such as Figure 2A , the timeline 222 includes more than one layer 224. Here, the layer 224 refers to the space in the timeline 222 that the action 210 can occupy when the action 210 affects a specific joint J assigned to the layer 224. In other words, the layer 224 defines the assignment of joints J of the robot 100. In some configurations, each joint J of the robot 100 is assigned to a specific layer 224 of the timeline 222. For example, the first layer 224, 224a corresponds to a leg joint, such as a knee joint J. K and hip joint J H . In some examples, one or more layers 224 do not have any joints J. An example of a layer 224 without any joints J may be when the layer 224 corresponds to a specific type of action 210, such as a body action 210b. In other words, there may be certain actions 210 that passively control joints J. In other words, when an action 210 is a modification to or input to the controller 172, the action 210 may affect joints J that have been specified or assigned to a specific layer 224. For example, in practical terms, movement of the body 110 occurs by changing at least one joint J of at least one leg 120 of the robot 100. However, when a joint J of a leg 120 has been assigned to a layer 224 (e.g., for a footstep action 210b), the body 110 may be moved by changing at least one joint J of at least one leg 120 of the robot 100. f or leg movements), the movement of the subject 110 may cause joint control interference between the two layers 224. In some examples, to prevent problems caused by joint interference, the UI 200 may b The subject motion 210 is transmitted as a suggested modification to a controller 172 of another layer 224 or a motion parameter of a particular controller 172. In other words, the subject motion 210 b is a secondary consideration that may affect joint control while it does not interfere with, for example, leg action. Here, an action 210 that can modify control of a joint J is referred to as a hint 210. H When prompted 210 H When it appears at a specific moment, prompt 210 H is considered by each other active controller 172 (eg, each time-current controller 172). When a controller 172 interprets the prompt 210 H When the controller 172 can determine the prompt 210 Hcorresponds to a joint J different from the joint J controlled by the controller 172, and ignores the prompt 210 H Modify its controller behavior. When prompted 210 H When corresponding to the same joint J, the controller 172 may H The motion or motion parameters 214 are incorporated into its own behavior.

[0040] In some examples, such as Figure 2B , the script builder 220 of the UI 200 allows the user 10 to have an action 210 on each layer 224 (e.g., shown as three layers 224, 224a-c) at each moment. When joints J are assigned to layers 224 to prevent interference, multiple actions 210 can be layered at the same moment on the timeline 222. In other words, through the layers 224, when the actions 210 only control certain parts of the robot 100, multiple actions 210 can be stacked on top of each other to participate in a wider range of dynamics of the robot 100. For example, prompt 210 H The designation of allows multiple actions 210 to be layered at the same moment on the timeline 222, as one layer 224 will be the primary consideration for control (e.g., leg actions), while another layer 224, such as layer 224 of body action 210b, 224b will be a secondary consideration for control. By having layers 224, actions 210 can be combined and / or ordered in various combinations to give the user 10 more options than the number of actions 210 available on a simple list 212. For example, Figure 2B The first action 210 on the first layer 224a (eg, the arm action layer) is shown. a , 210a, a second action 210 on a second layer 224b (eg, a main action layer) b , 210b, and a third action 210 on a third layer 224c (eg, a leg action layer) f , 210c.

[0041] Although the UI 200 allows hierarchical construction of actions 210, the μI 200 generally does not determine the feasibility of the combination of actions 210. Instead, as long as the hierarchical construction of actions 210 does not violate the rules at the μI 200 (e.g., at the script builder 220), the μI 200 transmits the action script 202 with the combination of layers 224 to the robot 100 (e.g., to the action system 300 of the robot 100). In some examples, the actions 210 are specific to a particular layer 224. More specifically, the UI 200 will not allow the user 10 to combine the main actions 210 b Placed in uncontrolled and subject action 210 b The layer 224 of the relevant joint J. For example, Figure 2CAs shown, the user 10 cannot place the swinging main body motion 210b affecting the joints J of one or more legs 120 on the joints J that only control the arm joints J. A In some implementations, when the user 10 adds an action 210 to the timeline 222 of the script builder 220, the action 210 is automatically populated into the layer(s) 224 associated with the joint J affected by the action 210 (e.g., the underlying motion required to control the action 210). Here, the user 10 controls the position of the action 210 on the timeline 222, not the layer 224 itself.

[0042] In some configurations, the action 210 is configured to operate on a joint J on more than one layer 224 of the timeline 222. In these configurations, the script builder 220 includes the action 210 on multiple layers 224 that occupy a default number of strips corresponding to the action 210. S Each layer 224 of the strip t S .For example, Figure 2D The second action 210b on the timeline 222 of the script builder 220 is shown to occupy twelve strips of the second layer 224, 224b simultaneously. S and the twelve strips t of the third layer 224, 224c S Here, the action 210 is shown as a single block so that the UI 200 prevents the user 10 from manipulating the same action 210 differently in different layers 224, which may cause an error when the robot 100 attempts to perform the action 210. Figure 2D , since the timeline 222 includes three layers 224a-c, another action 210 can be combined with the second action 210b by occupying the first layer 224a for a portion of the same time.

[0043] In some configurations, the UI 200 includes rules regarding the start state (i.e., the entry state 218s) and the end state (i.e., the exit state 218e) of the action 210. The UI 200 incorporates these rules to prevent possible errors when the robot 100 attempts the action script 202. For example, Figure 2D As shown, when the second action 210, 210b ends with the end state 218e of the robot 100 kneeling, the subsequent action 210 (for example, shown as the third action 210c) cannot assume that the robot 100 is immediately able to make the start state 218e based on the robot 100 standing up. S actions 210 (e.g., certain types of footstep actions 210 f ).

[0044] In order to overcome these potential problems between adjacent actions 210, each action 210 may include an entry state 218s and / or an exit state 218e. In the case where each action 210 includes an entry state 218s and / or an exit state 218e, the UI 200 (e.g., the script builder 220) may compare the transition states between two adjacent actions 210 (e.g., the exit state 218e of the first action 210a and the entry state 218s of the adjacent second action 210b) to determine whether these transition states are compatible. When these transition states are incompatible, the UI 200 communicates to the user 10 that the sequence of actions 210 is incompatible. In some examples, the UI 200 recognizes that a specific exit state 218e may be compatible with more than one entry state 218s.

[0045] Alternatively, UI 200 may make this determination when user 10 attempts to place or select adjacent action 210. Here, UI 200 does not allow user 10 to place or select action 210 when adjacent action 210 is incompatible. Figure 2D In the depicted example, the incompatible selected action 210c is grayed out and shown with an "X" to communicate to the user 10 that the action 210 is incompatible at that location in the timeline 222. In some implementations, the UI 200 displays a message at the UI 200 that communicates that the action 210 is incompatible with the location where the user 10 wishes to place the action 210 on the timeline 222. In some examples, the list 212 is dynamic, such that the list 212 displays actions 210 that are compatible with the previous selection. For example, the user 10 may select (i.e., click) an action 210 placed on the timeline 222, and the action list 212 displays compatible actions 210. In some examples, the UI 200 includes a selection by the user 10 to select the action 210 based on the entry state 218. s and / or exit status 218 e In addition or alternatively, when user 10 selects or attempts to place action 210 at an incompatible location on timeline 222, UI 200 may provide assistance to user 10. For example, UI 200 may suggest that user 10 add action 210 as a transition between incompatible actions 210 that are adjacent to each other in time on timeline 222. Here, UI 200 suggests that first action 210 be placed at the same location. a Exit status 218 e The second action 210b enters state 218 s Matching action 210.

[0046] In some examples, such as Figure 2E, the script builder 220 is scalable for use with more than one robot 100, 100a-b. For example, the script builder 220 includes a plurality of timelines 222, 222a-b, wherein each timeline 222 has a layer 224 specific to a robot 100. Figure 2E , the first timeline 222, 222a is for the first robot 100a, and the second timeline 222, 222b is for the second robot 100b. By coordinating the timelines 222 of multiple robots 100, the UI 200 can be used to generate coordinated movement routines between the robots 100. For example, the first robot 100a performs the first phase of the movement (multiple) 1 , pause, while the second robot 100b then performs the second phase of movement(s) p 2 , and then the first robot 100a performs the final third phase of the motion(s) together with the second robot 100b. In some implementations, the UI 200 can be compatible with different types of robots 100, such that a type or model of a robot 100 can be loaded with a robot profile specific profile. Here, the robot specific profile can load a timeline 222 with a plurality of layers 224 of a robot 100 specific to the robot specific profile. To illustrate, Figure 2E A first timeline 222a having three layers 224, 224a-c and a second timeline 222b having only two layers 224, 224a-b are depicted. The second timeline 224 has fewer layers 224 because the corresponding robot 100b does not include an arm 126, and thus a layer 224 corresponding to a joint J of the arm 126 is unnecessary. Figure 2E The timelines 222a-b are shown displayed in the same window, but each timeline 222 may have its own window (eg, the user 10 may switch between windows).

[0047] Return to reference Figure 2A-2E In some examples, the UI 200 also includes a simulator 230. The simulator 230 can be a main window of the UI 200 or a panel within its own window of the UI 200. In some implementations, the simulator 230 is configured to display a model 232 or avatar of the robot 100 as a visual simulation of the actions 210. For example, the user 10 selects one or more actions 210, and the simulator 230 displays a sample reproduction of the selected one or more actions 210. In some configurations, such as Figure 2D , the simulator 230 includes a GUI element 234 that allows the input or adjustment of a parameter 214 associated with the action 210. In these configurations, the simulator 230 can display the action 210 applying the parameter 214 input by the user 10. For example, in Figure 2DThe slider of the GUI element 234 displayed as the parameter 214 in the action 210 (eg, the main action 210) may be b ) during which the tilt or pitch of the main body 110 of the robot 100 is adjusted. Figure 2D A single set of parameters 214 is shown for selecting action 210n, but simulator 230 may be configured to display different GUI elements 234 corresponding to the parameters 214 of the selected action 210. For example, when user 10 selects the second action 210b, slider GUI element 234 may change to a simple input field.

[0048] In some configurations, the UI 200 is configured to transmit portions of the action script 202 to the robot 100. By transmitting portions of the action script 202, the user 10 can troubleshoot or debug sub-portions of the motion routine. For example, when the user 10 is frustrated that the action script 202 fails to execute successfully at the robot 100, the user 10 uses the UI 200 to select a portion of the action script 202 to communicate with the robot 100. In some implementations, the UI 200 requests the robot 100 to loop the action script 202 or a portion of the action script 202. Here, while the robot 100 loops the action script 202 or a portion thereof, the user 10 can update the action script 202 at the UI 200. In some configurations, the user 10 specifies where to loop the action script 202 in the timeline 222. For example, the user 10 places the loop widget 226 ( Figure 2E and 2F ) (eg, shown as an adjustable vertical bar) is inserted into the timeline 222 to specify a position in the action script 202 at which the motion routine should be repeated (eg, restarted). For illustration, Figure 2E The third stage of the action script 202a-b is shown. 3 The previous loop widget 226 allows the user 10 to 1 and the second stage p 2 Action 210, 210i-q performs troubleshooting.

[0049] In addition, the UI 200 may be configured to save and / or load action scripts 202 that have been built by the user 10 or are in the process of being built. For example, the UI 200 communicates with the computing resources 22, 24 of the user device 20 or the computing resources 162 of the remote system 160 to store or load a particular action script 202. In some examples, the UI 200 allows the user 10 to share the action script 202 with other users so that the action script 202 can be built collaboratively. In some implementations, the UI 200 includes a storage container (e.g., a file directory) that includes one or more action scripts 202 or a pointer to the location of one or more action scripts 202. Additionally or alternatively, the computing system 140 of the robot 100 may store the action script 202 (e.g., temporarily or for some extended period of time).

[0050] In some implementations, the action script 202 corresponds to a dance routine, where each action 210 is a dance move that the robot 100 can perform. Here, the time strip t S Can be used to coordinate with the tempo (ie, rate) of a particular song. For example, a song is identified (eg, by UI 200) as a particular number of beats per minute (BPM), and each strip t S The divisions correspond to a single beat b. Because the action 210 is scalable, the tempo of the song can be changed (eg, sped up or slowed down) or the actual song can be changed, and the execution of the action script 202 can adapt to the changed tempo.

[0051] In some implementations, script builder 220 includes a setting (e.g., GUI element 234) for setting timeline 222 to a particular tempo (e.g., BPM) to correspond to a particular song. For example, the setting may allow user 10 to map beat b of a song directly to action 210 or a portion of action 210. For example, Figure 2F Depicts footsteps 210 f Footsteps 210 on timeline 222 f The touchdown (TD) and / or liftoff (LO) positions are indicated in the box representation of . Using the TD / LO indication, the user 10 can align the TD or LO directly with the beat b of the song (or not align on the rising key). Figures 2A-2F In the example, timeline 222 passes through strip t S The darker bar between shows the beats of the song.

[0052] In some configurations, UI 200 (e.g., script builder 220) includes features that facilitate synchronization of action script 202 with audio from a song. Figure 2E, the UI 200 includes a slider 228 for the timeline 222. The slider 228 may be a time synchronization tool, wherein the user 10 may set the slider 228 to a custom position in the timeline 222. The position of the slider 228 may be used to indicate the start position at which the robot 100 should begin to execute a motion routine (e.g., a dance) of the action script 202. For example, Figure 2F Slider bar 228 is depicted prior to the first action 220a.

[0053] In some implementations, such as Figure 2F , the UI 200 sends the action script 202 as a configuration file to the robot 100, along with the current system time 204 of the UI 200 (e.g., a timestamp according to the clock of the user device 20) and the start time 206 at which the robot 100 should start the motion routine (e.g., as indicated by the position of the slider 228). Here, the robot 100, upon receiving the action script 202 with the system time 204 of the UI 200 and the start time 206 of the motion routine, will wait for the amount of time specified by the UI 200 and then start the motion routine (e.g., start dancing). In some examples, when the robot 100 receives the action script 202, the robot 100 assumes that the communication of the action script 202 between the UI 200 and the robot 100 has taken a certain amount of communication time, and synchronizes its own clock to take into account the amount of communication time. For example, the robot 100 synchronizes its own clock to the clock timestamp 204 received from the UI 200 plus an additional amount of the communication time. Based on this time synchronization, the robot 100 begins a motion routine (eg, a dance) at some future time (eg, at a time in the future equal to the difference between the system time 204 of the UI 200 and the specified start time 206).

[0054] Additionally or alternatively, the robot 100 provides feedback 208 to the UI 200 regarding the robot 100's perception of its temporal position in the movement routine (e.g., dancing). For example, the robot 100 provides its own clock time 208, 208a (e.g., in the form of a timestamp) Figure 3) as feedback 208. Here, the robot's time should be very similar to the system time 204 of the UI 200 because of the previous synchronization when the robot 100 received the action script 202. In some examples, when there is a difference between the system time 204 of the UI 200 and the clock time 208a of the robot 100, the UI 200 can resend its system time 204 to the robot 100 for resynchronization. In some implementations, when resynchronization occurs, the robot 100 can be configured to speed up or slow down the actions 210 of the action script 202. For example, the robot 100 identifies candidate actions 210 that the robot 100 can perform faster or slower to adjust the action routine 202 by resynchronization. In yet other examples, instead of feedback 208 from the robot 100, the UI 200 periodically transmits its current clock 204 to the robot 100 (e.g., at some specified frequency) so that the robot 100 checks the synchronization between the UI 200 and the robot 100. In these examples, the robot 100 performs resynchronization when periodic communications from the UI 200 indicate a time misalignment. In some configurations, the robot 100 resynchronizes when the difference between the system time 204 of the UI 200 and the clock 208a of the robot 100 meets a time offset threshold (e.g., exceeds a value set as the time offset threshold). This approach can prevent the robot 100 from frequently resynchronizing when there is a small time offset between the UI 200 and the robot 100.

[0055] refer to Figure 3 , the motion system 300 includes an interface 310 (such as an application programming interface (API)), a sequencer 320, and a dynamic planner 330. From an overall perspective, the goal of the motion system 300 is to receive a motion request (e.g., an action script 202 with (multiple) actions 210) as input and convert the input into a solution for generating motion for the robot 100. Here, even if the input is impossible (e.g., from a physics or robot dynamics perspective), the motion system 300 is configured to return a solution that is the closest possible solution for generating motion for the robot 100 based on the input. In other words, the motion system 300 is configured to determine the compliance or practical feasibility of the robot 100 to execute the actions 210 of the action script 202. This approach is particularly useful when a combination of actions 210 appear in the action script 202 at a given moment (i.e., the user 10 has hierarchical actions 210).

[0056] In some examples, the interface 310 serves as a means of communication between the UI 200 and the robot 100. In other words, the interface 310 is how the entire action script 202 is transferred from the UI 200 to the robot 100. In some implementations, the sequencer 320 interprets the action script 202 from the interface 310 to understand the timing of when the robot 100 should attempt to execute a particular action 210 of the action script 202. In some examples, the sequencer 320 operates as a scheduler of the actions 210 of the action script 202. In some configurations, the sequencer 320 operates by identifying the clock time 208a of the robot 100 and identifying the action(s) 210 that should be run at the identified time. For example, based on the clock 208a of the robot 100, the sequencer 320 identifies a particular stripe t of the action script 202. S , and determines whether the robot 100 should be in the time band t S Execute or attempt to execute action 210. When sequencer 320 identifies one or more actions 210 as being consistent with the identified time band t S When associated, the sequencer 320 may transmit (a plurality of) these actions 210 to the dynamic planner 330. For example, Figure 3 A time bar t having a clock symbol indicating the current time 208a of the robot 100 and an action script 202 corresponding to the current time 208a is depicted. S Here, three actions 210h-j occur in the time slice t S at (e.g., represented by a dashed box around the three actions 210h-j).

[0057] The motion system 300 is configured to generate commands 302 based on the (multiple) actions 210 of the action script 202 to control the motion of the robot 100 to move or attempt to move. The motion system 300 may use a dynamic planner 330 to generate all or part of the commands 302. In some examples, the dynamic planner 330 is configured to generate commands 302 for joints J of the robot 100 to control the motion of the robot (also referred to as joint commands 302). Here, the joint commands 302 may correspond to joint positions, joint forces (e.g., joint torques), joint angles, angular velocities associated with joints J, etc. When the controller 172 is activated by the action 210 at a particular moment, the controller 172 uses the dynamic planner 330 as a tool to determine the joint commands 302 (e.g., joint torques) to achieve or attempt to achieve the motion of the (multiple) actions 210. In an example where the robot 100 is a legged robot (e.g., a quadruped), the dynamic planner 330 is typically used to coordinate the leg actions 210 because leg motions are a common control type for the legged robot 100 (e.g., often stacked together). For the legged robot 100, the control system 170 may control the standing leg(s) 120 through force control (eg, joint torque). ST , and controlling the swing leg(s) 120 by position (e.g., using joint angles and angular velocities) SW .

[0058] To generate the joint commands 302, the dynamic planner 330 includes a solver 332. Here, the solver 332 refers to an optimization model that incorporates various constraints 334 (e.g., structural constraints, physical world constraints, user-defined constraints, etc.) of the robot 100 and / or the environment surrounding the robot 100. To coordinate the motion, the solver 332 of the dynamic planner 330 may receive or be pre-programmed to satisfy these constraints 334 so that the robot 100 can continue to function (e.g., maintain its balance) while performing the requested motion (e.g., requested by the action script 202). Some more specific examples of the constraints 334 include the range of motion constraints of the robot 100, constraints based on the structural dimensions of the robot 100 (e.g., leg length), estimates of friction on the robot 100 (e.g., at the feet 124 of the robot 100), etc.

[0059] To generate joint commands 302, dynamic planner 330 receives information about one or more actions 210, such as parameters 214, from motion script 202. In some examples, dynamic planner 330 identifies a cost 336 associated with an action 210 (e.g., each action 210 received at dynamic planner 330). Here, cost 336 refers to a value that specifies the importance of a motion or a particular portion of a motion for action 210. For example, cost 336 corresponds to a target event pose P of action 210.T The importance of the target event posture P T Typically refers to the desired position or orientation of the subject for a given motion. In some implementations, each action 210 received by the dynamic planner 330 includes a cost 336. Here, Figure 2A As shown, when constructing the action script 202, the user 10 can define the cost 336 as a parameter 214. In other words, the user 10 can define the cost 336 as a parameter 214 by assigning a value to the target event pose P of the action 210. T To define how critical a particular motion of the action 210 is.

[0060] By having costs 336 associated with actions 210 input into the solver 332, the solver 332 is configured to optimize multiple costs 336 to understand how to best perform more than one action 210 at a particular moment in the action script 202. In other words, when a first action 210a and a second action 210b occur at the same moment, each action 210a-b includes a cost 336, 336a-b that informs the solver 332 about what is important (e.g., to the user 10) for each action 210a-b. For example, the first action 210a is a footstep action 210b. f , where the user 10 does wish to achieve a specific contact position of the foot 124 during a step (e.g., during swing), but does not care about (or value) the speed of movement during the step or the foot movement 210 f Here, the user 10 can be a footstep action 210 f The cost 336 of the contact location is set to reflect this high importance (e.g., a large cost value). In this example, when the user 10 adds an additional action 210 (such as a swinging main action 210) b ) as a second motion 210b, the solver 332 may decide to reduce the sway amount of the second motion 210b so that it does not compromise the accuracy of the contact position of the first motion 210a. Here, the sway amount may have been designed as a parameter 214 at the UI 200. In this case, the second motion 210b may be configured with a cost 336b that reflects the lower importance of the sway amount. In this example, when each motion 210ab has a high cost 336 that causes a potential conflict, the solver 332 may attempt to optimize the motion result to resolve the conflict or compromise between the conflicting commands.

[0061] In some configurations, the solution is performed as a hint 210 before the dynamic planner 330 H In other words, the action system 300 and / or the control system 170 determines the prompt 210 HIs it compatible with other actions 210 occurring at the same time? In these configurations, when prompt 210 H With in and with tips 210 H When the actions (multiple) 210 occurring at the same time are compatible, the actions (multiple) 210 are modified to incorporate the prompt 210 H Movement or prompt 210 H Based on this process, the dynamic planner 330 may receive the hint 210 that has been provided by the solver 332. H Modified action 210.

[0062] In some examples, the controller 172 of the robot 100 performs the action 210 without using the dynamic planner 330. In other words, the solver 332 that generates the joint commands 302 is not necessary for a particular action 210. For example, a particular action 210 does not cause a joint control conflict. In other words, when the arm action 210 only affects the joint J of the arm 126, a and the leg motion 210 of the joint J which does not affect the arm 126 f At the same time, the arm movement 210 a The controller 172 may continue its own joint commands 302 to the arm 126 without utilizing the dynamic planner 330. In some examples, the motion 210 bypasses the dynamic planner 330 when optimization is not needed (e.g., when only one motion 210 occurs during the motion (multiple) of the motion 210 without potential joint interference). In some implementations, the motion system 300 is configured to not allow the motion 210 to bypass the dynamic planner 330, even for basic motions 210. In this approach, using the dynamic planner 330 even for basic motions 210 will help ensure that the robot 100 obeys real-world constraints (e.g., constraints 334) during the motion of the motion 210. This approach also enables the robot 100 to maintain balance and motion fluidity during motion.

[0063] Although not all motions 210 necessarily utilize the dynamic planner 330, in some implementations, bypassing the dynamic planner 330 is not based on using (multiple) controllers 172 with disjoint sets of joints J. Instead, in some examples, the motion system 300 may use trajectories (or "layers") to avoid situations where multiple controllers 172 attempt to command the same joint j. In some cases, many but not all controllers 172 are implemented using the dynamic planner 330. The dynamic planner 330 incorporates the hints 210. H To avoid parsing multiple controllers 172 that wish to command the same joint J.

[0064] In some implementations, the action 210 is a controller 172 or a prompt 210H Optionally, the action 210 may be a controller 172 and a prompt 210 H Both. The controller 172 directly controls the joint J of the robot 100. In the example shown, the controller 172 receives the prompt 210 H As a request to modify the behavior of the corresponding controller 172 in a particular way. A dance move (e.g., move 210) is assigned to one or more tracks (layers 224). Which track(s) a dance move is performed on is an inherent property of the type of that individual dance move, and is not something that the user 10 selects. For example, a "step" move might always be performed on the leg track, while a "sway" move might always be performed on the body track. The tracks that define their dance moves are the prompts 210. H Again, controller 172. In some cases, all leg and arm movements are controller 172, and all body movements are prompts 210. H If a dance move uses multiple tracks, if any of them is a controller track, it is the controller 172. The dynamic planner 330 is a tool used by many leg track controllers. It knows how to combine the cues 210 H to modify its behavior.

[0065] In some implementations, to generate the joint commands 302 for the joints J of the robot 100, the motion system 300 considers the current state 338 of the robot 100. Here, the current state 338 of the robot 100 may refer to any measurement related to the robot 100 or the surrounding environment of the robot 100. For example, the current state 338 includes sensor data 134 from the sensor system 130 regarding the time (e.g., the time identified by the sequencer 320) when the robot 100 attempts to perform the action 210. Some examples of the current state 338 include the position of the robot 100, the velocity of the robot 100, and / or the force that the robot 100 is experiencing. When the motion system 300 considers the current state 338, the motion system 300 (e.g., the dynamic planner 330) may update the constraints 334 based on the current state 338 to achieve accurate optimization of the solver 332.

[0066] Figure 4An example arrangement of operations of a method 400 for generating commands to control motion of a robot 100. In operation 402, the method 400 receives an action script 202, the action script 202 including a plurality of actions 210 for the legged robot 100 to perform. Here, each action 210 is associated with a cost 336. In operation 404, the method 400 identifies that two or more actions 210 of the plurality of actions 210 of the action script 202 occur at the same time. In operation 406, the method 400 determines a combined action performed by the legged robot 100 at the time based on the two or more actions 210 and the costs 336 associated with the two or more actions 210. In operation 408, the method 400 generates a joint command 302 to control motion of the legged robot 100 at the time, the joint command 302 commanding a set of joints J of the legged robot 100. Here, the set of joints J corresponds to the combined action.

[0067] In some examples, the plurality of actions 210 include a prompt corresponding to a body movement of a body 110 of the legged robot 100. Here, the method 400 determines whether the prompt is compatible with another action 210 in the plurality of actions 210 that occurs at the same time as the prompt, and when the prompt is compatible with the other action 210, modifies the other action 210 to incorporate the body movement of the prompt. In some implementations, receiving the action script 202 includes receiving the action script 202 from a user device 20 in communication with the data processing hardware. Here, the action script 202 is defined by a user 10 of the user device 20 at a user interface 200 running on the user device 20. In some configurations, when the action script 202 includes a dance script and each action 210 includes a dance move, the method 400 synchronizes each dance move of the dance script with the beat of a song.

[0068] Figure 5 Schematic diagram of an example computing device 500 that can be used to implement the systems (e.g., UI 200 and / or action system 300) and methods (e.g., method 400) described in this document. Computing device 500 is intended to represent various forms of digital computers, such as laptops, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown here, their connections and relationships, and their functions are merely exemplary and are not meant to limit the implementation of the inventions described and / or claimed in this document.

[0069] The computing device 500 includes a processor 510 (e.g., data processing hardware), a memory 520 (e.g., memory hardware), a storage device 530, a high-speed interface / controller 540 connected to the memory 520 and a high-speed expansion port 550, and a low-speed interface / controller 560 connected to a low-speed bus 570 and the storage device 530. Each of the components 510, 520, 530, 540, 550, and 560 are interconnected using various buses and can be mounted on a common motherboard or in other appropriate ways. The processor 510 can process instructions for running within the computing device 500 (including instructions stored in the memory 520 or on the storage device 530) to display graphical information of a graphical user interface (GUI) on an external input / output device (e.g., a display 480 connected to the high-speed interface 540). In other implementations, multiple processors and / or multiple buses, as well as multiple memories and multiple types of memories, can be used as appropriate. Furthermore, multiple computing devices 500 may be connected, with each device providing portions of the necessary operations (eg, as a server bank, a group of blade servers, or a multi-processor system).

[0070] Memory 520 stores information non-temporarily within computing device 500. Memory 520 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory unit(s). Non-temporary memory 520 may be a physical device used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by computing device 400. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0071] The storage device 530 can provide mass storage for the computing device 500. In some implementations, the storage device 530 is a computer-readable medium. In various implementations, the storage device 530 can be a floppy disk device, a hard disk device, an optical disk device or a tape device, a flash memory or other similar solid-state memory device, or an array of devices (including devices in a storage area network or other configurations). In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when performed, perform one or more methods (such as those described above). The information carrier is a computer or machine-readable medium, such as a memory 520, a storage device 530, or a memory on a processor 510.

[0072] The high-speed controller 540 manages bandwidth-intensive operations of the computing device 500, while the low-speed controller 560 manages less bandwidth-intensive operations. This division of responsibilities is exemplary only. In some implementations, the high-speed controller 540 is connected to the memory 520, the display 580 (e.g., through a graphics processor or accelerator), and to a high-speed expansion port 550 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 560 is connected to the storage device 530 and the low-speed expansion port 570. The low-speed expansion port 570, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be connected to one or more input / output devices (such as keyboards, pointing devices, scanners) or network devices (such as switches or routers) (e.g., through a network adapter).

[0073] As shown, computing device 500 can be implemented in a variety of different forms. For example, it can be implemented as a standard server 500a, or multiple times implemented as a laptop 500b in a group of such servers 500a, implemented as part of a rack server system 500c, or implemented as part of a robot 100.

[0074] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system that includes at least one programmable processor that may be either special-purpose or general-purpose. The programmable processor is connected to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to the storage system, at least one input device, and at least one output device.

[0075] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0076] The process and logic flow described in this specification can be performed by one or more programmable processors, which run one or more computer programs to perform functions by operating on input data and generating outputs. These processes and logic flows can also be performed by special logic circuits (e.g., FPGA (field programmable gate array) or ASIC (application-specific integrated circuit)). For example, processors suitable for running computer programs include both general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include or be operably connected to one or more large-capacity storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, to receive data from it or to transmit data to it, or both. However, the computer does not need to have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0077] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen for displaying information to the user, and an optional keyboard and pointing device (such as a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending files to and receiving files from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0078] Many implementations have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Therefore, other implementations are within the scope of the following claims.

Claims

1. A method (400) for generating joint commands, include: At data processing hardware, receiving an action script (202), the action script comprising a plurality of actions for execution by a legged robot (100), each action of the plurality of actions being associated with a cost (336); Recognizing by the data processing hardware that a first action and a second action of a plurality of actions in the action script (202) occur at the same time; determining, by the data processing hardware, a conflict between the first action and the second action, wherein the first action and the second action both command a first joint; determining, by the data processing hardware, a combined action performed by the legged robot (100) at the moment based on the first action and the second action; and The data processing hardware generates a joint command (302) based on the current state of the legged robot to control the movement of the legged robot (100) at the moment, the joint command (302) commands a group of joints (J) of the legged robot (100) including a first joint and a second joint, the group of joints (J) corresponding to the combined action, and the joint command controls the movement of the first joint to implement at least a part of the combined action.

2. The method (400) according to claim 1, in, The legged robot (100) corresponds to a quadruped robot.

3. The method (400) of claim 1, further comprising: include: determining, by the data processing hardware, the cue corresponding to a body movement of a body (110) of the legged robot (100) when the cue is compatible with the second motion; and When the cue is compatible with the second motion, the second motion is modified by the data processing hardware to incorporate the cue's body movement.

4. The method (400) according to claim 1, in, Each of the plurality of actions is configured to use an active controller (172) for the legged robot (100) or to modify an active controller (172) for the legged robot (100).

5. The method (400) according to claim 1, in, The cost (336) associated with each of the plurality of actions includes a user-defined cost (336) indicating how important it is for the legged robot (100) to perform the action.

6. The method (400) according to claim 1, in: Receiving the action script (202) includes receiving the action script (202) from a user device (20) in communication with the data processing hardware; and The action script (202) is defined by a user (10) of the user device (20) at a user interface (200) executed on the user device (20).

7. The method (400) according to claim 1, in, At least one of the multiple actions includes a footstep action, which includes the position and time at which the swinging leg of the legged robot (100) touches the ground or lifts off.

8. The method (400) according to claim 1, in, At least one of the plurality of actions comprises an arm action, wherein the arm action comprises a posture of the manipulator (126).

9. The method (400) according to claim 1, in, The action script (202) includes a dance script, and each of the plurality of actions includes a dance move.

10. The method (400) according to claim 9, further comprising: include: Each dance move of the dance script is synchronized with the beat of the song by the data processing hardware.

11. The method (400) of claim 1, further comprising: include: A determination is made, by the data processing hardware, that an exit state (218e) of one of the plurality of actions matches an entry state (218s) of a subsequent action in the plurality of actions.

12. The method according to claim 1, in, The current state of the legged robot includes sensor data captured from a sensor system of the legged robot.

13. The method according to claim 1, in, The current state of the legged robot includes at least one of a velocity of the legged robot or a force the legged robot is experiencing.

14. The method according to claim 1, in, The action script is received from a user interface, and the method further includes providing a clock time of the legged robot as feedback to the user interface.

15. The method according to claim 1, in, Modifying the behavior of at least one of the first controller or the second controller includes incorporating motion parameters of the third action into the behavior.

16. The method according to claim 1, in, The first action includes one or more parameters that can be modified to generate a set amount of change in the motion, and the combined action includes a modification of at least one of the one or more parameters of the first action.

17. The method according to claim 1, in, Determining the combined action of the robot includes a compromise between the first action and the second action.

18. A robot (100), include: Subject (110); two or more legs (120) connected to the body (110); and A computing system (170) in communication with the body (110) and the two or more legs (120), the computing system (170) comprising data processing hardware (510) and memory hardware (520) in communication with the data processing hardware (510), the memory hardware (520) storing instructions that, when executed on the data processing hardware (510), cause the data processing hardware (510) to perform operations including: receiving an action script (202), the action script (202) comprising a plurality of actions for the robot (100) to perform, each action of the plurality of actions being associated with a cost (336); Identify that a first action and a second action of a plurality of actions of the action script (202) occur at the same time; determining a conflict between the first action and the second action, wherein the first action and the second action both command a first joint; determining a combined action performed by the robot (100) at the moment based on the first action and the modified second action; and A joint command (302) is generated based on the current state of the robot to control the movement of the robot (100) at the moment, the joint command (302) commands a group of joints (J) of the legged robot (100) including a first joint and a second joint, the group of joints (J) corresponding to the combined action, and the joint command controls the movement of the first joint to implement at least a part of the combined action.

19. The robot (100) according to claim 18, in, The robot (100) corresponds to a quadruped robot.

20. The robot (100) according to claim 18, in, The operations also include: When the cue is compatible with the second motion, determining the cue corresponding to a body movement of a body (110) of the robot (100); and When the cue is compatible with the second motion, the second motion is modified to incorporate the subject movement of the cue.

21. The robot (100) according to claim 18, in, Each of the plurality of actions is configured to use an active controller (172) for the robot (100) or to modify an active controller (172) for the robot (100).

22. The robot (100) according to claim 18, in, The cost (336) associated with each of the plurality of actions includes a user-defined cost (336) indicating how important it is for the robot (100) to perform the action.

23. The robot (100) according to claim 18, in, The operation of receiving the action script (202) includes receiving the action script (202) from a user device (20) in communication with the data processing hardware, The action script (202) is defined by a user (10) of the user device (20) at a user interface (200) executed on the user device (20).

24. The robot (100) according to claim 18, in, At least one of the plurality of actions comprises a footstep action, wherein the footstep action comprises a position and a time at which a swinging leg of the robot (100) touches the ground or is lifted off.

25. The robot (100) according to claim 18, in, At least one of the plurality of actions comprises an arm action, wherein the arm action comprises a posture of the manipulator (126).

26. The robot (100) according to claim 18, in, The action script (202) includes a dance script, and each of the plurality of actions includes a dance move.

27. The robot (100) according to claim 26, in, The operation also includes synchronizing each dance move of the dance script with the beat of the song.

28. The robot (100) according to claim 18, in, The operations also include determining that an exit state (218e) of one of the plurality of actions complies with an entry state of a subsequent action in the plurality of actions.

29. The robot (100) of claim 18, further comprising a sensor system configured to capture data, wherein the current state comprises sensor data.