Robot Choreographer

Abstract

To provide the ability to program the robot for a variety of motion routines in a quick and efficient manner. SOLUTION: A method for generating joint instructions (302) includes receiving a strategy script (202) including a plurality of strategies (210) for a walking robot (100) to perform, each strategy (210) associated with a cost (336). The method further includes identifying that two or more of the strategies in the strategy script occur at the same time. The method also includes determining a combined strategy for the walking robot to perform at the same time based on the two or more strategies and costs associated with the two or more strategies. The method additionally includes generating joint instructions for controlling movement of the walking robot at times when the joint instructions command a set of joints (J) of the walking robot, where the set of joints corresponds to the combined strategy.

Description

[Technical field]

[0001] Technical Field

[0001] The present disclosure relates to robotic choreography. [Background technology]

[0002] Background technology

[0002] A robot is generally defined as a reprogrammable and multi-functional manipulator designed to move materials, parts, tools or specialized devices through variable programmed motions for the performance of a task. A robot can be a physically fixed manipulator (e.g., an industrial robot arm), a mobile robot that moves throughout the environment (e.g., by using legs, wheels or traction-based mechanisms), or some combination of manipulators and mobile robots. Robots are utilized in various industries, including, for example, manufacturing, transportation, hazardous environments, exploration and healthcare. Thus, the ability to program robots in a fast and efficient manner for various motion routines provides additional benefits to such industries. Summary of the Invention [Means for solving the problem]

[0003] Summary of the Invention

[0003] One aspect of the present disclosure provides a method for generating joint instructions. The method includes receiving, at the data processing hardware, a strategy script including a plurality of strategies for a legged robot to perform, where each strategy is associated with a cost. The method further includes identifying, by the data processing hardware, that two or more of the strategies of the strategy script occur at the same time. The method also includes determining, by the data processing hardware, a composite strategy for the legged robot to perform at the same time based on the two or more strategies and costs associated with the two or more strategies. The method additionally includes generating, by the data processing hardware, joint instructions for controlling movement of the legged robot at times when the joint instructions command a set of joints of the legged robot, where the set of joints corresponds to the composite strategy.

[0004]

[0004] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, the walking robot corresponds to a four-legged robot. In some examples, one of the multiple strategies includes a hint corresponding to a body motion of a body of the walking robot. In these examples, the method further includes determining, by the data processing hardware, whether the hint fits another strategy of the multiple strategies occurring at the same time as the hint, and modifying, by the data processing hardware, the other strategy to incorporate the body motion of the hint if the hint fits the other strategy. In some configurations, each strategy is configured to use or modify an active controller of the walking robot, and the strategy of modifying the active controller of the walking robot defines the hint. Optionally, a cost associated with each strategy includes a user-defined cost indicating an importance for the walking robot to perform the strategy.

[0005]

[0005] In some examples, at least one of the multiple strategies includes a footstep strategy including a location and time of landing of a swing leg of the walking robot. In some implementations, at least one of the multiple strategies includes an arm strategy including a pose of a manipulator of the walking robot. In some configurations, the strategy script includes a dance script, and each strategy includes a dance movement. Here, the method further includes synchronizing each dance movement of the dance script with a beat of the song by the data processing hardware. Additionally or alternatively, the method may also include determining, by the data processing hardware, that an output state of a first strategy of the multiple strategies of the strategy script follows an input state of a subsequent strategy.

[0006]

[0006] Another aspect of the present disclosure provides a robot capable of generating joint instructions. The robot includes a body, two or more legs coupled to the body, and a computing system in communication with the body and the two or more legs. The computing system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving a strategy script including a plurality of strategies for the robot to perform, where each strategy is associated with a cost. The operations further include identifying that two or more strategies of the plurality of strategies of the strategy script occur at the same time. The operations also include determining a composite strategy for the walking robot to perform at the same time based on the two or more strategies and the costs associated with the two or more strategies. The operations additionally include generating joint instructions for controlling a movement of the walking robot at a time when the joint instructions command a set of joints of the walking robot, where the set of joints corresponds to the composite strategy.

[0007]

[0007] This aspect may include one or more of the following optional features: In some implementations, the walking robot corresponds to a quadruped robot. In some examples, one of the multiple strategies includes a hint corresponding to a body motion of a body of the robot. In these examples, the manipulation further includes determining whether the hint fits another strategy of the multiple strategies occurring at the same time as the hint, and modifying the other strategy to incorporate the body motion of the hint if the hint fits the other strategy. In some configurations, each strategy is configured to use or modify an active controller of the robot, and the strategy that modifies the active controller of the robot defines the hint. Optionally, a cost associated with each strategy includes a user-defined cost indicating an importance for the robot to perform the strategy.

[0008]

[0008] In some examples, at least one of the multiple strategies includes a footstep strategy including a location and time of landing of a swing leg of the robot. In some implementations, at least one of the multiple strategies includes an arm strategy including a pose of a manipulator of the robot. In some configurations, the strategy script includes a dance script, and each strategy includes a dance movement. Here, the operations further include synchronizing each dance movement of the dance script with a beat of the song. Additionally or alternatively, the operations may also include determining that an output state of a first strategy of the multiple strategies of the strategy script follows an input state of a subsequent strategy.

[0009]

[0009] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the following specification. Other aspects, features, and advantages will become apparent from the specification and the accompanying drawings, and the claims. [Brief description of the drawings]

[0010] BRIEF DESCRIPTION OF THE DRAWINGS [Figure 1A]

[0010] FIG. 1 is a perspective view of an exemplary robot in a robotic environment. [Figure 1B]

[0011] FIG. 1B is a schematic diagram of an exemplary system of the robot of FIG. 1A. [Figure 1C] FIG. 1B is a schematic diagram of an exemplary system of the robot of FIG. 1A. [Figure 2A]

[0012] FIG. 1B is a schematic diagram of an exemplary user interface for interacting with the robot of FIG. 1A. [Figure 2B] FIG. 1B is a schematic diagram of an exemplary user interface for interacting with the robot of FIG. 1A. [Figure 2C] FIG. 1B is a schematic diagram of an exemplary user interface for interacting with the robot of FIG. 1A. [Figure 2D] FIG. 1B is a schematic diagram of an exemplary user interface for interacting with the robot of FIG. 1A. [Figure 2E] FIG. 1B is a schematic diagram of an exemplary user interface for interacting with the robot of FIG. 1A. [Figure 2F] FIG. 1B is a schematic diagram of an exemplary user interface for interacting with the robot of FIG. 1A. [Diagram 3]

[0013] FIG. 1B is a schematic diagram of an exemplary strategy system for the robot of FIG. 1A. [Figure 4]

[0014] 1 is an exemplary arrangement of operations for a robot to carry out a strategy. [Diagram 5]

[0015] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0011]

[0016] Like reference symbols in the various drawings indicate like elements. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] An operator who owns or controls a robot often wants to pre-program the robot's motions. For example, the operator wants the robot to perform repeatable motion routines. Unfortunately, pre-programmed motions may require hardcoding to generate the desired motions. Here, hardcoding refers to someone, such as a programmer, creating or editing a set of instructions written in a computer programming language. Hardcoding is generally time consuming and may therefore create a bottleneck such that only a limited number of people (e.g., programmers / coders) may be able to code the robot's motions. Furthermore, modifications to the coded motions or debugging the coded motions may create long feedback loops to implement the motion routines. In other words, robots often lack the means to quickly write robot behaviors. In addition, robot manufacturers may be prevented from the ability to hardcode new robot motions and / or edit already hardcoded code for existing robot motions.

[0013]

[0018] Another potential challenge with pre-programmed motion is that it may or may not take into account the environment around the robot and / or the physics of the robot itself. In other words, the person coding the pre-programmed motion may not realize that there may be other constraints on the robot during the pre-programmed motion. Without considering these other constraints, the pre-programmed routine may fail during execution, or may damage the robot during execution, or may damage the environment around the robot. For example, the coder of the pre-programmed motion routine may not realize that the robot will carry an additional 25 lbs of load during the routine, or that the routine may occur in a dynamic environment with static or dynamic obstacles. This clearly places a heavy burden on the operator or coder to consider all or a majority of the conditions that the robot will experience during the motion routine in order to achieve a successful motion routine.

[0014]

[0019] To overcome some of these challenges, referring to FIG. 1A, in a user interface (UI) 200 of the robot 100, the user 10 designs a motion routine. The robot 100 includes a strategy system 300 that communicates with the UI 200. Here, the UI 200 enables the user 10 of the robot 100 to generate a motion routine in the form of a strategy script 202. The strategy script 202 generally refers to multiple strategies 210, 210a-n (e.g., potentially overlapping in time) that the user 10 designs in the UI 200 for the robot 100 to perform within the robot environment 30. Here, the UI 200 allows the robot 100 to build motion without requiring any new hard coding. In other words, the user 10 uses the UI 200 to retrieve the robot 100's existing strategies 210 that can be layered or composited to generate the robot 100's motion routine in an intuitive and near real-time manner. Furthermore, some strategies 210 of the UI 200 are parameterized such that the user 10 may customize aspects of these strategies 210 in the UI 200 to suit his / her desired exercise routine. As will be apparent, the strategies 210 obtained and selected by the user 10 for the exercise routine may themselves be pre-programmed or hard-coded into the robot (e.g., by the manufacturer of the robot 100), but the composition and sequence of the strategies 210 in the strategy script 202 allows the robot 100 to perform / execute the exercise routine without requiring any hard-coding of the exercise routine by the user 10.

[0015]

[0020] In some examples, the user 10 executes the UI 200 on the user device 20. Here, the user device 20 executes the UI 200 by using computational resources 22, 24, such as data processing hardware 22 and memory hardware 24 in communication with the data processing hardware 22. Some examples of the user device 20 include a mobile device (e.g., a mobile phone, a tablet, a laptop, etc.), a personal computer (PC), a workstation, a terminal, or any other computing device having computational capabilities to execute the UI 200. The UI 200 can be web-based (e.g., a web-based application launched from a web browser), local to the user device 20 (e.g., configured and / or located to execute on the computational resources 22, 24 of the user device 20), or some combination of both. In some implementations, the user device 20 uses the UI 200 to communicate the strategy script 202 to the robot 100 by using a wired or wireless connection. For example, the UI 200 communicates the strategy script 202 to the robot 100 over a network (e.g., the network 150 shown in FIG. 1B).

[0016]

[0021] 1A , the robot 100 includes a body 110 having a motive base structure, such as legs 120a-d, coupled to the body 110 that allows the robot 100 to move about the environment 30. In some examples, each leg 120 is an articulatable structure such that one or more joints J allow the members 122 of the leg 120 to move. For example, each leg 120 may include an upper member 122, 122 of the leg 120, and an upper member 122 of the leg 120. U to the main body 110. H and the upper member 122 of the leg 120 U The lower member 122 of the leg 120 L Knee joint J K 1A depicts a quadruped robot having four legs 120a-d, the robot 100 may include any number of legs or motive-based structures (e.g., a bipedal robot or a humanoid robot having two legs) that provide a means for traversing terrain within the environment 30.

[0017]

[0022] To traverse a terrain, each leg 120 has a distal end 124 (i.e., a traction surface) that contacts the surface of the terrain. In other words, the distal end 124 of the leg 120 is the end of the leg 120 that is used by the robot 100 to turn, stand, or generally provide traction during movement of the robot 100. For example, the distal end 124 of the leg 120 may correspond to a foot of the robot 100. In some examples, not shown, the distal end 124 of the leg 120 may be disposed between the lower member 122 of the leg 120 and the distal end 124 of the leg 120. L The ankle joint J can be articulated to A Includes.

[0018]

[0023] In some examples, the robot 100 includes an arm 126 that functions as a robotic manipulator. The arm 126 may be configured to move about with multiple degrees of freedom to engage with elements of the environment 30 (e.g., objects in the environment 30) or to perform gestures (e.g., aesthetic gestures). In some examples, the arm 126 includes one or more members 128 that are coupled by a joint J such that the arm 126 can pivot or rotate about the joint J. For example, with two or more members 128, the arm 126 may be configured to extend or retract. To illustrate an example, FIG. 1A illustrates a lower member 128. L , upper member 128 U and hand parts 128 H 1 depicts an arm 126 having three members 128 corresponding to the lower member 128. L is a first arm joint J adjacent to the main body 110 A1 (e.g., where the arm 126 connects to the body 110 of the robot 100). L is the second arm joint J A2 In the upper member 128 U , and upper member 128 U is the third arm joint J A 3 Hand parts 128 H In some examples, the hand member 128 HThe hand includes additional members to allow for various types of gripping. These additional members include a simple two-part claw member 128 H More complex hand components that mimic the fingers of a human hand 128 H In some implementations, the arm 126 connects to the robot 100 at a socket on the body 110 of the robot 100. In some configurations, the socket is configured as a connector such that the arm 126 can be attached and detached to the robot 100 depending on whether the arm 126 is needed for manipulation.

[0019]

[0024] The robot 100 is configured to move along a vertical gravity axis (e.g., a z-axis A Z , denoted as θ and a center of gravity CM, which is a position corresponding to the average position of all parts of the robot 100 (where the parts are weighted according to their mass) (i.e., the point where the weighted relative position of the distributed mass of the robot 100 is zero). The robot 100 further has a vertical gravitational axis A to define a particular pose or stance that may be assumed by the robot 100. Z (i.e., a fixed reference frame relative to gravity). The posture of the robot 100 may be defined by the orientation or angular position of the robot 100 in space. Movement by the legs 120 relative to the body 110 modifies the pose P of the robot 100 (i.e., the combination of the location of the robot's CMs and the posture or orientation of the robot 100). Here, height generally refers to distance along the z direction. The sagittal plane of the robot 100 is aligned along the y direction axis A. Y and z-axis A Z In other words, the sagittal plane bisects the robot 100 into a left side and a right side. X and y-axis A YThe ground plane spans the XY plane by extending in the x-direction axis A. The ground plane refers to the ground surface 12 on which the distal ends 124 of the legs 120 of the robot 100 may generate traction to help the robot 100 move about the environment 30. Another anatomical plane of the robot 100 is the front, which spans the entire body 110 of the robot 100 (e.g., from the left side of the robot 100 with the first leg 120a to the right side of the robot 100 with the second leg 120b). The front is aligned along the x-direction axis A. X and z-axis A z The film spans the XZ plane by stretching in the direction of

[0020]

[0025] When the walking robot moves about the environment 30, each leg 120 of the robot 100 may be either in contact with the ground 12 or not in contact with the ground 12. When the leg 120 is in contact with the ground 12, the leg 120 is called a stance leg 120. ST When the leg 120 is not in contact with the ground 12, the leg 120 is called a swing leg 120. SW When the leg 120 lifts off the ground 12 (LO: lift-off), the leg 120 ST From free leg 120 SW Conversely, the swing leg transitions to 120 SW Also, when the swing leg lands on the ground 12 after not being in contact with the ground 12 (TD: touch down), the stance leg 120 ST Here, the leg 120 may transition to the swing leg 120 SW While the robot 100 functions as a standing leg 120, the other leg 120 of the robot 100 functions as a standing leg 120. ST (e.g., to maintain balance of the robot 100).

[0021]

[0026] To execute a strategy in the environment 30, the robot 100 includes a sensor system 130 having one or more sensors 132, 132a-n (e.g., shown as a first sensor 132, 132a and a second sensor 132, 132b). The sensors 132 may include vision / image sensors, inertial sensors (e.g., an inertial measurement unit (IMU)), force sensors, and / or kinematic sensors. Some examples of the vision / image sensors 132 include cameras, such as stereo cameras, scanning light-detection and ranging (LIDAR) sensors, or scanning laser-detection and ranging (LADAR) sensors. In some examples, the sensors 132 may have a corresponding field of view F that defines a sensing range or area corresponding to the sensors 132. v For example, FIG. 1A shows the field of view F of the robot 100. v Each sensor 132 depicts a field of view F, which may be centered, for example, around one or more axes (e.g., an x-axis, a y-axis, or a z-axis relative to a ground plane). v may be pivotable and / or rotatable so that the

[0022]

[0027] The sensor 132 measures the field of view F v When searching, the sensor system 130 has a field of view F vThe sensor system 130 generates sensor data 134 (also referred to as image data) corresponding to the robot 100. In some examples, the sensor data 134 is image data corresponding to a three-dimensional volumetric point cloud generated by the three-dimensional volumetric image sensor 132. Additionally or alternatively, the sensor system 130 collects pose data of the robot 100, including inertial measurement data (e.g., measured by an IMU) as the robot 100 performs a maneuver in the environment 30. In some examples, the pose data includes kinematic and / or orientation data about the robot 100 (e.g., kinematic and / or orientation data about the joints J of the legs 120 or other parts of the robot 100). The sensor data 134 may allow various systems of the robot 100 to use the sensor data 134 to define a current state of the robot 100 (e.g., of the kinematics of the robot 100) and / or a current state of the environment 30 around the robot 100.

[0023]

[0028] In some implementations, the sensor system 130 includes sensors 132 coupled to joints J. In some examples, these sensors 132 (e.g., sensors 132, 132a-b) are coupled to motors M that operate the joints J of the robot 100. Here, these sensors 132 generate joint dynamics in the form of joint-based sensor data 134. The joint dynamics collected as the joint-based sensor data 134 include joint angles (e.g., the joint angles of the lower member 122, L Upper member 122 for U ), joint velocities (e.g., joint angular velocities or joint angular accelerations), and / or forces experienced at the joints J (also referred to as joint forces). Here, the joint-based sensor data 134 generated by one or more sensors 132 may include raw sensor data, various types of joint dynamics 134 JD , which may be further processed to form a joint J, or some combination of both. For example, the sensor 132 may measure the joint position (or the position of the member 122 articulating at the joint J), and the system of the robot 100 may perform further processing to derive velocity and / or acceleration from the position data. In other examples, the sensor 132 may be configured to directly measure velocity and / or acceleration.

[0024]

[0029] As the sensor system 130 collects the sensor data 134, the computing system 140 is configured to store, process, and / or communicate the sensor data 134 to various systems of the robot 100 (e.g., the control system 170 and / or the strategy system 300). To perform computational tasks related to the sensor data 134, the computing system 140 of the robot 100 includes data processing hardware 142 and memory hardware 144. The data processing hardware 142 is configured to execute instructions stored in the memory hardware 144 to perform computational tasks related to activities (e.g., movement and / or movement-based activities) of the robot 100. Generally speaking, the computing system 140 refers to one or more locations of the data processing hardware 142 and / or the memory hardware 144.

[0025]

[0030] In some examples, the computing system 140 is a local system located on the robot 100. When located on the robot 100, the computing system 140 may be centralized (i.e., in a single location / area on the robot 100 (e.g., the body 110 of the robot 100)), or distributed (i.e., located at various locations around the robot 100), or a hybrid combination of both (e.g., a majority of centralized hardware and a minority of distributed hardware). To illustrate some differences, a distributed computing system 140 may allow processing to occur at the location of activity (e.g., in the motors that move the joints of the legs 120), while a centralized computing system 140 may allow a central processing hub that communicates with systems located at various locations on the robot 100 (e.g., that communicates with the motors that move the joints of the legs 120).

[0026]

[0031] Additionally or alternatively, the computing system 140 includes computing resources remote from the robot 100. For example, the computing system 140 communicates with a remote system 160 (e.g., a remote server or a cloud-based environment) via a network 150. Just like the computing system 140, the remote system 160 includes remote computing resources such as remote data processing hardware 162 and remote memory hardware 164. Here, the sensor data 134 or other processed data (e.g., data processed locally by the computing system 140) may be stored in the remote system 160 and may be accessible to the computing system 140. In some examples, the computing system 140 is configured to utilize the remote resources 162, 164 as an extension of the computing resources 142, 144 such that the resources of the computing system 140 may reside on the resources of the remote system 160. In some configurations, the remote system 160 and / or the computing system 140 host the UI 200 such that the user 10 accesses the UI 200 over the network 150 by using a web browser application.

[0027]

[0032] 1A-1C, the robot 100 includes a control system 170. The control system 170 may be configured to communicate with at least one sensor system 130 of the robot 100 and / or systems such as the strategy system 300. The control system 170 may perform some operations and other functions by using the hardware 140. The control system 170 includes at least one controller 172 configured to control the robot 100. For example, the controller 172 controls the movement of the robot 100 across the environment 30 based on input or feedback from systems of the robot 100 (e.g., the sensor system 130, the control system 170, and / or the strategy system 300). In some implementations, the controller 172 controls the movement of the robot 100 between poses and / or behaviors.

[0028]

[0033] In some examples, the controller 172 controls the robot 100 by controlling the motion about one or more joints J of the robot 100. In some configurations, the controller 172 is software having programming logic to control at least one joint J or a motor M that operates or is coupled to a joint J. For example, the controller 172 controls the amount of force (e.g., torque at a joint J) applied to a joint J. As a programmable controller 172, the number of joints J that the controller 172 controls is scalable and / or customizable for a particular control purpose. The controller 172 may control a single joint J of the robot 100 (e.g., control the torque at a single joint J) or may control multiple joints J. When controlling multiple joints J, the controller 172 may apply the same or different torques to each joint J controlled by the controller 172. By controlling multiple joints J, the controller 172 may coordinate the motion of a larger structure of the robot 100 (e.g., the body 110, one or more legs 120, arms 126). In other words, for some strategies 210, the controller 172 may be configured to control the motion of multiple parts of the robot 100 (e.g., two legs 120a-b, four legs 120a-d, or two legs 120a-b combined with an arm 126, etc.). For example, FIG. 1C depicts an example relationship between the motors M and joints J of the robot 100 that these motors control for a given configuration of the robot 100. Here, FIG. 1C shows four controllers 172, 172a-d set up to control several parts of the robot 100. The first controller 172, 172a controls the knee joint J associated with the first leg 120, 120a and the second leg 120, 120b. K and hip joint J H The second controller 172b controls the motor M of the knee joint J associated with a single leg, such as the third leg 120, 120c. K and hip joint J H The third controller 172c is shown configured to control the motors M associated with the third leg 120c and the fourth leg 120, 120d, and the arm joints J.A1 The fourth controller 172d is shown configured to control one of the motors M associated with only the arm 126 of the robot 100. For practical comparison, the third controller 172c may allow the robot 100 to lean to the side of the robot's body 110 on the third leg 120c and the fourth leg 120d while grasping something with the arm 180. In contrast to the third controller 172c, the robot 100 may perform leaning and grasping functions with three controllers 172 where two controllers 172 individually control the third and fourth legs 120c-d while the fourth controller 172d controls the arm 126. There may be several advantages or reasons for coordinating multiple controllers 172 (e.g., for fine motor control) or using a single controller 172 (e.g., for simplicity).

[0029]

[0034] 2A-2F, the UI 200 is configured to enable the user 10 to build an exercise routine in the form of a strategy script 202. In some examples, the strategy script 202 is a computer-readable format that the user 10 communicates with the robot 100 (e.g., via the UI 200) to enable the robot 100 to perform the exercise routine. In some implementations, the strategy script 202 is a series of movements over a set time period t that the user 10 requests the robot 100 to perform. To build the strategy script 202, the UI 200 provides multiple strategies 210a-n for the user 10 to select from. Here, the strategies 210 refer to the hard-coded movements of the robot 100 coordinated by the controller 172 of the control system 170. The strategies 210 may correspond to the controller 172 of the robot 100 (e.g., a programmed active controller for the control system 170), modifications to the controller 172 of the robot 100, one or more input parameters 214 of the controller 172 of the robot 100, or any combination thereof. In some examples, a strategy 210 defines a set of joints J required to perform the fundamental movement associated with the strategy 210 (e.g., by specifying the controller 172 to control one or more particular joints J). Some different types of strategies 210 include: footstep strategies (or leg strategies) 210, 210 that move the feet 124 of the robot 100. f , arm strategy 210, 210 for moving arm 126 of robot 100 a , and / or a body strategy 210, 210 for moving the body 110 of the robot 100 b Moreover, because the strategy 210 may correspond to the controller 172, the movements of the strategy 210 may correspond to any number of joints J, and may even correspond to joints J from different parts of the robot 100 (e.g., from two legs 120 on different sides of the robot's body 110). In other words, the strategy 210 may be configured to be a hybrid of different types of strategies. For example, the strategy 210 may be configured to be a footstep strategy 210. f and Arm Strategy 210 aIn some configurations, the strategy 210 is parameterized and hard-coded with parameters 214 that can be modified to create a set amount of customizable variation in the movement. For example, the footstep strategy 210 f includes parameters 214 such as specifying which foot 124 to step with, footstep coordinates / location (e.g., LO / TD location), swing height of the designated stepping foot 124, and / or step speed of the designated stepping foot 124. For body strategy 210b, parameters 214 may include swing area, swing location, swing direction, swing speed, and / or swing style.

[0030]

[0035] In some examples, to sequence one or more strategies 210 to form the strategy script 202, the UI 200 slices both the strategies 210 and the strategy script 202. S In other words, a slice of time from the strategy 210, t s The strategy 210 is one or more slices of the strategy script 202. S The time slice t of the strategy script 202 so as to occupy s Slice Strategy Script 202 s In other words, the UI 200 may scale the strategy script 202 over time, with each strategy 210 splitting it into a fixed number of slices t s For example, a script builder 222 for building the script script 202 of FIG. 2A in a timeline 222 may have four strategies 210, 210a-d, and a total of 40 slices t s A sustained strategy script 202 is shown, where the first strategy 210a is a 12 slice t s , and the second strategy 210b and the third strategy 210c each occupy 8 slices t s The fourth strategy 210d occupies 12 slices t sIf the user 10 wants to perform this 40 slice strategy script 202 after 40 seconds, the robot 100 allocates 12 seconds to each of the first strategy 210a and the fourth strategy 210d, and 8 seconds to each of the second strategy 210c and the third strategy 210d. In contrast, if the user 10 wants to perform this 40 slice strategy 202 after twice the time of 80 seconds, the robot 100 scales the slices of the strategies 210 such that the robot 100 allocates 24 seconds to each of the first strategy 210a and the fourth strategy 210d, and 16 seconds to each of the second strategy 210c and the third strategy 210d. Additionally or alternatively, each strategy 210 also has a set duration t having a start time and an end time. M Since the strategy script 202 may include unoccupied slices before, during, or after the strategy 210 as part of an exercise routine.

[0031]

[0036] In some implementations, the UI 200 includes a list 212 of strategies 210 that the user 10 may select to build the strategy script 202. As the user 10 selects from the list 212 (e.g., a selection shown in FIG. 2B as a bold outline around the strategy 210), the selection of the strategy 210 may display parameters 214 specific to the strategy 210 that the user 10 may customize (e.g., shown in the simulator 230). In some examples, the user 10 adds the strategy 210 to the strategy script 202 by selecting it or by selecting the strategy 210 and dragging it into the script builder 220. Some examples of selecting include double-clicking or selecting the strategy 210 and clicking a GUI element such as "Add" (e.g., as shown in FIG. 2B). Here, the script builder 220 may be a separate panel in the same window with the strategies list 212 or in a separate window.

[0032]

[0037] When a strategy 210 is added to the script builder 220 to build a strategy script 202, the strategy 210 may have a default size (e.g., number of slices) based on hard coding of the strategy 210. For example, a single strategy 210 that corresponds to a single exercise may have fewer slices by default than a more complex exercise (e.g., a handstand). s In either case, slice t of strategy 210 s The default number of may correspond to the minimum amount of time the robot 100 needs to successfully perform the strategy 210. In some implementations, the strategy 210 may be further modified by adding the strategy 210 to the script builder 220. For example, the user 10 may add a slice t s A slice of t with a number larger than the default number s In some examples, the script builder 220 resizes the strategy 210 to extend the strategy 210 over a segment (e.g., a slice t s 2, where a user 10 imports (e.g., by selection or a drag-and-drop process) a strategy 210 into the timeline 222 at a slice location within the timeline 222 that the user 10 specifies.

[0033]

[0038] 2A, the timeline 222 includes two or more layers 224, where a layer 224 refers to the space in the timeline 222 that the strategy 210 may occupy in affecting a particular joint J assigned to the layer 224. In other words, the layer 224 defines the allocation of the joints J of the robot 100. In some configurations, each joint J of the robot 100 is assigned to a particular layer 224 of the timeline 222. For example, a first layer 224, 224a, may be assigned to the knee joint J. K and hip joint J H In some examples, one or more layers 224 do not have joint J. An example of a layer 224 without joint J is when the layer 224 is bIn other words, there may be a strategy 210 that passively controls joint J. In other words, if the strategy 210 is a modification to or input to the controller 172, the strategy 210 may affect a joint J that has already been assigned or allocated to a particular layer 224. For example, the movement of the body 110 is effectively generated by making a modification to at least one joint J of at least one leg 120 of the robot 100. Furthermore, if a joint J of a leg 120 has already been assigned to a layer 224 (e.g., the footstep strategy 210), the strategy 210 may affect a joint J that has already been assigned or allocated to a particular layer 224. f For example, for a robotic or leg strategy, the motion of the body 110 may cause joint control interference between the two layers 224. In some examples, to prevent issues caused by joint interference, the UI 200 may use the body strategy 210 as a suggested modification to a controller 172 of another layer 224 or as a parameter of the motion for a particular controller 172. b In other words, the main strategy 210 b are secondary considerations that may affect joint control if they are not expected to interfere with, for example, the leg strategy. Here, the strategies 210 that may modify the control of joint J are hints 210 H It is called. Tip 210 H occurs at a specific time, hint 210 H is taken into account by each of the other active controllers 172 (e.g., each time-current controller 172). H When interpreting the hint 210, the controller 172 H corresponds to a joint J different from the joint J controlled by the controller 172," and H You can ignore modifying the controller's behavior. Tip 210 H corresponds to the same joint J, the controller 172 H The motion or motion parameters 214 may be incorporated into the user's behavior.

[0034]

[0039] In some examples, such as FIG. 2B , the script builder 220 of the UI 200 allows the user 10 to have a strategy 210 for each layer 224 (e.g., shown as three layers 224, 224a-c) at each time. When joints J are assigned to layers 224 to prevent interference, multiple strategies 210 can be layered at the same time on the timeline 222. In other words, when a strategy 210 controls only a few parts of the robot 100, multiple strategies 210 can be stacked to participate in a wide range of dynamics of the robot 100, depending on the layers 224. For example, hints 210 H The specification of the body strategy 210 while one layer 224 is the primary consideration for control (e.g., leg strategy). b Another layer 224, such as layer 224a for the first strategy 210, layer 224b for the second strategy 210, etc., allows multiple strategies 210 to be layered at the same time on the timeline 222 as the second strategy 210 becomes a secondary consideration for control. By having layers 224, the strategies 210 may be composited and / or ordered within various compositions to give the user 10 more options than simply the number of strategies 210 available on the list 212. For example, FIG. 2B shows a first strategy 210 on a first layer 224a (e.g., an arm strategy layer) and / or a second strategy 210 on a second layer 224b (e.g., an arm strategy layer). a , 210a to a second strategy 210 on a second layer 224b (e.g., a main strategy layer). b , 210b, and a third strategy 210 on a third layer 224c (e.g., a leg strategy layer). f , 210c.

[0035]

[0040] Although the UI 200 allows for layered construction of strategies 210, the UI 200 typically does not determine the feasibility of composition of strategies 210. Rather, the UI 200 communicates a strategy script 202 having a combination of layers 224 to the robot 100 (e.g., to the strategy system 300 of the robot 100) as long as the layering of strategies 210 does not violate any rules in the UI 200 (e.g., in the script builder 220). In some examples, strategies 210 are dedicated to a particular layer 224. Specifically, the UI 200 allows the user 10 to select a main strategy 210. b The main strategy 210 b2C, the user 10 may place a swing body strategy 210 that affects the joints J of one or more legs 120 on the non-controlling layer 224. b Arm joint J A In some implementations, when a user 10 adds a strategy 210 to the timeline 222 of the script builder 220, the strategy 210 automatically registers to the layer 224 associated with the joint J that the strategy 210 affects (e.g., requires control of the underlying movement of the strategy 210). Here, the user 10 controls where the strategy 210 is placed on the timeline 222, not the layer 224 itself.

[0036]

[0041] In some configurations, the strategy 210 is configured to operate on joints J on two or more layers 224 of the timeline 222. In these configurations, the script builder 220 may configure the strategy 210 to operate on a default number of slices t s Slice t of each layer 224 corresponding to s For example, FIG. 2D shows a second strategy 210b (12 slices of the second tier 224, 224b) on the timeline 222 of the script builder 220. s and 12 slices of the third layer 224, 224c. s 2D , a timeline 222 includes three layers 224a-c, so that another strategy 210 can be combined with a second strategy 210b by occupying a first layer 224a for some portion of the same time. Here, the strategy 210 is shown as a single block so that the UI 200 prevents the user 10 from operating the same strategy 210 differently in various layers 224 (which would likely cause errors in the robot 100 attempting to perform the strategy 210). Referring to FIG. 2D , the timeline 222 includes three layers 224a-c, so that another strategy 210 can be combined with a second strategy 210b by occupying a first layer 224a for some portion of the same time.

[0037]

[0042] In some configurations, the UI 200 includes rules regarding the starting state (i.e., input state 218s) and ending state (i.e., output state 218e) of the strategy 210. The UI 200 incorporates these rules to prevent possible errors when the robot 100 attempts the strategy script 202. For example, as shown in FIG. 2D, once the second strategy 210, 210b, ends with an ending state 218e of the robot 100 on its knees, the next strategy 210 (e.g., shown as a third strategy 210c) may be executed based on the starting state 218s and the robot 100 standing (e.g., some type of footstep strategy 210). f ) cannot be assumed.

[0038]

[0043] To overcome these potential issues between neighboring strategies 210, each strategy 210 may include input states 218s and / or output states 218e. With each strategy 210 including input states 218s and / or output states 218e, the UI 200 (e.g., the script builder 220) may compare the transition states between two neighboring strategies 210 (e.g., the output state 218e of a first strategy 210a and the input state 218s of an adjacent second strategy 210b) to determine whether these transition states are compatible. If these transition states are not compatible, the UI 200 communicates to the user 10 that the set of strategies 210 is incompatible. In some examples, the UI 200 recognizes that a particular output state 218e is compatible with two or more input states 218s.

[0039]

[0044] Alternatively, the UI 200 may make this determination when the user 10 attempts to place or select the adjacent strategy 210. Here, if the adjacent strategy 210 is not compatible, the UI 200 does not allow the user 10 to place or select the strategy 210. In the example depicted by FIG. 2D, the selected strategy 210c that is not compatible is grayed out and indicated by an "X" to communicate to the user 10 that the strategy 210 is not compatible at that location in the timeline 222. In some implementations, the UI 200 displays a message in the UI 200 that communicates that the strategy 210 is not compatible with respect to the location where the user 10 wants to place the strategy 210 on the timeline 222. In some examples, the list 212 is dynamic such that the list 212 displays strategies 210 that are compatible with the previous selection. For example, the user 10 selects (i.e., clicks) a strategy 210 placed on the timeline 222, and the strategy list 212 displays the compatible strategies 210. In some examples, the UI 200 includes a GUI element (e.g., from a menu or toolbar within the UI 200) that the user 10 selects to display adaptable strategies 210 based on input states 218s and / or output states 218e. Additionally or alternatively, the UI 200 may provide assistance to the user 10 as the user selects or attempts to place a strategy 210 in an incompatible position on the timeline 222. For example, the UI 200 may suggest that the user 10 add a strategy 210 as a transition between strategies 210 that are incompatible if they are adjacent to each other in time on the timeline 222. Here, the UI 200 suggests a strategy 210 that aligns the output state 218e of a first strategy 210a with the input state 218s of a second strategy 210b.

[0040]

[0045] In some examples, such as FIG. 2E, the script builder 220 is scalable for more than one robot 100, 100a-b. For example, the script builder 220 includes multiple timelines 222, 222a-b, each timeline 222 having a layer 224 that is specific to a robot 100. In FIG. 2E, a first timeline 222, 222a is for a first robot 100a, and a second timeline 222, 222b is for a second robot 100b. By coordinating the timelines 222 of multiple robots 100, the UI 200 can be used to generate coordinated exercise routines between the robots 100. For example, the first robot 100a performs a first phase p. 1 The second robot 100b performs the motion of the second phase p 2 The first robot 100a pauses during the motion of the third phase, and then the first robot 100a performs the final third phase motion together with the second robot 100b. In some implementations, the UI 200 can accommodate different types of robots 100 such that the type or model of the robot 100 can be loaded with a robot profile-specific profile. Here, the robot-specific profile can load a timeline 222 with multiple layers 224 specific to the robot 100 of the robot-specific profile. For example, FIG. 2E depicts the first timeline 222a with three layers 224, 224a-c, and the second timeline 222b with only two layers 224, 224a-b. The second timeline 224 has fewer layers 224 because the corresponding robot 100b does not include an arm 126, and therefore a layer 224 corresponding to the joint J of the arm 126 is unnecessary. Although FIG. 2E shows timelines 222a-b displayed in the same window, each timeline 222 may have its own window (eg, one that user 10 may toggle between).

[0041]

[0046] 2A-2E, in some examples, the UI 200 also includes a simulator 230. The simulator 230 may be a panel in the main window of the UI 200 or in its own window of the UI 200. In some implementations, the simulator 230 is configured to display a model 232 or avatar of the robot 100 as a visual simulation of the strategy 210. For example, the user 10 selects one or more strategies 210, and the simulator 230 displays a sample rendering of the selected strategy or strategies 210. In some configurations, such as FIG. 2D, the simulator 230 includes GUI elements 234 that may input or adjust parameters 214 associated with the strategy 210. In these configurations, the simulator 230 may display the strategy 210 by applying the parameters 214 input by the user 10. For example, sliders shown in FIG. 2D as GUI elements 234 for parameters 214 may be used to adjust the parameters 214 of the strategy 210 (e.g., the main body strategy 210). b 2D shows a single set of parameters 214 for selection of a strategy 210n, the simulator 230 may be configured to display various GUI elements 234 corresponding to the parameters 214 of the selected strategy 210. For example, the slider GUI element 234 may change to a simple input field when the user 10 selects the second strategy 210b.

[0042]

[0047] In some configurations, the UI 200 is configured to transmit a portion of the strategy script 202 to the robot 100. By transmitting a portion of the strategy script 202, the user 10 may troubleshoot or debug a sub-portion of the exercise routine. For example, if the user 10 becomes frustrated that the strategy script 202 is failing to execute successfully in the robot 100, the user 10 uses the UI 200 to select a portion of the strategy script 202 to transmit to the robot 100. In some implementations, the UI 200 requests the robot 100 to perform the strategy script 202 or a portion of the strategy script 202 in a loop. Here, the user 10 may update the strategy script 202 in the UI 200 while the robot 100 repeats an act or a portion of an act of the strategy script 202. In some configurations, the user 10 specifies where in the timeline 222 to repeat the strategy script 202. For example, the user 10 inserts a loop widget 226 (FIGS. 2E and 2F) (e.g., shown as an adjustable vertical bar) into the timeline 222 to specify the location in the strategy script 202 where the exercise routine should repeat (e.g., resume). 1 and the second phase p 2 The third phase p of the strategy script 202a-b can be used to troubleshoot the strategies 210, 210i-q. 3 Shows the previous loop widget 226.

[0043]

[0048] Additionally, the UI 200 may be configured to save and / or load strategy scripts 202 that have been built or are in the process of being built by the user 10. For example, the UI 200 communicates with the computational resources 22, 24 of the user device 20 or the computational resources 162 of the remote system 160 to store or load a particular strategy script 202. In some examples, the UI 200 allows the user 10 to share the strategy script 202 with other users so that the strategy script 202 may be built collaboratively. In some implementations, the UI 200 includes a storage container (e.g., a file directory) that includes one or more strategy scripts 202 or includes pointers to the location of one or more strategy scripts 202. Additionally or alternatively, the computational system 140 of the robot 100 may store the strategy script 202 (e.g., temporarily or for some extended period of time).

[0044]

[0049] In some implementations, the strategy script 202 corresponds to a dance routine, where each strategy 210 corresponds to a dance movement that the robot 100 can perform. Here, a time slice t s can be used to match the tempo (i.e., speed) of a particular song. For example, a song may be identified (e.g., by the UI 200) as having a particular number of beats per minute (BPM), and each slice t s corresponds to one division of a single beat b. Because the strategy 210 is scalable, the tempo of the song can be changed (e.g., sped up or slowed down) or the actual song can be altered and the performance of the strategy script 202 can be adapted to the altered tempo.

[0045]

[0050] In some implementations, the script builder 220 includes a setting (e.g., GUI element 234) to set the timeline 222 to a particular tempo (e.g., BPM) to correspond to a particular song. This setting may, for example, allow the user 10 to directly map a song's beat b to the strategy 210 or to a portion of the strategy 210. For example, FIG. 2F illustrates the footstep strategy 210. f Footstep Strategy 210 on Timeline 222 f2A-2F, the timeline 222 depicts a slice t s The black bar in between indicates the beat b of the song.

[0046]

[0051] In some configurations, the UI 200 (e.g., the script builder 220) includes features to facilitate synchronization of the strategy script 202 with audio from a song. In some examples, such as FIG. 2E, the UI 200 includes a slider bar 228 of a timeline 222. The slider bar 228 may be a time synchronization tool that allows the user 10 to set the slider bar 228 to a custom position within the timeline 222. The position of the slider bar 228 may be used to indicate a starting position where the robot 100 should begin performing the movement routine (e.g., a dance) of the strategy script 202. For example, FIG. 2F depicts the slider bar 228 before the first strategy 220a.

[0047]

[0052] In some implementations, such as FIG. 2F, the UI 200 sends the strategy script 202 to the robot 100 as a configuration file along with the UI 200's current system time 204 (e.g., a timestamp based on the clock of the user device 20) and the start time 206 at which the robot 100 should start the exercise routine (e.g., indicated by the position of the slider bar 228). Here, when the robot 100 receives the strategy script 202 along with the UI 200's system time 204 and the exercise routine start time 206, it will wait the amount of time defined by the UI 200, and then start the exercise routine (e.g., start dancing). In some examples, when the robot 100 receives the strategy script 202, the robot 100 assumes that the communication of the strategy script 202 between the UI 200 and the robot 100 took some communication time, and synchronizes its own clock to account for this amount of communication time. For example, the robot 100 synchronizes its own clock to be the clock timestamp 204 received from the UI 200 plus an additional amount of communication time. Based on this time synchronization, the robot 100 begins an exercise routine (eg, a dance) at some future time (eg, a future time equal to the difference between the system time 204 and the specified start time 206 of the UI 200).

[0048]

[0053] Additionally or alternatively, the robot 100 provides feedback 208 to the UI 200 regarding the time / location at which the robot 100 perceives that it is in an athletic routine (e.g., dancing). For example, the robot 100 provides its own clock time 208, 208a (FIG. 3) as feedback 208 (e.g., in the form of a timestamp), where the robot's time should closely resemble the system time 204 of the UI 200 for synchronization reasons when the robot 100 receives the strategy script 202. In some examples, if there is a mismatch between the system time 204 of the UI 200 and the robot 100's time 208a, the UI 200 may resend its system time 204 to the robot 100 for resynchronization. In some implementations, when resynchronization occurs, the robot 100 may be configured to speed up or slow down the strategies 210 of the strategy script 202. For example, the robot 100 identifies candidate strategies 210 that the robot 100 may perform more quickly or more slowly to adjust the strategy routine 202 due to resynchronization. In yet other examples, instead of feedback 208 from the robot 100, the UI 200 periodically (e.g., at some specified frequency) communicates its current time clock 204 to the robot 100 so that the robot 100 checks the synchronization between the UI 200 and the robot 100. In these examples, the robot 100 resynchronizes when the periodic communication from the UI 200 indicates a time misalignment. In some configurations, the robot 100 resynchronizes when the difference between the system time 204 of the UI 200 and the time clock 208a of the robot 100 satisfies a time shift threshold (e.g., exceeds a value set as the time shift threshold). This approach may prevent the robot 100 from resynchronizing frequently when a small time shift exists between the UI 200 and the robot 100.

[0049]

[0054] Referring to FIG. 3, the strategy system 300 includes an interface 310, such as an application program interface (API), a sequencer 320, and a dynamic planner 330. From an overall perspective, the goal of the strategy system 300 is to receive an action request (e.g., a strategy script 202 having strategies 210) as input and convert this input into a solution that generates a behavior of the robot 100. Here, even if this input is impossible (e.g., in terms of physics or robot dynamics), the strategy system 300 is configured to return a solution that is the closest possible solution to generate a behavior of the robot 100 based on the input. In other words, the strategy system 300 is configured to determine the compliance or actual feasibility of the robot 100 to perform the strategies 210 of the strategy script 202. This approach is particularly useful when a combination of strategies 210 occurs in the strategy script 202 at a given time (i.e., the user 10 has layered strategies 210).

[0050]

[0055] In some examples, the interface 310 serves as a means of communication between the UI 200 and the robot 100. In other words, the interface 310 is how the entire strategy script 202 is communicated from the UI 200 to the robot 100. In some implementations, the sequencer 320 interprets the strategy script 202 from the interface 310 to understand the time sequence of when the robot 100 should attempt to execute a particular strategy 210 of the strategy script 202. In some examples, the sequencer 320 operates as a schedule of the strategies 210 of the strategy script 202. In some configurations, the sequencer 320 operates by identifying a time 208a of the robot 100 and identifying a strategy 210 that should be running at the identified time. For example, based on the time clock 208a of the robot 100, the sequencer 320 may schedule a particular slice t of the strategy script 202. s and the robot 100 identifies the time slice t s t , the strategy 210 is executed or attempted to be executed in the identified time slice t s, the sequencer 320 may communicate these strategies 210 to the dynamic planner 330. For example, FIG. 3 illustrates a clock symbol identifying the current time 208a of the robot 100 and the time slice t of the strategy script 202 that corresponds to the current time 208a. s where the time slice t s There are three strategies 210h-j (eg, indicated by the dotted boxes around the three strategies 210h-j) that occur in

[0051]

[0056] The strategy system 300 is configured to generate instructions 302 for controlling the movement of the robot 100 to move or attempt to move based on the strategy 210 of the strategy script 202. The strategy system 300 may use a dynamic planner 330 to generate all or a portion of the instructions 302. In some examples, the dynamic planner 330 is configured to generate instructions 302 (also referred to as joint instructions 302) for joints J of the robot 100 to control the movement of the robot, where the joint instructions 302 may correspond to joint positions, joint forces (e.g., joint torques), joint angles, angular velocities, etc. associated with the joints J. The controller 172 uses the dynamic planner 330 as a utility when activated by the strategy 210 at a particular time to determine the joint instructions 302 (e.g., joint torques) for implementing or attempting to implement the movement of the strategy 210. In examples where the robot 100 is a walking robot (e.g., four-legged), a dynamic planner 330 is often used to coordinate (e.g., often hierarchical) leg strategies 210 since leg motion is a common type of control for walking robots 100. With a walking robot 100, the control system 170 controls the stance leg 120 via force control (e.g., joint torques). ST and the position (e.g., by joint angle and angular velocity) of the swing leg 120 SW can be controlled.

[0052]

[0057] To generate the joint commands 302, the dynamic planner 330 includes a solver 332, where solver 332 refers to an optimization model that incorporates various constraints 334 (e.g., structural constraints, physical world constraints, user-defined constraints, etc.) of the robot 100 and / or its surroundings. To coordinate movements, the solver 332 of the dynamic planner 330 may receive or be pre-programmed to satisfy these constraints 334 such that the robot 100 may remain functional (e.g., maintain its balance) while performing a requested movement (e.g., requested via the strategy script 202). More specific examples of constraints 334 include a set of motion constraints of the robot 100, constraints based on the structural dimensions (e.g., leg length) of the robot 100, an estimate of friction on the robot 100 (e.g., at the feet 124 of the robot 100), etc.

[0053]

[0058] To generate the joint instructions 302, the dynamic planner 330 receives information, such as parameters 214, for one or more strategies 210 from the strategy script 202. In some examples, the dynamic planner 330 identifies a cost 336 associated with the strategy 210 (e.g., with each strategy 210 received at the dynamic planner 330), where the cost 336 refers to a value that specifies the importance of a movement or a particular portion of a movement of the strategy 210. For example, the cost 336 may be a target event pose P T The importance of the target event pose P T Typically, refers to a desired body position or orientation for a given move. In some implementations, each strategy 210 received by the dynamic planner 330 includes a cost 336, where the user 10 may define the cost 336 as a parameter 214 when constructing the strategy script 202, as shown in FIG. 2A. In other words, the user 10 may determine how critical a particular move of the strategy 210 is by specifying the target event pose P of the strategy 210. T Define it by assigning a value to

[0054]

[0059] By having costs 336 associated with the strategies 210 input into the solver 332, the solver 332 is configured to optimize the multiple costs 336 to understand how to best perform two or more strategies 210 at a particular time in the strategy script 202. In other words, if a first strategy 210a and a second strategy 210b occur at the same time, each strategy 210a-b includes costs 336, 336a-b that inform the solver 332 what is important (e.g., to the user 10) about each strategy 210a-b. For example, the first strategy 210a may not want the user 10 to achieve a particular landing position of the foot 124 during a footstep (e.g., during a swing), but the footstep strategy 210 f A footstep strategy 210 that does not care (or emphasizes) the speed of movement during the footstep or orientation of the middle foot 124 f Here, the user 10 may select a footstep strategy 210 to reflect such high importance (e.g., high cost value). f In this example, the user 10 may set the cost 336 of the landing position as the second strategy 210b. b In the case of a hierarchy of strategies 210a-b, the solver 332 may determine to reduce the amount of sway of the second strategy 210b so as not to impair the accuracy of the landing position of the first strategy 210a. Here, the amount of sway may be designed as a parameter 214 in the UI 200. In this scenario, the second strategy 210b may be configured with a cost 336b that reflects a low importance of the amount of sway. In this example, if each strategy 210a-b has a high cost 336 that results in a potential conflict, the solver 332 may attempt to optimize the movement results to resolve the conflict or find a compromise between the conflicting instructions.

[0055]

[0060] In some configurations, Tip 210 H The strategy 210, which is H determines whether the hint 210 is compatible with this other strategy 210 that occurs at the same time.H But Hint 210 H and the strategy 210 occurs at the same time, the strategy 210 is a hint 210 H Exercise or Tips 210 H Based on this process, the dynamic planner 330 calculates the hints 210 in the solver 332. H The strategy 210 may have already been modified by

[0056]

[0061] In some examples, the controller 172 of the robot 100 executes the strategy 210 without utilizing the dynamic planner 330. In other words, the solver 332 that generates the joint commands 302 is not required for the specific strategy 210. For example, the specific strategy 210 does not result in a joint control conflict. In other words, the arm strategy 210 that affects only joint J of the arm 126 a and leg strategy 210 that does not affect joint J of arm 126 f If these occur at the same time, arm strategy 210 a The controller 172 that operates the arm 126 may proceed within its own joint commands 302 of the arm 126 without utilizing the dynamic planner 330. In some examples, the strategy 210 bypasses the dynamic planner 330 when optimization is not required (e.g., when only one strategy 210 occurs without the possibility of joint interference during execution of the movement for that strategy 210). In some implementations, the strategy system 300 is configured to not allow a strategy 210 (even a basic strategy 210) to bypass the dynamic planner 330. In this approach, using the dynamic planner 330 even for the basic strategy 210 helps ensure that the robot 100 follows real-world constraints (e.g., constraint 334) during the movement of the strategy 210. This approach also allows the robot 100 to maintain balance and movement fluidity during the movement.

[0057]

[0062] While not all strategies 210 necessarily utilize the dynamic planner 330, in some implementations bypassing the dynamic planner 330 is not based on controllers 172 using disjoint sets of joints J. Instead, in some examples, the strategy system 300 may use tracks (or "tiers") to avoid situations of multiple controllers 172 attempting to command the same joint J. In some instances, many, but not all, controllers 172 are implemented using the dynamic planner 330. The dynamic planner 330 may use hints 210 to avoid resolving multiple controllers 172 wanting to command the same joint J. H Import.

[0058]

[0063] In some implementations, the strategy 210 may be implemented by the controller 172 or the hints 210 H Optionally, the strategy 210 is either the controller 172 or the hint 210. H The controller 172 directly controls the joints J of the robot 100. In the example shown, the hint 210 H are received by the controllers 172 in response to a request to modify the behavior of each controller 172 in a particular way. Dance moves (e.g., strategies 210) are assigned to one or more tracks (tiers 224). Which track a dance move follows is an inherent property of that individual dance move type, not a choice of the user 10. For example, a "step" move might always follow on the legs track, and a "rock" move might always follow on the body track. The tracks determine which dance moves are followed by hints 210. H or Controller 172. In some cases, all leg and arm movements are Controller 172 and all body movements are Hints 210. H If a dance movement uses multiple tracks, the dance movement is a controller 172 if any of them are controller tracks. The dynamic planner 330 is a utility used by many of the leg track controllers. The dynamic planner 330 provides hints 210 to modify its behavior. HI know how to capture it.

[0059]

[0064] In some implementations, to generate the joint commands 302 for the joints J of the robot 100, the strategy system 300 considers the current state 338 of the robot 100, where the current state 338 of the robot 100 may refer to anything measured relative to the robot 100 or the surroundings of the robot 100. For example, the current state 338 includes the sensor data 134 from the sensor system 130 related to the time (e.g., the time identified by the sequencer 320) when the robot 100 is attempting to perform the strategy 210. Some examples of the current state 338 include the position of the robot 100, the velocity of the robot 100, and / or the forces the robot 100 is experiencing. When the strategy system 300 considers the current state 338, the strategy system 300 (e.g., the dynamic planner 330) may update the constraints 334 based on the current state 338 to enable accurate optimization of the solver 332.

[0060]

[0065] 4 is an exemplary arrangement of operations of a method 400 for generating instructions for controlling the motion of the robot 100. In operation 402, the method 400 receives a strategy script 202 including a plurality of strategies 210 for the walking robot 100 to perform, where each strategy 210 is associated with a cost 336. In operation 404, the method 400 identifies that two or more of the strategies 210 in the strategy script 202 occur at the same time. In operation 406, the method 400 determines a composite strategy for the walking robot 100 to perform at the same time based on the two or more strategies 210 and the costs 336 associated with the two or more strategies 210. In operation 408, the method 400 generates joint instructions 302 (to command a set of joints J of the walking robot 100) for controlling the motion of the walking robot 100 at the same time, where the set of joints J corresponds to the composite strategy.

[0061]

[0066] In some examples, the plurality of strategies 210 includes a hint corresponding to a body motion of the body 110 of the walking robot 100. Here, the method 400 determines whether the hint fits another strategy 210 of the plurality of strategies 210 occurring at the same time as the hint, and modifies the other strategy 210 to incorporate the body motion of the hint if the hint fits the other strategy 210. In some implementations, receiving the strategy script 202 includes receiving the strategy script 202 from a user device 20 in communication with the data processing hardware, where the strategy script 202 is defined by a user 10 of the user device 20 in a user interface 200 executing on the user device 20. In some configurations, the method 400 synchronizes each dance move of the dance script with a beat of the song when the strategy script 202 includes a dance script and each strategy 210 includes a dance move.

[0062]

[0067] 5 is a schematic diagram of an example computing device 500 that may be used to implement the systems (e.g., UI 200 and / or strategy system 300) and methods (e.g., method 400) described herein. Computing device 500 is intended to represent various types of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown, their connections and relationships, and their functions are intended to be merely exemplary, and thus are not meant to limit the implementation of the invention described and / or claimed in this document.

[0063]

[0068] The computing device 500 includes a processor 510 (e.g., data processing hardware), a memory 520 (e.g., memory hardware), a storage device 530, a high-speed interface / controller 540 that connects to the memory 520 and a high-speed expansion port 550, and a low-speed interface / controller 560 that connects to a low-speed bus 570 and the storage device 530. Each of the components 510, 520, 530, 540, 550, 560 are interconnected using various buses and may be mounted on a common motherboard or otherwise mounted as desired. The processor 510 may process instructions (including instructions stored in the memory 520 or on the storage device 530) for execution within the computing device 500 to display graphical information for a graphic user interface (GUI) on an external input / output device (such as a display 580 coupled to the high-speed interface 540). In other implementations, multiple processors and / or multiple buses may be used as needed, along with multiple memories and multiple types of memories. Also, multiple computing devices 500 may be connected (eg, as a bank of servers, a collection of blade servers, or a multi-processor system) with each device providing some portion of the required operations.

[0064]

[0069] The memory 520 stores information non-transiently within the computing device 500. The memory 520 may be a computer readable medium, a volatile memory unit, or a non-volatile memory unit. The non-transient memory 520 may be a physical device used to store programs (e.g., a sequence of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0065]

[0070] The storage device 530 can provide mass storage for the computer device 500. In some implementations, the storage device 530 is a computer-readable medium. In various different implementations, the storage device 530 can be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices, including a storage area network or other configuration of devices. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or a machine-readable medium, such as the memory 520, the storage device 530, or a memory on the processor 510.

[0066]

[0071] While the high-speed controller 540 manages the bandwidth-intensive operations of the computing device 500, the low-speed controller 560 manages the low-bandwidth intensive operations. This allocation of duties is merely exemplary. In some implementations, the high-speed controller 540 is coupled to the memory 520, to the display 580 (e.g., via a graphics processor or accelerator), and to a high-speed expansion port 550 that may accept various expansion cards (not shown). In some implementations, the low-speed controller 560 is coupled to the storage device 530 and to a low-speed expansion port 570. The low-speed expansion port 570, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled (e.g., via a network adapter) to one or more input / output devices such as a keyboard, a pointing device, a scanner, or a network device such as a switch or router.

[0067]

[0072] The computing device 500 may be implemented in many different forms as shown in the figure. For example, the computing device 500 may be implemented as a standard server 500a or a cluster of such servers 500a, as a laptop computer 500b, as part of a rack server system 500c, or as part of a robot 100.

[0068]

[0073] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system. The programmable system includes at least one programmable processor, which may be dedicated or general purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transmit data and instructions to the storage system, at least one input device, and at least one output device.

[0069]

[0074] These computer programs (also known as programs, software, software applications or codes) include machine instructions for a programmable processor and may be implemented in high-level procedural programming languages ​​and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus and / or device (e.g., magnetic disk, optical disk, memory, or programmable logic device (PLD), etc.) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0070]

[0075] The processes and logic flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output data. The processes and logic flows may also be implemented by special purpose logic circuitry, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). Processors suitable for executing a computer program include, by way of example, both general purpose and special purpose microprocessors, and any one or more processors of any type of digital computer. Typically, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic, magneto-optical or optical disks) for storing data, or be operatively coupled to receive data from or transfer data to the devices, or both. However, a computer need not have such devices. Suitable computer readable media for storing computer program instructions and data include, by way of example only, all types of non-volatile memory, media and memory devices, including semiconductor memory devices (e.g., EPROM, EEPROM, flash memory devices), magnetic disks (e.g., internal hard disks, removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, dedicated logic circuitry.

[0071]

[0076] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or a touch screen for displaying information to a user) and, optionally, a keyboard and a pointing device (e.g., a mouse or trackball) through which the user may provide input to the computer. Other types of devices may also be used to provide interaction to a user. For example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user may be received in any form, including acoustic, speech, or tactile input. In addition, the computer may interact with a user by sending documents to and receiving documents from a device used by the user (e.g., by sending web pages to a web browser on the user's client device in response to a request received from the web browser).

[0072]

[0077] A number of implementations have been described. Nevertheless, it will be understood that various modifications are possible without departing from the spirit and scope of the disclosure. Accordingly, other implementations are within the scope of the following claims.

Claims

1. A method of choreographing robots, The data processing hardware receives a strategy script containing multiple strategies employed by a walking robot, each of which is associated with a cost. The data processing hardware identifies that the first and second strategies of the aforementioned strategic script occur simultaneously. When both the first strategy and the second strategy command the first joint, the data processing hardware determines the conflict between the first strategy and the second strategy. The data processing hardware determines the combined strategy of the walking robot to be performed at the same time, based on the first strategy and the second strategy, as well as the first cost associated with the first strategy and the second cost associated with the second strategy, and Based on the current state of the walking robot, the data processing hardware generates joint commands for controlling the movement of the walking robot at the given time, wherein the joint commands command a pair of joints of the walking robot, including a first joint and a second joint, the pair of joints corresponding to the synthesized strategy, and the joint commands generate joint commands that control the movement of the first joint to execute at least a portion of the synthesized strategy. A method that includes this.

2. The method according to claim 1, wherein the walking robot corresponds to a quadruped robot.

3. One of the aforementioned strategies includes a third strategy configured to modify the control of the first joint, wherein the third strategy corresponds to the body motion of the walking robot body. The data processing hardware determines whether the third strategy is compatible with a fourth strategy among the plurality of strategies that occur at the same time as the third strategy, and When the third strategy is compatible with the fourth strategy, the data processing hardware modifies the fourth strategy to incorporate the body motion of the third strategy. The method according to claim 1 or 2, further comprising:

4. The method according to any one of claims 1 to 3, wherein each strategy is configured to use the active controller of the walking robot or to modify the control by the active controller of the walking robot.

5. The method according to any one of claims 1 to 4, wherein the cost associated with each strategy includes a user-defined cost indicating the importance of the walking robot performing the strategy.

6. Receiving the aforementioned strategy script includes receiving the strategy script from a user device that is in communication with the data processing hardware, The method according to any one of claims 1 to 5, wherein the strategy script is defined by the user of the user device in a user interface executed on the user device.

7. The method according to any one of claims 1 to 6, wherein at least one of the plurality of strategies includes a footstep strategy, and the footstep strategy includes the location and time of landing or lifting of the free leg of the walking robot.

8. The method according to any one of claims 1 to 7, wherein at least one of the plurality of strategies includes an arm strategy that includes manipulator poses.

9. The method according to any one of claims 1 to 8, wherein the strategy script includes a dance script, and each strategy includes a dance movement.

10. The method according to claim 9, further comprising synchronizing each dance movement of the dance script with the rhythm of the song using the data processing hardware.

11. The method according to any one of claims 1 to 10, further comprising determining by the data processing hardware that the output state of the first strategy of the plurality of strategies in the strategy script follows the input state of the subsequent strategies.

12. It is a robot, Main unit, Two or more legs attached to the main body, and A robot comprising the main body and a computing system in communication with the two or more legs, The computing system includes data processing hardware and memory hardware that is in communication with the data processing hardware. The memory hardware stores commands that cause the data processing hardware to perform an operation when executed on the data processing hardware, and the operation is: The robot receives a strategy script which includes multiple strategies, each of which is associated with a cost. Identifying that the first and second strategies of the aforementioned strategic script occur simultaneously. If both the first strategy and the second strategy command the first joint, determine the conflict between the first strategy and the second strategy. The combined strategy of the robot to be performed at the same time is determined based on the first strategy and the second strategy, as well as the first cost associated with the first strategy and the second cost associated with the second strategy, and A robot, comprising generating joint commands for controlling the movement of the robot at a given time based on the robot's current state, wherein the joint commands command a pair of joints of the robot, including a first joint and a second joint, the pair of joints corresponding to the synthesized strategy, and the joint commands control the movement of the first joint to execute at least a portion of the synthesized strategy.

13. The robot according to claim 12, wherein the robot corresponds to a quadruped robot.

14. One of the aforementioned strategies includes a third strategy configured to modify the control of the first joint, wherein the third strategy corresponds to the body motion of the robot body, and the operation further includes To determine whether the third strategy is compatible with a fourth strategy among the plurality of strategies that occur at the same time as the third strategy, and The robot according to claim 12 or 13, wherein the third strategy is adapted to the fourth strategy, the fourth strategy is modified to incorporate the body motion of the third strategy.

15. The robot according to any one of claims 12 to 14, wherein each strategy is configured to use the robot's active controller or to modify the control by the robot's active controller.

16. The robot according to any one of claims 12 to 15, wherein the cost associated with each strategy includes a user-defined cost indicating the importance of the robot performing the strategy.

17. The operation of receiving the strategy script includes receiving the strategy script from a user device that is in communication with the data processing hardware, The robot according to any one of claims 12 to 16, wherein the strategic script is defined by the user of the user device in a user interface executed on the user device.

18. The robot according to any one of claims 12 to 17, wherein at least one of the plurality of strategies includes a footstep strategy, the footstep strategy includes the location and time of landing or lifting of the robot's free leg.

19. The robot according to any one of claims 12 to 18, wherein at least one of the plurality of strategies includes an arm strategy that includes manipulator posing.

20. The robot according to any one of claims 12 to 19, wherein the strategy script includes a dance script, and each strategy includes a dance movement.

21. The robot according to claim 20, wherein the operation further comprises synchronizing each dance movement of the dance script with the beat of the song.

22. The robot according to any one of claims 12 to 21, further comprising determining that the output state of the first strategy of the plurality of strategies in the strategy script conforms to the input state of the subsequent strategies.

Images