Rapid design and animation of a free-walking robotic device

Abstract

A method of training a robotic device includes: parameterizing an input to the robotic device via a processing element. Parameterizing includes defining a range of values ​​for the input. The method further includes generating, via the processing element, a plurality of samples of the parameterized input from the range of values; training a control strategy via the processing element. The training includes: providing the plurality of samples to a control strategy, wherein the control strategy is adapted to operate an actuator of the robotic device, and generating, via the processing element, a policy behavior using the control strategy; transmitting the policy behavior to a robotic model, wherein the robotic model includes a physical model of the robotic device. The method further includes deploying one or more trained control strategies to an onboard controller for the robotic device.

Description

Technical Field

[0001] The present application relates to systems and methods for controlling a robotic character and generating effects with the robotic character. CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority under 35 U.S.C. §119(e) and 37 C.FR §1.78 to Provisional Application No. 63 / 542,225, filed on October 3, 2023, entitled “Rapid Design and Animation of Freely-Walking Robotic Characters,” which is hereby incorporated by reference in its entirety. Background Art

[0003] Creating and training robotic characters, such as freely moving robots, can be time-intensive and processing-intensive. Previous model-based processes for creating freely moving robots are often too closely tied to the physical hardware, making changes and improvements to the process or hardware difficult, i.e., new hardware requires entirely new control processes to achieve precise control and balance for the robotic device. Therefore, there is a need for an improved process that can enable rapid design of stable and creative robotic systems. Summary of the invention

[0004] In one embodiment, a method for training a robotic device includes: parameterizing an input to the robotic device via a processing element, wherein the parameterization includes limiting a range of values ​​for the input; generating a plurality of samples of the parameterized input from the range of values ​​via the processing element; training a control strategy via the processing element, wherein the training includes: providing the plurality of samples to the control strategy, wherein the control strategy is suitable for operating actuators of the robotic device, and generating a policy behavior using the control strategy via the processing element; transmitting the policy behavior to a robotic model, wherein the robotic model includes a physical model of the robotic device; and deploying the trained control strategy to an onboard controller for the robotic device.

[0005] Optionally, in some embodiments, the input to the robotic device includes at least one of: mass, torque, force, speed, number, type, or range of motion of components of the robotic device; a disturbance imparted to the robotic device; an operator command; or an environmental characteristic.

[0006] Optionally, in some embodiments, the control strategy is adapted to cause the robotic device to perform at least one of a permanent motion, a periodic motion, or an occasional motion.

[0007] Optionally, in some embodiments, the training further includes: simulating the action of the actuator using a physical model via a processing element; comparing the simulated action of the actuator with a reference action of the actuator via the processing element, wherein the reference action is based on multiple samples; and rewarding the control strategy based on the comparison via the processing element.

[0008] In one embodiment, a method for operating a robotic device includes: receiving user input at a processing element, wherein the processing element communicates with one or more actuators of the robotic device; comparing the user input with an animation database via the processing element; selecting an animation from the animation database based on the comparison via the processing element; actuating a control strategy for selecting the animation via the processing element, wherein the control strategy is trained by a reinforcement learning method; generating low-level controls suitable for controlling the robotic device actuators via the processing element; and controlling the robotic device actuators via the low-level controls.

[0009] Optionally, in some embodiments, the user input comprises a command to actuate a performance function of the robotic apparatus.

[0010] Optionally, in some embodiments, the performance functionality is independent of the robotic device actuators.

[0011] Optionally, in some embodiments, the performance function includes actuating at least one of lights, a movable antenna, eyes, or sound of the robotic device.

[0012] Optionally, in some embodiments, the reinforcement learning method includes: parameterizing an input to the robotic device via a second processing element, wherein the parameterization includes limiting a range of values ​​for the input; generating multiple samples of the parameterized input from the range of values ​​via the second processing element; providing the multiple samples to a control strategy via the second processing element; generating a policy behavior using the control strategy via the second processing element and transmitting the policy behavior to a robotic model, wherein the robotic model includes a physical model of the robotic device; simulating an action of an actuator of the robotic device using the physical model via the second processing element; comparing the simulated action of the actuator of the robotic device with a reference action of the actuator of the robotic device via the second processing element, wherein the reference action is based on multiple samples; and rewarding the control strategy based on the comparison of the simulated action of the actuator of the robotic device with the reference action via the processing element.

[0013] Optionally, in some embodiments, the input to the robotic device includes at least one of: mass, torque, force, speed, number, type, or range of motion of components of the robotic device; a disturbance imparted to the robotic device; an operator command; or an environmental characteristic.

[0014] Optionally, in some embodiments, the control strategy is adapted to cause the robotic device to perform at least one of a permanent motion, a periodic motion, or an occasional motion.

[0015] Optionally, in some embodiments, the selection animation includes one or more of a background animation or a trigger animation, and the method of operating the robotic device further includes laminating at least one of the background animation or the trigger animation with the remote control animation.

[0016] Optionally, in some embodiments, the remote control animation is based on user input received from the remote control.

[0017] In one embodiment, a robotic device includes: a plurality of modular hardware components; a processing element that communicates with the plurality of modular hardware components; and a plurality of control policies that are trained by a reinforcement learning method to control the plurality of modular hardware components.

[0018] Optionally, in some embodiments, the user input comprises a command to actuate a performance function of the robotic apparatus.

[0019] Optionally, in some embodiments, the performance functionality is independent of the robotic device actuators.

[0020] Optionally, in some embodiments, the selection animation includes one or more of a background animation or a trigger animation, and the robotic device can further operate by laminating at least one of the background animation or the trigger animation with the remote control animation.

[0021] Optionally, in some embodiments, the reinforcement learning method includes: parameterizing an input to the robotic device via a second processing element, wherein the parameterization includes limiting a range of values ​​for the input; generating multiple samples of the parameterized input from the range of values ​​via the second processing element; providing the multiple samples to multiple control strategies via the second processing element; generating a policy behavior using one of the multiple control strategies via the second processing element and transmitting the policy behavior to a robotic model, wherein the robotic model includes a physical model of the robotic device; simulating an action of an actuator using the physical model via the second processing element; comparing the simulated action of the actuator with a reference action of the actuator via the second processing element, wherein the reference action is based on multiple samples; and rewarding one of the multiple control strategies based on the comparison of the simulated action of the actuator with the reference action via the processing element.

[0022] Optionally, in some embodiments, the input to the robotic device includes at least one of: mass, torque, force, speed, number, type or range of motion of an actuator; a disturbance imparted to the robotic device; user input or environmental characteristics.

[0023] Optionally, in some embodiments, the plurality of strategies are adapted to cause the robotic device to perform at least one of a permanent action, a periodic action, or an occasional action.

[0024] Optionally, in some embodiments, the robotic device is capable of the following operations: receiving user input at a processing element; comparing the user input with an animation database via the processing element; selecting an animation from the animation database based on the comparison via the processing element; actuating a control strategy from a plurality of control strategies for selecting the animation via the processing element; generating a low-level control suitable for controlling an actuator via the processing element; and controlling a plurality of modular hardware components via the low-level control.

[0025] In one embodiment, a method for controlling a robotic device includes: generating a first strategy behavior suitable for performing a permanent action via a first training control strategy executed by a processing element; generating a second strategy behavior suitable for performing a periodic action via a second training control strategy executed by the processing element; generating a third strategy behavior suitable for performing an occasional action via a third training control strategy executed by the processing element; and deploying the first strategy behavior, the second strategy behavior, and the third strategy behavior to multiple actuators of the robotic device via the processing element, wherein the multiple actuators are suitable for performing permanent actions, periodic actions, and occasional actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1

[00136] An embodiment of a robotic device that may utilize the control architecture and training methods described herein is shown.

[0027] Figure 2 Shows that training is suitable for Figure 1 Embodiments of a method of using a strategy together with a robotic device.

[0028] Figure 3 The operation is shown to be suitable for Figure 1 Embodiments of a method of using a strategy together with a robotic device.

[0029] Figure 4A As shown by Figure 2 and Figure 3 Implementation of the strategy Figure 1 Embodiment of a permanent motion of a robotic device.

[0030] Figure 4B As shown by Figure 2 and Figure 3 Implementation of the strategy Figure 1 Embodiment of the periodic motion of the robotic device.

[0031] Figure 4C As shown by Figure 2 and Figure 3 Implementation of the strategy Figure 1 An embodiment of the incidental actions of a robotic device.

[0032] Figure 5A-5CShow use Figure 2 and Figure 3 Implementation of the strategy Figure 1 A method of layering animation of an embodiment of a robotic device.

[0033] Figure 6 Is training suitable for Figure 1 A flow chart of an embodiment of a method of using a strategy with a robotic device.

[0034] Figure 7 Shown is a method for generating Figure 1 System diagram of the control architecture of a robotic device.

[0035] Figure 8 shows the use of the architecture described in this article to control Figure 1 System diagram of the robotic device.

[0036] Fig. 9 is suitable for Figure 1 Robotic device or Figure 2 , Figure 3 and Figure 6 A simplified block diagram of components of a computing system for use with the method. DETAILED DESCRIPTION

[0037] The present disclosure includes embodiments for an integrated set of software and / or hardware tools for rapidly designing, building, animating, and / or controlling custom robotic devices. In some cases, these robotic devices may move freely, walk, and be configured to perform other movements, such as those that may be generated as part of a creative process. In some instances, the robotic devices may be selected to replicate a person both aesthetically and functionally. In some instances, the robotic design may be simulated in a simulated environment, and the learned model deployed to an onboard computer for the robotic device.

[0038] The robotic device can be generated from a set of modular hardware components. For example, various actuators, controllers, sensors, real-time computers, circuit boards, etc. can be combined in different ways to generate different robotic devices with different aesthetic appearances and functionality. In some embodiments, the modular hardware components are combined with parts made using various rapid prototyping methods, such as machining or additive manufacturing (e.g., 3D printing, etc.). The modular aspects of the hardware and machined or additively manufactured parts allow the robot to be created quickly and at a low cost compared to conventional custom robotic systems that can be used.

[0039] To allow the robot to perform walk cycles or other animations specified by the artist, despite the use of modular hardware (which may not be well suited to a particular functionality or include tolerances or other issues), reinforcement learning ("RL") techniques may be used. RL techniques use action policies (or control policies) that govern how a robotic device moves in a particular situation. To train a policy, the policy is asked to perform a certain action (e.g., walking), and the action output generated by the policy is compared to a reference action (i.e., the input or desired action). Policy-generated actions that match the reference action are rewarded or reinforced, while policy-driven actions that deviate from the reference action are penalized. Rewards strengthen the parts of the policy that lead to appropriate actions, while penalties weaken the parts of the policy that lead to incorrect or undesirable actions.

[0040] By using reinforcement learning techniques, the specific hardware used in the robotic device does not need to be clearly defined as in previous techniques, which typically require a custom control model for each hardware design. In various embodiments, a computer-aided design ("CAD") model can be generated that includes mass characteristics representative of the robotic device. A robotic model can be created based on the CAD model and placed in a simulation environment. In the simulation, the robotic model is instructed to imitate the actions specified by the artist while maintaining balance using a set of rewards.

[0041] Examples of control policy types include permanent actions that do not have a clear beginning and end (e.g., standing), periodic actions that animate repetitive behaviors (e.g., walking or running), and episodic actions that have a predetermined duration (e.g., expressive or heroic pose sequences). For example, a "hero" pose sequence may be one or more animations of the robotic device 100 that resemble or animate a character played by the robotic device 100. A heroic pose sequence may also be an emotional or expressive animation that resembles or displays an emotional state (e.g., sadness, excitement, happiness, etc.). Using different temporary actions, along with modular hardware and RL techniques and simulation, control policies for many different types of animations or desired behaviors can be quickly trained and deployed to robotic devices.

[0042] Go to the diagram, Figure 1An example of a robotic device 100 suitable for control and training using the methods and systems disclosed herein is shown. In some examples, the robotic device 100 may include a quasi-direct drive actuator with an integrated controller, a sensor (e.g., an inertial measurement unit or "IMU"), a custom electronic module to facilitate communication, an onboard real-time or near real-time processor, a battery or power module, and / or a performance function module that integrates a display (e.g., to represent the eyes of a character), a speaker, and other animation functions (e.g., eyes, movable antennas, or other features). In this example, the robotic device 100 has 5 degrees of freedom in each leg 112 and 4 degrees of freedom in the neck 118, enabling it to perform actions such as walking, and to independently move the neck and head and body as needed for different performances. Robots with more or fewer degrees of freedom can also be trained using the methods and systems described herein.

[0043] The robotic device includes performance or output features 126, such as illuminated eyes 122, antenna 104, speakers 120 in the body and in the head, and headlight 106. The performance features 126 may provide additional animation or expressiveness to the robotic device 100 without affecting the overall motion of its body. For example, the performance features 126 may be artistically controlled and synchronized with the overall motion of the robotic device, but without affecting the balance of the robotic device. For example, the eyes 122 may illuminate and / or the antenna 104 may move regardless of any motion of the legs 112 or neck 118 of the robotic device. In some instances, the robotic device may include more, fewer, or no performance features.

[0044] The robotic device has an onboard controller 102 that interfaces to actuator 114 hardware (such as actuators in the legs 112 and neck 118) via microcontroller-driven communications. The robotic device actuators have integrated motor drivers that implement low-level control loops (e.g., to control speed, position, torque, etc.) and have built-in encoders to measure their position (e.g., to generate state estimates 314 of one or more parts of the robotic device 100). The actuators 114 also interface to a communication interface 116. An inertial measurement unit 110 is coupled to the body 124 and interfaces to the communication interface 116. An onboard battery 108 provides power to the actuators 114 as well as the onboard controller 102 and other components.

[0045] The controller 102 runs a controller (e.g., a real-time controller) of the robotic device 100, receives commands for the robotic device 100 to move (e.g., from an external operator), and also receives measurements of the state of the robotic device (e.g., inertial measurements from the IMU 110 and measurements of position, velocity, torque, force, acceleration, etc. from the actuators 114). The controller 102 also receives additional diagnostic information from the actuators 114, such as temperature and battery voltage.

[0046] Figure 2 A reinforcement learning training method 200 for a policy 224 is shown, which includes one or more policies 202 for a robotic device (such as the robotic device 100). In various embodiments, a "policy" refers to a computing entity, executable code, or software program that interacts with an environment to achieve certain goals. Specifically, in reinforcement learning, a policy learns to make decisions by performing behaviors and receiving feedback in the form of rewards or penalties. The purpose of a policy is to learn a policy that maximizes cumulative rewards over time. In some embodiments, the policy does this by exploring the state space of the environment and learning from the results of the policy's behavior, such as effectively refining one or more policies based on trial and error or parameterization of inputs as discussed herein. In various embodiments, a policy represents a learning mechanism that iteratively adjusts the control strategy of the robotic device 100, aiming to optimize the performance of the robotic device when performing a specified task or a set of tasks (e.g., permanent actions, periodic actions, or sporadic actions, etc.). Although the exemplary method 200 depicts a specific sequence of operations, the sequence may be changed without departing from the scope of the present disclosure. For example, some of the depicted operations may be performed in parallel or in a different sequence that does not substantially affect the functionality of the method 200. In other examples, different components of an exemplary apparatus or system implementing method 200 may perform functions substantially simultaneously or in a particular sequence.

[0047] Prior to training the strategy 202, a physical computer model of the robotic device 100 (e.g., a computer-aided model or CAD solid model corresponding to the hardware components) may be used to develop a robotic model 212. The robotic model 212 represents the physics of the robotic device 100, its actuators 114, and the interaction of the robotic device with the environment. For example, the rigid body dynamics of the robotic device 100 may be simulated via the robotic model 212. In order to accurately describe the full dynamics of the physical hardware, custom actuator models 220 may be added to the robotic model 212. Such custom actuator models 220 are configured to represent the actuators used within the physical hardware. The models employed may be derived from first principles, with parameters obtained from system identification experiments of individual actuators 114.

[0048] The robotic device 100 may utilize the time-varying control input g tAt each time step, strategy 202 controls the strategy according to strategy 202, π(a t |s t ,φ t , g t ) produces behavior a t , Strategy 202,π(a t |s t ,φ t , g t ) Suppose there is an observable state s t , and an optional conditional input φ t and g t , where φ t is the phase signal. During training (e.g., in method 200), the robot model 212 generates the next state s t+1 , updates the phase signal, and returns a scalar reward r t = r(s t , a t ,s t+1 ,φ t , g t ). Strategies can be rewarded based on close imitation of artist-specified motion references and dynamic balance.

[0049] The RL training method 200 varies the parameters of the policy 202, which comprises the RL control policy 202 (typically a neural network), to maximize the reward within the environment. In order to collect a sufficient amount of training data, arbitrary robot models are created that accurately describe the physics of the robotic device 100, the actuators 114, the low-level software, and the interaction of the robotic device 100 with the environment. The parameters of these models are obtained from system identification experiments on the hardware. It is possible to randomize some of the parameter values, for example to account for changes in mass properties or the coefficient of friction of the ground on which the robotic device walks. These robot models 212 together provide the system state over time. In some examples, noise can be added to the random values ​​to provide further robustness.

[0050] The method 200 may begin with the generation of one or more random inputs 216 that are provided to the robot model in another operation of the method 200. While the robot model will not be a perfect representation of the real world, an advantage of the training method 200 is the ability to help bridge the "simulation to reality" gap and enable the robotic device 100 to achieve the desired performance when the control strategy is deployed to a real robot. The random inputs 216 may be of one or more types.

[0051] In some embodiments, the first type of stochastic input 216 may include physical characteristics or parameters of a component (e.g., an actuator) of the robotic device 100, such as size, velocity, acceleration, range of motion, mass, moment of inertia, spring rate, or other mass characteristics of the robotic device 100.

[0052] Examples of the first type of random input 216 may also include characteristics of the environment with which the robotic device interacts. For example, the environment has geometric characteristics (e.g., flat versus uneven terrain, obstacles, etc.) and physical characteristics (e.g., surface characteristics such as roughness or coefficient of friction). The simulated robot is trained in a terrain that includes random height field plots, stairs, etc. In some embodiments, the terrain is initialized and fixed during a particular training iteration. The surface characteristics of any part of the simulated environment may vary. For example, the dynamic and / or static coefficient of friction of any surface of the environment may be random. Training the robotic device 100 with this type of random input has the surprising benefit of enabling the robotic device to walk on a variety of surfaces (e.g., carpet, tile, ice, wood piles, stairs, etc.).

[0053] Examples of the second type of random input 216 may include randomizing one or more disturbances (such as input torques or forces) applied to the robotic device. Such disturbances may be imparted to the real robotic device 100 by the device colliding with an obstacle (moving or stationary), being pushed, pulled, or otherwise disturbed, or due to factors such as wind, rain, etc. Training the robotic device 100 with the second type of random input may have the surprising benefit of making the control strategy robust to external disturbances, such that the robotic device may perform the desired action even when disturbed.

[0054] An example of a third type of random input 216 may include a random operator input command. For example, from a remote control 304 ( Figure 3 One or more user inputs of the discussion) can be randomized about the relevant range and passed to the robot model for simulation. Relying on the third type of random input to train the robotic device 100 can have the surprising benefit of enabling the robotic device to balance under arbitrary or even conflicting operator commands.

[0055] During the training method 200, inputs are randomized, for example, within their full input range (e.g., within a predetermined range of values), and the robotic model 212 is perturbed with the randomized inputs 216. By randomizing within several domains (e.g., mass properties and the terrain on which the robotic device 100 is positioned or walking), and applying random perturbation forces and torques to individual components of the robotic device 100, the control strategy 202 can be trained to be robust to disturbances (e.g., pushing and pulling in any direction) and allow the robotic device 100 to walk stably on non-flat terrain (e.g., forest floors or gravel paths).

[0056] In one example, endpoints (e.g., maximum or minimum values) may be set for a given action or parameter (e.g., velocity of an actuator, mass, terrain, etc.), and the policy 202 may be repeatedly trained with respect to a reference action using random inputs 216 within the given endpoints. Thus, without knowing the specific values ​​of the hardware, environment, etc., the policy 202 may be quickly trained with respect to a variety of parameters, resulting in more robust and flexible control of the robotic device 100. The policy 202 is able to robustly execute the intended action under a wide distribution of states and user inputs.

[0057] The stochastic input 216 is converted into a reference action 204, which is transmitted to a strategy 202 including a plurality of strategies 202 and a comparison model 222. In this example, the strategy 202 generates an actuator 114 set point and receives system state data and user input. For example, the strategy 202 receives the stochastic input 216 and generates a command output that is supplied to a low-level control 218 (e.g., a control at the actuator 114 level).

[0058] By accumulating rewards received by the strategy 202 about the history of commands, states, and behaviors and maximizing these rewards, the strategy 202 will learn to match the desired input as closely as possible, while avoiding self-collisions (e.g., head hitting the body), and in addition to maintaining dynamic balance, complying with the limitations of the hardware. More emphasis is placed on complying with hardware limitations, meaning that if the artist's prescribed action will be infeasible (e.g., will result in self-collisions), the strategy 202 can adjust the action to resolve the collision. Because the entire robot motion is commanded by a single controller, non-local motion changes can be performed (e.g., shifting the neck to allow larger head motions), and it also handles non-local collisions generated by the movement of multiple joints. In this context, this functionality achieves a larger full range of motion, but it will also be applicable to other robot motion redirection applications.

[0059] The low-level control 218 generates outputs to the actuator model 220, which models the specific details of the actuator 114 (e.g., mass, velocity, torque, force, motion limits, etc.). The actuator model 220 enhances the physical property simulator. System identification can be performed by mounting a single actuator 114 on a test bench that measures the output torque. Examples of identification parameters are provided in Table 1.

[0060] In some examples, the proportional-derivative motor torque equation for the quasi-direct drive used in this work is calculated by:

[0061] where k P and k D is the gain, a is the joint set point, is the joint velocity, and Add the encoder offset ∈ q The joint position,

[0062] In some examples, the encoder offset is taken from a uniform distribution at the beginning of each RL event And resolve inaccuracies in our joint calibration.

[0063] In some instances, the actuators used in the head do not implement the same PD equation, but instead use

[0064] The derivative term numerically differentiates the set point error.

[0065] In some instances, friction in a joint is modeled as,

[0066] Among them, μ s is the static friction coefficient, where the activation parameter is And μ d is the coefficient of kinetic friction.

[0067] In some examples, the torque generated at the joint can then be calculated by applying a torque limit to τ m and subtract the friction to calculate,

[0068] in and are the minimum and maximum torques of the motor depending on speed. These limits include a constant limit torque τ for braking and low speeds max , and a linear limit above which the available torque decreases above the identified speed When the limit torque is 0Nm, the linear limit and speed intersect.

[0069] In some examples, the measured joint positions reported by the actuator model include encoder offsets (terms that model backlash and noise),

[0070] where b is the total motor backlash and the actuation parameter is τ b At the beginning of each event, b is drawn from a uniform distribution This term models the effect of the motor moving to the backlash side in the same direction as the applied motor torque. For the noise model, we fit a normal distribution with a standard deviation proportional to the motor rotation speed

[0071] In some examples, the reflected inertia of the actuator I m Added to robot models. In rigid body dynamics simulators this is done by setting the skeleton value of the corresponding joint. This value is randomized at the start of each event with a bias of up to 20%.

[0072] User input 310 may also be passed to rigid body simulation 226 (e.g., Figure 7 ), which simulates the kinematics of the robotic device 100. The actuator model 220 outputs the strategic behavior 208 to the robot model 212, which simulates the actions of the robotic device 100. The robot model 212 generates feedback 210 to the low-level control 218. For example, the feedback 210 may be based on the inertial measurement unit 110, the position encoders in the actuators 114, or both. The feedback 210 may be used to adjust future strategic behaviors 208.

[0073] As mentioned, the reference actions 204 are transmitted to the comparison model 222 as well as the policy 202. In the comparison model 222, the policy behavior 208 of the robotic device 100 is compared to the reference actions 204, and rewards / penalties 214 are applied. In a particular implementation, the rewards / penalties 214 are centered around following the artist's directed actions by tracking rewards while ensuring that the robotic device 100 is dynamically balanced by survival rewards. The rewards / penalties 214 are used to develop training data 206 that are transmitted back to the policies 202 to improve their functionality.

[0074] In some examples, training 206 is performed using a proximal policy optimization ("PPO") RL algorithm. PPO can improve the stability and efficiency of the training process compared to other techniques for policy 202 learning to make decisions by directly optimizing policy 202. PPO addresses challenges in training that can arise from the large number of updates used by older methods, which can make the training process unstable. In some embodiments, PPO limits the size of policy 202 updated during training to help ensure small incremental changes.

[0075] In some embodiments, the control strategy 202 may be trained offline, and after such offline training, the control strategy 202 may be deployed (e.g., implemented in a neural network) to the onboard controller 102 to control the motion of the legs 112 and neck 118 of the robotic device via a remote control device 304 (e.g., a joystick, mouse, trackpad, handheld computer, or other input mechanism). The control strategy 202 for a predetermined animation (e.g., episodic motion 406) may be triggered via an additional controller input (e.g., buttons on a remote control 304, such as those similar to a gamepad with an optional display). The control strategy 202 receives as input a high-level user input (e.g., for walking) and the current internal state of the robotic device 100, and outputs a signal (e.g., position, velocity, and / or torque) for the actuator 114 for the next time signal is synchronized. In some embodiments, the control strategy may be trained using multiple parallel simulations of the robotic device 100.

[0076] In some examples, the periodic motion 404 may be interfaced with the animation content by extracting motion references that define a target state of the device over time.

[0077] where p t is the global position of the torso, and θ t is its orientation expressed using quaternions. t and ω t are the linear and angular velocities of the trunk, and q t and are joint positions and velocities. Indicator variables and Limit the contact state of the left and right feet of the device.

[0078] In some examples, for each action type, the generator function f converts the path frame f t , and optionally a phase signal and type-dependent control inputs are mapped to the motion target state

[0079] In some examples, the reference generator for periodic action additionally outputs the phase rate of the drive phase signal This enables the step frequency to be changed during walking as the command changes.For episodic motions 406, the phase rate is determined by the duration of the motion.

[0080] In some examples, a permanent action 402 (e.g., a stand-up action) uses control inputs including head and torso commands that the associated control strategy can execute within the limits of the associated actuators of the robotic device. and The head height and orientation are commanded using the nominal configuration of the . The position and orientation of the torso are expressed in ZYX-Euler angles in path frame coordinates using the height and directional Come and command,

[0081] In some examples, for periodic motion 404, the same head commands as for episodic motion 406 may be used, as well as the two-dimensional velocity vector and angular rate (Both are expressed in path frame coordinates)

[0082] In some examples, for cyclical motion, the head commands are offset relative to the nominal head motion that the animator specifies as part of a cyclical walk cycle. This set of commands enables the artist to control velocity-dependent lower body motion, while the corresponding reference for the neck-head assembly can be changed to cause the robotic device to look in a specific direction during translation.

[0083] In some instances, the path frame plays a role in maintaining consistency during motion transitions. Each artist-designed motion is stored in path coordinates and mapped to world coordinates by a generator f based on the path frame state. During stance, the path frame slowly converges to the center of the feet. During walking, the next frame is calculated by integrating the path velocity command. For episodic motion, the path frame trajectory relative to the starting position is part of the artistic input. To prevent excessive deviation from the path, f t The maximum distance to project from the current torso state.

[0084] Method 200 is independent of the tools or techniques used by the artist to generate motion reference actions. To generate a permanent reference, inverse dynamics can be used to find a method that satisfies the command The pose is set and the optimal values ​​for the remaining degrees of freedom are solved so that the pressure center is located in the middle of the supporting polygon. For periodic motion 404, the artist provides reference gaits at several walking speeds. These gait samples are based on the command The new gait is programmed and combined. A model predictive controller is used to plan the desired center of mass and center of pressure.

[0085] In some examples, to prevent the rate at which reference actions can be generated from slowing down training, the reference generator may be densely sampled and implemented as a reference lookup table as an interpolation of these samples during RL training method 200 .

[0086] In some instances, the reward function combines an action imitation reward with additional regularization and survival rewards, e.g.

[0087] Imitation Reward It can be directly calculated by comparing the simulated posture to the target posture of the device. If the foot contact state matches the reference state, the policy 202 receives an additional reward. To mitigate vibration and unnecessary behavior, a regularization reward can be applied that penalizes joint torques and promotes behavioral smoothness. The survival reward provides a simple goal that motivates the device to maintain balance and prevents it from seeking early termination of events at the beginning of training. Early termination can be applied when the head or torso contacts the ground, or if a self-collision between the head and torso is detected. A detailed description of the weighted reward terms is provided in Table 1, where the hat (^) indicates the distance from the reference posture x t Where appropriate, the current time index t may be omitted, but the spatial index is included.

[0088] Table 1. Weighted reward items

[0089] In some examples, policy behavior 208, a t is the joint position setpoint for the proportional-derivative (PD) controller. In addition to optional action-specific phases and control commands, the strategy 202 receives as input the following states.

[0090] In order to make the state and policy invariant with respect to the global position of the robot, the horizontal (xy plane) position of the torso and directional The available path frame coordinates are expressed as, and the linear velocity of the torso is and angular velocity It can be expressed in body coordinates. Joint positions, joint velocities, and behaviors can be appended at the first two time steps.

[0091] Figure 3 An exemplary method 300 for operating a robotic device 100 utilizing a training strategy 202 is shown. Although the exemplary method 300 depicts a particular sequence of operations, the sequence may be varied without departing from the scope of the present disclosure. For example, some of the depicted operations may be performed in parallel or in a different sequence that does not materially affect the functionality of the method 300. In other examples, different components of an exemplary device or system implementing the method 300 may perform functions substantially simultaneously or in a particular sequence.

[0092] After offline training, the parameters of the policy 202 may be fixed and deployed to the onboard controller 102 of the robotic device 100. Rather than interacting with a simulator, the policy 202 interfaces with a real robotic device 100 (as opposed to a robotic simulation) and an onboard state estimate 314 that provides the state of the robotic device 100 based on inertial measurement unit 110 and actuator 114 measurements (e.g., encoder values).

[0093] The runtime system enables an operator to use a remote control interface of the remote control 304 to manipulate or otherwise provide animation control to the robotic device 100. In some embodiments, the remote control 304 includes various controls with which a user can interact to generate user input 310. For example, the remote control 304 may include one or more of the following: a simulated "thumbstick" that, for example, provides directional navigation for the head or body of the robotic device 100; action buttons located for easy access (e.g., adjacent to the thumbstick); a directional key or "cross pad" that enables precise control of the robotic device 100 and / or for navigating menus (e.g., in instances where the remote control 304 includes an optional display); a trackpad or touch screen that can provide mouse-like precision to user input 310; a trigger / bump that is typically suitable for operation by the index or middle finger, for example, on the shoulder of the remote control 304; a gyroscopic sensor or IMU that is suitable for capturing the movement of the remote control 304 and translating the movement into user input 310; and / or a grip button that is, for example, located on the back of the remote control 304 and allows more input options without moving fingers away from other controls.

[0094] The animation engine 308 maps associated manipulation commands (including policy switches, triggered animation events, and joystick inputs) to policy behaviors 208 , performance functions 126 signals, and / or audio signals.

[0095] The robotic device 100 may be remotely operated via a controller with an integrated game controller, with a separate remote control 304, or alternatively with a PC and a separate game controller. The remote control 304 may have user inputs, such as one or more joysticks for command continuous input, and buttons mapped to control various robotic device 100 functions or trigger actions or other events. Preferably, the remote control 304 communicates with the robotic device 100 via a wireless connection, such as Wi-Fi, radio, Bluetooth, or other suitable wireless connection. Communication between the remote control 304 and the robotic device 100 may be direct, or via an appropriate network, such as a local area network (LAN), a wide area network (WAN), and / or other networks. Commands may flow over a custom radio connection, such as long-range ("LoRa") LoRa technology is suitable for long-range, low-power communications between devices. LoRa technology typically operates in sub-gigahertz radio frequency bands, such as 868 MHz in Europe and 915 MHz in North America.

[0096] The method 300 may begin when a user generates one or more user inputs 310, for example, via a remote control 304. The user inputs 310 may be received by an animation engine 308. The animation engine 308 may compare the user inputs 310 to an animation database 302 that includes animations that the policy 202 is trained on. If the animation is found in the animation database 302, a control command may be issued by the animation engine 308 to the policy 202. Additionally, the user inputs 310 may be mixed with animation content by the animation engine 308 before the inputs are passed to the policy 202 for execution by the controller 102.

[0097] Any of the disclosed strategies 202 may implement automatic action redirection of animation input, which avoids conflicts in output, such as if a commanded animation is not found in the animation database 302 , or the commanded animation would exceed the physical limitations of the robotic device 100 .

[0098] The strategies 202 use the control commands to actuate the appropriate strategy 202 (eg, strategy for permanent action, periodic action, or occasional action). The strategies 202 output instructions to the low-level control 218 (eg, actuator 114 control), which then operates a given actuator 114 .

[0099] In some examples, the policy 202 outputs policy behavior 208 at one speed (e.g., 50 Hz), and the low-level control 218 operates at another higher speed (e.g., 600 Hz). To bridge this gap, a first-order hold can be performed on the policy behavior 208, for example, a linear interpolation of the previous and current policy behaviors 208, followed by a low-pass filter with a cutoff frequency of, for example, 37.5 Hz. The low-level control 218 can also implement the path frame dynamics and phase signal propagation described herein. These low-level control 218 aspects can be implemented identically in both RL and runtime environments.

[0100] The animation engine 308 also coordinates live animation content 312, such as performance functions 126 or other animations based on user input 310, such as live manipulation commands, with animation content retrieved from the animation database 302. Performance functions 126, such as lights and sounds, as well as small functions, such as antenna 104, may be included in the performance of the robotic device and may be specified as part of the artistic input. However, the performance functions 126 may be configured not to change the balance or overall body movement of the robotic device, so they may issue commands directly to the robotic device 100 (e.g., outside of the RL policy 202). Exemplary performance function parameters are shown in Table 2.

[0101] Table 2. Exemplary performance function parameters

[0102] Additionally, the user input 310 may be transmitted to the audio engine 306 to cause the robotic device 100 to play sounds, speak, etc. As with the performance function 126 , the audio engine 306 may issue commands directly to the robotic device 100 and bypass the policy 202 .

[0103] For example Figure 4A-4C As shown in , the strategies 202 and / or robotic device 100 may exhibit a particular type of motion and have an associated strategy (e.g., permanent motion 402, periodic motion 404, or episodic motion 406). In order to achieve a broad and diverse full range of performance, multiple and different approaches for controlling the walking and performance functions of the robotic device may be used. The system may switch between different RL control strategies 202 on the fly (e.g., dynamically) to create a seamless performance. In a specific example, the approaches and strategies include, but are not limited to:

[0104] A permanent action 402 that does not have a clear start and end (e.g., expressive idle). The robotic device 100 maintains balance and responds to measured states and a continuous stream of user inputs 310. For example, a permanent action 402 may be an expressive idle RL control strategy 202 for a standing robotic device 100 that maintains balance and is robust to disturbances. As input to the idle strategy 202, a head pose and a desired body pose may be provided. This allows the head and body 124 of the robotic device 100 to be animated by changing the commanded pose during execution. In yet another example, an idle strategy 202 may allow arbitrary command of the pose of the robotic device (head and body). Such a strategy 202 maintains robustness to disturbances, robustness to model errors, and adheres to the limitations of the robotic device 100 hardware.

[0105] Periodic actions 404 (e.g., expressive walking) are characterized by a periodic phase signal that is passed to the policy. In this mode, the phase signal loops, perhaps indefinitely. For example, the periodic action 404 can be an expressive walking RL control policy 202 for walking a robotic device that matches a desired walking direction, gait, and / or speed while being balanced and robust to disturbances. During training (e.g., in method 200), a program gait created by an artist is provided as a style reference. As an additional input to the walking policy 202, a desired head pose can be provided that enables the robotic device to walk stably while moving its head, and the body motion is automatically adapted to account for the head dynamics.

[0106] Episodic actions (e.g., hero actions) have a predetermined duration. The episodic policy 202 receives a monotonically increasing phase signal. Once the action ends, a transition to a new action is forced. For example, an episodic action RL control policy 202 may be specifically configured to execute an artist-specified input animation while balancing and maintaining robustness to disturbances. The policy 202 may not include additional inputs, i.e., it generates a controller for full-body 124 actions that cannot be changed by additional user input 310. Such an episodic action 406 policy 202 may include transferring artist-created animations to the robotic device 100 by training a specific RL policy 202 (e.g., character-specific animations, such as hero animations) to improve system performance. The actual actions performed by the robotic device 100 may be changed as needed to recover from any disturbances, maintain robustness to model errors, and comply with the limitations of the robotic device 100 hardware.

[0107] These different approaches trade off between adaptability and expressiveness. For highly dynamic, full-body motion, hero animations can be used that perform a sequence of motions specified by the artist. These motions may not change at runtime (e.g., while operating the robotic device 100), but can reach the limitations of the hardware. For less extreme motions, and for motions while walking, the permanent motion 402 and periodic motion 404 strategies 202 allow the motion to be changed at runtime, for example, so the robotic device 100 can look at objects or people in the scene while walking.

[0108] In another example, a method for creating a robust walking policy is disclosed that allows arbitrary command of a posture of a robotic device (e.g., a head). In addition, the method takes as input a prescribed walking style to follow as closely as possible. The method maintains robustness to perturbations, robustness to model errors, and adheres to the limitations of the robotic device hardware.

[0109] For example Figure 5A-5C As shown in , command inputs for expressing walking and expressing idle strategies can be generated by playing pre-authored artist content (e.g., permanent actions 402, periodic actions 404, or occasional actions 406) or by direct animation by an animator. In addition, command inputs can be layered to provide additional expressiveness (e.g., commanding the head to look up, and then performing a head shaking animation). For example Figure 5A , a background animation 502 (e.g., standing up) may be commanded by the user input 310 or as a default for the policy 202. The user may layer one or more additional animations over the background animation 502. For example, a triggered animation 504 (e.g., retracting the neck 118 and looking up) may be layered over the background animation 502. Additionally or alternatively, a remote control animation 506 may be layered over the background animation 502 or the triggered animation 504 (e.g., the robotic device 100 may look down, turn on its headlights 106, and appear to shake its head).

[0110] In some embodiments, the animation engine 308 generates animation commands procedurally based on the following three layers: t : Background animation (such as Figure 5A ), triggering animations (such as Figure 5B ) and animations derived from joystick input (such as Figure 5C ).

[0111] During standing and walking, the performance function 126 and the strategic behavior 208 are calculated by combining event-driven animation playback with live manipulation. To programmatically generate animation states, the robot device configuration can be defined as Extract control input from it. You can define extended animation commands y t =(v t , ct ), where v t Represents the performance function commands as summarized in Table 3.

[0112] Background animation 502: This layer depends on periodic background animation , which may be visible without additional input. Background animation 502 conveys a basic level of activity, including intermittent blinking and antenna 104 movements.

[0113] Trigger Animation 504: This layer blends operator triggered animation clips on top of background animation 502. Trigger Animation 504 can be selected from a library of artist specified clips and mapped to buttons on remote control 304. Trigger Animation 504 ranges from simple "yes-no" animations to complex "scan" clips. Indicates the current state of the triggered animation. The background and trigger target can be mixed as follows:

[0114] The interpolation is configured to be linear for position and joint angles, and spherical linear interpolation ("SLERP") for body orientation can be used. β and α are blending ratios that change over play time. Both ratios rise linearly from 0 to 1 for a given duration at the start of the animation, and return to 0 for the same duration before the end of the animation. The duration can be such that T β = 0.1s and T α = 0.35s, so that the facial expressions associated with the performance function 126 blend naturally faster than the body animation.

[0115] Remote control animation 506: The last layer transforms the blended animation state based on user input 310 from the operator (e.g., from remote control 304) Let u denote the joystick axis, trigger and button modifiers flowing from the controller. When standing, the target robot configuration is calculated as follows:

[0116] in is a nonlinear mapping that modifies the current animation state based on the command input. The joystick axes are mapped to Additional offsets for animated head and torso poses in , modifying the robot's gaze and pose when idle or performing triggered animations. Figure 5C An illustrative example is provided in .

[0117] When walking, the target robotic device 100 configuration is calculated using a mapping similar to the following,

[0118] This similar mapping additionally generates path speed commands for the cyclic strategy. For ease of use, gaze control can be similar; however, the joystick axes used for posture control are remapped to forward, lateral, and turning speed commands during walking. Performance function 126 modulation based on path speed. For example, when the robotic device 100 reaches top speed, the antenna 104 can retract and the eye 122 radius narrows to convey the effort during fast walking.

[0119] Once the animation output y is calculated t , performance function 126 using v t Direct control, while the strategic command signal comes from c t For the head, c t Can be compared with the nominal configuration of the robotic device 100 and the relative head commands extracted and While standing, The mid-torso height and orientation are extracted directly from the target configuration. When walking, the lower body motion is determined by the periodic policy and the commanded path velocity. Note that in both cases, the leg joint positions can be ignored and may not be part of the policy input.

[0120] The combination of user input and live animation content allows for simple, intuitive, and expressive remote operation of the robotic device 100. The remote operator will provide high-level commands that fit the robotic device's environment and the story being told (e.g., "go center stage" or "strike a hero pose"). The superimposed animation content provides subtle movement and details to express emotion and make the character seem alive. For example, a user can have the robotic device 100 look in the direction of a person, and then have this all superimposed with an animated "yes or no" head movement. The animation engine 308 will coordinate the synchronized control of the performance functions 126 (like antennas, eyes, and audio). Similarly, the coordination of the performance functions 126 with the triggered hero animations is also done by the animation engine 308.

[0121] Figure 6 An exemplary method 600 for training an RL policy is shown. Although the exemplary method 600 depicts a particular sequence of operations, the sequence may be changed without departing from the scope of the present disclosure. For example, some of the depicted operations may be performed in parallel or in a different sequence that does not materially affect the functionality of the method 600. In other examples, different components of an exemplary device or system implementing the method 600 may perform functions substantially simultaneously or in a particular sequence.

[0122] According to some examples, the method 600 includes parameterizing inputs at operation 602. Parameterization typically involves selecting possible values ​​for features that may be experienced in a real robotic device 100 or its environment. The parameterized inputs may be one or more inputs as discussed with respect to the method 200. As discussed, endpoints or maximum / minimum values ​​may be selected from a range of possible inputs. Inputs suitable for parameterization may include any physical aspect of the robotic device 100 or its environment that may affect its motion. For example, characteristics of the actuators 114, such as mass, torque, force, speed, number, type, friction, and / or range of motion, may be parameterized. Other parameterizable inputs may include the mass, height, degrees of freedom, number of appendages, etc. of the robotic device 100. Other parameterizable inputs may include disturbances imparted to the robotic device 100 (e.g., an object hitting or pushing the robotic device), or environmental features, such as uneven, slippery, or wet ground, the presence of obstacles, etc.

[0123] According to some examples, the method 600 includes sampling inputs within the parameterized range at operation 604. In operation 604, inputs within the range defined in operation 602 are selected or sampled. The inputs may be randomly selected (e.g., may be random inputs 216 selected at random or in a Monte Carlo process), or may be systematically selected according to a scheme (e.g., at regular intervals, etc.). The selected inputs may also be provided to the robot model 212 for later comparison.

[0124] According to some examples, the method 600 includes providing input to one or more strategies 202 at operation 606. For example, the random input 216 selected in operation 604 may be provided to the strategy 202. The strategy 202 may convert the input into one or more low-level control 218 commands suitable for operating the actuator 114 or simulating the operation of the actuator 114. These low-level control 218 commands ultimately result in the execution of one or more strategy behaviors 208 by the model of the robotic device 100 in the robotic model 212.

[0125] According to some examples, the method 600 includes evaluating the policy-induced policy behavior 208 against the reference action at operation 608 . In operation 608 , the policy behavior 208 generated in operation 606 is compared to the reference action 204 based on the selection input provided to the robot model 212 .

[0126] According to some examples, method 600 includes rewarding strategy 202 at operation 610. For example, in the case where strategy behavior 208 closely matches reference action 204, a reward 214 is generated that reinforces the portion of the strategy responsible for the desired action. In the case where strategy behavior 208 differs from reference action 204, a penalty 214 is generated that weakens the portion of strategy 202 responsible for the undesired action.

[0127] Figure 7A procedure for generating animation content for each of the operating modes (e.g., permanent motion 402, periodic motion 404, and episodic motion 406) of the robotic device 100 is shown. In some examples, embodiments are disclosed that include a design, construction, animation, and control pipeline to rapidly build a customized, free-walking robotic device with an integrated performance function, whereby the control of the robotic device and the control of the performance function 126 are considered as a combined system. Each mode takes a set of user inputs 310 and animation content (e.g., from an animation database 302), and combines these user inputs and animation content into a motion reference motion 204. The reference motion 204 is compared to the actuator 114 output from the policy 202 (which may be processed by one or more of the low-level control 218, the actuator model 220, and / or the rigid body simulation 226). Based on the comparison, one or more rewards / penalties 214 may be generated, such as a tracking reward (which estimates the degree to which the robotic device 100 tracks the reference motion 204), or a survival reward given to the policy for maintaining operation or surviving in the environment for a certain period of time or iteration.

[0128] Figure 8 An example of operating a robotic device 100 using a training policy 202 is shown. The system includes combining a standing controller (e.g., default policy behavior 208) with additional inputs (e.g., user input 310 from a remote controller 304) for expressive gesture control; a walking controller that follows a prescribed style, which has inputs for walking speed and direction and expressive gesture control; and a pre-trained controller for full-body hero animations (e.g., episodic motions 406).

[0129] The user input 310 may be generated by the remote control 304 from an operator or animator controlling the robotic device 100. The user input 310 may be passed to the animation engine 308, which determines the type of animation associated with the user input 310. In addition, the user input 310 may include live animation content 312, such as the operation of one or more of the performance functions 126. The user input 310 is passed to the strategy 202. The strategy 202 generates one or more actuator 114 commands, which are transmitted to the low-level control 218 (e.g., via the communication interface 116), which in turn operates the actuator 114. The actuator 114 and / or the inertial measurement unit 110 may generate state estimation 314 information (e.g., acceleration, velocity, position of one or more of the robotic device 100 or its actuators 114). In addition, as described, the live animation content 312 may be interpreted by the animation engine 308 to actuate one or more of the performance functions 126.

[0130] Fig. 9900 of the robotic device 100. The computing system 900 may also be used for the remote control 304 and / or a computer (e.g., a graphics processing unit) used to perform the training method 200 or other methods herein. For example, the processing element 902 and the memory component 908 may be located at one or several computing systems 900. The present disclosure contemplates any suitable number of such computing systems 900. For example, the computing system 900 may be a desktop computing system, a mainframe, a blade, a grid of computing systems 900, a laptop or notebook computing system 900, a tablet computing system 900, an embedded computing system 900, a system on a chip, a single board computing system 900, or a combination of two or more of these computing systems. Where appropriate, the computing system 900 may include one or more computing systems 900; be single or distributed; span multiple locations; span multiple machines; span multiple data centers; or reside in a cloud, which may include one or more cloud components in one or more networks. The computing system 900 may include one or more processing elements 902, an input / output I / O interface 904, one or more external devices 912, one or more memory components 908, and a network interface 910. Each of the various components may communicate with each other via one or more buses or communication networks, such as a wired or wireless network, e.g., the Internet. Fig. 9 The components in are merely exemplary. In various examples, the computing system 900 may include Fig. 9 Additional components and / or functionalities not shown.

[0131] Processing element 902 may be any type of electronic device capable of processing, receiving and / or transmitting instructions. For example, processing element 902 may be a central processing unit, a microprocessor, a processor, a graphics processing unit, or a microcontroller. In addition, it should be noted that some components of computing system 900 may be controlled by a first processing element 902, while other components may be controlled by a second processing element 902, wherein the first processing element and the second processing element may or may not communicate with each other.

[0132] I / O interface 904 allows a user to input data into computing system 900, and provides input / output for computing system 900 to communicate with other devices or services. I / O interface 904 may include one or more input buttons, a touch pad, a touch screen, or the like.

[0133] External device 912 is one or more devices that can be used to provide various inputs to computing system 900, such as a mouse, microphone, keyboard, touchpad, sensor elements (e.g., thermistor, humidity sensor, light detector, etc.). External device 912 can be local or remote and can be changed as desired. In some examples, external device 912 can also include one or more additional sensors.

[0134] The memory component 908 is used by the computing system 900 to store instructions and / or data for the processing element 902, such as the strategy 202, the actuator model 220, the robot model 212, the animation database 302, user preferences, alarms, etc. The memory component 908 can be, for example, magneto-optical storage, read-only memory, random access memory, erasable programmable memory, flash memory, or a combination of one or more types of memory components.

[0135] The network interface 910 provides communication to and from the computing system 900 to other devices. The network interface 910 includes one or more communication protocols, such as, but not limited to, Wi-Fi, Ethernet, Bluetooth, etc. The network interface 910 may also include one or more hardwired components, such as a Universal Serial Bus (USB) cable, etc. The configuration of the network interface 910 depends on the type of communication desired, and may be modified to communicate via Wi-Fi, Bluetooth, etc. For example, a network interface 910 in the robotic device 100 may communicate with a compatible network interface 910 in the remote control 304.

[0136] The display 906 provides visual output for the computing system 900 and may vary as needed based on the device. The display 906 may be configured to provide visual feedback to the user and may include a liquid crystal display, a light emitting diode screen, a plasma screen, etc. In some examples, the display 906 may be configured to act as an input element for the user through touch feedback, etc. The display may be optional, for example, the robotic device 100 may operate with a headless controller 102 without a display.

[0137] The description of certain embodiments included herein is merely exemplary in nature and is in no way intended to limit the scope of the present disclosure or its application or use. In the included detailed description of the embodiments of the present system and method, reference is made to the accompanying drawings forming part thereof, and the accompanying drawings are shown via an explanation specific to the embodiment, and the described system and method can be practiced in the embodiment. These embodiments are described in sufficient detail to enable those skilled in the art to practice the currently disclosed system and method, and it will be understood that other embodiments may be utilized, and structural and logical changes may be made without departing from the spirit and scope of the present disclosure. In addition, for the purpose of clarity, when certain features are obvious to those skilled in the art, the detailed description of these certain features will not be discussed, so as not to obscure the description of the embodiments of the present disclosure. Therefore, the included detailed description will not be carried out in a limiting sense, and the scope of the present disclosure is limited only by the appended claims.

[0138] From the foregoing it will be appreciated that, although specific embodiments of the invention are described herein for purposes of illustration, various modifications may be made without deviating from the spirit and scope of the invention.

[0139] The details shown herein are presented by way of example and for the purpose of illustrative discussion of the preferred embodiments of the present disclosure, and in order to provide what is considered to be the most useful and easily understood description of the principles and conceptual aspects of the various embodiments of the present invention. In this regard, no attempt is made to show the structural details of the present invention in more detail than is required for a basic understanding of the present invention, and the description using the drawings and / or examples makes it clear to those skilled in the art how several forms of the present invention may be implemented in practice.

[0140] As used herein, and unless otherwise indicated, the terms “a” or “an” mean “one,” “at least one,” or “one or more.” Unless otherwise required by context, as used herein, singular terms shall include pluralities and plural terms shall include the singular.

[0141] Unless the context clearly requires otherwise, throughout the specification and claims, the words "comprises," "including," and the like are to be interpreted in an inclusive sense, as opposed to an exclusive or exhaustive sense; that is, in the sense of "including, but not limited to." Words using the singular or plural number also include the plural and singular number, respectively. Additionally, when used in this application, the words "herein," "above," and "below," and words of similar meaning shall refer to this application as a whole and not to any particular portions of this application.

[0142] All relative, directional and ordinal references (including top, bottom, side, front, back, first, second, third, etc.) are given by way of example to aid the reader in understanding the examples described herein. Unless expressly set forth in the claims, they should not be read as requirements or limitations (particularly with respect to position, orientation, or use). Connection references (e.g., attachment, coupling, connection, linking, etc.) are to be interpreted broadly and may include intermediate components between the connection of elements and relative movement between elements. In this regard, unless expressly set forth in the claims, connection references do not necessarily imply that two elements are directly connected and are in a fixed relationship to each other.

[0143] Of course, it will be recognized that any of the examples, embodiments, or processes described herein may be combined with one or more other examples, embodiments, and / or processes, or separated and / or performed between separate devices or device portions according to the present systems, devices, and methods.

[0144] Finally, the above discussion is intended to be illustrative only of the present system and should not be construed as limiting the appended claims to any particular embodiment or group of embodiments. Therefore, while the present system has been described in particular detail with reference to the exemplary embodiments, it should also be appreciated that many modifications and alternative embodiments may be devised by those skilled in the art without departing from the broader and intended spirit and scope of the present system as set forth in the following claims. Accordingly, the specification and drawings are to be viewed in an illustrative manner and are not intended to limit the scope of the appended claims.

Claims

1. A method for training a robotic device, comprising: parameterizing, via a processing element, an input to the robotic device, wherein the parameterizing includes defining a range of values ​​for the input; generating, via the processing element, a plurality of samples of the parameterized input from within the range of values; Training a control strategy via the processing element, wherein the training comprises: providing the plurality of samples to the control strategy, wherein the control strategy is adapted to operate an actuator of the robotic device, and generating, via the processing element, a policy action using the control policy; transmitting the strategic behavior to a robot model, wherein the robot model includes a physical model of the robotic device; and The trained control policy is deployed to an onboard controller for the robotic device.

2. The method according to claim 1, wherein: The input to the robotic device includes at least one of: the mass, torque, force, speed, number, type, or range of motion of components of the robotic device; imparting a disturbance to said robotic device; Operator command; or Environmental characteristics.

3. The method according to claim 1, wherein: The control strategy is adapted to cause the robotic device to perform at least one of permanent motion, periodic motion, or sporadic motion.

4. The method according to claim 1, wherein: The training further includes: simulating, via the processing element, motion of the actuator using the physical model; comparing, via the processing element, the simulated motion of the actuator to a reference motion of the actuator, wherein the reference motion is based on the plurality of samples; and The control strategy is rewarded, via the processing element, based on the comparison.

5. A method of operating a robotic device, comprising: receiving user input at a processing element, wherein the processing element is in communication with one or more actuators of the robotic device; comparing, via the processing element, the user input with an animation database; selecting, via the processing element, an animation from the animation database based on the comparing; actuating, via the processing element, a control strategy for the selected animation, wherein the control strategy is trained by a reinforcement learning method; generating, via the processing element, low-level controls suitable for controlling actuators of the robotic device; The robotic device actuators are controlled via the low-level control.

6. The method of operating a robotic device according to claim 5, wherein: The user input includes commands to actuate a performance function of the robotic device.

7. The method of operating a robotic device according to claim 6, wherein: The performance functionality is independent of the robotic device actuators.

8. The method of operating a robotic device according to claim 6, wherein: The performance function includes actuating at least one of a light, a movable antenna, eyes, or a sound of the robotic device.

9. The method of operating a robotic device according to claim 5, wherein: The reinforcement learning method comprises: parameterizing an input to the robotic device via a second processing element, wherein the parameterizing includes defining a range of values ​​for the input; generating, via the second processing element, a plurality of samples of the parameterized input from the range of values; providing the plurality of samples to a control strategy via the second processing element; generating, via the second processing element, a policy behavior using the control policy and transmitting the policy behavior to a robot model, wherein the robot model includes a physical model of the robotic device; simulating, via the second processing element, motion of an actuator of the robotic device using the physical model; comparing, via the second processing element, the simulated motion of the robotic device actuator to a reference motion of the robotic device actuator, wherein the reference motion is based on the plurality of samples; and The control policy is rewarded, via the processing element, based on the comparison of the simulated action of the robotic device actuator to the reference action.

10. The method of operating a robotic device according to claim 9, wherein: The input to the robotic device includes at least one of: the mass, torque, force, speed, number, type, or range of motion of components of the robotic device; imparting a disturbance to said robotic device; Operator command; or Environmental characteristics.

11. The method of operating a robotic device according to claim 9, wherein: The control strategy is adapted to cause the robotic device to perform at least one of permanent motion, periodic motion, or sporadic motion.

12. The method of operating a robotic device according to claim 5, wherein: The selection animation includes one or more of a background animation or a trigger animation, and the method of operating the robotic device further includes laminating at least one of the background animation or the trigger animation with a remote control animation.

13. The method of operating a robotic device according to claim 12, wherein: The remote control animation is based on the user input received from a remote control.

14. A robotic device comprising: Multiple modular hardware components; a processing element in communication with the plurality of modular hardware components; A plurality of control strategies are trained by a reinforcement learning method to control the plurality of modular hardware components.

15. The robotic device according to claim 14, wherein: The user input includes commands to actuate a performance function of the robotic device.

16. The robotic device according to claim 15, wherein: The performance functionality is independent of the robotic device actuators.

17. The robotic device according to claim 14, wherein: The selection animation includes one or more of a background animation or a trigger animation, and the robot device is further operable by laminating at least one of the background animation or the trigger animation with a remote control animation.

18. The robotic device according to claim 14, wherein: The reinforcement learning method comprises: parameterizing an input to the robotic device via a second processing element, wherein the parameterizing includes defining a range of values ​​for the input; generating, via the second processing element, a plurality of samples of the parameterized input from within the range of values; providing the plurality of samples to a plurality of control strategies via the second processing element; generating, via the second processing element, a strategy behavior using one of the plurality of control strategies and transmitting the strategy behavior to a robot model, wherein the robot model includes a physical model of the robotic device; simulating, via the second processing element, motion of the actuator using the physical model; comparing, via the second processing element, the simulated motion of the actuator to a reference motion of the actuator, wherein the reference motion is based on the plurality of samples; and The one of the plurality of control strategies is rewarded, via the processing element, based on the comparison of the simulated action of the actuator with the reference action.

19. The robotic device according to claim 18, wherein: The input to the robotic device includes at least one of: the mass, torque, force, speed, number, type, or range of motion of components of the robotic device; imparting a disturbance to said robotic device; User input; or Environmental characteristics.

20. The robotic device according to claim 18, wherein: The plurality of policies are adapted to cause the robotic device to perform at least one of permanent actions, periodic actions, or sporadic actions.

21. The robotic device according to claim 14, wherein: The robotic device is capable of the following operations: receiving user input at the processing element; comparing, via the processing element, the user input with the animation database; selecting, via the processing element, an animation from the animation database based on the comparing; actuating, via the processing element, a control strategy from among the plurality of control strategies for the selected animation; generating, via the processing element, a low-level control suitable for controlling the actuator; The plurality of modular hardware components are controlled via the low-level control.

22. A method of controlling a robotic device, comprising: generating, via a first training control policy executed by a processing element, a first policy behavior suitable for performing a permanent action; generating, via a second training control policy executed by the processing element, a second policy behavior suitable for performing a periodic action; generating, via a third training control strategy executed by the processing element, a third policy behavior suitable for performing a sporadic action; as well as The first strategic behavior, the second strategic behavior, and the third strategic behavior are deployed via the processing element to a plurality of actuators of the robotic device, wherein the plurality of actuators are adapted to perform the permanent action, the periodic action, and the sporadic action.

Images