How to train a neural network and use it to stabilize a bipedal robot

Abstract

The invention relates to a method for training a neural network for stabilizing a bipedal robot (1) exhibiting multiple degrees of freedom actuated by actuators, characterized in that it comprises the implementation by data processing means (11) of a server (10) of the following steps: (a) applying a series of pushes to a virtual twin of the robot (1) in a simulation, (b) executing a reinforcement learning algorithm on said neural network, in which the neural network provides commands to said actuators of the virtual twin of the robot (1) in order to maximize a reward representing the recovery of the virtual twin of the robot (1) from each push.

Description

[Technical field]

[0001] The present invention relates to the field of exoskeleton robotics.

[0002] More specifically, it relates to a method for training a neural network and using the neural network to stabilize a bipedal robot, such as an exoskeleton. [Background technology]

[0003] In recent years, walking aids called exoskeletons have appeared for people with significant motor disabilities such as paraplegia. They are external robotic devices that are "worn" by the operator (human user) thanks to a system of fasteners that link the movements of the exoskeleton with his own. Exoskeletons of the lower limbs have several joints, generally at least at the knee and hip levels, to reproduce the locomotion. Actuators allow these joints to be moved, which in turn move the operator. An interface system allows the operator to give commands to the exoskeleton, and a control system converts these commands into commands for the actuators. Sensors generally complete the device.

[0004] These exoskeletons represent an advancement compared to wheelchairs, since they allow the operator to stand and walk on their own feet. The exoskeleton is no longer limited by wheels and can, in theory, evolve in most non-flat environments. Wheels, unlike legs, do not allow the user to clear significant obstacles such as steps, stairs or obstacles that are too high.

[0005] However, achieving dynamic stability of such exoskeletons remains a challenge. Vertical postures are inherently unstable, so continuous feedback control is required to maintain balance. Trajectory planning for bipedal robots has been successfully solved through whole-body optimization, and stable walking has been achieved in the applicant's exoskeleton Atalante, on perceptually flat ground and in the absence of significant obstacles, by using advanced conventional planning approaches. See, for example, U.S. Pat. Nos. 5,993,313 and 5,103,631.

[0006] However, the so-called "emergency recovery" is still an open problem. More specifically, although the exoskeleton is perfectly capable of maintaining a smooth and stable trajectory while walking, it may lose balance when subjected to a strong and unexpected disturbance, such as a push, which may result in injury to the human operator or damage to the robot.

[0007] For small perturbations, in-place recovery strategies that control the Center of Pressure (CoP), center of mass angular momentum, or use foot tilt are sufficient.

[0008] To deal with stronger perturbations, controllers based on Zero-Moment Point (ZMP) trajectory generation have been proposed along with Model Predictive Control (MPC) methods to control the ZMP, but their efficiency in practice is very limited.

[0009] Therefore, there remains a need for a purely reactive controller for standing push recovery on an exoskeleton that can be used as a last resort in emergency situations and feature diverse, human-like balance strategies while ensuring predictable, safe and smooth behavior. [Prior art documents] [Patent documents]

[0010] [Patent Document 1] Patent application EP3568266 [Patent Document 2] Patent application FR3101463 [Non-patent literature]

[0011] [Non-Patent Document 1] T. Gurriet, S. Finet, G. Boeris, A. Duburcq, A. Hereid, O. Harib, M. Masselin, J. Grizzle, and AD Ames, “Towards restoring locomotion for paraplegics: Realizing dynamically stable walking on exoskeletons,” 2018 IEEE International Conference on Robotics and Automation (ICRA), 2018, p.2804-2811. Summary of the Invention [Means for solving the problem]

[0012] To these ends, the invention provides, according to a first aspect, a method for training a neural network for stabilizing a bipedal robot exhibiting several degrees of freedom actuated by actuators, characterized in that it comprises the implementation by the data processing means of a server of the following steps: (a) A series of pushes are applied to the robot's virtual twin in a simulation. (b) running a reinforcement learning algorithm on said neural network, where the neural network provides commands to said actuators of the robot's virtual twin to maximize a reward representing the recovery of the robot's virtual twin from each push.

[0013] Preferred, but non-limiting, features of the present invention are as follows:

[0014] The simulation continues for a predetermined duration, and steps (a) and (b) are repeated for a number of simulations.

[0015] A push is applied periodically over the duration of the simulation, with a constant magnitude force being applied for a given duration.

[0016] The push is applied to the pelvis of the robot's virtual twin, with an orientation sampled from a spherical distribution.

[0017] During step (b), at least one terminal condition on the virtual twin of the robot is enforced.

[0018] The end conditions are selected from among a range of exoskeleton pelvis positions and / or orientations, a minimum distance between the robot's feet, a range of actuator positions and / or velocities, a maximum difference from expected trajectory, a maximum recovery duration, and a maximum power consumption.

[0019] The simulation outputs the state of the robot's virtual twin as a function of pushes and commands provided by the neural network.

[0020] The robot includes at least one sensor for observing a state of the robot, and the neural network takes as input in step (b) the state of the virtual twin of the robot output by the simulation.

[0021] The neural network provides target positions and / or velocities of the actuators as commands, and a control loop mechanism determines the torque to be applied by the actuators as a function of said target positions and / or velocities of the actuators.

[0022] The neural network provides commands at a first frequency and the control loop mechanism provides a torque at a second frequency that is higher than the first frequency.

[0023] Step (b) involves performing temporal and / or spatial regularization to improve the smoothness of the neural network's commands.

[0024] The method includes the step (c) of storing the trained neural network in a memory of the robot.

[0025] The bipedal robot is an exoskeleton that accommodates a human operator.

[0026] According to a second aspect, the invention proposes a method for stabilizing a biped robot exhibiting several degrees of freedom actuated by actuators, characterized in that it comprises the following steps: (d) providing commands to said actuators of the robot using a neural network trained using the method according to the first aspect.

[0027] According to a third aspect, the invention proposes a system comprising a server and a bipedal robot exhibiting multiple degrees of freedom actuated by actuators, each having data processing means, characterized in that said data processing means are adapted to implement, respectively, the method for training a neural network for stabilizing a robot according to the first aspect and the method for stabilizing an exoskeleton according to the second aspect.

[0028] According to a fourth and fifth aspect, the invention proposes a computer program product comprising code instructions for performing the method for training a neural network for stabilising a robot according to the first aspect or the method for stabilising a robot according to the second aspect; and a computer readable medium having stored thereon a computer program product comprising code instructions for performing the method for training a neural network for stabilising a robot according to the first aspect or the method for stabilising a robot according to the second aspect. [Brief description of the drawings]

[0029] Other characteristics and advantages of the invention will become apparent from reading the following description of preferred embodiments, which description is given with reference to the accompanying drawings, in which: [Figure 1] 1 shows an architecture for implementing the methods according to the invention. [Diagram 2] 1 represents an exoskeleton for use in methods according to the present invention. [Diagram 3] FIG. 1 illustrates preferred embodiments of the methods according to the present invention. [Figure 4] Represents a virtual twin of the exoskeleton in the simulation. [Diagram 5] 1 is an overview of the proposed control system in the methods according to the invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] Architecture According to two complementary aspects of the invention, it is proposed that: A method for training a neural network, in particular a "forward propagation" type neural network (FNN, Feedforward Neural Network), for stabilizing a bipedal robot 1; and A method for stabilizing a bipedal robot 1 (using a neural network advantageously trained thanks to the method described above).

[0031] These two types of processes are implemented within the architecture as shown in FIG.

[0032] The server 10 is a training server (implementing the first method) and the robot 1 implements the second method, in other words, the robot 1 directly applies said neural network, once trained, to stabilize the exoskeleton (in particular by performing an "emergency recovery" as explained above).

[0033] Although it is entirely possible for the server 10 to be integrated into the robot 1, in practice the server 10 is a remote server.

[0034] The server 10 is typically a computing device connected to the robot 1 via any kind of network 20, such as the Internet network, for data exchange, although in practice, once the neural network has been trained and deployed, i.e. provided to the robot 1, communication may be interrupted, at least intermittently.

[0035] Each of the server 10 and the exoskeleton 1 comprises data processing means 11, 11' of the processor type (in particular the data processing means 11 of the server 10 has high computing power, since training is long and complex compared to the simple application of a learned neural network to new data, called inference) and, where appropriate, data storage means 12, 12' such as a computer memory, for example a hard disk.

[0036] It should be noted that there may be multiple robots 1 each connected to the server 10.

[0037] By biped robot 1 is meant a multi-jointed mechanical system with two legs that can be actuated and commanded, more preferably an exoskeleton as shown in Fig. 2. This exoskeleton is more specifically a biped robot that receives a human operator so that his lower limbs are respectively integrated with the legs of the exoskeleton 1. The integration can be ensured in particular thanks to straps.

[0038] In the following specification we describe the preferred example of an exoskeleton, which is the most difficult type of bipedal robot to stabilize, but our method is efficient for any other bipedal robot, such as a humanoid robot.

[0039] "Stabilizing" the exoskeleton 1 means maintaining its balance as much as possible and preventing it from falling over thanks to emergency recovery, especially in case of strong perturbations. "Emergency recovery" means reflex movements of the exoskeleton 1 that should be utilized in case of emergency (i.e. used as a fallback of last resort) in response to a perturbation, like tilting the pelvis or taking a step. An emergency recovery is successful when the exoskeleton 1 returns to a safe posture, especially a stationary posture if it was not walking before the perturbation. Note that the recovery should be as "static" as possible (i.e. with minimal movements), but in case of strong perturbations it may require several steps to prevent a fall.

[0040] The exoskeleton 1 is provided with, for each leg, a foot structure including a support surface capable of supporting the foot of the leg of a person wearing the exoskeleton.

[0041] The exoskeleton 1 has multiple degrees of freedom, i.e. joints that can be transformed (typically via rotation), i.e. that can be moved relative to one another. Each of these can be either "actuated" (with actuation) or "non-actuated" (without actuation).

[0042] An actuated degree of freedom denotes a joint with an actuator controlled by the data processing means 11', i.e. this degree of freedom is controlled and can be acted upon. Conversely, a non-actuated degree of freedom denotes a joint without an actuator, i.e. this degree of freedom follows its own dynamics and the data processing means 11' does not directly control it (but a priori there is an indirect control via other actuated degrees of freedom). In the example of Fig. 2, the contact of the heel with the ground is point-like, so that the exoskeleton 2 is free to rotate about this contact point. The angle between the heel-to-hip axis and the vertical constitutes a non-actuated degree of freedom.

[0043] In a preferred embodiment, the exoskeleton 1 comprises six actuators in each leg (i.e. 12 actuators) and a set of sensors called "basic proprioceptive sensors", such as means for detecting foot impact on the ground 13 and / or at least one inertial measurement unit (IMU) 14. In Fig. 2 there is a "pelvis IMU" 14 on the rear of the exoskeleton, but there may be further IMUs such as left / right tibia IMUs and / or left / right foot IMUs. It is noted that there may be further sensors, in particular encoders (measuring actuator position) for each actuator.

[0044] The data processing means 11' is a computing device (typically a processor) suitable for processing instructions and generating the intended commands for the various actuators, preferably embedded in the exoskeleton 1, although it may be external if the exoskeleton 1 is "remotely controlled". As explained, said data processing means 11' of the exoskeleton 1 is configured to implement said neural network in order to control the degrees of freedom that are actuated (through actuators). Said actuators may be electric, hydraulic etc.

[0045] The operator may be equipped with a sensor vest 15 that detects the configuration of the torso (torso orientation). The direction the operator turns their chest is the direction they want to walk, and their speed is given by how hard they push their chest forward (how much they bend it).

[0046] The present application is not limited to any exoskeleton architecture 1 and takes as examples those described in WO 2015 / 140352 and WO 2015 / 140353.

[0047] However, one skilled in the art will know how to adapt the method to any other mechanical architecture.

[0048] Training Method According to a first aspect, the invention proposes a method, executed by data processing means 11 of a server 10, for training a neural network for stabilizing an exoskeleton 1 accepting a human operator and exhibiting multiple degrees of freedom of actuation.

[0049] Although the use of neural networks for stabilizing bipedal robots is known, the present invention proposes a unique training of such neural networks.

[0050] More precisely, said neural network is trained by running a reinforcement learning algorithm in a simulation specifically designed to trigger emergency recovery. In other words, this neural network simply acts as a last resort solution to stabilize the exoskeleton 1 in case of an emergency. It has no other role, such as applying trajectories.

[0051] Training a neural network involves generating efficient "policies" for the network to act in response to perturbations. θ This means defining appropriate values ​​for the neural network parameters θ so as to implement

[0052] As explained, the neural network could for example be a memoryless feedforward neural network with two hidden layers with 64 units each, an activation layer like LeakyReLU, and a linear output layer. Note that alternative architectures are also possible, such as with convolutional layers. For better generalization, an architecture can be chosen that minimizes the number of parameters needed to obtain a satisfactory performance. This avoids overfitting, which leads to a more predictable and robust behavior of the robot, even for unseen states.

[0053] As represented in Fig. 3, in a first step (a), the method comprises applying a series of pushes to a virtual twin of the exoskeleton 1 in said simulation. By "virtual twin" is meant a virtual replica of the exoskeleton 1 in the simulation, with the same actuators, sensors, properties etc. The human operator is advantageously considered to be rigidly fixed to the exoskeleton 1. In this respect, the exoskeleton-human system can be seen as a humanoid robot after integrating the respective mass distributions.

[0054] The open source simulator Jiminy can for example be used as a simulation environment (see Figure 4). It is tailor-made for control robotics problems and is designed for realistic simulation of legged robots. Jiminy includes motor inertia and friction, sensor noise and delays, accurate constraint-based ground reaction forces, and an advanced mechanical flexibility model.

[0055] By "push" we mean an external force applied to virtual twin exoskeleton 1 to disturb its stability. In order to learn sophisticated recovery strategies, the external push in the learning environment needs to be carefully designed: it must be strong enough to require stepping out at times, but pushing too hard will hinder learning. Furthermore, using curriculum learning to gradually increase the maximum force is risky as the policy is prone to getting stuck in local minima.

[0056] In a preferred embodiment, the series of pushes periodically applies a constant magnitude force of a given (short) duration to the pelvis, where the orientation is sampled from a spherical distribution, in other words, all the pushes in the series have approximately the same magnitude force and duration, but different orientations.

[0057] For example, these pushes can be applied every 3 seconds with a jitter of 2 seconds to learn to recover from successive pushes without overfitting to a fixed push scheme. These pushes are bell-shaped rather than uniform for numerical stability, have a peak magnitude of Fmax=800N, and are applied for a duration of 400ms.

[0058] Then, in step (b), processing unit 11 runs a reinforcement learning algorithm on said neural network, which provides commands to (i.e. controls) said actuators of the virtual twin of exoskeleton 1 so as to maximize a reward representing the recovery of said virtual twin of exoskeleton 1 from each push. In other words, the better the recovery, the higher the reward.

[0059] "Reinforcement Learning" (RL) refers to a machine learning paradigm in which an agent (a neural network) takes actions (emergency recovery) in an environment (a simulation) to maximize reward. The goal of RL is for the agent to learn a "policy", i.e. a function that defines the actions to take as a function of its state. Reinforcement learning differs from supervised learning in that it does not require labeled input / output pairs, nor does it require explicit correction of suboptimal actions. Instead, the focus is on finding a balance between exploration (unexplored state-action space) and exploitation (of current knowledge).

[0060] To find the appropriate action to perform, RL can now use any suitable optimization technique, in particular policy gradient methods and preferably Trust Region Policy Optimisation (TRPO), or even Proximal Policy Optimisation (PPO), which simplifies the optimization process with respect to TRPO while efficiently preventing “destructive policy updates” (updates that deviate too much from the policy).

[0061] Note that RL in simulations has already been used to train neural networks to control bipedal robots, and the transfer to reality has already been successfully achieved (i.e., application of the trained neural network to a real robot), but only for the walking task, i.e., reproducing the corresponding reference movements from a gait library. See non-patent document 2. In this document, the robot "Cassie" happens to show stability against some pushes, but this is mainly due to the design of Cassie - no upper body and almost weightless legs - and the neural network has not actually been trained for emergency recovery, and the same level of performance cannot be expected for an exoskeleton. [Non-Patent Document 2] Z. Li, X. Cheng, XB Peng, P. Abbeel, S. Levine, G. Berseth, and K. Sreenath, “Reinforcement learning for robust parameterized locomotion control of bipedal robots”, CoRR, vol. abs / 2103.14295, 2021

[0062] As described below, the neural network preferably controls a certain degree of freedom of said actuation of the virtual twin of the exoskeleton 1 by providing as commands target torques τ to be applied by the actuators. Note that the neural network may directly output torques or indirectly output other parameter(s), such as target positions / velocities of the actuators, from which torques can be calculated in a known manner using a control loop mechanism (such as a proportional-derivative controller as described below).

[0063] In a preferred embodiment of the present invention, the neural network and the environment interact according to FIG.

[0064] The state of the system at time t t is the pelvis position q of exoskeleton 1 x,y,z :=[q x ;q y ;q z ] T , orientation q ψ,ν,φ , linear velocity

number

number

number

[0065] Observation t ∈O⊂R n is calculated from the measurements of sensors 13, 14. This observation space can reach 49 dimensions (smaller than the state space, which can reach 68, since not all states are observable). The target motor position from the last time step

number

number

[0066] s0 is the state in which the agent starts an episode (i.e., a simulation) and can vary. The initial state distribution ρ0(s0):S→O defines the diversity of states s0 in which the agent can start. Choosing this distribution well is essential to explore a wide variety of states and discover robust recovery strategies. Indeed, to be efficient, we need to sample as many recoverable states as possible. Naive random sampling would either instantly trigger a terminal condition (see below) or lead to many infeasible configurations that are not recoverable. Therefore, the best solution is to sample s0 normally distributed around a diverse set of reference trajectories. This allows us to explore many different but feasible configurations.

[0067] The neural network determines the target position of the actuator.

number

[0068] As explained, the neural network is preferably configured to output some "high-level" targets (target positions and / or velocities of the actuators), which are then forwarded to a distributed low-level control loop mechanism (in particular a PD controller) that determines the torques τ at a higher second frequency (e.g. 100Hz). Such a hybrid control architecture improves efficiency, robustness and transferability on real hardware compared to directly predicting the torques. Decoupled control of each joint is well known to be robust to model uncertainties. Furthermore, these controllers can be tuned to trade off tracking accuracy and compliance, smoothing out oscillations and errors in the predicted action.

[0069] The target torques are applied to the virtual twin of the exoskeleton 1 in the simulation to update the state. The virtual twin “moves” and, as a result, the simulator outputs updated positions and velocities of the actuators to the PD controller and, most importantly, updated observations. t Output.

[0070] Thus, it is understood that steps (a) and (b) are actually simultaneous. The neural network attempts to perform emergency recovery for each push of the sequence by appropriately controlling the actuators.

[0071] The simulation preferably lasts for a predetermined duration, such as 60 seconds. Steps (a) and (b) are typically repeated for multiple, and potentially thousands or millions, of simulations called "episodes." Thus, the environment is reset for each new simulation. If the virtual twin falls over in one simulation, the virtual twin can be forced to reset or made to wait until the start of the next simulation.

[0072] Furthermore, at least one terminal condition on the virtual twin of the exoskeleton 1 is preferably enforced during step (b). Terminal conditions are considered as hard constraints, such as hardware limitations or safety. Moreover, they increase the robustness of the convergence by preventing getting into bad local minima.

[0073] Each end condition is advantageously selected from among a range of positions and / or orientations of the pelvis of the exoskeleton 1, a minimum distance between the feet of the exoskeleton 1, a range of actuator positions and / or velocities, a maximum difference from an expected trajectory, a maximum recovery duration, and a maximum power consumption. Preferably, more than one of these end conditions (possibly all of them) are enforced. 1) Range of pelvic positions and / or orientations. Pelvic movement is constrained so as not to scare the user and those around. For example, the following ranges may apply: q z >0.3m;-0.4rad ψ <0.4rad;-0.25rad φ <0.7rad 2) Minimum distance between the feet. For safety reasons, it is necessary to avoid foot collisions that could injure the human operator and damage the robot. For example, the following formula can be applied: D(CH r ;CH l )>0:02; where CH r;l is the convex hull of the left and right footprints, and D is the Euclidean distance. 3) The range of actuator positions and / or velocities. Hitting the mechanical bounds of actuator i at high velocities will result in catastrophic damage. In other words, the following equation applies:

number

number

number

number

[0074] Reward formation As mentioned before, RL maximizes a reward that represents the recovery of the virtual twin of exoskeleton 1 from each push. The reward can be seen as a score calculated for each simulation by the training algorithm and is sought to be as high as possible. The idea is that if the virtual twin exoskeleton performs an efficient recovery, i.e. maintains balance, it has a high reward and otherwise, e.g. falls, it has a low reward. In other words, the reward increases with the efficiency of the stabilization.

[0075] The rewards preferably include a set of reward components to obtain natural behavior that is pleasant for the user and to provide insight into how to maintain balance.

[0076] Reward can also be used as a means to trigger recovery as late as possible, as it is considered an emergency strategy of last resort.

[0077] Note that to be general, rewards can also be used for training a reference gait (i.e., following a reference trajectory).

[0078] In the case of multiple reward components, the total reward can be a weighted sum of the individual objective quantities: r t =Σ i ω i K(r i ) where r i is the reward component, and ω i are the weights and K is a kernel function intended to scale them uniformly. This is a radial basis function (RBF) (K(r i )=exp(-κr i 2 )∈[0,1]), etc.

[0079] The gradients become 0 at both very small and large values ​​as a side effect of this scaling. The cutoff parameter κ is used to adjust the operating range of each reward component.

[0080] There are many possible rewards (or reward components) representing recovery from each push of said virtual twin of exoskeleton 1, including something as simple as simply counting the number of seconds until the exoskeleton collapses. One skilled in the art is limited to a particular choice of reward(s) so long as they evaluate the ability of the virtual twin of exoskeleton 1 to recover from each push.

[0081] In the RBF example, K is r i = 0, i.e., the interval R + Note that k decreases in k = 1 k . This means that the reward component must in this case be chosen to decrease with respect to the ability to recover from each push (i.e., the better the recovery, the lower the reward component value, since the reward component value is formulating the error). This is the case in the following examples, but those skilled in the art can perfectly well use a differently constructed reward component that is an increasing function, using an appropriate increasing K function.

[0082] In the preferred embodiment, there may be three classes of rewards.

[0083] 1) Reference dynamics: We can extract a set of independent high-level features and define a reward component for each of them to facilitate arriving at a comfortable and stable resting pose for push recovery. Odometry. As explained, with regard to the safety of the push recovery step, the change in odometry is related to the pelvic position p b should be limited as much as possible, i.e. the remuneration component should be

number

number

number

number

number

[0084] 2) Transient dynamics: after a strong perturbation, a recovery step is performed to prevent tipping over in any case. Foot placement. The idea is that as soon as the Center of Pressure (CP) moves outside the support polygon, it acts by moving the pelvis towards the CP, encouraging it to return under the foot. For example, the reward component

number

number

[0085] 3) Safety and comfort: safety must be guaranteed during the operation of the exoskeleton 1. Comfort has the same priority for medical exoskeletons to enhance rehabilitation. Balanced contact forces. Even distribution of weight on both feet is important in natural standing. For example, the reward component

number

number

[0086] At the end of step (b), a neural network is obtained that is able to command the actuators of the exoskeleton 1 to recover from a real push.

[0087] Transition to reality In a next step (c), the trained neural network is stored in the memory 12' of the exoskeleton 1, typically from the memory 12 of the server 10. This is called "transitioning to reality", because up until now the neural network only recovered from pushes in the simulation, and once stored in the memory 12' the neural network is expected to provide commands to the real actuators of the real exoskeleton 1 to perform a real recovery of the exoskeleton 1 from an unexpected push.

[0088] Thus, in a second aspect, a method is proposed for stabilising an exoskeleton 1 accepting a human operator and exhibiting multiple degrees of freedom actuated by actuators, comprising a step (d) of providing commands to said actuators of the exoskeleton 1 using a neural network trained using the method according to the first aspect.

[0089] As explained, neural networks can be generated far in advance and transferred to multiple exoskeletons 1.

[0090] The second method (the stabilization method) can be run many times during the lifetime of the exoskeleton 1 and does not require new transfers, although the neural network may be updated if an improved policy becomes available.

[0091] Policy Regularization In many prior art attempts, the reality gap is largely overlooked and simulation results are rarely transferred to real hardware, failing to achieve real world success either because physics is oversimplified and hardware limitations are ignored, or because regularities are not guaranteed and unexpected and dangerous behaviors may occur.

[0092] Our training methods enable safe and predictable behavior, which is crucial for autonomous systems evolving in human environments, such as bipedal exoskeletons.

[0093] In the specific case of medical exoskeletons, comfort and smoothness become even more crucial: vibrations can cause anxiety and, more importantly, can lead to injuries over time, as patients' bones are fragile.

[0094] Therefore, step (b) preferably comprises performing a temporal and / or spatial regularization of the policy in order to improve the smoothness of the neural network's commands, also called "conditioning" the policy.

[0095] Typically in RL, smoothness can be promoted by adding a regularizer as a reward component, such as minimizing motor torque, motor speed, or power consumption. However, components in the reward function are not guaranteed to be optimal, since they have a small contribution in the actual loss function of the RL learning algorithm.

[0096] In contrast, injecting regularization as an additional term into the loss function directly gives control over how much enforcement is applied during training. [Non-Patent Document 3] S. Mysore, B. El Mabsout, R. Mancuso, and K. Saenko, “Regularizing action policies for smooth control with reinforcement learning”, 12 2020

[0097] In a preferred embodiment, the use of temporal and spatial regularization promotes smoothness of the neural network's learned state-to-action mapping, for example using the following terms:

number

[0098] These terms can be added to the objective function, possibly using weights, i.e., for PPO (see below): L(θ)=L PPO (θ)+λ T L T (θ)+λ S L T (θ) σ S is based on expected measurement noise and tolerances, and so its power is limited to robustness issues. However, its true power becomes apparent when smoothness is additionally used to shape and enforce regularities in policy behavior.

[0099] By choosing an appropriate standard deviation, a set of minimal but efficient recovery strategies can be learned in addition to being robust. Also, the responsiveness and reactivity of the policy on real devices can be tuned. For this purpose, σ S is typically comprised between 0.1 and 0.7.

[0100] A further possible improvement is the introduction of the L1 norm in the temporal regularization: this still ensures that the policy reacts only when necessary and recovers as fast as possible, while also avoiding penalizing too harshly peaks that are beneficial for surviving some pushes and smoothing out fast and highly dynamic movements.

[0101] Finally, regularization involves the actual policy output π θ Instead, it is the mean field

number

[0102] result Limiting the episode duration to 60s corresponds to T=1500 time steps. In practice, 100M sequential iterations are needed for asymptotic optimization under worst-case conditions, which corresponds to about one and a half months of experience with a real exoskeleton1. This takes 6 hours and 40 independent simulators on a single machine with 64 physical cores and one GPU Tesla V100 to obtain a satisfactory transferable policy.

[0103] The mean episode reward vs. duration training curves show the main contribution effect. Smoothness conditioning (regularization) slows down convergence slightly (60M vs. 50M) but does not prevent asymptotic reward. Using a simple reward, i.e. +1 per time step until early termination (tipping over), similar training performance can be observed up to 25M, thanks to a well-defined termination condition. After this point, convergence slows down, but the policy only slightly degrades in performance at the end. This result demonstrates the robustness of the convergence and provides insight into how the general idea of ​​using a reward representing the recovery of a virtual twin of the exoskeleton from each push restores balance. Without termination conditions for safety and portability, fast convergence can be achieved in about 30M. However, it would be impossible to use such a policy on a real device.

[0104] Further testing shows that smoothness conditioning improves learned behaviors, counteracts harmful oscillations and preserves highly dynamic movements, which in turn restore balance more efficiently by taking shorter, more minimal actions.

[0105] Finally, the trained neural network was evaluated on several real Atalante units, both for users and dummies. In contrast to the learning scenario of only pelvis-centered pushing, the policy is able to cope with many types of disturbances: the exoskeleton was pushed in real life at several different application points, with impressive results. The recovery strategy is reliable for all push deformations, but also for pulls. The transition to Atalante works out of the box despite hardware wear and inconsistencies with the simulation, especially ground friction, mechanical flexibility and patient disturbances.

[0106] system According to another aspect, the invention relates to a system of a server 10 and a bipedal robot 1 (preferably an exoskeleton accommodating a human operator) for the implementation of the method according to the first aspect (training the neural network) and / or the second aspect (stabilization of the robot).

[0107] As mentioned above, the server 10 and the robot 1 each comprise data processing means 11, 11' and data storage means 12, 12' (optionally external). The robot 1 typically also comprises sensors such as inertial measurement means 14 (inertial unit) and / or means for detecting foot impacts on the ground 13 (contact sensors or optionally pressure sensors).

[0108] The robot 1 has several degrees of freedom that are actuated by actuators controlled by data processing means 11'.

[0109] The data processing means 11 of the server 10 are configured to implement the method for training a neural network for stabilising a robot 1 according to the first aspect.

[0110] The data processing means 11' of the robot 1 are configured to implement the method for stabilising the robot 1 by means of a neural network according to the second aspect.

[0111] Computer Program Products According to a fourth and fifth aspect, the present invention relates to a computer program product comprising code instructions for executing (on processing means 11, 11') a method for training a neural network for stabilising a robot 1 according to the first aspect or a method for stabilising a robot 1 according to the second aspect, as well as storage means readable by a computing device (e.g. data storage means 12, 12') in which this computer program product is found.

Claims

1. A method of training a neural network for stabilizing a biped robot exhibiting multiple degrees of freedom actuated by actuators, the method comprising the steps of: (a) applying a series of pushes to a virtual twin of the robot in a simulation; (b) running a reinforcement learning algorithm on the neural network, wherein the neural network provides commands to the actuators of the virtual twin of the robot to maximize a reward representing recovery of the virtual twin of the robot from each push. method.

2. The method of claim 1 , wherein the simulation continues for a predetermined duration and steps (a) and (b) are repeated for multiple simulations.

3. The method of claim 2 , wherein a push is applied periodically over the duration of the simulation, and a constant magnitude force is applied for a predetermined duration.

4. The method of claim 1 , wherein pushes are applied to a pelvis of the virtual twin of the robot with orientations sampled from a spherical distribution.

5. 2. The method of claim 1, wherein during step (b), at least one terminal condition on the virtual twin of the robot is enforced.

6. 6. The method of claim 5, wherein the at least one terminal condition comprises at least one of a minimum distance between the robot's feet, a range of positions and / or velocities of the actuators, a maximum difference from an expected trajectory, a maximum recovery duration, and a maximum power consumption.

7. The minimum distance between the feet of the robot is calculated according to the formula D(CH r ;CH l )>0:02, where CH r;l 7. The method of claim 6, wherein x is the convex hull of the left and right footprints, respectively, and D is the Euclidean distance.

8. The method of claim 6 , wherein the range of velocities of the actuator comprises a range of velocities less than 0.6 rad / s.

9. 6. The method of claim 5, wherein the at least one terminal condition comprises a minimum pelvis height greater than 0.3 m, minimum and maximum pelvis yaw angles comprised between -0.4 rad and 0.4 rad, and minimum and maximum pelvis roll angles comprised between -0.25 rad and 0.7 rad.

10. The method of claim 1 , wherein the simulation outputs a state of the virtual twin of the robot as a function of the push and the commands provided by the neural network.

11. 11. The method of claim 10, wherein the robot is equipped with at least one sensor for observing a state of the robot, and the neural network receives as input in step (b) a state of the virtual twin of the robot output by the simulation.

12. 2. The method of claim 1, wherein the neural network provides target positions and / or velocities of the actuators as commands, and a control loop mechanism determines the torque to be applied by the actuators as a function of the target positions and / or velocities of the actuators.

13. 13. The method of claim 12, wherein the neural network provides commands at a first frequency and the control loop mechanism provides a torque at a second frequency that is higher than the first frequency.

14. 2. The method of claim 1 , wherein the neural network is trained to learn a policy, and step (b) comprises performing temporal and / or spatial regularization of the policy to improve smoothness of the commands of the neural network.

15. Step (b) is to set the first state s t The first value of the policy for and the second similar state [0010] performing spatial regularization using a spatial regularization term that is a function of a distance between the similar state [0025] is the first state s t Standard deviation σ centered on S Sampling function N(s t ,σ S 15. The method of claim 14, which is the result of

16. The standard deviation σ S 16. The method of claim 15, wherein is comprised between 0.1 and 0.

7.

17. Step (b) is the first state s t The first value of the policy for s and the second subsequent step s t+1 15. The method of claim 14, further comprising performing temporal regularization using a temporal regularization term that is a function of a distance between the first value of the policy and the second value of the policy for.

18. The method of claim 17 , wherein the distance uses the L1 norm.

19. The first value is the first state s t and the second value is an average value of the policy for the second state. [0030] 16. The method of claim 15, comprising an average value of the policies for

20. The method of claim 14 , wherein the regularization is applied to a mean field of the policy.

21. The method of claim 1 , further comprising the step (c) of storing the trained neural network in a memory of the robot.

22. The method of claim 1 , wherein the bipedal robot is an exoskeleton that accommodates a human operator.

23. 1. A method for stabilizing a biped robot exhibiting multiple degrees of freedom actuated by actuators, comprising: (d) using the neural network trained using the method of claim 1 to provide commands to the actuators of the robot. method.

24. 24. A system comprising a server and a bipedal robot exhibiting multiple degrees of freedom actuated by actuators, the server and the bipedal robot each comprising data processing means configured to perform, respectively, the method for training a neural network for stabilizing the robot described in any one of claims 1 to 22 and the method for stabilizing the exoskeleton described in claim 23.

25. A computer program product comprising code instructions for carrying out the method for training a neural network for stabilising a robot according to any one of claims 1 to 22 or the method for stabilising a robot according to claim 23.

26. 24. A storage means readable by a computing device, having a computer program product thereon comprising code instructions for carrying out the method for training a neural network for stabilising a robot according to any one of claims 1 to 22 or the method for stabilising a robot according to claim 23.

Images