Robot assembly skill learning method

Abstract

A method and system for robotic skill learning suitable for high precision assembly tasks employing compliance controllers. The actor-reviewer reinforcement learning controller is coupled to the compliance controller, where the actor neural network provides target position adjustment action data to the compliance controller based on robot state data feedback, and the reviewer neural network is used to train the actor. The reviewer neural network receives robot state data feedback and reward data from the robot, receives action data from the actor, and associates optimal actions associated with the state to maximize the reward. Then, the commentator adjusts the actor's parameters such that the actor produces an effective action in response to the status data, resulting in the compliance controller / robotic system quickly and reliably completing the assembly task. After the actors are fully trained, reviewers are not used any more.

Description

Technical Field

[0001] The present disclosure relates generally to methods for robotic skill learning, and more specifically to methods for robotic skill learning applicable to high-precision assembly tasks employing a compliant controller, wherein an actor-critic reinforcement learning controller is trained while providing nominal target position input corrections to the compliant controller, and after training is complete, the reinforcement learning controller is used in an actor-only mode. Background Art

[0002] The use of industrial robots to repeatedly perform a wide variety of manufacturing and assembly operations is well known. However, some types of close-tolerance assembly operations, such as installing a pin into a hole or inserting one component into another, remain problematic for robots to perform. These types of operations are typically performed manually because robots have difficulty detecting and correcting the complex misalignments that can occur in close-tolerance assembly tasks. That is, due to slight deviations in component pose caused by gripping and fixturing uncertainties, the robot cannot simply move the component to its nominal mounting position, but must instead "feel" in order to properly align and fit one component into another.

[0003] In order to make the assembly task robust to these unavoidable placement uncertainties, robotic systems typically utilize force controllers (also known as compliance control or admittance control), in which force and torque feedback is used to provide the motion commands required to complete the assembly operation. The traditional way to establish and adjust the force controller for robotic assembly tasks is through manual adjustment, in which a human operator programs the real robotic system for the assembly task, runs the program, and carefully adjusts the force control parameters in a trial and error manner. However, since manual trial and error must be performed, it is time-consuming and expensive to tune and set these force control functions using physical testing. In addition, when adjustments are performed on the actual system, if the actual system does not meet the requirements, redesign may be required. Parameter adjustments on a real physical test system are also dangerous because the robot is not compliant, and unexpected strong contact between components may therefore damage the robot, the components, or the surrounding fixtures or structures.

[0004] Systems exist for tuning force control parameters of robotic assembly in simulated environments, but these existing systems present several limitations including slowness and lack of robustness. Other existing systems attempt to apply learning capabilities to force controllers. However, existing simulated learning systems are not robust, and failure data often overwhelms the demonstration data in the learning controller. Existing reinforcement learning systems take a long time to train, and often experience many failed trials along the way, with the risks of component damage and dangerous behavior discussed previously.

[0005] In view of the above, there is a need for improved methods for robotic assembly skill learning in tight tolerance applications using compliant controllers. Summary of the invention

[0006] The following disclosure describes a method and system for robotic skill learning applicable to high-precision assembly tasks employing force or compliance controllers. An actor-critic reinforcement learning controller is coupled to a compliance controller, wherein an actor neural network provides action data in the form of target position adjustments to the compliance controller based on robot state data feedback, and a critic neural network is used to train the actor neural network. The critic neural network receives robot state data feedback and reward data from the robot, and action data from the actor neural network, and associates optimal actions associated with the states in order to maximize the reward. The critic then adjusts the parameters of the actor so that the actor produces effective actions in response to the state data, resulting in the compliant robot / robot system completing the assembly task quickly and reliably. After the actor neural network is sufficiently trained, the critic neural network is no longer used.

[0007] Additional features of the present disclosure will become apparent from the following description and appended claims, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is an illustration of a robotic assembly operation performed on a close tolerance part illustrating sources of part placement uncertainty that create challenges for the robotic assembly operation;

[0009] Figure 2 is an illustration of a component being assembled by a robot, wherein the component needs to be aligned in such a way that the robot performs a hole search in a plane perpendicular to the insertion axis;

[0010] Figure 3 is an illustration of a component being assembled by a robot, wherein the component needs to be aligned in such a way that the robot performs a phase search to find the proper rotational orientation about the insertion axis;

[0011] Figure 4 is a block diagram illustration of a system configured for robotic assembly operations using a compliant controller (ie, force or admittance control) as is known in the art;

[0012] Figure 5 is a block diagram illustration of a system configured for robotic assembly skill learning using a compliant controller with an actor-critic reinforcement learning module according to an embodiment of the present disclosure;

[0013] Figure 6 is a block diagram illustration of a reinforcement learning system as used in embodiments of the present disclosure;

[0014] Figure 7 is a block diagram illustration of a system configured for robotic assembly skill learning using a compliant controller and a robot with a trained actor-only reinforcement learning module according to an embodiment of the present disclosure;

[0015] Figure 8 is a flow chart of a method for performing robotic assembly skill learning using actor-critic reinforcement learning with a compliant controller / robot system according to an embodiment of the present disclosure;

[0016] Fig. 9 is a conceptual illustration of a system for robotic assembly skill learning combining human demonstration and reinforcement learning-based discovery according to an embodiment of the present disclosure;

[0017] Fig.10 is a block diagram illustration of a system configured for robotic assembly skill learning using a compliant controller with a reinforcement learning module in a co-training mode for human correction during ongoing self-learning in accordance with an embodiment of the present disclosure;

[0018] Fig.11 is a block diagram illustration of a system configured for robotic assembly skill learning using a compliant controller using a reinforcement learning module in an online self-learning mode according to an embodiment of the present disclosure; and

[0019] Fig.12 It is a flowchart of a method for using a compliance controller / robot system to learn robot assembly skills using a reinforcement learning controller according to an embodiment of the present disclosure, including offline pre-training using human demonstration data and online self-learning using human co-training. DETAILED DESCRIPTION

[0020] The following discussion of embodiments of the present disclosure involving methods for robotic skill learning using an actor-critic reinforcement learning controller used in an actor-only mode after training is merely exemplary in nature and is in no way intended to limit the disclosed technology or its applications or uses.

[0021] The use of industrial robots for various manufacturing and assembly operations is well known. The present disclosure is directed to overcoming the challenges encountered in many robotic assembly operations.

[0022] Figure 1is an illustration of a robotic assembly operation performed on a close tolerance component, which illustrates several sources of component placement uncertainty that challenge robotic assembly operations. A robot 100 with a fixture 102 grasps a first component 110 to be assembled with a second component 120. In this example, the first component 110 is a pin component and the second component 120 is a hole structure. The pin component 110 will be inserted into a hole in the hole structure 120. The tolerances of the components in the pin-in-hole assembly are generally very tight so that the assembly can work without excessive looseness after assembly. Some pin-in-hole assemblies have two coaxial pins on one component, or two pins with parallel axes on one component, which must be inserted into two holes on another component at the same time, which makes the assembly operation more difficult. Many other types of mating component assemblies, such as electrical connectors, complex planar shapes, etc., show similar close tolerances.

[0023] Assembly operations of the type described above are typically performed manually because robots have difficulty detecting and correcting complex misalignments that may occur in tight tolerance assembly tasks. That is, due to slight deviations in the pose of the components, the robot cannot simply move the components to their nominal installation positions, but must "feel" the alignment and fit of one piece to another. There are many possible sources of error and uncertainty in the pose of the components. First, the precise position and orientation (collectively referred to as "pose") of the pin component 110 captured in the fixture 102 can change from the expected pose by a small amount. Similarly, the precise pose of the hole component 120 in its fixture can also be different from the expected pose. In systems that use a camera 130 to provide an image of the workspace scene for position recognition, perception errors may also contribute to the uncertainty of relative component positioning. In addition, calibration errors in the placement of the robot 100 and the fixture that holds the component 120 in the workspace, as well as small robot joint position changes, may further contribute to component placement uncertainty. These factors combine to make it impossible for the robot 100 to simply pick up the pin component 110 and insert it into the hole structure 120 with a single motion.

[0024] Figure 2 is an illustration of a component being assembled by a robot, where the component needs to be aligned in such a way that the robot performs a hole search in a plane perpendicular to the insertion axis. Figure 1 The same manner shown grips the component 210 that must be inserted into the hole 220. The distance 230, which is exaggerated for visual effect, represents the uncertainty in the lateral position of the component 210 relative to the hole 220. In order to find the proper alignment of the component 210 with the hole 220, a robot may be required to perform a hole search in which the gripper 202 moves the component 210 back and forth in a zigzag pattern 240 in a plane perpendicular to the axis of the component 210.

[0025] Figure 3is an illustration of a component being assembled by a robot, where the component needs to be aligned in a manner that causes the robot to perform a phase (rotation) search to find the proper rotational orientation about the insertion axis. A clamp 302 grabs a first component 310 that must mate with a second component 320. In this case, the first and second components 310 / 320 are gears with teeth that must be precisely aligned. In order to find the proper alignment of component 310 relative to component 320, the robot may need to perform a phase search, where the clamp 302 finely adjusts the rotational position of component 310 about its pivot axis while attempting to lower component 310 into position and engage with component 320.

[0026] Although the above-described hole-seeking and phase-seeking capabilities exist in robotic systems, they can be inefficient and are generally ineffective when component assembly involves anything more complex than simple hole alignment or rotational alignment. Figure 1-3 The illustrations are merely exemplary, and there are many other types of close tolerance installation and assembly tasks that are difficult for robots to perform.

[0027] In order to make assembly tasks robust to these unavoidable positional uncertainties, robotic systems typically utilize force-control-based functions to "feel" for proper component fit. The traditional way to set up robotic assembly tasks is through manual tuning, where a human operator programs the actual robotic system for the assembly task, runs the program, and adjusts the force control parameters in a trial-and-error manner. However, using physical testing to tune and set up these force control functions is time consuming and expensive due to the manual trial and error that must be performed. Parameter adjustments on a real physical test system are also dangerous because the robot is not compliant, and unexpected high-force contact can damage the robot, the workpiece, and / or surrounding fixtures or structures.

[0028] Systems exist for adjusting force control parameters of robotic assemblies in simulated environments, but these existing systems exhibit several limitations. Some of these systems are designed to simulate only one specific type of assembly task, e.g. Figure 1-3 and cannot be applied to general assembly tasks because these techniques include pre-programmed motion strategies that are specialized for a particular type of assembly task. Other existing systems for parameter tuning in a simulation environment still require a great deal of human experience and expertise to guide the selection of force control parameters used in the simulation.

[0029] The present disclosure describes methods for robotic skill learning using improved reinforcement learning techniques that overcome the shortcomings of existing methods of programming or teaching robots to perform assembly tasks. One disclosed method uses a reinforcement learning controller that is pre-trained offline with human demonstration data and updated online with human override actions (co-training), while another disclosed method uses an actor-critic reinforcement learning controller that is trained and then used in an actor-only mode after training. Both disclosed methods are discussed in detail below.

[0030] Figure 4 is a block diagram illustration of a system 400 configured for robotic assembly operations using a compliance controller (ie, force or admittance control) as is known in the art. In the physical world, a robotic controller 410 interacts with objects such as Figure 1 The controller 410 may communicate with the robot 100 via a cable 412 as shown or wirelessly. As is known in the art and discussed below, the controller 410 provides joint motion commands to the robot 100 and receives status feedback from the robot 100. Figure 1 As shown, the robot 100 has a gripper 102 that grasps a first component 110, and the controller 410 provides commands for the purpose of assembling the first component 110 with (or into) a second component 120. The illustration of components 110 and 120 is merely exemplary. The compliant controller can be used to assemble any component that requires high precision, such as Figure 1-3 Those in, double pin parts with double hole parts, electrical connectors, etc.

[0031] Block 420 represents the controller 410 and the robot 100 in block diagram form. The controller 410 is configured as a compliance controller, the function of which is discussed below. Block 430 provides a nominal target position for the first part 110. The nominal target position can be predetermined and constant for a particular robot work cell, or the nominal target position can be provided by a vision system based on an observed position of the second part 120, or by some other means. For the purposes of discussion, it is assumed that the position of the second part 120 in the robot work cell is known, and the nominal target position from block 420 defines the position of the first part 110 to install it into the second part 120. Of course, the nominal target position of the first part 110 can then be transformed into fixture coordinates, which can then be converted into robot joint positions using inverse kinematics in a known manner.

[0032] After block 430, a summing junction 440 is included. Although Figure 4, node 440 does not have a second input, but a second input will be added in subsequent figures. Block 450 defines the motion limits of the robot 100. The motion limits ensure that the robot 100 does not take excessively large motion steps during the control process, where excessive motion may create a dangerous situation or cause forceful contact between the robot 100 and / or the parts 110 / 120 with each other or with other objects in the work cell. If the difference between the target position and the current position is greater than the motion limit, the motion limit prevails and limits the size of the step.

[0033] Block 460 includes an admittance control function that interacts with a robot in block 470 that performs an assembly task in block 480. Blocks 470 and 480 represent the physical actions of the robot 100 as it installs the first component 110 into the second component 120. The robot in block 470 provides state feedback to the admittance control function in block 460 on line 472. The state feedback provided on line 472 includes robot joint states (position / velocity) and contact forces and torques. Alternatively, position and velocity state data may be provided in Cartesian coordinates, which may be easily converted to joint coordinates by the above-mentioned transformation calculations, or vice versa. Force and torque sensors (not shown) are required in the operation of the robot 100 to measure the contact forces between the components 110 and 120, wherein the force and torque sensors may be placed between the robot 100 and the fixture 102, between the fixture 102 and the first component 110, or between the second component 120 and its "ground" (fixture). Contact forces and torques can also be measured from robot joint torque sensors or estimated from other signals such as motor currents.

[0034] The admittance control function in block 460 operates in the following manner, as known in the art and only briefly discussed here. Impedance control (or admittance control) is a method of dynamic control involving force and position. It is often used in applications where a manipulator interacts with its environment and the force-position relationship is of interest. Mechanical impedance is the ratio of force output to motion input. Controlling the impedance of a mechanism means controlling the resistance to external motion imposed by the environment. Mechanical admittance is the inverse of impedance, which defines the motion produced by the force input. The theory behind the impedance / admittance control method is to treat the environment as an admittance and the manipulator as an impedance.

[0035] Using the target position from node 440 and the motion limits from block 450, the admittance control function in block 460 calculates the target velocity (in six degrees of freedom) to move the workpiece from its current position to the target position (or motion-limited step size). The admittance control function then ... by using equations such as: The target velocity is adjusted using the force compensation term to calculate the command velocity, where V is the command velocity vector (this equation applies to translational motion) and V d is the target velocity vector, is the inverse of the admittance gain matrix, and F is the measured contact force vector from a force sensor mounted to the robot or workpiece. In this example, each vector includes three translational degrees of freedom. A similar equation is used to calculate the rotational command velocity ω using contact torque feedback.

[0036] The command velocities calculated as described above are then converted to command joint velocities for all robot joints by multiplying the inverse of the Jacobian matrix J by the transpose of the command velocity vector as follows: After the command joint velocity is calculated, a low pass filter may also be provided to ensure smoothness and feasibility of the commanded velocity. The calculated command joint velocity is provided to the robot which moves and measures the new contact force, and the target position is again compared to the current position, and the velocity calculation is repeated. In an attempt to reach the target position from the joint 440, the admittance control function in block 460 repeatedly provides motion commands to the robot in block 470 using force feedback and robot state feedback on line 472.

[0037] Units 430-460 are programmed as algorithms in the controller 410. The interaction of the controller 410 with the robot 100 occurs in the real world via cable 412 (or wirelessly), and this is represented in block 420 by the forward arrow (motion command) and feedback line 472 (joint states and contact forces) from block 460 to block 470.

[0038] As previously mentioned, conventional compliance control techniques are not effective for all types of assembly tasks. For example, in a tight tolerance assembly operation, the robot in block 470 may spend a long time in a feedback loop with the admittance control function in block 460, and may ultimately never complete the assembly task, including the possibility of component damage in the process. This can be somewhat mitigated by fine-tuning the impedance / admittance control parameters, but this is only effective in some cases and only for specific workpiece assembly operations. The technology of the present disclosure has been developed to address these shortcomings of existing compliance controllers.

[0039] Figure 5 is a block diagram illustration of a system 500 configured for robotic assembly skill learning using a compliant controller and a robot using an actor-critic reinforcement learning module according to an embodiment of the present disclosure. Figure 5 At the top of is block 420, which depicts Figure 4 The interaction of the compliant controller 410 and the robot 100 with the robotic assembly operation shown and discussed above. Figure 5 Omitted because it has been combined Figure 4 The roles of these physical units corresponding to the compliance control block 420 are explained.

[0040] In the disclosed technique, a reinforcement learning module 510 is coupled to the compliant controller / robotic system in block 420. The purpose of the reinforcement learning module 510 is to learn an effective high-level motion strategy for assembly operations and use this strategy to guide the compliant controller / robotic system. The specific technique used in the reinforcement learning module 510 is an actor-critic reinforcement learning structure, including an actor block 520 and a critic block 530. Figure 5 The reinforcement learning module 510 is depicted initially configured in training mode. The following figures depict how a trained reinforcement learning system may be deployed in inference mode for ongoing operation of a compliant controller / robotic system.

[0041] Figure 6 is a block diagram illustration of a reinforcement learning system 600 as used in embodiments of the present disclosure. The environment block 610 represents the physical system, in this case, a compliant controller / robot that performs an assembly task. Agent 620 is a reinforcement learning module. The basic concept of a reinforcement learning system is to observe the operating state of the system and the effectiveness of the system, and learn which input actions to the system lead to the best results.

[0042] To achieve this, the agent 620 receives state data S on line 630. t (t is the time step), and the reward data R is received on line 640 t For a robot system using a compliant controller, the state data S t Including robot position and velocity status, as well as contact force and torque. Reward data R t Typically a single numerical value corresponding to the success or failure of an assembly operation. For example, if the compliant controller / robotic system fails to install a first part into a second part, the reward may be assigned a small negative value such as -1, and if the compliant controller / robotic system successfully installs the first part into the second part, the reward may be assigned a large positive value such as 100.

[0043] Agent 620 (reinforcement learning module) can be modeled as a neural network. t and reward data R t When the agent 620 determines action A t , which is considered to be the most effective input to the system (environment 610) performing the assembly operation. In the case of a compliant controller / robot system, the action data A t It is the adjustment of the target position in six degrees of freedom (three orthogonal translations and three rotations). t, the compliant controller / robotic system (environment 610) performs the next step of the attempted assembly operation and provides a new value for the state data (S t+1 ) and the new value of the reward data (R t+1 ) as output, and the agent 620 then determines the next action based on the new input. This continues until the component completes successfully or eventually fails.

[0044] Agent 620 (reinforcement learning module) uses action data, state data, and reward data to train a neural network to learn the most effective action for a specific set of state data. This is done by framing the reinforcement learning problem as having the goal of maximizing the cumulative reward while accounting for stochasticity using a formula such as: where the formula tries to maximize the state s t and action a t The expected cumulative value of the reward r over the assembly operation (from time t=0 to t=T) as a function of , and where γ is the discount factor.

[0045] In the disclosed technology, a reinforcement learning system is configured for training using an actor-critic formulation. In the actor-critic formulation, the actor is a stochastic feedback controller (as discussed above). Figure 6 ), which uses a probability function to determine the action based on the state, and the critic trains the actor to maximize the reward. The actor and critic are approximated by neural networks.

[0046] The actor neural network determines the action a based on the strategy π, which is a function of the state s, namely: where the actor policy π is represented by a Gaussian distribution N with mean μ and standard deviation σ. Training of the actor policy involves adjusting the parameters of the probability function (mean μ and standard deviation σ) to provide actions a that lead to the most efficient outcome as determined by the rewards. The actor does not use reward values ​​from the robot / controller directly, but rather is trained by a critic who does use rewards.

[0047] The critic is a function of state and action called a Q function, which is used for training the actor. State, action, and reward data are required as input in order to train the critic. The critic neural network Q function is an approximation of the cumulative reward (as in Equation (1)), and is trained to minimize the temporal difference as follows: min[Q(s,a)-(r+γQ(s′,a′))] 2 (3) where Q(s,a) is the critic function Q (of state s and action a), r is the reward for the current step, and the last term is the reward for the next step (s′,a′) predicted by the Q function using a discount factor γ for the cumulative reward. The next action is determined by the actor’s policy; i.e., a′ ∼ π(·|s′).

[0048] The actor neural network is then trained to maximize the reward estimated by the critic. To do this, the critic adjusts the parameters (μ, σ) of the actor policy π in order to maximize the value of the Q function: max Q(s,a) (4) where action a is determined by policy π, as defined above in equation (2).

[0049] Using the techniques discussed in the previous paragraphs and in equations (1)-(4), the actor and critic neural networks are trained by simply observing and discovering the responses of the compliant controller / robot system to inputs, trying various input-action strategies dictated by the probabilistic nature of the actor and critic functions, and learning which action strategies lead to the most successful outcomes. At the beginning, the neural network is randomly initialized, and random actions are generated from the actor neural network to drive the robot. At each step in the assembly operation, the state s is measured and an action a is generated by the actor, then the action is evaluated, and the step reward r is obtained at the next step s′. The data at a step (s, a, s′, r) is recorded into a data set called a replay buffer, and the process is repeated. There is a maximum number of steps allowed for an assembly operation; the assembly operation is completed if the task succeeds or the maximum number of steps is reached. After performing one or several assembly operations, all data points (in the form of (s, a, s′, r)) are used to update the critic and actor neural networks.

[0050] return Figure 5 , and in accordance with the previous discussion of the actor-critic reinforcement learning system, the actor block 520 and the critic block 530 operate as follows. On line 522, the actor block 520 provides an action to the summing junction 440 in the form of a target position correction. Based on the state provided to the actor block 520 on line 542, the action is determined based on the actor policy. The target position correction from the actor block 520 is also provided to the critic block 530 on line 524. This is because the critic function requires action data as well as state and reward data for its training, as previously discussed.

[0051] The target position corrections provided to the summing junction 440 have the effect of changing the target position used by the admittance control block 460, thereby effectively controlling the strategy used by the compliant controller / robot system in performing the assembly operation. The admittance control block 460 interacts with the robot 470 at a relatively high frequency control cycle (e.g., 125 Hz), and the robot 470 provides feedback to the reinforcement learning module 510 at a relatively low frequency control cycle (e.g., 10 Hz). In other words, the inner loop (admittance control block 460 and robot 470) runs several or many steps ahead of each step of the outer loop (the reinforcement learning module 510 provides the target position corrections to the compliant controller / robot in block 420).

[0052] The robot 470 provides the state (robot pose or position, robot velocity, and contact forces / torques) to the actor block 520 on line 542, as previously described. Since the critic requires state data, the state is also provided to the critic block 530 on line 544. The reward data from the robot 470 (a small negative reward if the assembly step was unsuccessful, a large positive reward if the assembly step was successful) is provided to the critic block 530 on line 554. Thus, the critic block 530 receives all three inputs (actions, states, and rewards) that it needs for training. The critic block 530 performs ongoing training of the actor block 520 through parameter adjustments (of the actor neural network) on line 532.

[0053] use Figure 5 The system shown, during the assembly operation in the learning mode, uses all the data collected from the compliant controller / robotic system to perform training of the actor and critic neural networks (in blocks 520 and 530, respectively). This continues until the performance of the actor block 520 reaches a satisfactory level.

[0054] Figure 7 is a block diagram illustration of a system 700 configured for robotic assembly skill learning using a compliant controller and a robot with a trained actor-only reinforcement learning module according to an embodiment of the present disclosure. Figure 7 In the example, the reinforcement learning module 710 only includes Figure 5 The actor block 520 of FIG. 5 is a block of the actor neural network. In this case, the actor block 520 (actor neural network) has already been trained as described above to optimize the success of the robotic assembly operation. Therefore, the critic is no longer needed and the parameters of the actor neural network are no longer adjusted further. Figure 7 The assembly skill learning system is depicted as it is used for actual production work of the robot 100. As previously described, the robot 470 provides state data to the actor block 520 on line 542, and the actor block 520 uses the strategy to determine the action based on the state data. The action (target position adjustment or correction) is provided to the summing node 440 on line 522.

[0055] Figure 5 and 7 The system shown in (a compliant controller / robot system with a reinforcement learning module) has been implemented and demonstrated to provide superior performance. For example, in a two-pin / hole installation, the compliant controller / robot without a supplementary control strategy has a success rate of about 43% over an assembly operation time of about 35 seconds (350 attempted motion steps at 10 Hz). Figure 2 The same compliant controller / robot performing the same operation with the disclosed actor-critic reinforcement learning controller had a success rate of approximately 100% for an assembly operation time of approximately 12 seconds (120 attempted motion steps).

[0056] In one embodiment, the reinforcement learning module 510 / 710 is programmed into the robot controller 410 (along with the compliant control algorithms of the units 430-460). Other implementations are possible in which the reinforcement learning module 510 / 710 runs on a computer or cloud server separate from the robot controller 410; for example, the reinforcement learning module 510 / 710 may be executed on a device configured for efficient neural network computation. Another alternative is to train an actor-critic system running on a separate computer, then incorporate the trained actor-only reinforcement learning controller into the robot controller 410. Any combination of these implementations may be used as appropriate.

[0057] Figure 8 800 is a flow chart of a method for robotic assembly skill learning using actor-critic reinforcement learning with a compliant controller / robot system according to an embodiment of the present disclosure. At block 802, a compliant controller / robot system is provided, including an admittance control module that controls a robot performing an assembly task using force feedback, the force feedback being used by the admittance control module to determine the next robot motion toward a target position and within specified motion limits.

[0058] At block 804, an actor-critic reinforcement learning module is provided in communication with the compliance controller / robot system. The reinforcement learning module includes an actor neural network that receives state data feedback from the robot and determines actions in the form of target position corrections, which are provided as inputs to the admittance control module. The reinforcement learning module also includes a critic neural network that receives state data feedback and reward data feedback from the robot, and actions from the actor neural network. The critic neural network is trained using the actions, state data, and reward data to optimize the reward based on the state and action data. The critic neural network also adjusts parameters of the actor neural network to train the actor to provide actions that optimize the reward.

[0059] At box 806, the compliant controller / robotic system and reinforcement learning module operate in a learning mode, where the robotic system performs the assembly task and the reinforcement learning module trains the actor and critic neural networks. At decision diamond 808, it is determined whether the actor neural network has been sufficiently trained to reliably provide successful guidance of the robotic assembly operation. If not, then the learning mode continues.

[0060] At block 810, after the actor neural network demonstrates successful capability, the actor-only version of the compliant controller / robotic system and reinforcement learning module operates in production mode, where the robotic system performs the assembly task while being guided by target position corrections from the actor neural network. In production mode at block 808, the critic neural network is no longer used, and only the actor reinforcement learning module is used to provide feedback control to the compliant controller / robotic system, where the actor neural network operates in inference mode.

[0061] The actor-critic reinforcement learning technique described above provides significantly improved performance in robotic assembly operations using compliant controllers. The actor-critic reinforcement learning technique offers many advantages: it is fast because it optimally adjusts motion based on force feedback; it is flexible and requires no design or tuning of system parameters; it is easy to apply because the reinforcement learning system learns optimal behavior by using self-discovery of the real robotic system and does not require models or human tuning; and it is a general method that can be applied to simple or complex tasks.

[0062] Another type of assembly skill learning system uses human demonstrations of operations to pre-train a reinforcement learning system, which then performs ongoing training in a self-learning mode, with co-training provided by a human override to teach optimal behavior as needed. This system is discussed below.

[0063] Fig. 9is a conceptual illustration of a system 900 for robotic assembly skill learning combining human demonstration and reinforcement learning based discovery according to an embodiment of the present disclosure. As previously discussed, a robot such as the robot 100 with a compliant controller is configured to perform assembly operations.

[0064] In the first step of the process (see circled number 1), a human operator 910 collaborates with the robot 100 to demonstrate an assembly operation. One technique for demonstrating the operation involves placing the robot 100 in a teach mode, where the human 910 manually grabs the robot fixture and workpiece and moves the workpiece to an installation position in a second workpiece (while the robot and controller monitor the robot class and force states), or the human 910 uses a teach pendant to provide commands to the robot 100 to complete the workpiece installation. Another technique for demonstrating the operation is teleoperation. In one form of teleoperation, the human 910 manipulates a replica of the workpiece that the robot 100 is grabbing, and the human 910 uses visual feedback from the robot assembly operation and the human's own sense of touch while watching the robot 100 to move the replica workpiece (which is equipped and provides motion commands to the robot 100) to guide the robot 100 to successfully complete the assembly operation. In another form of teleoperation, the human 910 uses a joystick-type input device to provide motion instructions (translation and rotation) to the robot 100. These or other human demonstration techniques may be used.

[0065] The human demonstrator 910 preferably demonstrates the assembly operation several times so that several complete sets of state and motion data can be collected, each set resulting in a successful installation. The demonstration data (robot motion and force states and motions, as discussed in detail previously) is collected in a database 920.

[0066] In the second step, the demonstration data from the database 920 is provided to the reinforcement learning system 930. The reinforcement learning system 930 includes an agent (neural network) that learns what actions are effective for a set of robot and force states based on reward data. The demonstration data from the database 920 is provided to the reinforcement learning system 930 for pre-training. In this pre-training mode (the second step of the entire process, as shown by the circled number 2), there is no environment for the agent to interact with, as previously described in conjunction with Figure 6 In contrast, in pre-training mode, the agent neural network in the reinforcement learning system 930 is trained purely based on the demonstration data from the first step. Therefore, in the pre-training step, both the environment block and the feedback line are shown as dashed lines (not used).

[0067] In the third step of the process, the pre-trained reinforcement learning system 930 and compliance controller / robotic system 940 are placed in an online production mode where self-learning occurs. The compliance controller / robotic system 940 may use the same robot 100 as used for the human demonstration, or a separate instance of the same robot configuration. In the online self-learning mode, the compliance controller / robotic system 940 repeatedly performs the specified installation (assembly operation) in production mode, with additional control provided by the reinforcement learning system 930. An example of a production (or "online") operating mode is that a second part (perhaps by a second robot) is placed into a fixture or a second part arrives on a conveyor, the robot 100 picks up the first part and installs it into the second part, and the assembled unit continues to move for another assembly step or packaging, and the process is then repeated for the next two parts, and so on.

[0068] As reinforcement learning system 930 and compliant controller / robotic system 940 perform assembly operations in the third step, more cycles of learning data are accumulated and stored in database 922. Database 922 initially includes human demonstration data from database 920, and data from ongoing operations in the third step is added to database 922. As previously described, the data includes the action, state, and reward data required to train reinforcement learning system 930.

[0069] If the assembly operation is not particularly difficult, the reinforcement learning system 930 and the compliance controller / robotic system 940 can operate indefinitely in an online self-learning mode with a very high success rate. This is the case, for example, when a first part is installed into a second part with fairly loose tolerances, or when the parts include geometric features that mechanically guide one part into another. However, in a tight tolerance assembly operation, some attempted installations may be unsuccessful, and the failure data in the database 922 may begin to overwhelm the reinforcement learning system 930. That is, when the positive reward data is sparse, the neural network in the agent of the reinforcement learning system 930 cannot properly associate a valid action with a given state.

[0070] Due to the above, in the technology of the present disclosure, a fourth step is added for difficult assembly operations. The fourth step is a co-training mode, in which human correction is provided during the online self-learning mode. The human correction (co-training) stage includes monitoring the success rate of the assembly operation in the online self-learning mode. If the success rate drops below a predetermined threshold, or the attempted assembly operation exhibits a search behavior that deviates significantly from the baseline, the human operator 950 intervenes in the compliance controller / robotic system 940 and interacts with it to override the reinforcement learning system 930. The preferred mode of interaction between the human operator 950 and the compliance controller / robotic system 940 is remote operation, which has been discussed above.

[0071] By using the human intervention / correction step (co-training) described above, new successful learning cycles are added to the database 922 so that the high reward value provides beneficial updated training of the agent neural network in the reinforcement learning system 930 to identify effective actions for a given state. The human intervention / correction step can be performed for a period of time, with the human operator 950 monitoring and intervening as needed to ensure that each attempted assembly operation is successful. After this co-training period, it is expected that the reinforcement learning system 930 and the compliance controller / robotic system 940 will resume autonomous operation in the online self-learning production mode.

[0072] Fig.10 is a block diagram illustration of a system 1000 configured for robotic assembly skill learning using a compliant controller with a reinforcement learning module in a co-training mode for human correction during ongoing self-learning in accordance with an embodiment of the present disclosure. Fig.10 The system 1000 described in the above Fig. 9 The use of the common training model (step 4) is discussed.

[0073] exist Fig.10 At the top of is block 420, which depicts Figure 4 The interaction of the compliant controller with the robotic assembly operation shown and discussed previously. In the disclosed technology, a reinforcement learning module 1010 is coupled to the compliant controller / robotic system in block 420. The purpose of the reinforcement learning module 1010 is to learn an effective high-level motion strategy for the assembly operation and use the strategy to guide the compliant controller / robotic system. The reinforcement learning module 1010 corresponds to Fig. 9 For the purposes of this discussion, the database 920 and the agent neural network that learns the most effective assembly behavior are considered to be part of the reinforcement learning module 1010, such as Fig.10 As shown in block 1020 .

[0074] With about Figure 5 In the same manner discussed, the robot 470 provides state feedback (robot position and velocity, and contact forces / torques between components) to the block 1020 on line 1042, and provides reward data (large positive rewards for successful assembly; small negative rewards for unsuccessful attempts) on line 1044. The agent block 1020 stores the state data and reward data in a database and uses the data in the training of the agent neural network in the reinforcement learning module 1010.

[0075] Fig.10 depicts a system 1000 configured for human correction / intervention (co-training), while Fig.11 (discussed below) shows the same system 1000 configured for self-learning without human co-training. Fig.10 When configured for human co-training input as in , a human demonstrator 1050 guides the assembly operation using a technique such as teleoperation. A "switch" 1032 is positioned to receive motion data from a human demonstrator (eg, teleoperation) provided on line 1052.

[0076] The motion data from the human demonstration is provided to the compliance controller / robotic system in block 420 as a target position correction input to summing junction 440 on line 1062. During human co-training in the online self-learning phase, target position corrections are provided from the human demonstration (on line 1062) and from the reinforcement learning module 1010 (on line 1022 from the agent 1020), with the human demonstration data preferably being weighted higher than the reinforcement learning data. This arrangement enables the human demonstration input to guide the robot during assembly operations.

[0077] The motion data from the human demonstrations may also be provided to the agent 1020 in the reinforcement learning module 1010 as action data on line 1064. This connection provides all the action data needed by the agent 1020 in order to learn the correlations between states and actions that lead to successful assembly operations.

[0078] therefore, Fig.10 The configuration enables two things to occur; first, human demonstrations control the actual motion of the compliant controller / robotic system 420 during joint training; and second, action data from the human demonstrations is provided to the agent block 1020, which, together with the state data and reward data, allows the agent neural network to learn the correlation between state and valid actions.

[0079] Fig.10 The system 1000 can also be used in a simplified form for a human demonstration phase. This can be accomplished because there are connections from the human demonstrator 1050 to provide input to the compliance controller / robotic system 420, and connections from the robot 470 to provide state and reward feedback to the agent 1020 (which includes the database). If the system 1000 is used for a human demonstration phase ( Fig. 9420), only the human demonstration input is provided to the compliance controller in block 420, so the human demonstration exclusively controls the robot motion. In addition, the state, reward feedback, and human-demonstrated actions are collected in database 920 (as described above), but the agent in block 1020 does not provide actions to summing node 440. After the human demonstration data is collected, the offline pre-training of the agent in block 1020 can then continue as previously discussed without any interaction with the compliance controller / robotic system in block 420.

[0080] Fig.11 is a block diagram illustration of a system 1000 configured for robotic assembly skill learning using a compliant controller with a reinforcement learning module in an online self-learning mode according to an embodiment of the present disclosure. Fig.11 System 1000 with Fig.10 The system 1000 is identical to the system 1000, except in a different configuration.

[0081] Fig.11 A system 1000 configured for self-learning without human co-training input is depicted. Fig. 9 The online self-learning is performed in the third step of Fig.11 When configured for autonomous operation without human demonstration input as in block 1050, no human demonstrator 1050 is required, switch 1032 is positioned to not receive motion data on line 1052, and only motion data on line 1022 now flows to summing node 440. No motion data is provided to summing node 440 on line 1062, and no motion data is provided back to the agent in block 1020 on line 1064.

[0082] therefore, Fig.11 The configuration enables two things to happen for the autonomous operation of the reinforcement learning module 1010 and the compliant controller / robotic system in block 420; first, during the online self-learning phase, the agent neural network in block 1020 controls the actual movement of the compliant controller / robotic system; and second, the agent block 1020 inherently knows the action data, which together with the state data and reward data allows the agent neural network to continuously learn the correlation between state and valid actions.

[0083] Therefore, in Fig.10 and 11 The configured system 1000 provides such Fig. 9The system 1000 has all the capabilities required for human demonstration, offline pre-training, online self-learning, and human correction / intervention (co-training) as shown in the four steps of FIG. During the online production operation of the robot, the system 1000 can be easily switched to the autonomous operation of the reinforcement learning module 1010 simply by opening or closing the switch 1032 to allow human correction / intervention when necessary and returning to autonomous operation of the reinforcement learning module 1010 when appropriate. Fig.10 and 11 Switch between configurations 11.

[0084] The following discussion provides further explanation of the techniques used by the reinforcement learning module 1010 to learn and apply policies to effectively control the compliant controller / robotic system.

[0085] The general principles of reinforcement learning are referenced above Figure 6 is discussed. In typical reinforcement learning-based methods, low learning efficiency is the main drawback. Although reinforcement learning is well known for its ability to learn robust policies, the learning process requires a large amount of interaction data if no prior information is provided. When the reward data is sparse, and / or for tasks in which it is difficult to discover the complete state space (e.g., precise insertion tasks, where the chance of sampling an action leading to a successful insertion is very low), an unacceptably large amount of interaction data is required to learn a good control policy. In imitation learning-based methods, less interaction data is required. However, imitation learning results may be limited to specific tasks. In addition, since imitation learning does not generalize well to unseen situations, the learning results are not very robust to task changes.

[0086] The technology of the present disclosure is developed to overcome the above shortcomings. The first part of the currently disclosed technology uses offline pre-training of the reinforcement learning system. The reason for introducing offline reinforcement learning is that it is much faster than standard reinforcement learning. Standard reinforcement learning methods require continuous interaction with the "environment" (compliant controller / robotic system) in order to collect data along with training. Typically, the interactive data collection process is much slower than training. It is possible for offline reinforcement learning to perform training on a given data set (i.e., demonstration data) without further interactive data collection, and therefore offline reinforcement learning spends less time for pre-training.

[0087] The working principle of offline reinforcement learning is similar to that of standard reinforcement learning methods. However, if standard reinforcement learning methods are used to train a control policy on a given dataset without further interactive data collection, the training process becomes unstable. During the training process, overestimation of the Q function is inevitable, especially in unseen state-action spaces. This overestimation leads to blind optimism in unseen action spaces, and further leads to positive feedback of overestimation of the Q function in unseen state-action spaces, which ultimately leads to the divergence of the entire training process. In the currently disclosed offline reinforcement learning pre-training, this problem is solved by introducing additional constraints to the learning formula (i.e., to the critic (Q function) or actor in the actor-critic reinforcement learning system) to ensure that the updated policy does not deviate too much from the demonstration dataset.

[0088] An example of constraints on the actor of an actor-critic reinforcement learning system is given as: max a～π(·|s) [Q(s,a)-λD KL (π(·|s)||π demo (·|s))] (5)

[0089] In the above optimization problem, the objective function includes a loss function calculated by subtracting a divergence term from the reward calculated by the Q function. In the currently disclosed technology, the loss function includes a relative entropy (Kullback-Leibler) divergence calculation (D KL ), which is how one probability distribution [the actor’s control strategy, π(·|s)] differs from a second reference probability distribution [the training dataset, π demo (·|s)]. In equation (5), λ is a weighting constant. By subtracting the divergence term from the Q-function reward term, the objective function penalizes the actor's control policy for deviations from the training dataset by a small amount π. With this training constraint, overestimation can be avoided and the training process becomes stable on a given dataset.

[0090] When the pre-training of the control strategy is completed using the demonstration data as described above, the training process moves to the self-learning phase ( Fig. 9 Step 3). During the self-learning phase, the robot also interacts with the assembly task for evaluation of control performance and data collection. The collected interaction data is also used to fine-tune the control strategy in the agent neural network for performance improvement. In this phase, the human operator (950, 1050) can further assist the learning process by providing corrections or guidance (co-training), such as Fig. 9 and 10As shown. For tasks with process uncertainty, it is impossible for limited human demonstrations (step 1) to cover all task variations. Therefore, if the task variations are not covered in the demonstration dataset, the pre-trained controller may perform poorly. In this case, the standard reinforcement learning training process relies on random discovery and requires many random trials. In the technology disclosed in the present invention, a self-learning mode (co-training) is included. In the co-training mode, if the learning controller produces an undesirable action during the self-learning phase, the human operator is able to override the robot command. Using this guidance mechanism, appropriate state-action pairs can be recorded in the training dataset with fewer interactions, and the learning controller can learn to improve its performance in a short period of time.

[0091] Using the above techniques, Figure 9-11 The system described in has been demonstrated to produce significantly improved performance relative to standard or original reinforcement learning methods and relative to traditional imitation learning methods. Using the technology of the present disclosure, robotic assembly operations on various workpiece shapes have been demonstrated with higher success rates and shorter cycle times than the same operations performed using prior art methods.

[0092] Fig.12 1200 is a flow chart of a method for robotic assembly skill learning using a compliant controller / robot system using a reinforcement learning controller according to an embodiment of the present disclosure, including offline pre-training using human demonstration data and online self-learning using human co-training. At block 1202, a compliant controller / robot system is provided, including an admittance control module that uses force feedback to control a robot performing an assembly task, wherein the force feedback is used by the admittance control module to determine the next robot motion toward a target position within specified motion limits.

[0093] At block 1204, a reinforcement learning module is provided in communication with the compliance controller / robot system. The reinforcement learning module includes at least one neural network that receives state data feedback from the robot and determines an action in the form of a target position correction, which is provided as an input to the admittance control module. The reinforcement learning module also receives reward data feedback from the robot. The neural network in the reinforcement learning module (or both neural networks in an actor-critic system) is trained using the state data and reward data to optimize a reward based on the state data and corresponding action data.

[0094] At block 1206, a human demonstration of a prescribed assembly operation is performed, and pre-training of a reinforcement learning module is completed using data from the human demonstration. Block 1206 represents Fig. 9The first and second steps (upper half) of . As previously described, several repetitions of human demonstrations are preferably performed, with action, state, and reward data captured in a demonstration database 920. The demonstration data in database 920 is then used to perform initial pre-training of the reinforcement learning agent without any interaction with the compliant controller / robotic system. Pre-training includes using an optimization objective function that constrains the neural network in the reinforcement learning controller to behave as defined in the demonstration data.

[0095] At block 1208, the compliant controller / robotic system and the reinforcement learning module are placed in self-learning mode online. Fig. 9 The third step of , in which the robot and the compliance controller repeatedly perform the assembly operation with additional feedback control by the reinforcement learning system 930. In the self-learning mode at box 1208, the agent neural network in the reinforcement learning module continues to learn as data (actions, states, rewards) are collected in the database 922 (with initial data from the database 920). Fig.11 This self-learning mode is also described in , using a reinforcement learning module 1010 that provides feedback control to the compliant controller / robotic system in block 420.

[0096] At decision diamond 1210, it is determined whether the performance of the compliance controller / robotic system and the reinforcement learning module is acceptable. Performance is evaluated through a series of assembly operations and key performance indicators are measured. The performance indicators can be designed to include a combination of operation success rate, operation cycle time, and other factors determined by the operator. If the performance is acceptable, the online operation continues in the self-learning mode at box 1208.

[0097] If the performance is unacceptable at decision diamond 1210, then the online operation switches to a joint training mode at block 1212. Fig. 9 In the fourth step and Fig.10 Co-training is described in , where a human operator uses teleoperation to override inputs to a compliant controller / robotic system. During co-training, the reinforcement learning module receives the necessary action, state, and reward data to continue learning, and the reinforcement learning module may also contribute small amounts of input to the compliant controller / robotic system. Co-training with a human operator may continue for several assembly operations and / or for a certain period of time, at which point the performance of the system in self-learning mode is again evaluated at decision diamond 1210. As previously discussed, the co-training phase at box 1212 generates high reward training data from successful assembly operations, thereby providing additional training of the neural network in the reinforcement learning module to establish correlations between states and valid actions.

[0098] Figure 9-11A system that combines imitation learning (through human demonstration) and reinforcement learning (self-learning with human correction or co-training) offers many advantages in robotic skill learning. The system is flexible and can generally be applied to many different part types and shapes without modification. The system is also easy and intuitive, requiring no programming or fine-tuning of parameters. The system can be applied to relatively simple assembly tasks, where little or no human intervention is required for the self-learning operation to be performed, and / or to difficult assembly tasks with tight tolerances, where the use of human correction / intervention as needed ensures the necessary high success rate.

[0099] Throughout the foregoing discussion, various computers and controllers are described and implied. It should be understood that the software applications and modules of these computers and controllers are executed on one or more computing devices having processors and memory modules. In particular, this includes such Figure 4 The robot controller 410 shown controls the robot 100 to perform a robotic assembly task and performs Figure 4-11 The compliance controller block 420 and various reinforcement learning modules depicted are functional for the robot controller and optionally other computer processors.

[0100] The foregoing discussion discloses and describes only exemplary embodiments of the present disclosure. Those skilled in the art will readily recognize from such discussion and from the accompanying drawings and claims that various changes, modifications and variations may be made therein without departing from the spirit and scope of the present disclosure as defined in the appended claims.

Claims

1. A method for learning robot assembly skills, the method comprising: providing a robot controlled by the compliance controller, the robot configured to perform an assembly operation, wherein the robot moves a first component into an assembly position with a second component; providing a reinforcement learning controller in communication with the compliance controller and the robot, the reinforcement learning controller having an actor module and a critic module, wherein the actor module includes a neural network that defines a policy for determining actions in response to state data received as feedback from the robot, and the critic module includes a neural network that optimizes a critic function relating the state data to the actions so as to maximize reward data received as feedback from the robot; operating the robot to perform the assembly operation using the reinforcement learning controller in a learning mode, wherein the actor module provides the action as input to the compliance controller and the critic module updates parameters of the actor module's policy based on the critic function until the actor module demonstrates a predetermined level of performance of the assembly operation; as well as The robot is operated to perform the assembly operation using the reinforcement learning controller in a production mode, wherein the actor module provides the action as input to the compliance controller and the critic module is no longer used.

2. The method according to claim 1, wherein: The action is a target position adjustment provided to the compliance controller, the target position adjustment comprising three translations and three rotations of the first component relative to the second component.

3. The method according to claim 2, wherein: The target position adjustment is combined with a nominal target position, bounded by predetermined motion limits, and used as input to the compliance controller.

4. The method according to claim 1, wherein: The state data includes a plurality of robot positions and velocities, each robot position and velocity including three translational components and three rotational components, and a plurality of part-to-part contact forces and torques, each part-to-part contact force and torque being in three directions.

5. The method according to claim 1, wherein: The reward data includes a positive value when the assembling operation is successfully completed and a negative value when the assembling operation is not successfully completed, and the positive value has a magnitude greater than the negative value.

6. The method according to claim 1, wherein: The policy defined by the neural network in the actor module is a statistical distribution of actions over states, wherein the statistical distribution is defined by the parameters including a mean and a standard deviation.

7. The method according to claim 6, wherein: in, When the reinforcement learning controller operates in the production mode, the parameters of the policy in the actor module are fixed.

8. The method according to claim 6, wherein: Training the actor module by the critic module includes: using an optimization calculation that determines a desired action that maximizes the reward calculated by the critic function; and adjusting parameters of the policy in the actor module based on the desired action.

9. The method according to claim 1, wherein: The critic function calculates the reward for the current state and current action as equal to the current actual reward plus a term that includes the weighted expected rewards for future states and future actions.

10. The method according to claim 1, wherein: The compliance controller includes an admittance control routine that calculates a robot command velocity using a calculation having a first term and a second term, wherein the first term includes a difference between a current robot position and a target position adjusted by the reinforcement learning controller, and the second term includes an inverse of an admittance gain matrix multiplied by a contact force vector.

11. The method according to claim 1, wherein: The compliance controller controls the robot in a feedback loop having a first cycle frequency, and the reinforcement learning controller controls the compliance controller in a feedback loop having a second cycle frequency, and the first cycle frequency is higher than the second cycle frequency.

12. The method according to claim 1, wherein: Both the compliance controller and the reinforcement learning controller are executed on a robotic controller device, which provides joint motion commands to the robot and receives the state data from the robot as feedback.

13. A robot assembly skill learning system, the system comprising: a robot controlled by the compliant controller, the robot configured to perform an assembly operation wherein the robot moves a first component into an assembly position with a second component; as well as a reinforcement learning controller in communication with the compliance controller and the robot, the reinforcement learning controller having an actor module and a critic module, wherein the actor module includes a neural network that defines a policy for determining actions in response to state data received as feedback from the robot, and the critic module includes a neural network that optimizes a critic function relating the state data to the actions so as to maximize reward data received as feedback from the robot; wherein the robot performs an assembly operation using the reinforcement learning controller in a learning mode, wherein the actor module provides the actions as input to the compliance controller, and the critic module updates parameters of the policy of the actor module based on the critic function until the actor module demonstrates a predetermined performance level of the assembly operation, And wherein the robot performs the assembly operation using the reinforcement learning controller in a production mode, wherein the actor module provides the action as input to the compliance controller and the critic module is no longer used.

14. The system according to claim 13, wherein: The state data includes multiple robot positions and velocities and multiple part-to-part contact forces and torques, each robot position and velocity includes three translational components and three rotational components, and each part-to-part contact force and torque is in three directions, and the reward data includes a positive value when the assembly operation is successfully completed and a negative value when the assembly operation is not successfully completed.

15. The system of claim 13, wherein: The policy defined by the neural network in the actor module is a statistical distribution of actions over states, wherein the statistical distribution is defined by parameters including a mean and a standard deviation, and wherein the parameters of the policy in the actor module are fixed when the reinforcement learning controller operates in the production mode.

16. The system of claim 15, wherein the training of the actor module by the critic module comprises: using an optimization computation that determines a desired action that maximizes the reward computed by the critic function; and adjusting the parameters of the strategy in the actor module based on the desired action.

17. The system of claim 13, wherein: The action is a target position adjustment provided to the compliance controller, the target position adjustment comprising three translations and three rotations of the first component relative to the second component, and wherein the target position adjustment is combined with a nominal target position, bounded by predetermined motion limits, and used as input to the compliance controller.

18. The system of claim 13, wherein: The compliance controller controls the robot in a feedback loop having a first cycle frequency, and the reinforcement learning controller controls the compliance controller in a feedback loop having a second cycle frequency, and the first cycle frequency is higher than the second cycle frequency.

19. The system of claim 13, wherein: Both the compliance controller and the reinforcement learning controller are executed on a robotic controller device, which provides joint motion commands to the robot and receives the state data from the robot as feedback.

20. A robot assembly skill learning system, the system comprising: a robot controlled by the compliant controller, the robot configured to perform an assembly operation; as well as a reinforcement learning controller in communication with the compliance controller and the robot, the reinforcement learning controller having an actor module and a critic module, wherein the actor module defines a policy for determining actions in response to state data received as feedback from the robot, and the critic module optimizes a critic function relating the state data to the actions so as to maximize reward data received as feedback from the robot, The robot first performs the assembly operation using the reinforcement learning controller in a learning mode, wherein the actor module provides the action as input to the compliance controller and the critic module updates the parameters of the policy of the actor module, and the robot then performs the assembly operation using the reinforcement learning controller in a production mode, wherein the critic module is no longer used.

Images