Robot assembling skill learning method

Abstract

To provide a method and a system for robot assembling skill learning which are applicable to work.SOLUTION: An actor critic reinforcement learning controller is coupled to a compliance controller, here, actor neural network provides target position adjustment action data to the compliance controller on the basis of robot state data feedback, and critic neural network is used for training an actor. The critic neural network receives robot state data feedback and reward data from a robot together with action data from the actor, and correlates an optimal action associated with the state so as to maximize a reward.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present disclosure relates generally to a method for robotic skill learning, and more particularly to a method for robotic skill learning applicable to high-precision assembly tasks utilizing a compliant controller, in which an actor-critic reinforcement learning controller is trained while providing nominal target position input corrections to a compliant controller, and the reinforcement learning controller is used in actor-only mode after training is complete. [Background technology]

[0002] The use of industrial robots to perform a wide range of repetitive manufacturing and assembly operations is well known. However, some types of close-tolerance assembly operations, such as placing a peg in a hole or plugging one part into another, are still problematic for robots to perform. These types of operations are often performed manually because robots have difficulty detecting and correcting the complex misalignments that can occur in close-tolerance assembly operations. That is, due to small deviations in part pose caused by uncertainties in both gripping and fixturing, the robot cannot simply move the part to its nominal installed position, but rather must "feel" for the proper alignment and fit of one part within the other.

[0003] To make assembly operations robust against these unavoidable positioning uncertainties, robotic systems typically use force controllers (also known as compliance or admittance control), where force and torque feedback is used to provide the motion commands required to complete the assembly operations. The traditional method for setting up and tuning force controllers for robotic assembly operations is by manual tuning, where a human operator programs the actual robotic system for the assembly operation, runs the program, and carefully adjusts the force control parameters in a trial and error manner. However, tuning and setting up these force control functions using physical testing is time-consuming and costly due to the need for manual trial and error. Similarly, if tuning is performed on the actual system, a redesign may be required if the actual system does not meet the requirements. Parameter tuning on an actual physical test system can also be risky, as the robot is not compliant and therefore unexpected hard contact between parts can cause damage to the robot, the parts, or the surrounding fixtures or structures.

[0004] Although systems exist for tuning force control parameters for robotic assembly in a simulated environment, these existing systems exhibit several limitations, including slowness and lack of robustness. Other existing systems attempt to apply learning functions to the force controller. However, existing limited learning systems are not robust and fault data typically overwhelms the live demonstration data in the learning controller. Existing reinforcement learning systems take a long time to train and typically experience many failed attempts along the way, entailing the risks of part damage and dangerous behavior as discussed above.

[0005] In view of the above, there is a need for improved methods for robotic assembly skill learning in tight tolerance applications using compliant controllers. Summary of the Invention

[0006] The following disclosure describes a method and system for robotic assembly skill learning applicable to high-precision assembly tasks utilizing a force or compliance controller. An actor-critic reinforcement learning controller is coupled to the compliance controller, where the actor neural network provides behavior data in the form of target position adjustments to the compliance controller based on robot state data feedback, and the critic neural network is used to train the actor neural network. The critic neural network receives robot state data feedback and reward data from the robot along with behavior data from the actor neural network, and correlates optimal behaviors associated with the state to optimize rewards. The critic then adjusts the actor's parameters such that the actor generates highly effective behaviors in response to the state data, leading to fast and reliable completion of the assembly task by the compliance controller / robot system. After the actor neural network is properly trained, the critic neural network is no longer used.

[0007] Additional features of the present disclosure will become apparent from consideration of the following specification and appended claims in conjunction with the accompanying drawings. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is an illustration of a robotic assembly operation being performed on tight tolerance parts illustrating sources of part positioning uncertainty that create challenges for robotic assembly operations. [Diagram 2] FIG. 2 is an illustration of a part being assembled by a robot where the part requires alignment in a way that causes the robot to perform a hole search in a plane perpendicular to the insertion axis. [Diagram 3] FIG. 3 is an illustration of a part being assembled by a robot where the part requires alignment in a way that causes the robot to perform a phase search to find the proper rotational orientation about the insertion axis. [Figure 4] FIG. 4 is a block diagram illustration of a system configured for robotic assembly operations using compliance control (ie, force or admittance control) as known in the art. [Diagram 5] FIG. 5 is a block diagram illustration of a system configured for robotic assembly skill learning using a compliant controller including an actor-critic reinforcement learning module in accordance with one embodiment of the present disclosure. [Figure 6] FIG. 6 is a block diagram illustration of a reinforcement learning system used in embodiments of the present disclosure. [Figure 7] FIG. 7 is a block diagram illustration of a system configured for robotic assembly skill learning using a compliance controller and a robot including a trained actor-only reinforcement learning module in accordance with one embodiment of the present disclosure. [Figure 8] FIG. 8 is a flow diagram of a method for robotic assembly skill learning using actor-critic reinforcement learning with a compliant controller / robot system according to one embodiment of the present disclosure. [Figure 9] FIG. 9 is a conceptual illustration of a system for robotic assembly skill learning that combines human demonstration and reinforcement learning-based discovery in accordance with one embodiment of the present disclosure. [Figure 10] FIG. 10 is a block diagram illustration of a system configured for robotic assembly skill learning using a compliance controller including a reinforcement learning module in a concurrent training mode for human correction during ongoing self-learning in accordance with one embodiment of the present disclosure. [Figure 11] FIG. 11 is a block diagram illustration of a system configured for robotic assembly skill learning using a compliance controller including a reinforcement learning module in an online self-learning mode in accordance with one embodiment of the present disclosure. [Figure 12]FIG. 12 is a flow diagram of a method for robotic assembly skill learning using a reinforcement learning controller with a compliance controller / robot system including offline pre-training using human demonstration data and online self-learning including simultaneous human training according to one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The following discussion of embodiments of the present disclosure directed to methods for robotic skill learning using an actor-critic used in post-training actor-only mode is merely exemplary in nature and is not intended to limit in any way the disclosed technology or its applications or uses.

[0010] The use of industrial robots for a variety of manufacturing and assembly operations is well known. The present disclosure is directed to overcoming challenges encountered in many robotic assembly operations.

[0011] FIG. 1 is an illustration of a robotic assembly operation being performed on parts with tight tolerances, illustrating some sources of part positioning inaccuracies that create challenges for the robotic assembly operation. A robot 100 with a gripper 102 grasps a first part 110 to be assembled with a second part 120. In this example, the first part 110 is a peg part and the second part 120 is a hole structure. The peg part 110 is to be inserted into a hole in the hole structure 120. The tolerances of the parts in a peg-in-hole assembly are typically quite tight so that the assembly can operate without excessive play after it is assembled. Some peg-in-hole assemblies have two coaxial pegs on one part, or two parallel-axis pegs on one part, which must be inserted simultaneously into two holes on another part, making the assembly operation even more difficult. Many other types of mating part assemblies, such as electrical connectors, complex planar shapes, etc., exhibit similarly tight tolerances.

[0012] The above-mentioned types of assembly operations are often performed manually because robots have difficulty detecting and correcting complex misalignments that may occur in tight tolerance assembly operations. That is, due to small deviations in part pose, the robot cannot simply move the part to its nominal installed position, but rather must "probe" the alignment and fit of one part within the other. There are many possible sources of part pose error and uncertainty. First, the exact position and orientation (collectively "pose") of the peg part 110 grasped in the gripper 102 may vary slightly from the expected pose. Similarly, the exact pose of the hole part 120 in its fixture may vary from the expected pose as well. In systems where a camera 130 is used to provide an image of the workspace scene for location identification, perception errors may also contribute to relative part positioning uncertainty. Additionally, calibration errors in the placement of the robot 100 and the fixture that holds the part 120 in the workspace, and slight robot joint position variations may all further contribute to part positioning uncertainty. These factors combine to make it impossible for the robot 100 to simply pick up the peg component 110 and insert it into the hole structure 120 in a single motion.

[0013] 2 is an illustration of a part being assembled by a robot where the part needs to be aligned in a way that will cause the robot to perform a hole search in a plane perpendicular to the insertion axis. Gripper 202 grasps part 210 which must be inserted into hole 220 in the same manner as shown in FIG. 1. Distance 230, exaggerated for visual effect, represents the uncertainty in the lateral position of part 210 relative to hole 220. To find the proper alignment of part 210 and hole 220, the robot may be required to perform a hole search, where gripper 202 moves part 210 back and forth in a zigzag pattern 240 in a plane perpendicular to the axis of part 210.

[0014] 3 is an illustration of parts being assembled by a robot where the parts need to be aligned in a way that causes the robot to perform a phase (rotation) search to find the proper rotational orientation about the insertion axis. Gripper 302 grips a first part 310 that must mesh with a second part 320. In this case, the first and second parts 310 / 320 are gears with teeth that must be precisely aligned. To find the proper alignment of part 310 relative to part 320, the robot may be required to perform a phase search, where gripper 302 fine-tunes the rotational position of part 310 about its pivot axis while attempting to lower part 310 into position and into engagement with part 320.

[0015] While the hole-finding and phase-finding capabilities described above exist in robotic systems, they can be inefficient and are often ineffective when part assembly involves something more complicated than simple hole or rotational alignment. The illustrations of Figures 1-3 are merely one example, and there are many other types of close tolerance installation and assembly tasks that are difficult to perform with a robot.

[0016] To make assembly operations robust to these unavoidable positional uncertainties, robotic systems typically utilize force control-based features to "feel" the proper part fit. The traditional method of setting up a robotic assembly operation is by manual tuning, where a human operator programs the actual robotic system for the assembly operation, runs the program, and adjusts the force control parameters in a trial-and-error fashion. However, tuning and setting up these force control features using physical testing is time-consuming and costly due to the manual trial-and-error that must be performed. Tuning parameters on an actual physical test system can also be dangerous, as the robot may not be compliant and unexpected hard contacts may cause damage to the robot, the workpiece, and / or surrounding fixtures or structures.

[0017] Although systems exist for tuning force control parameters for robotic assembly in a simulated environment, these existing systems exhibit several limitations. Some of these systems are designed to simulate only one particular type of assembly task, such as one of those shown in Figures 1-3, and the techniques include pre-programmed motion strategies specific to the particular type of assembly task, making these systems inapplicable to assembly tasks in general. Other existing systems for parameter tuning in a simulated environment still require significant human experience and expertise to guide the selection of force control parameters used in the simulation.

[0018] This disclosure describes methods for robot skill learning using improved reinforcement learning techniques that overcome the shortcomings of existing methods for programming or teaching robots to perform assembly tasks. One disclosed method uses a reinforcement learning controller that is pre-trained offline with human demonstration data and self-trained updates online with human override behaviors (concurrent training), and another disclosed method uses an actor-critic reinforcement learning controller that is trained and then used in a post-training actor-only mode. Both of these disclosed methods are described in detail below.

[0019] FIG. 4 is a block diagram illustration of a system 400 configured for robot assembly operations using compliance control (i.e., force or admittance control) as known in the art. In the physical world, a robot controller 410 communicates with a robot, such as the robot 100 of FIG. 1. The controller 410 can communicate with the robot 100 via a cable 412 as shown or wirelessly. The controller 410 provides articulation commands to the robot 100 and receives state feedback from the robot 100, as known in the art and discussed below. As illustrated in FIG. 1, the robot 100 has a gripper 102 that grips a first part 110, and the controller 410 provides commands to assemble the first part 110 with (into) a second part 120. The illustration of the parts 110 and 120 is merely an example. The compliance controller may be used for the assembly of any component that requires high precision, such as those shown in Figures 1-3, dual peg and two hole components, electrical connectors, etc.

[0020] Block 420 represents the controller 410 and the robot 100 in block diagram form. The controller 410 is configured as a compliance controller with functionality discussed below. Block 430 provides a nominal target position for the first part 110. It is contemplated that the nominal target position may be predefined and invariant for a particular robot work cell, or the nominal target position may be provided by a vision system or by some other means based on an observed position of the second part 120. For purposes of this discussion, it is assumed that the position of the second part 120 in the robot work cell is known, and the nominal target position from block 420 defines a position of the first part 110 for placing the first part in the second part 120. The nominal target position of the first part 110 may then of course be transformed into gripper coordinates, which can then be transformed into robot joint positions using inverse kinematics in a known manner.

[0021] After block 430, summing junction 440 is included. Although junction 440 does not have a second input in FIG. 4, a second input will be added in later figures. Block 450 defines the motion limits of robot 100. The motion limits ensure that robot 100 does not take excessively large motion steps during the control process, where excessively large movements could create dangerous situations or result in forced contact of robot 100 and / or parts 110 / 120 with each other or with other objects in the work cell. If the difference between the target position and the current position is larger than the motion limits, the motion limits will take precedence and limit the size of the steps.

[0022] Block 460 includes an admittance control that interacts with the robot in block 470 and performs the assembly task in block 480. Blocks 470 and 480 represent the physical behavior of the robot 100 as it places the first part 110 in the second part 120. The robot in block 470 provides state feedback on line 472 to the admittance control function in block 460. The state feedback provided on line 472 includes the robot joint states (position / velocity) along with contact forces and torques. Position and velocity state data may alternatively be provided in Cartesian coordinates, which can be easily converted to joint coordinates and vice versa via the conversion calculations described above. Force and torque sensors (not shown) are required in the operation of the robot 100 to measure contact forces between the parts 110 and 120, where the force and torque sensors could be positioned between the robot 100 and the gripper 102, the gripper 102 and the first part 110, or between the second part 120 and its "ground" (fixture device). Contact forces and torques could also be measured from robot joint torque sensors or estimated from other signals such as motor currents.

[0023] The admittance control function in block 460, as known in the art and discussed only briefly here, operates in the following manner: Impedance control (or admittance control) is an approach to dynamic control that relates force to position. It is often used in applications where a manipulator interacts with its environment and the relationship between force and position is of interest. Mechanical impedance is the ratio of force output to motion input. Controlling the impedance of a mechanism means controlling the resistance to an external motion imposed by the environment. Mechanical admittance is the inverse of impedance and defines the resulting motion resulting from a force input. The theory behind the impedance / admittance control method is to treat the environment as an admittance and the manipulator as an impedance.

[0024] Using the target position from connection 440 and the motion limits from block 450, the admittance control function in block 460 calculates the target velocity (in six degrees of freedom) to move the workpiece from its current position to the target position (or motion-constrained step size), where V is the command velocity vector (equation applies to translations) and V d is the target velocity vector, K v -1 V = V where V is the inverse of the admittance gain matrix and F is the measured contact force vector from a force sensor attached to the robot or workpiece. d +K v -1 The command velocity is calculated by adjusting the desired velocity with a force compensation term using the equation F. All vectors contain three translational degrees of freedom in this example. A similar equation is used to calculate the rotational command velocity ω using contact torque feedback.

[0025] The command velocities calculated as above are then multiplied by the inverse of the Jacobian matrix J by the transpose of the command velocity vector as follows:

number

number

[0026] The elements 430-460 are programmed as algorithms in the controller 410. The interaction between the controller 410 and the robot 100 occurs in the real world via cables 412 (or wirelessly), which is represented in block 420 by the forward arrows (movement commands) and feedback lines 472 (joint states and contact forces) from block 460 to block 470.

[0027] As previously discussed, conventional compliance control techniques are not effective for all types of assembly operations. For example, in assembly operations with tight tolerances, the robot in block 470 may spend a long time in a feedback loop that includes the admittance control function in block 460, and may ultimately not complete the assembly operation, including the possibility of part damage in the process. Although this situation may be somewhat mitigated by fine tuning the impedance / admittance control parameters, this is effective only in some situations and for assembly operations of certain workpieces. The techniques of the present disclosure have been developed to address these shortcomings of existing compliance controllers.

[0028] Figure 5 is a block diagram illustration of a system 500 configured for robotic assembly skill learning using a compliance controller and a robot including an actor-critic reinforcement learning module according to one embodiment of the present disclosure. At the top of Figure 5 is a block diagram 420 depicting the interaction of the compliance controller with the robot assembly operations illustrated in Figure 4 and discussed above. Because the roles of the controller 410 and physical elements such as the robot 100 corresponding to the compliance control block 420 have already been described in relation to Figure 4, these depictions are omitted from Figure 5.

[0029] In the disclosed technique, a reinforcement learning module 510 is coupled to the compliance controller / robot system in block 420. The purpose of the reinforcement learning module 510 is to learn effective high-level motion strategies for the assembly operations and use these strategies to guide the compliance controller / robot system. The particular technique used within the reinforcement learning module 510 is an actor-critic reinforcement learning structure that includes an actor block 520 and a critic block 530. Figure 5 illustrates the reinforcement learning module 510 as initially configured in training mode. Subsequent figures illustrate how the trained reinforcement learning system is deployed in inference mode for ongoing operation of the compliance controller / robot system.

[0030] 6 is a block diagram illustration of a reinforcement learning system 600 used in embodiments of the present disclosure. The environment block 610 represents a physical system, in this case a compliant controller / robot performing an assembly task. The agent 620 is a reinforcement learning module. The basic idea of ​​a reinforcement learning system is to observe the operating state of the system, as well as the effectiveness of the system, and learn which input actions to the system lead to the best results.

[0031] To achieve this, the agent 620 transmits state data S t(t is the time step), and on line 640, the reward data R t For a robot system using a compliance controller, the state data S t contains the position and velocity state of the robot, along with the contact forces and torques. The reward data R t is typically a single numeric value corresponding to the success or lack thereof of an assembly operation. For example, if the compliance controller / robotic system fails to place the first part within the second part, the reward may be assigned a small negative value, such as -1, and if the compliance controller / robotic system successfully places the first part within the second part, the reward may be assigned a large positive value, such as 100.

[0032] The agent 620 (reinforcement learning module) may be modeled as a neural network. t and Reward Data R t Upon receiving the action A, the agent 620 selects the action A that is considered to be the most effective input to the system (environment 610) to perform the assembly operation. t In the case of a compliance controller / robot system, the behavior data A t is the adjustment to the target position in six degrees of freedom (three orthogonal translations and three rotations). t Upon receiving , the compliance controller / robot system (environment 610) executes the next step of attempting the assembly operation and obtains the new value of the state data (S t+1 ) and the new value of the reward data (R t+1 ) as output, and agent 620 then determines its next action based on the new input. This continues until the build is either successfully completed or ultimately fails.

[0033] Agent 620 (a reinforcement learning module) uses the action, state, and reward data to train a neural network to learn the most effective actions for a particular state data set. This is done by formulating a reinforcement learning problem with the goal of maximizing cumulative reward while taking into account contingencies, using a formula such as:

number

[0034] In the disclosed technique, a reinforcement learning system is configured to train using an actor-critic formulation, in which the actor is a probabilistic feedback controller (as shown in FIG. 6 above) that determines actions based on states using a probability function, and the critic trains the actor to maximize rewards. The actor and the critic are approximated by neural networks.

[0035] The actor neural network determines an action a based on a policy π, which is a function of the state s:

number

[0036] A critic is a function of state and action, called a Q-function, used in training an actor. State, action, and reward data are required as input to train a critic. The critic neural network Q-function is an approximation of the cumulative reward (as in equation (1)) and is trained to minimize the time lag, as follows:

number

[0037] The actor neural network is then trained to maximize the reward estimated by the critic. To do this, the critic adjusts the parameters (μ,σ) of the actor policy π with the goal of maximizing the value of the Q-function.

number

[0038] Using the techniques discussed in the previous section and equations (1)-(4), the actor and critic neural networks are trained by simply observing and discovering the response of the compliance controller / robot system to the inputs, trying different input action strategies dictated by the stochastic properties of the actor and critic functions, and learning which action strategy leads to the most successful outcome. First, the neural network is randomly initialized and random actions are generated from the actor neural network to drive the robot. At each step of the assembly operation, the state s is measured and an action a is generated by the actor, after which the action is evaluated and at the next step s', a step reward r is obtained. The data (s, a, s', r) at one step is recorded in a data set called the replay buffer, and then the process is repeated. For one assembly operation, there is a maximum number of steps allowed, and the assembly operation ends if the operation is successful or the maximum number of steps is reached. After performing one or more assembly operations, all data points (in the form (s,a,s',r)) are used to update the critic and actor neural networks.

[0039] Returning to Figure 5, and in accordance with the previous discussion of the actor-critic reinforcement learning system, the actor block 520 and critic block 530 operate as follows: On line 522, the actor block 520 provides an action to the summing junction 440 in the form of a goal position correction. The action is determined based on the actor policy as a function of the state, which is provided to the actor block 520 on line 542. The goal position correction from the actor block 520 is likewise provided to the critic block 530 on line 524. This is because, as discussed above, the critic function requires action data along with state and reward data for its training.

[0040] The target position corrections provided to summing junction 440 have the effect of modifying the target position used by admittance control block 460, which effectively controls the strategy used by the compliance controller / robot system in performing the assembly operations. The admittance control block 460 interacts with the robot 470 at a relatively high frequency control cycle (e.g., 125 Hz), and the robot 470 provides feedback to the reinforcement learning module 510 at a relatively low frequency control cycle (e.g., 10 Hz). In other words, the inner loop (admittance control block 460 and robot 470) executes several or many steps before each step of the outer loop (the reinforcement learning module 510 that provides the target position corrections to the compliance controller / robot in block 420).

[0041] The robot 470 provides state (robot pose or position, robot velocity, and contact forces / torques) to the actor block 520 on line 542 as mentioned above. State is also provided to the critic block 530 on line 544 since the critic needs state data. Reward data from the robot 470 (small negative reward if the assembly step is not successful, large positive reward if the assembly step is successful) is provided to the critic block 530 on line 554. Thus, the critic block 530 receives all three of the inputs required for training (action, state, and reward). The critic block 530 provides ongoing training of the actor block 520 by parameter tuning (of the actor neural network) on line 532.

[0042] 5, training of the actor and critic neural networks (in blocks 520 and 530, respectively) is performed using all data collected from the compliance controller / robot system during the assembly operation in learning mode, continuing until the performance of the actor block 520 reaches a satisfactory level.

[0043] FIG. 7 is a block diagram illustration of a system 700 configured for robotic assembly skill learning using a compliance controller and a robot including a trained actor-only reinforcement learning module according to one embodiment of the present disclosure. In FIG. 7, the reinforcement learning module 710 includes only the actor block 520 of FIG. 5. In this case, the actor block 520 (actor neural network) has been trained as described above to optimize successful robotic assembly operations. Thus, a critic is no longer needed and the actor neural network parameters are not further adjusted. FIG. 7 illustrates the assembly skill learning system being used for an actual production task with the robot 100. As before, the robot 470 provides state data to the actor block 520 on line 542, and the actor block uses a strategy to determine an action based on the state data. The action (target position adjustment or correction) is provided to the summing junction 440 on line 522.

[0044] The system illustrated in Figures 5 and 7 (compliance controller / robot system including reinforcement learning module) has been implemented and demonstrated to provide better performance. For example, in two peg / hole placement, the compliance controller / robot without any supplementary control strategy had a success rate of about 43% in an assembly operation time of about 35 seconds (350 motion step attempts at 10 Hz). The same compliance controller / robot performing the same operation with an unguided supplementary search strategy (as in Figure 2) had a success rate of about 83% in an assembly operation time of about 19 seconds (190 motion step attempts). The same compliance controller / robot performing the same operation with the actor-critic reinforcement learning controller of the present disclosure had a success rate of 100% in an assembly operation time of about 12 seconds (120 motion step attempts).

[0045] In one embodiment, the reinforcement learning module 510 / 710 is programmed into the robot controller 410 (along with the compliance control algorithms of elements 430-460). Other implementations are possible, where the reinforcement learning module 510 / 710 runs on a computer or cloud server separate from the robot controller 410, for example, the reinforcement learning module 510 / 710 could run on a device configured for efficient neural network computation. Another alternative is to train an actor-critic system running on a separate computer and then incorporate a trained actor-only reinforcement learning controller in the robot controller 410. Any combination of these implementations may be used as preferred.

[0046] 8 is a flow diagram 800 of a method for robotic assembly skill learning using actor-critic reinforcement learning with a compliance controller / robot system according to one embodiment of the disclosure. In box 802, a compliance controller / robot system is provided that includes an admittance control module that controls a robot performing an assembly task using force feedback that the admittance control module uses to determine the next robot motion toward a target position and within specified motion limits.

[0047] In box 804, an actor-critic reinforcement learning module is provided in communication with the compliance controller / robot system. The reinforcement learning module includes an actor neural network that receives state data feedback from the robot and determines an action in the form of a target position correction that is provided as an input to the admittance control module. The reinforcement learning module also includes a critic neural network that receives state data feedback and reward data feedback from the robot along with actions from the actor neural network. The critic neural network uses the action, state data, and reward data for training to optimize rewards based on the state and action data. The critic neural network also adjusts parameters of the actor neural network to train the actor to provide actions that optimize rewards.

[0048] In box 806, the compliance controller / robot system and the reinforcement learning module are operated in a learning mode, where the robotic system performs an assembly task and the reinforcement learning module trains the actor and critic neural networks. In decision diamond 808, it is determined whether the actor neural network has been trained sufficiently to reliably provide successful guidance of the robotic assembly tasks. If not, the learning mode continues.

[0049] In box 810, after the actor neural network has demonstrated successful performance, the compliance controller / robot system and the actor-only version of the reinforcement learning module are operated in production mode, where the robotic system performs an assembly task while guided by the target position corrections from the actor neural network. In production mode, in box 808, the critic neural network is no longer used, and the actor-only reinforcement learning module is used to provide feedback control to the compliance controller / robot system, with the actor neural network operating in inference mode.

[0050] The actor-critic reinforcement learning technique described above provides significantly improved performance in robotic assembly operations using compliant controllers. The actor-critic reinforcement learning technique offers many advantages: it is fast since it adjusts the operation in an optimal way based on force feedback; it is versatile since there is no need to design or tune the system parameters; it is easy to apply since the reinforcement learning system learns optimal behavior by self-discovery using the actual robotic system and no model or human tuning is required; it is a general approach that can be applied to simple or complex tasks.

[0051] Another type of assembly skill learning system uses a human demonstration of an action to pre-train a reinforcement learning system, which then performs ongoing training in a self-learning mode with concurrent training provided by human override as needed to teach optimal behavior.

[0052] 9 is a conceptual illustration of a system 900 for robotic assembly skill learning combining human demonstration and reinforcement learning-based discovery in accordance with one embodiment of the present disclosure. A robot, such as robot 100 including a compliance controller, is configured to perform assembly operations as discussed above.

[0053] In the first step of the process (see circled number 1), a human operator 910 demonstrates an assembly operation in cooperation with the robot 100. One technique for demonstrating the operation involves placing the robot 100 in a teach mode, where the human 910 either manually grasps the robot gripper and workpiece and moves it to a placement position in a second workpiece (while the robot and controller monitor the robot and force states) or uses a teach pendant to provide commands to the robot 100 to complete the workpiece placement. Another technique for demonstrating the operation is teleoperation. In one form of teleoperation, the human 910 operates a replica of the workpiece being gripped by the robot 100 and moves this replica workpiece (providing motion commands to the instrumented robot 100) while watching the robot 100, using visual feedback from the robot assembly operation and the human's own haptic feel to guide the robot 100 in completing the correct assembly operation. In another form of teleoperation, the human 910 uses a joystick-type input device to provide motion commands (translation and rotation) to the robot 100. These or other human demonstration techniques can be used.

[0054] The human demonstrator 910 preferably demonstrates the assembly action multiple times, thus allowing multiple complete sets of state and action data to be collected, each leading to a successful installation. The demonstration data (robot motion and force states and actions as discussed at length above) are collected in a database 920.

[0055] In a second step, the demonstration data from the database 920 is provided to the reinforcement learning system 930. The reinforcement learning system 930 includes an agent (neural network) that learns which actions are effective in correlation with the robot and force state set based on the reward data. The demonstration data from the database 920 is provided to the reinforcement learning system 930 for pre-training. In this pre-training mode (the second step of the overall process, marked by the circled number 2), there is no environment to interact with the agent as discussed above in connection with FIG. 6. Instead, in the pre-training mode, the agent neural network in the reinforcement learning system 930 is trained purely based on the demonstration data from step 1. Thus, the environment blocks and feedback lines are all shown as dashed lines (unused) in the pre-training step.

[0056] In a third step of the process, the pre-trained reinforcement learning system 930 and the compliance controller / robot system 940 are placed in an online production mode where self-learning occurs. The compliance controller / robot system 940 may use the same robot 100 used for the human demonstration, or a separate instance of the same robot configuration. In the online self-learning mode, the compliance controller / robot system 940 iteratively performs a given installation (assembly operation) in production mode with additional control provided by the reinforcement learning system 930. An example of a production (or "online" mode of operation is when a second part is placed in a fixture (possibly by a second robot) or a second part arrives on a conveyor, the robot 100 picks up the first part and places it in the second part, the unit of assembly continues to move for another assembly step or packaging, and then the process repeats for the next two parts.

[0057] As the reinforcement learning system 930 and compliance controller / robot system 940 perform assembly operations in the third step, more learning cycle data accumulates and is stored in database 922. Database 922 initially contains the human performance data from database 920, and data from ongoing operations in the third step is added to database 922. The data includes the action, state, and reward data needed to train the reinforcement learning system 930, as discussed above.

[0058] If the assembly operation is not particularly challenging, the reinforcement learning system 930 and compliance controller / robot system 940 can continue to run indefinitely in an online self-learning mode with a very high success rate. This may be the case if the tolerances for the installation of a first part into a second part are fairly loose, or if the parts contain geometric features that mechanically guide one part into the other, for example. However, in assembly operations with tight tolerances, some installation attempts may not be successful, and the failure data in the database 922 may begin to overwhelm the reinforcement learning system 930. That is, if the positive reward data is sparse, the neural network in the agent of the reinforcement learning system 930 may not be able to properly correlate valid actions to a given state.

[0059] Due to the above circumstances, in the technique of this disclosure, a fourth step is added for difficult assembly operations. The fourth step is a concurrent training mode, where human correction is provided during the online learning mode. The human correction (concurrent training) stage includes monitoring the success rate of the assembly operations in the online self-learning mode. If the success rate drops below a predefined threshold or the assembly operation attempt shows a clearly incorrect exploration behavior, the human operator 950 steps in and interacts with the compliance controller / robot system 940 to override the reinforcement learning system 930. The preferred interaction mode between the human operator 950 and the compliance controller / robot system 940 is teleoperation, as discussed above.

[0060] By using the human intervention / correction step (co-training) described above, new successful learning cycles are added to the database 922, and thus high reward values ​​provide useful updated training for the agent neural network in the reinforcement learning system 930 to identify effective actions for a given state. The human intervention / correction step can be performed for a period of time, with a human operator 950 monitoring and intervening as necessary to ensure that each assembly motion attempt is successful. After this period of co-training, the reinforcement learning system 930 and the compliance controller / robot system 940 would be expected to resume autonomous operation in an online self-learning production mode.

[0061] 10 is a block diagram illustration of a system 1000 configured for robotic assembly skill learning using a compliance controller including a reinforcement learning module in a concurrent training mode for human correction during ongoing self-learning, according to an embodiment of the present invention. Use of the system 1000 depicted in FIG. 10 for the concurrent training mode (step 4) discussed above in connection with FIG. 9 is described below.

[0062] At the top of FIG. 10 is block 420, which depicts the interaction of the compliance controller with the robot assembly operations, as illustrated in FIG. 4 and discussed above. In the techniques of this disclosure, a reinforcement learning module 1010 is coupled to the compliance controller / robot system in block 420. The purpose of the reinforcement learning module 1010 is to learn effective high-level motion strategies for the assembly operations and use these strategies to guide the compliance controller / robot system. The reinforcement learning module 1010 corresponds to the reinforcement learning system 930 of FIG. 9. For the purposes of this discussion, the database 920 and the agent neural network that learns the most effective assembly behavior are considered to be part of the reinforcement learning module 1010, as shown in block 1020 of FIG. 10.

[0063] 5, the robot 470 provides state feedback (robot position and velocity and contact forces / torques between parts) to block 1020 on line 1042, and reward data (large positive rewards for successful assemblies, small negative rewards for unsuccessful attempts) on line 1044. The agent block 1020 stores the state and reward data in a database and uses the data in training an agent neural network in the reinforcement learning module 1010.

[0064] Figure 10 depicts a system 1000 configured for human correction / intervention (concurrent training), while Figure 11 (discussed below) shows the same system 1000 configured for self-learning without concurrent human training. When configured for concurrent human training input as in Figure 10, a human demonstrator 1050 guides the assembly operations using techniques such as teleoperation. A "switch" 1032 is positioned to receive operation data from human demonstration (e.g., teleoperation) provided on line 1052.

[0065] Motion data from the human demonstration is provided to the compliance controller / robot system in block 420 as target position correction input on line 1062 to summing junction 440. During concurrent human training in the online self-learning phase, target position corrections are provided from both the human demonstration (on line 1062) and the reinforcement learning module 1010 (from agent 1020 on line 1022), where the human demonstration data is preferably weighted higher than the reinforcement learning data. This arrangement allows the human demonstration input to guide the robot during assembly operations.

[0066] Motion data derived from human demonstrations may also be provided as behavioral data on line 1064 to the agent 1020 in the reinforcement learning module 1010. This connection provides all of the behavioral data the agent 1020 needs to learn the correlations between actions and states that lead to successful assembly operations.

[0067] 10 configuration thus allows two things to happen: first, human demonstration controls the actual behavior of the compliance controller / robot system 420 during concurrent training, and second, behavioral data derived from the human demonstration is provided to the agent block 1020, which, together with state and reward data, enables the agent neural network to learn correlations between states and effective actions.

[0068] It is believed that the system of FIG. 10 can be used in a simplified form for the human demonstration phase as well. This may be done because there is a connection from the human performer 1050 to provide input to the compliance controller / robot system 420, and a connection from the robot 470 to provide state and reward feedback to the agent 1020 (including the database). When the system 1000 is used for the human demonstration phase (first step of FIG. 9), only human demonstration input is provided to the compliance controller in block 420, and thus the robot motion is controlled exclusively by human demonstration. Furthermore, the state, reward feedback and human demonstrated actions are collected in the database 920 (discussed above), but the agent in block 1020 does not provide actions to the summing junction 440. It is believed that after collecting the human demonstration data, the offline pre-training of the agent in block 1020 may then proceed as discussed above, without any interaction with the compliance controller / robot system in block 420.

[0069] 11 is a block diagram illustration of a system 1000 configured for robotic assembly skill learning using a compliance controller including a reinforcement learning module in an online self-learning mode in accordance with one embodiment of the present disclosure. The system 1000 of FIG. 11 is the same as the system 1000 of FIG. 10, except for a different configuration.

[0070] Figure 11 illustrates system 1000 configured for autonomous learning without concurrent human training input. This is the on-line self-learning that took place in the third step of Figure 9. When configured for autonomous operation without any human demonstration input as in Figure 11, a human performer 1050 is not required and switch 1032 is positioned to receive no performance data on line 1052, and only the behavioral data on line 1022 then flows to summing junction 440. No behavioral data is provided to summing junction 440 on line 1062, and no behavioral data is provided back to the agent in block 1020 on line 1064.

[0071] 11 configuration thus enables two things to happen for the autonomous operation of the reinforcement learning module 1010 including the compliance controller / robot system in block 420: First, the agent neural network in block 1020 controls the actual operation of the compliance controller / robot system in an online self-learning phase, and second, behavioral data is inherently known by the agent block 1020, which together with state and reward data allows the agent neural network to continuously learn correlations between states and effective actions.

[0072] Thus, system 1000 in the configurations of Figures 10 and 11 provides all of the capabilities needed for human demonstration, offline pre-training, online self-learning and human correction / intervention (concurrent training) as illustrated in the four steps of Figure 9. During online productive operation of the robot, system 1000 can be easily switched between the configurations of Figures 10 and 11 by simply opening and closing switch 1032 to allow for human correction / intervention when necessary and to return to autonomous operation of reinforcement learning module 1010 when appropriate.

[0073] The following discussion provides further explanation of the techniques used by the reinforcement learning module 1010 to learn and apply policies for the purpose of effectively controlling the compliance controller / robotic system.

[0074] The general principles of reinforcement learning have been discussed above with respect to FIG. 6. In typical reinforcement learning-based approaches, low learning efficiency is a major drawback. Although reinforcement learning is well known for its ability to learn robust policies, the learning process requires a lot of interaction data when no a priori information is provided. For tasks where reward data is sparse and / or where it is difficult to discover the entire state space (e.g., precise insertion tasks where the probability of sampling actions that lead to successful insertions is very low), an unacceptably large amount of interaction data is required to learn a good control policy. Imitation learning-based approaches require less interaction data. However, the results of imitation learning may be limited to a specific task. Similarly, imitation learning does not generalize well to unknown situations, so the learning results are relatively less robust to task variations.

[0075] The technique of the present disclosure has been developed to overcome the above-mentioned shortcomings. The first part of the technique disclosed herein uses offline pre-training of a reinforcement learning system. The reason for introducing offline reinforcement learning is that it is much faster than standard reinforcement learning. Standard reinforcement learning approaches require continuous interaction with the "environment" (compliance controller / robot system) for data collection as well as training. Typically, the interactive data collection process is much slower than training. Offline reinforcement learning can train on a given data set (i.e., demonstration data) without further interactive data collection, and therefore the time it takes for offline reinforcement learning to pre-train is relatively short.

[0076] The working principle of offline reinforcement learning is similar to standard reinforcement learning approaches. However, when standard reinforcement learning approaches are used to train a control policy on a given dataset without further interactive data collection, the training process becomes unstable. During the training process, an overestimation of the Q-function is inevitable, especially in the unknown state-action space. This overestimation leads to blind optimism in the unknown action space, which in turn leads to a positive feedback of overestimation of the Q-function in the unknown state-action space, which ultimately causes the divergence of the entire training process. In the offline reinforcement learning pre-training disclosed herein, this issue is addressed by introducing additional constraints on the learning formulation (i.e., either the critic (Q-function) or the actor in an actor-critic reinforcement learning system) to ensure that the updated policy does not deviate too much from the demonstration dataset.

[0077] An example of a constraint on an actor in an actor-critic reinforcement learning system is shown below:

number

[0078] In the optimization problem described above, the objective function includes a loss function calculated by subtracting a divergence term from the reward calculated by the Q-function. In the technology disclosed herein, the loss function is calculated by multiplying one probability distribution [the actor's control policy, π(·|s)] by a second reference probability distribution [the training data set, π demo The Kullback-Leibler divergence calculation (D KL ), where λ is a weighting constant. By subtracting the divergence term from the Q-function reward term, the objective function penalizes the behavior of the actor control policy π that deviates from the training dataset. This training constraint allows us to avoid overestimation and makes the training process stable for a given dataset.

[0079] If the control strategy is pre-trained using demonstration data as described above, the training process moves to a self-learning stage (step 3 in FIG. 9). In the self-learning stage, the robot further interacts with the assembly task for evaluation of control performance and data collection. The collected interaction data is further utilized to fine-tune the control strategy in the agent neural network for improved performance. At this stage, a human operator (950, 1050) can further assist the learning process by providing correction or guidance (concurrent training), as illustrated in FIGS. 9 and 10. For tasks with process uncertainty, limited human demonstration (step 1) is not possible to cover all task variations. Thus, the performance of the pre-trained controller may be poor if task variations are not covered in the demonstration data set. Under this circumstance, the progress of standard reinforcement learning training is dominated by random discovery, requiring many random trials. The technology of the present disclosure includes a self-learning mode (concurrent training). In the concurrent training mode, the human operator can override the robot commands if inappropriate actions are produced by the learning controller during the self-learning phase. This guidance mechanism allows the correct state-action pairs to be recorded in the training dataset with fewer interactions, and the learning controller can learn to improve its performance in a short time.

[0080] Using the techniques described above, the system depicted in Figures 9-11 has been demonstrated to produce dramatically improved performance compared to standard or simple reinforcement learning approaches, and even conventional imitation learning approaches. Using the techniques of the present disclosure, robotic assembly operations on a variety of workpiece geometries have been demonstrated to exhibit higher success rates and shorter cycle times compared to the same operations performed using prior art methods.

[0081] 12 is a flow diagram 1200 of a method for robotic assembly skill learning using a reinforcement learning controller including a compliance controller / robot system including offline pre-training using human demonstration data and online self-learning including human concurrent training according to one embodiment of the disclosure. In box 1202, a compliance controller / robot system is provided that includes an admittance control module that controls a robot performing an assembly task with force feedback that is used by the admittance control module to determine a next robot motion toward a target position within specified motion limits.

[0082] In box 1204, a reinforcement learning module in communication with the compliance controller / robot system is provided. The reinforcement learning module includes at least one neural network that receives state data feedback from the robot and determines an action in the form of a target position correction that is provided as an input to the admittance control module. The reinforcement learning module also receives reward data feedback from the robot. The neural network in the reinforcement learning module (or two neural networks in the actor-critic system) uses the state data and reward data to train to optimize rewards based on the state data and corresponding action data.

[0083] In box 1206, a human demonstration of a given assembly operation is performed, and the data from the human demonstration is used to complete pre-training of the reinforcement learning module. Box 1206 represents the first and second steps (top half) of FIG. 9. As discussed above, multiple iterations of the human demonstration are preferably performed using the action, state and reward data captured in the demonstration database 920. The demonstration data in the database 920 is then used to initially pre-train the reinforcement learning agent without any interaction with the compliance controller / robot system. Pre-training involves the use of an optimization objective function that constrains the neural network in the reinforcement learning controller to the behaviors defined in the demonstration data.

[0084] In box 1208, the compliance controller / robot system including the reinforcement learning module is put online in self-learning mode. This is the third step in FIG. 9, where the robot including the compliance controller performs an assembly operation with additional feedback control by the reinforcement learning system 930. In the self-learning mode in box 1208, the agent neural network in the reinforcement learning module continues to learn as data (actions, states, rewards) is collected in database 922 (along with the initial data from database 920). This self-learning mode is also depicted in FIG. 11 with the reinforcement learning module 1010 providing feedback control to the compliance controller / robot system in block 420.

[0085] At decision diamond 1210, it is determined whether the performance of the compliance controller / robot system including the reinforcement learning module is acceptable. Performance is evaluated through a series of assembly operations and key performance indices are measured. The performance indices can be designed to include a combination of operation success rate, operation cycle time, and other factors determined by the operator. If performance is acceptable, online operation continues in self-learning mode at box 1208.

[0086] If the performance is not acceptable at decision diamond 1210, the online operation switches to a concurrent training mode at box 1212. Concurrent training, where a human operator uses teleoperation to override inputs to the compliance controller / robot system, is also depicted in FIG. 10 as the fourth step of FIG. 9. During concurrent training, the reinforcement learning module receives the necessary action, state and reward data to continue learning, and the reinforcement learning module can also contribute small amounts of inputs to the compliance controller / robot system. Concurrent training with the human operator can continue for multiple assembly operations and / or for a period of time, at which point the performance of the system in self-learning mode is evaluated again at decision diamond 1210. As discussed above, the concurrent training phase at box 1212 generates high-reward training data from successful assembly operations, thus providing additional training of the neural network in the reinforcement learning module to build correlations between states and valid actions.

[0087] Combining imitation learning (through human demonstration) and reinforcement learning (self-learning or simultaneous training with human correction), the system of Figs. 9-11 offers many advantages in robot skill learning. The system is versatile, being generally available for many different part types and shapes without modification. The system is also simple and intuitive, requiring no programming or delicate fine-tuning of parameters. The system is applicable to relatively simple assembly tasks that require little or no human intervention for ongoing self-learning behavior, and / or difficult assembly tasks with tight tolerances where the use of human correction / intervention as needed ensures a high success rate.

[0088] Various computers and controllers are described or implied throughout the above discussion. It should be understood that the software applications and modules of these computers and controllers execute on one or more computing devices having a processor and memory modules. In particular, this includes the processor in the robot controller 410 that controls the robot 100 performing the robotic assembly tasks shown in FIG. 4, and the robot controller and any other computers that perform the functions of the various reinforcement learning modules and compliance controller blocks 420 depicted in FIGS. 4-11.

[0089] The foregoing discussion discloses and describes merely exemplary embodiments of the present disclosure, and those skilled in the art will readily recognize from such discussion and the accompanying drawings and claims that various changes, modifications, and variations can be made therein without departing from the spirit and scope of the present disclosure, as defined in the following claims.

Claims

1. A robot assembly skill learning method comprising: providing a robot controlled by a compliance controller, the robot configured to perform an assembly operation in which the robot moves a first part to an assembly position with a second part; providing a reinforcement learning controller in communication with the compliance controller and the robot, the reinforcement learning controller including an actor module and a critic module, the actor module including a neural network defining a strategy for determining an action in response to state data received as feedback from the robot, and the critic module including a neural network optimizing a critic function that correlates the state data and the action with the objective of maximizing reward data received as feedback from the robot; operating the robot to perform an assembly task using the reinforcement learning controller in a learning mode, the actor module providing the behaviors as inputs to a compliance controller and the critic module updating parameters of the actor module's strategy based on the critic function until the actor module demonstrates a predefined performance level for the assembly task; and operating the robot to perform the assembly operations using the reinforcement learning controller in a production mode, wherein the actor module provides the behaviors as inputs to the compliance controller and the critic module is not used.

2. The method of claim 1 , wherein the action is a target position adjustment including three translations and three rotations of the first part relative to the second part that is provided to the compliance controller.

3. The method of claim 2 , wherein the target position adjustment is combined with a nominal target position bounded by predefined operating limits and used as the input to the compliance controller.

4. The method of claim 1 , wherein the state data includes the robot's positions and velocities, each having three translational and three rotational components, and part-to-part contact forces and torques in each of three directions.

5. 2. The method of claim 1, wherein reward data includes a positive value if the assembly operation is completed successfully and a negative value if the assembly operation is completed unsuccessfully, the positive value having a greater absolute value than the negative value.

6. 2. The method of claim 1, wherein the strategy defined by the neural network in the actor module is a statistical distribution of actions with respect to states, the statistical distribution being defined by the parameters including a mean and a standard deviation.

7. The method of claim 6 , wherein the parameters of the policy in the actor module are fixed when the reinforcement learning controller is operating in the production mode.

8. 7. The method of claim 6, wherein training the actor module by the critic module includes using an optimization calculation to determine a desired action that maximizes the reward calculated by a critic function, and adjusting the parameters of the policy in the actor module based on the desired action.

9. 2. The method of claim 1, wherein the critic function calculates a reward for a current state and current action as equal to a current actual reward plus a term that includes a weighted expected reward for future states and future actions.

10. 2. The method of claim 1 , wherein the compliance controller includes an admittance control routine that calculates a robot command velocity using a calculation where a first term includes a difference between a target position adjusted by the reinforcement learning controller and a current robot position, and a second term includes the inverse of an admittance gain matrix multiplied by a contact force vector.

11. 2. The method of claim 1, wherein the compliance controller controls the robot in a feedback loop having a first cycle frequency and the reinforcement learning controller controls the compliance controller in a feedback loop having a second cycle frequency, the first cycle frequency being higher than the second cycle frequency.

12. The method of claim 1 , wherein the compliance controller and the reinforcement learning controller are both executed on a robot controller device that provides articulation commands to the robot and receives the state data from the robot as feedback.

13. In the robot assembly skill learning system, a robot controlled by a compliance controller, the robot configured to perform an assembly operation to move a first part to an assembly position with a second part; a reinforcement learning controller in communication with the compliance controller and the robot, the reinforcement learning controller having an actor module and a critic module, the actor module including a neural network that defines a strategy for determining an action in response to state data received as feedback from the robot, and the critic module including a neural network that optimizes a critic function that correlates the state data and the action with an objective of maximizing reward data received as feedback from the robot, the robot performs the assembly task using the reinforcement learning controller in a learning mode, the actor module providing the behaviors as inputs to the compliance controller, and the critic module updating parameters of the actor module's strategy based on the critic function until the actor module demonstrates a predefined performance level for the assembly task; A system in which a robot performs the assembly operations using the reinforcement learning controller in a production mode, the actor module provides the behaviors as inputs to the compliance controller, and the critic module is not used.

14. 14. The system of claim 13, wherein the state data includes positions and velocities of the robot, each having three translational components and three rotational components, and part-to-part contact forces and torques in three directions, and the reward data includes a positive value if the assembly operation is completed successfully and a negative value if the assembly operation is completed unsuccessfully.

15. 14. The system of claim 13, wherein the policy defined by the neural network in the actor module is a statistical distribution of actions with respect to states, the statistical distribution being defined by the parameters including a mean and a standard deviation, and the parameters of the policy in the actor module are fixed when the reinforcement learning controller is operated in the production mode.

16. 16. The system of claim 15, wherein training the actor module by the critic module includes using an optimization calculation to determine a desired action that maximizes a reward calculated by a critic function, and adjusting the parameters of a policy in the actor module based on the desired action.

17. 14. The system of claim 13, wherein the action is a target position adjustment including three translations and three rotations of the first part relative to the second part that is provided to the compliance controller, the target position adjustments combined with a nominal target position bounded by predefined operating limits and used as the input to the compliance controller.

18. 14. The system of claim 13, wherein the compliance controller controls the robot in a feedback loop having a first cycle frequency and the reinforcement learning controller controls the compliance controller in a feedback loop having a second cycle frequency, the first cycle frequency being higher than the second cycle frequency.

19. The system of claim 13 , wherein the compliance controller and the reinforcement learning controller both execute on a robot controller device that provides articulation commands to the robot and receives the state data as feedback from the robot.

20. In the robot assembly skill learning system, a robot controlled by a compliance controller, the robot configured to perform an assembly operation; a reinforcement learning controller in communication with the compliance controller and the robot, the reinforcement learning controller having an actor module and a critic module, the actor module defining a strategy for determining an action in response to state data received as feedback from the robot, and the critic module optimizing a critic function correlating the state data and the action with the objective of maximizing reward data received as feedback from the robot, A system in which the robot first performs the assembly operations using the reinforcement learning controller in a learning mode, the actor module providing the actions as inputs to the compliance controller and the critic module updating parameters of the actor module's policy, and the robot performs the assembly operations using the reinforcement learning controller in a production mode, the critic module not being used.

Images