System and method for robust pivoting to reorient parts during robotic assembly - Patents.com

Abstract

A manipulation controller for reorienting an object by a manipulator of a robotic system is provided, the manipulation controller comprising an interface controller configured to acquire measurement data from sensors disposed on the robotic system, at least one processor, and a memory configured to store a computer-implemented method, the instructions of which include the steps of acquiring measurement data from vision and force sensors disposed on the robotic system, determining an input-output relationship for the object based on a nonlinear static model representing the input-output relationship between contact forces and movements of the object on the worktable, representing the interaction between the object and the manipulator using complementarity constraints to capture the contact state between the object and the manipulator, formulating a representation of the frictional stability of the object based on the nonlinear static model of the external contact with the worktable, formulating a two-level optimization problem to maximize the frictional stability over a position trajectory of the object being manipulated on the worktable, estimating uncertainty values ​​contained in the physical parameters to be compensated by executing the two-level optimization problem, and solving the two-level optimization problem using a nonlinear optimization solver to generate control data for a set of contact forces being applied to the object by using the manipulator.

Description

[Technical field]

[0001] The present invention relates generally to robotic manipulations, and more particularly to a method of robust optimization for pivot manipulations that can be utilized to reorient parts during a robotic assembly process. [Background technology]

[0002] Contact plays a role in all robotic manipulation tasks. While selective use of contact can enable a robot to reason about and manipulate its environment, contact-based reasoning and control tend to be very challenging. Contact models are analytically, algorithmically, and computationally challenging. As a result, little progress has been made in principle on model-based manipulation. The hybrid dynamics of the underlying friction interactions, and the uncertainties surrounding friction parameters, make efficient design of model-based controllers for manipulation challenging.

[0003] Robotic systems are very good at performing pick-and-place operations in structured environments where parts are always delivered in a known orientation, and often only need to grasp an object by applying a constant force and move it to a known location. However, when the robot must touch these objects in order to manipulate them, or when there is uncertainty about the structure made for the object, these systems are very vulnerable.

[0004] In today's state-of-the-art assembly lines, robots are always fed with parts in the desired poses, and then the assembly robot picks and places the parts, fitting them together, and assembles them. This is achieved by using specially designed mechanisms that allow passive control of the parts by design. However, these mechanisms are quite domain-specific, and designing and commissioning these systems is very expensive. Conversely, we want to design modular assembly systems that can perform the desired manipulation of parts without the aid of specially designed part feeders. This can be achieved by designing a robotic system that can utilize planning and control to reorient the various parts during assembly with minimal requirements for specially designed jigs and fixtures. However, manipulation is an underactuated control problem, which is generally very difficult to model and control. It is also desirable for such systems to be able to take into account uncertainties in the physical properties of the parts during operation. For example, it is desirable for the robot to be able to handle parts with uncertainties in their physical properties. For example, manipulation depends on the object's shape, mass, friction properties (e.g., friction coefficient), and the external environment. To allow for generalization, it is desirable for a manipulation system to be able to handle unknown objects even if the estimates of their physical parameters are imprecise.

[0005] The present invention is directed to the design of such a fundamental manipulation method that can be utilized by a robotic system to reorient parts during assembly or packaging. In particular, the present invention presents a pivoting manipulation with two external contacts with the environment. Although pivoting manipulations have been studied before, all previous analyses presented in the known literature consider a sticky contact with the external environment. In contrast, we present a pivoting manipulation that requires a persistent sliding contact with the external environment. Controlling the sliding contacts in the presence of uncertainty is challenging as it leads to an ill-posed optimization problem. Thus, the present invention presents an optimization formulation that can be utilized to perform a robust pivoting by utilizing friction. Summary of the Invention

[0006] It is an object of some embodiments to provide a robotic system configured to optimize the trajectory of a motion of an object during manipulation by optimizing a set of control forces acting on the object while manipulating the object.

[0007] Some embodiments of the present disclosure are based on the recognition that most state-of-the-art assembly systems utilize meticulously engineered part orientation and feed mechanisms.

[0008] Some embodiments are based on the realization that a robot can utilize its external contact to gain additional dexterity for manipulating the pose of objects in its environment. These manipulation problems are important in many applications where a robot is expected to manipulate various initial state parts to accomplish a desired task.

[0009] Some embodiments are based on the recognition that robots are expected to interact with novel workpieces during these manipulation tasks. Without such capabilities, manipulation algorithms are brittle in the presence of uncertainty, and such systems cannot generalize to objects with shapes different from those for which the controller was designed. Currently, only a few technologies enable pose manipulation of objects with unknown characteristics, and the use of these algorithms is limited to very few controlled environments.

[0010] Some embodiments are based on the recognition that a robot can utilize pivot manipulation as a basic primitive to manipulate the pose of an object for the underlying task by utilizing external contacts. However, pivot manipulation under uncertainty is a complex manipulation problem due to the inherent contact constraints in the presence of uncertainties. Currently, there is no known technique capable of performing robust optimization for manipulation with external contacts that can be utilized to perform reliable robust manipulation. This makes optimization and control of robotic manipulation under parameter uncertainty difficult.

[0011] To that end, this disclosure proposes an optimization method for robust trajectory optimization during maneuvers. To solve the underlying maneuvering problem, we propose a two-level trajectory optimization method for generating trajectories in the presence of uncertain parameters. The proposed formulation introduces a novel concept of frictional stability, which is defined as the amount of parameter uncertainty that can be accommodated by redistributing contact forces to maintain the stability of a particular contact configuration during maneuvers.

[0012] In some embodiments, frictional stability is calculated by considering the static equilibrium of the manipulated object in the presence of external contact. The key realization is that the contact forces can be redistributed over multiple contact points, providing a margin of stability to compensate for uncertainties in gravity and moment. Estimating this margin allows for the generation of trajectories that are robust to uncertainties in the parameters of the manipulated object.

[0013] Some embodiments are based on the realization that to ensure the stability of the manipulation task, the robot needs to keep the object in static equilibrium along the planned trajectory. The contact forces at the two external contact points vary depending on the control forces that the robot applies to the object-robot contact points. As a result, the friction stability margin varies over the entire manipulation trajectory, since it depends on the static equilibrium pose of the object.

[0014] Some embodiments of the proposed disclosure are based on the recognition that during a pivot maneuver, an object needs to maintain sliding contact at two external contact points. As a result, the friction force always remains at the boundary of the friction cone during the maneuver trajectory. This leads to an inherent equality constraint of the robust optimization problem for the underlying maneuver problem. The equality constraint leads to an ill-posed optimization problem to generate a robust trajectory for the maneuver. As a result, a novel optimization method for generating robust control trajectories to compensate for uncertainties during two-point pivot maneuvers is required.

[0015] Some embodiments of the present disclosure are based on the recognition that during two-point pivoting, the frictional stability along the maneuver trajectory varies, so that a minimum margin during the maneuver can be estimated. To maximize the stability margin for this maneuver, we can maximize this minimum margin to ensure maneuvering under uncertainty in the parameters of the object being manipulated. This leads to a novel two-level optimization problem that can be solved to compute a robust maneuver trajectory.

[0016] According to some embodiments of the present invention, there is provided a manipulation controller for reorienting an object by a manipulator of a robotic system utilizing external contact of the object during manipulation, the manipulation controller may comprise a computer-implemented method including an interface controller configured to acquire measurement data from sensors disposed on the robotic system, at least one processor, a nonlinear programming module, a nonlinear optimization solver, and an optimization module, and a memory configured to store a nonlinear static model representing an input-output relationship between contact forces and movements for a given object, the method utilizing at least one processor coupled to the memory storing instructions for performing the method, the instructions, when executed by the at least one processor, performing the steps of the method. The steps include obtaining measurement data from vision sensors and force sensors arranged on the robot system; determining an input-output relationship for the object based on a nonlinear static model representing the input-output relationship between contact forces and movement of the object on the worktable; representing the interaction between the object and the manipulator using complementarity constraints to capture the contact state between the object and the manipulator; formulating an expression for the frictional stability of the object based on the nonlinear static model of the external contact with the worktable; formulating a two-level optimization problem to maximize the frictional stability over a position trajectory of the object being manipulated on the worktable; estimating uncertainty values ​​contained in the physical parameters to be compensated using the frictional stability by executing the two-level optimization problem; solving the two-level optimization problem using a nonlinear optimization solver to generate control data for a series of contact forces being applied to the object by using the manipulator; and transmitting control data instructing the manipulator to reorient the object on the worktable according to the series of contact forces.

[0017] Additionally, some embodiments of the present invention provide a computer-implemented method for performing a desired operation by reorienting an object on a worktable by a manipulator of a robotic system utilizing external contact of the object with a portion of the worktable, the method utilizing a processor coupled to a memory storing instructions for performing the method, the instructions, when executed by the processor, performing the steps of the method. The steps include obtaining measurement data from vision sensors and force sensors arranged on the robot system; determining an input-output relationship for the object based on a nonlinear static model representing the input-output relationship between contact forces and movement of the object on the worktable; representing the interaction between the object and the manipulator using complementarity constraints to capture the contact state between the object and the manipulator; formulating an expression of frictional stability of the object based on the nonlinear static model of external contact with the worktable; formulating a two-level optimization problem to maximize the frictional stability over a position trajectory of the object being manipulated on the worktable; estimating uncertainty values ​​contained in the physical parameters to be compensated using the frictional stability by executing the two-level optimization problem; solving the two-level optimization problem using a nonlinear optimization solver to generate control data for a series of contact forces being applied to the object by using the manipulator; and transmitting control data instructing the manipulator to reorient the object on the worktable according to the series of contact forces to obtain a target position of the object.

[0018] In the detailed description that follows, the present disclosure will be further described by way of non-limiting examples of exemplary embodiments of the present disclosure with reference to a number of drawings, in which like reference numerals represent like parts throughout the several views of the drawings. The drawings shown are not necessarily to scale, with emphasis instead generally being placed upon illustrating the principles of the presently disclosed embodiments. [Brief description of the drawings]

[0019] [Figure 1A]10A-10C illustrate several manipulation scenarios in which a part is reoriented by a manipulator arm utilizing an external contact surface, according to an embodiment of the present invention. [Figure 1B] 10A-10C illustrate several manipulation scenarios in which a part is reoriented by a manipulator arm utilizing an external contact surface, according to an embodiment of the present invention. [Diagram 2] FIG. 1 illustrates a free-body diagram of an object in a particular contact configuration in which the object is in contact with two outer walls and with the fingers of a robotic manipulator, in accordance with an embodiment of the present invention. [Figure 3A] FIG. 13 illustrates contact force redistribution along the boundary of a friction cone with uncertainty in the object's mass, according to an embodiment of the invention. [Figure 3B] FIG. 13 illustrates the redistribution of contact forces along the boundary of a friction cone with uncertainty in the location of the object's center of gravity, according to an embodiment of the present invention. [Figure 4] FIG. 13 illustrates an object mass uncertainty margin that can be compensated for by contact force redistribution, according to an embodiment of the present invention. [Diagram 5] FIG. 4 illustrates the variation of stability margins over time and across a particular operating trajectory in accordance with an embodiment of the present invention. [Figure 6] FIG. 2 illustrates a max-min optimization problem to be solved to provide robustness to operations with uncertainty in parameters of the object being manipulated, according to an embodiment of the present invention. [Figure 7] FIG. 2 is a conceptual visualization of a two-level optimization problem that needs to be solved to compensate for uncertainties in the location of an object's mass and center of gravity, according to an embodiment of the present invention. [Figure 8] FIG. 13 illustrates an assembly scenario in which a robotic system is provided with several parts on a tabletop with additional contact surfaces on which the robot can assemble a desired object using pivoting operations, according to an embodiment of the present invention. [Figure 9] FIG. 1 illustrates a packing scenario in which a robotic system is provided with blocks of different sizes that need to be stacked in a desired manner using a proposed pivoting operation, according to an embodiment of the present invention. [Figure 10] FIG. 2 illustrates a sequence of steps used by a manipulation system to perform a desired manipulation task using the proposed method, according to an embodiment of the present invention. [Figure 11A] FIG. 2 is a block diagram illustrating a controller configured to control an actuator system of a robot, according to some embodiments of the present invention. [Figure 11B] FIG. 1 illustrates a proposed manipulation system for creating a robust trajectory optimization problem according to an embodiment of the present invention. [Figure 12] 11A-11C are diagrams illustrating examples of reorienting parts for assembly using pivot manipulation primitives, in accordance with an embodiment of the present invention. [Figure 13] FIG. 13 shows the number of successful pivot tests out of 10 attempts for gear 1 for two different methods. [Figure 14] FIG. 13 is a snapshot of a hardware experiment, showing white pegs and gears. [Figure 15] 14 is a diagram showing parameters of an object, where m, l, and w are the mass, length, and width of the object, respectively. Here, the first elements in l and w are l1 and w1, respectively, shown in FIG. 14, and the second elements in l and w are l2 and w2, respectively, shown in FIG. 14.

[0020] Although the above drawings illustrate the presently disclosed embodiments, other embodiments are contemplated, as noted in the description. This disclosure presents exemplary embodiments by way of representation, and not by way of limitation. Numerous other variations and other embodiments may be devised by those skilled in the art that fall within the scope and spirit of the principles of the presently disclosed embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that the present disclosure may be practiced without these specific details. In other instances, devices and methods are shown only as block diagrams in order to avoid obscuring the present disclosure.

[0022] As used herein and in the appended claims, the terms "for example," "for instance," and "such as," as well as the verbs "comprising," "having," "including," and other verb forms thereof, when used in conjunction with a list of one or more components or other items, should be construed as open-ended, meaning that the list should not be considered to exclude other additional components or other items. The term "based on" means based, at least in part, on. Moreover, it is to be understood that the phraseology and terminology used herein are for purposes of description and should not be regarded as limiting. The headings used herein are for convenience only and do not have any legal or limiting effect.

[0023] Contact is central to most manipulation tasks, since it gives a robot additional dexterity to interact with its environment. Designing a controller that is robust for frictional interactions with objects whose physical properties are uncertain is challenging, since the mechanical stability of the objects depends on these physical properties. Inspired by this challenge, we consider herein the task of pivoting manipulation. In particular, the invention considers the problem of reorienting a part with uncertain mass and Center of Mass (CoM) location using pivoting. The invention concerns ensuring mechanical stability through friction to compensate for uncertainties in the physical properties of the objects.

[0024] Some embodiments of the present disclosure are based on the recognition that designing robust controllers for friction interaction systems is challenging due to the hybrid nature of the underlying friction dynamics. As a result, many traditional robust planning and control techniques cannot be applied to these systems in the presence of uncertainty. Although the concepts of stability margins and Lyapunov stability have been studied considerably in the context of nonlinear dynamical system controller design, such concepts have never been explored in contact-rich manipulation problems. This is most likely due to the fact that controllers must reason about the mechanical stability constraints of friction interactions to ensure stability. Since mechanical stability is highly dependent on the contact configuration during manipulation, controllers must ensure that the desired contact configuration is maintained during the task or that stability can be maintained even when a set of contacts is disturbed. Analyzing such systems is challenging in the presence of friction, leading to differential inclusions.

[0025] Some embodiments of the present disclosure are based on the recognition that friction provides a mechanical stability margin during contact-heavy tasks. The mechanical stability provided by friction is referred to as frictional stability. This frictional stability can be exploited during optimization to enable stability of manipulation even under uncertainty. The present disclosure relates to pivoting manipulation tasks where an object being manipulated must maintain sliding contact with two external surfaces. A robot uses this manipulation to reorient parts that are on a flat surface and manipulate the object into a desired pose to enable grasping or to assist in assembly. FIG. 1A illustrates a tabletop manipulation scenario where a round object 112 is placed on a flat table surface (tabletop) 110. The tabletop is configured to have or be attached with an additional external contact surface 111. In some cases, the additional external contact surface 111 can be provided by a solid object such as a metal block(s) having a substantially flat surface to make contact A and contact B and a mass that is sufficiently heavier than the mass of the object being manipulated. 11A and 11B, the manipulation task of the robot arm of the robot may include pivoting an object 112 from an initial pose 112 to a desired pose 113 using an external contact surface 111 according to a computer-implemented method stored in memory 1130B of the controller (manipulator controller) 1100 of the robot 1150. Similarly, in FIG. 1B, the manipulation task of the robot 1150 may be to manipulate the pose of a peg 120 on a flat surface 110 to a desired pose 121 using a proposed pivot manipulation.

[0026] Some embodiments of the invention provide a computer-implemented method using a processor coupled to a memory storing instructions for performing the computer-implemented method, the processor configured to execute the steps of the instructions according to the method, the computer-implemented method configured to operate a robotic manipulator such that an object performs a pivoting motion that maintains sliding contact with two exterior surfaces.

[0027]

number

[0028]

number

[0029]

number

[0030]

number

[0031]

number

[0032] Equations (1)-(7) above define a non-linear input-output relationship that determines the movement of object 201 on the worktable when a contact force is applied (ie, represents the input-output relationship between contact force and movement of the object).

[0033]

number

[0034] 3A illustrates the variation of contact forces experienced by an object 301 when there is uncertainty about the mass of the object. If the actual mass 330 is less than the estimated one, the friction force at point A 310 will be larger but the friction force at point B 320 will be smaller compared to the nominal case. In contrast, if the actual mass 330 of the object is heavier than the estimated mass, the object may roll down along point B in the clockwise direction. In this case, one can imagine that the friction force at point A 310 will be smaller and the friction force at point B 320 will be larger. However, as long as the friction force is non-zero, the object will be able to stay in contact with the external environment.

[0035] A similar argument could be made for the uncertainty in the location of the Center of Mass (CoM). Figure 3B shows the case where the CoM location 351 is uncertain, but the mass m 331 is known. This also leads to the contact force being redistributed to points A 311 and B 321 compared to the normal case.

[0036] Some embodiments of the present disclosure are based on the realization that the ability to redistribute frictional forces over the two contact points provides a margin of stability to compensate for uncertain gravity and moments. This margin is called frictional stability.

[0037] 4 illustrates the concept of frictional stability margin 460 in the presence of uncertainty in the mass 450, 430 of an object 401. A control contact force 440 is applied to the object 401 at contact point 441. Note that the contact forces 410 and 420 can be redistributed from the nominal contact force to compensate for the uncertainty in the mass of the object 401.

[0038]

number

[0039]

number

[0040]

number

[0041]

number

[0042]

number

[0043]

number

[0044]

number

[0045]

number

[0046]

number

[0047] The above equations (14)-(15) and (21)-(22) represent the nonlinear relationship of the frictional stability of an object by utilizing the redistribution of contact forces for the static equilibrium of the object.

[0048] Some embodiments of the present disclosure are based on the recognition that an optimal control problem of a manipulation task for pivoting an object needs to be designed and solved. In some cases, the contact state and the object state may be captured using a sensor 1101 such as a vision sensor(s) 1102 and a tactile sensor. The object state is represented by the angular position θ 250 of the object in FIG. 2, which can be measured using a vision sensor. There are two types of contacts made by the object during manipulation. There are two contacts between the outer surface and the object, which we call external contacts. For example, their locations are shown as points A and B in FIG. 2, FIG. 3A, FIG. 3B, FIG. 4, and FIG. 7. There is one contact point between the manipulator and the object. For example, this is shown as point P in FIG. 2, FIG. 3A, FIG. 3B, FIG. 4, and FIG. 7. The contact state can be measured either directly using the tactile sensor and the vision sensor or using an estimation method. The tactile sensor 1101 and the vision sensor 1102 can be used to measure the contact state between the manipulator and an object and to monitor slippage between the object and the manipulator. These sensors can also be used to measure the contact state between the object and an external contact surface (external contact) and to monitor slippage therebetween.

[0049]

number

[0050]

number

[0051] In the above equations, matrices Q and R are cost matrices of state and control inputs made by the user to the optimization routine based on the challenge of the manipulation problem. The constraints of the optimization problem (23) specified in (24) are obtained by the conditions of static equilibrium of the object in any contact configuration and the kinematic constraints of the manipulation problem. The parameters of these static equilibrium conditions include the mass of the object and the friction coefficients of the various surfaces. The mass of the object during the manipulation is provided by the user with a certain uncertainty. However, the proposed method requires perfect knowledge of the friction coefficients. These can be obtained by performing system identification experiments with the object and can be provided as shown in Figure 15. The kinematic constraints require a description of the shape of the object, which can be obtained using the vision sensor 1120.

[0052] The output of the optimization algorithm is the optimal sequence of control, state, and contact force trajectories, as shown in the formulation of Eq. (23).

[0053] Some embodiments of the present disclosure are based on the recognition that trajectory optimization formulations need to consider incorporating frictional stability to obtain robustness. In some cases, frictional stability defines the degree to which multiple contact points can compensate for gravity and moments in the presence of uncertainties in the positions of the mass and CoM.

[0054] An important realization in this disclosure is that if (10)-(12) and / or (18)-(22) were simply added to (23)-(25), the optimization problem would be ill-posed since there is no u to satisfy all uncertain realizations in the equality constraints. Thus, this disclosure presents a novel optimization formulation that leads to a formulation that can be solved using nonlinear programming software. For example, the nonlinear programming software may be a nonlinear optimization solver such as Interior Point OPTimizer (IPOPT) and Sparse Nonlinear OPTimizer (SNOPT).

[0055]

number

[0056]

number

[0057] Some embodiments of the present disclosure are based on the recognition that robust manipulation leads to a two-level optimization problem. Figure 7 illustrates a method of the proposed two-level optimization problem 770. For an uncertain mass 701, in the presence of control forces 740, the contact forces 710 and 720 are redistributed to compensate for the uncertainty in the mass of the object 760. Note that the uncertainty (uncertainty value) can be in a positive or negative direction 750. Similarly, for an uncertain CoM position 702 and fixed mass 761, the contact forces 711, 721 are redistributed to compensate for the uncertainty 751 in the CoM position. Depending on the type of uncertainty, the two-level optimization problem maximizes the minimum friction stability over the entire manipulation trajectory.

[0058]

number

[0059]

number

number

[0060] FIG. 8 illustrates an assembly operation scenario where a robot operates a tabletop 800 to assemble a product 840 with parts 820, 830. The desired task is to insert a peg 830 into a hole 850 and then insert a circular gear 820 onto the peg 830. However, the robot cannot grasp the circular gear 820 in its current configuration and cannot insert the peg 830 in its current pose. The flat table 800 has an external vertical flat surface 810 that is fixed but very heavy and cannot be moved by the robot. The robot then calculates a set of control forces using the proposed optimization method to pivot 820 and 830 against the surface 810. The robot can then grasp 830 in the correct insertion pose to insert it into the hole 850. The robot can then grasp the circular part 820 to insert it onto the peg 830 (already in the hole 830).

[0061] Figure 9 illustrates a packing scenario on a flat tabletop 900, where the manipulation task is to pack a block 920 between blocks 930 and 940 in a pose 920. As can be seen, the robot needs to manipulate the pose of the block 920 before it can be packed in the desired pose 920'. The flat platform 900 has an external vertical flat surface 910 that is fixed but very heavy and cannot be moved by the robot. The robot then uses the optimization method proposed in this disclosure to calculate an optimal set of control forces (set of contact forces) to pivot 920 against 910 so that the pose of the object 920 can be manipulated to the desired pose 920'.

[0062] An example of a computer-implemented method for robust optimization for pivoting operations that can be used to reorient parts during a robotic assembly process is described below with reference to FIG. 10 and FIGS. 11A and 11B.

[0063] 10 is a diagram illustrating a computer-implemented method that describes a series of steps used by a manipulation system 1155 of a robotic system 1150, sometimes referred to as a robot 1150, to perform a desired manipulation task, in accordance with an embodiment of the present invention.

[0064] The robot system 1150 (FIG. 11B) includes a manipulator arm 1155, a number of force sensors 1101 arranged on the manipulator arm 1155, and a vision system 1102 (at least one camera). The number of force sensors 1101 (sometimes referred to as at least one force sensor) are configured to detect a force applied by the manipulator arm 1155 to an object at a contact point between the object and the manipulator. The vision system 1102 may be at least one camera or multiple cameras, a depth camera, a distance camera, etc. The vision system 1102 is arranged at a fixed position so that it can observe an object state representing a positional relationship between the object, the tabletop, and the additional contact surface. The vision system 1102 is configured to estimate a pose of an object on the tabletop 900 with the additional contact surface 910 in the environment of the robot system 1150.

[0065] In step 1020, the vision system 1102 detects and estimates the pose of the object to be manipulated. In step 1030, the robot controller 1100 is configured to determine if the part needs to be reoriented before being used for a desired task (e.g., assembly). The controller 1100 is configured to calculate a set of control forces to be applied to the object using a two-level optimization algorithm. In step 1040, the robot 1150 applies a set of control forces (a set of contact forces) to the object against an external contact surface (e.g., 810).

[0066] In step 1050, the manipulation controller 1100 is configured to generate and send control data including instructions for the calculated set of control forces to an underlying robot controller (e.g., a manipulator actuator controller), which causes the manipulator to apply the calculated set of control forces (contact forces) to the object. Then, in step 1060, the robot grasps the reoriented part for use in the desired task (assembly or packaging).

[0067] FIG. 11A is a block diagram illustrating a robot control system 1100 for controlling the movement of a robot, according to an embodiment of the present invention.

[0068] 11B illustrates a robotic system (robot) 1150 for manipulating an object on a tabletop 900 according to a trajectory generated by the proposed robust trajectory optimization problem, according to an embodiment of the present invention. The robot control system 1100 is configured to control the actuator system 1103 of the robot 1150. The robot control system 1100 may be referred to as a control system of the robot or a robot controller.

[0069] The robot control system 1100 may include an interface controller 1110B, a processor 1120, and a storage unit 1130B. The storage unit 1130B is configured to store a computer-implemented method including a nonlinear programming module, a nonlinear optimization solver, and an optimization module, and a nonlinear static model that represents an input-output relationship between contact force and movement for a given object. The processor 1120 may be one or more processor units, and the storage unit 1130B may be a storage device, a data storage device, etc. The interface controller 1110B may be an interface circuit and includes a sensor 1101 including a force sensor and a vision sensor(s) 1102, as well as an analog-to-digital (A / D) converter and a digital-to-analog (D / A) converter for signal / data communication with the motion controller 1150B of the robot 1150. Furthermore, the interface controller 1110B may include a memory for storing data used by the A / D converter or the D / A converter. Sensors 1101 are placed on the robot's joints (robot arm(s) or manipulator) or object picking mechanism 1104 (e.g. fingers) to measure contact states with the robot. These vision sensors can be placed in any location that provides a vantage point to observe / measure object states that represent the positional relationships between the object, the tabletop, and additional contact surfaces.

[0070] The control system 1100 includes an actuator controller (device / circuit) 1150B. The actuator controller 1150B includes a policy unit 1151B for generating action parameters for controlling the robot system 1150 that controls the robot arm 1155, the handling mechanism, or a combination of the arms 1103 including the handling mechanisms 1103-1, 1103-2, 1103-3, and 1103-#n depending on the number of joints or handling fingers. For example, the sensor 1101 may include an acceleration sensor, an angle sensor, a force sensor, or a tactile sensor for measuring the position of the object as well as the external force. For example, the interaction between the robot arm of the robot system and the object can be represented using a complementarity constraint to capture the contact state between the robot arm of the robot system and the object. That is, these interactions are based on the contact state represented by the relationship between the sliding velocity of the object on the tabletop and the friction of the object with the tabletop when the object is moved by the robot arm.

[0071] The interface controller 1110B is also connected to a sensor 1101 mounted on the robot that measures / acquires the state of the robot's motion. In some cases, if the actuators are electric motors, the actuator controller 1150B may control the angle of the robot arm or individual electric motors that drive the handling of the object by the handling mechanism. In some cases, the actuator controller 1150B may control the rotation of individual arms arranged on the arm to smoothly accelerate or safely decelerate the robot's motion in response to policy parameters generated from the computer-implemented method 1000 stored in the storage unit 1130B including the modeling module 1101B for optimization and the optimization module 1140B for control signals. Furthermore, depending on the design of the object handling mechanism, the actuator controller 1150B may control the length of the actuator in response to policy parameters according to instructions generated by the computer-implemented method 1000.

[0072] The control system 1100 is connected to an imager or vision sensor 1102 that provides an RGBD image. In alternative embodiments, the vision sensor 1102 may include a depth camera, a thermal camera, an RGB camera, a computer, a scanner, a mobile device, a webcam, or any combination thereof. In some cases, the vision sensor 1102 may be referred to as a vision system 1102. Signals from the vision sensor 1102 are processed and utilized to classify, recognize, or measure object conditions.

[0073] FIG. 12 illustrates an example of a controlled trajectory of an object during a pivot operation according to the method 1000 of the control system 1100 included in the robotic system 1150.

[0074] These figures show a sequence of actions, poses, and contact configurations (1)-(2)-(3) and (1)-(4)-(5) for reorienting various objects / parts for assembly using pivot manipulation primitives. Such reorientations may be necessary when the part being assembled is not easy to grasp in the initial pose or when the part being inserted during assembly is not in the desired pose (e.g., a peg). The figures show some examples of controller implementations for reorienting thick and thin gears. In FIG. 12, the robot 1150 reorients a part on a flat surface using a manipulator arm with fingers 1104 to manipulate the object into a desired pose for grasping or to assist in assembly. The desired pose may be the target position of the object on the worktable. Note that this manipulation is challenging as it requires controlled sliding, so it is essential to consider the robustness of the control trajectory. Experimental Example

[0075] To demonstrate the effectiveness of the proposed method for gear 1, the controller is implemented using a 6DoF manipulator. To evaluate the robustness for objects with unknown mass, the optimization is solved using a mass different from the true mass of the object and the obtained trajectory is implemented for the object. The implementation results for the gear are shown in Figure 13. To test the proposed algorithm, the real mass of the gear is 140 grams and the controlled trajectory is calculated using the mass listed in Figure 13. It is found that the controlled trajectory calculated using the proposed two-level optimization algorithm is able to succeed in all test trials pivoting an object with considerable uncertainty in its mass.

[0076] 13 shows examples of controlled trajectories of objects during pivot operations according to method 1000 of control system 1100 included in robotic system 1150. These figures show a sequence of actions, poses, and contact configurations for four different objects in FIG. 14 (I)-(IV).

[0077] Figure 15 shows a list of object dimensions and friction coefficients of various surfaces used in calculating the controlled trajectory using the proposed method. In particular, Figure 15 shows the dimensions, masses, and friction coefficients for the four objects shown in Figure 14. Note that the three coefficients of friction in Figure 15 correspond to the three points of contact A, B, and P already shown in Figures 2, 3A, 3B, 4A, 4B, and 7.

[0078] The embodiments of the present disclosure as described above may be implemented in any of numerous ways. For example, the embodiments may be implemented using hardware, software, or a combination thereof. If implemented in software, the software code may be executed on any suitable processor or collection of processors, whether located on one computer or distributed among multiple computers. Such a processor may be implemented as an integrated circuit including one or more processors on a single integrated circuit component. However, a processor may be implemented using circuitry in any suitable format.

[0079] Additionally, embodiments of the invention may be implemented as described methods. The operations performed as part of the method may be ordered as appropriate. As such, embodiments may be constructed that perform operations in an order different from that illustrated, and may include performing some operations simultaneously even though the illustrated embodiments show operations as sequential.

[0080] Moreover, the use of ordinal numbers such as "first," "second," and the like to modify claim elements in the appended claims does not, by itself, imply a preference, precedence, or order in which one claim element is preceded by another claim element, nor does it imply a chronological order in which the actions involved in a method should be performed, but rather merely distinguishes between claim elements (but not the use of ordinal numbers) as a label to distinguish one claim element of a given name from other claim elements of the same name.

[0081] Although the invention has been described by way of examples of preferred embodiments, it will be understood that various other adaptations and modifications may be made without departing from the spirit and scope of the invention.

[0082] Therefore, it is the object of the appended claims to cover all such variations and modifications as come within the true spirit and scope of the invention.

Claims

1. 1. A computer-implemented method for performing a desired operation by reorienting an object on a worktable by a manipulator of a robotic system utilizing external contact of the object with a portion of the worktable, the method utilizing a processor coupled to a memory storing instructions for performing the method, the instructions, when executed by the processor, performing steps of the method, the steps comprising: acquiring measurement data from vision and force sensors disposed on the robotic system; determining an input / output relationship for the object based on a non-linear static model representing an input / output relationship between contact force and movement of the object on the worktable; expressing the interaction between the object and the manipulator using complementarity constraints to capture the contact state between the object and the manipulator; formulating a representation of the frictional stability of the object based on the nonlinear static model of the external contact with the worktable; formulating a bilevel optimization problem to maximize the frictional stability over a position trajectory of the object being manipulated on the worktable; using said friction stability to estimate uncertainty values ​​contained in the physical parameters to be compensated; solving the two-level optimization problem using a nonlinear optimization solver to generate control data for a set of the contact forces being applied to the object by using the manipulator; and transmitting the control data instructing the manipulator to reorient the object on the worktable according to the series of contact forces to obtain a target position of the object.

2. The method of claim 1 , wherein the component of the worktable is positioned to contact at least two external contacts of the object.

3. The method of claim 1 , further comprising generating a positional relationship for estimating the frictional stability taking into account a contact configuration of the object with the work surface.

4. The method of claim 1 , further comprising converting the two-level optimization problem into a single-level large-scale nonlinear programming problem using Karush-Kuhn-Tucker (KKT) conditions and an epigraph trick.

5. The method of claim 1 , wherein the sensor comprises a force sensor (F / T sensor).

6. The method of claim 1 , wherein the uncertainty value comprises an erroneous coefficient of friction.

7. The method of claim 1 , wherein the manipulator is a Six Degree of Freedom (6DoF) manipulator.

8. The method of claim 1 , wherein the manipulator includes a servo motor and a gripper.

9. The method of claim 1 , wherein the step of solving the bi-level optimization problem calculates a frictional stability margin for the object.

10. The method of claim 1 , wherein the bi-level optimization problem is composed of two sub-optimization problems.

11. 1. A manipulation controller for reorienting an object by a manipulator of a robotic system utilizing external contact of the object during manipulation, the manipulation controller comprising: an interface controller configured to acquire measurement data from sensors disposed on the robotic system; At least one processor; 1. A computer-implemented method comprising: a nonlinear programming module; a nonlinear optimization solver; and an optimization module; and a memory configured to store a nonlinear static model representing an input-output relationship between contact forces and movements for a given object, the method utilizing at least one processor coupled to the memory storing instructions for performing the method, the instructions, when executed by the at least one processor, perform steps of the method, the steps including: acquiring measurement data from vision and force sensors disposed on the robotic system; determining an input / output relationship for the object based on a non-linear static model representing an input / output relationship between contact force and movement of the object on the worktable; expressing the interaction between the object and the manipulator using complementarity constraints to capture the contact state between the object and the manipulator; formulating a representation of the frictional stability of the object based on the nonlinear static model of the external contact with the worktable; formulating a bilevel optimization problem to maximize the frictional stability over a position trajectory of the object being manipulated on the worktable; using said friction stability to estimate uncertainty values ​​contained in the physical parameters to be compensated; solving the two-level optimization problem with the nonlinear optimization solver to generate control data for a set of the contact forces being applied to the object by using the manipulator; and transmitting the control data to instruct the manipulator to reorient the object on the worktable in accordance with the series of contact forces.

12. The controller of claim 11 , wherein the controller further generates a position relationship for estimating the friction stability taking into account a contact configuration of the object with the environment.

13. The controller of claim 11 , further comprising: converting the two-level optimization problem into a single-level large-scale nonlinear programming problem using KKT conditions and an epigraph trick.

14. The controller of claim 11 , wherein the component of the worktable is positioned to contact at least two external contacts of the object.

15. The controller of claim 11 , further comprising: generating a position relationship for estimating the friction stability taking into account a contact configuration of the object with the work surface.

16. The controller of claim 11 , further comprising: converting the two-level optimization problem into a single-level large-scale nonlinear programming problem using Karush-Kuhn-Tucker (KKT) conditions and an epigraph trick.

17. The controller of claim 11 , wherein the sensor includes a force sensor (F / T sensor).

18. The controller of claim 11 , wherein the uncertainty value comprises an erroneous coefficient of friction.

19. The controller of claim 11 , wherein the manipulator is a Six Degree of Freedom (6DoF) manipulator.

20. The controller of claim 11 , wherein the manipulator includes a servo motor and a gripper.

21. The controller of claim 11 , wherein solving the bi-level optimization problem calculates a frictional stability margin for the object.

22. The controller of claim 11 , wherein the bi-level optimization problem is composed of two sub-optimization problems.

Images