Systems and methods for robust pivoting to reorient parts during robotic assembly

Abstract

A manipulation controller for reorienting an object by a manipulator of a robotic system is provided, the manipulation controller comprising an interface controller configured to acquire measurement data from sensors disposed on the robotic system, at least one processor, and a memory configured to store a computer-implemented method, the instructions of which include the steps of acquiring measurement data from vision and force sensors disposed on the robotic system, determining an input-output relationship for the object based on a nonlinear static model representing the input-output relationship between contact forces and movements of the object on the worktable, representing the interaction between the object and the manipulator using complementarity constraints to capture the contact state between the object and the manipulator, formulating a representation of the frictional stability of the object based on the nonlinear static model of the external contact with the worktable, formulating a two-level optimization problem to maximize the frictional stability over a position trajectory of the object being manipulated on the worktable, estimating uncertainty values ​​contained in the physical parameters to be compensated by executing the two-level optimization problem, and solving the two-level optimization problem using a nonlinear optimization solver to generate control data for a set of contact forces being applied to the object by using the manipulator.

Description

[Technical Field]

[0001] The present invention relates generally to robotic manipulation, and more particularly to a method for robust optimization for pivot manipulation that can be utilized to reorient parts during a robotic assembly process. [Background technology]

[0002] Contact plays a role in all robotic manipulation tasks. While selective use of contact allows a robot to reason about and manipulate its environment, contact-based reasoning and control tends to be very challenging. Contact models are analytically, algorithmically, and computationally challenging. As a result, little progress has been made in principle on model-based manipulation techniques. The hybrid dynamics of the underlying friction interactions and the uncertainties surrounding friction parameters make efficient design of model-based controllers for manipulation challenging.

[0003] Robotic systems are very good at performing pick-and-place operations in structured environments where parts are always delivered in a known orientation, often simply grasping an object by applying a constant force and moving it to a known location. However, when the robot must come into contact with these objects to manipulate them, or when there is uncertainty in the structure created for the object, these systems are very vulnerable.

[0004] In today's state-of-the-art assembly lines, robots are constantly fed parts in desired poses, and then the assembly robot picks and places the parts, fitting them together, and assembles them. This is achieved using specially designed mechanisms that allow for passive control of the parts. However, these mechanisms are highly domain-specific, and the design and commissioning of these systems is expensive. Conversely, we want to design modular assembly systems that can perform desired manipulations of parts without the aid of specially designed part feeders. This can be achieved by designing a robotic system that can utilize planning and control to reorient various parts during assembly, with minimal requirements for specially designed jigs and fixtures. However, manipulation is an underactuated control problem, which is generally very difficult to model and control. It is also desirable for such systems to account for uncertainty in the material properties of the parts during operation. For example, it is desirable for robots to be able to handle parts with uncertain material properties. For example, manipulation depends on the object's shape, mass, frictional properties (e.g., friction coefficient), and the external environment. To allow for generalization, it is desirable for the manipulation system to be able to handle unknown objects even if the estimates of their physical parameters are imprecise.

[0005] The present invention relates to the design of such a fundamental manipulation method that a robotic system can use to reorient parts during assembly or packaging. In particular, the present invention presents a pivoting manipulation involving two external contacts with the environment. While pivoting manipulation has been studied before, all previous analyses presented in the literature consider solid contact with the external environment. In contrast, we present a pivoting manipulation that requires persistent sliding contact with the external environment. Controlling sliding contact in the presence of uncertainty is challenging, as it leads to an ill-posed optimization problem. Therefore, the present invention presents an optimization formulation that can be used to implement robust pivoting by utilizing friction. Summary of the Invention

[0006] It is an object of some embodiments to provide a robotic system configured to optimize the trajectory of a motion of an object during manipulation by optimizing a set of control forces acting on the object while manipulating the object.

[0007] Some embodiments of the present disclosure are based on the recognition that most state-of-the-art assembly systems utilize meticulously designed part orientation and feeding mechanisms.

[0008] Some embodiments are based on the recognition that a robot can utilize its external contact to gain additional dexterity for manipulating the pose of objects in its environment. These manipulation problems are important in many applications where a robot is expected to manipulate various initial state parts to accomplish a desired task.

[0009] Some embodiments recognize that robots are expected to interact with novel workpieces during these manipulation tasks. Without such capabilities, manipulation algorithms are brittle in the presence of uncertainty, and such systems cannot generalize to objects with shapes different from those for which the controller was designed. Currently, very few technologies enable pose manipulation of objects with unknown properties, and the use of these algorithms is limited to very few controlled environments.

[0010] Some embodiments are based on the recognition that robots can utilize pivoting manipulation as a fundamental primitive to manipulate the pose of an object for the underlying task by utilizing external contact. However, pivoting manipulation under uncertainty is a complex manipulation problem due to the inherent contact constraints in the presence of uncertainty. Currently, there are no known techniques capable of performing robust optimization for manipulation with external contact that can be utilized to perform reliable and robust manipulation. This makes optimization and control of robotic manipulation under parameter uncertainty challenging.

[0011] To this end, this disclosure proposes an optimization method for robust trajectory optimization during maneuvers. To solve the underlying maneuvering problem, we propose a two-level trajectory optimization method for generating trajectories in the presence of parameter uncertainty. The proposed formulation introduces a novel concept of frictional stability, which is defined as the amount of parameter uncertainty that can be accommodated by redistributing contact forces to maintain the stability of a particular contact configuration during maneuvers.

[0012] In some embodiments, frictional stability is calculated by considering the static equilibrium of the manipulated object in the presence of external contact. The key realization is that the ability to redistribute contact forces over multiple contact points provides a margin of stability to compensate for uncertainties in gravity and momentum. Estimating this margin allows for the generation of trajectories that are robust to uncertainties in the parameters of the manipulated object.

[0013] Some embodiments are based on the recognition that to ensure the stability of a manipulation task, a robot needs to maintain an object in static equilibrium along a planned trajectory. The contact forces at the two external contact points vary depending on the control forces that the robot applies to the object-robot contact points. As a result, the friction stability margin varies with the static equilibrium pose of the object and therefore varies throughout the manipulation trajectory.

[0014] Some embodiments of the proposed disclosure are based on the recognition that during a pivot maneuver, an object needs to maintain sliding contact at two external contact points. As a result, friction forces always remain at the boundary of the friction cone during the maneuver trajectory. This leads to inherent equality constraints in the robust optimization problem for the underlying maneuver problem. The equality constraints lead to an ill-posed optimization problem generating a robust trajectory for the maneuver. As a result, a novel optimization method for generating robust control trajectories to compensate for uncertainties during two-point pivot maneuvers is needed.

[0015] Some embodiments of the present disclosure are based on the recognition that during a two-point pivot, frictional stability along the maneuver trajectory varies, so that a minimum margin can be estimated during the maneuver. To maximize the stability margin for this maneuver, we can maximize this minimum margin so that the maneuver can be reliably operated under uncertainty in the parameters of the object being manipulated. This leads to a novel two-level optimization problem that can be solved to compute a robust maneuver trajectory.

[0016] According to some embodiments of the present invention, there is provided a manipulation controller for reorienting an object by a manipulator of a robotic system using external contact of the object during manipulation, the manipulation controller may comprise an interface controller configured to acquire measurement data from sensors disposed on the robotic system, a computer-implemented method including at least one processor, a nonlinear programming module, a nonlinear optimization solver, and an optimization module, and a memory configured to store a nonlinear static model representing an input-output relationship between contact forces and movements for a given object, the method utilizing at least one processor coupled to the memory storing instructions for performing the method, the instructions, when executed by the at least one processor, performing the steps of the method. The steps include acquiring measurement data from vision sensors and force sensors disposed on the robot system; determining an input-output relationship for the object based on a nonlinear static model representing the input-output relationship between contact forces and movement of the object on the worktable; representing the interaction between the object and the manipulator using complementarity constraints to capture the contact state between the object and the manipulator; formulating an expression for the frictional stability of the object based on the nonlinear static model of external contact with the worktable; formulating a two-level optimization problem to maximize the frictional stability over a position trajectory of the object being manipulated on the worktable; using the frictional stability to estimate uncertainty values ​​contained in the physical parameters to be compensated by executing the two-level optimization problem; solving the two-level optimization problem using a nonlinear optimization solver to generate control data for a series of contact forces being applied to the object by using the manipulator; and transmitting the control data to instruct the manipulator to reorient the object on the worktable according to the series of contact forces.

[0017] Furthermore, some embodiments of the present invention provide a computer-implemented method for performing a desired operation by reorienting an object on a work surface by a manipulator of a robotic system using external contact of the object with a portion of the work surface, the method utilizing a processor coupled to a memory storing instructions for performing the method, the instructions, when executed by the processor, performing the steps of the method. The steps include acquiring measurement data from vision sensors and force sensors disposed on the robot system; determining an input-output relationship for the object based on a nonlinear static model representing the input-output relationship between contact force and movement of the object on the worktable; representing the interaction between the object and the manipulator using complementarity constraints to capture the contact state between the object and the manipulator; formulating an expression for the frictional stability of the object based on the nonlinear static model of external contact with the worktable; formulating a two-level optimization problem to maximize the frictional stability over a position trajectory of the object being manipulated on the worktable; using the frictional stability to estimate uncertainty values ​​contained in the physical parameters to be compensated by executing the two-level optimization problem; solving the two-level optimization problem using a nonlinear optimization solver to generate control data for a series of contact forces being applied to the object by using the manipulator; and transmitting the control data to instruct the manipulator to reorient the object on the worktable according to the series of contact forces to obtain a target position of the object.

[0018] In the detailed description that follows, the present disclosure will be further described by way of non-limiting examples of exemplary embodiments of the present disclosure with reference to the drawings, in which like reference numerals represent like parts throughout the several views of the drawings. The drawings shown are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the presently disclosed embodiments. [Brief explanation of the drawings]

[0019] [Figure 1A]10A-10C illustrate several manipulation scenarios in which a part is reoriented by a manipulator arm utilizing an external contact surface, according to an embodiment of the present invention. [Figure 1B] 10A-10C illustrate several manipulation scenarios in which a part is reoriented by a manipulator arm utilizing an external contact surface, according to an embodiment of the present invention. [Figure 2] 1A is a free-body diagram of an object in a particular contact configuration, where the object is in contact with two outer walls and the fingers of a robotic manipulator, according to an embodiment of the present invention. [Figure 3A] FIG. 10 illustrates contact force redistribution along the boundary of a friction cone with uncertainty in the object's mass, according to an embodiment of the present invention. [Figure 3B] FIG. 10 illustrates contact force redistribution along the boundary of a friction cone with uncertainty in the location of the object's center of gravity, according to an embodiment of the present invention. [Figure 4] FIG. 10 illustrates the object mass uncertainty margin that can be compensated for by contact force redistribution, according to an embodiment of the present invention. [Figure 5] FIG. 10 illustrates the variation of stability margins over time and across a particular operating trajectory, according to an embodiment of the present invention. [Figure 6] FIG. 2 illustrates a max-min optimization problem to be solved to provide robustness to operations with uncertainty in the parameters of the object being manipulated, according to an embodiment of the present invention. [Figure 7] FIG. 2 is a conceptual visualization of a two-level optimization problem that needs to be solved to compensate for uncertainties in the mass and center of gravity location of an object, according to an embodiment of the present invention. [Figure 8] FIG. 10 illustrates an assembly scenario in which a robotic system is provided with several parts on a tabletop with additional contact surfaces on which the robot can assemble a desired object using pivoting operations, according to an embodiment of the present invention. [Figure 9] 1 illustrates a packing scenario in which a robotic system is provided with blocks of different sizes that need to be stacked in a desired manner using a proposed pivoting operation, according to an embodiment of the present invention. [Figure 10] FIG. 2 illustrates a sequence of steps used by a manipulation system to perform a desired manipulation task using the proposed method, according to an embodiment of the present invention. [Figure 11A] FIG. 2 is a block diagram illustrating a controller configured to control an actuator system of a robot, according to some embodiments of the present invention. [Figure 11B] FIG. 1 illustrates a proposed operating system for creating a robust trajectory optimization problem according to an embodiment of the present invention. [Figure 12] 10A-10C illustrate examples of reorienting parts for assembly using pivot manipulation primitives, in accordance with an embodiment of the present invention. [Figure 13] FIG. 10 shows the number of successful pivot tests out of 10 attempts for gear 1 for two different methods. [Figure 14] FIG. 10 is a diagram showing a snapshot of a hardware experiment, showing white pegs and gears. [Figure 15] 14 is a diagram showing parameters of an object. m, l, and w are a diagram showing the mass, length, and width of the object, respectively. Here, the first elements of l and w are l1 and w1, respectively, shown in FIG. 14, and the second elements of l and w are l2 and w2, respectively, shown in FIG. 14.

[0020] While the above drawings illustrate the presently disclosed embodiments, other embodiments are contemplated, as noted in the description. This disclosure presents exemplary embodiments by way of representation and not limitation. Numerous other variations and embodiments may be devised by those skilled in the art that fall within the scope and spirit of the principles of the presently disclosed embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0021] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details. In other instances, devices and methods are shown only in block diagram form in order to avoid obscuring the present disclosure.

[0022] As used in this specification and the appended claims, the terms "for example," "for instance," and "such as," and the verbs "comprising," "having," and "including" and other verb forms thereof, when used in conjunction with a list of one or more components or other items, should each be construed as open-ended, meaning that the list should not be considered to exclude other, additional components or other items. The term "based on" means based, at least in part, on. Furthermore, it should be understood that the phraseology and terminology used herein are for descriptive purposes and should not be regarded as limiting. The headings used herein are for convenience only and do not have any legal or limiting effect.

[0023] Contact is central to most manipulation tasks, as it provides additional dexterity for a robot to interact with its environment. Designing a controller that is robust for frictional interactions with objects whose physical properties are uncertain is challenging, as the mechanical stability of the object depends on these physical properties. Inspired by this challenge, this paper considers the task of pivotal manipulation. In particular, the present invention considers the problem of using pivots to reorient a part whose mass and center of mass (CoM) location are uncertain. The present invention is concerned with ensuring mechanical stability through friction to compensate for uncertainties in the object's physical properties.

[0024] Some embodiments of the present disclosure recognize that designing robust controllers for frictional interaction systems is challenging due to the hybrid nature of the underlying frictional dynamics. As a result, many traditional robust planning and control techniques cannot be applied to these systems in the presence of uncertainty. While the concepts of stability margins and Lyapunov stability have been extensively studied in the context of nonlinear dynamical system controller design, these concepts have not been explored in contact-rich manipulation problems. This is likely due to the fact that, in most cases, controllers must reason about the mechanical stability constraints of frictional interactions to ensure stability. Because mechanical stability is highly dependent on the contact configuration during manipulation, controllers must ensure that the desired contact configuration is maintained during the task or that stability can be maintained even when a series of contacts are disturbed. Analyzing such systems is challenging in the presence of friction, leading to differential inclusions.

[0025] Some embodiments of the present disclosure recognize that friction provides a mechanical stability margin during contact-intensive tasks. This mechanical stability is referred to as frictional stability. This frictional stability can be utilized during optimization to enable manipulation stability even under uncertainty. This disclosure relates to pivotal manipulation tasks in which an object being manipulated must maintain sliding contact with two external surfaces. A robot can use this manipulation to reorient parts on a flat surface and manipulate the object into a desired pose for grasping or to assist in assembly. FIG. 1A illustrates a tabletop manipulation scenario in which a round object 112 is placed on a flat table surface (tabletop) 110. The tabletop is configured to have or be attached to an additional external contact surface 111. In some cases, the additional external contact surface 111 can be provided by a solid object, such as a metal block(s), with a substantially flat surface to create contact A and contact B and a mass significantly heavier than the mass of the object being manipulated. 11A and 11B, the manipulation task of the robot's robotic arm may include utilizing an external contact surface 111 to pivot an object 112 from an initial pose 112 to a desired pose 113 according to a computer-implemented method stored in memory 1130B of the controller (manipulator controller) 1100 of the robot 1150. Similarly, in FIG. 1B, the manipulation task of the robot 1150 may be to manipulate the pose of a peg 120 resting on a flat surface 110 to a desired pose 121 using a proposed pivoting manipulation.

[0026] Some embodiments of the present invention provide a computer-implemented method using a processor coupled to a memory storing instructions for performing the computer-implemented method, the processor configured to perform the steps of the instructions according to the method, the computer-implemented method configured to operate a robotic manipulator to perform a pivoting motion in which an object maintains sliding contact with two exterior surfaces.

[0027]

number

[0028]

number

[0029]

number

[0030]

number

[0031]

number

[0032] The above equations (1) to (7) define a nonlinear input-output relationship that determines the movement of the object 201 on the worktable when a contact force is applied (ie, they represent the input-output relationship between the contact force and the movement of the object).

[0033]

number

[0034] 3A illustrates the variation in contact force experienced by an object 301 when there is uncertainty in the object's mass. If the actual mass 330 is less than estimated, the friction force at point A 310 will be larger but the friction force at point B 320 will be smaller compared to the nominal case. In contrast, if the object's actual mass 330 is heavier than the estimated mass, the object may roll clockwise along point B. In this case, one can imagine that the friction force at point A 310 will be smaller and the friction force at point B 320 will be larger. However, as long as the friction force is non-zero, the object will remain in contact with the external environment.

[0035] A similar argument can be made for uncertainty in the location of the Center of Mass (CoM). Figure 3B shows the case where the CoM location 351 is uncertain, but the mass m 331 is known. This also leads to a redistribution of the contact force to points A 311 and B 321 compared to the normal case.

[0036] Some embodiments of the present disclosure are based on the recognition that the ability to redistribute frictional forces over two contact points provides a margin of stability to compensate for uncertain gravitational forces and moments. This margin is called frictional stability.

[0037] Figure 4 illustrates the concept of frictional stability margin 460 when there is uncertainty in the mass 450, 430 of object 401. A control contact force 440 is applied to object 401 at contact point 441. Note that contact forces 410 and 420 can be redistributed from the nominal contact force to compensate for the uncertainty in the mass of object 401.

[0038]

number

[0039]

number

[0040]

number

[0041]

number

[0042]

number

[0043]

number

[0044]

number

[0045]

number

[0046]

number

[0047] The above equations (14)-(15) and (21)-(22) represent the nonlinear relationship of the frictional stability of an object by utilizing the redistribution of contact forces for the static equilibrium of the object.

[0048] Some embodiments of the present disclosure are based on the recognition that it is necessary to design and solve an optimal control problem for a manipulation task to pivot an object. In some cases, contact and object states may be captured using sensors 1101, such as vision sensor(s) 1102 and tactile sensors. The object state is represented by the object's angular position θ 250 in FIG. 2 , which can be measured using the vision sensors. There are two types of contacts made by the object during manipulation. There are two contacts between the outer surface and the object, which we call external contacts. For example, their locations are shown as points A and B in FIGS. 2, 3A, 3B, 4, and 7. There is one contact point between the manipulator and the object. For example, this is shown as point P in FIGS. 2, 3A, 3B, 4, and 7. The contact state can be measured either directly using the tactile and vision sensors or by using estimation methods. The tactile sensor 1101 and the vision sensor 1102 can be used to measure the contact state between the manipulator and an object and to monitor slippage between the object and the manipulator. These sensors can also be used to measure the contact state between the object and an external contact surface (external contact) and to monitor slippage therebetween.

[0049]

number

[0050]

number

[0051] In the above equations, matrices Q and R are cost matrices of state and control inputs provided to the optimization routine by the user based on the manipulation problem problem. The constraints of the optimization problem (23) specified in (24) are obtained by the static equilibrium conditions of the object in any contact configuration and the kinematic constraints of the manipulation problem. The parameters of these static equilibrium conditions include the mass of the object and the friction coefficients of various surfaces. The mass of the object during manipulation is provided by the user with a certain uncertainty. However, the proposed method requires perfect knowledge of the friction coefficients. These can be obtained by conducting system identification experiments with the object, as shown in Figure 15. The kinematic constraints require a description of the object's shape, which can be obtained using the vision sensor 1120.

[0052] The output of the optimization algorithm is the optimal sequence of control, state, and contact force trajectories, as shown in the formulation of Equation (23).

[0053] Some embodiments of the present disclosure recognize that trajectory optimization formulations must consider incorporating frictional stability to obtain robustness. In some cases, frictional stability defines the degree to which multiple contact points can compensate for gravity and moment forces in the presence of uncertainty in the mass and CoM positions.

[0054] A key realization in this disclosure is that simply adding (10)-(12) and / or (18)-(22) to (23)-(25) results in an ill-posed optimization problem, since there is no u to satisfy all uncertain realizations in the equality constraints. Thus, this disclosure presents a novel optimization formulation that leads to a formulation that can be solved using nonlinear programming software. For example, the nonlinear programming software can be a nonlinear optimization solver such as Interior Point OPTimizer (IPOPT) and Sparse Nonlinear OPTimizer (SNOPT).

[0055]

number

[0056]

number

[0057] Some embodiments of the present disclosure are based on the recognition that robust manipulation leads to a two-level optimization problem. FIG. 7 illustrates a proposed two-level optimization method 770. For an uncertain mass 701, in the presence of control forces 740, contact forces 710 and 720 are redistributed to compensate for the uncertainty in the mass of the object 760. Note that the uncertainty (uncertainty value) can exist in either a positive or negative direction 750. Similarly, for an uncertain CoM position 702 and fixed mass 761, contact forces 711, 721 are redistributed to compensate for the uncertainty 751 in the CoM position. Depending on the type of uncertainty, the two-level optimization problem maximizes minimum friction stability over the entire manipulation trajectory.

[0058]

number

[0059]

number

number

[0060] FIG. 8 illustrates an assembly scenario in which a robot operates a tabletop 800 to assemble a product 840 using parts 820 and 830. The desired task is to insert a peg 830 into a hole 850 and then insert a circular gear 820 onto the peg 830. However, the robot cannot grasp the circular gear 820 in its current configuration and cannot insert the peg 830 in its current pose. The flat table 800 has an external vertical flat surface 810 that is fixed but too heavy for the robot to move. The robot then uses the proposed optimization method to calculate a set of control forces to pivot parts 820 and 830 against surface 810. The robot can then grasp part 830 in the correct insertion pose and insert it into hole 850. The robot can then grasp the circular part 820 and insert it onto the peg 830 (which is already in hole 830).

[0061] 9 illustrates a packing scenario on a flat tabletop 900, where the manipulation task is to pack block 920 between blocks 930 and 940 in pose 920. As can be seen, the robot needs to manipulate the pose of block 920 before it can be packed in the desired pose 920'. The flat tabletop 900 has an external vertical flat surface 910 that is fixed but too heavy for the robot to move. The robot then uses the optimization method proposed in this disclosure to calculate an optimal set of control forces (a set of contact forces) to apply and pivot 920 against 910 so that the pose of object 920 can be manipulated to the desired pose 920'.

[0062] An example of a computer-implemented method for robust optimization for pivoting operations that can be used to reorient parts during a robotic assembly process is described below with reference to FIG. 10 and FIGS. 11A and 11B.

[0063] 10 is a diagram illustrating a computer-implemented method that describes a series of steps used by a manipulation system 1155 of a robotic system 1150, sometimes referred to as a robot 1150, to perform a desired manipulation task, according to an embodiment of the present invention.

[0064] The robot system 1150 (FIG. 11B) includes a manipulator arm 1155, multiple force sensors 1101 arranged on the manipulator arm 1155, and a vision system 1102 (at least one camera). The multiple force sensors 1101 (sometimes referred to as "at least one force sensor") are configured to detect forces applied by the manipulator arm 1155 to an object at a contact point between the object and the manipulator. The vision system 1102 may be at least one camera or multiple cameras, a depth camera, a distance camera, etc. The vision system 1102 is positioned at a fixed position so that it can observe an object state representing the positional relationship between the object, the tabletop, and the additional contact surface. The vision system 1102 is configured to estimate the pose of an object on the tabletop 900 with the additional contact surface 910 in the environment of the robot system 1150.

[0065] In step 1020, the vision system 1102 detects and estimates the pose of the object to be manipulated. In step 1030, the robot controller 1100 is configured to determine whether the part needs to be reoriented before being used for a desired task (e.g., assembly). The controller 1100 is configured to calculate a set of control forces to be applied to the object using a two-level optimization algorithm. In step 1040, the robot 1150 applies a set of control forces (a set of contact forces) to the object against an external contact surface (e.g., 810).

[0066] In step 1050, the manipulation controller 1100 is configured to generate and send control data including instructions for the calculated set of control forces to a subordinate robot controller (e.g., a manipulator actuator controller), which causes the manipulator to apply the calculated set of control forces (contact forces) to the object. Thereafter, in step 1060, the robot grasps the reoriented part so that it can be used for the desired task (assembly or packaging).

[0067] FIG. 11A is a block diagram illustrating a robot control system 1100 for controlling the movement of a robot, according to an embodiment of the present invention.

[0068] 11B illustrates a robotic system (robot) 1150 for manipulating an object on a tabletop 900 according to a trajectory generated by the proposed robust trajectory optimization problem, according to an embodiment of the present invention. The robot control system 1100 is configured to control the actuator system 1103 of the robot 1150. The robot control system 1100 may also be referred to as the control system of the robot or the robot controller.

[0069] The robot control system 1100 may include an interface controller 1110B, a processor 1120, and a memory 1130B. The memory 1130B is configured to store a computer-implemented method, including a nonlinear programming module, a nonlinear optimization solver, and an optimization module, as well as a nonlinear static model representing the input-output relationship between contact force and movement for a given object. The processor 1120 may be one or more processor units, and the memory 1130B may be a memory device, a data storage device, or the like. The interface controller 1110B may be an interface circuit and includes sensors 1101, including force sensors and vision sensor(s) 1102, as well as an analog-to-digital (A / D) converter and a digital-to-analog (D / A) converter for signal / data communication with the motion controller 1150B of the robot 1150. Furthermore, the interface controller 1110B may include a memory for storing data used by the A / D converter or D / A converter. Sensors 1101 are placed on the robot's joints (robot arm(s) or manipulator) or object picking mechanism 1104 (e.g., fingers) to measure contact states with the robot. These vision sensors can be placed in any position that provides a vantage point for observing / measuring object states that represent the positional relationship between the object, the tabletop, and additional contact surfaces.

[0070] The control system 1100 includes an actuator controller (device / circuit) 1150B. The actuator controller 1150B includes a policy unit 1151B for generating action parameters for controlling the robot system 1150, which controls a robot arm 1155, a handling mechanism, or a combination of arms 1103 including handling mechanisms 1103-1, 1103-2, 1103-3, and 1103-#n depending on the number of joints or handling fingers. For example, the sensor 1101 may include an acceleration sensor, an angle sensor, a force sensor, or a tactile sensor for measuring the position of an object as well as external forces. For example, the interaction between the robot arm of the robot system and an object can be expressed using a complementarity constraint to capture the contact state between the robot arm of the robot system and the object. That is, these interactions are based on the contact state represented by the relationship between the sliding velocity of the object on a tabletop when the object is moved by the robot arm and the friction of the object with the tabletop.

[0071] The interface controller 1110B is also connected to sensors 1101 mounted on the robot that measure / acquire the state of the robot's motion. In some cases, if the actuators are electric motors, the actuator controller 1150B may control the angle of the robot arm or individual electric motors that drive the handling of an object by the handling mechanism. In some cases, the actuator controller 1150B may control the rotation of individual arms disposed on the arm to smoothly accelerate or safely decelerate the robot's motion in response to policy parameters generated from a computer-implemented method 1000 stored in a memory unit 1130B that includes a modeling module for optimization 1101B and an optimization module for control signals 1140B. Furthermore, depending on the design of the object handling mechanism, the actuator controller 1150B may control the length of the actuator in response to policy parameters corresponding to instructions generated by the computer-implemented method 1000.

[0072] The control system 1100 is connected to an image capture device or vision sensor 1102 that provides an RGBD image. In alternative embodiments, the vision sensor 1102 may include a depth camera, a thermal camera, an RGB camera, a computer, a scanner, a mobile device, a webcam, or any combination thereof. In some cases, the vision sensor 1102 may be referred to as a vision system 1102. Signals from the vision sensor 1102 are processed and used to classify, recognize, or measure object conditions.

[0073] FIG. 12 illustrates an example of a controlled trajectory of an object during a pivoting operation according to the method 1000 of the control system 1100 included in the robotic system 1150.

[0074] These figures show a series of actions, poses, and contact configurations (1)-(2)-(3) and (1)-(4)-(5) for reorienting various objects / components for assembly using pivot manipulation primitives. Such reorientations may be necessary when the component being assembled is not easily grasped in its initial pose or when the component being inserted during assembly is not in the desired pose (e.g., a peg). The figures show several examples of controller implementations for reorienting thick and thin gears. In Figure 12, a robot 1150 uses a manipulator arm with fingers 1104 to reorient a component on a flat surface, manipulating the object into a desired pose for grasping or to assist in assembly. The desired pose can be the target position of an object on a worktable. Note that this manipulation is challenging because it requires controlled sliding, making it essential to consider the robustness of the control trajectory. Experimental example

[0075] To demonstrate the effectiveness of the proposed method for gear 1, the controller is implemented using a 6DoF manipulator. To evaluate the robustness for an object with unknown mass, the optimization is solved using a mass different from the true mass of the object, and the resulting trajectory is implemented for the object. The implementation results for the gear are shown in Figure 13. The actual mass of the gear is 140 grams, and the controlled trajectory is calculated using the mass listed in Figure 13 to test the proposed algorithm. It was found that the controlled trajectory calculated using the proposed two-level optimization algorithm can succeed in all test trials of pivoting an object with considerable uncertainty in its mass.

[0076] 13A-13C are diagrams illustrating examples of controlled trajectories of objects during pivoting operations according to method 1000 of control system 1100 included in robotic system 1150. These diagrams show a series of actions, poses, and contact configurations for four different objects, (I)-(IV) of FIG.

[0077] Figure 15 shows a list of the dimensions of the objects and the friction coefficients of various surfaces used in calculating the controlled trajectory using the proposed method. In particular, Figure 15 shows the dimensions, mass, and friction coefficients for the four objects shown in Figure 14. Note that the three coefficients of friction in Figure 15 correspond to the three points of contact A, B, and P already shown in Figures 2, 3A, 3B, 4A, 4B, and 7.

[0078] The embodiments of the present disclosure described above may be implemented in any of numerous ways. For example, embodiments may be implemented using hardware, software, or a combination thereof. If implemented in software, the software code may be executed on any suitable processor or collection of processors, whether located on a single computer or distributed among multiple computers. Such a processor may be implemented as an integrated circuit including one or more processors on a single integrated circuit component. However, a processor may be implemented using circuitry in any suitable format.

[0079] Additionally, embodiments of the present invention may be implemented as described methods. The actions performed as part of the method may be ordered as appropriate. Thus, embodiments may be constructed that perform actions in an order different from the illustrated order, and may include performing some actions simultaneously even though the illustrated embodiments show sequential actions.

[0080] Furthermore, the use of ordinal numbers such as "first," "second," etc. to modify claim elements in the appended claims does not, by itself, imply a preference, precedence, or order in which one claim element precedes another, nor does it imply a chronological order in which actions within a method should be performed; rather, it merely serves to distinguish claim elements (however ordinal numbers are used) as a label to distinguish one named claim element from another named claim element.

[0081] Although the invention has been described by way of examples of preferred embodiments, it will be understood that various other adaptations and modifications may be made without departing from the spirit and scope of the invention.

[0082] Therefore, it is the object of the appended claims to cover all such variations and modifications as come within the true spirit and scope of the invention.

Claims

1. 1. A computer-implemented method of operation for performing a desired operation by reorienting an object on a worktable by a manipulator of a robotic system utilizing external contact of the object with an additional external contact surface of the worktable, the method utilizing a processor coupled to a memory storing instructions for performing the method, the instructions, when executed by the processor, performing steps of the method, the steps comprising: The method includes a step of acquiring measurement data from a vision sensor and a force sensor arranged on the robot system, the vision sensor being arranged at an arbitrary position from which a viewpoint for observing / measuring an object state representing a positional relationship between the object, the work table, and the additional external contact surface can be obtained, and the force sensor is configured to detect a force applied by the manipulator to the object at a contact point between the object and the manipulator, and the step further includes: determining an input-output relationship for the object based on a nonlinear static model representing an input-output relationship between contact force and movement of the object on the worktable; expressing the interaction between the object and the manipulator using complementarity constraints to capture contact states between the object and the manipulator; formulating a representation of the frictional stability of the object based on the nonlinear static model of the external contact with the worktable; formulating a bilevel optimization problem to maximize the frictional stability over a position trajectory of the object being manipulated on the worktable; using said friction stability to estimate uncertainty values ​​contained in the physical parameters to be compensated; solving the two-level optimization problem using a nonlinear optimization solver to generate control data for the set of contact forces being applied to the object by using the manipulator; transmitting the control data instructing the manipulator to reorient the object on the worktable in accordance with the series of contact forces to obtain a target position of the object; The method further includes generating a positional relationship for estimating the frictional stability taking into account a contact configuration of the object with the work surface.

2. 1. A computer-implemented method of operation for performing a desired operation by reorienting an object on a worktable by a manipulator of a robotic system utilizing external contact of the object with an additional external contact surface of the worktable, the method utilizing a processor coupled to a memory storing instructions for performing the method, the instructions, when executed by the processor, performing steps of the method, the steps comprising: The method includes a step of acquiring measurement data from a vision sensor and a force sensor arranged on the robot system, the vision sensor being arranged at an arbitrary position from which a viewpoint for observing / measuring an object state representing a positional relationship between the object, the work table, and the additional external contact surface can be obtained, and the force sensor is configured to detect a force applied by the manipulator to the object at a contact point between the object and the manipulator, and the step further includes: determining an input-output relationship for the object based on a nonlinear static model representing an input-output relationship between contact force and movement of the object on the worktable; expressing the interaction between the object and the manipulator using complementarity constraints to capture contact states between the object and the manipulator; formulating a representation of the frictional stability of the object based on the nonlinear static model of the external contact with the worktable; formulating a bilevel optimization problem to maximize the frictional stability over a position trajectory of the object being manipulated on the worktable; using said friction stability to estimate uncertainty values ​​contained in the physical parameters to be compensated; solving the two-level optimization problem using a nonlinear optimization solver to generate control data for the set of contact forces being applied to the object by using the manipulator; transmitting the control data instructing the manipulator to reorient the object on the worktable in accordance with the series of contact forces to obtain a target position of the object; The method further comprises converting the two-level optimization problem into a single-level large-scale nonlinear programming problem using KKT (Karush-Kuhn-Tucker) conditions and an epigraph trick.

3. The method of claim 1 or 2, wherein the worktable components are arranged to contact at least two external contacts of the object.

4. The method of claim 1 or 2, wherein the uncertainty value comprises an erroneous coefficient of friction.

5. the manipulator is a 6DoF (Six Degree of Freedom) manipulator, and / or The method of claim 1 or 2, wherein the manipulator includes a servo motor and a gripper.

6. The step of solving the bilevel optimization problem includes calculating a friction stability margin of the object; and / or The method of claim 1 or 2, wherein the two-level optimization problem is composed of two sub-optimization problems.

7. 1. A manipulation controller for reorienting an object by a manipulator of a robotic system utilizing external contact of the object with an additional external contact surface of a worktable during manipulation, the manipulation controller comprising: an interface controller configured to acquire measurement data from sensors disposed on the robotic system; at least one processor; 1. A computer-implemented method comprising: a nonlinear programming module; a nonlinear optimization solver; and an optimization module; and a memory configured to store a nonlinear static model representing an input-output relationship between contact forces and movements for a given object, the method utilizing at least one processor coupled to the memory storing instructions for performing the method, the instructions, when executed by the at least one processor, performing steps of the method, the steps including: The method includes a step of acquiring measurement data from a vision sensor and a force sensor arranged on the robot system, the vision sensor being arranged at an arbitrary position from which a viewpoint for observing / measuring an object state representing a positional relationship between the object and the additional external contact surface can be obtained, and the force sensor being configured to detect a force applied by the manipulator to the object at a contact point between the object and the manipulator, and the step further includes: determining an input-output relationship for the object based on a nonlinear static model representing an input-output relationship between contact force and movement of the object on the worktable; expressing the interaction between the object and the manipulator using complementarity constraints to capture contact states between the object and the manipulator; formulating a representation of the frictional stability of the object based on the nonlinear static model of the external contact with the external contact of the worktable; formulating a bilevel optimization problem to maximize the frictional stability over a position trajectory of the object being manipulated on the worktable; using said friction stability to estimate uncertainty values ​​contained in the physical parameters to be compensated; solving the two-level optimization problem using the nonlinear optimization solver to generate control data for a set of the contact forces being applied to the object by using the manipulator; transmitting the control data instructing the manipulator to reorient the object on the worktable in accordance with the series of contact forces; The operation controller further generates a positional relationship for estimating the frictional stability taking into account a contact configuration of the object with the worktable.

8. 1. A manipulation controller for reorienting an object by a manipulator of a robotic system utilizing external contact of the object with an additional external contact surface of a worktable during manipulation, the manipulation controller comprising: an interface controller configured to acquire measurement data from sensors disposed on the robotic system; at least one processor; 1. A computer-implemented method comprising: a nonlinear programming module; a nonlinear optimization solver; and an optimization module; and a memory configured to store a nonlinear static model representing an input-output relationship between contact forces and movements for a given object, the method utilizing at least one processor coupled to the memory storing instructions for performing the method, the instructions, when executed by the at least one processor, performing steps of the method, the steps including: The method includes a step of acquiring measurement data from a vision sensor and a force sensor arranged on the robot system, the vision sensor being arranged at an arbitrary position from which a viewpoint for observing / measuring an object state representing a positional relationship between the object and the additional external contact surface can be obtained, and the force sensor being configured to detect a force applied by the manipulator to the object at a contact point between the object and the manipulator, and the step further includes: determining an input-output relationship for the object based on a nonlinear static model representing an input-output relationship between contact force and movement of the object on the worktable; expressing the interaction between the object and the manipulator using complementarity constraints to capture contact states between the object and the manipulator; formulating a representation of the frictional stability of the object based on the nonlinear static model of the external contact with the external contact of the worktable; formulating a bilevel optimization problem to maximize the frictional stability over a position trajectory of the object being manipulated on the worktable; using said friction stability to estimate uncertainty values ​​contained in the physical parameters to be compensated; solving the two-level optimization problem using the nonlinear optimization solver to generate control data for a set of the contact forces being applied to the object by using the manipulator; transmitting the control data instructing the manipulator to reorient the object on the worktable in accordance with the series of contact forces; The manipulation controller further transforms the two-level optimization problem into a single-level large-scale nonlinear programming problem using KKT conditions and an epigraph trick.

9. 9. A controller according to claim 7 or 8, wherein the worktable components are arranged to contact at least two external contacts of the object.

10. The controller according to claim 7 or 8, further comprising: generating a positional relationship for estimating the friction stability by taking into account a contact configuration of the object with the worktable.

11. The sensor includes a force sensor (F / T sensor), and / or The controller of claim 7 or 8, wherein the uncertainty value comprises an erroneous coefficient of friction.

12. The controller according to claim 7 or 8, wherein the manipulator is a 6DoF (Six Degree of Freedom) manipulator.

13. The manipulator includes a servo motor and a gripper, and / or The controller of claim 7 or 8, wherein the step of solving the bilevel optimization problem calculates a friction stability margin of the object.

14. The controller of claim 7 or 8, wherein the two-level optimization problem is composed of two sub-optimization problems.

Images