SYSTEM AND METHOD FOR CONTROLLING THE OPERATION OF AN OPERATING SYSTEM - Patent application

Abstract

The present disclosure provides a system and method for controlling the operation of a manipulation system, the method including: formulating an optimization problem based on a sample-average approximation method and a Stochastic Discrete-time Linear Complementarity Model (SDLCM) for a manipulation task; solving the formulated optimization problem using a critical particle algorithm to calculate an optimal state trajectory, an optimal feedforward control trajectory, an optimal complementary variable trajectory, a state feedback gain, and a complementary feedback gain; collecting measurements indicative of a current state trajectory and a current complementary variable trajectory; determining online control inputs based on the optimal feedforward control trajectory, the deviation of the current state trajectory from the optimal state trajectory, the deviation of the current complementary variable trajectory from the optimal complementary variable trajectory, the state feedback gain, and the complementary feedback gain; and controlling actuators of the manipulation system according to the online control inputs.

Description

[Technical Field]

[0001] The present disclosure relates generally to robotic manipulation, and more particularly to systems and methods for controlling the operation of a manipulation system. [Background technology]

[0002] In a robotic system, a motion planning technique is used to determine a trajectory for a robot to perform a manipulation task of moving an object in an environment to a target pose. The robotic system includes, for example, a robot arm for manipulating the object and moving it to a target pose. Manipulating the object leads to the formation of multiple contacts, such as contact between the robot arm and the object and contact between the object and the environment. Efficient use of these contacts can provide additional dexterity to the robotic system while performing complex manipulation tasks. To achieve this, it is desirable to incorporate constraints imposed by these contacts into the motion planning and control of the robotic system.

[0003] However, uncertainties exist in robotic systems, such as uncertainty in contact forces and uncertainty in friction coefficients at these contact points. Motion planning and control of such uncertain robotic systems become challenging because propagation of uncertainties is difficult for uncertain robotic systems. Therefore, there is a need for a system and method for controlling a robotic system to perform manipulation tasks under uncertainties. Summary of the Invention

[0004] It is an object of some embodiments to provide a system and method for controlling a manipulation system to perform a manipulation task in an environment. The manipulation system includes a robotic arm for manipulating an object in the environment. The manipulation task corresponds to manipulating the object, such as pushing the object or changing the object's orientation. Manipulating the object may result in the formation of multiple contacts, e.g., contact between the object and the environment, and contact between the robotic arm and the object. Some embodiments recognize that such contact-rich manipulation systems can be efficiently modeled using complementarity constraints. To this end, embodiments model the manipulation task using a Stochastic Discrete-time Linear Complementarity Model (SDLCM) that includes complementarity constraints.

[0005] However, there exists uncertainty in the manipulation system, for example, uncertainty in the contact force and the friction coefficient. The uncertainty in the manipulation system leads to a stochastic complementarity system. Some embodiments are based on the recognition that designing a controller for an SDLCM with a stochastic complementarity constraint is difficult, and the states of the manipulation system and the complementarity variables are implicitly related via the complementarity constraint (i.e., uncertainty in one leads to the stochastic evolution of the other), which makes uncertainty propagation difficult.

[0006] To that end, an objective of some embodiments is to formulate an optimization problem for covariance control of a stochastic complementarity-constrained SDLCM. In some embodiments, for a manipulation task, it is desirable to constrain states within a specific set with a certain probability. Such constraints can be formulated as chance constraints. The chance constraints must be satisfied during control of the manipulation system. Therefore, an objective of some embodiments is to formulate a chance-constrained optimization problem that satisfies the chance constraints for a stochastic complementarity-constrained SDLCM.

[0007] To achieve this goal, we first formulate a stochastic constrained optimization problem determined by stochastic constraints. By sampling the uncertainty, we use SAA (Sample Average Approximation) to approximately solve the stochastic constrained optimization problem. In particular, by sampling from the uncertainty distribution, we can obtain N realizations (also called particles) of the uncertainty. That is, we approximate the uncertainty distribution using a finite-dimensional distribution. The uncertainty distribution follows a uniform variance on the uncertainty samples. We use SDLCM to propagate each particle and predict the mean and covariance of the state.

[0008] Furthermore, based on N particles and the mean and covariance of the predicted states, we reformulate the stochastic constrained optimization to create a particle-based optimization problem for covariance control of SDLCM. There is coupling between the state and the complementary variables, since uncertainty in the state leads to uncertainty in the complementary variables, and uncertainty in the complementary variables leads to uncertainty in the state. Previous approaches ignore coupling during uncertainty propagation for the manipulated system. Therefore, particles are used for uncertainty propagation using SAA.

[0009] Additionally, some embodiments recognize that the control input of a manipulation system includes a feedforward term and a feedback term. The feedforward term controls the mean state of the manipulation system, and the feedback term controls the covariance of the manipulation system. The feedback term is a function of the difference between the current state of the manipulation system and the desired / optimal state, i.e., the deviation of the state from the desired state. However, the stochastic complementarity-constrained SDLCM includes complementary variables, and controlling both the state and the complementary variables is important for a manipulation system with many contacts. Therefore, the feedback term is formulated as a function of the deviation of the state from the desired state and the deviation of the complementary variable from the desired complementary variable. To this end, the control input is a function of the feedforward term, the deviation of the state from the desired state, and the deviation of the complementary variable from the desired complementary variable. Furthermore, the control input is based on a state feedback gain, which controls the deviation of the state from the desired state, and a feedback gain of the complementary variable (also referred to as a complementary feedback gain), which controls the deviation of the complementary variable from the desired complementary variable.

[0010] The control input in a particle-based optimization problem is replaced with a control input that is a function of the feedforward term, state feedback gain, complementary feedback gain, state deviation, and complementary variable deviation. Solving this particle-based optimization problem determines the optimal state trajectory, optimal feedforward control trajectory, optimal complementary variable trajectory, state feedback gain, and complementary feedback gain. However, solving this particle-based optimization problem is computationally expensive because the particle-based optimization problem must be evaluated for each particle. Furthermore, solving this particle-based optimization problem is difficult because each optimization problem is accompanied by a complementarity constraint. Therefore, all optimization problems become mathematical programming problems with complementarity constraints (MPCC), which are computationally difficult to solve.

[0011] Some embodiments of the present disclosure provide an important-particle algorithm for solving particle-based optimization problems in a computationally cost-effective manner. The important-particle algorithm samples important particles that are likely to provide the most information about stochastic constraint violations. For example, the important-particle algorithm starts with a relatively small number of particles and solves the particle-based optimization problem to determine a controller for an SDLCM. Furthermore, using Monte Carlo simulation, a controller is tested for each of multiple test particles sampled from an uncertainty distribution to identify particles that violate stochastic constraints. A fixed number of worst-particles that violate the stochastic constraints are then added to the N particles used to solve the particle-based optimization problem. The particle-based optimization problem is then solved again using the N particles with the added worst-particles. This process is repeated until a termination condition is met. In this way, the number of particles used to solve the particle-based optimization problem is reduced, thereby reducing the computational load.

[0012] During online control, measurements indicating a current state trajectory and a current complementary variable trajectory are collected from one or more sensors (e.g., tactile or force sensors) associated with the manipulation system. A deviation of the current state trajectory from an optimal state trajectory and a deviation of the current complementary variable trajectory from an optimal complementary variable trajectory are calculated. Online control inputs are determined as a combination of an optimal feedforward control trajectory and a feedback control for controlling the state covariance and complementary variables of the manipulation system. The feedback control is based on the deviation of the current state trajectory from the optimal state trajectory, the deviation of the current complementary variable trajectory from the optimal complementary variable trajectory, the state feedback gain, and the complementary feedback gain. The determined online control inputs are sent to actuators of the manipulation system to control the operation of the manipulation system.

[0013] Accordingly, one embodiment discloses a control system for controlling operation of a manipulation system, the manipulation system comprising a robotic arm for performing a manipulation task to manipulate an object, the control system comprising at least one processor and a memory storing instructions, the instructions causing the at least one processor to formulate a particle-based optimization problem for covariance control of the manipulation system based on a sample mean approximation method and a Stochastic Discrete-time Linear Complementarity Model (SDLCM) of the manipulation task, solve the formulated particle-based optimization problem using a key particle algorithm to calculate an optimal state trajectory, an optimal feedforward control trajectory, an optimal complementary variable trajectory, a state feedback gain, and a complementary feedback gain, collect measurements indicative of a current state trajectory and a current complementary variable trajectory from one or more sensors associated with the manipulation system, calculate deviations of the current state trajectory from the optimal state trajectory and deviations of the current complementary variable trajectory from the optimal complementary variable trajectory, and calculate the optimal feedforward control trajectory. determining online control inputs based on the control trajectory and feedback control for controlling the state covariances and complementary variables of the manipulation system, the feedback control being based on deviations of the current state trajectory from an optimal state trajectory, deviations of the current complementary variable trajectories from the optimal complementary variable trajectories, state feedback gains, and complementary feedback gains; and further causing the at least one processor to generate control commands for actuators of the robot arm based on the determined online control inputs and operate the actuators of the robot arm in accordance with the determined online control inputs by sending control signals of the control commands to the robot arm.

[0014] Accordingly, another embodiment discloses a method for controlling operation of a manipulation system, the manipulation system comprising a robotic arm for performing a manipulation task to manipulate an object, the method including: formulating a particle-based optimization problem for covariance control of the manipulation system based on a sample mean approximation method and a Stochastic Discrete-time Linear Complementarity Model (SDLCM) of the manipulation task; solving the formulated particle-based optimization problem using a key particle algorithm to calculate an optimal state trajectory, an optimal feedforward control trajectory, an optimal complementary variable trajectory, a state feedback gain, and a complementary feedback gain; collecting measurements indicative of a current state trajectory and a current complementary variable trajectory from one or more sensors associated with the manipulation system; calculating deviations of the current state trajectory from the optimal state trajectory and deviations of the current complementary variable trajectory from the optimal complementary variable trajectory; determining online control inputs based on the forward control trajectory and feedback control for controlling the state covariances and complementary variables of the manipulation system, the feedback control being based on a deviation of the current state trajectory from an optimal state trajectory, a deviation of the current complementary variable trajectory from an optimal complementary variable trajectory, a state feedback gain, and a complementary feedback gain, the method further including generating control commands for actuators of the robot arm based on the determined online control inputs, and operating the actuators of the robot arm in accordance with the determined online control inputs by sending control signals of the control commands to the robot arm.

[0015] Accordingly, yet another embodiment discloses a non-transitory computer-readable storage medium embodied with a program executable by a processor to execute a method for controlling operation of a manipulation system, the manipulation system comprising a robotic arm for performing a manipulation task to manipulate an object, the method including: formulating a particle-based optimization problem for covariance control of the manipulation system based on a sample mean approximation method and a Stochastic Discrete-time Linear Complementarity Model (SDLCM) of the manipulation task; solving the formulated particle-based optimization problem using a key particle algorithm to calculate an optimal state trajectory, an optimal feedforward control trajectory, an optimal complementary variable trajectory, a state feedback gain, and a complementary feedback gain; collecting measurements indicative of a current state trajectory and a current complementary variable trajectory from one or more sensors associated with the manipulation system; calculating deviations of the current state trajectory from the optimal state trajectory and deviations of the current complementary variable trajectory from the optimal complementary variable trajectory; determining online control inputs based on the forward control trajectory and feedback control for controlling the state covariances and complementary variables of the manipulation system, the feedback control being based on a deviation of the current state trajectory from an optimal state trajectory, a deviation of the current complementary variable trajectory from an optimal complementary variable trajectory, a state feedback gain, and a complementary feedback gain, the method further including generating control commands for actuators of the robot arm based on the determined online control inputs, and operating the actuators of the robot arm in accordance with the determined online control inputs by sending control signals of the control commands to the robot arm.

[0016] Accordingly, yet another embodiment discloses a control system for controlling operation of a manipulation system, the manipulation system comprising a robotic arm for performing a manipulation task to manipulate an object. The control system comprises at least one processor and a memory storing instructions, the instructions causing the at least one processor to collect measurements indicative of a current state trajectory and a current complementary variable trajectory from one or more sensors associated with the manipulation system, and calculate a deviation of the current state trajectory from an optimal state trajectory and a deviation of the current complementary variable trajectory from an optimal complementary variable trajectory. A particle-based optimization problem is solved using a key particle algorithm to obtain an optimal state trajectory, an optimal complementary variable trajectory, an optimal feedforward control trajectory, a state feedback gain, and a complementary feedback gain, the particle-based optimization problem being based on a sample mean approximation method and a Stochastic Discrete-time Linear Complementarity Model (SDLCM) for the manipulation task. The at least one processor further determines online control inputs based on the optimal feedforward control trajectory and feedback control for controlling the state covariances and complementary variables of the manipulation system, the feedback control being based on the deviation of the current state trajectory from the optimal state trajectory, the deviation of the current complementary variable trajectory from the optimal complementary variable trajectory, the state feedback gain, and the complementary feedback gain, and the at least one processor further generates control commands for actuators of the robot arm based on the determined online control inputs and operates the actuators of the robot arm in accordance with the determined online control inputs by sending control signals of the control commands to the robot arm. [Brief explanation of the drawings]

[0017] [Figure 1A] FIG. 1 illustrates a control system according to some embodiments of the present disclosure. [Figure 1B] FIG. 1 illustrates a block diagram of a control system for controlling the operation of an operating system, according to some embodiments of the present disclosure. [Figure 1C]FIG. 1 illustrates a block diagram of a function performed by a control system for determining online control inputs for controlling the operation of an operating system, according to an embodiment of the present disclosure. [Figure 1D] FIG. 1 illustrates a block diagram of a function performed by a control system for controlling the operation of an operating system based on determined online control inputs, according to an embodiment of the present disclosure. [Figure 2A] FIG. 1 illustrates a block diagram of functions performed during the offline phase, according to an embodiment of the present disclosure. [Figure 2B] FIG. 10 shows a block diagram of functions performed during the online phase according to another embodiment of the present disclosure. [Figure 3] FIG. 1 illustrates a significant particle algorithm according to an embodiment of the present disclosure. [Figure 4] FIG. 10 illustrates an operation task of pushing an object to a target position according to an embodiment of the present disclosure. [Figure 5A] 10A-10C illustrate bolt orientation changes according to some embodiments of the present disclosure. [Figure 5B] 10A-10C illustrate bottle reorientation according to some embodiments of the present disclosure. [Figure 5C] FIG. 1 illustrates an object placement task, according to some embodiments of the present disclosure. [Figure 5D] FIG. 1 illustrates an assembly task, according to some embodiments of the present disclosure. [Figure 6] FIG. 1 is a schematic diagram illustrating a computing device for implementing the control system of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0018] The presently disclosed embodiments will be further described with reference to the accompanying drawings, in which the drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the presently disclosed embodiments.

[0019] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details. In other instances, devices and methods are shown only in block diagram form in order to avoid obscuring the present disclosure.

[0020] As used in this specification and the appended claims, the terms "for example," "for instance," and "such as," and the verbs "comprising," "having," and "including" and other verb forms thereof, when used in conjunction with a list of one or more components or other items, should each be construed as open-ended, meaning that the list should not be considered to exclude other, additional components or other items. The term "based on" means based, at least in part, on. Furthermore, it should be understood that the phraseology and terminology used herein are for descriptive purposes and should not be regarded as limiting. The headings used herein are for convenience only and do not have any legal or limiting effect.

[0021] FIG. 1A illustrates a manipulation system 100 according to some embodiments of the present disclosure. The manipulation system 100 includes a robotic arm 101 configured to perform a manipulation task of manipulating an object 103. The manipulation task of manipulating the object 103 may correspond to pushing the object 103 or changing the orientation of the object 103. Manipulating the object 103 results in the formation of multiple contacts, e.g., contact 105 between the robotic arm 101 and the object 103, and contact 107 between the object 103 and the environment 109. An objective of some embodiments is to design a control system for controlling the operation of the manipulation system 100 with such a high number of contacts. Such a control system is described below in FIG. 1B.

[0022] FIG. 1B illustrates a block diagram of a control system 111 for controlling the operation of the operating system 100, according to some embodiments of the present disclosure. The control system 111 includes a processor 113 and a memory 115. The processor 113 may be a single-core processor, a multi-core processor, a computer cluster, or any number of other configurations. The memory 115 may include random access memory (RAM), read-only memory (ROM), flash memory, or other suitable memory systems. Additionally, in some embodiments, the memory 115 may be implemented using a hard drive, an optical drive, a thumb drive, an array of drives, or any combination thereof. The control system 111 is communicatively coupled to the operating system 100. The control system 111 performs functions for controlling the operation of the operating system 100, as described below with reference to FIGS. 1C and 1D.

[0023] 1C illustrates a block diagram 117 of a function executed by the control system 111 for determining online control inputs for controlling the operation of the manipulation system 100, according to an embodiment of the present disclosure. In block 119, the processor 113 formulates a particle-based optimization problem for covariance control of the manipulation system 100 based on a sample average approximation (SAA) and a stochastic discrete-time linear complementarity model (SDLCM) of the manipulation task. Some embodiments recognize that a manipulation system 100 with many contacts can be efficiently modeled using complementarity constraints. To this end, embodiments model the manipulation task using an SDLCM including complementarity constraints. According to an embodiment, the SDLCM may be given by:

number

[0024] Due to uncertainties in the manipulation system 100, for example, uncertainties in contact forces and friction coefficients, the uncertainty in the manipulation system 100 leads to a stochastic complementarity system. Some embodiments are based on the recognition that designing a controller for an SDLCM with a stochastic complementarity constraint is difficult, and the states of the manipulation system 100 and the complementarity variables are implicitly related via the complementarity constraint (i.e., uncertainty in one leads to the stochastic evolution of the other), which makes uncertainty propagation challenging.

[0025] To that end, an objective of some embodiments is to formulate an optimization problem for covariance control of a stochastic complementarity-constrained SDLCM. In some embodiments, for a manipulation task, it is desirable to constrain states within a particular set with a certain probability. Such constraints may be formulated as chance constraints. The chance constraints must be satisfied during control of the manipulation system 100. Therefore, an objective of some embodiments is to formulate a chance-constrained optimization problem that satisfies the chance constraints for a stochastic complementarity-constrained SDLCM.

[0026] To achieve this goal, we first formulate a stochastic constrained optimization that is determined by stochastic constraints as given below.

number

[0027] The processor 113 uses SAA to approximately solve the stochastic constrained optimization by sampling the uncertainty. Specifically, N realizations (also referred to as particles) of the uncertainty are obtained by sampling from the distribution of the uncertainty. That is, a finite-dimensional distribution is used to approximate the uncertainty distribution. The finite-dimensional distribution follows uniform variance on the uncertainty samples. Each particle is propagated using SDLCM to predict the mean and covariance of the state. Furthermore, based on the N particles and the predicted mean and covariance of the state, the stochastic constrained optimization is reformulated to create a particle-based optimization problem for covariance control of the SDLCM.

[0028]

number

[0029]

number

[0030] Thus, we formulate the particle-based optimization problem (9)–(14) based on SDLCM (1)–(2) and SAA (6)–(8).

[0031]

number

[0032]

number

[0033]

number

[0034] However, solving the above particle-based optimization problems (17)-(20) is computationally expensive because the particle-based optimization problem must be evaluated for each particle. Furthermore, solving such particle-based optimization problems is difficult because each optimization problem involves complementarity constraints. Therefore, all optimization problems become mathematical programming problems with complementarity constraints (MPCC), which are computationally difficult to solve.

[0035] Some embodiments of the present disclosure provide an Important-Particle Algorithm for solving particle-based optimization problems in a computationally cost-effective manner. The Important-Particle Algorithm is described in detail in FIG. 3.

[0036]

number

[0037] Further, in block 125, the processor 113 collects measurements indicative of a current state trajectory and a current complementary variable trajectory from one or more sensors associated with the manipulation system 100. For example, the one or more sensors associated with the manipulation system 100 correspond to tactile sensors, force sensors, or torque sensors. The one or more sensors may be mounted on the robot arm 101. For example, a tactile sensor may be co-located with the fingers of a gripper of the robot arm 101. The processor 113 collects measurements from the one or more sensors. The collected measurements indicate a current state trajectory and a current complementary variable trajectory. The current state trajectory includes a current pose of the object 103, and the current complementary variable trajectory includes a current contact force between the robot arm 101 and the object 103.

[0038] In block 127, processor 113 calculates the deviation of the current state trajectory from the optimal state trajectory and the deviation of the current complementary variable trajectory from the optimal complementary variable trajectory.

[0039]

number

[0040] 1D shows a block diagram of functions executed by the control system 111 for controlling the operation of the manipulation system 100 based on the determined online control inputs, according to an embodiment of the present disclosure. In block 131, the processor 113 generates control commands for the actuators of the robot arm 101 based on the determined online control inputs. In block 133, the processor 113 operates the actuators of the robot arm 101 according to the determined online control inputs by sending control signals for the control commands to the robot arm 101, thereby controlling the operation of the manipulation system 100, e.g., a manipulation task, even under uncertainty.

[0041] Instead, in some embodiments, the particle-based optimization problems (17)-(20) are formulated and solved offline, i.e., in advance, and the online control inputs are determined online, i.e., during real-time operation. Such embodiments are described below with reference to Figures 2A and 2B.

[0042] FIG. 2A shows a block diagram 200 of functions performed during the offline phase according to an embodiment of the present disclosure.

[0043]

number

[0044]

number

[0045] Furthermore, the important particle algorithm used to solve the particle-based optimization problems (17)-(20) is described below with reference to FIG.

[0046]

number

[0047]

number

[0048]

number

[0049] Some embodiments are based on the further understanding that nonlinear programming (NLP) can also be used to solve the particle-based optimization problems (17)-(20). To solve the particle-based optimization problems (17)-(20) using NLP, the integer constraints (14) must be solved in an NLP manner. To solve the integer constraints (14) in an NLP manner, the following two-level optimization problem is formulated:

number

[0050]

number

[0051]

number

[0052]

number

[0053] Some embodiments are based on the understanding that the control system 111 can be used to control a manipulation task of pushing an object to a target position even under uncertainty. FIG. 4 illustrates a manipulation task of pushing an object 403 to a target position 405, according to an embodiment of the present disclosure. A robotic arm 401 is communicatively coupled to the control system 111. The manipulation task of pushing the object 403 to the target position 405 results in the formation of multiple contacts, including a contact 407 between the robotic arm 401 and the object 403 and a contact 409 between the object 403 and the environment 411. There are uncertainties in the contact forces and friction coefficients at the contact points between the contacts 407 and 409. As described above in FIG. 1C , the control system 111 determines online control inputs. Furthermore, the control system 111 controls the robotic arm 401 based on the determined online control inputs to cause the robotic arm 401 to push the object 403 to the target position 405 under uncertainty.

[0054] Additionally, the control system 111 can be used to change the orientation of objects of different sizes and shapes.

[0055] 5A is a diagram illustrating changing the orientation of a bolt 501, according to some embodiments of the present disclosure. A control system 111 (not shown) is communicatively coupled to a robotic arm 503. As can be seen in FIG. 5A , the control system 111 controls the robotic arm 503 to move the bolt 501 to a target pose 505 without losing contact 507 between the tool 509 held by the robotic arm 503 and the bolt 501 and contact 511 between the bolt 501 and the environment 513, and with uncertainty in the contact force at the contact point between contact 507 and contact 511.

[0056] 5B is a diagram illustrating a change in orientation of bottle 515, according to some embodiments of the present disclosure. As can be seen in FIG. 5B, control system 111 (not shown) controls robot arm 503 to move bottle 515 to target pose 517 without losing contact 519 between tool 521 held by robot arm 503 and bottle 515 and contact 523 between bottle 515 and environment 527, and with uncertainty in the contact force at the contact point between contact 519 and contact 523.

[0057] The control system 111 can be used to control the robotic arm 503 to perform object placement tasks, as the control system 111 can control the robotic arm 503 to manipulate multiple objects of different sizes and shapes, such as the bolt 501 and the bottle 515, without losing contact and with uncertainty during manipulation.

[0058] 5C illustrates an object placement task, according to some embodiments of the present disclosure. Objects such as bolt 529, bottle 531, and box 533 are being placed on table 535. Control system 111 controls robotic arm 503 to move each object to its respective target pose and place it on table 535. For example, bolt 529, bottle 531, and box 533 are transferred to target poses 537, 539, and 541, respectively. To do so, control system 111 places each object according to its respective target pose, regardless of the object's initial pose.

[0059] Additionally, the control system 111 can be used to manipulate the objects to assemble them using one or a combination of pushing, orienting, and grasping to produce a product, such an embodiment being described below in Figure 5D.

[0060] 5D illustrates an assembly task, according to some embodiments of the present disclosure. An object, such as a peg 543, is placed on a tabletop 545, where some coefficient of friction is known. The assembly task involves manipulating peg 543 from initial configuration 547 to configuration 549 and then to configuration 551. Control system 111 controls robotic arm 553 and gripper 555 such that gripper 555 holds and reorients peg 543 to configuration 549 and then to configuration 551, thereby performing the assembly task.

[0061] 6 is a schematic diagram illustrating a computing device 600 for implementing the control system 111 and method of the present disclosure. The computing device 600 includes a power supply 601, a processor 603, a memory 605, and a storage device 607, all connected to a bus 609. Additionally, a high-speed interface 611, a low-speed interface 613, a high-speed expansion port 615, and a low-speed connection port 617 may be connected to the bus 609. Additionally, a low-speed expansion port 619 is connected to the bus 609. Additionally, an input interface 621 may be connected to an external receiver 623 and an output interface 625 via the bus 609. A receiver 627 may be connected to an external transmitter 629 and a transmitter 631 via the bus 609. An external memory 633, an external sensor 635, a machine(s) 637, and an environment 639 may also be connected to the bus 609. For example, machine(s) 637 may be manipulation system 100 including a robotic arm 101 for performing manipulation tasks such as pushing an object (such as object 103), orienting an object, grasping an object, or manipulating an object to assemble it using a combination of pushing, orienting, and grasping to manufacture a product. Additionally, one or more external input / output devices 641 may be connected to bus 609. A NIC (Network Interface Controller) 643 may be adapted to connect to a network 645 through bus 609, where image data or other data may be rendered, among other things, on a third-party display device, a third-party imaging device, and / or a third-party printing device external to computing device 600.

[0062] The memory 605 can store instructions executable by the computing device 600 and any data that can be utilized by the methods and systems of the present disclosure. The memory 605 can include a random access memory (RAM), a read-only memory (ROM), flash memory, or other suitable memory system. The memory 605 can be one or more volatile storage devices and / or one or more non-volatile storage devices. The memory 605 can also be another form of computer-readable medium, such as a magnetic disk or an optical disk.

[0063] The storage device 607 may be adapted to store auxiliary data and / or software modules utilized by the computing device 600. The storage device 607 may include a hard drive, an optical drive, a thumb drive, an array of drives, or any combination thereof. Additionally, the storage device 607 may include a computer-readable medium such as a floppy disk drive, a hard disk drive, an optical disk drive, or an array of devices, including a tape drive, a flash memory or other similar solid-state memory device, or a device included in a storage area network or other configuration. The information carrier may store instructions that, when executed by one or more processing units (e.g., processor 603), perform one or more methods, such as those described above.

[0064]

number

[0065] Computing device 600 can optionally be linked through bus 609 to a display interface or user interface (HMI) 647 adapted to connect computing device 600 to a display device 649 and keyboard 651, where display device 649 may include, among other devices, a computer monitor, a camera, a television, a projector, or a mobile device. In some implementations, computing device 600 can include a printer interface for connecting to a printing device, where the printing device may include, among other devices, a liquid inkjet printer, a solid ink printer, a large-scale commercial printer, a thermal printer, a UV printer, or a dye-sublimation printer.

[0066] The high-speed interface 611 manages operations requiring more bandwidth for the computing device 600, while the low-speed interface 613 manages operations requiring less bandwidth. This allocation of functionality is merely exemplary. In some implementations, the high-speed interface 611 can be coupled to memory 605, to a user interface (HMI) 647, to a keyboard 651 and a display 649 (e.g., through a graphics processor or accelerator), and to a high-speed expansion port 615. The high-speed expansion port 615 can accept various expansion cards via a bus 609. In one implementation, the low-speed interface 613 is coupled to storage 607 and a low-speed expansion port 617 via the bus 609. The low-speed expansion port 617, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input / output devices 641. The computing device 600 may be connected to a server 653 and a rack server 655. The computing device 600 may be implemented in several different forms. For example, the computing device 600 may be implemented as part of a rack server 655 .

[0067] The above description provides only exemplary embodiments and does not limit the scope, applicability, or configuration of the present disclosure. Rather, the description of the exemplary embodiments above provides one of ordinary skill in the art with an enabling description for implementing one or more exemplary embodiments. Various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the subject matter as set forth in the appended claims.

[0068] Specific details have been set forth in the above description to provide a thorough understanding of the embodiments. However, those skilled in the art will understand that the embodiments may be practiced without these specific details. For example, systems, processes, and other elements of the disclosed subject matter may be illustrated in block diagrams as components to avoid obscuring the embodiments in unnecessary detail. In other instances, well-known processes, structures, and techniques may be illustrated without unnecessary detail to avoid obscuring the embodiments. Furthermore, like reference numbers and symbols in the various drawings indicate like elements.

[0069] Individual embodiments may also be described as a process and illustrated as a flowchart, flow diagram, data flow diagram, structure diagram, or block diagram. A flowchart may depict operations as a sequential process, but many of the operations may be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process may terminate when its operations are completed, but may include additional steps not included in the diagram. Furthermore, not all operations included in a specifically described process may be performed in all embodiments. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, the end of the function may correspond to the function returning to a calling function or a main function.

[0070] Furthermore, at least some of the embodiments of the disclosed subject matter may be implemented manually or automatically. The manual or automatic implementation may be performed or at least assisted by the use of machine, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored on a machine-readable medium. A processor(s) may perform the necessary tasks.

[0071] The various methods or steps outlined herein may be coded as software executable on one or more processors employing any one of a variety of operating systems or platforms. Additionally, such software may be written using any number of suitable programming languages ​​and / or programming or scripting tools, or may be compiled as executable machine language code or intermediate code that runs on a framework or virtual machine. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0072] Embodiments of the present disclosure may be implemented as methods, examples of which have been provided. The actions performed as part of the method may be ordered as appropriate. Thus, embodiments may be constructed in which actions are performed in an order different from the illustrated order, and may involve performing some actions simultaneously, even though the illustrated embodiment shows them as sequential actions.

[0073] Furthermore, embodiments of the present disclosure and functional operations may be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, or in computer hardware combinations, including the structures disclosed herein and their structural equivalents. Furthermore, some embodiments of the present disclosure may be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by or to control the operation of a data processing apparatus. Furthermore, the program instructions may be encoded on an artificially generated, propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a receiving device suitable for execution by a data processing apparatus. The computer storage medium may be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory device, or one or more combinations thereof.

[0074] According to embodiments of the present disclosure, the term "data processing device" may encompass any type of device, apparatus, or machine for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. The device may include dedicated logic circuitry, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). In addition to hardware, the device may also include code that creates an execution environment for the computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0075] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script, or code) may be written in any type of programming language, such as a compiled or interpreted language, or a declarative or procedural language, and may be deployed in any form, such as a stand-alone program or as a module, component, subroutine, or other structural unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in part of a file holding other programs or data, for example, one or more scripts stored in a document written in a markup language, in a file dedicated to the program, or in several associated files, for example, files storing one or more modules, subprograms, or portions of code.

[0076] A computer program can be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communications network. Computers suitable for running computer programs can include, by way of example, general-purpose or application-specific microprocessors or both, or other types of central processing units (CPUs). Typically, a CPU receives instructions and data from read-only memory (ROM), random-access memory (RAM), or both. The essential components of a computer are a CPU for performing or executing instructions and one or more memory elements for storing instructions and data.

[0077] Typically, a CPU receives instructions and data from Read Only Memory (ROM), Random Access Memory (RAM), or both. The essential components of a computer are a CPU for performing or executing instructions and one or more memory elements for storing instructions and data. Typically, a computer includes one or more mass storage devices, e.g., magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operably connected to receive, transmit, or both data from one or more such mass storage devices. However, a computer need not have such devices. A computer can also be incorporated into another device, e.g., a mobile phone, a personal digital assistant (PDA), a portable audio or video player, a game controller, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a Universal Serial Bus (USB) flash drive), to name a few.

[0078] To enable user interaction, embodiments of the inventive subject matter described herein may be implemented on a computer having a display device, such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor, for displaying information to a user, and a keyboard and pointing device, such as a mouse or trackball, for allowing a user to provide input to the computer. Other types of devices may also be used to provide user interaction. For example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic, speech, or tactile input. Additionally, the computer may interact with a user by sending and receiving documents from devices used by the user, for example, by sending web pages to a web browser on the user's client device in response to a request received from the web browser.

[0079] Embodiments of the inventive subject matter described herein may be implemented in a computer system having back-end components, such as a data server; a computer system having middleware components, such as an application server; a computer system having front-end components, such as a client computer having a graphical user interface or web browser through which a user can interact with embodiments of the inventive subject matter described herein; or a computer system having any combination of one or more back-end, middleware, and front-end components. The components of the system may be interconnected by any form or medium of digital data communication, such as a communications network. Communications networks include local area networks ("LANs") and wide area networks ("WANs"), e.g., the Internet.

[0080] A computer system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server exists by virtue of computer programs running on the respective computers and having a client-server relationship.

[0081] Although the present disclosure has been described by way of examples of certain preferred embodiments, it will be understood that various other adaptations and modifications may be made without departing from the spirit and scope of the present disclosure, and it is, therefore, the feature of the appended claims to cover all such variations and modifications that come within the spirit and scope of the present disclosure.

Claims

1. 1. A control system for controlling operation of a manipulation system, the manipulation system comprising a robotic arm for performing a manipulation task of manipulating an object, the control system comprising at least one processor and a memory storing instructions, the instructions causing the at least one processor to: Formulating a particle-based optimization problem for covariance control of the manipulation system based on a sample mean approximation method and a stochastic discrete-time linear complementarity model (SDLCM) of the manipulation task; using a critical particle algorithm to solve the formulated particle-based optimization problem to calculate an optimal state trajectory, an optimal feedforward control trajectory, an optimal complementary variable trajectory, a state feedback gain, and a complementary feedback gain; collecting measurements from one or more sensors associated with the operational system indicative of a current state trajectory and a current complementary variable trajectory; calculating deviations of the current state trajectory from the optimal state trajectory and deviations of the current complementary variable trajectories from the optimal complementary variable trajectories; determining an online control input based on the optimal feedforward control trajectory and a feedback control for controlling a covariance of states and complementary variables of the manipulation system, the feedback control being based on the deviation of the current state trajectory from the optimal state trajectory, the deviation of the current complementary variable trajectories from the optimal complementary variable trajectory, the state feedback gain, and the complementary feedback gain; creating a control command for an actuator of the robot arm based on the determined online control input; a control system that transmits a control signal of the control command to the robot arm, thereby causing the actuator of the robot arm to operate in accordance with the determined online control input.

2. The control system of claim 1 , wherein the particle-based optimization problem corresponds to a stochastically constrained optimization problem governed by stochastic complementarity constraints and stochastic constraints.

3. The control system of claim 2 , wherein the probabilistic constraints are approximated using the sample mean approximation method.

4. The important particle algorithm is executed iteratively until a termination condition is met, and to execute one iteration, the processor further comprises: solving the particle-based optimization problem using a plurality of particles to determine a controller; performing a Monte Carlo simulation for each test particle of a plurality of test particles sampled from the uncertainty distribution based on the controller to identify particles that violate the probabilistic constraints; selecting a plurality of worst particles from the identified particles; The control system of claim 2 , configured to add the plurality of worst particles to the plurality of particles.

5. The control system of claim 1 , wherein the current state trajectory includes a current pose of the object and the current complementary variable trajectory includes a current contact force between the robot arm and the object.

6. The control system of claim 1 , wherein the state of the manipulation system corresponds to a pose of the object, and the complementary variable of the manipulation system corresponds to a contact force between the robot arm and the object.

7. 2. The control system of claim 1, wherein the manipulation task corresponds to pushing the object, orienting the object, or grasping the object, or manipulating the object to assemble the object using a combination of pushing, orienting, and grasping to manufacture a product.

8. The control system of claim 1 , wherein the one or more sensors associated with the manipulation system correspond to a tactile sensor, a force sensor, or a torque sensor.

9. 1. A method for controlling the operation of a manipulation system, the manipulation system comprising a robotic arm for performing a manipulation task to manipulate an object, the method comprising: Formulating a particle-based optimization problem for covariance control of the manipulation system based on a sample mean approximation method and a stochastic discrete-time linear complementarity model (SDLCM) for the manipulation task; solving the formulated particle-based optimization problem using a critical particle algorithm to calculate an optimal state trajectory, an optimal feedforward control trajectory, an optimal complementary variable trajectory, a state feedback gain, and a complementary feedback gain; collecting measurements indicative of a current state trajectory and a current complementary variable trajectory from one or more sensors associated with the operational system; calculating deviations of the current state trajectory from the optimal state trajectory and deviations of the current complementary variable trajectories from the optimal complementary variable trajectories; determining online control inputs based on the optimal feedforward control trajectory and feedback control for controlling state covariances and complementary variables of the manipulated system, the feedback control being based on the deviation of the current state trajectory from the optimal state trajectory, the deviation of the current complementary variable trajectories from the optimal complementary variable trajectory, the state feedback gain, and the complementary feedback gain; the method further comprising: generating a control command for an actuator of the robot arm based on the determined online control input; and operating the actuator of the robot arm in accordance with the determined online control input by transmitting a control signal of the control command to the robot arm.

10. The method of claim 9 , wherein the particle-based optimization problem corresponds to a stochastically constrained optimization problem governed by stochastic complementarity constraints and stochastic constraints.

11. The method of claim 10 , wherein the probabilistic constraints are approximated using the sample mean approximation method.

12. The important particle algorithm is executed iteratively until a termination condition is met, and the execution of one iteration is solving the particle-based optimization problem using a plurality of particles to determine a controller; performing a Monte Carlo simulation for each test particle of a plurality of test particles sampled from the uncertainty distribution based on the controller to identify particles that violate the probabilistic constraints; selecting a plurality of worst particles from the identified particles; and adding the plurality of worst particles to the plurality of particles.

13. The method of claim 9 , wherein the current state trajectory includes a current pose of the object and the current complementary variable trajectory includes a current contact force between the robot arm and the object.

14. The method of claim 9 , wherein the state of the manipulation system corresponds to a pose of the object and the complementary variable of the manipulation system corresponds to a contact force between the robot arm and the object.

15. The method of claim 9 , wherein the manipulation task corresponds to one of pushing the object and changing the orientation of the object.

16. The method of claim 9 , wherein the one or more sensors associated with the manipulation system correspond to a tactile sensor, a force sensor, or a torque sensor.

17. 1. A non-transitory computer-readable storage medium embodied with a program executable by a processor to perform a method for controlling the operation of a manipulation system, the manipulation system comprising a robotic arm for performing manipulation tasks that manipulate objects, the method comprising: Formulating a particle-based optimization problem for covariance control of the manipulation system based on a sample mean approximation method and a stochastic discrete-time linear complementarity model (SDLCM) for the manipulation task; solving the formulated particle-based optimization problem using a critical particle algorithm to calculate an optimal state trajectory, an optimal feedforward control trajectory, an optimal complementary variable trajectory, a state feedback gain, and a complementary feedback gain; collecting measurements indicative of a current state trajectory and a current complementary variable trajectory from one or more sensors associated with the operational system; calculating deviations of the current state trajectory from the optimal state trajectory and deviations of the current complementary variable trajectories from the optimal complementary variable trajectories; determining online control inputs based on the optimal feedforward control trajectory and feedback control for controlling state covariances and complementary variables of the manipulated system, the feedback control being based on the deviation of the current state trajectory from the optimal state trajectory, the deviation of the current complementary variable trajectories from the optimal complementary variable trajectory, the state feedback gain, and the complementary feedback gain; the method further comprising: generating a control command for an actuator of the robot arm based on the determined online control input; and operating the actuator of the robot arm in accordance with the determined online control input by transmitting a control signal of the control command to the robot arm.

18. 20. The non-transitory computer-readable storage medium of claim 17, wherein the particle-based optimization problem corresponds to a probabilistic constrained optimization problem governed by a probabilistic complementarity constraint and a probabilistic constraint.

19. The non-transitory computer-readable storage medium of claim 18 , wherein the probabilistic constraints are approximated using the sample mean approximation method.

20. 1. A control system for controlling operation of a manipulation system, the manipulation system comprising a robotic arm for performing a manipulation task of manipulating an object, the control system comprising at least one processor and a memory storing instructions, the instructions causing the at least one processor to: collecting measurements from one or more sensors associated with the operational system indicative of a current state trajectory and a current complementary variable trajectory; calculating deviations of the current state trajectory from an optimal state trajectory and deviations of the current complementary variable trajectories from an optimal complementary variable trajectory; a particle-based optimization problem is solved using a critical particle algorithm to obtain the optimal state trajectory, the optimal complementary variable trajectory, the optimal feedforward control trajectory, the state feedback gain, and the complementary feedback gain, wherein the particle-based optimization problem is based on a sample mean approximation method and a stochastic discrete-time linear complementary model (SDLCM) of the manipulation task; and the at least one processor further comprises: determining an online control input based on the optimal feedforward control trajectory and a feedback control for controlling a covariance of states and complementary variables of the manipulation system, the feedback control being based on the deviation of the current state trajectory from the optimal state trajectory, the deviation of the current complementary variable trajectories from the optimal complementary variable trajectory, the state feedback gain, and the complementary feedback gain; creating a control command for an actuator of the robot arm based on the determined online control input; a control system that transmits a control signal of the control command to the robot arm, thereby causing the actuator of the robot arm to operate in accordance with the determined online control input.

Images