Systems and methods for controlling the movement of a robotic arm

Abstract

The present disclosure discloses a system and method for controlling the movement of a robotic arm holding a tool for manipulating an object. The method comprises collecting measurements of a tactile sensor associated with the robotic arm, estimating a feedback signal indicative of a pose of the object based on the collected measurements and constraints imposed by a model predictive controller (MPC), and executing the MPC configured to generate control commands for actuators of the robotic arm based on the pose of the object by optimizing a cost function that minimizes deviation of the pose of the object from a target pose of the object. The optimization of the cost function is subject to constraints, where the constraints limit one or more forces acting on the object at one or more contact points to be within corresponding friction regions. The method further comprises controlling the actuators in accordance with the control commands.

Description

[Technical Field]

[0001] The present disclosure relates generally to robotic manipulation, and more particularly to systems and methods for controlling the movement of a robotic arm that holds a tool for manipulating an object. [Background technology]

[0002] In a robotic system, trajectory optimization is used to determine a control trajectory for a robot to perform a manipulation task of moving an object from a given initial pose to a target pose. The robotic system includes, for example, a robot arm holding a tool for manipulating the object and moving it to the target pose. The manipulation of the object with the tool causes multiple contacts, for example, contact between the robot arm and the tool, contact between the tool and the object, and contact between the object and the environment. By efficiently using the contacts, the robotic system becomes more dexterous while performing complex manipulation tasks.

[0003] However, most current-generation robotic systems avoid contact with the environment because contact introduces complex and discontinuous dynamics into their movement and interaction. To do so, the constraints imposed by contact must be incorporated into the planning and control of manipulation tasks. However, incorporating constraints into manipulation planning and control is difficult due to discontinuities. Therefore, classical control theory results for smooth and hybrid systems cannot be directly applied to manipulation. Therefore, a control system is needed to control a robotic system to perform a manipulation task while satisfying the constraints imposed by contact. Summary of the Invention

[0004] It is an objective of some embodiments to provide a system and method for controlling a manipulation system to perform manipulation of an object in an environment. The manipulation system includes a robotic arm holding a tool for manipulating the object. The manipulation of the object may correspond to a reorientation of the object, a pick-and-place operation, or an assembly operation. Manipulation of the object with the tool causes multiple contacts, for example, contact between the robotic arm and the tool, contact between the tool and the object, and contact between the object and the environment. These contacts must be maintained during the manipulation. Therefore, planning for the manipulation of the object needs to incorporate constraints imposed by the contacts to ensure that the contacts are maintained during the manipulation. Incorporating constraints in planning the manipulation is difficult.

[0005] An objective of some embodiments is to design a closed-loop controller that can reliably maintain contact during manipulation. It is also an objective of some embodiments to enable robust execution of planned manipulations via a closed-loop controller that uses tactile sensors. Some embodiments provide a control system for implementing such a closed-loop controller. The control system includes a processor and a memory. The processor may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. The memory may include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. The memory stores a tactile estimator and a model predictive controller (MPC). The tactile estimator and the MPC are executed by the processor.

[0006] The control system is communicatively coupled to the manipulation system. The tactile sensors are mounted on the robotic arm. For example, in one embodiment, the tactile sensors may both be positioned on the fingers of a gripper of the robotic arm. The control system collects tactile sensor measurements. The tactile sensor measurements are input to a tactile estimator. The tactile estimator is configured to estimate a feedback signal indicative of a state of the object. The state of the object may include a pose of the object. The estimated pose of the object is further input to an MPC. The MPC is configured to generate control commands for actuators of the robotic arm of the manipulation system based on the pose of the object by optimizing a cost function that minimizes deviation of the pose of the object from a target pose of the object while satisfying contact constraints required for the desired manipulation task.

[0007] The optimization of the cost function is subject to constraints that limit the forces acting on the objects at the contact points to satisfy the physics of frictional interactions. For example, in the case of stick contact, the friction forces must lie within the corresponding friction cone to ensure that contact is maintained during the operation. For example, constraints limit the forces acting at the contact between the tool and the object within the corresponding friction region, depending on the friction coefficient between the tool and the object, and limit the forces acting at the contact between the object and the environment within the corresponding friction region to ensure that such contact is maintained during the operation. The contact forces acting at the contact points depend on the type of contact formation and the physical properties of the objects. For example, in the case of stick contact, the friction forces must obey Coulomb's friction law. This law states that the friction forces lie within the friction cone for stick contact and exactly on the boundary for slip contact (i.e., when the objects slide against each other).

[0008] Furthermore, the control system controls the actuators of the robot arm according to the control commands, thus controlling the operation while maintaining contact.

[0009] Constraints that limit the forces acting on the object at the contact points to be within the corresponding friction regions are formulated using a contact model of the operation. Some embodiments are based on the understanding that the contact model of the operation can be obtained using the principles of quasi-static equilibrium, under the assumption that the object and the tool remain in contact during the operation. In other words, a contact model designed using the principles of quasi-static equilibrium assumes zero slip, i.e., no slip, at the contact points, such as between the tool and the object, and between the object and the environment.

[0010] Some embodiments are based on the recognition that a control system can be used to recover from deviations from a planned trajectory or in the event of unexpected contact. During online control, the control system collects measurements from tactile sensors and estimates the pose of the object using a tactile estimator. The control system further executes an MPC that determines control commands to recover from deviations from the planned trajectory based on the estimated pose of the object. Furthermore, the control system can be used to manipulate objects of different sizes and shapes, such as bolts and bottles, without losing contact during the manipulation. Furthermore, in some embodiments, the control system can perform tool manipulations with different object-tool-environment pairs. Because the control system can manipulate objects of different sizes and shapes without losing contact during the manipulation, the control system can be used to control a robotic arm to perform object placement tasks.

[0011] Some embodiments are based on the recognition that because a robotic arm controls an object using a tool, the pose of the controlled object is not directly observable during manipulation. There are no sensors that directly observe the object's pose. Therefore, to perform feedback control, the control system uses a haptic estimator that estimates the object's pose using tactile sensor measurements and constraints. The MPC and haptic estimator work synchronously to provide stable control of the manipulation. The MPC uses the pose estimated by the haptic estimator to calculate control signals that ensure contact is maintained throughout the manipulation. The haptic estimator estimates the object's pose, relying on the assumption that constraints are satisfied during the manipulation.

[0012] Accordingly, one embodiment discloses a control system for controlling the movement of a robotic arm holding a tool for manipulating an object. The control system comprises at least one processor and a memory having instructions stored thereon. The instructions cause the at least one processor of the control system to execute a model predictive controller (MPC) configured to collect measurements of at least one tactile sensor associated with the robotic arm, estimate a feedback signal indicative of a pose of the object based on the collected measurements and constraints imposed by an MPC, and generate control commands for actuators of the robotic arm based on the pose of the object by optimizing a cost function that minimizes deviation of the pose of the object from a target pose of the object. The optimization of the cost function is subject to the constraints. The constraints limit one or more forces acting on the object at one or more contact points to be within corresponding friction regions. The instructions further cause the at least one processor to control the actuators of the robotic arm in accordance with the control commands.

[0013] Accordingly, another embodiment discloses a method for controlling the movement of a robotic arm holding a tool for manipulating an object. The method comprises collecting measurements of at least one tactile sensor associated with the robotic arm; estimating a feedback signal indicative of a pose of the object based on the collected measurements and constraints imposed by a model predictive controller (MPC); and executing the MPC configured to generate control commands for actuators of the robotic arm based on the pose of the object by optimizing a cost function that minimizes deviation of the pose of the object from a target pose of the object. The optimization of the cost function is subject to constraints, where the constraints limit one or more forces acting on the object at one or more contact points to be within corresponding friction regions. The method further comprises controlling the actuators of the robotic arm in accordance with the control commands.

[0014] Accordingly, yet another embodiment discloses a non-transitory computer-readable storage medium having embodied thereon a program executable by a processor to perform a method for controlling movement of a robotic arm holding a tool for manipulating an object, the method including: collecting measurements of at least one tactile sensor associated with the robotic arm; estimating a feedback signal indicative of a pose of the object based on the collected measurements and constraints imposed by a model predictive controller (MPC); and executing the MPC configured to generate control commands for actuators of the robotic arm based on the pose of the object by optimizing a cost function that minimizes deviation of the pose of the object from a target pose of the object, the optimization of the cost function being subject to the constraints, the constraints limiting one or more forces acting on the object at one or more contact points to be within corresponding friction regions, the method further including controlling the actuators of the robotic arm in accordance with the control commands. [Brief explanation of the drawings]

[0015] [Figure 1A] FIG. 1 illustrates an operating system according to some embodiments of the present disclosure. [Figure 1B] FIG. 1 is a block diagram illustrating a control system for controlling the movement of a robotic arm of a manipulation system, according to some embodiments of the present disclosure. [Figure 1C] 1A-1C illustrate different forces acting on an object at a contact point according to some embodiments of the present disclosure. [Figure 2A] FIG. 1 illustrates a simplified 3D contact model for object manipulation according to some embodiments of the present disclosure. [Figure 2B] FIG. 1 illustrates a 2D contact model for object manipulation according to some embodiments of the present disclosure. [Figure 3A] FIG. 1 illustrates a haptic estimator for estimating the pose of an object, according to some embodiments of the present disclosure. [Figure 3B] 10A-10C illustrate relative frame orientations at the grip center (θ S ), according to some embodiments of the present disclosure. [Figure 4A] FIG. 1 is a block diagram illustrating an overall closed-loop operation according to some embodiments of the present disclosure. [Figure 4B] FIG. 1 is a block diagram illustrating the synchronized operation of a haptic estimator and a model predictive controller (MPC) according to some embodiments of the present disclosure. [Figure 5A] 10A-10D illustrate the operation of a bolt according to some embodiments of the present disclosure. [Figure 5B] 1A-1C illustrate the operation of a bottle according to some embodiments of the present disclosure. [Figure 5C] FIG. 1 illustrates an object placement task according to some embodiments of the present disclosure. [Figure 6] FIG. 1 is a block diagram illustrating a method for controlling the movement of a robotic arm holding a tool for manipulating an object, according to some embodiments of the present disclosure. [Figure 7] FIG. 1 is a schematic diagram illustrating a computing device for implementing the methods and systems of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0016] [Description of the embodiment] The present invention will now be described in detail with reference to the accompanying drawings, in which the drawings are not necessarily to scale, with emphasis instead generally being placed upon illustrating the principles of embodiments of the present disclosure.

[0017] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details. In other instances, devices and methods are shown only in block diagram form in order to avoid obscuring the present disclosure.

[0018] As used in this specification and claims, when used in conjunction with a list of one or more components or other items, the terms “for example,” “for instance,” and “such as,” as well as the verbs “comprising,” “having,” and “including” and other verb forms thereof, respectively, are to be construed as open-ended, meaning that such a list should not be viewed as excluding other additional components or items. The term “based on” means based at least in part on. Furthermore, it is to be understood that the phraseology and terminology used herein are for purposes of description and should not be viewed as limiting. Any headings utilized within this description are for convenience only and have no legal or restrictive effect.

[0019] FIG. 1A illustrates a manipulation system 100 according to some embodiments of the present disclosure. The manipulation system 100 includes a robotic arm 101 configured to manipulate an object 103. In one embodiment, the robotic arm 101 is configured to hold a tool 105 for manipulating the object 103 within an environment. The manipulation of the object 103 may correspond to a reorientation of the object 103, a pick-and-place operation, or an assembly operation. The manipulation of the object 103 by the tool 105 results in multiple contacts, such as a contact 107 between the robotic arm 101 and the tool 105, a contact 109 between the tool 105 and the object 103, and a contact 111 between the object 103 and the environment 113. The contacts 107, 109, and 111 must be maintained during the manipulation. Therefore, planning for the manipulation of the object 103 using the tool 105 must incorporate constraints imposed by the contacts 107, 109, and 111 to ensure that the contacts 107, 109, and 111 are maintained during the manipulation. Incorporating constraints into planning operations is difficult.

[0020] An objective of some embodiments is to design a closed-loop controller that can reliably maintain contact during manipulation. An objective of some embodiments is to enable robust implementation of planned manipulation via a closed-loop controller that uses tactile sensors. Such a closed-loop controller is described below in FIG. 1B.

[0021] 1B is a block diagram illustrating a control system 115 for controlling the manipulation of an object 103, according to some embodiments of the present disclosure. The control system 115 includes a processor 117 and a memory 119. The processor 117 may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. The memory 119 may include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. Furthermore, in some embodiments, the memory 119 may be implemented using a hard drive, an optical drive, a thumb drive, an array of drives, or any combination thereof. The memory 119 stores a haptic estimator 119a and a model predictive controller (MPC) 119b. The haptic estimator 119a and the MPC 119b are executed by the processor 117.

[0022] Control system 115 is communicatively coupled to manipulation system 100. Tactile sensors are mounted on robotic arm 101. For example, in one embodiment, the tactile sensors may be located together on the fingers of a gripper of robotic arm 101. Control system 115 collects tactile sensor measurements. The tactile sensor measurements are input to tactile estimator 119a. Tactile estimator 119a is configured to estimate a feedback signal indicative of a state of object 103. The state of object 103 may include a pose of object 103. Further, the estimated pose of object 103 is input to MPC 119b. MPC 119b is configured to generate control commands for actuators of robotic arm 101 of manipulation system 100 based on the pose of the object by optimizing a cost function that minimizes the deviation of the pose of object 103 from a target pose of object 103. The optimization of the cost function is subject to constraints that limit the forces acting on the object 103 at the contact points 109 and 111 to be within the corresponding friction regions to ensure that contact 109 and 111 is maintained during manipulation. The forces acting on the object 103 at the contact points 109 and 111 are described below in FIG. 1C.

[0023] 1C illustrates different forces acting on object 103 at contact points 109 and 111, according to some embodiments of the present disclosure. Forces 121 and 123 act on contact points 109 and 111, respectively. MPC 119b generates control commands for actuators of robot arm 101 based on the pose of object 103 by optimizing a cost function that minimizes the deviation of the pose of object 103 from a target pose 125 of object 103. The optimization of the cost function is subject to constraints that limit force 121 acting at contact point 109 to be within friction region 127 and force 123 acting at contact point 111 to be within friction region 129 to ensure that contact 109 and 111 are maintained during manipulation. The shapes of friction regions 127 and 129 depend on the coefficient of friction between the two surfaces. Here, the shapes of friction regions 127 and 129 may be conical.

[0024] Furthermore, the control system 115 controls the actuators of the robot arm 101 according to the control commands. In this way, the control system 115 controls the operation while maintaining the contacts 107, 109 and 111.

[0025]

number

[0026]

number

[0027]

number

[0028]

number

[0029] The line contact at B exists on a specific plane P217 created by the tool 203. The plane P217 is used to determine the friction cone 219 between the object 205 and the tool 203 because slippage can only occur along the plane P217. Therefore, by changing the orientation of the tool 203, the orientation of the plane P217 also changes. Such a change does not affect the friction cone constraint (3), but does affect the object 205 through static equilibrium. Furthermore, different tools have different tip geometries. Based on the kinematics of the tool 203, the definition of the local forces changes.

[0030]

number

[0031] The quasi-static equilibrium of tool 203 and object 205 given in (1a)-(2b), the friction cone constraint (3), and the inequality constraint (4) are imposed as trajectory optimization constraints for the manipulation and control of the manipulation by MPC 119b.

[0032]

number

[0033]

number

[0034] 3A illustrates a tactile estimator 119a for estimating the pose of an object 205, according to some embodiments of the present disclosure. Some embodiments are based on the recognition that tactile sensors are deformable and therefore may have non-linear stiffness. Because the tactile sensors are deformable, the stiffness of the tactile sensors is determined to accurately estimate the slippage of the object 205 in a grasp.

[0035]

number

[0036]

number

[0037]

number

[0038]

number

[0039] 4A is a block diagram illustrating an overall closed-loop manipulation, according to some embodiments of the present disclosure. Based on the dynamics of the tool manipulation, constraints on the manipulation (e.g., constraints (1a), (1b), (2a), (2b), (3), and (4)) are derived. The constraints are enforced during trajectory optimization (5) and then used to calculate feasible trajectories for the manipulation. During online control, the control system 115 estimates the pose of the object 205 using the haptic estimator 119a. Furthermore, the control system 115 executes the MPC 119b, which determines a sequence of new feasible control commands based on the estimated pose of the object 205. The control commands are applied to the actuators of the manipulation system 100 to control the manipulation.

[0040] Some embodiments are based on the recognition that the control system 115 can be used to recover from deviations from a planned trajectory or in the event of an unexpected contact. During online control, the control system 115 collects measurements from tactile sensors and estimates the pose of the object using a tactile estimator 119a. Based on the estimated pose of the object 205, the control system 115 executes an MPC 119b that determines control commands to recover from deviations from a planned trajectory.

[0041] 4B is a block diagram illustrating the synchronized operation of the haptic estimator 119a and the MPC 119b for stable operation of the closed-loop control system 115. During online control, the haptic estimator 119a provides a feedback signal 405 to the MPC 119b that indicates the pose of the object being manipulated. The feedback signal 405 is used for stable operation of the MPC 119b. Based on the pose of the object, the MPC 119b determines control signals that can enforce the constraints 407 required for manipulation. The haptic estimator 119a operates assuming that the constraints 407 are imposed during manipulation. Thus, the haptic estimator 119a and the MPC 119b operate synchronously, contributing to each other's stable operation.

[0042] Furthermore, the control system 115 can be used to manipulate objects of different sizes and shapes without losing contact during manipulation. Additionally, in some embodiments, the control system 115 can perform tool manipulations on different object-tool-environment pairs.

[0043] 5A illustrates the operation of a bolt 501 according to some embodiments of the present disclosure. A control system 115 (not shown) is communicatively coupled to a robotic arm 503. As can be seen in FIG. 5A , the control system 115 controls the robotic arm 503 such that the bolt 501 is moved to a target pose 505 without losing contact 507 between a tool 509 held by the robotic arm 503 and the bolt 501, and contact 511 between the bolt 501 and the environment 513.

[0044] 5B illustrates the manipulation of a bottle 515 according to some embodiments of the present disclosure. As can be seen in FIG. 5B, the control system 115 (not shown) controls the robotic arm 503 such that the bottle 515 is moved to a target pose 517 without losing contact 519 between the bottle 515 and a tool 521 held by the robotic arm 503, and contact 523 between the bottle 515 and the environment 527.

[0045] Because the control system 115 can control the robotic arm 503 to manipulate objects of different sizes and shapes, such as the bolt 501 and the bottle 515, without losing contact during the manipulation, the control system 115 can be used to control the robotic arm 503 to perform object placement tasks.

[0046] 5C illustrates an object placement task according to some embodiments of the present disclosure. Objects such as bolt 529, bottle 531, and box 533 are placed on table 535. Control system 115 controls robot arm 503 to move each object to its respective target pose, thereby placing the objects on table 535. For example, bolt 529, bottle 531, and box 533 are moved to target poses 537, 539, and 541, respectively. To achieve this, control system 115 can place each object according to its respective target pose, regardless of the initial pose of each object.

[0047] 6 is a block diagram illustrating a method 600 for controlling the movement of a robotic arm 101 holding a tool 105 for manipulating an object 103, according to some embodiments of the present disclosure. At block 601, the method 600 includes collecting measurements of at least one tactile sensor associated with the robotic arm 101. At block 603, the method 600 includes estimating a feedback signal indicative of a pose of the object 103 based on the collected measurements.

[0048] At block 605, the method 600 includes executing an MPC 119b configured to generate control commands for actuators of the robot arm 101 based on the pose of the object by optimizing a cost function that minimizes the deviation of the object's pose from a target pose of the object. The optimization of the cost function is subject to constraints (1a), (1b), (2a), (2b), (3), and (4) that limit one or more forces acting on the object 103 at one or more contact points to be within corresponding friction regions.

[0049] At block 607, the method 600 includes controlling an actuator of the robot arm 101 according to the control command.

[0050] 7 is a schematic diagram illustrating a computing device 700 for implementing the method and system of the present disclosure. The computing device 700 includes a power supply 701, a processor 703, a memory 705, and a storage device 707, all of which are connected to a bus 709. A high-speed interface 711, a low-speed interface 713, a high-speed expansion port 715, and a low-speed connection port 719 can be connected to the bus 709. A low-speed expansion port 717 is also connected to the bus 709. An input interface 721 can be connected to an external receiver 723 and an output interface 725 via the bus 709. The receiver 727 can be connected to an external transmitter 729 and a transmitter 731 via the bus 709. An external memory 733, an external sensor 735, a machine(s) 737, and an environment 739 can also be connected to the bus 709. One or more external input / output devices 741 can also be connected to the bus 709. A network interface controller (NIC) 743 may be adapted to connect to a network 745 via bus 709 and may render data or other data to, among other things, a third-party display device, a third-party imaging device, and / or a third-party printing device external to computing device 700.

[0051] The memory 705 can store instructions executable by the computing device 700, as well as any data that may be utilized by the methods and systems of the present disclosure. The memory 705 may include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. The memory 705 may be one or more volatile and / or non-volatile memory units. The memory 705 may also be another form of computer-readable medium, such as a magnetic disk or an optical disk.

[0052] The storage device 707 may be adapted to store supplemental data and / or software modules used by the computing device 700. The storage device 707 may include a hard drive, an optical drive, a thumb drive, an array of drives, or any combination thereof. Additionally, the storage device 707 may include an array of devices, including computer-readable media such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. The instructions may be stored on an information carrier. When executed by one or more processing devices (e.g., processor 703), the instructions perform one or more methods, such as those described above.

[0053] The computing device 700 may be optionally coupled via bus 709 to a display interface or user interface (HMI) 747 adapted to connect the computing device 700 to a display device 749 and a keyboard 751. The display device 749 may include, among other things, a computer monitor, a camera, a television, a projector, or a mobile device. In some implementations, the computing device 700 may include a printer interface for connecting to a printing device. The printing device may include, among other things, a liquid inkjet printer, a solid ink printer, a large-scale commercial printer, a thermal printer, a UV printer, or a dye-sublimation printer.

[0054] The high-speed interface 711 manages bandwidth-intensive operations for the computing device 700, while the low-speed interface 713 manages less bandwidth-intensive operations. Such an allocation of functionality is merely an example. In some implementations, the high-speed interface 711 can be coupled to memory 705 and a user interface (HMI) 747, and can further be coupled to a keyboard 751 and a display 749 (e.g., via a graphics processor or accelerator), and can further be coupled to a high-speed expansion port 715 that can accept various expansion cards via a bus 709. In one implementation, the low-speed interface 713 is coupled to a storage device 707 and a low-speed expansion port 717 via a bus 709. The low-speed expansion port 717, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices 741. The computing device 700 can be connected to a server 753 and a rack server 755. The computing device 700 may be implemented in several different forms. For example, the computing device 700 may be implemented as part of a rack server 755 .

[0055] The description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments will provide those skilled in the art with an enabling description for implementing one or more exemplary embodiments. Various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the disclosed subject matter as set forth in the appended claims.

[0056] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be understood by those skilled in the art that the embodiments may be practiced without these specific details. For example, systems, processes, and other elements in the disclosed subject matter may be shown as components in block diagram form to avoid obscuring the embodiments in unnecessary detail. In other instances, well-known processes, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments. Additionally, the same reference numbers and names in the various drawings refer to the same elements.

[0057] Also, particular embodiments may be described as a process that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. While a flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. Additionally, the order of operations may be rearranged. A process may be terminated when its operations are completed, but may have additional steps not discussed or included in the diagram. Moreover, not all operations in any process that are specifically described may occur in all embodiments. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, the termination of the function may correspond to a return of the function to the calling function or the main function.

[0058] Furthermore, embodiments of the disclosed subject matter may be implemented, at least in part, either manually or automatically. The manual or automatic implementation may be performed or at least assisted by a machine, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored on a machine-readable medium. A processor(s) may perform the necessary tasks.

[0059] The various methods or processes outlined herein may be coded as software executable on one or more processors employing any one of a variety of operating systems or platforms. Additionally, such software may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and compiled as executable machine code or intermediate code to run on a framework or virtual machine. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0060] Embodiments of the present disclosure may be implemented as a method, an example of which is provided. The order of operations performed as part of this method may be determined in any suitable manner. Thus, embodiments may be configured to perform operations in an order different from that illustrated, which may include performing some operations simultaneously, even though they are shown as a sequence in the illustrated embodiment.

[0061] Furthermore, embodiments of the present disclosure and the functional operations described herein can be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed herein and their structural equivalents, or in one or more combinations thereof. Furthermore, some embodiments of the present disclosure can be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by or to control the operation of a data processing apparatus. Still further, the program instructions can be encoded on an artificially generated propagated signal, for example, a machine-generated electrical, optical, or electromagnetic signal. The propagated signal is generated to encode information that is transmitted to a suitable receiving device for execution by a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random access memory device, or a serial access memory device, or one or more combinations thereof.

[0062] According to embodiments of the present disclosure, the term "data processing apparatus" may encompass all types of apparatus, devices, and machines that process data, including, by way of example, a programmable processor, computer, or multiple processors or computers. The apparatus may include special-purpose logic circuitry, such as an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or one or more combinations thereof.

[0063] A computer program (which may also be called or described as a program, software, software application, module, software module, script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, such as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in part of a file that holds other programs or data, for example, in one or more scripts stored in a markup language document, in a single file dedicated to the program, or in multiple coordinated files, for example, a file that stores one or more modules, subprograms, or portions of code.

[0064] A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communications network. Computers suitable for running computer programs may, by way of example, be based on general-purpose or special-purpose microprocessors or both, or any other type of central processing unit. Typically, a central processing unit receives instructions and data from a read-only memory or a random-access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data.

[0065] Typically, a computer also includes one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or is operatively coupled to such disks to receive and / or transfer data therefrom. However, a computer need not have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0066] To provide for user interaction, embodiments of the subject matter described herein can be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, that allows the user to provide input to the computer. Other types of devices can also be used to provide user interaction. For example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic input, speech input, or tactile input. Additionally, the computer can provide for user interaction by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0067] Embodiments of the subject matter described herein can be implemented in a computing system that includes a back-end component, e.g., a data server, or includes a middleware component, e.g., an application server, or includes a front-end component, e.g., a client computer having a graphical user interface or web browser that allows a user to interact with an implementation of the subject matter described herein, or includes any combination of one or more of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communications network. Communications networks include, for example, local area networks ("LANs") and wide area networks ("WANs"), e.g., the Internet.

[0068] A computing system may include clients and servers. Clients and servers are generally remote and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a mutual relationship between the client and server.

[0069] Although the present disclosure has been described with reference to certain preferred embodiments, it is to be understood that various other adaptations and modifications can be made within the spirit and scope of the disclosure. It is, therefore, the object of the appended claims to cover all such variations and modifications as come within the true spirit and scope of the present disclosure.

Claims

1. 1. A control system for controlling the movement of a robotic arm holding a tool for manipulating an object, the control system comprising: at least one processor; and a memory storing instructions, the instructions causing the at least one processor of the control system to: collecting measurements of at least one tactile sensor associated with the robotic arm; estimating a feedback signal indicative of the pose of the object based on the collected measurements of the at least one tactile sensor and constraints imposed by a Model Predictive Controller (MPC); and executing the MPC configured to generate control commands for actuators of the robot arm based on the estimated pose of the object by optimizing a cost function that minimizes deviation of the pose of the object from a target pose of the object, the optimization of the cost function being subject to the constraints, the constraints limiting one or more forces acting on the object at one or more contact points to be within corresponding friction regions; and the instructions further include: a control system that controls the actuators of the robotic arm according to the control commands.

2. The control system of claim 1 , wherein the one or more contact points include contact between the object and the tool and contact between the object and an environment.

3. The control system of claim 1 , wherein the constraints include quasi-static equilibrium between the tool and the object with zero slip at the one or more contact points.

4. The control system of claim 1 , wherein the shape of the friction area is based on the shape of the object.

5. The control system of claim 1 , wherein the processor is further configured to collect robot proprioception measurements and measurements of the at least one tactile sensor.

6. The control system of claim 5 , wherein the robot proprioception measurements include one or more of motor speeds, robot arm joint angles, and battery voltage of the robot arm.

7. The processor further comprises: calculating a relative orientation of a frame at a grasp center of the object based on measurements of the at least one tactile sensor; 6. The control system of claim 5, configured to perform a least squares regression using the robot proprioception measurements and the relative orientation of the frame at the object's grasp center to estimate the pose of the object.

8. 1. A method for controlling the movement of a robotic arm holding a tool for manipulating an object, comprising: collecting measurements of at least one tactile sensor associated with the robotic arm; estimating a feedback signal indicative of the pose of the object based on collected measurements of the at least one tactile sensor and constraints imposed by a model predictive controller (MPC); executing the MPC configured to generate control commands for actuators of the robot arm based on the estimated pose of the object by optimizing a cost function that minimizes deviation of the pose of the object from a target pose of the object, the optimization of the cost function being subject to the constraints, the constraints limiting one or more forces acting on the object at one or more contact points to be within corresponding friction regions; controlling the actuator of the robotic arm according to the control command.

9. The method of claim 8 , wherein the one or more contact points include contact between the object and the tool and contact between the object and the environment.

10. The method of claim 8 , wherein the constraints include quasi-static equilibrium between the tool and the object with zero slip at the one or more contact points.

11. The method of claim 8 , wherein the shape of the friction area is based on the shape of the object.

12. The method of claim 8 , further comprising collecting robot proprioception measurements and the at least one tactile sensor measurements.

13. The method of claim 12 , wherein the robot proprioception measurements include one or more of the following: motor speed, robot arm joint angle, and battery voltage of the robot arm.

14. calculating a relative orientation of a frame at a grasp center of the object based on measurements of the at least one tactile sensor; 13. The method of claim 12, further comprising: performing a least squares regression using the robot proprioception measurements and the relative orientation of the frame at the grasp center of the object to estimate the pose of the object.

15. 1. A non-transitory computer-readable storage medium having embodied thereon a program executable by a processor to perform a method for controlling the movement of a robotic arm holding a tool for manipulating an object, the method comprising: collecting measurements of at least one tactile sensor associated with the robotic arm; estimating a feedback signal indicative of the pose of the object based on collected measurements of the at least one tactile sensor and constraints imposed by a model predictive controller (MPC); and executing the MPC configured to generate control commands for actuators of the robot arm based on the estimated pose of the object by optimizing a cost function that minimizes deviation of the pose of the object from a target pose of the object, the optimization of the cost function being subject to the constraints, the constraints limiting one or more forces acting on the object at one or more contact points to be within corresponding friction regions, the method further comprising: and controlling the actuator of the robotic arm according to the control commands.

16. The non-transitory computer-readable storage medium of claim 15 , wherein the one or more contact points include contact between the object and the tool and contact between the object and an environment.

17. The non-transitory computer-readable storage medium of claim 15 , wherein the constraints include quasi-static equilibrium between the tool and the object with zero slip at the one or more contact points.

18. The non-transitory computer-readable storage medium of claim 15 , wherein the shape of the friction area is based on the shape of the object.

19. The non-transitory computer-readable storage medium of claim 15 , wherein the method further comprises collecting robot proprioception measurements and the at least one tactile sensor measurements.

20. The method further comprises: calculating a relative orientation of a frame at a grasp center of the object based on measurements of the at least one tactile sensor; and performing a least squares regression using the robot proprioception measurements and the relative orientation of the frame at the grasp center of the object to estimate the pose of the object.

Images