System and method for controlling the movement of a robotic arm

Abstract

The present disclosure discloses a system and method for controlling the movement of a robotic arm holding a tool for manipulating an object. The method comprises collecting measurements of a tactile sensor associated with the robotic arm, estimating a feedback signal indicative of a pose of the object based on the collected measurements and constraints imposed by a model predictive controller (MPC), and executing the MPC configured to generate control commands for actuators of the robotic arm based on the pose of the object by optimizing a cost function that minimizes deviation of the pose of the object from a target pose of the object. The optimization of the cost function is subject to constraints, where the constraints limit one or more forces acting on the object at one or more contact points to be within corresponding friction regions. The method further comprises controlling the actuators in accordance with the control commands.

Description

Technical Field

[0001] The present disclosure generally relates to robot manipulation, and more specifically, to a system and method for controlling the operation of a robotic arm that holds a tool for manipulating an object.

Background Art

[0002] In a robot system, trajectory optimization is used to determine a control trajectory for a robot to execute an operation task of moving an object from a given initial pose to a target pose. The robot system includes, for example, a robotic arm that holds a tool for manipulating an object to move the object to the target pose. The manipulation of the object by the tool causes multiple contacts, such as contacts between the robotic arm and the tool, between the tool and the object, and between the object and the environment. By efficiently using the contacts, the robot system becomes more skillful while executing complex operation tasks.

[0003] However, most modern robot systems avoid contact with the environment because the dynamics during their movement and interaction become complex and discontinuous due to the contact. To do so, it is necessary to incorporate the constraints imposed by the contact into the planning and control of the operation task. However, incorporating constraints into the planning and control of the operation is difficult due to the discontinuity. Therefore, the results of classical control theory for smooth and hybrid systems cannot be directly used for the operation. Therefore, a control system is needed to control the robot system to execute the operation task while satisfying the constraints imposed by the contact.

Summary of the Invention

[0004] The objective of some embodiments is to provide systems and methods for controlling an operating system to perform operations on objects in an environment. The operating system includes a robotic arm holding a tool for manipulating the object. The manipulation of the object may correspond to object reorientation, pick-and-place operations, or assembly operations. Manipulating an object with a tool causes multiple contacts, e.g., contact between the robotic arm and the tool, contact between the tool and the object, and contact between the object and the environment. These contacts must be maintained during the operation. Therefore, the planning of the object's operation must incorporate constraints imposed by the contacts to ensure that the contacts are maintained during the operation. Incorporating constraints into the planning of operations is challenging.

[0005] The objective of some embodiments is to design a closed-loop controller capable of reliably maintaining contact during operation. Also, the objective of some embodiments is to enable robust execution of planned operations via a closed-loop controller using tactile sensors. Some embodiments provide a control system for implementing such a closed-loop controller. The control system includes a processor and memory. The processor may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. The memory may include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. The memory stores a tactile estimator and a model predictive controller (MPC). The tactile estimator and MPC are executed by the processor.

[0006] The control system is communicatively coupled to the operating system. Tactile sensors are mounted on the robot arm. For example, in one embodiment, the tactile sensors may be positioned together with the fingers of the gripper of the robot arm. The control system collects measurements from the tactile sensors. The measurements from the tactile sensors are input to a tactile estimator. The tactile estimator is configured to estimate a feedback signal indicating the state of an object. The state of an object may include the object's orientation. Furthermore, the estimated orientation of the object is input to the MPC. The MPC is configured to generate control commands for the operating system's robot arm actuators based on the object's orientation by optimizing a cost function that minimizes the deviation of the object's orientation from the object's target orientation while satisfying the contact constraints required for the desired operating task.

[0007] The optimization of the cost function is constrained by limiting the forces acting on the object at the point of contact to satisfy the physics of frictional interaction. For example, in the case of fixed contact, the frictional force must be within the corresponding friction cone so that the contact is reliably maintained during operation. For example, to ensure that the aforementioned contact is reliably maintained during operation, the constraint limits the forces acting at the contact between the tool and the object within the corresponding friction region, depending on the coefficient of friction between the tool and the object, and limits the forces acting at the contact between the object and the environment within the corresponding friction region. The contact forces acting at the point of contact are determined by the type of contact formation and the physical properties of the objects. For example, in the case of fixed contact, the frictional force must obey Coulomb's law of friction, which states that the frictional force is within the friction cone for fixed contact and precisely at the boundary for slip contact (i.e., when the objects slide against each other).

[0008] Furthermore, the control system controls the actuators of the robot arm according to control commands. In this way, the control system controls the operation while maintaining contact.

[0009] The constraint that limits the force acting on an object at a point of contact to be within the corresponding friction region is formulated using an operational contact model. Some embodiments are based on the understanding that the operational contact model can be obtained using the principle of quasi-static equilibrium, under the assumption that the object and the tool remain in contact during the operation. In other words, a contact model designed using the principle of quasi-static equilibrium assumes zero slip, i.e., no slip, at points of contact such as between the tool and the object, and between the object and the environment.

[0010] Some embodiments of the control system are based on the recognition that it can be used to recover from deviations from a planned trajectory or in the event of unexpected contact. During online control, the control system collects measurements from tactile sensors and estimates the object's pose using a tactile estimator. The control system further executes an MPC that determines a control command to recover from deviations from the planned trajectory based on the estimated object's pose. Furthermore, the control system can be used to manipulate objects of different sizes and shapes, such as bolts and bottles, without losing contact during operation. In addition, in some embodiments, the control system can perform tool operations with different object-tool-environment pairs. Because the control system can manipulate objects of different sizes and shapes without losing contact during operation, the control system can be used to control a robotic arm to perform object placement tasks.

[0011] Some embodiments are based on the understanding that the pose of the object being controlled during an operation is not directly observable because the robotic arm is controlling the object using a tool. There are no sensors to directly observe the object's pose. Therefore, to perform feedback control, the control system uses a tactile estimator that estimates the object's pose using tactile sensor measurements and constraints. The MPC and the tactile estimator work synchronously to provide stable control of the operation. The MPC uses the pose estimated by the tactile estimator to compute control signals that ensure contact is maintained throughout the operation. The tactile estimator estimates the object's pose by taking advantage of the assumption that constraints are met during the operation.

[0012] Accordingly, one embodiment discloses a control system for controlling the movement of a robotic arm holding a tool for manipulating an object. The control system comprises at least one processor and a memory storing instructions. Instructions cause at least one processor in the control system to collect measurements from at least one tactile sensor associated with the robotic arm, to estimate a feedback signal indicating the object's pose based on the collected measurements and constraints imposed by the Model Predictive Controller (MPC), and to generate control commands for the robotic arm's actuators based on the object's pose by optimizing a cost function that minimizes the deviation of the object's pose from the object's target pose. The optimization of the cost function is constrained. The constraints limit one or more forces acting on the object at one or more contact points to be within the corresponding friction region. Instructions further cause at least one processor to control the robotic arm's actuators according to the control commands.

[0013] Accordingly, another embodiment discloses a method for controlling the movement of a robotic arm holding a tool for manipulating an object. The method comprises: collecting measurements from at least one tactile sensor associated with the robotic arm; estimating a feedback signal indicating the object's pose based on the collected measurements and constraints imposed by a Model Predictive Controller (MPC); and running the MPC configured to generate control commands for the robotic arm's actuators based on the object's pose by optimizing a cost function that minimizes the deviation of the object's pose from the object's target pose. The optimization of the cost function is constrained, the constraints limiting one or more forces acting on the object at one or more contact points to be within the corresponding friction region. The method further comprises controlling the robotic arm's actuators according to the control commands.

[0014] Accordingly, yet another embodiment discloses a non-temporary computer-readable storage medium embodying a program executable by a processor to perform a method for controlling the movement of a robotic arm holding a tool for manipulating an object. The method includes: collecting measurements from at least one tactile sensor associated with the robotic arm; estimating a feedback signal indicating the pose of an object based on the collected measurements and constraints imposed by a Model Predictive Controller (MPC); and running the MPC configured to generate control commands for the actuators of the robotic arm based on the pose of the object by optimizing a cost function that minimizes the deviation of the object's pose from the object's target pose, the optimization of the cost function being constrained, the constraints limiting one or more forces acting on the object at one or more contacts to be within the corresponding friction region; and the method further includes controlling the actuators of the robotic arm in accordance with the control commands. [Brief explanation of the drawing]

[0015] [Figure 1A] This figure shows an operating system according to some embodiments of the present disclosure. [Figure 1B] This is a block diagram showing a control system for controlling the movement of a robot arm in an operating system, according to some embodiments of the present disclosure. [Figure 1C] This figure shows different forces acting on an object at a point of contact, according to some embodiments of the present disclosure. [Figure 2A] This figure shows a simplified 3D contact model of object manipulation according to some embodiments of the present disclosure. [Figure 2B] This figure shows a 2D contact model of object manipulation according to some embodiments of the present disclosure. [Figure 3A] This figure shows a tactile estimator for estimating the pose of an object, according to some embodiments of the present disclosure. [Figure 3B] This figure shows the relative orientation of the frame at the gripping center (θS) according to some embodiments of the present disclosure. [Figure 4A] This is a block diagram showing an overall closed-loop operation according to some embodiments of the present disclosure. [Figure 4B] This block diagram shows the synchronous operation of a tactile estimator and a model predictive controller (MPC) according to some embodiments of the present disclosure. [Figure 5A] This figure shows the operation of a bolt according to some embodiments of the present disclosure. [Figure 5B] This figure shows the operation of a bottle according to some embodiments of the present disclosure. [Figure 5C] This figure shows an object placement task according to some embodiments of the present disclosure. [Figure 6] This diagram shows a method for controlling the movement of a robotic arm holding a tool for manipulating an object, according to some embodiments of the present disclosure. [Figure 7] This is a schematic diagram showing a computing device for implementing the method and system of this disclosure. [Mode for Carrying Out the Invention]

[0016] [Description of Embodiments] Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. The drawings shown are not necessarily to scale, and instead, generally, emphasis is placed on explaining the principles of the embodiments of the present disclosure.

[0017] In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without these specific details. In other instances, devices and methods are shown only in block diagram form in order to avoid obscuring the present disclosure.

[0018] As used in this specification and the claims, when used in conjunction with a list of one or more components or other items, the terms "for example", "for instance", and "such as", as well as the verbs "comprising", "having", "including", and other verb forms thereof, should each be interpreted as open-ended, meaning that such listings should not be regarded as excluding other additional components or items."Based on" means based at least in part on. Further, it should be understood that the expressions and terms used in this specification are for the purpose of explanation and should not be regarded as limiting. Any headings utilized within this description are for convenience only and have no legal or limiting effect.

[0019] Figure 1A shows an operating system 100 according to some embodiments of the present disclosure. The operating system 100 includes a robotic arm 101 configured to manipulate an object 103. In some embodiments, the robotic arm 101 is configured to hold a tool 105 for manipulating the object 103 within an environment. The manipulation of the object 103 may correspond to reorienting the object 103, a pick-and-place operation, or an assembly operation. The manipulation of the object 103 by the tool 105 causes a plurality of contacts, such as a contact 107 between the robotic arm 101 and the tool 105, a contact 109 between the tool 105 and the object 103, and a contact 111 between the object 103 and the environment 113. The contacts 107, 109, and 111 must be maintained during the operation. Thus, the planning of the manipulation of the object 103 using the tool 105 needs to incorporate the constraints imposed by the contacts 107, 109, and 111 such that the contacts 107, 109, and 111 are reliably maintained during the operation. Incorporating the constraints in the manipulation planning is difficult.

[0020] The objective of some embodiments is to design a closed-loop controller that can reliably maintain the contacts during the operation. Also, the objective of some embodiments is to enable a robust implementation of the planned manipulation via a closed-loop controller that uses tactile sensors. Such a closed-loop controller is described below with reference to Figure 1B.

[0021] Figure 1B is a block diagram showing a control system 115 for controlling the operation of an object 103 according to some embodiments of the present disclosure. The control system 115 includes a processor 117 and a memory 119. The processor 117 may be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. The memory 119 may include random-access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. Furthermore, in some embodiments, the memory 119 may be implemented using a hard drive, an optical drive, a thumb drive, an array of drives, or any combination thereof. The memory 119 stores a tactile estimator 119a and a model prediction controller (MPC) 119b. The tactile estimator 119a and the MPC 119b are executed by the processor 117.

[0022] The control system 115 is communicatively coupled to the operating system 100. A tactile sensor is mounted on the robot arm 101. For example, in one embodiment, the tactile sensor may be positioned together with the fingers of the gripper of the robot arm 101. The control system 115 collects measurements from the tactile sensor. The measurements from the tactile sensor are input to the tactile estimator 119a. The tactile estimator 119a is configured to estimate a feedback signal indicating the state of object 103. The state of object 103 may include the pose of object 103. Furthermore, the estimated pose of object 103 is input to the MPC 119b. The MPC 119b is configured to generate control commands for the actuators of the robot arm 101 of the operating system 100 based on the object's pose by optimizing a cost function that minimizes the deviation of the object's pose from the target pose of object 103. The optimization of the cost function is constrained to limit the forces acting on object 103 at contact points 109 and 111 to within the corresponding friction regions, so that contacts 109 and 111 are reliably maintained during operation. The forces acting on object 103 at contact points 109 and 111 are described below in Figure 1C.

[0023] Figure 1C shows different forces acting on object 103 at contacts 109 and 111 according to some embodiments of the present disclosure. Forces 121 and 123 act on contacts 109 and 111, respectively. The MPC 119b generates control commands for the actuators of the robot arm 101 based on the orientation of object 103 by optimizing a cost function that minimizes the deviation of the orientation of object 103 from the target orientation 125 of object 103. The optimization of the cost function is constrained to limit the force 121 acting at contact 109 to be within the friction region 127 and the force 123 acting at contact 111 to be within the friction region 129, so that contacts 109 and 111 are reliably maintained during operation. The shapes of the friction regions 127 and 129 are determined by the coefficient of friction between the two surfaces, where the shapes of the friction regions 127 and 129 may be conical.

[0024] Furthermore, the control system 115 controls the actuators of the robot arm 101 according to control commands. In this way, the control system 115 controls the operation while maintaining contacts 107, 109, and 111.

[0025]

number

[0026]

number

[0027]

number

[0028]

number

[0029] The line contact at B lies on a specific plane P217 created by tool 203. Plane P217 is used to determine the friction cone 219 between object 205 and tool 203, since sliding can only occur along plane P217. Therefore, changing the orientation of tool 203 also changes the orientation of plane P217. Such a change does not affect the friction cone constraint (3), but it does affect object 205 by static equilibrium. Furthermore, different tools have different tip shapes. Based on the kinematics of tool 203, the definition of local forces changes.

[0030]

number

[0031] The quasi-static equilibrium of tool 203 and object 205 given in (1a) to (2b), the friction cone constraint (3), and the inequality constraint (4) are imposed as constraints on trajectory optimization for the operation and control of the operation by MPC119b.

[0032]

number

[0033]

number

[0034] Figure 3A shows a tactile estimator 119a for estimating the orientation of an object 205, according to some embodiments of the present disclosure. Some embodiments are based on the recognition that the tactile sensor is deformable and therefore may have nonlinear stiffness. Because the tactile sensor is deformable, the stiffness of the tactile sensor is determined to accurately estimate the slip of the object 205 in gripping.

[0035]

number

[0036]

number

[0037]

number

[0038]

number

[0039] Figure 4A is a block diagram showing an overall closed-loop operation according to some embodiments of the present disclosure. Based on the mechanics of the tool operation, constraints on the operation (e.g., constraints (1a), (1b), (2a), (2b), (3), and (4)) are obtained. The constraints are implemented during trajectory optimization (5) and then used to calculate feasible trajectories for the operation. During online control, the control system 115 estimates the attitude of object 205 using the tactile estimator 119a. Furthermore, the control system 115 executes an MPC 119b that determines a new feasible sequence of control commands based on the estimated attitude of object 205. The control commands are applied to the actuators of the operation system 100 to control the operation.

[0040] Some embodiments are based on the recognition that the control system 115 is available to recover from deviations from a planned trajectory or in the event of an unexpected contact. During online control, the control system 115 collects measurements from tactile sensors and estimates the object's attitude using the tactile estimator 119a. Based on the estimated attitude of the object 205, the control system 115 executes the MPC 119b, which determines a control command to recover from deviations from the planned trajectory.

[0041] Figure 4B is a block diagram illustrating the synchronous operation of the tactile estimator 119a and MPC 119b in relation to the stable operation of the closed-loop control system 115. During online control, the tactile estimator 119a provides the MPC 119b with a feedback signal 405 indicating the pose of the object being manipulated. The feedback signal 405 is used for the stable operation of the MPC 119b. Based on the object's pose, the MPC 119b determines a control signal that can enforce the constraints 407 required for the operation. The tactile estimator 119a operates under the assumption that the constraints 407 will be imposed during the operation. Therefore, the tactile estimator 119a and the MPC 119b operate synchronously to contribute to the stable operation of each other.

[0042] Furthermore, the control system 115 can be used to manipulate objects of different sizes and shapes without losing contact during operation. In addition, in some embodiments, the control system 115 can perform tool operations with different object-tool-environment pairs.

[0043] Figure 5A illustrates the operation of a bolt 501 according to several embodiments of the present disclosure. A control system 115 (not shown) is communicatively coupled to a robot arm 503. As can be seen from Figure 5A, the control system 115 controls the robot arm 503 so that the bolt 501 is moved to a target position 505 without losing contact 507 between the tool 509 held by the robot arm 503 and the bolt 501, and contact 511 between the bolt 501 and the environment 513.

[0044] Figure 5B illustrates the operation of a bottle 515 according to some embodiments of the present disclosure. As can be seen from Figure 5B, a control system 115 (not shown) controls the robot arm 503 so that the bottle 515 is moved to a target position 517 without losing contact 519 between the tool 521 held by the robot arm 503 and the bottle 515, and contact 523 between the bottle 515 and the environment 527.

[0045] The control system 115 can control the robot arm 503 to manipulate objects of different sizes and shapes, such as bolts 501 and bottles 515, without losing contact during operation. Therefore, the control system 115 can be used to control the robot arm 503 to perform object placement tasks.

[0046] Figure 5C illustrates an object placement task according to some embodiments of the present disclosure. Objects such as a bolt 529, a bottle 531, and a box 533 are placed on a table 535. A control system 115 places the objects on the table 535 by controlling a robotic arm 503 to move each object to its respective target orientation. For example, the bolt 529, bottle 531, and box 533 are moved to target orientations 537, 539, and 541, respectively. To this end, the control system 115 can place each object according to its respective target orientation, regardless of its initial orientation.

[0047] Figure 6 is a block diagram showing a method 600 for controlling the movement of a robotic arm 101 holding a tool 105 for manipulating an object 103, according to some embodiments of the present disclosure. In block 601, the method 600 includes collecting measurements from at least one tactile sensor associated with the robotic arm 101. In block 603, the method 600 includes estimating a feedback signal indicating the pose of the object 103 based on the collected measurements.

[0048] In block 605, method 600 includes executing MPC 119b configured to generate control commands for the actuators of the robot arm 101 based on the object's orientation by optimizing a cost function that minimizes the deviation of the object's orientation from the object's target orientation. The optimization of the cost function is subject to constraints (1a), (1b), (2a), (2b), (3) and (4) that limit one or more forces acting on the object 103 at one or more contact points to be within the corresponding friction region.

[0049] In block 607, method 600 includes controlling the actuators of the robot arm 101 according to control commands.

[0050] Figure 7 is a schematic diagram showing a computing device 700 for implementing the method and system of the present disclosure. The computing device 700 includes a power supply 701, a processor 703, memory 705, and a storage device 707, all of which are connected to a bus 709. Furthermore, a high-speed interface 711, a low-speed interface 713, a high-speed expansion port 715, and a low-speed connection port 719 can be connected to the bus 709. A low-speed expansion port 717 is also connected to the bus 709. In addition, an input interface 721 can be connected to an external receiver 723 and an output interface 725 via the bus 709. Receiver 727 can be connected to an external transmitter 729 and a transmitter 731 via the bus 709. Furthermore, an external memory 733, an external sensor 735, a machine 737, and an environment 739 can be connected to the bus 709. In addition, one or more external input / output devices 741 can be connected to the bus 709. The network interface controller (NIC) 743 can be adapted to connect to network 745 via bus 709, and data or other data can be rendered, among other things, to third-party display devices, third-party imaging devices, and / or third-party printing devices outside of the computer device 700.

[0051] Memory 705 can store instructions executable by computer device 700, as well as any data that may be utilized by the methods and systems of this disclosure. Memory 705 may include random access memory (RAM), read-only memory (ROM), flash memory, or any other suitable memory system. Memory 705 may be one or more volatile memory units and / or non-volatile memory units. Memory 705 may also be another form of computer-readable medium, such as a magnetic disk or an optical disk.

[0052] The storage device 707 may be adapted to store supplemental data and / or software modules used by the computer device 700. The storage device 707 may include a hard drive, optical drive, thumb drive, array of drives, or any combination thereof. Furthermore, the storage device 707 may include an array of devices including computer-readable media such as floppy disk devices, hard disk devices, optical disk devices, or tape devices, flash memory or other similar solid-state memory devices, or devices in a storage area network or other configuration. Instructions may be stored on the information carrier. When an instruction is executed by one or more processing devices (e.g., processor 703), it performs one or more of the methods described above.

[0053] The computing device 700 may optionally be connected via bus 709 to a display interface or user interface (HMI) 747 adapted to connect the computing device 700 to a display device 749 and a keyboard 751. The display device 749 may include, among other things, a computer monitor, a camera, a television, a projector, or a mobile device. In some implementations, the computer device 700 may also include a printer interface for connecting to a printing device. The printing device may include, among other things, a liquid inkjet printer, a solid ink printer, a large-scale commercial printer, a thermal printer, a UV printer, or a dye-sublimation printer.

[0054] The high-speed interface 711 manages bandwidth-intensive operations related to the computing device 700, while the low-speed interface 713 manages less bandwidth-intensive operations. Such function assignments are merely examples. In some implementations, the high-speed interface 711 can be coupled to memory 705 and user interface (HMI) 747, and further (e.g., via a graphics processor or accelerator) to keyboard 751 and display 749, and further to a high-speed expansion port 715 that can accept various expansion cards via bus 709. In one implementation, the low-speed interface 713 is coupled to storage device 707 and low-speed expansion port 717 via bus 709. The low-speed expansion port 717, which may include various communication ports (e.g., USB, Bluetooth®, Ethernet®, wireless Ethernet), can be coupled to one or more input / output devices 741. The computing device 700 can be connected to server 753 and rack server 755. The computing device 700 may be implemented in several different forms. For example, computing device 700 can be implemented as part of rack server 755.

[0055] The description provides only exemplary embodiments and is not intended to limit the scope, availability, or configuration of the disclosure. Rather, the following description of exemplary embodiments will provide a practicable description for realizing one or more exemplary embodiments for those skilled in the art. Various modifications may be made in the function and arrangement of the elements without departing from the spirit and scope of the subject matter disclosed as set forth in the appended claims.

[0056] To provide a complete understanding of the embodiments, certain details are given in the following description. However, it will be understood by those skilled in the art that the embodiments can be practiced without these specific details. For example, to avoid obscuring the embodiments by describing them in unnecessary detail, systems, processes, and other elements in the disclosed subject matter may be shown as components in the form of block diagrams. In other cases, to avoid obscuring the embodiments, well-known processes, structures, and techniques may be shown without unnecessary detail. Also, the same reference numerals and names in different drawings refer to the same elements.

[0057] Furthermore, individual embodiments may be described as processes shown as flowcharts, flow diagrams, data flow diagrams, structural diagrams, or block diagrams. While flowcharts may describe operations as sequential processes, many operations can be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process may terminate when its operations are complete, but it may have additional steps that are not discussed or included in the diagrams. Moreover, not all operations in any specifically described process can occur in all embodiments. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. If a process corresponds to a function, the termination of the function may correspond to returning the function to a calling function or main function.

[0058] Furthermore, embodiments of the disclosed subject matter may be implemented, at least in part, manually or automatically. Manual or automatic implementation may be performed, or at least assisted, by a machine, hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. If implemented in software, firmware, middleware, or microcode, the program code or code segments for performing the required tasks may be stored in a machine-readable medium. A processor(s) may perform the required tasks.

[0059] The various methods or processes outlined herein may be encoded as software executable on one or more processors employing any one of a variety of operating systems or platforms. Furthermore, such software may be written using any of several suitable programming languages ​​and / or programming or scripting tools, and may be compiled as executable machine language code or intermediate code that runs on a framework or virtual machine. Typically, the functions of program modules may be combined or distributed as desired in various embodiments.

[0060] Embodiments of the present disclosure may be implemented as methods, and an example thereof is provided. The order of operations performed as part of this method may be determined in any suitable manner. Thus, embodiments may be configured such that operations are performed in an order different from the order illustrated, which may include performing some operations simultaneously, although they are shown as a series of operations in the illustrated embodiments.

[0061] Furthermore, embodiments of the present disclosure and the functional operations described herein can be implemented in digital electronic circuits, in tangibly implemented computer software or firmware, in computer hardware including the structures disclosed herein and their structural equivalents, or in one or more combinations thereof. Furthermore, some embodiments of the present disclosure can be implemented as one or more computer programs, i.e., as one or more modules of computer program instructions encoded on a tangible, non-temporary program carrier for execution by a data processing device or for controlling the operation of a data processing device. Furthermore, program instructions can be encoded on artificially generated propagating signals, for example, on machine-generated electrical, optical, or electromagnetic signals. The propagating signals are generated to encode information to be transmitted to a suitable receiving device for execution by the data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage board, a random-access memory device, or a serial-access memory device, or one or more combinations thereof.

[0062] According to embodiments of this disclosure, the term “data processing device” may encompass all types of devices, machines, and apparatus that process data, including, for example, a programmable processor, a computer, or multiple processors or computers. The apparatus may include dedicated logic circuits, such as FPGAs (field-programmable gate arrays) or ASICs (application-specific integrated circuits). In addition to hardware, the apparatus may also include code that generates the execution environment for the computer program, such as processor firmware, a protocol stack, a database management system, an operating system, or code comprising one or more of these.

[0063] Computer programs (sometimes called, or described as, programs, software, software applications, modules, software modules, scripts, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, as standalone programs, or as modules, components, subroutines, or other units suitable for use in a computing environment. Computer programs may, but may not, correspond to files in a file system. A program may be stored in part of a file that holds other programs or data, for example, one or more scripts stored in a markup language document, a single file dedicated to the program, or a coordinated set of files, for example, one or more modules, subprograms, or parts of code.

[0064] Computer programs can be deployed to run on a single computer, or on multiple computers located in one place or distributed across multiple locations and interconnected by a communication network. A computer suitable for running a computer program may, for example, be based on a general-purpose microprocessor, a dedicated microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory, random-access memory, or both. Essential elements of a computer are a central processing unit for executing or running instructions, and one or more memory devices for storing instructions and data.

[0065] Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operably coupled to such disks to receive data from them, transfer data to them, or both. However, a computer does not have to have such devices. Furthermore, a computer can be embedded in another device. For example, it can be embedded in a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive.

[0066] To provide user interaction, embodiments of the subject matter described herein can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, that allows the user to provide input to the computer. User interaction may be provided using other types of devices. For example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user may be received in any form, including acoustic input, voice input, or tactile input. In addition, the computer can implement user interaction by sending documents to and receiving documents from a device used by the user, for example, by sending web pages to a web browser on the user's client device in response to a request received from the web browser.

[0067] Embodiments of the subject matter described herein can be implemented in a computing system that includes, for example, a backend component as a data server, or a middleware component, such as an application server, or a frontend component, such as a client computer having a graphical user interface or a web browser that allows a user to interact with an implementation of the subject matter described herein, or any combination of one or more such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), such as the Internet.

[0068] A computing system may include clients and servers. Clients and servers are generally geographically separated and typically communicate through a communication network. The relationship between a client and a server arises from computer programs running on each computer that have a reciprocal relationship between them.

[0069] While this disclosure has been described with reference to certain preferred embodiments, it should be understood that various other adaptations and modifications can be made within the spirit and scope of this disclosure. Therefore, it is the nature of the appended claims to cover all such variations and modifications that fall within the true spirit and scope of this disclosure.

Claims

1. A control system for controlling the movement of a robotic arm holding a tool for manipulating an object, comprising at least one processor and a memory storing instructions, wherein the instructions are transmitted to the at least one processor of the control system. The robot arm collects measurements from at least one tactile sensor associated with it. We collected measurements of robot proprioception, Based on the measurements from the at least one tactile sensor, the relative orientation of the frame at the center of the object's grip is calculated. Based on the measurements from the at least one tactile sensor collected and the constraints imposed by the Model Predictive Controller (MPC), a feedback signal indicating the object's pose is estimated. Using the measured values ​​of the robot's proprioception and the relative orientation of the frame at the object's gripping center, least-squares regression is performed to estimate the object's posture. The MPC is configured to generate control commands for the actuators of the robot arm based on the estimated pose of the object by optimizing a cost function that minimizes the deviation of the object's pose from its target pose, the optimization of the cost function is subject to the constraints, the constraints limiting one or more forces acting on the object at one or more contacts to be within the corresponding friction region, and the instruction further causes the at least one processor to A control system that controls the actuator of the robot arm according to the control command.

2. The control system according to claim 1, wherein the one or more contacts include contact between the object and the tool, and contact between the object and the environment.

3. The control system according to claim 1, wherein the constraint includes a quasi-static equilibrium between the tool and the object with zero slip at one or more contacts.

4. The control system according to claim 1, wherein the shape of the friction region is based on the shape of the object.

5. The control system according to claim 1, wherein the measured value of the robot proprioception includes one or more values ​​from among motor speed, robot arm joint angle, and the battery voltage of the robot arm.

6. A method for controlling the movement of a robotic arm that holds a tool for manipulating an object, Collecting measurements from at least one tactile sensor associated with the robot arm, To collect measurements of robot proprioception, Based on the measurements of at least one of the tactile sensors, the relative orientation of the frame at the center of the object's grip is calculated, Based on the measurements of at least one tactile sensor collected and the constraints imposed by the model predictive controller (MPC), a feedback signal indicating the pose of the object is estimated. Using the measured values ​​of the robot's proprioception and the relative orientation of the frame at the object's gripping center, least-squares regression is performed to estimate the object's posture. The method further comprises: executing the MPC configured to generate control commands for the actuators of the robot arm based on the estimated posture of the object by optimizing a cost function that minimizes the deviation of the object's posture from its target posture, wherein the optimization of the cost function is subject to the constraints that limit one or more forces acting on the object at one or more contact points to be within the corresponding friction region, and the method further A method comprising controlling the actuator of the robot arm in accordance with the control command.

7. The method according to claim 6, wherein the one or more contacts include contact between the object and the tool, and contact between the object and the environment.

8. The method according to claim 6, wherein the constraint includes a quasi-static equilibrium between the tool and the object with zero slip at one or more contact points.

9. The method according to claim 6, wherein the shape of the friction region is based on the shape of the object.

10. The method according to claim 6, wherein the measured value of the robot proprioception includes one or more values ​​from among motor speed, robot arm joint angle, and battery voltage of the robot arm.

11. A non-temporary computer-readable storage medium in which a program executable by a processor is embodied for performing a method for controlling the movement of a robotic arm that holds a tool for manipulating an object, wherein the method is Collecting measurements from at least one tactile sensor associated with the robot arm, To collect measurements of robot proprioception, Based on the measurements of at least one of the tactile sensors, the relative orientation of the frame at the center of the object's grip is calculated, Based on the measurements of at least one tactile sensor collected and the constraints imposed by the Model Predictive Controller (MPC), a feedback signal indicating the pose of the object is estimated. Using the measured values ​​of the robot's proprioception and the relative orientation of the frame at the object's gripping center, least-squares regression is performed to estimate the object's posture. The method includes performing the MPC configured to generate control commands for the actuators of the robot arm based on the estimated posture of the object by optimizing a cost function that minimizes the deviation of the object's posture from its target posture, wherein the optimization of the cost function is subject to the constraints that limit one or more forces acting on the object at one or more contact points to be within the corresponding friction region, and the method further includes, A non-temporary computer-readable storage medium that includes controlling the actuator of the robot arm in accordance with the control command.

12. The non-temporary computer-readable storage medium according to claim 11, wherein the one or more contacts include contact between the object and the tool, and contact between the object and the environment.

13. The non-temporary computer-readable storage medium according to claim 11, wherein the constraint includes a quasi-static equilibrium between the tool and the object with zero slip at one or more contacts.

14. The shape of the friction region is based on the shape of the object, as described in claim 11, for a non-temporary computer-readable storage medium.

Images