A method for controlling a robotic device for inserting a first object into a second object.

Abstract

This relates to a method for controlling a robotic device for inserting a first object into a second object. [Solution] The method involves moving a first object near a second object, detecting an image showing the second object and the first object located near the second object, determining a first transformation which is a transformation between the coordinate system of the first object and the coordinate system of a gripping tool using the image and / or another image, determining a second transformation by inputting the image as input and the first transformation as condition to a conditional neural network, wherein the second transformation represents a transformation between the coordinate system of the gripping tool and the coordinate system of the second object, determining a robotic trajectory for inserting the first object into the second object using the second transformation, and controlling a robotic device according to the robotic trajectory.

Description

Technical Field

[0001] Background Art Inserting an object into another object using a robotic arm is an important task in robotics, for example, in the automated manufacturing of components such as household robots.

Background Art

[0002] A specific example is inserting a plug into a socket. In this case, it is required that the robot can detect the plug, can receive (e.g., grip) the plug, and can then move the received plug toward the socket and insert it accurately therein. In this regard, "InsertionNet - A Scalable Solution for Insertion" by O. Spector and D. Di Castro, arXiv:2104.1422, 2021 (hereinafter referred to as Reference [1]), describes a two-stage approach in which the received plug is first moved to the vicinity of the socket, then the difference between the coordinate system of the plug and the coordinate system of the socket is determined based on an image, and the plug is inserted into the socket according to this difference.

[0003] Disclosure of the Invention The model described in Reference [1] is trained for a specific use case (e.g., inserting a specific plug into a specific socket), and assumes a standard gripping configuration for the plug.

[0004] It has been shown that if the plug is gripped even slightly differently from the standard gripping configuration, the performance of the model degrades significantly, and the performance of the model also degrades significantly when the model is applied to other, but similar use cases (e.g., other (even slightly different) types of plugs).

Prior Art Documents

Non-Patent Documents

[0005] [Non-Patent Document 1] O. Spector and D. Di Castro, “InsertionNet - A Scalable Solution for Insertion”, arXiv:2104.1422, 2021. [Overview of the project] [Problems that the invention aims to solve]

[0006] This disclosure relates to a method for controlling a robotic device for inserting a first object into a second object. This method maintains high performance even with the modifications described above. Specifically, a more robust method for inserting one object into another is provided. This is achieved, for example, by using a conditional neural network (also known as a conditional neural network) that uses information about how the object is grasped by a grasping tool as a condition. [Means for solving the problem]

[0007] Various embodiments relate to a method for controlling a robotic device for inserting a first object into a second object, and the method is Controlling the robotic device (according to the receiving robot configuration) to receive a first object using the robotic device's gripping tool, To move the first object to the vicinity of the second object (according to the initial robot trajectory), To detect at least one image showing a second object and a first object located in the vicinity of the second object (which is received using a grasping tool), Using at least one image and / or at least one other image showing the first object and the gripping tool, determine a first transformation which is a transformation between the coordinate system of the first object and the coordinate system of the gripping tool. The method involves determining a second transformation by inputting at least one image as input and a first transformation as a condition into a conditional neural network, wherein the second transformation represents a transformation between the coordinate system of the grasping tool and the coordinate system of the second object. The second transformation is used to determine the robot trajectory for inserting the first object into the second object, Controlling the robotic device according to the robot's trajectory, Includes.

[0008] Various examples are shown below.

[0009] Example 1 is a method for controlling a robotic device for inserting a first object into a second object, as described above.

[0010] Example 2 is configured according to Example 1, and the first transformation is determined using at least one image and / or at least one other image. This includes inputting the above-mentioned at least one image and / or the above-mentioned at least one other image into a machine learning model configured to output a first transformation in response to at least one image being input.

[0011] Specifically, the machine learning model may be trained to output a transformation between the object's coordinate system and the gripping tool's coordinate system, based on one (or more) images showing an object being gripped by the gripping tool.

[0012] In Example 3, the method according to Example 1 or 2 optionally further moves the first object near the second object according to the initial robot trajectory, The force and torque at the moment the gripping tool receives (e.g., holds, grips, etc.) the first object are detected (e.g., continuously) (using at least one force-torque sensor), To determine whether the force at that time is above the force threshold, and whether the torque at that time is above the torque threshold, It may also include, The (adapted) robot trajectory is determined by determining the first and second transformations, and the robot device is controlled according to the robot trajectory if the force at that time is greater than or equal to the force threshold and the torque at that time is greater than or equal to the torque threshold.

[0013] Specifically, the first object can be inserted into the second object by moving it according to a initially determined robot trajectory. In this case, forces and torques exceeding their respective thresholds may indicate that a collision has occurred (e.g., a collision between the first object and / or the gripping tool and the second object). If this is the case, a new, adapted robot trajectory can be determined to insert the first object into the second object without collision, according to the method described herein (e.g., Example 1). Specifically, the method described herein can function as a correction because it can determine the robot trajectory with greater accuracy than the initial robot trajectory.

[0014] Example 4 is configured according to any of Examples 1 to 3, and the conditional neural network has multiple layers, each of which is configured to apply a corresponding affine transformation to each feature determined within the layer, depending on the input condition (i.e., the first transformation).

[0015] Such conditional neural networks have been shown to have a particularly high success rate (and consequently, high accuracy in the determined robot trajectory) when controlling robot arms.

[0016] Embodiment 5 is configured according to any one of Examples 1 to 4, The second transformation is a transformation between the coordinate system of the gripping tool and the coordinate system of the second object, or, The second transformation is a transformation between the coordinate system of the first object and the coordinate system of the second object, and indicates the transformation between the coordinate system of the gripping tool and the coordinate system of the second object based on the first transformation.

[0017] Specifically, the second transformation may directly or indirectly indicate the transformation between the coordinate system of the gripping tool and the coordinate system of the second object.

[0018] Embodiment 6 is configured according to any one of Embodiments 1 to 5, wherein controlling the robot apparatus according to the robot trajectory includes controlling the robot arm of the robot apparatus having the gripping tool (as an end effector) according to the robot trajectory.

[0019] Embodiment 7 is a control apparatus configured to perform the method according to any one of Embodiments 1 to 6.

[0020] Embodiment 8 is a robot apparatus comprising: a robot arm having a gripping tool; and the control apparatus according to Embodiment 7.

[0021] Embodiment 9 is a computer program comprising instructions which, when executed by a processor, cause the processor to perform the method according to any one of Embodiments 1 to 6.

[0022] Embodiment 10 is a computer-readable medium storing instructions which, when executed by a processor, cause the processor to perform the method according to any one of Embodiments 1 to 6.

[0023] In the drawings, similar reference numerals generally refer to the same parts throughout the various drawings. The drawings are not necessarily to scale and instead generally focus on illustrating the principles of the invention. In the following description, various embodiments are described with reference to the following drawings. [Brief explanation of the drawing]

[0024] [Figure 1] This figure shows robot device assemblies according to various embodiments. [Figure 2] This figure shows a flowchart of a method for controlling a robotic device for inserting a first object into a second object, according to various embodiments. [Figure 3] This figure shows a schematic flowchart for determining a robot trajectory for inserting a first object into a second object, according to various configurations. [Modes for carrying out the invention]

[0025] The following detailed description refers to the accompanying drawings, which illustrate detailed aspects and embodiments of the present disclosure in which the present invention may be carried out. Other embodiments may be used and structural, logical, and electrical modifications may be made without departing from the scope of protection of the present invention. The various embodiments of the present disclosure are not necessarily mutually exclusive, as they may be combined with one or more other embodiments of the present disclosure to form new embodiments.

[0026] Various embodiments are described in more detail below.

[0027] Figure 1 shows robot device assembly 100 according to various embodiments. The robot device assembly 100 may have a robot device 101 (abbreviated as: robot). The robot device 101 may have a robot arm 120 for receiving an object, moving the received object, and inserting the object into another object.

[0028] In this specification, "inserting" one object into another object can be understood to mean any kind of robot-assisted insertion task in which one object is introduced into another object, such as embedding, setting up, or plugging in.

[0029] For illustrative purposes, the insertion task in various embodiments herein is described as inserting a plug 114 into a socket 116. This is for illustrative purposes only, and it should be understood that “insertion” as described herein can be any kind of insertion in which one object is inserted into another, for example, as part of robot-assisted manufacturing, assembly, processing, etc., as part of maintenance, as part of a task of a household robot, as part of a medical task of a medical robot, etc. Specifically, the robotic device 101 described herein can be any kind of robotic device having a robotic arm 120, for example, a manufacturing robot, a maintenance robot, a household robot, a medical robot, etc.

[0030] The robot arm 120 may have robot links 102, 103, and 104, and a base (or generally a mount) 105 that supports the robot links 102, 103, and 104. The term "robot link" may refer to a movable part of the robot device 101 whose movement enables physical interaction with the surroundings, for example, to perform an insertion task.

[0031] To control the robotic device 101, the robotic device assembly 100 may have a (robot) control device 106 configured to enable interaction with the surroundings according to a control program. The last element 104 of the robot links 102, 103, 104 (as seen from the base 105) is also referred to as the end effector 104. The type of end effector 104 may determine whether the robotic device 101 can perform gripping and / or non-gripping operations of objects. A gripping operation of an object may refer to the operation of an object in which an object is gripped, while a non-gripping operation of an object may refer to the operation of an object in which an object is not gripped. Thus, if the end effector 104 has at least one gripping tool (which may be a suction device (e.g., a suction head) or similar), the robotic device 101 may be capable of gripping operations of objects. In either case, the robotic device 101 may be capable of non-gripping operations of objects, for example, by pushing the object (without gripping it), for example, by changing its lateral position and / or orientation.

[0032] The other robot links 102 and 103 (closer to the base 105) may form a positioning device, thereby providing a robot arm 120 (or articulated arm) with the end effector 104 at its end, together with the end effector 104. The robot arm 120 may be a mechanical arm capable of performing functions similar to a human arm (possibly equipped with a tool at its end).

[0033] The robot arm 120 may have connecting elements 107, 108, and 109 that connect the robot links 102, 103, and 104 to each other and to the base 105. The connecting elements 107, 108, and 109 may have one or more joints, each of which may provide rotational and / or translational motion (i.e., parallel movement) of the associated robot links relative to each other. The motion of the robot links 102, 103, and 104 may be initiated with the assistance of actuators controlled by the control device 106.

[0034] The term "actuator" can be understood to mean a component suitable for influencing a mechanism in response to being driven. An actuator can convert instructions (so-called activations) issued from the control device 106 into mechanical motion. An actuator, such as an electromechanical transducer, may be configured to convert electrical energy into mechanical energy in response to its drive control.

[0035] The term “control device” (also referred to as “control unit”) can be understood to mean any kind of logic implementation unit that can, for example, execute software, firmware, or a combination thereof stored on a storage medium, and can include circuits and / or processors that can give instructions to the actuators in this embodiment. The control device may be configured, for example, to control the operation of a system, in this example, a robot, by program code (e.g., software).

[0036] In this example, the control device 106 may include a computer 110 and a memory 111, the memory 111 which stores code and data, and the computer 110 controls the robot device 101 based on these. According to various embodiments, the control device 106 may control the robot device 101 based on a robot control model 112 stored in the memory 111.

[0037] For example, the gripping task of the robotic device 101 may be an insertion task, i.e., receiving (e.g., gripping or sucking) a plug 114, moving the received plug 114, and inserting the plug 114 into a socket 116. To perform this task, the control device 106 may use images of the work area of ​​the robotic device 101 where the plug 114 and socket 116 are located.

[0038] These images of the area around the robotic device 101 may be supplied by one or more imaging sensors 113 (for example, mounted on the robotic arm 120, or otherwise mounted so that the control device 106 can control the viewing angle of one or more imaging sensors 113).

[0039] Image sensors used in this specification may include, for example, cameras (e.g., standard cameras, digital cameras, infrared cameras, stereo cameras, etc.), radar sensors, LiDAR sensors, ultrasonic sensors, etc. Therefore, images may be RGB images, RGB-D images, or depth images (also referred to as D images). Depth images as described herein may be any type of image containing depth information. Specifically, depth images may contain three-dimensional information relating to one or more objects. Depth images as described herein may include, for example, point clouds provided by LiDAR sensors and / or radar sensors. Depth images may be, for example, images with depth information provided by LiDAR sensors.

[0040] The control device 106 may be configured to control the robot arm 120 based on the output of the robot control model 112 in response to at least one image being input to the robot control model 112.

[0041] The robot control model 112 may include a machine learning model. According to various embodiments, the machine learning model may be generated (e.g., learned or trained) while the robot device 101 is not operating. The generated machine learning model may then be used to determine the skills to be performed by the robot device 101 while it is operating. Optionally, the generated machine learning model may be further adjusted (e.g., trained) (also referred to as online learning) while the robot device 101 is operating.

[0042] In various embodiments, a machine learning model may include, or may be, a conditional neural network 306. The conditional neural network 306 described herein may be any kind of conditional neural network, i.e., any kind of neural network in which conditions (e.g., context) may be input in addition to the input. For example, each layer of the conditional neural network 306 may be configured to apply a corresponding affine transformation to each feature determined within the layer, depending on the input conditions. Specifically, each layer may perform a feature-by-feature affine transformation of features depending on the conditions. For example, the conditional neural network 306 may include the FiLM architecture, as described in "FiLM: Visual Reasoning with a General Conditioning Layer" by E. Perez, arXiv:1709.07871, 2017. This is an example for illustrative purposes, and it is understood that the conditional neural network 306 may include any other architecture.

[0043] Figure 2 shows a flowchart of a (computer-implemented) method 200 for controlling a robotic device for inserting a first object into a second object, according to various embodiments.

[0044] Method 200 may include (in 202) controlling the robotic device to receive a first object (according to the receiving robot configuration) using a gripping tool of the robotic device.

[0045] Method 200 may include (in 204) moving the first object into the vicinity of the second object (according to the initial robot trajectory).

[0046] Method 200 may include (in 206) detecting at least one image showing a second object and a first object located in the vicinity of the second object (received using a grasping tool).

[0047] Method 200 may include (in 208) determining a first transformation, which is a transformation between the coordinate system of the first object and the coordinate system of the gripping tool, using the above-mentioned at least one image and / or at least one other image showing the first object and the gripping tool.

[0048] Method 200 may include (in 210) determining a second transformation by inputting at least one image as input to a conditional neural network, conditionally on a first transformation. The second transformation may (directly or indirectly) represent a transformation between the coordinate system of the grasping tool and the coordinate system of a second object.

[0049] Method 200 may include (in 212) using a second transformation to determine a robot trajectory for inserting the first object into the second object.

[0050] Method 200 may include (in 214) controlling the robotic device according to the robotic trajectory.

[0051] The following describes in more detail various aspects of Method 200, with the first object being described as a plug 114 and the second object as a socket 116 for illustrative purposes.

[0052] In various embodiments, one or more images of the area around the robot device 101 can be captured (using one or more imaging sensors 113). These one or more images may depict the plug 114 and the socket. Based on these one or more images, the control device 106 can recognize (e.g., locate) the plug 114 and the socket 116, determine a receiving robot configuration for receiving (e.g., grasping) the plug 114 using a gripping tool 104, determine an initial robot trajectory for moving the received plug 114 near the socket 116, and control the robot arm 120 accordingly.

[0053] In some embodiments, steps 206 to 214 of method 200 can always be performed as soon as the received plug 114 is located near the socket 116. In other embodiments, these steps 206 to 214 of method 200 can only be performed under predefined conditions. For example, as described in reference [1], while moving the plug 114 (in 204), the force and torque at the time the gripping tool 104 receives (e.g., holds, grips, sucks, etc.) the plug 114 can be detected (e.g., continuously) using at least one force-torque sensor. Steps 206 to 214 of method 200 can be performed if the force at the time is greater than or equal to a force threshold and / or the torque at the time is greater than or equal to a torque threshold. Specifically, forces and torques exceeding their respective thresholds may indicate a collision between the plug 114 and the socket 116, and using method 200, it can then be determined how the plug 114 can be inserted into the socket 116 without collision. Optionally, the forces and / or torques may be input to a conditional neural network 306 (in addition to at least one image).

[0054] Figure 3 shows a schematic flowchart 300 for determining the robot trajectory for inserting the plug 114 (as an exemplary first object) into the socket 116 (as an exemplary second object) according to various embodiments.

[0055] In 206, at least one image 302 showing the plug 114 in the vicinity of the socket 116, received by the gripping tool 104, may be detected. In some embodiments, one image may be used for this purpose. In other embodiments, multiple images (e.g., from different viewpoints) may be used for this purpose.

[0056] In 208, a first transformation 304 between the coordinate system of the plug 114 and the coordinate system of the gripping tool 104 can be determined. For this purpose, at least one image 302 (as illustrated in Figure 3), one or more images from a plurality of images if the at least one image 302 is a plurality of images, or one or more other images can be used. Specifically, one or more images showing at least the plug 114 and the gripping tool 104 holding the plug 114 can be used to determine the first transformation 304. These one or more images can be detected, for example, before and / or while moving the plug 114 (in 204) near the socket 116. Specifically, the first transformation 304 may show how the plug 114 is gripped by the gripping tool 104.

[0057] The first transformation 304 may be determined using a model 303 configured to output the first transformation 304 in response to one or more images being input. For example, model 303 may be a machine learning model. If at least one image 302 is used to determine the first transformation 304, the machine learning model of the control model 112 may include an additional prediction head in addition to the conditional neural network 306, which is configured to predict the first transformation 304, which is then input to the conditional neural network 306 as a condition. Alternatively, the control model 112 may include an additional model (e.g., a machine learning model) configured to predict the first transformation 304 based on one or more images.

[0058] In 210, the second transformation 308 is determined (e.g., predicted) by inputting at least one image 302 as input and the first transformation 304 as condition to the conditional neural network 306. Specifically, the second transformation 308 may be determined by considering the circumstances under which the plug 114 is grasped by the grasping tool 104.

[0059] The second transformation 308 directly or indirectly represents the transformation between the coordinate system of the gripping tool 104 and the coordinate system of the socket 116. For example, the second transformation 308 may be the transformation between the coordinate system of the plug 114 and the coordinate system of the socket 116, and the transformation between the coordinate system of the gripping tool 104 and the coordinate system of the socket 116 may be determined based on the first transformation 304. The transformation between the coordinate system of the gripping tool 104 and the coordinate system of the socket 116 may also be referred to as the insertion difference or insertion differential.

[0060] In 212, a robotic trajectory 310 for inserting the plug 114 into the socket 116 can then be determined based on the second conversion 308.

Claims

1. A method (200) for controlling a robotic device (101) for inserting a first object (114) into a second object (116), The above method (200) is, - Controlling the robot device (101) to receive the first object (114) using the gripping tool (104) of the robot device (101) (202), - Moving the first object (114) to the vicinity of the second object (116) (204), - Detecting (206) at least one image (302) showing the second object (116) and the first object (114) located in the vicinity of the second object (116), - Using at least one image (302) and / or at least one other image showing the first object (114) and the gripping tool (104), determine a first transformation (304) which is a transformation between the coordinate system of the first object (114) and the coordinate system of the gripping tool (104) (208), - Determining a second transformation (308) by inputting at least one image (302) as input and the first transformation (304) as condition to a conditional neural network (306) (210), wherein the second transformation (308) represents a transformation between the coordinate system of the gripping tool (104) and the coordinate system of the second object (116) (210), - Using the second transformation (308), determine (212) a robot trajectory (310) for inserting the first object (114) into the second object (116), - Controlling the robot device (101) according to the robot trajectory (310) (214), Method (200), including the method (200).

2. Determining the first transformation (304) using the at least one image (302) and / or the at least one other image (208) is, This includes inputting the at least one image (302) and / or the at least one other image into a machine learning model configured to output the first transformation (304) in response to the input of the image, The method according to claim 1 (200).

3. While moving the first object (114) to the vicinity of the second object (116) according to the initial robot trajectory (204), The gripping tool (104) detects the force and torque at the time it receives the first object (114), To determine whether the force at the aforementioned time point is greater than or equal to the force threshold, and whether the torque at the aforementioned time point is greater than or equal to the torque threshold, It further includes, The robot trajectory (310) is determined by determining the first transformation (304) and the second transformation (308), The robot device (101) is controlled according to the robot trajectory (310) when the force at that time is greater than or equal to the force threshold and the torque at that time is greater than or equal to the torque threshold. The method according to claim 1 or 2 (200).

4. The conditional neural network (306) has multiple layers, and each of these layers is configured to apply a respective affine transformation to each feature determined within that layer, depending on the input conditions. The method according to any one of claims 1 to 3 (200).

5. The second transformation (308) is a transformation between the coordinate system of the gripping tool (104) and the coordinate system of the second object (116), Or, The second transformation (308) is a transformation between the coordinate system of the first object (114) and the coordinate system of the second object (116), and based on the first transformation (304), the transformation between the coordinate system of the gripping tool (104) and the coordinate system of the second object (116) is shown. The method according to any one of claims 1 to 4 (200).

6. Controlling the robot device (101) according to the robot trajectory (310) (214) is, This includes controlling the robot arm (120) of the robot device (101) having the gripping tool (104) according to the robot trajectory (310), The method according to any one of claims 1 to 5 (200).

7. A control device (106) configured to carry out the method (200) according to any one of claims 1 to 6.

8. A robot arm (120) having the gripping tool (104), The control device (106) according to claim 7, A robotic device (101) equipped with the following:

9. A computer program that, when executed by a processor, includes instructions causing the processor to perform the method (200) according to any one of claims 1 to 6.

10. A computer-readable medium storing instructions for causing a processor to perform the method (200) according to any one of claims 1 to 6, when executed by the processor.

Images