Control device and control method for robot that picks up an object, computer program, and computer-readable medium

Abstract

To provide a method for controlling a robot for picking up an object.SOLUTION: A method includes: receiving a camera image of an object; ascertaining an image region in the camera image showing an area of the object where it may not be picked up, by conveying the camera image to a machine learning model which is trained to allocate values to regions in camera images that represent whether the regions show areas of an object where it may not be picked up; allocating the ascertained image region to a spatial region; and controlling the robot to grasp the object in a spatial region other than the ascertained spatial region.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present disclosure relates to a control device and a control method for a robot that picks up an object. [Background technology]

[0002] To enable flexible manufacturing or processing of objects by a robot, it is desirable for the robot to be able to handle the object regardless of where the object is located in the robot's operating space. Therefore, the robot should be able to identify the location of the object, at least as far as picking up (e.g., grasping) the object, i.e., determine the object's pickup pose (e.g., grasping pose), so that it can correspondingly properly orient its end effector (e.g., equipped with a gripper) and move the end effector to the correct position. However, secondary situations may arise in which an object cannot be grasped at every location. This is because, for example, the gripper may obscure the barcode or the object may be damaged if grasped at a sensitive location. Therefore, a control method for a robotic device that picks up objects at various locations is desired, which avoids grasping at certain locations on the object. Summary of the Invention [Means for solving the problem]

[0003] According to various embodiments, a method for controlling a robot for picking up an object is provided, the method including receiving a camera image of the object and identifying image regions in the camera image that indicate locations on the object where the object should not be picked up by feeding the camera image to a machine learning model that is trained to assign values ​​to regions in the camera image that indicate whether the regions indicate locations on the object where the object should not be picked up. and, assigning the identified image region to a spatial region, and controlling the robot to grasp the object in a spatial region different from the identified spatial region.

[0004] The above-described method allows for reliable pick-up (eg, grasping) of an object at any position on the object, thereby avoiding grasping an object where it should not be grasped.

[0005] In the following, various examples are given.

[0006] The first embodiment is a method for controlling a robot that picks up objects at various positions as described above.

[0007] Example 2 is the method of Example 1, the method including, for an object, identifying an image region by training a machine learning model to map a camera image of the object onto a descriptor image, where the descriptor image onto which the camera image is mapped has, for a portion of the object shown in the camera image at the image position, a descriptor value of a portion of the object at the image position, the method including: obtaining descriptor values ​​of portions of the object where the object should not be picked up; mapping the camera image onto the descriptor image using the trained machine learning model; identifying a region in the descriptor image having the obtained descriptor value; and identifying the image region as a region of the camera image at the image position that corresponds to the identified region in the descriptor image.

[0008] Training such a machine learning model to identify image regions of an object that indicate where the object should not be picked up makes it possible (at little additional cost) to identify the object in space and thus appropriately control a robot to pick up the object.

[0009] Example 3 is a method as described in Example 2, the method including: obtaining descriptor values ​​of locations on the object where the object should not be picked up; using a machine learning model to map a camera image onto the descriptor image in which areas are marked as areas indicating where the object should not be picked up; and selecting descriptor values ​​of the marked areas from the descriptor image.

[0010] For example, a user can mark an area once (i.e., for one camera image), and then areas of an object indicating where the object should not be picked up can be identified across descriptor values ​​for all other camera images.

[0011] Example 4 is the method of example 2 or 3, wherein the identified image regions are assigned to spatial regions using a trained machine learning model by: identifying a 3D model of the object having a grid of vertices to which descriptor values ​​are assigned; identifying correspondences between positions in the camera image and vertices of the 3D model by assigning, to positions, vertices that have descriptor values ​​that are equivalent to those that the descriptor image has at that position; and assigning the identified image regions to locations on the object according to the identified correspondences between the positions in the camera image and the vertices of the 3D model.

[0012] In this way, the machine learning model also enables the assignment of image regions to corresponding spatial regions at little additional cost.

[0013] Example 5 is a method according to Example 1, including: identifying image regions using a plurality of camera images and marking one or more image regions in the camera images that indicate where the object should not be picked up, by training a machine learning model to identify image regions in the camera images that indicate where the object should not be picked up; and identifying the image regions by supplying the camera images to the trained machine learning model.

[0014] For example, if training data, images with examples of barcode objects that should not be obscured, is available for such training of a machine learning model, this provides an efficient approach to identifying image regions of the object that indicate where the object should not be picked up.

[0015] Example 6 is the method according to any one of Examples 1 to 5, further comprising receiving depth information for image regions of a camera image and allocating the identified image regions to spatial domains using the depth information.

[0016] For example, by using the depth information provided by an RGB-D camera, it becomes possible to assign identified image regions to spatial domains with low computational cost.

[0017] Example 7 is a robot controller configured to perform the method according to any one of Examples 1 to 6.

[0018] Example 8 is a computer program comprising instructions that, when executed by a processor, cause the processor to perform the method according to any one of Examples 1 to 6.

[0019] Example 9 is a computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method according to any one of Examples 1 to 6.

[0020] In these drawings, like reference numerals refer to like parts generally throughout the various views. The drawings are not necessarily to scale, emphasis instead being placed generally upon illustrating the principles of the invention. In the following description, various aspects are described with reference to the following drawings: [Brief explanation of the drawings]

[0021] [Figure 1] FIG. 1 is a diagram illustrating a robot. [Figure 2] FIG. 1 illustrates training of a neural network, according to an embodiment. [Figure 3] FIG. 10 illustrates a method for identifying a grasping pose. [Figure 4] FIG. 4 illustrates training for the method described in connection with FIG. 3 in the case where a dense object network is used. [Figure 5] FIG. 4 illustrates training for the method described in connection with FIG. 3 in the case where a machine learning model is trained to identify non-grasped regions in camera images. [Figure 6] FIG. 10 is a diagram illustrating a method for controlling a robot that picks up an object. DETAILED DESCRIPTION OF THE INVENTION

[0022] The following detailed description refers to the accompanying drawings, which show, by way of example, specific details and aspects of the present disclosure in which the present invention may be practiced. Other aspects may be utilized and structural, logical, and electrical changes may be made without departing from the scope of protection of the present invention. of The various aspects of the present disclosure are not necessarily mutually exclusive, as aspects may be combined with one or more other aspects of the present disclosure to form new aspects.

[0023] Various examples are described in more detail below.

[0024] FIG. 1 shows a robot 100 .

[0025] The robot 100 includes a robotic arm 101, e.g., an industrial robotic arm, for handling or attaching a workpiece (or one or more other objects). The robotic arm 101 includes manipulators 102, 103, and 104 and a base (or holder) 105, which holds the manipulators 102, 103, and 104. The term "manipulator" refers to a movable component of the robotic arm 101, the operation of which allows physical interaction with the surroundings, e.g., to perform a task. For control, the robot 100 includes a (robot) controller 106, which is designed to execute the interaction with the surroundings according to a control program. The final component 104 of the manipulators 102, 103, and 104 (which is furthest from the holder 105), also referred to as the end effector 104, may include one or more tools, e.g., a welding torch, a gripping tool, a painting device, etc.

[0026] The other manipulators 102, 103 (which are closer to the base 105) can form a positioning device, resulting in a robotic arm 101 with an end effector 104 at its end, along with an end effector 104. The robotic arm 101 is a mechanical arm that can provide similar functionality to a human arm (possibly with a tool at its end).

[0027] The robotic arm 101 may include joint elements 107, 108, and 109 that connect the manipulators 102, 103, and 104 to one another and to a holder 105. The joint elements 107, 108, and 109 may have one or more joints, each capable of providing rotatable (i.e., rotational) and / or translational (i.e., translational) motion to the associated manipulator relative to one another. Movement of the manipulators 102, 103, and 104 may be initiated using actuators controlled by a controller 106.

[0028] The expression "actuator" may be understood as a component configured to respond to its own driving force to effect a mechanism or process. An actuator can realize a command (so-called activation) made by the controller 106 in a mechanical movement. An actuator, for example an electromechanical converter, may be designed to convert electrical energy into mechanical energy in response to its own driving force.

[0029] The expression "controller" may be understood as any type of logic-implementing entity, which may include, for example, a circuit and / or a processor capable of executing software, firmware, or a combination thereof stored in a storage medium, and which may, for example, issue commands to actuators in this example. The controller may, for example, be constructed by program code (e.g., software), which controls the operation of a system, in this example, the operation of a robot.

[0030] In this example, the controller 106 includes one or more processors 110 and a memory 111. The memory 111 stores program code and data based on which the processor 110 controls the robotic arm 101. In various embodiments, the controller 106 controls the robotic arm 101 based on a machine learning model 112 stored in the memory 111.

[0031] In various embodiments, the machine learning model 112 is designed and trained to enable the robot 100 (particularly the controller) to identify portions of the object 113 where the object 113 should not be grasped. For example, the object 113 may have a fragile portion 115 (e.g., a box-shaped object may have a window cut out that can be easily damaged), or the object 113 may have a barcode (or QR code) 116. The barcode 116 should not be obscured by the end effector 104 because the barcode 116 should be read when the robot arm 101 holds the object 113. Identifying portions of the object 113 where the object 113 should not be grasped (hereinafter also referred to as non-grasp portions) enables the robot 100 to handle objects with such portions, which occur in many applications.

[0032] The robot 100 may be equipped with, for example, one or more cameras 114, which enable the robot 100 to take images of its operating space. The cameras 114 are, for example, fixed to the robot arm 101, and the robot can create images of the object 113 from different viewpoints by rotating the robot arm 101.

[0033] This allows, on the one hand, the robot 100 to determine the pose of the object 113, and, on the other hand, the machine learning model 112 to be trained to identify regions in the camera image that indicate non-grasped locations. Knowledge of the pose and non-grasped locations of the object 113 allows the controller 106 to determine a grasping pose (i.e., a position and orientation of the end effector 104) of the robot for grasping (or generally for picking up) the object 113 that avoids the robot grasping the non-grasped locations of the object 113.

[0034] The camera 114 may, for example, provide images with depth information (e.g., RGB-D images), which may enable the control device 106 to determine the pose of the object 113 from one or more camera images (possibly from different viewpoints).

[0035] However, the controller 106 may also implement a machine learning model 112 whose output the controller 106 can use to identify a pickup pose (e.g., a grasping pose or a suction pose) for the object 113.

[0036] An example of such a machine learning model 112 for object identification is a dense object network, which maps an image (e.g., an RGB image provided by a camera 114) onto a descriptor space image of any dimension (dimension D). However, other machine learning models 112 can also be used to identify the grasp pose of an object 113, particularly machine learning models 112 that do not necessarily generate "dense" feature maps, but simply assign descriptor values ​​to specific points (e.g., corners) of the object.

[0037] A dense object network is a neural network that is trained using self-supervised learning to output a descriptor space image for an input image of an image. If a 3D model of the object (e.g., a CAD (Computer Aided Design)-model) is known, as is typically the case in industrial assembly or processing tasks, the dense object network may also be trained using supervised learning.

[0038] For this purpose, for example, a target image is generated for each camera image, i.e., a pair of camera image and target image is formed, and this pair of training input images and the associated target image is used as training data for training the neural network, as shown in Figure 2.

[0039] FIG. 2 illustrates the training of a neural network 200 according to an embodiment.

[0040] Neural network 200 is a fully convolutional network, which maps an h×w×3 tensor (input image) onto an h×w×D tensor (output image).

[0041] It comprises multiple stages of convolutional layers 204, followed by pooling layers, upsampling layers 205 and skip connections 206, which allow the outputs of the various layers to be combined.

[0042] For training, the neural network 200 receives training input images 201 and outputs an output image 202 comprising pixel values ​​in descriptor space (e.g., color components according to descriptor vector components). A training loss is computed between the output image 202 and a target image 203 associated with the training input image. This is done for each stack of training input images, the training loss is averaged across multiple training input images, and the weights of the neural network 200 are trained using stochastic gradient descent using this training loss. The training loss computed between the output image 202 and the target image 203 is, for example, an L2 loss function (which minimizes the least squared error per pixel between the target image 203 and the output image 202).

[0043] The training input images 201 show the object, and the target and output images contain vectors in descriptor space, which may be mapped onto colors, so that the output image 202 (and target image 203) resembles a heat map of the object.

[0044] The vectors in the descriptor space (also called (dense) descriptors) are d-dimensional vectors (e.g., d is 1, 2, or 3) that are assigned to each pixel in each image (e.g., each pixel of the input image 201, assuming that the input image 201 and the output image 202 have the same dimensions). The dense descriptors implicitly encode the surface geometry of the object shown in the input image 201, invariant to its location or camera position.

[0045] If the machine learning model 112 generates camera images from descriptor images (such as a dense object network), it may proceed, for example, as follows. a. Register camera images to an object type (e.g., a special box) from different viewpoints. This can be done using a robot with a fixed camera, or by a human with a handheld camera. b. Train the machine learning model 112 to output a descriptor image for camera images of this object type. This results in a machine learning model that assigns a descriptor value (e.g., a feature vector) to each surface point of the object, regardless of the viewpoint in the camera image. c. In a representative camera image of the object, the user marks a no-grasp region (or multiple no-grasp regions) for the indicated object. d. By tracking the surface points of the object marked by the user in this manner using the descriptor values, the control device 106 can automatically identify non-grasped regions (i.e., regions that indicate non-grasped locations on the object) in newly captured camera images.

[0046] Alternatively to the above approach, the machine learning model 112 may be trained to directly identify non-grasp regions in the camera image. For example, the machine learning model is a convolutional network (e.g., Mask-RCNN) that is trained to correspondingly segment the camera image. This is possible if training images with markings of non-grasp regions are available. This may be suitable, for example, for cases where the non-grasp regions are non-grasp regions that contain barcodes, because in such cases the machine learning model 112 may be trained to find barcodes in the image. In this case, for example, a target image showing a segmentation of the input camera image 201 (e.g., into barcode and non-barcode regions) replaces the target image 203 with descriptor values ​​in FIG. 2 . The architecture of the neural network may be adapted to this task.

[0047] FIG. 3 illustrates a method for determining a grasp pose, implemented, for example, by the controller 106.

[0048] The camera 114 captures a camera image 301, e.g., an RGB-D image, of the object 113 to be grasped. This image is fed to a trained machine learning model 302, e.g., a dense object network or neural network, which identifies non-grasped regions in the camera image. From the output of the neural network (via a segmentation of the camera image output from the neural network, or directly via descriptor values ​​assigned to various image regions), the controller identifies non-grasped regions 303 in the camera image.

[0049] In 304, the control device 106 projects each non-grasped region 303 onto 3D coordinates, e.g., onto a non-grasped location of the object, or into 3D coordinates in the robot arm 101's working area, e.g., into 3D coordinates in the robot cell's coordinate system (using the known geometry of the robot-working area and the intrinsic and extrinsic calibration of the camera). This can be done using depth information. Optionally, this can be done via descriptor values, where an object pose that matches the observed camera image is identified (so that the descriptor value appears at the correct location in the camera image or the corresponding descriptor image). Furthermore, a corresponding PnP (Perspective-n-Point) problem is solved.

[0050] The controller then excludes the identified 3D coordinate from possible grasp poses at 305 (i.e., grasp poses that would grasp the object at the point where they intersect with the identified 3D coordinate).

[0051] The controller (eg, grip plan module) then identifies a safe grasping pose 306 in which non-grasped portions of the object are not grasped.

[0052] FIG. 4 illustrates training for the method described in connection with FIG. 3 in the case where a dense object network (or generally a machine learning model that maps camera images onto descriptor images) is used.

[0053] At 401, camera images are taken from different perspectives of the object type to be grasped.

[0054] At 402, a dense object network is trained using the camera images to map them onto the descriptor images (according to the assignment of the object's surface points (e.g., vertices) to descriptor values, which may be predetermined in the case of supervised learning, or learned concurrently in the case of unsupervised learning). Here, as is conventional in the context of 3D models, the grid points of the 3D object model are referred to as "Vertices" (singular Vertex).

[0055] At 403, the user defines a non-grasp area in one image, for example, by indicating a rectangular frame of the non-grasp area with the mouse.

[0056] This results in a trained learning model 404 that can identify non-grasped portions of an object in newly captured camera images, regardless of viewpoint, as indicated by the user's definition of non-grasped regions in the camera images.

[0057] FIG. 5 illustrates training for the method described in connection with FIG. 3 in the case where a machine learning model is trained (supervised) to identify (directly) non-grasped regions in the camera image, i.e., to correspondingly segment the input camera image.

[0058] At 501, images containing examples for non-grasped regions (e.g., barcodes) are collected. At 502, each of the collected images is assigned a marking of the non-grasped regions in the image (e.g., a corresponding segmented image). At 503, a neural network is trained for the detection of non-grasped regions in newly captured camera images using the training data thus generated.

[0060] In summary, according to various embodiments, a method is provided as illustrated in FIG.

[0061] FIG. 6 shows a flowchart of a method for controlling a robot to pick up an object, which is implemented, for example, by the control device 106.

[0062] At 601, a camera image of an object is received (eg, captured by a camera).

[0063] At 602, image regions in the camera image that indicate locations on the object where the object should not be picked up are identified by feeding the camera image to a machine learning model that has been trained to assign values ​​to regions in the camera image that indicate whether those regions indicate locations on the object where the object should not be picked up.

[0064] This value may be, for example, a descriptor value, or may be a value indicative of a segmentation of the camera image (which may be generated, for example, by a convolutional network trained for segmentation).

[0065] At 603, the identified image regions are assigned to spatial domains.

[0066] At 604, the robot is controlled to grasp the object in a spatial region different from the identified spatial region.

[0067] In other words, according to various embodiments, a machine learning model (e.g., using a neural network) is used to identify a region of the object in the camera image that indicates where the object should not be picked up (e.g., grasped or sucked). This region of the camera image is then mapped onto a spatial domain, e.g., via depth information or via solving a PnP problem. This spatial domain (i.e., the location of the object in space indicated in the identified region) is then excluded from pick-up. That is, for example, a grasping pose that would grasp (or suck) the object at this location is excluded from the set of grasping poses from which, e.g., a planning software module, may select.

[0068] "Picking up" is understood to mean, for example, grasping by a gripper. However, other types of holding mechanisms may also be used, such as, for example, a suction cup for suctioning the object. Furthermore, "picking up" should not necessarily mean that the object is simply moved, for example, a part of a larger structure may be removed and bent without being separated from the larger structure.

[0069] The machine learning model is, for example, a neural network, but other correspondingly trained machine learning models may also be used.

[0070] In various embodiments, the machine learning model assigns pixels of the object (in the image plane of each camera image) to descriptors. This can be considered an implicit encoding of the object's surface geometry. This association between descriptors and surface geometry can be made explicitly by rendering to map the descriptors onto the image plane. Note that descriptor values ​​at faces (i.e., non-vertex points) of the object model can be determined using interpolation. For example, if a face is given by three vertices of the object model, each with its own descriptor value y1, y2, and y3, then at any point on the face, the descriptor value y can be calculated as the weighted sum of these values: w1·y1+w2·y2+w3·y3. In other words, the descriptor values ​​at the vertices are interpolated.

[0071] In various embodiments, the machine learning model is trained with training data image pairs, each training data image pair including a training input image and a target image of an object, where the target image is generated by projecting descriptors of vertices visible in the training input image onto the training input image plane according to their locations on the object in the training input image.

[0072] In another embodiment, the machine learning model is trained with training data images, each of which includes, as ground truth, markings of areas of the training data image that indicate where on the object the object should not be grasped.

[0073] The images with their associated target images or markings are used for supervised training of the machine learning model.

[0074] The method of FIG. 6 can be implemented by one or more computers, where the one or more computers include one or more data processing devices. Components of a data processing device may be realized by one or more circuits. In some embodiments, a "circuit" should be understood as a unit that implements logic, and may be hardware, software, firmware, or a combination thereof. Thus, in some embodiments, a "circuit" may be understood as a hardwired logic circuit or a programmable logic circuit, such as a programmable processor. A "circuit" may also be understood as a processor that executes software, such as any type of computer program, e.g., a computer program in the form of programming code for a virtual machine. In some embodiments, a "circuit" may be understood as any type of implementation of the functionality described herein.

[0075] The camera images are, for example, RGB or RGB-D images, but may also be other types of camera images, for example, thermal images. A grasping pose is identified in a robot cell, for example, to control a robot to pick up an object (for example, from a box). This is done, for example, for assembling a larger object from sub-objects, moving an object, etc.

[0076] Although specific embodiments have been shown and described herein, a person of average skill in the art will recognize that various alternative and / or equivalent implementations can be substituted for the specific embodiments shown and described without departing from the scope of protection of the present invention. This application is intended to cover any adaptations or variations of the specific embodiments discussed herein. Accordingly, it is intended that the present invention be limited only by the claims and their equivalents.

Claims

1. A method for controlling a robot that picks up an object, comprising: receiving a camera image of an object; identifying image regions in the camera image that indicate locations on the object where the object should not be picked up by feeding the camera image to a machine learning model that is trained to assign values ​​to regions in the camera image that indicate whether the regions indicate locations on the object where the object should not be picked up; assigning the identified image regions to spatial domains; controlling the robot to grasp the object in a spatial region different from the identified spatial region; Including, and identifying the image region for the object by training the machine learning model to map a camera image of the object onto a descriptor image, where the descriptor image onto which the camera image is mapped has, for a location of the object shown by the camera image at the image position, a descriptor value of the location of the object at the image position; obtaining a descriptor value of the object at the location where the object should not be picked up; mapping the camera image onto a descriptor image using the trained machine learning model; identifying regions within the descriptor image having the obtained descriptor values; identifying the image region as the region of the camera image at the image position that corresponds to the identified region in the descriptor image; By identifying a 3D model of the object having a grid of vertices to which descriptor values ​​have been assigned; by identifying correspondences between positions in the camera image and vertices of the 3D model by assigning to positions vertices that have the same descriptor values ​​as the descriptor image has at those positions; and assigning the identified image regions to locations of the object according to the identified correspondences between positions in the camera image and vertices of the 3D model; The method further comprises using the trained machine learning model to assign the identified image regions to the spatial regions.

2. obtaining a descriptor value of the object at the location where the object should not be picked up; using the machine learning model to map a camera image onto a descriptor image in which areas are marked as areas indicating where the object should not be picked up; selecting the descriptor values ​​for the marked regions from the descriptor image; The method of claim 1 , comprising:

3. identifying image regions of an object indicative of where the object should not be picked up using a plurality of camera images and marking one or more image regions in the camera images indicative of where the object should not be picked up by training the machine learning model; identifying the image region by feeding the camera image to the trained machine learning model; The method of claim 1 , comprising:

4. The method of claim 1 , further comprising receiving depth information for the image regions of the camera image and allocating the identified image regions to the spatial regions using the depth information.

5. A robot controller configured to carry out the method according to any one of claims 1 to 4.

6. A computer program comprising instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 4.

7. A computer readable medium storing instructions which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 4.

Images