Attitude detection system and attitude detection method

Abstract

To provide an attitude detection system for a workpiece which has high accuracy while increasing a processing speed.SOLUTION: The attitude detection system comprises: a mask image acquisition unit which acquires a mask image from an RGB image of an object captured by an imaging apparatus; a depth estimation model which estimates a depth to a position of the object on the basis of RGB images of the object captured by a plurality of imaging apparatuses; and an attitude detection unit which hollows out the object from the mask image and detects an attitude of the object on the basis of an image of the hollowed out object and data of the depth.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an attitude detection system and an attitude detection method. [Background technology]

[0002] A method for recognizing objects by image processing has been developed for picking up objects by a robot end effector. Patent Document 1 discloses a method for using a 3D camera to acquire a 2D trichromatic (RGB) image of a box and a 2D depth map image. The method uses an image segmentation process that uses a simplified Mask R-CNN (convolutional neural network) executable by a CPU to predict which pixels in the RGB image are associated with each box. The pixels associated with each box are given a unique label that is linked to define a mask for the box. The method then uses the segmentation image to identify a position for picking up the box. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2023-81310 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the case of an advanced estimation model, the estimation cycle was slow. Also, in the case of a model with fast processing, the accuracy and robustness of the estimation was low. Therefore, the object of the present disclosure is to provide a workpiece posture detection system with high accuracy while increasing the processing speed. [Means for solving the problem]

[0005] The posture detection system of the present disclosure comprises: a mask image acquisition unit that acquires a mask image from an RGB image of an object captured by an imaging device; a depth estimation model that estimates a depth to a position of the object based on RGB images of the object captured by a plurality of image capture devices; The posture detection system includes a posture detection unit that cuts out the object from the mask image and detects the posture of the object based on the image of the cut-out object and the depth data.

[0006] The above configuration provides a workpiece orientation detection system that can increase processing speed and achieve high accuracy.

[0007] The posture detection system of the present disclosure comprises: The mask image acquisition unit a first machine learning machine that uses a rough mask image of the object obtained by arranging a silhouette image of an object in the mask image of the object as input data, learns a clean mask image of the object as teaching data, and outputs a clean mask image of the object from the mask image of the object; a second machine learning machine that receives an RGB image of the object as input, learns a clean mask image of the object as teaching data, and outputs a clean mask image of the object from the RGB image of the object.

[0008] With the above configuration, a clear mask image of the object can be obtained from the RGB image of the object.

[0009] The posture detection system of the present disclosure comprises: The depth estimation model is obtaining a mask image from the RGB image of the object; The mask image is processed into a square image using padding. Detecting the object from the square image using an object detection model; The detected object is input into a machine learning machine to estimate the depth.

[0010] With the above configuration, the distance from the end effector to the object can be estimated.

[0011] The posture detection system of the present disclosure comprises: The attitude detection unit is Detecting a two-dimensional bounding box position from the mask image and estimating two-dimensional coordinates of the object; The image of the segment mask of the object is cut out from the mask image and processed into a square image using padding. Detecting the object from the square image using an object detection model; The detected object is input into a machine learning machine to estimate its three-dimensional size; Calculating vertex coordinates of a 3D bounding box of the object from the three-dimensional size; Calculating the rotation coordinates of the 3D Bounding Box from the vertex coordinates; The posture of the object is detected from the depth data, the two-dimensional coordinates, the three-dimensional size, and the rotation coordinates.

[0012] With the above configuration, the posture of the target object can be detected.

[0013] The posture detection method of the present disclosure includes: A mask image is obtained from an RGB image of the object captured by an imaging device. estimating a depth to a position of the object based on RGB images of the object captured by a plurality of image capture devices; The method is a posture detection method for detecting the posture of the object by cutting out the object from the mask image and detecting the posture of the object based on the cut-out image of the object and the depth data.

[0014] The above configuration provides a method for detecting the orientation of a workpiece with high accuracy while increasing the processing speed. Effect of the Invention

[0015] The present disclosure provides a workpiece orientation detection system that achieves high-precision processing speed. [Brief description of the drawings]

[0016] [Figure 1] 1 is a block diagram of a posture detection system according to an embodiment; [Diagram 2] 11 is a part of a flowchart of generation of a mask image acquisition unit according to an embodiment. [Diagram 3] 11 is a part of a flowchart of generation of a mask image acquisition unit according to an embodiment. [Figure 4] 4 is a part of a flowchart of a posture detection method according to an embodiment. [Diagram 5] 4 is a part of a flowchart of a posture detection method according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Embodiment Hereinafter, an embodiment of the present invention will be described with reference to the drawings. However, the invention according to the claims is not limited to the following embodiment. Furthermore, not all of the configurations described in the embodiment are necessarily essential as means for solving the problem. For clarity of explanation, the following description and drawings are omitted and simplified as appropriate. In each drawing, the same elements are given the same reference numerals, and repeated explanations are omitted as necessary.

[0018] (Description of Attitude Detection System and Attitude Detection Method According to the Embodiment) FIG. 1 is a block diagram of a posture detection system according to an embodiment. FIG. 2 is a part of a flowchart of generation of a mask image acquisition unit according to an embodiment. FIG. 3 is a part of a flowchart of generation of a mask image acquisition unit according to an embodiment. FIG. 4 is a part of a flowchart of a posture detection method according to an embodiment. FIG. 5 is a part of a flowchart of a posture detection method according to an embodiment. The posture detection system according to the embodiment will be described with reference to FIG. 1 to FIG. 5.

[0019] As shown in FIG. 1 , the posture detection system according to the embodiment includes a mask image acquisition unit Cnet101, a depth estimation model including contour extraction 102, a 2D bounding box 103, a raw image 104, a mask image 105, padding 106, a mask image 107, a mask image 108, padding 109, a network 110, an object detection model 1 111, an object detection model 1 112, and a linear neural network 113, and a posture detection unit including raw image 104, mask image 105, padding 106, mask image 107, mask image 108, padding 109, VGG110, object detection model 2 114, object detection model 2 115, a linear neural network 116, an object detection model 2 117, an object detection model 2 118, a linear neural network 119, and a 3D bounding box 123.

[0020] Cnet101, which is a mask image acquisition unit, acquires a mask image from an RGB image of an object captured by at least one imaging device. The imaging device is an imaging device that can acquire an RGB image. The object corresponds to, for example, a gripping part of a workpiece. The mask image is a binary image of the same size as the captured image, in which pixels to be processed are colored white and other pixels are colored black.

[0021] Cnet101 includes a first machine learning machine that receives as input data a rough mask image of an object in which a silhouette image of the object is arranged, learns as teaching data a mask image of a clean object, and outputs a clean mask image of the object from the mask image of the object. The machine learning machine is a so-called artificial intelligence (AI (Artificial Intelligence)). The machine learning machine learns using a convolutional neural network or the like. A silhouette image is a binary image in which an image of an object is grayed out. Cnet101 also includes a second machine learning machine that receives as input an RGB image of an object, learns as teaching data a mask image of a clean object, and outputs a clean mask image of the object from the RGB image of the object. Cnet101 receives as input an RGB image of an object, and outputs a clean mask image of the object from the RGB image of the object using the first and second machine learning machines.

[0022] 2, the Cnet 101, which is a mask image acquisition unit, first inputs rough mask data and trains the autoencoder A using a clean mask as training (step S201). Here, the autoencoder A is a first machine learning device.

[0023] Next, an RGB image from a camera is input, and an autoencoder B is trained using a clean mask as training (step S202). Here, autoencoder B is a second learning device.

[0024] 3, next, autoencoders A and B are combined to generate an assembled learning model C (step S301). Steps S201, S202, and S301 generate a learning device that receives an RGB image of an object as input and outputs a clean mask image of the object.

[0025] Next, the depth estimation model will be described. The depth estimation model estimates the depth to the position of an object based on RGB images of the object captured by multiple imaging devices. The depth estimation model acquires a mask image from the RGB image of the object, processes the mask image into a square image using padding, detects the object from the square image using an object detection model, and inputs the detected object into a machine learning device to estimate the depth.

[0026] 4, a contour is extracted from an input segment mask (step S401). A plurality of RGB images of an object are input to Cnet 101 to obtain a mask image 105 of the object. A contour of the object is extracted 102 from the mask image.

[0027] Next, a list of object contours is created (step S402). Next, an image with the same size as the input data and all values ​​of 0 is generated (step S403). Next, the coordinates of the contours are read (step S404). Next, a mask image is generated for each object, filling the inside of the object contour with a value of 255 (step S405). Next, the RGB image and depth image mask are input to a machine learning device, and the objects are acquired 120, 121, 122 (step S406). Next, the number of objects is incremented by 1 (step S407). Steps S404 to S407 are repeated until the number of contours is equal to the number of objects (step S408). If the number of contours is equal to the number of objects (YES in step S408), the process ends. In this way, a mask image is acquired from the RGB image of the object.

[0028] As shown in FIG. 5, the mask image is made into a square shape using padding 106, 109 (step S501). The squared image is input to network 110. Network 110 is a network that narrows down characteristic values ​​from the image and classifies the images, for example, a CNN (convolutional neural network). Next, an object is detected using object detection model 1 111, 112 (step S502). Next, the output layer is input to Linear Neural Network 113 (step S503). Then, the z position of the center of the object is estimated (step S504). The z position is the depth of the center position of the object. In this way, the depth is estimated from multiple mask images using a depth estimation model.

[0029] Next, the orientation detection unit will be described. The orientation detection unit cuts out the object from the mask image, and detects the orientation of the object based on the image of the cut-out object and depth data.

[0030] The orientation detection unit detects the 2D bounding box position from the mask image, estimates the 2D coordinates of the object, cuts out the segment mask image of the object from the mask image, and processes it into a square image using padding. The orientation detection unit then detects the object from the square image using an object detection model, inputs the detected object into a machine learning machine to estimate its 3D size, calculates the vertex coordinates of the object's 3D bounding box from the 3D size, and calculates the rotation coordinates of the 3D bounding box from the vertex coordinates. The orientation detection unit detects the orientation of the object from the depth data, 2D coordinates, 3D size, and rotation coordinates.

[0031] As shown in FIG. 5, first, the position of the 2D bounding box 103 is detected from the segment mask (step S506). The x, y coordinates obtained here are sent to step S505. Next, the image of the object is cut out (step S507). Next, the cut-out image is made into a square using padding 106, 109 (step S508). The squared image is input to the network 110. Next, the object is detected using object detection model 2 114, 115, 117, 118 (step S509). Next, the output layer is input to each machine learning machine, Linear Neural Network 116, 119 (step S510). This allows the three-dimensional size to be calculated. Next, the eight vertex coordinates of the 3D bounding box 123 are calculated from the size on the origin (step S511). Next, the rotation coordinates are calculated from the rotation angles of the vertex coordinates, and the 3D box is rotated around the origin (step S512). The 3D bounding box is rotated to obtain the rotation angle of the object, while the rotational orientation of the object is calculated.

[0032] In parallel, the center coordinates (x, y, z) of the object are calculated (step S505). Finally, the estimated object center coordinates are added to the coordinates of each vertex (step S513). In this manner, the orientation detection unit detects the orientation of the target object. In this manner, by cutting out a portion of the image, a highly accurate workpiece orientation detection system is provided while speeding up processing.

[0033] The present invention is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit and scope of the present invention. [Explanation of symbols]

[0034] 101 Cnet, 102 Extracting contours, 103 2D bounding box, 104 Raw image, 105 Mask image, 106 Padding, 107 Mask image, 108 Mask image, 109 Padding, 110 VGG, 111 Object detection model 1, 112 Object detection model 1, 113 Linear neural network, 114 Object detection model 2, 115 Object detection model 2, 116 Linear neural network, 117 Object detection model 2, 118 Object detection model 2, 119 Linear neural network, 120 Obtaining object, 121 Obtaining object, 122 Obtaining object, 123 3D bounding box

Claims

1. a mask image acquisition unit that acquires a mask image from an RGB image of an object captured by an imaging device; a depth estimation model that estimates a depth to a position of the object based on RGB images of the object captured by a plurality of image capture devices; a posture detection unit that cuts out the object from the mask image and detects the posture of the object based on the cut-out image of the object and the depth data.

2. The mask image acquisition unit a first machine learning machine that uses a rough mask image of the object obtained by arranging a silhouette image of an object in the mask image of the object as input data, learns a clean mask image of the object as teaching data, and outputs the clean mask image of the object from the mask image of the object; a second machine learning machine that receives an RGB image of the object as an input, learns a clean mask image of the object as teaching data, and outputs a clean mask image of the object from the RGB image of the object.

3. The depth estimation model is obtaining a mask image from the RGB image of the object; The mask image is processed into a square image by padding, Detecting the object from the square image using an object detection model; The posture detection system according to claim 1 , wherein the detected object is input to a machine learning machine to estimate the depth.

4. The attitude detection unit is Detecting a two-dimensional bounding box position from the mask image and estimating two-dimensional coordinates of the object; The image of the segment mask of the object is cut out from the mask image, and processed into a square image by padding; Detecting the object from the square image using an object detection model; The detected object is input to a machine learning machine to estimate its three-dimensional size; Calculating vertex coordinates of a 3D bounding box of the object from the three-dimensional size; Calculating rotation coordinates of the 3D Bounding Box from the vertex coordinates; The posture detection system according to claim 1 , wherein the posture of the object is detected from the depth data, the two-dimensional coordinates, the three-dimensional size, and the rotation coordinates.

5. A mask image is obtained from an RGB image of an object captured by an imaging device; Estimating a depth to a position of the object based on RGB images of the object captured by a plurality of image capture devices; The posture detection method includes: cutting out the object from the mask image; and detecting the posture of the object based on the cut-out image of the object and the depth data.

Images