Fine-grained grabbing detection method for in-class packaged articles

Abstract

The invention discloses a fine-grained grabbing detection method for in-class packaged articles, and belongs to the technical field of robot target object grabbing methods. A Mask R-CNN is adopted as a basic framework, fine-grained feature extraction and learning are carried out, an FGM R-CNN target detection framework is constructed, and target objects in classes are classified; the target object grabbing detection part comprises the following steps: for a target detection object, taking an identified single picture as input information, calculating a central point coordinate of a grabbed object by using the size of a detection frame, inputting an RGB-D picture of the single target detection object into a CNN optimization network framework, directly generating grabbing attitude information of a central point, and performing grabbing detection on the grabbing attitude information; the grabbing parameters comprise the grabbing width and the grabbing angle, and a grabbing rectangle of the pixel points is generated. According to the method, the problems that in the prior art, similar objects are not accurately grabbed, detected and recognized, and the position recognition precision of the grabbed rectangular frame is low are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of robot target object grasping methods, and in particular relates to a fine-grained grasping and detection method for intra-class packaged items. Background Art

[0002] With the rapid development of machine vision technology, robot grasping and detection has become a hot topic for manufacturing and transportation companies, and it has also become a major aspect of robotics research. At present, most of the grasping and detection tasks in the field of robot grasping and detection are for single objects. There are few studies on grasping and detection of multiple objects, and even fewer studies on grasping and detection of similar objects. However, in the process of sorting packaged products, there is a large demand for sorting similar objects, and it is highly dependent on manual labor. Therefore, fine-grained classification of packaged items has gradually become a research hotspot.

[0003] Fine-grained image classification has been a hot research topic in the fields of computer vision and pattern recognition in recent years. Its purpose is to perform more detailed sub-class division for coarse-grained objects within a large category, so as to identify objects within the category with similar features. However, due to the subtle inter-class differences and large intra-class differences between subcategories, fine-grained image classification is more difficult than ordinary image classification tasks. Traditional grasping and detection mainly targets inter-class objects, that is, objects with large feature differences, such as plastic bottles, paper boxes, packaging bags and other packaging materials with large shape differences; in actual sorting situations, more detection is needed for intra-class objects, that is, objects with similar features, such as beverages of similar shapes, snacks and other packaging products with similar features. Therefore, grasping and detection of similar packaged objects has become a research hotspot in the grasping process. Summary of the invention

[0004] The purpose of the present invention is to provide a fine-grained grasping detection method for packaged items within a category, which solves the problems of inaccurate grasping detection and recognition of similar objects and low position recognition accuracy of grasping rectangular frames in the prior art.

[0005] The technical solution adopted by the present invention is: a fine-grained grasping and detection method for packaged items within a category, including two parts: fine-grained target detection and target object grasping and detection; Fine-grained target detection: Mask R-CNN is used as the basic framework, and fine-grained features are extracted through self-supervised learning. Then, data enhancement is used for fine-grained feature learning. The FGM R-CNN (Fine-grained Mask R-CNN) target detection framework is constructed, and the FRP dataset is constructed. Training is performed within the target detection framework to classify the target objects within the class. Target object grasping and detection part: For the target detection object, the identified single image is used as input information and input into the FGM R-CNN target detection framework. The center point coordinates of the grasped object are calculated using the detection frame size. The RGB-D image of a single target detection object is input into the CNN optimization network framework to directly generate the grasping posture information of the center point, which includes the grasping width and grasping angle, thereby generating a grasping rectangle of the pixel point, performing grasping detection on the target detection object, and verifying the grasping detection effect.

[0006] The technical solution adopted by the present invention is also characterized in that: Furthermore, the fine-grained grasping and detection method for packaged items within a category is implemented in the following steps: Step 1: Based on the deep learning network model of Mask R-CNN, self-supervised learning and fine-grained enhancement are introduced to extract target features and perform fine-grained feature learning to construct a deep learning network FGM R-CNN suitable for fine-grained target detection, namely the FGM R-CNN target detection framework; Step 2: Use the target detection images to annotate the dataset images in a fine-grained format according to the annotation format of fine-grained classification, obtain the FRP (Fine-grained Retail Product) dataset, and train it within the target detection framework; Step 3: For the target detection object, input the object image into the FGM R-CNN target detection framework in step 1 and the CNN-based grasp detection optimization network to generate the grasping posture information of the center point, perform grasping detection on it, and conduct training and testing in combination with the Cornell dataset and the FRP dataset to verify the robustness and generalization ability of the grasping detection optimization method.

[0007] Furthermore, in step 1, the target features are extracted through the ResNet50 feature extraction network. During the feature extraction, each layer extractor except the first layer is input into the original RPN module. The RPN module determines the difference between the background and the object, extracts the area containing the target information and generates the candidate box. Finally, target feature candidate boxes of different sizes are obtained. Feature candidate boxes of different sizes are adjusted to a uniform size through ROIAlign to facilitate subsequent positioning and classification work.

[0008] Furthermore, in step 1, after obtaining the uniform size feature map of each object, it is cropped to obtain a local image of the object, and the cropped image is re-input into the ResNet50 feature extraction network as the input image of self-supervised learning to form a self-supervised learning module, and the weights during self-supervised learning are shared with the weights of feature extraction to achieve better stability and obtain the tiny features of the target; the identified fine-grained features are respectively input into the convolution layer for full connection and full convolution processing, and data enhancement operations are performed during fusion. After completion, the detection box, category prediction and mask output are performed respectively to obtain the FGM R-CNN target detection framework that can perform fine-grained image classification.

[0009] Furthermore, in step 2, the existing RPC retail product dataset is used as a standard to sample and organize common products, the COCO dataset format is used as a reference, and the Labelme software is used to perform fine-grained category annotation on the images. The number of images in the training set and the test set are allocated in a ratio of 5:1 to obtain the FRP dataset.

[0010] Furthermore, in step 3, the center point is calculated using the detected target detection frame coordinates. In order to reduce the amount of calculation and improve efficiency, the calculated coordinates are used as the grabbing coordinates. , as shown in formulas (1) and (2):

[0011] in,( ) is the coordinate value of the target detection box, is the target detection box points.

[0012] Further, in step 3, the coordinate values ​​obtained are As the grasping point, the grasping posture of the point is predicted; in the grasping detection process, the improved CNN is mainly processed through six convolutional layers; among them, the first three convolutional layers are combined with the pooling layer for feature extraction. For subsequent regression, the last three deconvolutional layers use Transpose to increase the size so that the output size is the same as the input size; thus outputting the grasping width And the grab angle , the expression of the grab box is shown in formula (3):

[0013] in, is the capture width corresponding to the pixel position, is the grabbing angle, and are the length and width of the end effector itself, is the target detection box points, R is the angle and grasping width set when n, W and H are determined, grasping angle Can be used as well as Two parameters are described. The value range is .

[0014] Furthermore, the equipment used in the detection method includes a 3D depth camera, a KUKA KR10 robotic arm, an industrial computer and communication equipment therebetween.

[0015] The beneficial effects of the present invention are: 1. The present invention proposes a fine-grained grasping and detection method for intra-class packaged items. In the detection part of fine-grained targets, the Mask R-CNN instance segmentation network is selected as the basic framework, and the feature map of a single item is used for self-supervised learning, and the minute features of the item are identified by the data enhancement method. This method pays more attention to the minute features of the item and further improves the accuracy of fine-grained classification. In the grasping and detection part of the target object, CNN is used as a feature extractor, and according to the results of the fine-grained target detection part, the target object image is convolved and deconvolved to quickly obtain the grasping detection frame of similar target objects.

[0016] 2. After training and testing the Cornell dataset and the FRP dataset, the results show that the method of the present invention can accurately predict the corresponding grasping boxes for similar objects of different sizes. Based on these advantages, the method proposed in the present invention has strong robustness and can be widely used in grasping and detecting tasks of various similar objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 The overall flow chart of the method of the present invention is as follows; Figure 2 A flow chart of the method of the present invention for detecting fine-grained targets within a class; Figure 3 This is an example diagram of the intra-class item dataset; Figure 4 This is a flow chart of the grasping point detection of the grasped object by the method of the present invention; Figure 5 The method of the present invention captures the heat map output of the angle and width of the detection node; Figure 6 This is a grasping detection prediction graph of a single object by the grasping network of the method of the present invention. DETAILED DESCRIPTION

[0018] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings.

[0019] The present invention provides a fine-grained grasping detection method for intra-class packaged items, including a new grasping detection network.

[0020] The grasping detection network proposed in the present invention can be divided into two stages, which include the first stage target detection and the second stage grasping point prediction. The purpose of the present invention is to solve the problems of target detection errors and low recognition for similar objects. The first stage target detection is based on the Mask R-CNN target detection network framework, and a target detection algorithm FGM R-CNN target detection framework combined with fine-grained classification is proposed. By collecting product packaging image samples, annotating their contours and feature information, and using data enhancement to find tiny features, self-supervised learning is used to perform cropping of different sizes and feature loops to train a deep learning model. The second stage grasping point detection uses the detection box of target detection to calculate the grasping point, inputs the image into the CNN network, uses convolution and deconvolution to predict the grasping posture, performs data enhancement on the product packaging image samples and the Cornell data set, and uses the trained grasping detection model to predict and obtain the corresponding grasping detection box.

[0021] The present invention proposes a fine-grained grasping and detection method for packaged items within a category, such as Figure 1 As shown, the method comprises the following steps: 1) Based on the deep learning model of Mask R-CNN, self-supervised learning and fine-grained enhancement are introduced to extract target features and perform fine-grained feature learning to construct a deep learning model FGMR-CNN (Fine-grained Mask R-CNN) suitable for fine-grained target detection, namely the FGM R-CNN target detection framework; 2) Using the target detection images, we annotate the dataset images in a fine-grained format according to the annotation format of fine-grained classification, and obtain the fine-grained retail product FRP (Fine-grained Retail Product) dataset; 3) For the target detection object, the object image is input into the FGM R-CNN target detection framework and the CNN-based grasp detection optimization network to generate the grasping posture information of the center point, and grasp detection is performed on it. The FGM R-CNN and the optimized CNN grasp detection network are combined with the Cornell dataset and the FRP dataset for training and testing to verify its generalization ability and robustness.

[0022] In step 1), the original image is input into the network framework, and the original feature map is extracted after CNN processing. Then, RPN is used to identify the background and object features to generate a bounding box, and ROIAlign is used to process the extracted feature map to obtain the initial feature map. The initial feature map is cropped regularly to obtain object detail images of the same size. The object detail images are input into the backbone extraction network to extract local features of the objects in the image and amplify the feature differences between subcategories. The FGM R-CNN algorithm can accurately distinguish the features between subcategories through deep learning methods.

[0023] In step 1), the feature map of the same size generated by ROIAlign is used as the raw data for self-supervised learning preprocessing fine-grained enhancement, and is cropped before being input into CNN. The cropped image will exclude interference from background factors and other target factors, and focus on the tiny features of the target for recognition. Based on the fine-grained features output by the FGM R-CNN target detection framework, the fully connected layer is used to generate bounding boxes and category classifications.

[0024] In step 2), the images in the dataset are selected to be objects with similar graspable shape features. 600 non-single-item images are selected from the RPC dataset, with pixels ranging from 400 to 2000 pixels. The images obtained contain grasped objects in the same class with similar sizes, angles, and shapes, so as to improve the generalization ability of the network model.

[0025] In step 3), the coordinate information of the detection frame of the target detection is output, where each target outputs four detection frame coordinates, and then select The coordinates of each point are used as input, and the coordinates of the grasping center are calculated using a geometric algorithm.

[0026] In step 3), the RGB-D image of the object is input into the grasping detection neural network according to the grasping center coordinates. The grasping detection network improved based on CNN extracts the features near the grasping point through convolution and predicts a candidate box containing the grasping point features. The candidate box contains the grasping width. And the grab angle .

[0027] The composition of the present invention is further described below by way of examples and drawings.

[0028] Example 1 like Figure 1 As shown, this embodiment provides a fine-grained grasping detection method for packaged items within a category, specifically a grasping posture generation method for items within a category with similar shape features based on the combination of target detection and grasping detection methods, including the following steps: Step 1: Based on the deep learning network model of Mask R-CNN, self-supervised learning and fine-grained enhancement are introduced to extract target features and perform fine-grained feature learning to construct a deep learning network FGM R-CNN suitable for fine-grained target detection, namely the FGM R-CNN target detection framework; Step 2: Based on the pictures of objects in the same class with similar shape features, the contours of the target objects in the pictures are marked to obtain the FRP dataset and trained in the object detection network; In step 3, for the target detection object, the object image is input into the FGM R-CNN target detection framework in step 1 and the CNN-based grasp detection optimization network to generate the grasping posture information of the center point, perform grasping detection on it, and conduct training and testing in combination with the Cornell dataset and the FRP dataset to verify the robustness and generalization ability of the grasping detection optimization method.

[0029] In actual operation, the equipment needed includes 3D depth camera, KUKA KR10 robotic arm, industrial computer and the communication equipment between them.

[0030] Example 2 This embodiment provides a fine-grained grasping and detection method for packaged items within a category, which is specifically implemented in the following steps: Step 1: Based on the deep learning network model of Mask R-CNN, self-supervised learning and fine-grained enhancement are introduced to extract target features and perform fine-grained feature learning to construct a deep learning network FGM R-CNN suitable for fine-grained target detection, namely the FGM R-CNN target detection framework; Step 2: Using the target detection images, annotate the dataset images in a fine-grained format according to the annotation format of fine-grained classification to obtain the FRP dataset, and train it in the target detection network; Step 3: For the target detection object, input the object image into the FGM R-CNN target detection framework in step 1 and the CNN-based grasp detection optimization network to generate the grasping posture information of the center point, perform grasping detection on it, and conduct training and testing in combination with the Cornell dataset and the FRP dataset to verify the robustness and generalization ability of the grasping detection optimization method.

[0031] In actual operation, the equipment needed includes 3D depth camera, KUKA KR10 robotic arm, industrial computer and the communication equipment between them.

[0032] Example 3 This embodiment provides a fine-grained grasping and detection method for packaged items within a category, which is specifically implemented in the following steps: Step 1: Based on the deep learning network model of Mask R-CNN, self-supervised learning and fine-grained enhancement are introduced to extract target features and perform fine-grained feature learning to construct a deep learning network FGM R-CNN suitable for fine-grained target detection, namely the FGM R-CNN target detection framework; The target features are extracted through the ResNet50 feature extraction network. During the feature extraction, each layer extractor except the first layer is input into the RPN module. The RPN module determines the difference between the background and the object, extracts the area that may contain the target information and generates the candidate box. Finally, the target feature candidate box diagrams of different sizes are obtained. The feature candidate box diagrams of different sizes are adjusted to a uniform size through ROIAlign to facilitate the subsequent positioning and classification work; After obtaining the uniform size feature map of each object, it is cropped to obtain the local image of the object. The cropped image is re-input into the ResNet50 feature extraction network as the input image of self-supervised learning to form a self-supervised learning module. The weights of self-supervised learning and feature extraction are shared to achieve better stability and obtain the tiny features of the target; the identified fine-grained features are input into the convolution layer for full connection and full convolution processing, and data enhancement operations are performed during fusion. After completion, the detection box, category prediction and mask output are performed separately to obtain the FGM R-CNN target detection framework that can perform fine-grained image classification; Step 2: Using the target detection images, annotate the dataset images in a fine-grained format according to the annotation format of fine-grained classification to obtain the FRP dataset, and train it in the target detection network; Step 3: For the target detection object, input the object image into the FGM R-CNN target detection framework in step 1 and the CNN-based grasp detection optimization network to generate the grasping posture information of the center point, perform grasping detection on it, and conduct training and testing in combination with the Cornell dataset and the FRP dataset to verify the robustness and generalization ability of the grasping detection optimization method.

[0033] In actual operation, the equipment needed includes 3D depth camera, KUKA KR10 robotic arm, industrial computer and the communication equipment between them.

[0034] Example 4 This embodiment provides a fine-grained grasping and detection method for packaged items within a category, which is specifically implemented in the following steps: Step 1: Based on the deep learning network model of Mask R-CNN, self-supervised learning and fine-grained enhancement are introduced to extract target features and perform fine-grained feature learning to construct a deep learning network FGM R-CNN suitable for fine-grained target detection, namely the FGM R-CNN target detection framework; The target features are extracted through the ResNet50 feature extraction network. During the feature extraction, each layer extractor except the first layer is input into the RPN module. The RPN module determines the difference between the background and the object, extracts the area that may contain the target information and generates the candidate box. Finally, the target feature candidate box diagrams of different sizes are obtained. The feature candidate box diagrams of different sizes are adjusted to a uniform size through ROIAlign to facilitate the subsequent positioning and classification work; After obtaining the uniform size feature map of each object, it is cropped to obtain the local image of the object. The cropped image is re-input into the ResNet50 feature extraction network as the input image of self-supervised learning to form a self-supervised learning module. The weights of self-supervised learning and feature extraction are shared to achieve better stability and obtain the tiny features of the target; the identified fine-grained features are input into the convolution layer for full connection and full convolution processing, and data enhancement operations are performed during fusion. After completion, the detection box, category prediction and mask output are performed separately to obtain the FGM R-CNN target detection framework that can perform fine-grained image classification; Step 2: Using the target detection images, annotate the dataset images in a fine-grained format according to the annotation format of fine-grained classification to obtain the FRP dataset, and train it in the target detection network; Using the existing RPC retail product dataset as a standard, we sampled and sorted common products, used the COCO dataset format as a reference, used Labelme software to annotate images with fine-grained categories, and allocated the number of images in the training set and test set in a ratio of 5:1 to obtain the FRP dataset. Step 3: For the target detection object, input the object image into the FGM R-CNN target detection framework in step 1 and the CNN-based grasp detection optimization network to generate the grasping posture information of the center point, perform grasping detection on it, and conduct training and testing in combination with the Cornell dataset and the FRP dataset to verify the robustness and generalization ability of the grasping detection optimization method.

[0035] In actual operation, the equipment needed includes 3D depth camera, KUKA KR10 robotic arm, industrial computer and the communication equipment between them.

[0036] Example 5 This embodiment provides a fine-grained grasping and detection method for packaged items within a category, which is specifically implemented in the following steps: Step 1: Based on the deep learning network model of Mask R-CNN, self-supervised learning and fine-grained enhancement are introduced to extract target features and perform fine-grained feature learning to construct a deep learning network FGM R-CNN suitable for fine-grained target detection, namely the FGM R-CNN target detection framework; The target features are extracted through the ResNet50 feature extraction network. During the feature extraction, each layer extractor except the first layer is input into the RPN module. The RPN module determines the difference between the background and the object, extracts the area that may contain the target information and generates the candidate box. Finally, the target feature candidate box diagrams of different sizes are obtained. The feature candidate box diagrams of different sizes are adjusted to a uniform size through ROIAlign to facilitate the subsequent positioning and classification work; After obtaining the uniform size feature map of each object, it is cropped to obtain the local image of the object. The cropped image is re-input into the ResNet50 feature extraction network as the input image of self-supervised learning to form a self-supervised learning module. The weights of self-supervised learning and feature extraction are shared to achieve better stability and obtain the tiny features of the target; the identified fine-grained features are input into the convolution layer for full connection and full convolution processing, and data enhancement operations are performed during fusion. After completion, the detection box, category prediction and mask output are performed separately to obtain the FGM R-CNN target detection framework that can perform fine-grained image classification; Step 2: Using the target detection images, annotate the dataset images in a fine-grained format according to the annotation format of fine-grained classification to obtain the FRP dataset, and train it in the target detection network; Using the existing RPC retail product dataset as a standard, we sampled and sorted common products, used the COCO dataset format as a reference, used Labelme software to annotate images with fine-grained categories, and allocated the number of images in the training set and test set in a ratio of 5:1 to obtain the FRP dataset. Step 3: For the target detection object, input the object image into the FGM R-CNN target detection framework and the CNN-based grasp detection optimization network in step 1 to generate the grasping posture information of the center point, perform grasping detection on it, and conduct training and testing in combination with the Cornell dataset and the FRP dataset to verify the robustness and generalization ability of the grasping detection optimization method; The center point is calculated using the detected target detection frame coordinates. In order to reduce the amount of calculation and improve efficiency, the calculated coordinates are used as the grabbing coordinates. , as shown in formulas (1) and (2):

[0037] in,( ) is the coordinate value of the target detection box, is the target detection box points; The coordinate value obtained As the grasping point, the grasping posture of the point is predicted; in the grasping detection process, the improved CNN is mainly processed through six convolutional layers; among them, the first three convolutional layers are combined with the pooling layer for feature extraction. For subsequent regression, the last three deconvolutional layers use Transpose to increase the size so that the output size is the same as the input size; thus outputting the grasping width And the grab angle , the expression of the grab box is shown in formula (3):

[0038] in, is the capture width corresponding to the pixel position, is the grabbing angle, and are the length and width of the end effector itself, is the target detection box points, R is the angle and grasping width set when n, W and H are determined, grasping angle Can be used as well as Two parameters are described. The value range is .

[0039] In actual operation, the equipment needed includes 3D depth camera, KUKA KR10 robotic arm, industrial computer and the communication equipment between them.

[0040] Example 6 like Figure 1 As shown, this embodiment provides a fine-grained grasping detection method for packaged items within a category, specifically a grasping posture generation method for items within a category with similar shape features based on the combination of target detection and grasping detection methods, including the following steps: Step 1: Based on the RGB information of the target object, FGM R-CNN uses a deep learning neural network to perform fine-grained classification of the target object to obtain the position and posture information of the target object; Step 2: Based on the pictures of objects in the same class with similar shape features, the contours of the target objects in the pictures are marked to obtain the FRP dataset and trained in the object detection network; Step 3: Based on the RGB-D information of the target object obtained by the camera and the pose information of the target object extracted by the neural network, the target object’s grasping detection frame is obtained by calculating and extracting features through the optimized CNN.

[0041] In the actual operation of this embodiment, the equipment required includes a 3D depth camera, a KUKA KR10 robotic arm, an industrial computer and communication equipment therebetween.

[0042] The above specific steps of this embodiment are: In step 1, the 3D camera carried by the robotic arm grasping system obtains point cloud information of the object and the environment. The camera can be fixed at a position outside the robotic arm body, or fixed next to the end effector of the robotic arm and move with it.

[0043] The 3D camera of this embodiment is mounted above the workbench, and can obtain RGB-D image feature information of the target object after initialization.

[0044] In step 1, the input image passes through ResNet50 and then enters RPN to generate 3×3 anchor boxes. The anchor boxes recognize the image and classify the objects using the softmax matrix. The difference between the background and the object is determined for binary classification. Subsequently, RPN uses bounding box regression to fine-tune the anchor box so that the positive anchor is closer to the real box. Then, the generated ROI and the probability that the ROI contains the object are output. The specific process is as follows: Figure 2 As shown in the RPN module.

[0045] In step 1, Figure 2 The function of ROIAlign in is to adjust feature maps of different sizes to a uniform size. The specific operation is as follows Figure 3 As shown in the figure, the input feature map is divided into sections of size H×W, and four areas are sampled for each section. The red × in the figure represents the center position; bilinear interpolation is performed on the values ​​of the four centers of each section, and max pooling is performed on the values ​​calculated for each section to obtain the result output of the required size.

[0046] In step 1, the fully connected layer passes the output of the convolution activation function through a filter to obtain the output of a neuron. At the end of the traversal, a layer of neurons is obtained. The neurons in each layer are connected to all the neurons in the previous layer. Through the connection, the detailed features of the object can be obtained and these features can be combined.

[0047] In step 2, during the annotation process, the image is divided into three different objects: large, medium, and small, and annotated separately, such as Figure 3 As shown, (a-1) to (c-1) are the placement situations of the target object larger than 96×96, the target object between 32×32 and 96×96, and the target object smaller than 32×32, respectively, and (d-1) is the mixed placement of the three objects; Figure 3 The figure shows the marked outline. If there is any occlusion in the figure, the occluded part will not be marked.

[0048] In step 2, the labeling results show that when the packages are randomly stacked, it is possible to effectively identify and label different categories of items, such as Figure 3 As shown in the figure; during the labeling process, rectangular frames are used to identify and label regular objects, while for irregular objects, such as bottles, personalized packaging bags and other packaging products, polygons can be used to cover and label the contours; this method can determine and identify occlusion and stratification, and label the full contour of the top-level object, which is convenient for the next step of layered grasping process.

[0049] In step 3, if Figure 4 As shown in the figure, the object image detected by the target is converted into 300×300 pixels as input, and CNN is used to generate a captured image based on a known depth map. First, OpenCV is used to preprocess the image and fill in invalid depth values ​​to reduce the impact of the background. The convolution layer is used to extract features to improve the accuracy of the segmentation task. After convolution is used for feature extraction, ROIAlign is used for pooling to reduce the amount of model calculation and improve the robustness of the model when processing images. The output image size is then adjusted to the same as the input image by replacing the connection layer with a deconvolution layer.

[0050] In step 3, the description of the angle is that the grasping angle is symmetrical. The range is In the deep learning network, the grasping angle is not predicted directly, but encoded into two components: and , these two variables can ensure the continuity of the angle And provide a unique angle value in it , the final grasping angle output is ; The heat map of the angle and width output of the grab detection node is as follows Figure 5 shown.

[0051] In step 3, the Cornell grabbing dataset and the FRP dataset are used for mixed training. A part of the Cornell dataset is randomly extracted without duplication, and the FRP dataset is used as the remaining part for mixed training to maximize the high robustness of the network model. The generalization ability and accuracy of the grab detection box data output are guaranteed. The accurate grab detection results are as follows: Figure 6 shown.

Claims

1. A fine-grained grasping and detection method for packaged items within a category, characterized in that: It includes two parts: fine-grained target detection and target object grasping detection; Fine-grained target detection: Mask R-CNN is used as the basic framework, and fine-grained features are extracted through self-supervised learning. Then, data enhancement is used for fine-grained feature learning. The FGM R-CNN target detection framework is constructed, and the FRP dataset is constructed. The training is performed within the target detection framework to classify the target objects within the class. Target object grasping and detection part: For the target detection object, the identified single image is used as input information and input into the FGM R-CNN target detection framework. The center point coordinates of the grasped object are calculated using the detection frame size. The RGB-D image of a single target detection object is input into the CNN optimization network framework to directly generate the grasping posture information of the center point, which includes the grasping width and grasping angle, thereby generating a grasping rectangle of the pixel point, performing grasping detection on the target detection object, and verifying the grasping detection effect.

2. The fine-grained grasping and detection method for inner-packaged items according to claim 1 is characterized in that: Please follow the steps below to implement: Step 1: Based on the deep learning network model of Mask R-CNN, self-supervised learning and fine-grained enhancement are introduced to extract target features and perform fine-grained feature learning to construct a deep learning network FGM R-CNN suitable for fine-grained target detection, namely the FGM R-CNN target detection framework; Step 2: Using the target detection images, we annotate the dataset images in a fine-grained format according to the annotation format of fine-grained classification to obtain the FRP dataset, and train it in the target detection framework. Step 3: For the target detection object, input the object image into the FGM R-CNN target detection framework in step 1 and the CNN-based grasp detection optimization network to generate the grasping posture information of the center point, perform grasping detection on it, and conduct training and testing in combination with the Cornell dataset and the FRP dataset to verify the robustness and generalization ability of the grasping detection optimization method.

3. The fine-grained grasping and detection method for inner-packaged items according to claim 2 is characterized in that: In the step 1, the target features are extracted by the ResNet50 feature extraction network. During the feature extraction, each layer extractor except the first layer is input into the original RPN module. The RPN module determines the difference between the background and the object, extracts the area containing the target information and generates a candidate frame, and finally obtains target feature candidate frame diagrams of different sizes. The feature candidate frame diagrams of different sizes are adjusted to a uniform size through ROIAlign to facilitate subsequent positioning and classification work.

4. The fine-grained grasping and detection method for inner-packaged items according to claim 3 is characterized in that: In the step 1, after obtaining the uniform size feature map of each object, the uniform size feature map is cropped to obtain a local image of the object, and the cropped image is re-input into the ResNet50 feature extraction network as an input image for self-supervised learning to form a self-supervised learning module, and the weights during self-supervised learning are shared with the weights of feature extraction to achieve better stability and obtain the tiny features of the target; The identified fine-grained features are input into the convolutional layers for full connection and full convolution processing, and data enhancement operations are performed during fusion. After completion, the detection box, category prediction and mask output are performed separately to obtain the FGM R-CNN target detection framework that can perform fine-grained image classification.

5. The fine-grained grasping and detection method for inner-packaged items according to claim 2 is characterized in that: In step 2, the existing RPC retail commodity dataset is used as a standard to sample and organize common commodities, the COCO dataset format is used as a reference, the Labelme software is used to perform fine-grained category annotation on the images, and the number of images in the training set and the test set are allocated in a ratio of 5:1 to obtain the FRP dataset.

6. The fine-grained grasping and detection method for inner-packaged items according to claim 2 is characterized in that: In step 3, the center point is calculated using the detected target detection frame coordinates. In order to reduce the amount of calculation and improve efficiency, the calculated coordinates are used as the grabbing coordinates. , as shown in formulas (1) and (2): in,( ) is the coordinate value of the target detection box, is the target detection box points.

7. The fine-grained grasping and detection method for inner-packaged items according to claim 6 is characterized in that: In step 3, the coordinate values ​​obtained are As the grasping point, the grasping posture of the point is predicted; in the grasping detection process, the improved CNN is mainly processed through six convolutional layers; among them, the first three convolutional layers are combined with the pooling layer for feature extraction. For subsequent regression, the last three deconvolutional layers use Transpose to increase the size so that the output size is the same as the input size; thus outputting the grasping width And the grab angle , the expression of the grab box is shown in formula (3): in, is the capture width corresponding to the pixel position, is the grabbing angle, and are the length and width of the end effector itself, is the target detection box points, R is the angle and grasping width set when n, W and H are determined, grasping angle Can be used as well as Two parameters are described. The value range is .

8. The fine-grained grasping and detection method for inner-packaged items according to any one of claims 1 to 7, characterized in that: The equipment used in the detection method includes a 3D depth camera, a KUKA KR10 robotic arm, an industrial computer and communication equipment therebetween.

Images