A multi-object grasp prediction method based on improved Cascade R-CNN

Abstract

The present invention discloses a multi-object grasp prediction method based on an improved Cascade R-CNN, comprising the following steps: 1: fixing a camera to the end of a robotic arm for hand-eye calibration and camera intrinsic parameter calibration; 2: capturing images containing multiple objects through a camera to form an original image dataset, drawing a grasping frame for the original image and performing expansion processing to obtain a multi-object grasping image dataset; 3: constructing an improved Cascade R-CNN network and training the network using the multi-object grasping image dataset; 4: acquiring an RGB image through a camera and inputting it into the improved Cascade R-CNN network to predict a feasible grasping frame for each object in the image; and 5: using the robotic arm to determine the object's position and grasp it based on the predicted feasible grasping frame. The method can quickly and accurately obtain the grasping posture of each object in a multi-object scene, thereby achieving high-accuracy grasping of unknown objects.

Description

Technical Field

[0001] The present invention belongs to the technical field of image detection and robot grasping, and relates to a multi-object grasping prediction method based on an improved Cascade R-CNN. Background Art

[0002] For the robot's planar grasping task, the first thing that needs to be calculated is the pose of the given object. Faced with objects of different types, shapes, and colors, the objects need to be grasped and labeled. Using a learning method to allow the robot to simulate human grasping of objects is an important research. Cornell University first proposed a grasping dataset (Cornell GraspDataset) based on a learning method, which uses five parameters to express the grasping pose and annotates a rotating grasping frame for each object in the image. It also achieved good performance in actual grasping experiments. This research is called grasp detection. Many existing implementation solutions for object grasping are mainly aimed at grasping a single object, and there is little research on grasping multiple objects. Therefore, there is an urgent need for a multi-object grasping method to solve the problem of robot grasping and classification of multiple objects in complex environments, improve the interaction ability of collaborative robots with the environment, and complete precise grasping. Summary of the Invention

[0003] To solve the above technical problems, the present invention provides a multi-object grasping prediction method based on an improved Cascade R-CNN, which can achieve accurate grasping detection of multiple objects in multi-object scenes.

[0004] The present invention provides a multi-object grasp prediction method based on an improved Cascade R-CNN, comprising:

[0005] Step 1: Fix the camera at the end of the robotic arm to perform hand-eye calibration and camera intrinsic parameter calibration;

[0006] Step 2: Use the camera to capture images containing multiple objects to form an original image dataset. Draw a capture box for the original image and perform expansion processing to obtain a multi-target capture image dataset.

[0007] Step 3: Build an improved Cascade R-CNN network and train it using a multi-object capture image dataset.

[0008] Step 4: Obtain an RGB image through the camera and input it into the improved Cascade R-CNN network to predict feasible grasping boxes for each object in the image;

[0009] Step 5: The robotic arm determines the location of the object and grabs it based on the predicted feasible grabbing frame.

[0010] In the multi-object grasping method based on a multi-stage neural network of the present invention, the hand-eye calibration in step 1 is specifically as follows:

[0011] 1) Print a calibration plate and stick it on a flat surface;

[0012] 2) Move the robotic arm to a certain position and record the posture of the end of the robotic arm at the current position;

[0013] 3) Use the camera on the robotic arm to capture the position of the calibration plate in the camera and record it;

[0014] 4) The pose X of the manipulator and the pose of the calibration plate captured by the camera when the manipulator is in pose X form a set of pose information;

[0015] 5) Repeat 2) and 3) until more than 17 sets of posture information are collected;

[0016] 6) Calculate based on the collected multiple sets of posture information to obtain the relationship transformation matrix.

[0017] In the multi-object grasping method based on a multi-stage neural network of the present invention, the camera intrinsic parameter calibration in step 1 is specifically as follows:

[0018] 1) Print a chessboard and stick it on a flat surface as a calibration object;

[0019] 2) Take photos of the calibration object in different orientations by adjusting the orientation of the calibration object or the camera.

[0020] 3) Extracting checkerboard corners from the photo;

[0021] 4) Estimate five intrinsic parameters and six extrinsic parameters for an ideal distortion-free scenario;

[0022] 5) Apply the least squares method to estimate the distortion coefficient under actual radial distortion;

[0023] 6) According to the distortion coefficient under radial distortion, the maximum likelihood method is used to perform optimization estimation to obtain the camera's intrinsic parameter matrix, radial distortion, and tangential distortion.

[0024] In the multi-object grasping method using a multi-stage neural network of the present invention, step 2 includes:

[0025] Step 2.1: Select 32 categories of objects, place different combinations of objects in different postures, and use the camera at the end of the robotic arm to capture original images to form the original image dataset;

[0026] Step 2.2: Use LabMel software to annotate the grab box for each object in the original image, so that each image has a grab label, including the position parameters and angle classification of the grab box;

[0027] Step 2.3: First, crop the original 640×480 image from the center to the surrounding square to obtain a 351×351 image;

[0028] Step 2.4: Then expand the image by 75 pixels in all directions using the replicate mode in OpenCV to obtain a 501×501 image.

[0029] Step 2.5: Shift the pixels horizontally and vertically, perform five random rotations within 180°, and finally resize the image to obtain multiple 320×320 images, thereby achieving a 100-fold expansion relative to the original dataset.

[0030] In the multi-object grasping method using a multi-stage neural network of the present invention, the position parameters in step 2.2 include: the height, width, center of mass coordinates of the grasping box, and the grasping angle, where the grasping angle is the angle between the height direction of the grasping box and the horizontal axis of the image;

[0031] The angle classification is as follows: the angle between the grabbing frame of each object and the horizontal axis of the image is divided into 18 categories within the range of 0° to 180°.

[0032] In the multi-object grasping method based on a multi-stage neural network of the present invention, the improved CascadeR-CNN network in step 3 includes: a ResFPNet feature extraction network, a region proposal network and a cascade detection network; the ResFPNet feature extraction network is used to perform feature extraction on the input RGB image to generate a multi-scale fused feature map; the multi-scale fused feature map is input into the region proposal network to generate a candidate grasping frame; the candidate grasping frame is projected onto the multi-scale fused feature map and input into the cascade detection network; the cascade detection network is used to perform angle classification evaluation and candidate grasping frame prediction, and finally extract the optimal detection grasping frame.

[0033] In the multi-object grasping method based on a multi-stage neural network of the present invention, the ResFPNet feature extraction network adds a spatial pyramid pooling module with dilated convolution on the basis of the ResNeXt network. During the feature fusion process, the spatial pyramid pooling module with dilated convolution is used to upsample the shallow feature map, and then fuse it with the deep feature map before upsampling.

[0034] In the multi-object grasping method based on a multi-stage neural network of the present invention, the cascade detection network includes three detectors, and the detector of each branch includes: a pooling layer, a fully connected layer and a convolutional layer. The pooling layer is used to process the feature map into a fixed size, the fully connected layer is used to perform angle classification prediction, and the convolutional layer is used to predict the candidate grasping box. The candidate grasping box output by the convolutional layer of the previous detector is projected onto the multi-scale fusion feature map and then input into the next detector for angle classification prediction and candidate grasping box prediction.

[0035] In the multi-object grasping method using a multi-stage neural network of the present invention, the loss function for angle classification prediction in the improved Cascade R-CNN network is cross entropy, which is expressed as follows:

[0036]

[0037] Among them, θ i It represents the angle classification label value. It represents the predicted value of angle classification;

[0038] The loss function for the position prediction of the grab box is expressed as follows using square error:

[0039]

[0040] Among them, x i ,y i It represents the coordinate label value of the center of mass of the grab box. It represents the predicted value of the center of mass coordinate of the grab box; h i It represents the value of the grab box height label. It represents the predicted value of the grab box height; w i It represents the value of the grab box width tag. It represents the predicted value of the grab box width.

[0041] The present invention provides a multi-object grasping prediction method based on an improved Cascade R-CNN, and proposes an improved Cascade R-CNN network. The network is composed of a ResFPNet feature extraction network, a region proposal network, and a cascade detection network. The ResFPNet feature extraction network adds a spatial pyramid pooling module with a dilated convolution on the basis of the ResNeXt network. When performing feature fusion, the shallow feature map is upsampled and then fused with the deep feature map before upsampling, which can reduce feature loss. The cascade detection network includes three detectors, each of which includes two branches: a fully connected layer and a convolutional layer, and can simultaneously realize angle classification prediction and candidate grasping box prediction. The method of the present invention can quickly and accurately obtain the grasping posture of each object in a multi-object scene, thereby realizing high-accuracy grasping of unknown objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flowchart of a multi-object grasp prediction method based on an improved Cascade R-CNN of the present invention;

[0043] Figure 2 This is the improved Cascade R-CNN network structure diagram of the present invention. DETAILED DESCRIPTION

[0044] like Figure 1 As shown, the present invention provides a multi-object grasp prediction method based on an improved Cascade R-CNN, comprising:

[0045] Step 1: Fix the camera at the end of the robotic arm to perform hand-eye calibration and camera intrinsic parameter calibration;

[0046] In specific implementation, the hand-eye calibration is as follows:

[0047] 1) Print a calibration plate and stick it on a flat surface;

[0048] 2) Move the robotic arm to a certain position and record the posture of the end of the robotic arm at the current position;

[0049] 3) Use the camera on the robotic arm to capture the position of the calibration plate in the camera and record it;

[0050] 4) The pose X of the manipulator and the pose of the calibration plate captured by the camera when the manipulator is in pose X form a set of pose information;

[0051] 5) Repeat 2) and 3) until more than 17 sets of posture information are collected;

[0052] 6) Calculate based on the collected multiple sets of posture information to obtain the relationship transformation matrix.

[0053] In specific implementation, the camera internal parameter calibration is specifically as follows:

[0054] 1) Print a chessboard and stick it on a flat surface as a calibration object;

[0055] 2) Take photos of the calibration object in different orientations by adjusting the orientation of the calibration object or the camera.

[0056] 3) Extracting checkerboard corners from the photo;

[0057] 4) Estimate five intrinsic parameters and six extrinsic parameters for an ideal distortion-free scenario;

[0058] 5) Apply the least squares method to estimate the distortion coefficient under actual radial distortion;

[0059] 6) According to the distortion coefficient under radial distortion, the maximum likelihood method is used to perform optimization estimation to obtain the camera's intrinsic parameter matrix, radial distortion, and tangential distortion.

[0060] Step 2: Use a camera to capture images containing multiple objects to form an original image dataset, draw a capture frame for the original image and perform expansion processing to obtain a multi-target capture image dataset. Step 2 includes:

[0061] Step 2.1: Select 32 categories of objects, place different combinations of objects in different postures, and use the camera at the end of the robotic arm to capture original images to form the original image dataset;

[0062] In practice, we collected 32 common categories of objects in daily life. Within each category, we selected multiple items of varying styles. Each image could contain one or more objects, with the objects positioned in different positions across the image. A total of 700 images were collected.

[0063] Step 2.2: Use LabMel software to annotate the grab box for each object in the original image, so that each image has a grab label, including the position parameters and angle classification of the grab box;

[0064] The position parameters include: the height, width, centroid coordinates and grab angle of the grab box, where the grab angle is the angle between the height direction of the grab box and the horizontal axis of the image;

[0065] Angle classification is as follows: the angle between the grabbing frame of each object and the horizontal axis of the image is divided into 18 categories within the range of 0° to 180°.

[0066] Step 2.3: First, crop the original 640×480 image from the center to the surrounding square to obtain a 351×351 image;

[0067] Step 2.4: Then expand the image by 75 pixels in all directions using the replicate mode in OpenCV to obtain a 501×501 image.

[0068] Step 2.5: Shift the pixels horizontally and vertically, perform five random rotations within 180°, and finally resize the image to obtain multiple 320×320 images, thereby achieving a 100-fold expansion relative to the original dataset.

[0069] Due to the limitations of image sample types and quantity as well as time, the amount of data produced does not meet the standards of large-scale datasets, which can easily lead to underfitting of the model and the inability of the network to learn effective features. Therefore, before network training, the dataset needs to be enhanced. The data enhancement method is basically the same as that of single-target grasping detection. At the same time, it is necessary to consider that the field of view of multiple targets in the image accounts for a larger proportion than that of a single target. In the case of rotation and translation, the object may exceed the boundary and the corresponding grasping box is invalid. Therefore, before this, the image needs to be resized and padded to ensure that the object does not deform while keeping the object in the field of view as much as possible. The invalid grasping box after enhancement is removed. The collected image size is 640×480, and the enhanced image size is 320×320. Finally, the generated image data is imported into a json file in accordance with the COCO target detection dataset format. 45,000 training set images and 15,000 test set images are generated. Each image contains the position parameters of the grasping box, angle classification and item category information.

[0070] Step 3: Build an improved Cascade R-CNN network and train it using a multi-object capture image dataset.

[0071] like Figure 2 The figure shows an improved Cascade R-CNN network, comprising a ResFPNet feature extraction network, a Region Proposal Network (RPN), and a cascaded detection network. The ResFPNet feature extraction network extracts features from the input RGB image and generates a multi-scale fused feature map. This multi-scale fused feature map is fed into the RPN to generate candidate bounding boxes. The candidate bounding boxes are projected onto the multi-scale fused feature map and fed into the cascaded detection network. The cascaded detection network evaluates angle classification and predicts candidate bounding boxes, ultimately extracting the optimal detection bounding box.

[0072] The ResFPNet feature extraction network builds on the ResNeXt network by adding a spatial pyramid pooling module with dilated convolutions. During feature fusion, this module is used to upsample shallow feature maps before fusing them with the pre-upsampled deep feature maps to minimize feature loss. Compared to standard convolution, dilated convolution has an additional parameter, the dilation rate, which controls the size of the receptive field. A larger rate increases the receptive field.

[0073] The spatial pyramid pooling module with dilated convolution can not only expand the receptive field, but also does not affect the resolution of the image and does not increase the amount of additional calculations. Therefore, in view of the differences in the receptive fields corresponding to targets of different scales, the present invention connects three dilated convolutions with different dilation rates in parallel, and then upsamples them, so that information of different ranges and sizes around the target can be obtained. Different dilated convolutions have different ranges, so after convolution, features of different ranges are retained, effectively reducing feature loss. The spatial pyramid pooling module with dilated convolution does not change the size of the feature map, and the receptive field range is controlled accordingly, which is conducive to extracting multi-scale information.

[0074] The cascade detection network includes three detectors. The detector of each branch includes: a pooling layer, a fully connected layer and a convolutional layer. The pooling layer is used to process the feature map into a fixed size, the fully connected layer is used to perform angle classification prediction, and the convolutional layer is used to predict the candidate grasping box. The candidate grasping box output by the convolutional layer of the previous detector is projected onto the multi-scale fusion feature map and then input into the next detector for angle classification prediction and candidate grasping box prediction.

[0075] The loss function for angle classification prediction in the improved Cascade R-CNN network is cross entropy, which is expressed as follows:

[0076]

[0077] Among them, θ i It represents the angle classification label value. It represents the predicted value of angle classification;

[0078] The loss function for the position prediction of the grab box is expressed as follows using square error:

[0079]

[0080] Among them, x i ,y i It represents the coordinate label value of the center of mass of the grab box. It represents the predicted value of the center of mass coordinate of the grab box; h i It represents the value of the grab box height label. It represents the predicted value of the grab box height; w i It represents the value of the grab box width tag. It represents the predicted value of the grab box width.

[0081] Step 4: Obtain an RGB image through the camera and input it into the improved Cascade R-CNN network to predict feasible grasping boxes for each object in the image;

[0082] Step 5: The robotic arm determines the location of the object and grabs it based on the predicted feasible grabbing frame.

[0083] The above description is only a preferred embodiment of the present invention and is not intended to limit the concept of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-object grasp prediction method based on improved Cascade R-CNN, characterized by: include: Step 1: Fix the camera at the end of the robotic arm to perform hand-eye calibration and camera intrinsic parameter calibration; Step 2: Use the camera to capture images containing multiple objects to form an original image dataset. Draw a capture box for the original image and perform expansion processing to obtain a multi-target capture image dataset. Step 3: Build an improved Cascade R-CNN network and train it using a multi-object capture image dataset. Step 4: Obtain an RGB image through the camera and input it into the improved Cascade R-CNN network to predict feasible grasping boxes for each object in the image; Step 5: The robotic arm determines the object's location and grabs it based on the predicted feasible grabbing frame; The improved Cascade R-CNN network in step 3 includes: a ResFPNet feature extraction network, a region proposal network, and a cascade detection network; the ResFPNet feature extraction network is used to extract features from the input RGB image and generate a multi-scale fused feature map; the multi-scale fused feature map is input into the region proposal network to generate a candidate grab box; the candidate grab box is projected onto the multi-scale fused feature map and input into the cascade detection network; the cascade detection network is used to perform angle classification prediction and candidate grab box prediction, and finally extract the optimal detection grab box; The ResFPNet feature extraction network adds a spatial pyramid pooling module with dilated convolutions to the ResNeXt network. During the feature fusion process, the spatial pyramid pooling module with dilated convolutions is used to upsample the shallow feature maps and then fuse them with the deep feature maps before upsampling. The cascade detection network includes three detectors. The detector of each branch includes: a pooling layer, a fully connected layer and a convolutional layer. The pooling layer is used to process the feature map into a fixed size, the fully connected layer is used to perform angle classification prediction, and the convolutional layer is used to predict the candidate grasping box. The candidate grasping box output by the convolutional layer of the previous detector is projected onto the multi-scale fusion feature map and then input into the next detector for angle classification prediction and candidate grasping box prediction.

2. The multi-object grasp prediction method based on the improved Cascade R-CNN according to claim 1, characterized in that: The hand-eye calibration in step 1 is specifically as follows: 1) Print a calibration plate and stick it on a flat surface; 2) Move the robotic arm to a certain position and record the posture of the end of the robotic arm at the current position; 3) Use the camera on the robotic arm to capture the position of the calibration plate in the camera and record it; 4) The pose X of the manipulator and the pose of the calibration plate captured by the camera when the manipulator is in pose X form a set of pose information; 5) Repeat 2) and 3) until more than 17 sets of posture information are collected; 6) Calculate based on the collected multiple sets of posture information to obtain the relationship transformation matrix.

3. The multi-object grasp prediction method based on the improved Cascade R-CNN according to claim 1, characterized in that: The camera intrinsic parameter calibration in step 1 is specifically as follows: 1) Print a chessboard and stick it on a flat surface as a calibration object; 2) Take photos of the calibration object in different orientations by adjusting the orientation of the calibration object or the camera. 3) Extracting checkerboard corners from the photo; 4) Estimate five intrinsic parameters and six extrinsic parameters for an ideal distortion-free scenario; 5) Apply the least squares method to estimate the distortion coefficient under actual radial distortion; 6) According to the distortion coefficient under radial distortion, the maximum likelihood method is used to perform optimization estimation to obtain the camera's intrinsic parameter matrix, radial distortion, and tangential distortion.

4. The multi-object grasp prediction method based on the improved Cascade R-CNN according to claim 1, characterized in that: The step 2 includes: Step 2.1: Select 32 categories of objects, place different combinations of objects in different postures, and use the camera at the end of the robotic arm to capture original images to form the original image dataset; Step 2.2: Use the LabelMe software to annotate each object in the original image with a grab box, so that each image has a grab label, including the position parameters and angle classification of the grab box; Step 2.3: First, crop the original 640×480 image from the center to the surrounding square to obtain a 351×351 image; Step 2.4: Then expand the image by 75 pixels in all directions using the replicate mode in OpenCV to obtain a 501×501 image. Step 2.5: Shift the pixels horizontally and vertically, perform five random rotations within 180°, and finally resize the image to obtain multiple 320×320 images, thereby achieving a 100-fold expansion of the original image dataset.

5. The multi-object grasp prediction method based on the improved Cascade R-CNN according to claim 4, characterized in that: The position parameters in step 2.2 include: the height, width, centroid coordinates and grab angle of the grab box, where the grab angle is the angle between the height direction of the grab box and the horizontal axis of the image; The angle classification is as follows: the angle between the grabbing frame of each object and the horizontal axis of the image is divided into 18 categories within the range of 0° to 180°.

6. The multi-object grasp prediction method based on the improved Cascade R-CNN according to claim 1, characterized in that: The loss function for angle classification prediction in the improved Cascade R-CNN network is cross entropy, which is expressed as follows: Among them, θ i It represents the angle classification label value. It represents the predicted value of angle classification; The loss function for the position prediction of the candidate grab box is expressed as follows using square error: Among them, x i ,y i It represents the coordinate label value of the center of mass of the grab box. It represents the predicted value of the center of mass coordinate of the grab box; h i It represents the value of the grab box height label. It represents the predicted value of the grab box height; w i It represents the value of the grab box width tag. It represents the predicted value of the grab box width.

Images