A method for unknown object instance segmentation for complex desktop scenes

Abstract

The application provides a kind of unknown object instance segmentation method for complex desktop scene, it is related to instance segmentation technical field, for the shape scale change of object, mutual occlusion between objects, unknown target object etc. in desktop scene, first, a kind of deep metric learning network is proposed, it contains multi-level feature fusion and multi-scale feature prediction.Then a feature fusion module is proposed to effectively fuse complementary color and depth information.Attention mechanism and skip connection can improve the representation ability of the feature map after fusion.By deploying feature fusion modules at different resolution stages of feature extraction, the robustness of the neural network to object size changes is improved.Finally, the application designs a loss function containing multi-scale metric learning mechanism, which considers the feature embedding distance of different feature levels, calculates the intra-class and inter-class distance at multiple scales, and improves the learning ability of the network to metric distance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of instance segmentation, and particularly relates to a method for unknown object instance segmentation in a complex desktop scene. BACKGROUND

[0002] Instance segmentation algorithm is widely concerned because it can obtain the pixel-level location information of the target object in the photographed scene picture. In recent years, with the continuous development of artificial intelligence and robot technology, computer vision has developed vigorously in the field of robots, and its application range has gradually expanded from the initially structured simple factory assembly line environment to the unstructured complex environment of human daily life. Using the visual sensor equipped by the household service robot, the object instance segmentation algorithm can calculate the contour and position information of the target object using the obtained visual image, which greatly improves the intelligence and practicality of the service robot, and provides a solid visual foundation for subsequent human-computer interaction and object manipulation and other downstream tasks. In practical applications, the image content obtained by the camera sensor does not only contain single target object information, and multiple target objects of different shapes and sizes often appear in the photographed image. The instance segmentation algorithm can generate a corresponding pixel-level object mask for each target object to accurately distinguish different objects, which is of great significance for robot manipulation tasks of specific objects. However, the desktop environment in the indoor scene is complex and changeable, and various irrelevant objects and interference factors may appear in the background environment, which poses a great challenge to the instance segmentation algorithm. In addition, the objects on the desktop include regular-shaped objects such as cups and bottles, and irregular-shaped objects such as scissors and spoons, which further increases the difficulty of instance segmentation. In addition, since humans inevitably place new objects on the desktop in daily life, which are unknown to the robot in advance, whether the instance segmentation algorithm can still have relatively accurate segmentation performance under such circumstances is also an important factor for measuring the performance of the desktop object instance segmentation algorithm. Unfortunately, most instance segmentation algorithms are designed for known object situations such as indoor tables, chairs, electrical appliances or outdoor pedestrians and vehicles, and they are difficult to meet the needs of household service robots for desktop object instance segmentation. Although some excellent object instance segmentation algorithms have emerged, they are difficult to adapt to changes in object shape and size, mutual occlusion between objects, and the presence of unknown objects, and thus cannot exhibit good adaptability and generalization performance.

[0003] The paper "Danielczuk M, Matl M, Gupta S, et al. Segmenting unknown 3d objects from real depth images using mask r-cnn trained on synthetic data [C] / / 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019: 7283-7290." only uses depth images to train the Mask R-CNN network and applies it to tabletop object instance segmentation. Subsequently, the paper "Xie C, Xiang Y, Mousavian A, et al. Unseen object instance segmentation for robotic environments [J]. IEEE Transactions on Robotics, 2021, 37(5): 1343-1359." first generates an initial mask using depth images, and then optimizes the initial mask using RGB images to achieve better results. The paper "Xiang Y, Xie C, Mousavian A, et al. Learning rgb-d feature embeddings for unseen object instance segmentation [C] / / Conference on Robot Learning. PMLR, 2021: 461-470." proposes a method that simultaneously learns RGB-D feature embeddings in synthetic data for unknown object instance segmentation. In the paper "Zhou H, Qi L, Huang H, et al. CANet: Co-attention network for RGB-D semantic segmentation [J]. Pattern Recognition, 2022, 124: 108468.", the encoder part is used to extract RGB, depth, and mixed features. Then a co-attention fusion part consisting of a position co-attention fusion module (PCFM), a channel co-attention fusion module (CCFM), and a fusion co-attention module (FCM) is designed to fuse the above three features.The paper “Zhang M, Shi M, Li L. MFNet: Multiclass Few-Shot Segmentation Network With Pixel-Wise Metric Learning [J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32(12): 8586-8598.” proposes a multi-path encoding and decoding architecture that effectively fuses multi-scale query information and multi-class support information into one query-support embedding. Furthermore, the method uses a pixel metric learning module to improve the learning performance of query and support images in the embedding space.

[0004] The documents “Danielczuk M, Matl M, Gupta S, et al. Segmenting unknown 3d objects from real depth images using mask r-cnn trained on synthetic data [C] / / 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019: 7283-7290.”, “Xie C, Xiang Y, Mousavian A, et al. Unseen object instance segmentation for robotic environments [J]. IEEE Transactions on Robotics, 2021, 37(5): 1343-1359.” and “Xiang Y, Xie C, Mousavian A, et al. Learning rgb-d feature embeddings for unseen object instance segmentation [C] / / Conference on Robot Learning. PMLR, 2021: 461-470.” propose three representative unseen object instance segmentation models, although they can complete the object segmentation task in the relevant experimental scenarios, the first two documents do not consider the mutual fusion of color features and depth features, and the third document only uses a simple addition operation in the last stage of the feature extraction network to complete feature fusion, often difficult to fully extract and use different modal features to improve the segmentation and generalization performance of the algorithm. The document “Zhou H, Qi L, Huang H, et al. CANet: Co-attention network for RGB-D semantic segmentation [J]. Pattern Recognition, 2022, 124: 108468.” only considers the feature fusion operation of different modalities in the encoder structure, without multi-scale feature transmission between the encoder and the decoder.Although the document "Zhang M, Shi M, Li L. MFNet: Multiclass Few-Shot Segmentation Network With Pixel-Wise Metric Learning [J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32(12): 8586-8598." can realize unknown object segmentation using a small number of samples, it is not designed for a desktop object scene, and it needs to support the assistance of images in the inference process, which limits the application range of the algorithm. In summary, the current research cannot well solve the problem of unknown object instance segmentation for complex desktop scenes. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application provides a kind of unknown object instance segmentation method for complex desktop scene, the present application realizes accurate instance segmentation under complex conditions such as object shape scale change, mutual occlusion between objects, unknown target object, to provide visual information support for specific object manipulation and grasping task of household service robot.

[0006] An unknown object instance segmentation method for complex desktop scene, comprising the following steps:

[0007] Step 1: extract color-depth multi-modal features of the RGB-D image pair using a dual-stream residual network:

[0008] The dual-stream residual network is a ResNet-34 network composed of residual units stacked, and the color-depth multi-modal features are extracted by the ResNet-34 network;

[0009] Step 1.1: Given an input RGB-D image pair with height H and width W And Where H, W represent the RGB-D image pair with height H and width W, respectively, and the depth map D is converted to a point cloud using the camera intrinsic parameters that can be obtained in advance After conversion, R and P have the same size and channel number;

[0010] Step 1.2: send the RGB image R into the ResNet-34 network to extract color features of different scales; Specifically, after processing through ResNet network layer 1 to network layer 4, color features with sizes of W / 4, W / 8, W / 16 and W / 32 are obtained;

[0011] Step 1.3: The point cloud P is input into a ResNet-34 network to extract deep features of different scales; specifically, after processing through ResNet network layers 1 to 4, deep features of sizes W / 4, W / 8, W / 16, and W / 32 are obtained;

[0012] Step 2: Use a multi-scale decoder to restore the low-resolution color-depth multi-modal features to the input image size:

[0013] The decoder uses three cascaded decoding modules, and the steps of each decoding module include:

[0014] Step 2.1: Use a 3x3 convolution to reduce the channel number of the color-depth multi-modal features from 512, 256, and 128 to 64;

[0015] Step 2.2: Use nearest neighbor upsampling to improve the resolution of the feature map obtained in step 2.1;

[0016] Step 2.3: Use a 3x3 channel-wise convolution to integrate adjacent features in a sliding window manner on the two-dimensional feature layer.

[0017] For each decoder module, a 1x1 convolution with 64 output channels is used to perform multiply-add operations on features of sizes W / 32, W / 16, and W / 8 to reduce their channel numbers.

[0018] Step 3: Use a feature fusion module to fully fuse color features and depth features: a feature fusion module containing a self-attention layer and a squeeze-excitation structure is designed to complete multi-modal feature fusion in unknown object instance segmentation, including two stages of RGB-D feature fusion and encoder-decoder feature fusion:

[0019] Step 3.1: For RGB-D feature fusion: first, use color features and depth features of sizes W / 4, W / 8, and W / 16 extracted from a dual-stream residual network for feature connection, use position embedding to mark the relative position information of the feature sequence, and input the connected feature map with position embedding into the self-attention layer; after passing through the self-attention layer, the connected feature map is split to obtain the output color feature map and depth feature map, respectively;

[0020] Step 3.2: For encoder-decoder feature fusion: first, use a squeeze-excitation structure to weight the color feature map and the depth feature map, respectively, then use element-wise addition to obtain the output feature map, and input the output feature map through a jump connection into the corresponding resolution decoder module, and add it to the existing features in the decoder module to obtain the fused features, thereby refining the feature map in the decoder.

[0021] Step 4: Calculate the loss function using the multi-scale metric learning strategy, adjust and optimize the parameters in it by training, and use the mean shift algorithm to cluster the output features to obtain the instance segmentation result:

[0022] The basic intra-class metric loss function used is:

[0023]

[0024] Where K is the number of target objects in the input image, N is the number of selected pixels for each object, d represents the cosine distance, 1{} represents the indicator function, α is the intra-class distance margin, f i k represents the feature embedding of the i-th pixel of the k-th object, and μ is the cluster center.

[0025] The basic inter-class metric loss function is as follows:

[0026]

[0027] Where [·] + represents replacing the actual value with 0 when the value in the brackets is less than 0, k and k' represent two different target objects, and δ represents the inter-class distance margin.

[0028] Through the multi-scale strategy, the basic intra-class metric loss function and the basic inter-class metric loss function are used at different feature scales; let f d32 , f d16 , f d8 be the multi-scale feature maps obtained in step 2; f final be the final output feature map, then the multi-scale metric learning loss function is represented as:

[0029]

[0030] Where Up 32 , Up 16 , and Up8 represent up-sampling operations of different multiples; L d32 , L d16 , L d8 are loss functions with scales W / 32, W / 16, and W / 8, and L final means the loss function with scale W. Summing the above loss functions gives the overall multi-scale metric learning loss function:

[0031] L MFE = λ1L d32 + λ2L d16 + λ3L d8 + λ4L final

[0032] wherein λ1, λ2, λ3 and λ4 represent constants, L MFE denotes the overall multi-scale metric learning loss function.

[0033] The technical scheme has the beneficial effects that:

[0034] The unknown object instance segmentation method for a complex desktop scene provided by the application has the advantages compared with the prior art that:

[0035] 1. A color-depth multi-modal feature fusion module is provided, which uses an attention layer and a skip connection to improve the expression ability of the fused feature map. By using this feature fusion module on features of different sizes, the robustness of the method to object size changes can be improved.

[0036] 2. A loss function containing a multi-scale metric learning strategy is provided, which calculates the intra-class and inter-class distances at multiple scales to improve the metric learning ability of the method in the feature embedding space. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 The unknown object instance segmentation method flowchart in the embodiment of the application is provided.

[0038] Figure 2 The feature fusion module structure diagram in the embodiment of the application is provided. DETAILED DESCRIPTION

[0039] The specific embodiments of the application will be further described in detail below in conjunction with the drawings and examples. The following examples are used to illustrate the application, but are not used to limit the scope of the application.

[0040] An unknown object instance segmentation method for a complex desktop scene, as shown in Figure 1 includes the following steps:

[0041] Step 1: Extract color-depth multi-modal features of the RGB-D image pair using a dual-stream residual network:

[0042] A residual unit is a mainstream feature extraction module used in the field of computer vision, which can effectively solve the network degradation problem in the process of increasing the number of layers, and has good function fitting performance. The dual-stream residual network is a ResNet-34 network composed of residual units stacked, and the color-depth multi-modal features are extracted by the ResNet-34 network;

[0043] Step 1.1: Given an input RGB-D image pair and where H, W represent the RGB-D image pair with height H and width W, the depth map D is converted into point cloud using the camera intrinsic parameters which can be obtained in advance After conversion, although the channel numbers of P and D are 3 and 1 respectively, both of them contain the same three-dimensional environment information. In this way, R and P have the same size and channel number, which facilitates the use of the same backbone network to extract features;

[0044] Step 1.2: The RGB image R is sent into the ResNet-34 network to extract color features of different scales; specifically, after processing through ResNet network layer 1 to network layer 4, color features with sizes of W / 4, W / 8, W / 16 and W / 32 are obtained;

[0045] Step 1.3: The point cloud P is sent into the ResNet-34 network to extract depth features of different scales; specifically, after processing through ResNet network layer 1 to network layer 4, depth features with sizes of W / 4, W / 8, W / 16 and W / 32 are obtained;

[0046] Step 2: Use a multi-scale decoder to restore the low-resolution color-depth multi-modal features to the input image size:

[0047] The decoder part is used to gradually restore the resolution of the feature map to the same size as the original image. The decoder uses three cascaded decoding modules to increase the resolution of the feature map while reducing the number of channels. The steps of each decoding module include:

[0048] Step 2.1: Use 3x3 convolution to reduce the channel number of the color-depth multi-modal feature from 512, 256 and 128 to 64;

[0049] Step 2.2: Use nearest neighbor upsampling to improve the resolution of the feature map obtained in step 2.1;

[0050] Step 2.3: Use 3x3 channel-wise convolution to integrate adjacent features in a sliding window manner on the two-dimensional feature layer.

[0051] The color-depth multi-modal features of different sizes are used to calculate the metric loss corresponding to each decoder module, rather than only using the feature map obtained in the last stage. For each decoder module, a 1x1 convolution with 64 output channels is used to perform multiply-add operations on the W / 32, W / 16, W / 8 size features to reduce their channel numbers.

[0052] Step 3: Use a feature fusion module to fully fuse color features and depth features:

[0053] Although great progress has been made in deep learning-based feature fusion methods recently, it is still challenging to directly apply them to the unknown object instance segmentation task. First, some feature fusion methods can only fuse low-level and high-level features extracted from single-modal data. Due to the limited perception ability of single-modal data to the environment, this will result in poor generalization ability of the network. Second, there may be target objects with variable shapes and cluttered backgrounds in the tabletop scene, which makes it difficult to segment unknown target objects. Therefore, a feature fusion module containing a self-attention layer and a squeeze-and-excitation structure is designed to complete the multi-modal feature fusion in the unknown object instance segmentation, as shown in the specific structure of Figure 2 as shown. Specifically, it includes two stages of RGB-D feature fusion and encoder-decoder feature fusion:

[0054] Step 3.1: For RGB-D feature fusion: first, use the color features and depth features with sizes of W / 4, W / 8, and W / 16 extracted from the dual-stream residual network to perform feature connection operations, use position embedding to mark the relative position information of the feature sequence, and input the connected feature map with position embedding into the self-attention layer; After passing through the self-attention layer, the connected feature map is split to obtain the output color feature map and depth feature map, respectively;

[0055] Step 3.2: For encoder-decoder feature fusion: first, the color feature map and the depth feature map are weighted through the squeeze-and-excitation structure, respectively, and then the element-wise addition operation is used to obtain the output feature map. The output feature map is sent to the corresponding decoder module through the jump connection, added to the existing features in the decoder module to obtain the fused features, thereby refining the feature maps in the decoder.

[0056] Step 4: Use the multi-scale metric learning strategy to calculate the loss function, adjust and optimize the parameters in it through training, and use the mean shift algorithm to cluster the output features to obtain the instance segmentation results:

[0057] The purpose of the metric learning strategy is to make the features belonging to the same object in the feature embedding space close to each other, while making the features of different objects far away from each other. To this end, an effective metric loss function needs to be used to guide the distance between features. The basic intra-class metric loss function used is:

[0058]

[0059] where K is the number of target objects in the input image, N is the number of selected pixels for each object, d represents the cosine distance, 1{} represents the indicator function, and a is the intra-class distance margin, is the feature embedding of the i-th pixel of the k-th object, and μ is the cluster center; the loss function aims to reduce the distance between the object feature embedding f and the cluster center μ according to the predefined distance margin α;

[0060] The basic inter-class metric loss function is as follows:

[0061]

[0062] wherein [·] + represents that when the value in the brackets is less than 0, the actual value is replaced with 0, k and k' represent two different target objects, and δ represents the inter-class distance margin; the loss function aims to make the distance between different cluster centers greater than the margin δ.

[0063] Through a multi-scale strategy, the basic intra-class metric loss function and the basic inter-class metric loss function are used at different feature scales; let f d32 , f d16 , and f d8 be the multi-scale feature maps obtained in step 2; and f final be the final output feature map, then the multi-scale metric learning loss function is represented as:

[0064]

[0065] wherein Up 32 , Up 16 , and Up8 respectively represent up-sampling operations of different multiples; L d32 , L d16 , and L d8 are loss functions with scales of W / 32, W / 16, and W / 8, and L final is a loss function with a scale of W. The sum of the above loss functions is obtained to obtain the overall multi-scale metric learning loss function:

[0066] L MFE = λ1L d32 + λ2L d16 + λ3L d8 + λ4L final

[0067] wherein λ1, λ2, λ3, and λ4 represent constants, L MFE represents the overall multi-scale metric learning loss function.

[0068] To verify the effectiveness of the algorithm, the present application is tested on the commonly used desktop object OSD dataset. The OSD dataset contains 111 pairs of RGB-D images taken in real scenes. The algorithm of the present application does not use the objects existing in the OSD dataset in the training process. Therefore, for the present application, the objects in the OSD dataset are all unknown objects. The experimental equipment platform adopted is NVIDIA RTX3090 GPU and Intel Xeon(R) Silver 4210R CPU@2.40GHz. Following previous work, the algorithm of the present application uses overlap precision, overlap recall, overlap F-measure, boundary precision, boundary recall, boundary F-measure, and overlap F-measure≥75%(F@.75) as evaluation indicators of the effect of the algorithm. On the OSD dataset, the results achieved by the present application on the above indicators are 86.9%, 90.0%, 88.4%, 80.3%, 77.8%, 78.8%, and 83.8%, respectively. As can be seen, the algorithm of the present application exhibits good instance segmentation performance in real desktop environments with different unknown target objects.

[0069] The above description is merely the preferred embodiments of the present disclosure and the explanation of the principles of the applied technology. It should be understood by those skilled in the art that the scope of the application involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combinations of the above technical features or their equivalent features without departing from the above inventive concept. For example, the technical solutions formed by mutually replacing the above features and the technical features disclosed in the embodiments of the present disclosure (but not limited to) having similar functions.

Claims

1. A method for unknown object instance segmentation for complex tabletop scenes, comprising: The method comprises the following steps: Step 1: color-depth multi-modal features of the RGB-D image pair are extracted by using a double-stream residual network; Step 2: low-resolution color-depth multi-modal features are restored to the input image size by using a multi-scale decoder; Step 3: color features and depth features are fully fused by using a feature fusion module; Step 4: a loss function is calculated by using a multi-scale metric learning strategy, parameters in the loss function are adjusted and optimized by training, and output features are clustered by using a mean shift algorithm to obtain instance segmentation results; The double-stream residual network in step 1 is a ResNet-34 network composed of residual units stacked together, and the color-depth multi-modal features are extracted by the ResNet-34 network; The step 1 specifically comprises the following steps: Step 1.1: Given an RGB-D image pair of height H and width W and where H, W denote the RGB-D image pair of height H and width W, the depth map D is converted to point cloud using the camera intrinsic which can be acquired in advance After conversion, R and P have the same size and number of channels; Step 1.2: the RGB image R is sent into the ResNet-34 network to extract color features of different scales; specifically, after the processing of ResNet network layers 1 to 4, color features of sizes W / 4, W / 8, W / 16 and W / 32 are obtained; Step 1.3: the point cloud P is sent into the ResNet-34 network to extract depth features of different scales; specifically, after the processing of ResNet network layers 1 to 4, depth features of sizes W / 4, W / 8, W / 16 and W / 32 are obtained; The decoder in step 2 adopts three cascaded decoding modules, and each decoding module comprises the following steps: Step 2.1: 3x3 convolution is used to reduce the channel number of the color-depth multi-modal features from 512, 256 and 128 to 64; Step 2.2: the resolution of the feature map obtained in step 2.1 is improved by using nearest neighbor upsampling; Step 2.3: 3x3 channel-wise convolution is used to integrate adjacent features in a sliding window manner in the two-dimensional feature layer; For each decoder module, a 1x1 convolution with an output channel of 64 is used to perform multiplication and addition operations on the W / 32, W / 16 and W / 8 size features to reduce the channel number; The step 3 specifically comprises the following steps: The step 3 specifically comprises the following steps: Step 3.1: for RGB-D feature fusion: first, the color features and depth features of sizes W / 4, W / 8 and W / 16 extracted from the double-stream residual network are connected by using position embedding to mark the relative position information of the feature sequence, and the connected feature map with position embedding is input into the self-attention layer; after passing through the self-attention layer, the connected feature map is split to obtain the output color feature map and depth feature map, respectively; Step 3.2: for encoder-decoder feature fusion: first, the color features and depth features of sizes W / 4, W / 8 and W / 16 extracted from the double-stream residual network are connected by using position embedding to mark the relative position information of the feature sequence, and the connected feature map with position embedding is input into the self-attention layer; after passing through the self-attention layer, the connected feature map is split to obtain the output color feature map and depth feature map, respectively. Step 3.2: For encoder-decoder feature fusion: firstly, the color feature map and the depth feature map are weighted respectively by squeezing-excitation structure, then the output feature map is obtained by element-wise addition operation, and the output feature map is sent into the corresponding resolution decoder module through the jump connection, added with the existing features in the decoder module to obtain the fusion features, so as to refine the feature map in the decoder; The step 4 is specifically as follows: the base intra-class metric loss function used is as follows: where K is the number of target objects in the input image, N is the number of selected pixels for each object, d denotes the cosine distance, 1{} denotes the indicator function, and a is the intra-class distance margin, denotes the feature embedding of the i-th pixel of the k-th object, and μ is the cluster center. The base inter-class metric loss function is as follows: wherein [·] + denotes that when the value in the bracket is less than 0, the actual value is replaced by 0, k and k' denote two different target objects, and δ denotes an inter-class distance margin; Through a multi-scale strategy, intra-class and inter-class loss functions are applied at different feature scales; let f d32 f d16 f d8 f represents the multi-scale feature map obtained in step 2; final If the final output feature map is used, then the multi-scale metric learning loss function is expressed as: wherein Up 32 , Up 16 , and Up8represent up-sampling operations of different multiples; L d32 , L d16 , L d8 are loss functions of scales W / 32, W / 16, W / 8, respectively, and L final denotes a loss function of scale W. The loss functions are summed to obtain the overall multi-scale metric learning loss function: L MFE = λ1L d32 + λ2L d16 + λ3L d8 + λ4L final wherein λ1, λ2, λ3, and λ4represent constants, L MFE represents the overall multi-scale metric learning loss function.

Images