A fast segmentation method for metal parts based on deep learning

Abstract

The invention discloses a fast segmentation method for metal parts based on deep learning, comprising: establishing a metal parts data set through a Labelme image annotation tool; inputting a single image, using an object detection algorithm based on an FPN network to predict the target region of interest and candidate frame coordinates (x, y, w, h), and using Resnet‑50 and a feature pyramid as a basic network to obtain features of the entire input image; modeling the target region of interest as two other overlapping layers, namely a Top Layer and a Bottom Layer, and performing a convolution operation to further extract features; using a RoI Align algorithm to accurately extract a feature subgraph of the target region of interest in a feature map of the entire image according to the position of the object detection frame, and using it as an input of a dual-image convolutional neural network for final object segmentation; updating model parameters according to a custom loss function, iterating the above steps, and calculating whether a preset accuracy is achieved.

Description

Technical Field

[0001] The present invention relates to the field of industrial machine vision, and in particular to a method for fast segmentation of metal parts based on deep learning. Background Art

[0002] In the industrial production process, a large number of parts need to be identified and positioned. Positioning accuracy and real-time performance have become an important part of improving product production quality and efficiency. In recent years, with the development of artificial intelligence and deep learning, a large number of excellent target detection frameworks have emerged, such as YOLO, SSD, Faster-CNN, etc. Metal parts segmentation belongs to instance segmentation, which is a basic task of image and video scene understanding. This task organically combines object detection and semantic segmentation. It not only needs to predict whether each pixel of the input image belongs to an object, but also needs to distinguish the pixels contained in different objects. Instance segmentation methods represented by Mask R-CNN usually follow the paradigm of detection first and then segmentation, that is, first obtain the detection box of the target of interest, and then extract the mask of the pixels in the area, and have achieved leading performance in the COCO dataset. We have noticed that most subsequent improved algorithms such as PANet, HTC, BlendMask, CenterMask, etc. focus on designing a better network backbone, a fusion mechanism or cascade structure of high and low-level features, but ignore the role of mask prediction branches. In fact, many large-scale instance segmentation errors are due to the overlapping objects contained in the same RoL confusing the true contours of different objects, especially when the occluding and occluded targets belong to the same category or their textures and colors are similar.

[0003] Instance segmentation is one of the basic tasks in computer vision. However, it is rarely used in the industrial field, especially when it comes to metal parts segmentation. The main reasons are as follows: (1) Some parts have complex structures and severe occlusion between parts, which can lead to inaccurate segmentation; (2) The color and texture information of metal parts is single, which increases the difficulty of part segmentation; (3) In the metal parts segmentation task, there is a lack of dedicated annotated datasets, and the quality of the annotated dataset often determines the quality of the segmentation model. Summary of the invention

[0004] To solve the above technical problems, the present invention proposes a fast segmentation method for metal parts based on deep learning. Specifically, an instance segmentation network based on a two-stage detection algorithm is adopted. The boundaries of occluding and occluded objects are naturally decoupled through display modeling, and the mutual influence of occlusion relationships is considered while predicting object boundaries and masks.

[0005] The purpose of the present invention can be achieved by the following technical solution: A method for rapid segmentation of metal parts based on deep learning, comprising the following steps:

[0006] S1. Establish a metal parts dataset using the Labelme image annotation tool;

[0007] S2. Input a single image and use the object detection algorithm based on the FPN network to predict the target area of ​​interest and the coordinates of the candidate box (x, y, w, h). Resnet-50 and feature pyramid are used as the basic network to obtain the features of the entire input image.

[0008] S3, the target area of ​​interest is modeled as two other overlapping layers, namely Top Layer and Bottom Layer, and a convolution operation is performed to further extract features;

[0009] S4. Use the RoL Align algorithm to accurately extract the feature sub-image of the target area of ​​interest in the feature map of the entire image according to the position of the object detection frame, and use it as the input of the dual-image convolutional neural network for the final object segmentation;

[0010] S5. Update the model parameters according to the custom loss function, iterate the above steps, and calculate whether the preset accuracy is achieved. If the preset accuracy is achieved, output the segmentation result. Otherwise, repeat steps S2 to S4.

[0011] In step S4, the instance segmentation network is composed of a cascaded dual-graph convolutional neural network:

[0012] a. The first layer models the shape and appearance of the occluded object in the target region of interest. This layer of graph convolutional network contains four layers, namely the convolutional layer, the graph convolutional layer and the two convolutions at the end. The first graph convolutional network inputs the features of the target region of interest and outputs the boundary and mask of the occluded object in the target box of interest;

[0013] b. The second layer combines the occluded object information extracted by the first graph convolutional network. Specifically, the features of the target region of interest obtained in step S2 are added to the features after the last convolution in the first graph convolutional network to obtain new features, which are used as the input of the second graph convolutional network. Finally, the segmentation result of the occluded target object in the target region of interest is output. The second graph convolutional network has the same structure as the first graph convolutional network, forming a cascade network relationship.

[0014] c. Implementation of graph convolutional layer using non-local operator operations.

[0015] Furthermore, in step S2, the parameters of the FPN network are set: a sliding convolution operation is performed on the feature map using a convolution kernel of size 3*3, and each operation generates a 256-dimensional feature vector, which is input into the classification layer and the regression layer to perform the part target classification task and the part position box regression task.

[0016] Furthermore, during convolution, anchor points of five sizes and two aspect ratios are generated at the center of each sliding window; according to the proportion of the parts to be identified in the image, the anchor point size is set to 0.5 and the aspect ratio is set to 1 or 2, with a total of 42 sliding window sizes.

[0017] Compared with the prior art, the present invention has the following advantages: the image segmentation algorithm under complex occlusion proposed in the present invention, based on the existing two-stage segmentation model, replaces the traditional single fully convolutional mask prediction branch network with a graph neural network model composed of a cascade of two layers. In the target area of ​​interest, the front layer Top Layer models and outputs the position and shape of the occluded object, and the rear layer Bottom Layer finally outputs the corresponding mask of the occluded object based on the front layer, so that the instance segmentation algorithm can still maintain high performance and high speed on the server side under occlusion, thereby enabling finer segmentation boundaries to be obtained on the metal parts data set, thereby improving the segmentation performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The specific implementation modes of the present invention are further described in detail below in conjunction with the accompanying drawings.

[0019] Figure 1 It is a schematic flow chart of the method of the present invention;

[0020] Figure 2 This is a schematic diagram of the current common FPN network structure. DETAILED DESCRIPTION

[0021] In order to explain the present invention more clearly, the present invention is further described below in conjunction with preferred embodiments. It should be understood by those skilled in the art that the following specific description is illustrative rather than restrictive, and should not be used to limit the scope of protection of the present invention.

[0022] like Figure 1 As shown, a fast segmentation method for metal parts based on deep learning includes the following steps:

[0023] S1. Establish a metal parts dataset using the Labelme image annotation tool;

[0024] S2, input a single image, using the FPN network (see Figure 2The object detection algorithm of the FPN network structure diagram shown in the figure predicts the target region of interest (RoI) and the candidate box coordinates (x, y, w, h), and uses Resnet-50 and feature pyramid as the basic network to obtain the features of the entire input image;

[0025] S3, the target area of ​​interest is modeled as two other overlapping layers, namely Top Layer and Bottom Layer, and a convolution operation is performed to further extract features;

[0026] S4. Use the RoI Align algorithm to accurately extract the feature sub-image of the target area of ​​interest in the feature map of the entire image according to the position of the object detection frame, and use it as the input of the dual-image convolutional neural network for the final object segmentation;

[0027] S5. Update the model parameters according to the customized loss function, iterate the above steps, and calculate whether the preset accuracy is achieved. If the preset accuracy is achieved, output the segmentation result. Otherwise, repeat steps S2 to S4.

[0028] In the above step S4, the instance segmentation network is composed of a cascaded dual-graph convolutional neural network:

[0029] a. The first layer, namely the Top Layer, models the shape and appearance of the occluded object in the target area of ​​interest. This layer of graph convolutional network contains four layers, namely the convolutional layer (convolution kernel size is 3*3), the graph convolution layer (non-local layer) and the last two convolutions (convolution kernel size is 3*3). The first graph convolutional network inputs the features of the target area of ​​interest and outputs the boundary and mask of the occluded object in the target box of interest;

[0030] b. The second layer, namely the Bottom Layer, combines the occluded object information (including the boundary and mask of the occluded object) extracted by the first graph convolutional network. Specifically, the features of the target region of interest obtained in step S2 are added to the features after the last convolution in the first graph convolutional network to obtain new features, which are used as the input of the second graph convolutional network. Finally, the segmentation result of the occluded target object in the target region of interest is output. The second graph convolutional network has the same structure as the first graph convolutional network, forming a cascade network relationship.

[0031] c. In order to reduce the number of model parameters, the present invention uses non-local operator operations to implement the graph convolution layer, which specifically includes three convolution layers with a convolution kernel size of 1*1 and a softmax operator, which effectively associates the pixel points in the image space according to the similarity of the corresponding feature vectors, realizes the re-aggregation of the input target area features, and can better solve the problem of discontinuity caused by the occlusion and truncation of the pixels of the same object in space.

[0032] The above segmentation algorithm takes the relationship between the occluding object and the occluded target object into consideration at the same time, and can effectively distinguish the adjacent object boundaries between the occluding object and the occluded target object, and finally outputs the segmentation result of the occluded target object in the target region of interest.

[0033] The entire instance segmentation network can be represented end-to-end using a multi-task loss function:

[0034]

[0035]

[0036]

[0037] in, and denote the losses of boundary detection and mask segmentation respectively. λ1, λ2, λ3, λ4, and λ5 are hyperparameters of the balanced loss function and can be replaced by the initial values ​​{1, 0.5, 0.25, 0.5, 1.0} on the validation set.

[0038] In the above step S2, the following parameters are set for the FPN network: a sliding convolution operation is performed on the feature map using a convolution kernel of size 3*3. Each operation generates a 256-dimensional feature vector, which is input into the classification layer and regression layer for part target classification tasks and part position box regression tasks.

[0039] During convolution, five anchor points of different sizes and two aspect ratios are generated at the center of each sliding window. According to the proportion of the parts to be identified in the image, the anchor point size is set to 0.5 and the aspect ratio is set to 1 or 2, with a total of 42 sliding window sizes.

[0040] It is worth noting that the RoI Align algorithm uses bilinear interpolation to obtain pixel values ​​at floating-point pixel points. No quantization is performed before traversing the pre-selected box and generating units. Finally, the bilinear difference method is used to obtain 8 position coordinates of each unit, and the maximum pooling operation is performed. The back propagation formula of the RoI Align algorithm is:

[0041]

[0042] Here, x i Represents the pixel point on the feature map before pooling; y rj represents the jth point of the rth candidate region after pooling; i*(r,j) represents point y rj The source of the pixel value (the coordinates of the point where the maximum pixel value is selected during maximum pooling); x i*(r,j) is a floating point coordinate position (the sampling point calculated during forward propagation). In the feature map before pooling, each i*(r,j) Points with both horizontal and vertical coordinates less than 1 should accept the corresponding point y rj The returned gradient; d(i,i*(r,j)) represents the distance between the two points, Δh and Δw represent x i With x i (r,j) The difference between the horizontal and vertical coordinates.

[0043] Based on the existing two-stage segmentation model, the present invention replaces the traditional single fully convolutional mask prediction branch network with a graph neural network model composed of a cascade of two layers. In the target region of interest (RoI), the front layer models and outputs the position and shape of the occluded object, and the rear layer finally outputs the corresponding mask of the occluded object based on the front layer, so that the instance segmentation algorithm can still maintain high performance and high speed on the server side under occlusion.

[0044] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not limitations on the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the protection scope of the present invention.

Claims

1. A fast segmentation method for metal parts based on deep learning, characterized in that: include: S1. Pass Labelme Image annotation tools to build metal parts datasets; S2, input a single image, use the FPN The network's object detection algorithm predicts the target region of interest and the coordinates of the candidate box ( x,y,w,h ),use Resnet-50 And feature pyramid as the basic network to obtain the features of the entire input image; S3, the target area of ​​interest is modeled as two other overlapping layers, namely Top Layer and Bottom Layer , and perform convolution operations to further extract features; S4. Use RoI Align The algorithm accurately extracts the feature sub-graph of the target area of ​​interest in the feature map of the entire image according to the position of the object detection frame, and uses it as the input of the dual-image convolutional neural network for the final object segmentation; S5. Update the model parameters according to the custom loss function, iterate the above steps, and calculate whether the preset accuracy is achieved. If the preset accuracy is achieved, output the segmentation result. Otherwise, repeat steps S2 to S4. In step S4, the instance segmentation network is composed of a cascaded dual-graph convolutional neural network: a. The first layer models the shape and appearance of the occluded object in the target region of interest. This layer of graph convolutional network contains four layers, namely the convolutional layer, the graph convolutional layer and the two convolutions at the end. The first graph convolutional network inputs the features of the target region of interest and outputs the boundary and mask of the occluded object in the target box of interest; b. The second layer combines the occluded object information extracted by the first graph convolutional network. Specifically, the features of the target region of interest obtained in step S2 are added to the features after the last convolution in the first graph convolutional network to obtain new features, which are used as the input of the second graph convolutional network. Finally, the segmentation result of the occluded target object in the target region of interest is output. The second graph convolutional network has the same structure as the first graph convolutional network, forming a cascade network relationship. c. Implementation of graph convolutional layer using non-local operator operations.

2. According to the method of fast metal parts segmentation based on deep learning in claim 1, it is characterized in that: In step S2, FPN Network parameter setting: Use a size of 3 * The convolution kernel of 3 performs sliding convolution operation on the feature map. Each operation generates a 256-dimensional feature vector, which is input into the classification layer and regression layer for part target classification task and part position box regression task.

3. The method for rapid segmentation of metal parts based on deep learning according to claim 2 is characterized in that: During convolution, five anchor points of different sizes and two aspect ratios are generated at the center of each sliding window. According to the proportion of the parts to be identified in the image, the anchor point size is set to 0.5 and the aspect ratio is set to 1 or 2, with a total of 42 sliding window sizes.

Images