A mechanical parts image segmentation algorithm based on improved DeepLabV3+ network

Abstract

The present invention relates to a mechanical parts image segmentation algorithm based on an improved DeepLabV3+ network. It includes: first, introducing an attention mechanism in the backbone network to enhance the importance of effective feature information and enhance the learning ability of component targets. Secondly, increasing the underlying feature source to make full use of the feature fusion branch to adaptively learn the spatial information of low-level features, and decomposing the original 4-fold upsampling operation into layer-by-layer upsampling, reducing the loss of important pixel information. Finally, using asymmetric convolution to enhance the core skeleton part of the 3×3 convolution in the decoding layer, the processing power of the convolution kernel and the model accuracy are improved. The method of the present invention realizes the accurate segmentation of the component area in the component image, effectively making up for the problem of blurred edge segmentation and incomplete segmentation in the traditional DeepLabV3+ segmentation of the component image, and at the same time has a certain real-time performance, thereby improving the industrialization efficiency.

Description

Technical Field

[0001] The present invention relates to the field of deep learning and computer vision technology, and specifically to a mechanical parts image segmentation algorithm based on an improved DeepLabV3+ network. Background Art

[0002] Parts recognition and segmentation is an important part of the intelligent grasping vision task of industrial robots, and its segmentation results will have a great impact on the smooth progress of the grasping task. In the research on parts recognition and segmentation, traditional algorithms such as HOG algorithm, Canny edge detection and Sobel operator were first used for feature extraction. However, the accuracy of part feature recognition of these traditional methods mainly depends on the quality of the manually selected feature extraction method, and the recognition effect is not good in multi-target mechanical parts images; as convolutional neural network algorithms are gradually proposed, the method of extracting parts image areas is often target detection or semantic segmentation. R-CNN, SS and YOLO series are commonly used in target detection methods, but due to the wide variety of parts in actual scenes, target detection methods are difficult to extract target contours and specific areas, but semantic segmentation methods can, and commonly used methods are SegNet, U-Net and DeepLab series, etc., but in the process of segmentation, due to the similarity of some parts images with the background, local occlusion and other problems, the segmentation difficulty is increased. Summary of the invention

[0003] Aiming at the problem that the traditional DeepLabV3+ has fuzzy part edge segmentation and incomplete segmentation when segmenting parts images, resulting in low accuracy of final image segmentation, a mechanical parts image segmentation algorithm based on the improved DeepLabV3+ network is proposed to reduce the fuzziness of boundary information in the parts image segmentation results and improve the accuracy of the parts segmentation algorithm to adapt to the current industrial parts image segmentation. The experimental results show that the algorithm model in this paper can achieve good segmentation results and has a certain reference significance for parts image segmentation.

[0004] The technical solution adopted by the present invention to achieve the above-mentioned purpose is: a mechanical parts image segmentation algorithm based on an improved DeepLabV3+ network, comprising the following steps:

[0005] Step 1: Collect the original component image; segment and mark one or more component boundaries appearing in the original component image to obtain an artificial segmentation image, and use the original image and the artificial segmentation image to jointly establish a component image dataset;

[0006] Step 2: Perform data augmentation processing on the component images and the manually segmented images to expand the sample images, and then randomly divide them into training set, validation set and test set according to the proportion;

[0007] Step 3: Build an improved semantic segmentation model of DeepLabV3+, including encoding layer and decoding layer; the encoding layer includes MobileNetV2 and channel attention module, adaptive spatial feature fusion module ASFF, spatial pyramid pool module ASPP, 1×1 convolution module, which is used to reduce the size of feature map and extract information for component boundary segmentation; the decoding layer includes 1×1 convolution module, upsampling operation, asymmetric convolution module, which is used to gradually restore the spatial dimension and output the boundary segmentation result map of the same size;

[0008] Step 4: Set the network training parameters, input the training set images into the improved DeepLabV3+ semantic segmentation network for training, and use the validation set for verification to obtain the optimized network model;

[0009] Step 5: Use the optimized DeepLabV3+ semantic segmentation model to identify and predict the component images in the test set to obtain the mechanical component segmentation result map.

[0010] The data enhancement includes rotation, translation and noise addition operations.

[0011] The rotation includes horizontal flipping, vertical flipping, rotation of ±60°, ±90°, ±210°, and ±240°; the translation includes shifting right by 100 pixels and shifting down by 100 pixels; and adding salt and pepper noise of 0.001, 0.002, and 0.003; finally, the enhanced dataset samples are randomly divided into training set, validation set, and test set in a ratio of 8:1:1.

[0012] The MobileNetV2 and channel attention modules process the input sample image and output the original Figure 1 The underlying feature map of size / 4, the feature level maps of three different sizes, and the original Figure 1 / 16 size high-level semantic feature map; three feature level maps of different sizes are input into the ASFF module and then the fused feature map is output to the decoding layer; the high-level semantic features are input into the ASPP structure, and in parallel, 1×1 convolutions, dilated convolutions with dilated rates of 6, 12, and 18 are used to extract features, and the feature maps are fused after global average pooling and then output to the 1×1 convolution module, and the obtained high-level semantic feature map enters the decoding layer;

[0013] The bottom feature map output by the coding layer is added with the ASFF fusion feature map after 1×1 convolution; then, it is fused with the feature map after bilinear interpolation upsampling layer by layer by a factor of 2; then, asymmetric convolution is performed on the fused feature map, and the final segmentation prediction result map is output after a factor of 2 upsampling layer by layer.

[0014] The steps of processing by the ASFF module include:

[0015] Take three feature level graphs of different sizes as input, denoted as x1, x2, and x3 respectively, and perform downsampling operation through 3×3 convolution to adjust the feature levels x2 and x3 to the same size as x1;

[0016] The resized feature level graphs x1, x2, and x3 are adaptively learned according to the fusion feature formula to obtain the fusion feature graph y l ; The formula for the level l fusion feature is as follows:

[0017]

[0018] in, Represents the output feature map y l The (i,j)th eigenvector of Represents the spatial importance weights of the three different levels to level l, and this weight information is shared by all channels. It is represented as the feature vector at position (i, j) on the feature map adjusted from level m to level l.

[0019] The layer-by-layer 2-fold bilinear interpolation upsampling replaces the original 4-fold upsampling for pixel continuity and reduces the loss of important pixel information.

[0020] The asymmetric convolution module includes 3×3, 1×3, and 3×1 convolutions. For the jth convolution kernel, let F (j) , and Respectively represent the output results of 3×3, 1×3 and 3×1 convolution kernels, and the final fusion result It can be expressed as:

[0021]

[0022] In the formula, σ j , Corresponding to the standard deviation values ​​of 3×3, 1×3, and 3×1 convolutions, γ j , and β j , are the scaling factors and offsets of 3×3, 1×3, and 3×1 convolutions, respectively. j represents the bias, and the formula is:

[0023]

[0024] In the formula, μ j , It is the channel average of batch normalization of 3×3, 1×3, and 3×1 convolutions, so the following asymmetric convolution fusion formula can be obtained:

[0025]

[0026] In the formula, O :,:,j , They represent the original outputs of the three branches: 3×3, 1×3, and 3×1. * represents the two-dimensional convolution operator. M :,:,k is the feature map of size U×V on the k-th channel of M, represents the convolution kernel j acting on the kth channel after fusion, and C represents the number of channels.

[0027] The network training parameters include: the number of training epochs is set to 100, the initial learning rate is set to 0.007, the learning strategy is "poly", the batch size is set to 2, the initial momentum is 0.9, the cross entropy loss function is used, and the stochastic gradient descent optimization algorithm is selected for training.

[0028] The present invention has the following beneficial effects and advantages:

[0029] The method in this paper replaces the original backbone network with MobilenetV2 to meet the real-time requirements in industrial scenarios; the channel attention mechanism added to the backbone network can enhance the network's ability to learn the target; the added adaptive spatial feature fusion module makes full use of the feature information of multi-scale features to obtain more target information; the layer-by-layer upsampling operation can ensure pixel continuity and reduce the loss of important pixel information; finally, asymmetric convolution is used to enhance the core skeleton part of the 3×3 convolution, improve the processing power of the convolution kernel and the model accuracy, and finally the method in this paper can segment the component images with higher accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the overall process of the method of the present invention;

[0031] Figure 2 Schematic diagram of a channel attention module used in the method of the present invention;

[0032] Figure 3 A schematic diagram of the structure of the method of the present invention based on an improved DeepLabV3+ network;

[0033] Figure 4 This is a schematic diagram of the effect of identifying occluded parts using a mechanical parts image segmentation algorithm based on an improved DeepLabV3+ network in a method in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation method of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the invention, so the present invention is not limited by the specific implementation disclosed below.

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0036] Figure 1 It is a flow chart of the present invention.

[0037] Step 1: Data preparation

[0038] The parts are randomly placed, and then the original part images are obtained using an industrial camera. The Labelme software is used to classify and annotate the part boundaries and the center circle contours of some parts in the original part images (that is, the parts to be identified are labeled according to their types) to obtain artificially segmented images. The original images and the artificially segmented images are used together to establish a part image dataset.

[0039] Step 2: Data enhancement processing

[0040] The parts images and manually segmented images were subjected to data enhancement processing such as rotation, translation and adding noise to obtain sample images, and all sample images were randomly divided into training set, validation set and test set in proportion; data enhancement refers to the method of expanding the original data set through a series of random transformations to increase the data volume. This system uses some random image processing methods to expand the original data set, including horizontal flipping, vertical flipping, rotation of ±60°, ±90°, ±210°, ±240°, right shift of 100 pixels, down shift of 100 pixels, and adding salt and pepper noise of 0.001, 0.002, and 0.003 to improve the generalization ability of the network; finally, the enhanced data set samples are randomly divided into training set, validation set and test set in the ratio of 8:1:1.

[0041] Step 3: Build a segmentation model

[0042] The overall structure of DeepLabV3+ can be divided into two parts: the encoding layer and the decoding layer. The main function of the encoding layer is to reduce the size of the feature map and extract more information for accurate segmentation. The decoding layer gradually restores the spatial dimension and completes the output of the same size as much as possible. In the encoding layer, MobileNetV2 replaces the original Xception as the backbone network and adds a channel attention module to enhance the learning ability of the target. The channel attention structure is as follows Figure 2 The input component image first passes through this backbone network to obtain the original Figure 1 The underlying features of size / 4, the feature hierarchy diagrams of three different sizes, and the original Figure 1 / 16 size of three parts of high-level semantic features, of which three different sizes of feature level maps are input into the Adaptively Spatial Feature Fusion (ASFF) module, and the high-level semantic feature part is input into the Atrous Spatial Pyramid Pooling (ASPP). In the ASPP structure, 1×1 convolution is performed in parallel, and the hole convolution with hole rates of 6, 12, and 18 continues to extract features and global average pooling to fuse the feature maps to avoid information loss. Then 1×1 convolution is used to reduce the number of channels, and the obtained feature map enters the decoding layer; in the decoding layer, the underlying feature map generated by the backbone network is added with the feature map fused with ASFF after 1×1 convolution to increase the feature information carried by the number of channels, and then the feature map obtained by the encoding layer is fused with the feature map after bilinear interpolation upsampling layer by layer by 2 times. After that, asymmetric convolution is used on the fused feature map, and then upsampling layer by layer by 2 times is performed, and finally the prediction result of the network is obtained. The improved network structure based on DeepLabV3+ is as follows Figure 3 shown.

[0043] In this embodiment, Xception is replaced by MobileNetV2 because the MobileNetV2 network can solve the problems of large convolutional neural network and insufficient hardware training that occur in the model training process. Its core is the deep separable convolution, which includes two parts: longitudinal convolution and point convolution. It can greatly reduce the amount of calculation when the convolution kernel size is the same, ensuring the real-time performance of the algorithm in industrial scenarios.

[0044] At the same time, an effective channel attention module is introduced before the jump connection of the inverted residual block in the backbone network. This module W×H×C When only the interaction between the channel and the one-dimensional convolution kernel k is considered, the shared weight ω i After performing global average pooling on the channel, we get a 1×1×C feature map. Since the size of the one-dimensional convolution k is related to the channel dimension C, The mapping relationship between the convolution kernel and the kernel size k can be determined by C, so the size of k and the range of cross-channel information interaction can be determined by C. The output can be obtained by multiplying the mapped 1×1×C feature map with the initial input feature map. In this way, the convolution kernel size k can be adaptively determined to capture local cross-channel interactions, reduce model complexity, and improve efficiency.

[0045] The ASFF module is designed to make full use of the underlying information. This operation is to add a backbone network to generate 1 / 2, 1 / 4, and 1 / 8 feature maps as input feature levels, denoted as x1, x2, and x3, respectively. The number of input channels is 32, 24, and 16, respectively. Then, a 3×3 convolution is used to downsample the feature levels x2 and x3 to the same size as x1. Finally, x1 is output as the result because this level has richer semantic and spatial information than other levels. Then, the adaptive fusion operation is used to adaptively learn the spatial weight information and then fuse it. It is represented as the feature vector at position (i, j) on the feature map adjusted from level m to level l. The corresponding formula for fusion feature at level l is:

[0046]

[0047] in, Represents the output feature map y l The (i,j)th eigenvector of . Indicates the spatial importance weights of different three levels to level l, and the weight information is shared by all channels to improve efficiency;

[0048] The original 4x upsampling is replaced by 2x bilinear interpolation upsampling layer by layer to ensure the continuity of pixels and reduce the loss of important pixel information.

[0049] Asymmetric convolution fusion module, the use of asymmetric convolution is to enhance the core skeleton of 3×3 convolution, enhance the processing capability of convolution kernel, and improve the accuracy of the model. The convolution includes 3×3, 1×3, and 3×1 convolution. For the jth convolution kernel, let F (j) , and They represent the output results of 3×3, 1×3 and 3×1 convolution kernels respectively, and the final result of convolution kernel fusion It is expressed as:

[0050]

[0051] In the formula, σ j , Corresponding to the standard deviation values ​​of 3×3, 1×3, and 3×1 convolutions respectively. j , and β j , are the scaling factors and offsets of 3×3, 1×3, and 3×1 convolutions, respectively. j represents the bias, and the formula is:

[0052]

[0053] In the formula, μ j , is the channel average of batch normalization of 3×3, 1×3, and 3×1 convolutions, and the following asymmetric convolution fusion formula can be obtained:

[0054]

[0055] In the formula, O :,:,j , They represent the original outputs of the three branches of 3×3, 1×3 and 3×1, which are summed up as the output of the asymmetric convolution fusion module. * represents the two-dimensional convolution operator. M :,:,k is the feature map of size U×V on the k-th channel of M, represents the convolution kernel j acting on the kth channel after fusion, and C represents the number of channels.

[0056] Step 4: Train the semantic segmentation model

[0057] Set the network training parameters, and then input the training set images divided in step 2 into the improved DeepLabV3+ semantic segmentation network constructed in step 3 for training.

[0058] In this embodiment, the network training parameters are that the epoch of the training network is set to 100, the initial learning rate is set to 0.007, the learning strategy is "poly", the batch size is set to 2, the initial momentum is 0.9, the cross entropy loss function is used, and the stochastic gradient descent (SGD) optimization algorithm is selected.

[0059] Step 5: Testing

[0060] The DeepLabV3+ semantic segmentation model is used to identify and segment the component images in the test set to obtain the segmentation results of the network, such as Figure 4 c). Among them, Figure 4 a) shows the original image. Figure 4 b) shows a manually annotated image. The edges of the parts identified by this method are highly consistent with the manually annotated image, indicating that the segmentation effect of this method is good.

[0061] In this embodiment, the average intersection over union (MIoU) index is used to judge the image segmentation performance. The higher the value, the more accurate the network prediction.

[0062]

[0063] Where n represents the total number of categories, that is, the component category to which the pixel is manually labeled; p ij represents the number of pixels that should be in class i but are predicted to be in class j; p ii Indicates the number of pixels predicted correctly. ij and p ji They represent false positives and false negatives respectively. The test set is used to verify the experimental results. The experimental results show that the average intersection-over-union ratio of the part images reaches 93.77%.

[0064] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any simple modification, change and equivalent structural change made to the above embodiment based on the technical essence of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A mechanical parts image segmentation algorithm based on an improved DeepLabV3+ network, characterized in that: The following steps are involved: Step 1: Collect the original component image; segment and mark one or more component boundaries appearing in the original component image to obtain an artificial segmentation image, and use the original image and the artificial segmentation image to jointly establish a component image dataset; Step 2: Perform data augmentation processing on the component images and the manually segmented images to expand the sample images, and then randomly divide them into training set, validation set and test set according to the proportion; Step 3: Build an improved semantic segmentation model of DeepLabV3+, including encoding layer and decoding layer; the encoding layer includes MobileNetV2 and channel attention module, adaptive spatial feature fusion module ASFF, spatial pyramid pool module ASPP, 1×1 convolution module, which is used to reduce the size of feature map and extract information for component boundary segmentation; the decoding layer includes 1×1 convolution module, upsampling operation, asymmetric convolution module, which is used to gradually restore the spatial dimension and output the boundary segmentation result map of the same size; Step 4: Set the network training parameters, input the training set images into the improved DeepLabV3+ semantic segmentation network for training, and use the validation set for verification to obtain the optimized network model; Step 5: Use the optimized DeepLabV3+ semantic segmentation model to identify and predict the component images in the test set to obtain the mechanical component segmentation result map.

2. The mechanical parts image segmentation algorithm based on the improved DeepLabV3+ network according to claim 1, characterized in that: The data enhancement includes rotation, translation and noise addition operations.

3. The mechanical parts image segmentation algorithm based on the improved DeepLabV3+ network according to claim 2, characterized in that: The rotation includes horizontal flipping, vertical flipping, rotation of ±60°, ±90°, ±210°, and ±240°; the translation includes shifting right by 100 pixels and shifting down by 100 pixels; and adding salt and pepper noise of 0.001, 0.002, and 0.003; finally, the enhanced dataset samples are randomly divided into training set, validation set, and test set in a ratio of 8:1:

1.

4. The mechanical parts image segmentation algorithm based on the improved DeepLabV3+ network according to claim 1, characterized in that: The MobileNetV2 and channel attention modules process the input sample images and output the bottom feature map of 1 / 4 the size of the original image, three feature level maps of different sizes and the high-level semantic feature map of 1 / 16 the size of the original image; the three feature level maps of different sizes are input into the ASFF module and then the fused feature map is output to the decoding layer; the high-level semantic features are input into the ASPP structure, and the features are further extracted by 1×1 convolution in parallel, and the feature maps are fused after global average pooling and then output to the 1×1 convolution module, and the obtained high-level semantic feature map enters the decoding layer; The bottom feature map output by the coding layer is added with the ASFF fusion feature map after 1×1 convolution; then, it is fused with the feature map after bilinear interpolation upsampling layer by layer by a factor of 2; then, asymmetric convolution is performed on the fused feature map, and the final segmentation prediction result map is output after a factor of 2 upsampling layer by layer.

5. The mechanical parts image segmentation algorithm based on the improved DeepLabV3+ network according to claim 4, characterized in that: The steps of processing by the ASFF module include: Take three feature level graphs of different sizes as input, denoted as x1, x2, and x3 respectively, and perform downsampling operation through 3×3 convolution to adjust the feature levels x2 and x3 to the same size as x1; The resized feature level graphs x1, x2, and x3 are adaptively learned according to the fusion feature formula to obtain the fusion feature graph y l ; The formula for level l fusion features is as follows: in, Represents the output feature map y l The (i,j)th eigenvector of Represents the spatial importance weights of the three different levels to level l, and this weight information is shared by all channels. It is represented as the feature vector at position (i, j) on the feature map adjusted from level m to level l.

6. The mechanical parts image segmentation algorithm based on the improved DeepLabV3+ network according to claim 1, characterized in that: Layer-by-layer 2x bilinear interpolation upsampling replaces the original 4x upsampling for pixel continuity and reduces the loss of important pixel information.

7. The mechanical parts image segmentation algorithm based on the improved DeepLabV3+ network according to claim 1, characterized in that: The asymmetric convolution module includes 3×3, 1×3, and 3×1 convolutions. For the jth convolution kernel, let F (j) , and Respectively represent the output results of 3×3, 1×3 and 3×1 convolution kernels, and the final fusion result It can be expressed as: In the formula, They correspond to the standard deviation values ​​of 3×3, 1×3, and 3×1 convolutions respectively. and are the scaling factors and offsets of 3×3, 1×3, and 3×1 convolutions, respectively. j represents the bias, and the formula is: In the formula, It is the channel average of batch normalization of 3×3, 1×3, and 3×1 convolutions, so the following asymmetric convolution fusion formula can be obtained: In the formula, They represent the original outputs of the three branches: 3×3, 1×3, and 3×1. * represents the two-dimensional convolution operator. M :,:,k is the feature map of size U×V on the k-th channel of M, represents the convolution kernel j acting on the kth channel after fusion, and C represents the number of channels.

8. The mechanical parts image segmentation algorithm based on the improved DeepLabV3+ network according to claim 1, characterized in that: The network training parameters include: the number of training epochs is set to 100, the initial learning rate is set to 0.007, the learning strategy is "poly", the batch size is set to 2, the initial momentum is 0.9, the cross entropy loss function is used, and the stochastic gradient descent optimization algorithm is selected for training.

Images