A superpixel segmentation method based on deep learning

Abstract

The present invention discloses a superpixel segmentation method based on deep learning, specifically including network over-parameterization, network pruning for the encoder part and TDT (Top-down-top) decoding structure for the decoder part. The present invention is based on deep learning and has end-to-end learning ability, and has more comprehensive integration ability with other visual tasks, thereby promoting the rapid development of other visual tasks. The method steps of the invention are roughly divided into three stages: the network over-parameterization stage in the network search strategy; the network pruning stage in the network search strategy; the network decoding ability enhancement and retraining stage. The present invention improves the boundary capture ability in the encoder feature extraction process and the structure retention ability in the decoder superpixel segmentation process, which can effectively reduce the occurrence of wrong segmentation. At the same time, it has good interpretability in network over-parameterization, network pruning and TPT decoder structure design, and its method has high computational efficiency, and the superpixel segmentation speed can reach real-time.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and computer vision, and relates to a superpixel segmentation method based on deep learning. Background Art

[0002] Superpixel segmentation is an image over-segmentation technology that quickly divides an image into a certain number of sub-regions based on the underlying features of the image, such as space, color, and texture. Compared with the basic image processing unit in traditional methods, superpixels are more conducive to the extraction of local features and the expression of structural information, and can significantly reduce the computational cost of subsequent visual tasks. This technology is an important topic in computer vision research. Superpixel segmentation technology is widely used in computer vision tasks such as semantic segmentation, optical flow calculation, and depth estimation that require accurate target boundaries and real-time requirements.

[0003] Low-level features in convolutional neural networks, such as color, gradient, and texture, are conducive to extracting edges, while high-level semantic features are conducive to capturing target boundaries. Based on this advantage, Jampani et al. proposed superpixel segmentation algorithms based on convolutional neural networks in their paper "Superpixel Sampling Networks" published at ECCV in 2018 and Tu et al. proposed superpixel segmentation algorithms based on convolutional neural networks in their paper "Learning Superpixels with Segmentation-Aware Affinity Loss" published at CVPR in 2018. The idea of ​​both algorithms is to encode traditional image space and color features through an encoder based on a convolutional neural network, use the encoded features to replace traditional manual features, and then complete the superpixel segmentation task using traditional methods. Yang et al. proposed a superpixel segmentation algorithm based on a fully convolutional encoder-decoder network in their paper "SpixelFCN: Superpixel Segmentation with Fully Convolutional Network" published at CVPR in 2020. Different from the above-mentioned strategy of fusing the working encoder with the traditional superpixel segmentation algorithm, superpixel segmentation is independently completed by the convolutional neural network composed of the encoder and decoder.

[0004] However, each of the above methods has its own drawbacks. The encoder proposed by the above method still has serious loss problems in boundary capture, and the use of traditional superpixel segmentation methods on high-dimensional features greatly increases the computational cost of the algorithm; on the other hand, the decoder based on the convolutional neural network structure is prone to feature information loss during the decoding process, which in turn causes structural information loss in superpixel segmentation.

[0005] In summary, the current superpixel segmentation algorithms based on deep learning have problems such as serious boundary loss, excessive computational cost, and structural information loss, which are not conducive to the integration of superpixel segmentation algorithms with other computer vision tasks. How to improve the above problems at the same time has become a current research hotspot. Summary of the invention

[0006] In view of the shortcomings of the above-mentioned prior art, the main purpose of the present invention is to provide a superpixel segmentation method that is superior in three aspects: capturing target boundaries, retaining structural information, and computational efficiency. The method is based on deep learning and has end-to-end training and learning capabilities. As an image preprocessing technology, it has the ability to be more fully integrated with other computer vision tasks.

[0007] The present invention provides a superpixel segmentation method based on deep learning, which comprises the following steps:

[0008] S1: Network model initialization. First, a U-type codec network is selected as the initial network. The U-type codec network structure consists of an encoder and a decoder, and has cross-layer connections at the same scale position of the encoder and the decoder;

[0009] S2: Over-parameterization of network model. Design a multi-branch feature extraction module to replace any number of feature extraction layers in the initial network encoder to achieve over-parameterization of the network model. The multi-branch feature extraction module is a parallel multi-branch structure. The input and output of the module are not required to be of the same size. Different branches are composed of convolutional neural networks with different structures. The input of each branch is the same as each other, the output size of each branch is the same as each other, and each branch has an initialized learnable weight. The outputs of different branches are added element by element through the learnable weights of each branch, followed by batch normalization and activation function, thereby forming a parallel multi-branch feature extraction structure;

[0010] S3: Network pruning. The over-parameterized network obtained by replacing any number of feature extraction layers in the initial network encoder with a multi-branch feature extraction module is trained until the loss function curve tends to be flat, and the training is stopped. At this time, the network model has the ability to perform superpixel segmentation. The pruning threshold is set according to the number of training iterations, the learning rate, and the initialization value of the learnable weight. According to the learned network model parameters, a pruning judgment is made for each branch of each multi-branch feature extraction module to complete the over-parameterized network pruning and obtain an encoder for the superpixel segmentation task. The network pruning strategy is to prune each branch in the multi-branch feature extraction module according to the network model parameters obtained through training, when the learnable weight corresponding to the branch is less than the set threshold, the branch is pruned, otherwise the branch is retained. When the weights of all branches are less than the threshold, only the branch with the largest weight is retained;

[0011] S4: Network decoding enhancement. Add cross-layer feature fusion to the decoder structure after network pruning to realize a top-down and then bottom-up decoding structure. The enhanced decoder is referred to as TDT (Top-down-top) decoder. The cross-layer feature fusion is located after the top-down decoding stage of the U-type codec network, and is used to realize cross-scale feature fusion from micro to broad, thereby improving the quality of the original scale feature map. The cross-scale feature fusion process is a bottom-up decoding structure. The specific operation of the cross-scale feature fusion is: under the original framework of the U-type codec network, after upsampling the output features of any layer under the large receptive field of the decoder front end, it is spliced ​​with the output features of the original scale network layer of the decoder back end and followed by a convolution operation to realize cross-scale information fusion between the two layers, realize network decoding enhancement, and then follow with convolution and Softmax activation function, thereby realizing the calculation of superpixel segmentation results based on probability. By connecting the top-down decoding structure of the U-type codec network and the bottom-up decoding structure of the cross-scale feature fusion, the construction of the overall decoding structure of the TDT decoder from top to bottom and then from bottom to top is realized;

[0012] S5: Network model training, and use the trained model to perform superpixel segmentation on the image. The encoder and decoder with TDT decoding structure after the network selection obtained after the above steps s1 to s4 are trained until the loss function is fully converged, and the training is stopped to obtain the trained model parameters, and the superpixel segmentation results are calculated based on the model parameters.

[0013] Beneficial Effects

[0014] Due to the adoption of the above method, the present invention has the following advantages compared with the prior art:

[0015] 1) The network over-parameterization method proposed in the present invention can add multiple feature extraction modules based on manual experience without changing the network structure of the encoder itself, enrich the search space of the network, and enable the encoder itself to have the ability to extract features that are more suitable for superpixel segmentation tasks;

[0016] 2) The network search strategy proposed in the present invention can effectively screen the effective branches in the multi-branch feature extraction module, while reducing the overfitting of the network, retaining the feature extraction branches suitable for the superpixel segmentation task, prompting the encoder to extract more powerful encoding features, thereby enhancing the boundary capture ability of the encoder and promoting the boundary accuracy of the superpixel segmentation result;

[0017] 3) The TDT decoder proposed in the present invention adopts a top-down and then bottom-up decoding structure. This structure avoids the problem that when the top-down decoding structure performs feature decoding at a large receptive field position, decoding errors are caused due to the excessively large receptive field, thereby affecting the subsequent superpixel segmentation results; conversely, in the bottom-up decoding structure, when the decoder performs feature decoding at a small receptive field position, decoding errors are caused due to the overly local decoding information, thereby affecting the subsequent superpixel segmentation results. The decoder proposed in the present invention realizes feature decoding from broad to fine and then from fine to broad through a bottom-up and then top-down TDT decoding structure, which can effectively avoid the above problems, and the superpixel segmentation results have better structure retention capabilities;

[0018] 4) The deep learning-based superpixel segmentation method proposed in this invention provides a superpixel segmentation image preprocessing technology with fast calculation speed, accurate boundary capture, and sufficient structural information retention for other visual tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of the method of the present invention;

[0020] Figure 2 Schematic diagram of the structure of an initialized U-type codec network constructed based on SpixelFCN in the embodiment of the present invention;

[0021] Figure 3 A schematic diagram of an over-parameterized network structure according to an embodiment of the present invention;

[0022] Figure 4 It is a structural schematic diagram of a multi-branch feature extraction module in the present invention;

[0023] Figure 5 A schematic diagram of the operation of each branch in the multi-branch feature extraction module of the embodiment of the present invention;

[0024] Figure 6 A schematic diagram of the structure of a codec network for finally performing superpixel segmentation reasoning in the embodiment of the present invention;

[0025] FIG. 7( a ) is a superpixel number-ASA score curve of an embodiment of the present invention;

[0026] FIG. 7( b ) is a boundary recall rate-boundary accuracy rate curve of an embodiment of the present invention;

[0027] FIG. 7( c ) is a curve of superpixel number-average calculation time according to an embodiment of the present invention; DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present invention more clearly visible, the technical solution of the embodiment of the present invention is described in more detail and completely in conjunction with the accompanying drawings. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0029] The present invention proposes a superpixel segmentation algorithm based on deep learning, the flow chart is as follows Figure 1 As shown, the specific steps include:

[0030] S1: Select SpixelFCN based on the U-type codec structure as the U-type codec network initialized in this embodiment, and the network consists of an encoder and a decoder. According to the expansion and contraction of the receptive field during the network forward transmission, the part of the network receptive field that expands the encoding features is defined as the encoder, and the part of the receptive field that contracts the decoding features is defined as the decoder. The encoder and the decoder have a cross-layer connection, which enables the decoder to restore the local detail information lost during the expansion of the encoder's receptive field. The U-type codec network structure initialized in this embodiment is as follows: Figure 2 As shown in the figure, the encoder is mainly composed of convolution operations of different step lengths to achieve feature extraction and expansion of the receptive field; the decoder is mainly composed of convolution and deconvolution operations, where convolution mainly realizes the decoding of encoded features, and deconvolution realizes the spatial scale recovery of features;

[0031] S2: Design a multi-branch feature extraction module to replace any number of feature extraction layers in the encoder of the initialized codec network to achieve network over-parameterization. In this embodiment, the second feature extraction layer at each scale in the encoder is selected as the replacement layer of the multi-branch feature extraction module. The over-parameterized network structure after replacement is as follows: Figure 3 In addition, the first feature extraction layer in each scale may be replaced, or only some feature extraction layers in some scales may be replaced, and no additional restrictions are imposed here.

[0032] The structure of the multi-branch feature extraction module is as follows: Figure 4As shown, the input and output of this module can be features of different sizes, where the operations of different branches are composed of convolutional neural networks of different structures, where the input of each branch is the same as each other, the output size of each branch is the same as each other, and the outputs of different branches are added element by element through the learnable weights pre-initialized by each branch, followed by batch normalization and activation function, thereby forming a parallel multi-branch feature extraction structure. Through the multi-branch feature extraction module to achieve network over-parameterization, the network can extract richer and more multi-level coding features, and the learnable weights of each branch obtained through training can adaptively select feature extraction branches that are more suitable for the task. The operations of each branch selected in this embodiment are as follows: Figure 5 As shown in the figure, each branch is given the same initialization weight w=1, and the multi-branch feature extraction module composed of them has a variety of receptive fields and network depths, which can effectively extract rich coding features;

[0033] S3: The over-parameterized codec network is trained according to the reconstruction loss, and the training is stopped when the loss function curve tends to be flat, so that the network has superpixel segmentation ability. In this embodiment, the reconstruction loss of the target segmentation true value labeled image is used for network training. Specifically, the true value labeled image is downsampled according to the superpixel segmentation result, and then upsampled according to the superpixel segmentation result to obtain the reconstructed true value labeled image. According to the information difference between the original image and the reconstructed image, a reconstruction loss based on cross entropy and Euclidean distance is constructed as a supervision means for network training.

[0034] A pruning threshold λ is established according to the number of network training iterations, the learning rate, and the initialization weight w. In this work, [0.85w, 0.95w] is generally selected as the candidate interval of the threshold λ. In this embodiment, λ=0.95 is selected. A pruning threshold λ is established, and the network is pruned according to the pruning strategy based on the network model parameters obtained after training to obtain a dedicated encoder that is more suitable for superpixel segmentation tasks. The network pruning strategy is to determine whether to retain or prune each branch in each multi-branch feature extraction module based on the model parameters obtained from network training. When the learnable weight of the current branch is less than the threshold, that is, w<λ, the branch is pruned; conversely, when the learnable weight of the branch is greater than or equal to the threshold, that is, w≥λ, the branch is retained; when the learnable weights of all branches are less than the threshold, only the branch with the largest weight is retained, minus other branches;

[0035] S4: Add cross-layer feature fusion to the decoder structure after network pruning to realize a top-to-bottom and then bottom-to-top decoding structure, thereby promoting the decoding capability of the decoder and reducing the decoding error. The enhanced decoder is referred to as a TDT (Top-down-top) decoder. The cross-layer feature fusion is located after the top-to-bottom decoding stage of the U-type codec network, and is used to realize cross-scale feature fusion from micro to broad, thereby improving the quality of the original scale feature map. The cross-scale feature fusion process is a bottom-to-top decoding structure. By connecting the top-to-bottom decoding structure of the U-type codec network and the bottom-to-top decoding structure of the cross-scale feature fusion in series, the construction of the overall decoding structure of the TDT decoder from top to bottom and then from bottom to top is realized.

[0036] In this embodiment, the final encoding and decoding network structure diagram is as follows: Figure 6 As shown, the top-to-bottom stage of the TDT decoder adopts the top-to-bottom structure of the U-type codec network itself, and the bottom-to-top stage is realized by cross-scale feature fusion. The specific operation of the cross-scale feature fusion is: under the original framework of the U-type codec network, the output features of any layer under the large receptive field of the decoder front end are upsampled, and then spliced ​​with the output features of the original scale network layer of the decoder back end and followed by a convolution operation to achieve cross-scale information fusion between the two layers, network decoding enhancement, and followed by convolution and Softmax activation functions, thereby realizing the calculation of superpixel segmentation results based on probability. In the bottom-to-top stage of the TDT decoder in this embodiment, the cross-scale feature fusion selects the convolution layer of the decoder at the 1 / 8 scale and the original scale network layer for information fusion. At the same time, the convolution layer of the decoder at the 1 / 4 scale or the convolution layer at the 1 / 16 scale can also be selected to perform cross-scale feature fusion with the original scale network layer. No additional restrictions are made here;

[0037] S5: The codec network obtained after the above steps s1 to s4 can effectively calculate the probability of the superpixel index to which the pixel belongs, and train it until the loss function fully converges, so as to stop the training and obtain the model parameters suitable for the superpixel segmentation task, and use this model parameters to calculate the final superpixel segmentation result.

[0038] S5-1: This embodiment selects the BSDS (Berkeley Segmentation Dataset) data set as the experimental data set, which contains 200 training images, 100 verification images and 200 test images, each of which corresponds to multiple target level segmentation annotations. After preprocessing the data set, 1633 pairs of training images and 1063 pairs of test images can be obtained as the training data and test data of this embodiment.

[0039] S5-2: Data augmentation methods during training include random cropping and random horizontal / vertical flipping operations.

[0040] S5-3: The Adam optimizer is selected during network training, where the optimizer parameters β1 = 0.9 and β2 = 0.999. The batch size is 4, the initial learning rate is 5e-5, and it decays to 2.5e-5 when the number of iterations reaches 200,000. The weight decay coefficient is 4e-4, the randomly cropped image size is 208×208, the number of classified superpixels during network training is 169, the number of training iterations in the network pruning S3 stage is 10,000, and the number of iterations in the model training S5 stage is 300,000.

[0041] Usage phase.

[0042] According to the above method, the network structure is constructed, the model parameters are initialized, and the data training set is prepared. When the training is completed, the image to be detected is input into the trained overall network, and the superpixel segmentation result of the image is output.

[0043] Method testing.

[0044] The method disclosed in the present invention is verified in the above-mentioned embodiments and compared with the currently disclosed methods to verify that the superpixel segmentation result obtained by the present invention has relative advantages in boundary capture, structural loss and computational efficiency.

[0045] The quantitative indicators used are ASA (Achievable Segmentation Accuracy), BR-BP (Boundary Recall-Boundary Precision) and calculation time. The image is segmented into superpixels to obtain K superpixels, and its set Image annotation divides the image into several regions based on information such as the target object or semantics. Where J is the number of segmented regions in the labeled image.

[0046] The indicator formula of ASA is:

[0047]

[0048] All superpixels S in the ASA process k Each of them traverses the set G and finds the G with the largest overlapping area. j And calculate the number of overlapping pixels between the two, and finally all S k The number of overlapping pixels is added and then divided by the total number of image pixels for normalization, and the ASA score is finally obtained. Since its statistical element is the image area, and the quantization result has a strong positive correlation with whether the superpixel crosses the boundary frequently, it can be used as a quantitative indicator to measure the structural loss of superpixel segmentation.

[0049] The indicator formulas for boundary recall BR (Boundary Recall) and boundary precision BP (Boundary Precision) are:

[0050]

[0051]

[0052] Among them, TP is true positive, FP is false positive, TN is true negative, and FP is false negative. Different from the regional comparison method of ASA, BR-BP quantifies superpixel segmentation with pixels as elements, and its boundary accuracy and recall rate can effectively reflect the boundary capture ability of the superpixel segmentation algorithm.

[0053] The quantitative results of the comparison are shown in Figure 7(a), Figure 7(b), and Figure 7(c). The ASA score is mainly used to measure the degree of retention of structural information of the superpixel segmentation results. The larger the value, the better. The boundary recall-accuracy is mainly used to measure the accuracy and recall of the superpixel segmentation results for the target boundary, reflecting its boundary capture ability. The closer the curve is to the upper right, the stronger the method is. Figure 7(c) shows the calculation time of different superpixel segmentation methods with different numbers of superpixels. The less calculation time, the better.

[0054] In this embodiment, we select the SpixelFCN encoding and decoding network in "SpixelFCN: SuperpixelSegmentation with Fully Convolutional Network" published by Yang et al. at CVPR 2020 as our initial network. On this basis, through a series of steps such as network over-parameterization, network pruning and TDT decoding enhancement proposed in the present invention, as shown in Figure 7(a) and Figure 7(b), the ASA and boundary recall-precision are significantly improved compared with the initial network.

[0055] In addition, compared with the SSN method proposed in "Superpixel Sampling Networks" and the SEAL method proposed in "Learning Superpixels with Segmentation-Aware Affinity Loss", the quantization results of the present invention further shorten the gap with SSN in ASA, which measures structural loss, and are only slightly worse than SEAL when the number of superpixels is small; in terms of boundary recall rate-accuracy, which measures the ability to capture table boundaries, the SSN method is significantly ahead of all other methods currently disclosed, and the method proposed in the present invention performs equally well with SSN in this indicator, and is far superior to the SEAL method. At the same time, in terms of computational efficiency, the present invention has an absolute advantage over the SSN and SEAL methods, and is basically the same as SpixelFCN, but is far superior to SpixleFCN in terms of structural information retention and boundary capture capabilities.

[0056] In summary, based on the network model over-parameterization, network pruning and TDT decoding enhancement proposed in the present invention, the boundary capture ability and structural information retention ability of the embodiment using SpixelFCN based on the U-type codec structure as the initial network construction are significantly improved. According to the quantification results of this embodiment, through the comprehensive comparison of the three quantitative indicators of boundary capture ability, structural information retention and computational efficiency with other algorithms, it can be seen that this method is the optimal algorithm at present.

Claims

1. A superpixel segmentation method based on deep learning, characterized in that The following steps are involved: S1: Network model initialization: First, a U-type codec network structure is selected as the initial network. The U-type codec network structure consists of an encoder and a decoder, and includes cross-layer connections at the same scale position of the encoder and the decoder; S2: Over-parameterization of network model: Design a multi-branch feature extraction module to replace any number of feature extraction layers in the initial network encoder to achieve over-parameterization of the network model; wherein the multi-branch feature extraction module is a parallel multi-branch structure, the input and output of the module are not required to be of the same size, wherein different branches are composed of convolutional neural networks with different structures, the input of each branch is the same as each other, the output size of each branch is the same as each other, and each branch has an initialized learnable weight, the outputs of different branches are element-by-element added through the learnable weights of each branch, followed by batch normalization and activation function, thereby forming a parallel multi-branch feature extraction structure; S3: Network pruning: Train the over-parameterized network obtained by replacing any number of feature extraction layers in the initial network encoder with a multi-branch feature extraction module, stop training when the loss function curve tends to be flat, and use the network pruning strategy to complete the over-parameterized network pruning of the trained network model to obtain an encoder for the superpixel segmentation task; wherein the network pruning strategy is for each branch in the multi-branch feature extraction module, when the corresponding learnable weight obtained by the branch training is less than the set threshold, the branch is pruned, otherwise the branch is retained, and when the weights of all branches are less than the threshold, only the branch with the largest weight is retained; S4: Add cross-layer feature fusion to the decoder structure after network pruning to realize a top-down and then bottom-up decoding structure. The enhanced decoder is referred to as TDT (Top-down-top) decoder; the cross-layer feature fusion is located after the top-down decoding stage of the U-type codec network, and is used to realize cross-scale feature fusion from micro to wide, thereby improving the quality of the original scale feature map. The cross-scale feature fusion process is a bottom-up decoding structure; the specific operation of the cross-scale feature fusion is: under the original framework of the U-type codec network, after upsampling the output features of any layer under the large receptive field of the decoder front end, it is spliced ​​with the output features of the original scale network layer of the decoder back end and followed by a convolution operation to realize cross-scale information fusion between the two layers, realize network decoding enhancement, and then follow the convolution and Softmax activation function to realize the calculation of superpixel segmentation results based on probability; by connecting the top-down decoding structure of the U-type codec network and the bottom-up decoding structure of the cross-scale feature fusion in series, the construction of the overall decoding structure of the TDT decoder from top to bottom and then from bottom to top is realized; S5: Train the decoded enhanced network and use the trained network to perform superpixel segmentation on the image.

2. The superpixel segmentation method based on deep learning according to claim 1, characterized in that: The threshold value in step S3 ranges from [0.85w, 0.95w], where w is the initialization weight.

3. The superpixel segmentation method based on deep learning according to claim 1, characterized in that: The U-type codec network is a SpixelFCN codec network.

Images