A real-time semantic segmentation method based on hybrid depthwise separable convolution
By adopting mixed depth separable convolution in semantic segmentation method, combining multi-scale feature expression and feature rearrangement, the problem of local receptive field limitation in the existing technology is solved, efficient and accurate semantic segmentation is achieved, and practical application is promoted. .
Patent Information
- Application Number
- CN202110179063.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-02-09
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-02-09
AI Technical Summary
The existing semantic segmentation methods are difficult to meet the real-time requirements while improving the segmentation accuracy. Especially due to the limitation of local receptive fields, it is difficult to fuse multi-scale features of the image, resulting in a low accuracy rate.
Using mixed depth separable convolution, mix convolution kernels of different sizes into a single depth separable convolution unit, design parallel convolutions of different sizes to enhance the multi-scale target feature expression capability, and eliminate channels through feature rearrangement The mutual independence of information between them breaks through the limitations of local receptive fields.
While maintaining high efficiency, the segmentation accuracy is significantly improved, a good balance between segmentation efficiency and accuracy is achieved, and the quality of the final feature map is improved.
Smart Images

Figure CN112861727B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of real-time semantic segmentation of images in computer vision, and is a method for performing semantic segmentation using a convolutional neural network. Background Art
[0002] In the field of computer vision, image semantic segmentation is a key task and a hot topic of research. The purpose of semantic segmentation is to segment the scene image captured by the camera into a series of non-overlapping image regions and assign a specific category to each pixel in the image. These categories usually include countable objects (such as bicycles, cars, and people) and uncountable objects (such as the sky, roads, and grass). With the rise of deep learning, image semantic segmentation technology has paved the way for scene analysis and plays a vital role in autonomous driving, augmented reality, and video surveillance.
[0003] However, existing semantic segmentation methods mainly focus on improving performance. In order to improve the accuracy of segmentation, these algorithms usually use more complex backbone networks for feature encoders, and semantic decoders also use intensive computing network structures, such as spatial pyramid pooling, shallow and deep feature fusion, and setting dilated convolutions with different expansion rates to expand the receptive field. Therefore, the network model is large in scale and the scene segmentation efficiency is low. However, for practical applications, the algorithm needs to maintain a high segmentation accuracy, and the real-time requirements must also be met. Although some real-time semantic segmentation algorithms (such as the ICNet algorithm) are fast, they are limited by the local receptive field mechanism and lack a full understanding of the target features of different scales in the scene image. It is difficult to obtain long-term memory of various features, resulting in low accuracy. Therefore, it is necessary to break through the limitations of the local receptive field, fuse the multi-scale features of the image for prediction, improve the accuracy of scene segmentation, and meet the real-time requirements.
[0004] In order to solve the above problems, this paper proposes a real-time semantic segmentation method based on hybrid depth separable convolution, which uses hybrid depth separable convolution to mix multiple convolution kernels of different sizes into a single depth separable convolution unit, designs parallel convolutions of different sizes to enhance the expression ability of multi-scale target features, and uses feature rearrangement to eliminate the situation where information between channels is independent of each other, breaking through the limitation of local receptive field, improving the feature extraction ability of different sizes, and thus improving the quality of the final feature map. The method proposed by the present invention achieves a good balance between segmentation efficiency and segmentation accuracy. Summary of the invention
[0005] In order to solve the problem of unbalanced segmentation efficiency and segmentation accuracy in the field of image semantic segmentation, the present invention designs a real-time semantic segmentation method based on hybrid depthwise separable convolution, which achieves a better balance between prediction efficiency and prediction accuracy, and further promotes the practical application of semantic segmentation. Figure 1 shown.
[0006] A real-time semantic segmentation method based on hybrid depthwise separable convolution includes the following steps:
[0007] (1) Data preprocessing and data enhancement: The present invention selects 3475 images from a public urban landscape dataset, of which 2975 images are used as a training set and 500 images are used as a validation set; the training set is used to train the network model, and the validation dataset is used to select the best training result; categories that meet practical applications are selected from the annotation information, and inapplicable data categories are eliminated; these data are normalized and augmented, and six different data augmentation methods are used for data augmentation, including random cropping and scaling, random horizontal flipping, random brightness, random saturation, random contrast, and Gaussian blur.
[0008] (2) Design of hybrid depth-separable convolution unit: In order to break through the limitation of local receptive field of ordinary convolution, improve the multi-scale feature expression of the above image data and meet the requirements of real-time prediction, the hybrid depth-separable convolution of the present invention is as follows: Figure 2 As shown in the figure, the feature maps to be processed are first divided into 4 groups according to the channels and the edges of the 4 groups of feature maps are filled with 0; secondly, the 4 groups are respectively extracted using odd-numbered convolution kernels of different sizes; then the obtained 4 groups of features are concatenated according to the channels; finally, the feature maps are alternately rearranged to eliminate the problem of independent information between channels.
[0009] (3) Constructing a hybrid depth-separable convolution module: The hybrid depth-separable convolution module of the present invention is as follows: Figure 3 As shown in the figure, it consists of 4 mixed depth-wise separable convolutional units, 4 normalization layers, and 4 nonlinear activation layers stacked in series. Three skip connections are used within the module, and 1×1 convolution residual connections are used between modules to alleviate the problem of gradient disappearance.
[0010] (4) Building a hybrid deep separable convolutional semantic segmentation network: The network structure diagram of the present invention is as follows Figure 4 As shown in the figure, the backbone network includes a hybrid depth separable convolution, a normalization layer, and a nonlinear activation layer. It is composed of 4 hybrid depth separable convolution modules connected in series, and each convolution block generates a feature map. In order to obtain a feature map with higher resolution, the downsampling operation is removed in the last two hybrid depth separable convolution modules, so that the final feature map size is 1 / 8 of the input image, thereby retaining more details of the feature map.
[0011] (5) Train and verify the semantic segmentation network of hybrid deep separable convolution: input the processed image into the designed network, output the predicted segmentation result through the forward calculation of the network, and use the cross-entropy loss function and the corresponding pixel-level label to calculate the loss value; use the momentum stochastic gradient descent optimizer, the momentum coefficient is set to 0.9, the weight decay hyperparameter factor is set to 0.00005, and the learning rate is warmed up 5 times and increased from 0 to 0.003; iterate the training until the cross-entropy loss converges and verify the performance on the validation set.
[0012] This paper designs a real-time semantic segmentation method of hybrid depth-separable convolution. On the urban landscape verification dataset, the backbone network constructed using hybrid depth-separable convolution has the advantages of high prediction accuracy, small number of parameters, fast and lightweight, and has achieved a good balance between prediction efficiency and prediction accuracy. It can achieve efficient scene perception tasks and further promote its application in many fields such as autonomous driving, augmented reality, and video surveillance. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 It is an overall schematic diagram of the method of the present invention.
[0014] Figure 2 Schematic diagram of the hybrid depth-wise separable convolutional unit proposed in the present invention.
[0015] Figure 3 This is a structural diagram of the hybrid depth-separable convolution module proposed in the present invention.
[0016] Figure 4 This is the structural diagram of the semantic segmentation network based on hybrid depth-separable convolution proposed in the present invention. DETAILED DESCRIPTION
[0017] The present invention is further described below in conjunction with the accompanying drawings:
[0018] The present invention is a real-time semantic segmentation method based on hybrid depth-separable convolution, such as Figure 1 As shown in the figure, the specific process of this method is as follows: first, 3475 images of the public urban landscape dataset are selected, and these data are preprocessed and enhanced; then, in order to improve the multi-scale feature expression in these image data and meet the requirements of real-time prediction, a hybrid depth-separable convolution unit is designed; then a hybrid depth-separable convolution module is constructed to further improve the multi-scale feature expression capability while eliminating the aliasing effect of multi-scale features; then a semantic segmentation network of hybrid depth-separable convolution is built; finally, the semantic segmentation network of hybrid depth-separable convolution is trained and verified, and the images processed in the first step are input into the network for training, and the performance is verified through the forward reasoning of the network and the predicted segmentation results are output.
[0019] The present invention has been verified to have good segmentation performance on public data sets, and the specific implementation method includes the following steps:
[0020] Step 1: Data preprocessing and data enhancement
[0021] (1) Data preparation. We selected 3,475 images from a public urban landscape dataset, of which 2,975 images were from the training set and the remaining 500 images were from the validation set. These images contain actual traffic scenarios, images of driving scenes in 50 different cities, and have high-quality pixel-level annotation information. We selected categories that meet practical applications from the annotation information, and discarded inappropriate categories and set them as ignored categories.
[0022] (2) Normalization and data enhancement. First, the RGB (red, green, and blue) images are normalized to eliminate the adverse effects that may be caused by singular sample data. Secondly, in order to improve the segmentation accuracy and generalization ability of the model, six different data enhancement methods are used on the prepared data: random cropping and scaling, random horizontal flipping, random brightness, random saturation, random contrast, and Gaussian blur.
[0023] Step 2: Designing Hybrid Depthwise Separable Convolutional Units
[0024] Since the images selected above contain many objects of different sizes, a single convolutional mapping often lacks a full understanding of multi-scale features. To this end, a hybrid depthwise separable convolutional unit is designed, which Figure 2 As shown in the figure, the feature maps to be processed are first divided into 4 groups by channel. These 4 groups of feature maps are fed in parallel to odd-numbered convolution kernels of different sizes to extract multi-scale feature maps. These convolution kernels of different sizes are 3×3, 5×5, 7×7, and 9×9 depth-separable convolutions. In order to align the size of the output feature map, the edges of the 4 groups of input feature maps need to be padded with 0 before convolution. Then the obtained 4 groups of feature maps are spliced according to the channel dimension; since the depth-separable convolution belongs to group convolution, it is easy to cause the information between channels to be independent of each other, and lack of feature fusion between groups and channels. Therefore, after splicing, the features are rearranged and the 4 groups of feature maps are shuffled. The shuffling process is to stack them together in the natural order of the channels of these 4 groups of feature maps. The purpose of this is to eliminate the problem of independent information between channels, which is conducive to the improvement of feature extraction capabilities, and thus improve the network's ability to extract features for targets of different scales.
[0025] Step 3: Build a hybrid depth-wise separable convolutional module
[0026] The hybrid depth-separable convolution module constructed by the present invention is as follows Figure 3 As shown. Based on the second step, it is composed of 4 mixed depth separable convolution units, 4 normalization layers, and 4 nonlinear activation layers stacked in series. Specifically, the mixed depth separable convolution module is composed of mixed depth separable convolution units, normalization layers, and nonlinear activation layers stacked in series 4 times in sequence, where the number of channels of the feature maps output by the first, second, and third mixed depth separable convolution units are the same, and the output feature map sizes are the same; the number of feature map channels output by the fourth mixed depth separable convolution unit is twice the number of feature channels of the input, and the output feature map size is half the input feature size. At the same time, a residual structure with 3 jump connections is used to alleviate the problem of gradient disappearance. In the first hybrid depth-wise separable convolution unit, the four groups of convolutions use 3×3, 5×5, 7×7, and 9×9 depth-wise separable convolution kernels from left to right; in the second hybrid depth-wise separable convolution unit, the four groups use 5×5, 7×7, 9×9, and 3×3 depth-wise separable convolution kernels from left to right; in the third hybrid depth-wise separable convolution unit, the four groups use 7×7, 9×9, 3×3, and 5×5 depth-wise separable convolution kernels from left to right; in the fourth hybrid depth-wise separable convolution unit, the four groups use 9×9, 3×3, 5×5, and 7×7 depth-wise separable convolution kernels from left to right; the purpose of using different orders is to eliminate the aliasing effect of multi-scale features and further enhance the multi-scale feature expression capability.
[0027] Step 4: Build a semantic segmentation network with hybrid depth-separable convolution
[0028] In order to achieve a better balance between prediction efficiency and prediction accuracy, the present invention builds a semantic segmentation network of hybrid depth-separable convolution on the basis of the above. Figure 4As shown in the figure, the backbone network responsible for extracting features is composed of 18 convolutional layers in series. The first layer is a 7×7 convolution with 3 input channels, 64 output channels, and a stride of 2, followed by a normalization layer and a nonlinear activation layer. The second to fifth layers are the first hybrid depthwise separable convolutional module, which has 64 input channels and 128 output channels. The second to fourth layers are hybrid depthwise separable convolutional units with 64 input channels, 64 output channels, and a stride of 1. The fifth layer is a hybrid depthwise separable convolutional unit with 64 input channels, 128 output channels, and a stride of 2. The 6th to 9th layers are the second hybrid depth separable convolution module, which has 128 input channels and 256 output channels. The 6th to 8th layers have 128 input channels and 128 output channels, and a hybrid depth separable convolution unit with a stride of 1. The 9th layer has 128 input channels and 256 output channels, and a hybrid depth separable convolution unit with a stride of 2. The 10th to 13th layers are the third hybrid depth separable convolution module, which has 256 input channels and 512 output channels. The 10th to 12th layers have 256 input channels and 256 output channels, and a hybrid depth separable convolution unit with a stride of 1. The 13th layer has 256 input channels and 512 output channels, and a hybrid depth separable convolution unit with a stride of 1, in order to improve the resolution of the features. The 14th to 17th layers are the fourth hybrid depth separable convolution module, which has 512 input channels and 1024 output channels. The 14th to 16th layers have 512 input channels, 512 output channels, and a hybrid depth separable convolution unit with a step size of 1. The 17th layer has 512 input channels and 1024 output channels. It uses a hybrid depth separable convolution unit with a step size of 1, and the purpose is also to improve the resolution of the feature map. The 18th layer has 1024 input channels, and the output channel size is the number of categories of the dataset label. A 3×3 convolution with a step size of 1 is used as the final classification layer. At the same time, a residual structure with 3 jump connections is used in each hybrid depth separable convolution module to alleviate the gradient disappearance. Each hybrid depth separable convolution module is connected by a 1×1 convolution to improve training efficiency and avoid the problem of gradient disappearance. Finally, the output result of the 18th layer is passed through an upsampling layer, and the feature map size is enlarged by 8 times using bilinear interpolation. The bilinear interpolation function is shown in Formula 1.
[0029] f(u+i,v+j)=(1-i)(1-j)f(u,v)+i(1-j)f(u+1,v)+(1-i)jf(u,v+1)+ijf(u+1,v+1) (1)
[0030] Where f(u, v) is the pixel value of the feature map at the position (u, v). The pixel coordinates of the bilinear interpolation result are (u+i, v+j), where u and v are the integer part of the floating-point coordinates, and i and j are the decimal part of the floating-point coordinates. The bilinear interpolation result pixel is represented as f(u+i, v+j), which is calculated by weighting the pixel values of the four neighboring positions f(u, v), f(u, v+1), f(u+1, v), and f(u+1, v+1) in proportion.
[0031] Step 5: Train and validate the semantic segmentation network with hybrid depthwise separable convolutions
[0032] (1) Initialize the parameters of the semantic segmentation network of the hybrid depth-separable convolution. These parameters include the parameters of each convolution layer and the parameters of the normalization layer. The Xavier method is used for random initialization.
[0033] (2) Configure network hyperparameters. The initial learning rate is set to 0.003, and a learning rate warm-up strategy is adopted. After 5 warm-ups, the learning rate increases from 0 to the initial learning rate value. The momentum stochastic gradient descent optimizer is used, the momentum coefficient is set to 0.9, and the weight decay hyperparameter factor is set to 0.00005. The cross entropy loss function is used, and the loss weight coefficient defaults to 1.
[0034] (3) Model training. The training images processed in the first step are input into the network. The predicted segmentation results are output through the forward calculation of the backbone network. The loss value is calculated using the cross entropy loss function and the pixel-level label corresponding to the image. The gradient of the parameters of each layer is calculated through the back propagation of the loss value, and these parameters are updated using the momentum stochastic gradient descent method. Iterations are performed continuously until the cross entropy loss fluctuates around a certain value. At this time, the model is considered to have converged.
[0035] The cross entropy loss function is shown in formula (2):
[0036] Loss=-|y GT logy pted +(1-y GT )log(1-y pted )| (2)
[0037] where y GT represents the manually labeled category, y pted It is expressed as the probability value of the sample category predicted by the network, and Loss is the result of the cross entropy loss calculation.
[0038] (4) Verify the segmentation performance of the model. In order to verify the segmentation performance of the real-time semantic segmentation method based on hybrid depth-separable convolution, a set of comparative experiments was designed. The above-mentioned semantic segmentation network was constructed using ordinary 3×3 convolution and hybrid depth-separable convolution units respectively. Both were tested on the urban landscape verification set through the above-mentioned iterative training. The experimental results show that compared with the results of the ordinary convolution method, the method of the present invention has a parameter amount reduced by about three-fifths and a calculation amount reduced by about three-quarters, which meets the real-time performance while maintaining a high prediction accuracy. The average intersection-over-union (MIOU) result of the backbone network constructed by the hybrid depth-separable convolution used in the present invention is 73.48%, which has the advantages of high prediction accuracy, small parameter amount, fast and lightweight. Compared with the MIOU result of 71.7% of the classic real-time semantic segmentation network ICNet, it has a performance advantage of 1.78%. The evaluation index average intersection-over-union (MIOU) is shown in formula (3):
[0039]
[0040] Among them, TO represents true positive examples, FO represents false positive examples, and FN represents false negative examples.
Claims
1. A real-time semantic segmentation method based on hybrid depthwise separable convolution, It is characterized in that The method comprises the following steps: Step S1, 3475 images are selected from the public urban landscape dataset, suitable categories are selected and inappropriate categories are set as ignored categories, and the selected data are preprocessed and data enhanced; Step S2, in view of the requirements of multi-scale feature expression and real-time prediction of the image data in step S1, a hybrid depth-wise separable convolution unit is designed. The unit first divides the input feature map into 4 groups on average, uses 4 groups of depth-wise separable convolution kernels of different sizes for feature extraction, then concatenates the feature maps, and finally rearranges the feature maps alternately; Step S3, constructing a hybrid depthwise separable convolution module based on step S2, which stacks four hybrid depthwise separable convolution units with different convolution kernel orders in series to eliminate multi-scale aliasing effects; Step S4: Based on step S3, a hybrid depth-wise separable convolutional semantic segmentation network is constructed. The network consists of four hybrid depth-wise separable convolutional modules stacked in series. Three skip connections are used within the module, and 1×1 convolutions are used between modules for residual connection. Step S5, training and verifying the semantic segmentation network of the hybrid depth-separable convolution, using the image processed in step S1 as input into the network for training, verifying the performance through forward reasoning of the network and outputting the predicted segmentation result; The hybrid depth-wise separable convolutional unit designed in step S2 divides the feature maps to be processed into 4 groups by channel, and fills the edges of the 4 groups of feature maps with 0 to align the resolution of the 4 output feature maps; These four groups use different sizes of odd-numbered depth-separable convolution kernels for feature extraction. These different sizes of convolution kernels are 3×3, 5×5, 7×7, and 9×9 depth-separable convolution kernels. The four groups of features with the same resolution are concatenated by channel. Finally, the feature maps are rearranged alternately. The hybrid depth-wise separable convolution module constructed in step S3 is composed of the hybrid depth-wise separable convolution unit, normalization layer, and nonlinear activation layer designed in step S2, which are stacked in series four times in sequence, wherein the number of channels of the feature maps output by the first, second, and third hybrid depth-wise separable convolution units are the same, and the output feature map sizes are the same; the number of feature map channels output by the fourth hybrid depth-wise separable convolution unit is twice the number of feature channels of the input, and the output feature map size is half the input feature size; at the same time, a residual structure with three jump connections is used to alleviate the problem of gradient disappearance and improve the training speed.
2. The real-time semantic segmentation method based on hybrid depth-separable convolution according to claim 1, Features: When constructing the hybrid depth-wise separable convolution module, four hybrid depth-wise separable convolution units are stacked; in the first hybrid depth-wise separable convolution unit, these four groups of convolutions use 3×3, 5×5, 7×7, and 9×9 depth-wise separable convolution kernels from left to right; in the second hybrid depth-wise separable convolution unit, these four groups use 5×5, 7×7, 9×9, and 3×3 depth-wise separable convolution kernels from left to right; in the third hybrid depth-wise separable convolution unit, these four groups use 7×7, 9×9, 3×3, and 5×5 depth-wise separable convolution kernels from left to right; in the fourth hybrid depth-wise separable convolution unit, these four groups use 9×9, 3×3, 5×5, and 7×7 depth-wise separable convolution kernels from left to right, eliminating the aliasing effect of multi-scale features and improving the multi-scale feature expression capability.
3. The real-time semantic segmentation method based on hybrid depth-separable convolution according to claim 1, Features: The network for hybrid depth-wise separable convolutional semantic segmentation built in step S4; the hybrid depth-wise separable convolutional modules designed in step S3 are stacked in series 4 times, and a total of 16 hybrid depth-wise separable convolutional units are used to improve the feature semantic capabilities of the image; a residual structure with 3 jump connections is used in the hybrid depth-wise separable convolutional module to alleviate the gradient disappearance; 1×1 convolution is used for residual connection between hybrid depth-wise separable convolutional modules; finally, 3×3 convolution is used to output the final segmentation result.
4. The real-time semantic segmentation method based on hybrid depth-separable convolution according to claim 1, Features: Step S5 trains and verifies the semantic segmentation network of the hybrid deep separable convolution; the training image processed in step S1 is input into the network, the performance is verified through the forward calculation of the hybrid deep separable convolutional network and the predicted segmentation result is output, and the cross entropy loss function and the corresponding pixel-level label are used to calculate the loss value for supervised training.
