A semantic segmentation method and system based on deep convolutional neural network

Abstract

The present invention relates to a semantic segmentation method and system based on a deep convolutional neural network. The method includes model training and model application. The model training includes the following steps: obtaining a semantic segmentation image data set, preprocessing the images in the semantic segmentation data set; building a semantic segmentation network model, using a variant ResNet50 backbone network as an encoder, and using a multi-scale hybrid pool structure and a feature attention fusion module as a decoder; using the preprocessed semantic segmentation image data set to perform model training and network parameter setting of the semantic segmentation network model to obtain a trained semantic segmentation network model. Compared with the prior art, the present invention can effectively realize the connection between encoding and decoding, fully extract relevant information between different stages and within the same stage, can effectively fuse low-level and high-level features, obtain remote dependencies and rich context information, and can segment related images more efficiently and accurately.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and neural network, and in particular to a semantic segmentation method and system based on a deep convolutional neural network. Background Art

[0002] Semantic segmentation is an important research field in computer vision and one of the key technologies for achieving scene understanding. Its purpose is to assign semantic category labels to all pixels in an image and to segment and parse scene images into different regions associated with semantic categories. It is widely used in autonomous driving, medical image analysis, and target detection. Semantic segmentation is a challenging task because it requires a combination of dense pixel-level accuracy and multi-scale contextual reasoning, in addition to considering the huge differences in content, shape, and scale within the same object, as well as the high similarity between objects of different categories, and also needs to pay attention to objects in easily confused boundary areas.

[0003] In recent years, most advanced semantic segmentation methods have often made extensive use of deep convolutional neural networks (CNNs), which have shown impressive capabilities in solving various complex challenges. Convolutional neural networks are able to achieve end-to-end full-image segmentation with better accuracy than any traditional methods. Dense semantic representations are extracted from input images and pixel-level labels are predicted. After proper training, deep CNNs can obtain rich scene information using multiple convolution operations, nonlinear pooling, and activation functions. However, due to the local nature of CNNs, convolutional local features usually have limited receptive fields. Although some methods use large receptive fields, the extracted features mainly describe the core area of ​​the object, while largely ignoring the context around its boundaries. In addition, objects of different categories may have similar local features. For example, tables and chairs may share similar local textures. Therefore, these detailed information brings great difficulties to deep network learning, which in turn leads to poor image segmentation results. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a semantic segmentation method and system based on deep convolutional neural network.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A semantic segmentation method based on a deep convolutional neural network includes model training and model application. The model training includes the following steps:

[0007] Obtain a semantic segmentation image dataset and preprocess the images in the semantic segmentation dataset; build a semantic segmentation network model, use the variant ResNet50 backbone network as the encoder, and use a multi-scale mixed pool structure and a feature attention fusion module as the decoder. The encoder extracts features from the input image to obtain a feature map, and the decoder obtains a segmentation result map based on the feature map; use the preprocessed semantic segmentation image dataset to perform model training and network parameter setting for the semantic segmentation network model to obtain a trained semantic segmentation network model;

[0008] The specific application of the model is: use the trained semantic segmentation network model to perform semantic segmentation on the image and obtain the segmentation result map.

[0009] Furthermore, the constructed semantic segmentation network model uses a variant ResNet50 backbone network as an encoder. The ResNet50 backbone network includes 4 layers. The input image is sent to the first layer, and the output of each layer is sent to the next layer. The variant ResNet50 backbone network is a ResNet50 backbone network including a channel attention module and the third and fourth layers have hole convolutions. The outputs of the second and third layers of the backbone network are sent to the channel attention module, and the output of the channel attention module and the output of the third layer of the backbone network are combined using the Element-wise Sum method and sent to the fourth layer of the backbone network together. The output of the fourth layer of the backbone network is the feature map extracted by the encoder.

[0010] Furthermore, the constructed semantic segmentation network model adopts a multi-scale mixed pool structure and a feature attention fusion module as a decoder. The input of the multi-scale mixed pool structure is the output of the fourth layer of the backbone network, and the output of the multi-scale mixed pool structure and the output of the third layer of the backbone network are sent to the feature attention fusion module together; the multi-scale mixed pool structure includes a conventional pooling module and an unconventional pooling module, which respectively target the characteristics of different objects and have a certain improvement in the segmentation of objects with irregular shapes.

[0011] Furthermore, the conventional pooling module includes m k×k adaptive average pooling blocks, k>0, and can be 1, 2, ..., 6, etc. The feature map output by the fourth layer of the backbone network is sent to the m adaptive average pooling blocks after dimensionality reduction; the unconventional pooling module includes s×1 pooling blocks and 1×s pooling blocks, and the feature map output by the fourth layer of the backbone network is sent to s×1 pooling blocks and 1×s pooling blocks after dimensionality reduction, and then the output of the unconventional pooling module is restored to the size of s×s after matrix multiplication and convolution; finally, the feature map output by the conventional pooling module and the feature map output by the unconventional pooling module are restored to the size of the input feature map after upsampling, and the output of the multi-scale mixed pooling structure, that is, the feature map after mixed pooling, is obtained after merging and dimensionality reduction.

[0012] Furthermore, the feature attention fusion module is specifically described as follows: the output of the multi-scale mixed pool structure is the feature map X1, the output of the third layer of the backbone network is the feature map X2, and the feature map X2 undergoes global average pooling, convolution, batch normalization and Sigmoid activation function in the feature attention fusion module to generate an attention coefficient, which is used to adjust each channel of the feature map X1. The feature attention fusion module compresses and excites the feature map output by the third layer of the backbone network, and generates an attention coefficient for weighting the feature map output by the multi-scale mixed pool structure, further enhancing the segmentation accuracy of the semantic segmentation network model.

[0013] Furthermore, the constructed semantic segmentation network model also includes a context embedding block, a global average pooling layer and a classification module. The feature map output by the encoder is sent to the context embedding block, and then passes through the global average pooling layer and the classification module in sequence. Among them, the classification module includes a fully connected layer and a Sigmoid function. In the network training stage, a fully connected layer is used to learn the global information of the corresponding features to emphasize beneficial features and suppress useless features. The output of the classification module and the output of the feature attention fusion module are element-wise multiplied to adjust the relevant parameters of the semantic segmentation network model, optimize the network performance, refine the image segmentation results, and obtain the segmentation result map.

[0014] Furthermore, two types of auxiliary losses are introduced. The output of the classification module is used together with the Classification label to assist in classification and further refine the image segmentation results. The output of the semantic segmentation network model, i.e. the segmentation result map, is used together with the Segmentation label to assist in segmentation and adjust the training parameters of the semantic segmentation network model.

[0015] Furthermore, the images in the semantic segmentation dataset are preprocessed by shearing, flipping, translating and scaling the images and masks in the semantic segmentation dataset to expand the semantic segmentation dataset.

[0016] Furthermore, the model training and network parameter setting of the semantic segmentation network model using the preprocessed semantic segmentation image dataset are as follows:

[0017] The images in the preprocessed semantic segmentation image dataset are used as input images, and the size of the input images is unified. The weight initialization method in the semantic segmentation network model is Kaiming. The semantic segmentation network model is trained using the stochastic gradient descent algorithm with momentum. The number of iterations is 30,000, the weight decay is 1e-5, the momentum is 0.9, the batch size is 4, the initial learning rate is 0.001, and the learning rate iteration strategy is "Poly".

[0018] A semantic segmentation system based on deep convolutional neural network, comprising:

[0019] An image acquisition module is used to acquire scene images and pre-process the scene images;

[0020] The semantic segmentation module uses the trained semantic segmentation network model to perform semantic segmentation on the scene image and output the segmentation result map;

[0021] The training process of the semantic segmentation network model is specifically as follows:

[0022] A semantic segmentation image dataset is obtained, and images in the semantic segmentation dataset are preprocessed; a semantic segmentation network model is built, a variant ResNet50 backbone network is used as an encoder, and a multi-scale mixed pool structure and a feature attention fusion module are used as a decoder. The encoder extracts features from the input image to obtain a feature map, and the decoder obtains a segmentation result map based on the feature map; the preprocessed semantic segmentation image dataset is used to perform model training and network parameter setting of the semantic segmentation network model to obtain a trained semantic segmentation network model.

[0023] Furthermore, the constructed semantic segmentation network model uses a variant of the ResNet50 backbone network as an encoder. The variant of the ResNet50 backbone network is a ResNet50 backbone network that includes a channel attention module and has hole convolutions in the third and fourth layers. The outputs of the second and third layers of the backbone network are sent to the channel attention module, and the output of the channel attention module and the output of the third layer of the backbone network are combined using the Element-wise Sum method and sent to the fourth layer of the backbone network together. The output of the fourth layer of the backbone network is the feature map extracted by the encoder.

[0024] Compared with the prior art, the present invention has improved the encoding and decoding structures, and designed some special modules to fully extract the relevant information between and within features. These modules can jointly extract rich multi-scale information and global information of different images, and can realize dense classification of each pixel. Among them, the two-input channel attention module can capture the internal correlation of the encoding stage; the multi-scale mixed pool structure can effectively capture the multi-scale features of different images; the feature attention fusion module can capture the external correlation between the encoding and decoding stages; the classification module can further optimize the network performance and refine the image segmentation results. Therefore, the present invention can efficiently and accurately perform semantic segmentation on images. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Schematic diagram of the semantic segmentation network model;

[0026] Figure 2Schematic diagram of the multi-scale mixing pool structure;

[0027] Figure 3 This is the training flow chart of the semantic segmentation network model;

[0028] Figure 4 This is the segmentation result map obtained using the semantic segmentation network model;

[0029] Figure 5 This is the segmentation result map obtained using the semantic segmentation network model;

[0030] Figure 6 This is the segmentation result map obtained using the semantic segmentation network model;

[0031] Figure 7 This is the segmentation result map obtained using the semantic segmentation network model; DETAILED DESCRIPTION

[0032] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0033] Embodiment 1:

[0034] A semantic segmentation method based on deep convolutional neural network, including model training and model application, model training such as Figure 3 As shown, this embodiment uses Python language as the basis and open source Pytorch as the neural network framework to complete the construction of the semantic segmentation network model. The semantic segmentation network model is trained through the semantic segmentation image dataset to find the optimal model parameters, including the following steps:

[0035] Obtain a semantic segmentation image dataset and preprocess the images in the semantic segmentation dataset; build a semantic segmentation network model, such as Figure 1 As shown in the figure, the variant ResNet50 backbone network is used as the encoder, and the multi-scale mixed pool structure and feature attention fusion module are used as the decoder. The encoder extracts features from the input image to obtain a feature map, and the decoder obtains a segmentation result map based on the feature map. The preprocessed semantic segmentation image dataset is used to perform model training and network parameter setting of the semantic segmentation network model to obtain a trained semantic segmentation network model.

[0036] The specific application of the model is: use the trained semantic segmentation network model to perform semantic segmentation on the image and obtain the segmentation result map.

[0037] Among them, the preprocessing of images in the semantic segmentation dataset is specifically: cutting, flipping, translating and scaling the images and their masks in the semantic segmentation dataset, such as scaling the length and width according to a preset scaling ratio, etc., expanding the semantic segmentation dataset can improve the generalization ability of the semantic segmentation network model.

[0038] Among them, the semantic segmentation network model built is as follows Figure 1 As shown:

[0039] 1) The variant ResNet50 backbone network is used as the encoder. The ResNet50 backbone network consists of 4 layers. The input image is sent to the first layer, and the output of each layer is sent to the next layer. The variant ResNet50 backbone network is a ResNet50 backbone network containing a channel attention module and the third and fourth layers have dilated convolutions. The outputs of the second and third layers of the backbone network are sent to the channel attention module. The output of the channel attention module and the output of the third layer of the backbone network are combined using the Element-wiseSum method and sent together to the fourth layer of the backbone network. The output of the fourth layer of the backbone network is the feature map extracted by the encoder.

[0040] 2) A multi-scale mixed pool structure and a feature attention fusion module are used as the decoder. The input of the multi-scale mixed pool structure is the output of the fourth layer of the backbone network. The output of the multi-scale mixed pool structure and the output of the third layer of the backbone network are sent to the feature attention fusion module together; the multi-scale mixed pool structure includes a conventional pooling module and an unconventional pooling module, which target the characteristics of different objects respectively, and have a certain improvement in the segmentation of objects with irregular shapes.

[0041] The multi-scale mixing pool structure is as follows Figure 2 As shown in the figure, the feature map output by the encoder is fed into the multi-scale hybrid pooling structure. The conventional pooling module includes m k×k adaptive average pooling blocks, k>0, which can be 1, 2, ..., 6, etc. The feature map output by the fourth layer of the backbone network is fed into the m adaptive average pooling blocks after dimension reduction; the unconventional pooling module includes s×1 pooling blocks and 1×s pooling blocks. The feature map output by the fourth layer of the backbone network is fed into s×1 pooling blocks and 1×s pooling blocks after dimension reduction, and then the output of the unconventional pooling module is restored to the size of s×s after matrix multiplication and convolution; finally, the feature map output by the conventional pooling module and the feature map output by the unconventional pooling module are restored to the size of the input feature map after upsampling, that is, the same size as the feature map output by the encoder, and then merged and reduced in dimension to obtain the output of the multi-scale hybrid pooling structure, that is, the feature map after hybrid pooling, which is fed into the feature attention fusion module.

[0042] The feature attention fusion module is described as follows: the output of the multi-scale mixed pool structure is the feature map X1, and the output of the third layer of the backbone network is the feature map X2. The feature map X2 undergoes global average pooling, convolution, batch normalization, and Sigmoid activation function in the feature attention fusion module to generate an attention coefficient, which is used to adjust each channel of the feature map X1. The feature attention fusion module compresses and excites the feature map output by the third layer of the backbone network, and generates an attention coefficient to weight the feature map output by the multi-scale mixed pool structure, further enhancing the segmentation accuracy of the semantic segmentation network model.

[0043] 3) The constructed semantic segmentation network model also includes a context embedding block, a global average pooling layer and a classification module. The feature map output by the encoder is sent to the context embedding block, and then passes through the global average pooling layer and the classification module in sequence. Among them, the classification module includes a fully connected layer and a Sigmoid function. In the network training stage, a fully connected layer is used to learn the global information of the corresponding features to emphasize beneficial features and suppress useless features. The output of the classification module and the output of the feature attention fusion module are combined using Element-wise product, that is, element-wise multiplication is performed to adjust the relevant parameters of the semantic segmentation network model, optimize network performance, refine the image segmentation results, and obtain the segmentation result map. The size of the segmentation result map is restored to the size of the input image.

[0044] Two types of auxiliary losses are introduced: Classification label is the classification label, and Segmentation label is the semantic label. The output of the classification module is used with the Classification label to assist in classification, and further refine the image segmentation results. The output of the semantic segmentation network model, that is, the segmentation result map, is used with the Segmentation label to assist in segmentation and adjust the training parameters of the semantic segmentation network model.

[0045] The images in the semantic segmentation dataset are preprocessed by shearing, flipping, translating and scaling the images and masks in the semantic segmentation dataset to expand the semantic segmentation dataset.

[0046] The model training and network parameter setting of the semantic segmentation network model using the preprocessed semantic segmentation image dataset are as follows:

[0047] The images in the preprocessed semantic segmentation image dataset are used as input images, and the size of the input images is unified. The weight initialization method in the semantic segmentation network model is Kaiming. The semantic segmentation network model is trained using the stochastic gradient descent algorithm with momentum. The number of iterations is 30,000, the weight decay is 1e-5, the momentum is 0.9, the batch size is 4, the initial learning rate is 0.001, and the learning rate iteration strategy is "Poly".

[0048] A semantic segmentation system based on deep convolutional neural network, comprising:

[0049] An image acquisition module is used to acquire scene images and pre-process the scene images;

[0050] The semantic segmentation module uses the trained semantic segmentation network model to perform semantic segmentation on the scene image and output the segmentation result map;

[0051] The training process of the semantic segmentation network model is as follows:

[0052] Obtain a semantic segmentation image dataset and preprocess the images in the semantic segmentation dataset; build a semantic segmentation network model, use the variant ResNet50 backbone network as the encoder, and adopt a multi-scale mixed pool structure and feature attention fusion module as the decoder. The encoder extracts features from the input image to obtain a feature map, and the decoder obtains the segmentation result map based on the feature map; use the preprocessed semantic segmentation image dataset to perform model training and network parameter setting of the semantic segmentation network model to obtain a trained semantic segmentation network model.

[0053] Taking the real-time semantic segmentation of video images as an example, the camera function in the OpenCV library is used to read the real-time picture, and then the read video is processed frame by frame and the image format is converted. The preprocessed video image is input frame by frame into the trained semantic segmentation network model, and the corresponding segmentation mask is output in real time. Figure 4 , Figure 5 , Figure 6 and Figure 7 The segmentation results of different street view images are given.

[0054] The present invention proposes segmentation methods such as a variant-based ResNet50 backbone network and hybrid pooling. During the training process, the preprocessed semantic segmentation image is sent to a deeper encoder in an effort to enable the network to learn more image feature information. In addition, a channel attention module is added, and transfer learning is used to initialize the network architecture to accelerate the learning of image feature information, thereby accelerating model convergence.

[0055] In the decoding stage, a multi-scale mixed pooling structure is designed, including conventional pooling modules and unconventional pooling modules. These mixed pooling modules (Multi-scale Mixed Pooling module, MMP) of different scales are connected in parallel to achieve efficient aggregation of multi-scale feature maps. A feature attention fusion module is also built after the multi-scale mixed pooling structure to compress and stimulate the feature map output by the third layer of the backbone network in the encoding stage, and generate attention coefficients for weighting the feature map output by the multi-scale mixed pooling structure in the decoding stage, further enhancing the accuracy of network segmentation.

[0056] In addition, a context embedding block, a global average pooling layer and a classification module are established in parallel with the decoder. The output of the classification module is subjected to an element-wise product operation with the output of the feature attention fusion module, that is, element-wise multiplication, to obtain a segmentation result map, which enhances the segmentation effect.

[0057] Generally speaking, although objects of different categories may have similar local features, and related detail information increases the difficulty of segmentation, the present invention can effectively realize the connection between encoding and decoding, fully extract relevant information between different stages and within the same stage, effectively fuse low-level and high-level features, obtain long-range dependencies and rich context information, and can segment related images more efficiently and accurately. The present invention can efficiently and accurately realize the semantic segmentation task of related images.

[0058] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A semantic segmentation method based on deep convolutional neural network, characterized in that: Including model training and model application, model training includes the following steps: Obtain a semantic segmentation image dataset and preprocess the images in the semantic segmentation dataset; build a semantic segmentation network model, use the variant ResNet50 backbone network as the encoder, and use a multi-scale mixed pool structure and a feature attention fusion module as the decoder. The encoder extracts features from the input image to obtain a feature map, and the decoder obtains a segmentation result map based on the feature map; use the preprocessed semantic segmentation image dataset to perform model training and network parameter setting for the semantic segmentation network model to obtain a trained semantic segmentation network model; The specific application of the model is: use the trained semantic segmentation network model to perform semantic segmentation on the image and obtain the segmentation result map; The constructed semantic segmentation network model uses a variant ResNet50 backbone network as an encoder. The variant ResNet50 backbone network is a ResNet50 backbone network that includes a channel attention module and has dilated convolutions in the third and fourth layers. The outputs of the second and third layers of the backbone network are sent to the channel attention module. The output of the channel attention module and the output of the third layer of the backbone network are combined in an Element-wise Sum manner and sent together to the fourth layer of the backbone network. The output of the fourth layer of the backbone network is the feature map extracted by the encoder. The constructed semantic segmentation network model adopts a multi-scale mixed pool structure and a feature attention fusion module as a decoder. The multi-scale mixed pool structure includes a conventional pooling module and an unconventional pooling module. The input of the multi-scale mixed pool structure is the output of the fourth layer of the backbone network. The output of the multi-scale mixed pool structure and the output of the third layer of the backbone network are sent to the feature attention fusion module together. The conventional pooling module includes m k×k adaptive average poolings, k>0, and the unconventional pooling module adopts s×1 and 1×s pooling modes.

2. The semantic segmentation method based on deep convolutional neural network according to claim 1, characterized in that: The feature attention fusion module is specifically described as follows: the output of the multi-scale mixed pool structure is the feature map X1, the output of the third layer of the backbone network is the feature map X2, and the feature map X2 undergoes global average pooling, convolution, batch normalization and Sigmoid activation function in the feature attention fusion module to generate an attention coefficient, which is used to adjust each channel of the feature map X1.

3. The semantic segmentation method based on deep convolutional neural network according to claim 1, characterized in that: The constructed semantic segmentation network model also includes a context embedding block, a global average pooling layer and a classification module. The feature map output by the encoder is sent to the context embedding block, and then passes through the global average pooling layer and the classification module in sequence. The classification module includes a fully connected layer and a Sigmoid function; the output of the classification module and the output of the feature attention fusion module are element-wise multiplied to obtain the segmentation result map.

4. The semantic segmentation method based on deep convolutional neural network according to claim 1, characterized in that: The images in the semantic segmentation dataset are preprocessed by shearing, flipping, translating and scaling the images and masks in the semantic segmentation dataset to expand the semantic segmentation dataset.

5. The semantic segmentation method based on deep convolutional neural network according to claim 1, characterized in that: The model training and network parameter setting of the semantic segmentation network model using the preprocessed semantic segmentation image dataset are as follows: The images in the preprocessed semantic segmentation image dataset are used as input images, and the size of the input images is unified. The weight initialization method in the semantic segmentation network model is Kaiming. The semantic segmentation network model is trained using the stochastic gradient descent algorithm with momentum. The number of iterations is 30,000, the weight decay is 1e-5, the momentum is 0.9, the batch size is 4, the initial learning rate is 0.001, and the learning rate iteration strategy is "Poly".

6. A semantic segmentation system based on deep convolutional neural network, characterized in that: A semantic segmentation method based on a deep convolutional neural network according to any one of claims 1 to 5, comprising: An image acquisition module is used to acquire scene images and pre-process the scene images; The semantic segmentation module uses the trained semantic segmentation network model to perform semantic segmentation on the scene image and output the segmentation result map; The training process of the semantic segmentation network model is specifically as follows: A semantic segmentation image dataset is obtained, and images in the semantic segmentation dataset are preprocessed; a semantic segmentation network model is built, a variant ResNet50 backbone network is used as an encoder, and a multi-scale mixed pool structure and a feature attention fusion module are used as a decoder. The encoder extracts features from the input image to obtain a feature map, and the decoder obtains a segmentation result map based on the feature map; the preprocessed semantic segmentation image dataset is used to perform model training and network parameter setting of the semantic segmentation network model to obtain a trained semantic segmentation network model.

Images