A semantic segmentation method based on same down-sampling frequency

Abstract

This invention discloses a semantic segmentation method based on the same downsampling frequency. To address the problem that existing neural networks cannot effectively achieve multi-scale feature fusion, this method uses ResNet-50 for feature extraction, performing five downsampling passes to preserve the main feature information in the image. After each downsampling, the information passes through a corresponding feature processing network to increase the receptive field of the features. The low-level information after the first downsampling by the ResNet50 network is processed by a U-Net network with four downsampling passes, and then merged with the upsampling information through cross-layer connections. The information after the second downsampling by the ResNet50 network is processed by a U-Net network with three downsampling passes, and then information is fused. During feature upsampling, cross-layer connections are used to fuse the low-level fine-grained information and high-level semantic information processed in the semantic segmentation network.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of image recognition, and particularly relates to a semantic segmentation method based on the same down-sampling frequency, which is used to realize target segmentation in an image recognition task. BACKGROUND

[0002] With the continuous convolution and pooling of the deep neural network, and since the features of each layer of the neural network are calculated from the previous features, the receptive field of the features will become larger and larger, and the semantic information representation ability will become stronger and stronger. However, the continuous convolution and pooling will also cause the loss of image information. The deeper the network is, the larger the receptive field of the features is, and the greater the loss of information is. Therefore, the following conclusion can be drawn: the receptive field of the shallow features is small, and the semantic information representation ability is not strong, but the shallow features have strong geometric detail information representation ability because they do not lose too much image information, and thus are suitable for processing small targets; the receptive field of the high-level features is large, and the semantic representation ability is strong, but since more geometric detail information is lost, the high-level features are suitable for processing large targets.

[0003] In 2016, Liu W et al. proposed the SSD algorithm to realize the target detection task. The SSD algorithm uses multi-scale feature maps to predict targets, and uses high-level feature information with a large receptive field to detect large targets, and uses shallow feature information with a small receptive field to detect small targets. However, when using shallow feature information to detect small targets, since the semantic information representation ability of the shallow feature information is weak, and lacks the guidance of high-level semantic information, the classification result of the SSD for small targets is not good.

[0004] ASPP is another way to obtain multi-scale features. The principle of ASPP is to perform parallel sampling on the given input with different sampling rates of atrous convolution to obtain features with different receptive fields. In 2018, Chen et al. proposed the DeepLabv3+ algorithm, which obtains multi-scale features through the ASPP structure, and finally achieves good semantic segmentation effect. However, the problem of ASPP is that the atrous convolution will cause the discontinuity of image features in spatial information, which may ignore the image information of small targets, so that the ASPP can well segment large targets, but the segmentation effect of small targets is not satisfactory.

[0005] The skip connection of the same level of the network is a better way to fuse multi-scale features. The U-Net algorithm fuses the fine-grained surface information of the bottom layer and the semantic information of the high layer through the skip connection of the same level, thereby fusing the features of different receptive fields, and inputting the fused features to the next layer of the network, and finally obtaining a relatively ideal segmentation image. Due to the ability of the U-Net algorithm to fuse multi-scale features, and the structure of the skip connection to a certain extent to make up for the information loss caused by down-sampling, the network can finally obtain a clear edge segmentation image, so that the U-Net algorithm is widely used in the field of image segmentation.

[0006] The U-Net network also has disadvantages, mainly in that the U-Net network can only fuse the multi-scale features of the bottom layer through the skip connection when down-sampling each time, and the number of fusions depends on the number of down-sampling. Since the receptive field of the feature is too small when the network is initially down-sampled, the recognition ability of the target is weak. In order to reduce the calculation loss, the network generally needs to be down-sampled for 2-3 times before the skip connection is performed, which leads to insufficient multi-scale features obtained by the U-Net network. If the U-Net network can fuse more multi-scale feature information while avoiding image information loss, the recognition ability of the network for different scale targets can be greatly improved.

[0007] In summary, it is necessary to design a semantic segmentation method that fuses multi-scale features. SUMMARY

[0008] The present application aims at the shortcomings of the existing semantic segmentation method and the problem that the existing neural network algorithm cannot well fuse multi-scale features, and provides a new method of multi-scale feature fusion, namely a semantic segmentation method based on the same down-sampling frequency, which realizes accurate segmentation of the target to be detected in the picture, and has better integrity for the segmentation of large targets, and can realize accurate identification of small targets in the picture, avoiding ignoring small targets in image recognition.

[0009] The technical scheme adopted by the present application comprises the following steps:

[0010] The semantic segmentation method based on the same down-sampling frequency of the present application has the specific process as follows:

[0011] Step 1, constructing a semantic segmentation data set and labeling the pictures in the data set.

[0012] Step 2, inputting a three-channel picture to be identified into the semantic segmentation network.

[0013] Step 3, using the image feature extraction network ResNet-50 to realize high-level semantic information extraction of the picture to be identified.

[0014] Step 3.1, in the process of using ResNet-50 to extract features of the picture to be identified, 5 times of down-sampling are used to retain the main feature information of the picture and reduce the calculation loss of the neural network.

[0015] Step 3.2, after each down-sampling, the information will pass through a corresponding feature processing network to improve the receptive field of the down-sampled features. The bottom layer information after the first down-sampling of the ResNet50 network will pass through the U-Net network with 4 times of down-sampling, and then be combined with the information after up-sampling through cross-layer connection. The information after the second down-sampling of the ResNet50 network will pass through the U-Net network with 3 times of down-sampling, and then be combined with the information after up-sampling, and so on.

[0016] Step 4, the bilinear interpolation method is used to up-sample the extracted high-level semantic information to restore the resolution of the picture.

[0017] Step 5, a total of five times of up-sampling will be performed, and feature fusion will be performed after each up-sampling.

[0018] Step 5.1, in the process of up-sampling the features, the cross-layer connection method is used to fuse the processed bottom layer fine-grained information and high-level semantic information in the semantic segmentation network. For example, the features after the first up-sampling will be combined with the features after the fourth down-sampling, the features after the second up-sampling will be combined with the features after the third down-sampling, and so on.

[0019] Step 6, a result image with the same resolution as the input picture is generated, and the difference between the result image and the target image is calculated using the loss function Dice Loss, and the error is returned to the network, and the parameters of the neural network are adjusted using the back propagation algorithm, and finally the semantic segmentation network can correctly segment the target in the picture to be identified.

[0020] Advantages of the present application

[0021] (1) The image feature fusion method proposed in the present application can fuse multi-scale feature information, fuse high-level semantic information and bottom layer fine-grained surface information, use high-level semantic information to guide the bottom layer fine-grained information to correctly detect small targets, and solve the problem that the existing method cannot well fuse multi-scale features.

[0022] (2) The semantic segmentation method based on the same down-sampling frequency proposed in the present application can effectively solve the problem of insufficient integrity of the semantic segmentation method for target segmentation, and can accurately identify small targets in the picture, solving the problem that the existing algorithm cannot well identify small targets. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is an ASPP module schematic diagram;

[0024] Figure 2 is a cross-layer connection schematic diagram;

[0025] Figure 3 is an algorithm framework flow chart. DETAILED DESCRIPTION

[0026] In a neural network, the existing multi-scale feature fusion method is mainly through the feature pyramid and the cross-layer connection. The pyramid structure expands the receptive field of the feature by using different sampling methods on the same feature map, so that the network has better recognition effect on large targets, but is not conducive to small target segmentation. The cross-layer connection (Skip Connection) of the same level of the network is a better way to fuse multi-scale features, which fuses high-level semantic information and low-level fine-grained surface information, uses high-level semantic information to guide the low-level fine-grained information to detect small targets correctly, and can better solve the problem that the above algorithm is not conducive to small target segmentation. However, the problem of cross-layer connection is that the high-level semantic information with large receptive field is directly fused with the low-level fine-grained surface information, the fusion method is too rough, which will bring a lot of noise, and finally affect the accuracy of the semantic segmentation task.

[0027] The present application fully considers the shortcomings of feature fusion in the existing semantic segmentation network. In the process of using the ResNet-50 network to complete feature extraction, the feature information after each downsampling is processed by a corresponding neural network, so as to improve the receptive field of the low-level fine-grained surface information, reduce the noise when fusing the high-level semantic information and the low-level fine-grained surface information, and obtain more multi-scale information, so that the semantic segmentation network can have excellent recognition ability for large targets and small targets, and improve the accuracy of the network.

[0028] The specific process of the semantic segmentation method based on the same downsampling frequency is as follows:

[0029] Step 1, construct a semantic segmentation data set, and label the pictures in the data set.

[0030] Step 2, input a three-channel picture to be recognized into the semantic segmentation network.

[0031] Step 3, use the image feature extraction network ResNet-50 to realize high-level semantic information extraction of the picture to be recognized.

[0032] Step 3.1. In the process of feature extraction using ResNet-50 for the picture to be identified, 5 times of down-sampling are performed to preserve the main feature information of the picture and reduce the computational loss of the neural network.

[0033] Step 3.2. After each down-sampling, the information is processed through a corresponding feature processing network to improve the receptive field of the down-sampled features. The bottom layer information after the first down-sampling of the ResNet50 network is processed through a U-Net network with 4 times of down-sampling, and then merged with the up-sampled information through cross-layer connection. The information after the second down-sampling of the ResNet50 network is processed through a U-Net network with 3 times of down-sampling, and then merged with the up-sampled information, and so on.

[0034] Step 4. The bilinear interpolation method is used to up-sample the extracted high-level semantic information to restore the resolution of the picture.

[0035] Step 5. A total of 5 times of up-sampling are performed, and feature fusion is performed after each up-sampling.

[0036] Step 5.1. In the process of up-sampling the features, the cross-layer connection method is used to fuse the processed bottom layer fine-grained information and high-level semantic information in the semantic segmentation network. For example, the features after the first up-sampling are fused with the processed fourth down-sampled feature information, the features after the second up-sampling are fused with the processed third down-sampled feature information, and so on.

[0037] Step 6. A result image with the same resolution as the input picture is generated, and the difference between the result image and the target image is calculated using the loss function Dice Loss. The error is returned to the network, and the parameters of the neural network are adjusted using the backpropagation algorithm. Finally, the semantic segmentation network can correctly segment the target in the picture to be identified.

Claims

1. A method for semantic segmentation based on the same down-sampling frequency, characterized in that, The steps are: step 1, constructing a semantic segmentation dataset, and labeling the pictures in the dataset; step 2, inputting a three-channel picture to be identified into the semantic segmentation network; step 3, using the image feature extraction network ResNet-50 to realize high-level semantic information extraction of the picture to be identified; step 4, using the bilinear interpolation method to up-sample the extracted high-level semantic information to restore the resolution of the picture; step 5, a total of five up-sampling will be performed, and feature fusion will be performed after each up-sampling; step 6, generating a result picture with the same resolution as the input picture, and using the loss function Dice Loss to calculate the difference between the result picture and the target picture, and returning the error to the network, adjusting the parameters of the neural network using the back propagation algorithm, and finally the semantic segmentation network can correctly segment the target in the picture to be identified; Step 3 uses ResNet-50 to extract features from the picture to be identified, which will go through 5 times of down-sampling to preserve the main feature information in the picture and reduce the calculation loss of the neural network. The information after each down-sampling will be processed by a corresponding feature processing network to improve the receptive field of the down-sampled features. The bottom layer information after the first down-sampling of the ResNet50 network will be processed by the U-Net network with 4 times of down-sampling, and then merged with the up-sampled information through cross-layer connection. The information after the second down-sampling of the ResNet50 network will be processed by the U-Net network with 3 times of down-sampling, and then information fusion will be performed. The same applies to the subsequent steps. The algorithm structure of the semantic segmentation network consists of an encoder, a multi-scale feature enhancement module and a decoder. The input three-channel image first enters the encoder with ResNet-50 as the backbone, and then goes through one convolution and one maximum pooling for preliminary down-sampling, and then completes five times of down-sampling through five stages of residual blocks to extract multi-scale features from shallow texture to deep semantics. The features after each down-sampling will be sent to a simplified U-Net feature processing network. The down-sampling times of different layers of the simplified U-Net are 4, 3, 2, 1 and 0 respectively to ensure that all branches reach the same down-sampling depth. The decoder part performs five times of up-sampling. Each up-sampling first uses bilinear interpolation to double the current feature, and then performs cross-layer connection with the feature processed by the simplified U-Net in the corresponding layer of the encoder. After concatenation in the channel dimension, convolution fusion is performed to recover the image resolution layer by layer. The specific correspondence is: the first up-sampling fuses the fourth down-sampled feature, the second fuses the third, the third fuses the second, the fourth fuses the first, and the fifth up-sampling is only used to restore the original image size and does not perform fusion. Finally, a 1x1 convolution is used to map to the class channel to output the final segmentation result.

Images