A semantic segmentation method and system based on feature clustering

Abstract

The application provides a feature clustering-based semantic segmentation method and system, which comprises the following steps: a feature extraction layer in a semantic segmentation model extracts deep semantic features and shallow detail features of a training image; a first convolution classifier in the semantic segmentation model classifies the deep semantic features to obtain a first classification result of each pixel in the training image; the first classification result is normalized to obtain a probability of each pixel belonging to a category; a dot product of the probability and the deep semantic features is calculated to obtain a class feature of each category; the deep semantic features and the shallow detail features are fused to obtain pixel-level features; a category corresponding to a maximum similarity of each pixel in the training image to each class feature is taken as a second classification result according to the pixel-level features; a second convolution classifier in the semantic segmentation model classifies the pixel-level features to obtain a third classification result of each pixel in the training image; and the semantic segmentation model is trained according to a category label and each classification result to perform semantic segmentation on a specified image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of planar image semantic segmentation in computer vision, and particularly relates to a semantic segmentation method and system based on feature clustering. BACKGROUND

[0002] Semantic segmentation is to classify each pixel in an image. Semantic segmentation needs long-range context information, and the receptive field of single-layer convolution is limited, so context modules are often used after the backbone network to extract context information, and then the enhanced features are classified by a convolution classifier to realize pixel classification. Currently, two kinds of context information, spatial context and semantic context, are commonly used.

[0003] The existing scheme 1 is spatial context. Spatial context refers to using context of different positions and different scales to enhance the features extracted by the backbone network (convolution network). Common spatial context networks are as shown in the following figure: Figure 1 The input image is first subjected to feature extraction by the backbone network, then different scale features are extracted by using multiple different size pooling layers, and finally the enhanced features are subjected to a convolution classifier to obtain the semantic segmentation result.

[0004] The existing scheme 2 is semantic context. Semantic context refers to a way of enhancing the features extracted by the backbone network using the features of pixels / regions of the same semantics as the pixels. Common semantic contexts are as shown in the following figure: Figure 2 The similarity of pixel features of different positions and different channels is calculated for the features extracted by the backbone network, and the enhanced features are obtained by weighting the features of other positions based on the similarity, and finally a convolution classifier is used to obtain the semantic segmentation result.

[0005] The above existing technologies mainly have two problems:

[0006] 1) The context module in the above method directly acts on the features extracted by the backbone network, lacks effective supervision, and is easy to introduce irrelevant context information, leading to feature confusion;

[0007] 2) The above method directly uses the enhanced features to obtain the semantic segmentation result by a convolution classifier, and the classifier does not consider the context relationship of the pixels. SUMMARY

[0008] The present application aims to solve the problems of feature confusion caused by the context module and the lack of context relationship in the classification stage, and proposes a semantic segmentation method based on feature clustering, which comprises:

[0009] Step 1, obtaining a training image with a labeled class label, a feature extraction layer in the semantic segmentation model extracts deep semantic features and shallow detail features of the training image, a first convolutional classifier in the semantic segmentation model classifies the deep semantic features to obtain a first classification result of each pixel in the training image, normalizes the first classification result to obtain the probability of each pixel belonging to each semantic class, and performs dot product on the probability and the deep semantic features to obtain the class feature of each class;

[0010] Step 2, fusing the deep semantic features and the shallow detail features to obtain pixel-level features; according to the pixel-level features, calculating the similarity of each pixel in the training image to each class feature, and taking the class corresponding to the maximum similarity as the second classification result of the current pixel; a second convolutional classifier in the semantic segmentation model classifies the pixel-level features to obtain a third classification result of the pixels in the training image;

[0011] Step 3, constructing a loss function according to the class label of the training image and the first classification result, the second classification result and the third classification result, training the semantic segmentation model, and performing semantic segmentation on the specified image using the trained semantic segmentation model.

[0012] The feature extraction layer includes a backbone network and a context module connected in series, and the context module is used to enhance the deep network features output by the backbone network to obtain the deep semantic features.

[0013] The step 1 includes:

[0014] The deep semantic features are pre-segmented by the following formula:

[0015] r=S(f h )

[0016] Where f h represents deep semantic features, r represents the first classification result, and S is the first convolutional classifier; and the first classification result is normalized by softmax to obtain the probability of each pixel belonging to each class:

[0017]

[0018] Where k represents a class, and i represents a pixel.

[0019] Dot product is performed on the deep semantic features f h and the probability m ik to obtain the class feature f k of each class:

[0020]

[0021] This step 2 includes:

[0022] The deep semantic feature and the shallow detail feature are fused by the following formula, thereby constructing the high-resolution pixel-level feature:

[0023] f p =C R (concat(C R (f l ),upsample(C R (f h )))

[0024] Where f p represents the final pixel feature, f h , f l represent deep semantic feature and shallow feature respectively, C R represent up-sampling convolution;

[0025] The similarity is obtained by the following formula:

[0026]

[0027] class i =argmax j (p i1 ,p i2 ,…p ij ,...,p iK )

[0028] Where p ik represents the similarity of pixel i and the kth class, class i represents the second classification result of pixel i.

[0029] The loss function includes cluster center loss, feature consistency loss, pre-segmentation loss and regularization loss.

[0030] Cluster center loss:

[0031]

[0032] Feature consistency loss:

[0033]

[0034] Pre-segmentation loss:

[0035]

[0036] Regularization loss:

[0037]

[0038] The final loss function is:

[0039]

[0040] The application also provides a semantic segmentation system based on feature clustering, comprising:

[0041] The feature extraction module is configured to obtain a training image with a labeled class label, and a feature extraction layer in the semantic segmentation model extracts deep semantic features and shallow detail features of the training image, a first convolutional classifier in the semantic segmentation model classifies the deep semantic features to obtain a first classification result of each pixel in the training image, and the first classification result is normalized to obtain a probability that each pixel belongs to each semantic class, and the probability and the deep semantic features are dot multiplied to obtain a class feature of each class.

[0042] The feature fusion module is configured to fuse the deep semantic features and the shallow detail features to obtain pixel-level features, calculate the similarity of each pixel in the training image to each class feature according to the pixel-level features, and take the class corresponding to the maximum similarity as a second classification result of the current pixel.

[0043] The model training module is configured to construct a loss function according to the class label of the training image and the first classification result, the second classification result, and the third classification result, train the semantic segmentation model, and perform semantic segmentation on a specified image using the trained semantic segmentation model.

[0044] The semantic segmentation system based on feature clustering, wherein the feature extraction layer comprises a backbone network and a context module connected in series, and the context module is configured to enhance the deep network features output by the backbone network to obtain the deep semantic features.

[0045] The semantic segmentation system based on feature clustering, wherein the feature extraction module is configured to:

[0046] The deep semantic features are pre-segmented by the following formula:

[0047] r=S(f h )

[0048] Where f h represents deep semantic features, r represents the first classification result, and S is the first convolutional classifier. Then, the first classification result is normalized by softmax to obtain the probability that each pixel belongs to each class:

[0049]

[0050] wherein k represents a class, and i represents a pixel;

[0051] performing dot product on the deep semantic feature f h and the probability m ik to obtain a class feature f k of each class:

[0052]

[0053] The feature fusion module is used to:

[0054] The deep semantic feature and the shallow detail feature are fused by the following formula, so as to construct a high-resolution pixel-level feature:

[0055] f p =C R (concat(C R (f l ), upsample(C R (f h )))

[0056] wherein f p represents a final pixel feature, f h , f l represent a deep semantic feature and a shallow feature respectively, and C R represents an up-sampling convolution;

[0057] The similarity is obtained by the following formula:

[0058]

[0059] class i =argmax j (p i1 , p i2 ,...p ij ,..., p iK )

[0060] wherein p ik represents a similarity between a pixel i and a k-th class, and class i represents the second classification result of the pixel i.

[0061] The semantic segmentation system based on feature clustering, wherein the loss function comprises a clustering center loss, a feature consistency loss, a pre-segmentation loss and a regularization loss;

[0062] The clustering center loss is:

[0063]

[0064] Feature consistency loss:

[0065]

[0066] Pre-segmentation loss:

[0067]

[0068] Regularization loss:

[0069]

[0070] Final loss function:

[0071]

[0072] The application further provides a storage medium for storing a program for executing any one of the feature clustering based semantic segmentation methods.

[0073] The application further provides a client for any one of the feature clustering based semantic segmentation systems.

[0074] From the above scheme, the application has the following advantages:

[0075] 1. The application forms a more compact remote sensing image feature space, reducing the difficulty of semantic segmentation. Figure 3 As shown, the features extracted at the same position by different methods are visualized, the left is the baseline method PSPNet, and the right is the result of the application, wherein the same gray represents the features of the same semantic category, and it can be seen from the result that, compared with the baseline method, the same pixel features extracted by the application are more compact, and the different pixel features are more dispersed, which greatly reduces the difficulty of semantic segmentation.

[0076] 2. The application significantly improves the accuracy of semantic segmentation in quantitative experiments.

[0077] Comparative experiment: Figure 4 The first row is the baseline method, and the second row is the application, wherein the convolution classifier in the baseline method is replaced by the proposed feature clustering method and supervised, and the experimental results show that, under the condition that the parameter quantity and the calculation amount of the application are close to those of the baseline method, the application significantly improves the performance of mIoU and mean-F1.

[0078] Loss function experiment: The baseline method directly uses cross-entropy to supervise the segmentation results of the convolution classifier, achieving a segmentation accuracy mIoU of 60.2%. Building upon this, the supervised clustering framework of this invention is introduced. The distance from pixel features to class features is calculated to obtain the final segmentation result, and supervision is provided through cluster center loss and pre-segmentation loss. This improves the segmentation accuracy from 60.2% to 62.1%. Furthermore, a regularization loss is introduced to map pixel features to a feature space suitable for classification, further improving the segmentation accuracy to 63.0%. Finally, the feature consistency loss compares the similarity between pixel features and class features in the feature space, forming a more compact remote sensing image feature space, thereby further improving the segmentation accuracy to 63.3%. This experiment fully demonstrates the importance of each loss in the supervision function of this invention.

[0079] 3. For example Figure 5 The invention shown improves the visualization results of semantic segmentation. It can be seen that, compared with the baseline scheme, the invention effectively improves the accuracy of segmentation of difficult semantic categories such as small objects and playgrounds.

[0080] In summary, this invention significantly improves semantic segmentation performance while maintaining a similar computational and parameter load to existing technologies. Attached Figure Description

[0081] Figure 1 This is a schematic diagram of the spatial context network structure;

[0082] Figure 2 This is a schematic diagram of a semantic context network structure;

[0083] Figure 3 This is a spatial comparison map of remote sensing image features;

[0084] Figure 4 and Figure 5 These are all comparison charts showing the accuracy of semantic segmentation.

[0085] Figure 6 A comparison chart showing the visualization results of semantic segmentation;

[0086] Figure 7 This is a schematic diagram of the overall structure of the present invention. Detailed Implementation

[0087] The inventors found that the features extracted by the context module of the existing method may be confused due to lack of effective supervision, and the network classifier module does not consider the context relationship when classifying pixels, which can be solved by modeling semantic segmentation as a feature clustering method, and a feature clustering network and a loss function for the clustering network are proposed. The method considers the context features when classifying, and uses real semantic segmentation labels to supervise it. Specifically, the application includes the following key technical points:

[0088] Key point 1: The application models semantic segmentation as a feature clustering problem, and proposes a feature clustering network structure, which considers the context relationship when classifying and effectively reduces the difficulty of pixel classification.

[0089] Key point 2: The application proposes a loss function for the clustering network, including a clustering center loss and a feature consistency loss, which forms a more compact feature space.

[0090] In order to make the above features and effects of the application more clear and easy to understand, the following embodiments are described in detail below, together with the accompanying drawings.

[0091] The overall structure of the application is shown in Figure 7 , which includes a feature clustering network structure and a feature clustering loss function. In the application, the modules that need to be trained include a backbone network, a context module and two convolutional classification modules, i.e. the semantic segmentation model of the application is composed of the above four modules.

[0092] Feature clustering network structure: including feature extraction module and supervised clustering module. The network extracts image features from the input image through the feature extraction module, and then classifies pixels through feature clustering. The feature clustering steps are as follows:

[0093] 1) Construct class features

[0094] Since the features extracted by the deep network ( Figure 7 The backbone network and the context module) have more rich semantic information and contain less noise, the application uses the features to construct robust features for each semantic class. First, the deep semantic features are classified by a plurality of layers of convolution, and the real semantic segmentation labels are used for supervision, so as to obtain the probability that each pixel belongs to a certain class. Then, the pixel probability and the deep semantic feature are dot multiplied to obtain the feature of each class. The above process is expressed by formula, first, the deep semantic feature is pre-segmented:

[0095] r=S(f h )

[0096] Where f hrepresents deep semantic features, r represents the coarse segmentation result, and S is the convolution classifier.

[0097] The coarse segmentation result is then normalized using softmax to obtain the probability that each pixel belongs to a certain semantic category:

[0098]

[0099] Where k represents the category, i represents the pixel, and HW represents the length and width of the input image, respectively.

[0100] Finally, the deep semantic features f h The probability m of a pixel belonging to a certain category ik By performing dot products, features for each class are constructed:

[0101]

[0102] 2) Construct pixel features

[0103] Semantic segmentation is a pixel-by-pixel task, and more image detail features are needed to obtain high-resolution prediction results. Therefore, this invention fuses deep semantic features and shallow detail features to construct high-resolution pixel-level features. This can be expressed as:

[0104] f p =C R (concat(C R (f l ), upsample(C R (f h )))

[0105] Where f p f represents the final pixel feature. h f l C represents deep semantic features and shallow features, respectively. R This represents an upsampled 1x1 convolution.

[0106] 3) Pixel classification

[0107] This invention obtains pixel categories by calculating the distance between pixel features and class features and then performs supervision, thereby grouping pixels of the same category together in the feature space. Specifically, after constructing effective pixel features and class features, the cosine similarity from each pixel to each semantic category feature is calculated, and the pixel with the highest similarity is taken as its semantic category. This can be expressed by the formula:

[0108]

[0109] class i =argmax j(p i1 , p i2 ,... p ij ,..., p iK )

[0110] where p ik represents the similarity of pixel i and the k-th class feature.class i represents the semantic class of pixel i.

[0111] Feature clustering loss function: In order to accurately and effectively supervise the feature clustering network, the present application proposes a loss function for the clustering network. The loss function includes four parts: cluster center loss, feature consistency loss, pre-segmentation loss and regularization loss.

[0112] Cluster center loss: for the clustering problem, first of all, the distance between pixel features and cluster centers needs to be shortened, so the present application supervises the similarity of pixel features and class features through the cross-entropy function. Since the softmax function in the cross-entropy function has the effect of highlighting the maximum value and suppressing other values, this loss can effectively shorten the distance between pixel features and features of the same class. In the formula is the class label of the i-th pixel in the training image. Specifically is a one-hot mode label, and k represents the real class of the i-th pixel is k. For example, if the real class of the pixel has a total of 5 classes, k = 2 means that the 2nd position is 1,

[0113]

[0114] Feature consistency loss: for feature clustering, pixel features from the same class should appear in the adjacent positions in the feature space, so the present application supervises the extracted global class features and pixel features through the contrastive learning loss.

[0115]

[0116] In the formula, q represents the feature of the sampled pixel, f+ represents the positive sample pixel feature of other positions belonging to the same class as the pixel, f- represents the negative sample pixel feature of other positions belonging to different classes as the pixel, t represents the temperature hyperparameter, which is used to control the attention degree to difficult negative sample features, and s represents the sample space, all pixel features.

[0117] Pre-segmentation loss: in order to construct class features, it is necessary to first pre-segment the image to obtain the rough position of each class of pixels, so the present application uses the cross-entropy function to supervise the pre-segmentation result y aux1i of the image.

[0118]

[0119] Regularization loss: the present application uses cross-entropy to directly classify the pixel features through several layers of convolution y aux2i Supervision is performed to map the pixel features to the class feature space, thereby playing a regularization role.

[0120]

[0121] Therefore, the total loss function is defined as follows:

[0122]

[0123] where λ1, λ2 and λ3 are all preset weights.

[0124] The following is a system embodiment corresponding to the above method embodiment, and the present embodiment can be implemented in cooperation with the above embodiments. The related technical details mentioned in the above embodiments are still valid in the present embodiment. In order to reduce repetition, they will not be described again. Correspondingly, the related technical details mentioned in the present embodiment can also be applied in the above embodiments.

[0125] The present application also proposes a feature clustering-based semantic segmentation system, which comprises:

[0126] The feature extraction module is configured to obtain a training image with a labeled class label, and a feature extraction layer in the semantic segmentation model extracts deep semantic features and shallow detail features of the training image. A first convolutional classifier in the semantic segmentation model classifies the deep semantic features to obtain a first classification result of each pixel in the training image. The first classification result is normalized to obtain a probability that each pixel belongs to each semantic class. The probability and the deep semantic features are dot multiplied to obtain a class feature of each class.

[0127] The feature fusion module is configured to fuse the deep semantic features and the shallow detail features to obtain pixel-level features. According to the pixel-level features, the similarity of each pixel in the training image to each class feature is calculated, and the class corresponding to the maximum similarity is taken as a second classification result of the current pixel. A second convolutional classifier in the semantic segmentation model classifies the pixel-level features to obtain a third classification result of the pixels in the training image. The second classification result is the final semantic segmentation result, and the third classification result is an auxiliary supervision loss.

[0128] The model training module is configured to construct a loss function according to the class label of the training image and the first classification result, the second classification result and the third classification result, train the semantic segmentation model, and perform semantic segmentation on a specified image using the trained semantic segmentation model.

[0129] The feature clustering-based semantic segmentation system, wherein the feature extraction layer comprises a backbone network and a context module connected in series, and the context module is configured to enhance network deep features output by the backbone network to obtain the deep semantic features.

[0130] The feature clustering-based semantic segmentation system, wherein the feature extraction module is configured to:

[0131] The deep semantic features are pre-segmented by the following formula:

[0132] r = S(f h )

[0133] wherein f h represents the deep semantic features, r represents the first classification result, and S is the first convolutional classifier; the first classification result is then normalized by softmax to obtain the probability of each pixel belonging to each class:

[0134]

[0135] wherein k represents a class, and i represents a pixel;

[0136] The deep semantic features f h and the probability m ik are dot multiplied to obtain class features f k of each class:

[0137]

[0138] The feature fusion module is configured to:

[0139] The deep semantic features and the shallow detail features are fused by the following formula to construct high-resolution pixel-level features:

[0140] f p = C R (concat(C R (f l ), upsample(C R (f h )))

[0141] wherein f p represents the final pixel features, f h and f l represent the deep semantic features and the shallow features respectively, C R represents a 1x1 convolution to reduce the amount of calculation, and upsample represents up-sampling to align the resolution of the deep semantic features and the shallow features.

[0142] The similarity is obtained by the following formula:

[0143]

[0144] class i = argmax j (p i1 , p i2 ,... p ij ,..., p iK )

[0145] where p ik represents the similarity of pixel i and the k-th class, class i represents the second classification result of pixel i.

[0146] The feature clustering-based semantic segmentation system, wherein the loss function comprises a clustering center loss, a feature consistency loss, a pre-segmentation loss and a regularization loss.

[0147] The clustering center loss:

[0148]

[0149] The feature consistency loss:

[0150]

[0151] The pre-segmentation loss:

[0152]

[0153] The regularization loss:

[0154]

[0155] The final loss function:

[0156]

[0157] The application further provides a storage medium for storing a program for executing any one of the feature clustering-based semantic segmentation methods.

[0158] The application further provides a client for any one of the feature clustering-based semantic segmentation systems.

Claims

1. A method of semantic segmentation based on feature clustering, characterized in that, Comprise: Step 1, obtain a training image with labeled class labels, a feature extraction layer in the semantic segmentation model extracts deep semantic features and shallow detail features of the training image, a first convolutional classifier in the semantic segmentation model classifies the deep semantic features to obtain a first classification result of each pixel in the training image, normalizes the first classification result to obtain the probability of each pixel belonging to each semantic class, and performs dot product on the probability and the deep semantic features to obtain the class feature of each class; Step 2, fuse the deep semantic features and the shallow detail features to obtain pixel-level features; According to the pixel-level features, calculate the similarity of each pixel in the training image to each class feature, and take the class corresponding to the maximum similarity as the second classification result of the current pixel; A second convolutional classifier in the semantic segmentation model classifies the pixel-level features to obtain a third classification result of the pixels in the training image; Step 3, construct a loss function according to the class labels of the training image and the first, second and third classification results, train the semantic segmentation model, and perform semantic segmentation on the specified image using the trained semantic segmentation model; Wherein, the step 1 comprises: Pre-segment the deep semantic features by the following formula: where f h represent deep semantic features, r represents the first classification result, and S is the first convolutional classifier; the first classification result is then normalized by softmax to obtain the probability of each pixel belonging to each class: Where k represents the class, and i represents the pixel; Deep semantic features f h and probabilities m ik are dot producted to obtain class features f k for each class. The step 2 comprises: Fuse the deep semantic features and the shallow detail features by the following formula to construct high-resolution pixel-level features: where f p represents the final pixel feature, f h , f l represent deep semantic feature and shallow feature respectively, C R represents 1x1 convolution to reduce the amount of calculation, and upsample represents up-sampling to align the resolution of deep feature and shallow feature. The similarity is obtained by the following formula: where p ik represents the similarity of pixel i and the kth class, class i represents the second classification result of pixel i; The loss function comprises a clustering center loss, a feature consistency loss, a pre-segmentation loss and a regularization loss; Clustering center loss: Feature consistency loss: Pre-segmentation loss: Regularization loss: Final loss function: 。 2. The feature clustering based semantic segmentation method of claim 1, wherein, The feature extraction layer comprises a backbone network and a context module connected in series, the context module is used to enhance the network deep features output by the backbone network to obtain the deep semantic features; And the second classification result is the final semantic segmentation result, and the third classification result is an auxiliary supervision loss.

3. A feature clustering based semantic segmentation system, characterized in that, Comprise: A feature extraction module is configured to obtain a training image with labeled class labels, a feature extraction layer in the semantic segmentation model extracts deep semantic features and shallow detail features of the training image, a first convolutional classifier in the semantic segmentation model classifies the deep semantic features to obtain a first classification result of each pixel in the training image, normalizes the first classification result to obtain the probability of each pixel belonging to each semantic class, and performs dot product on the probability and the deep semantic features to obtain the class feature of each class; A feature fusion module is configured to fuse the deep semantic features and the shallow detail features to obtain pixel-level features; According to the pixel-level features, calculate the similarity of each pixel in the training image to each class feature, and take the class corresponding to the maximum similarity as the second classification result of the current pixel; A second convolutional classifier in the semantic segmentation model classifies the pixel-level features to obtain a third classification result of the pixels in the training image; The model training module is configured to construct a loss function according to the class label of the training image and the first classification result, the second classification result and the third classification result, train the semantic segmentation model, and perform semantic segmentation on a specified image by using the trained semantic segmentation model. The feature extraction module is configured to: The deep semantic feature is pre-segmented by the following formula: where f h represent deep semantic features, r represents the first classification result, and S is the first convolutional classifier. The first classification result is then normalized by softmax to obtain the probability of each pixel belonging to each class: Wherein k represents a class, and i represents a pixel. Deep semantic features f h and probabilities m ik are multiplied to obtain class features f k for each class. The feature fusion module is configured to: The deep semantic feature and the shallow detail feature are fused by the following formula, thereby constructing a high-resolution pixel-level feature: where f p represents the final pixel feature, f h , f l represent deep semantic feature and shallow feature respectively, C R represents 1x1 convolution to reduce the amount of calculation, and upsample represents up-sampling to align the resolution of deep feature and shallow feature. The similarity is obtained by the following formula: where p ik represents the similarity of pixel i and the k-th class, class i represents the second classification result of pixel i; The loss function includes a clustering center loss, a feature consistency loss, a pre-segmentation loss and a regularization loss. The clustering center loss is: The feature consistency loss is: The pre-segmentation loss is: The regularization loss is: The final loss function is: 。 4. The feature clustering based semantic segmentation system of claim 3, wherein, The feature extraction layer includes a backbone network and a context module connected in series, and the context module is configured to enhance the network deep feature output by the backbone network to obtain the deep semantic feature. The second classification result is a final semantic segmentation result, and the third classification result is an auxiliary supervision loss.

5. A storage medium for storing a program for executing the feature clustering based semantic segmentation method according to claim 1 or 2.

6. A client for the feature clustering based semantic segmentation system according to claim 3 or 4.

Images