A multi-classifier based image semantic segmentation method

Abstract

The application discloses a multi-classifier-based image semantic segmentation method, which comprises the following steps: S100: three different classifiers are respectively designed; in the first classifier, semantic features of an input image are extracted through a backbone network, and then input into a coordinate attention-based cavity space pyramid pooling C-ASPP module to capture semantic information of different receptive fields; in the second classifier, semantic features of the input image are obtained through the backbone network and the ASPP module, and then a feature fusion-based decoder is used to give weights to shallow features and deep features respectively and fuse the features; a new loss function is introduced in the third classifier to supervise detail information; S200: the three different classifiers are fused through an ensemble learning method to generate a final segmentation result. The method generates the final segmentation result through ensemble learning of the three different network structure classifiers, enhances semantic features, makes up for detail loss, and improves segmentation performance.

Description

TECHNICAL FIELD

[0001] The present disclosure belongs to the technical field of computer vision and image processing, and particularly relates to an image semantic segmentation method based on multiple classifiers. BACKGROUND

[0002] Semantic segmentation is to classify each pixel in an image according to a label, which can distinguish different objects in the image, delineate the contours between different individuals and backgrounds, and give the same color to the objects of the same class. As a hot issue in the field of computer vision, image semantic segmentation has a wide range of applications in practical scenarios such as autonomous driving, video conferencing, and medical image diagnosis.

[0003] Traditional image segmentation techniques such as threshold-based and edge-based segmentation methods are difficult to obtain good segmentation results and cannot meet the requirements of practical applications in terms of segmentation accuracy. In recent years, with the development of deep learning, convolutional neural networks have shown strong feature extraction and object representation capabilities, and a series of classic network models such as AlexNet, VGG, and GoogLeNet have emerged, making image segmentation models based on convolutional neural networks achieve good segmentation performance. FCN, as a pioneering work of applying convolutional neural networks to image semantic segmentation, realizes end-to-end training and achieves good performance on various benchmark datasets. Its core idea is to replace the fully connected layer with a convolutional layer and use a skip connection to apply the semantic information of the intermediate layer to improve the final prediction results. Since the segmentation results of FCN are relatively rough, in order to refine the output, a structured prediction module is used on the network output and a probabilistic graph model such as Conditional Random Field (CRF) or Markov Random Field (MRF) is utilized. SegNet uses an index to record the spatial position information lost during the pooling operation, so as to recover the corresponding pixel information during upsampling. In order to obtain more diverse features, PSPNet constructs a pyramid pooling module to aggregate context information from different regions. Deeplabv2 proposes an Atrous Spatial Pyramid Pooling (ASPP) module, which parallelly applies different rates of atrous convolution to capture multi-scale semantic features. Deeplabv3 optimizes the ASPP module and introduces a global average pooling to further capture global semantic information of deep features. DeepLabv3plus based on DeepLabv3 combines atrous convolution and depthwise separable convolution instead of ordinary convolution, which greatly reduces the parameter quantity of the model and improves the segmentation accuracy.

[0004] In the Deeplabv3plus network structure, in order to obtain multi-scale semantic information, the empty space pyramid pooling module is further optimized, the information output by the backbone network is globally pooled after parallel convolution with different empty rates, and then the size is restored by upsampling, and finally the feature maps with the same size are connected in the channel. However, the use of empty convolution makes the input information not fully utilized, and the loss of part of the information causes the wrong segmentation result.

[0005] Deeplabv3plus further expands Deeplabv3 by adding a simple and effective decoder, which refines the segmentation result at the image details such as target outline and boundary. In this decoder, the semantic features output by ASPP are upsampled by 4, and are connected with the shallow features in the backbone network with the same resolution. However, this operation does not fully utilize the shallow features, which may cause inaccurate edge segmentation and partial pixel segmentation errors. SUMMARY

[0006] In order to solve the above problems, the present disclosure provides an image semantic segmentation method based on multiple classifiers, comprising the following steps:

[0007] S100: three different classifiers are designed respectively; in the first classifier, the input image is extracted by the backbone network to obtain semantic features, and then input into the coordinate attention based empty space pyramid pooling C-ASPP module to capture semantic information of different receptive fields; in the second classifier, the input image is input into the backbone network and the empty space pyramid pooling ASPP module to obtain semantic features, and then a decoder based on feature fusion is used to fuse the shallow features and the deep features by assigning weights to them respectively; a new loss function is introduced in the third classifier to supervise the detail information;

[0008] S200: the above three different classifiers are fused by the method of ensemble learning to generate the final segmentation result.

[0009] By the technical solution, semantic features are enhanced while details of the shallow network are utilized. The method proposes three different classifiers based on Deeplabv3plus, and the three different classifiers are integrated to generate a final segmentation map. In the first classifier, a spatial pyramid pooling module based on attention is designed to replace the ASPP module to fully utilize the output information of the backbone network to obtain rich semantic information. A branch path is newly constructed based on the ASPP to enhance the global semantic features by using the attention mechanism, and the output of the branch is fused with the output of the ASPP to improve the segmentation accuracy. In the second classifier, a decoder based on a feature fusion module is redesigned. The decoder is simple and effective, and can fully fuse the shallow features and the deep features. The fused result can accurately retain rich detail information, and is beneficial to generate a more fine segmentation map. In the third classifier, a new loss function is introduced to supervise the detail information. The new loss function introduced at the shallow feature layer can make the classifier encode more spatial details and improve the segmentation accuracy. Based on the three different classifiers, an integrated learning system is built to achieve better segmentation performance. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 FIG. 1 is a flowchart of an image semantic segmentation method based on a multi-classifier according to an embodiment of the present disclosure;

[0011] Figure 2 FIG. 2 is a schematic diagram of a network structure according to an embodiment of the present disclosure;

[0012] Figure 3 FIG. 3 is a schematic diagram of a coordinate attention based atrous spatial pyramid pooling C-ASPP structure according to an embodiment of the present disclosure;

[0013] Figure 4 FIG. 4 is a schematic diagram of a feature fusion based decoder FF-Decoder structure according to an embodiment of the present disclosure;

[0014] Figure 5 FIG. 5 is a visualization diagram of segmentation results on a PASCAL VOC 2012 dataset according to an embodiment of the present disclosure;

[0015] Figure 6 FIG. 6 is a comparison diagram of specific class IoU of the C-ASPP structure and a baseline on a PASCAL VOC 2012 validation set according to an embodiment of the present disclosure;

[0016] Figure 7 FIG. 7 is a visualization diagram of segmentation results of the C-ASPP structure on the PASCAL VOC 2012 dataset according to an embodiment of the present disclosure;

[0017] Figure 8 is a specific class IoU comparison chart of the loss function introduced in one embodiment of the present disclosure on the PASCAL VOC 2012 validation set with the baseline;

[0018] Figure 9 is a segmentation result visualization chart of the loss function introduced in one embodiment of the present disclosure on the PASCAL VOC 2012 dataset. DETAILED DESCRIPTION

[0019] In one embodiment, as shown in Figure 1 , a multi-classifier-based image semantic segmentation method is disclosed, comprising the following steps:

[0020] S100: Three different classifiers are designed respectively; in the first classifier, the input image is extracted by the backbone network to obtain semantic features, and then input into the coordinate attention-based empty space pyramid pooling C-ASPP module to capture semantic information of different receptive fields; in the second classifier, the input image is input into the backbone network and the empty space pyramid pooling ASPP module to obtain semantic features, and then a feature fusion-based decoder is used to fuse the shallow features and deep features by assigning weights to them respectively; a new loss function is introduced in the third classifier to supervise the detail information;

[0021] S200: The above three different classifiers are fused by the method of ensemble learning to generate the final segmentation result.

[0022] For this embodiment, three different classifiers are constructed based on Deeplabv3plus, and the segmentation performance of the algorithm is improved through ensemble learning of different classifiers. The first classifier relies on the proposed attention-based empty space pyramid pooling module (C-ASPP), which captures more rich global semantic information through parallel attention mechanism and empty space pyramid pooling module. In the second classifier, a new decoder (FF-Decoder) is designed, which adopts feature fusion to obtain the edge shape and other detail information possessed by the shallow network. In the third classifier, a new loss function is introduced to supervise the detail information loss. Through ensemble learning of the three different network structure classifiers, the final segmentation result is generated, which enhances the semantic features while making up for the detail loss and improves the segmentation performance.

[0023] The network structure used in the method is shown in Figure 2 , three different classifiers are designed based on Deeplabv3plus, and the three different classifiers are fused by the method of ensemble learning to generate the final segmentation result. As shown in Figure 2 , three different classifiers are designed based on Deeplabv3plus, and the three different classifiers are fused by the method of ensemble learning to generate the final segmentation result.where ensemble means that different classifiers are combined by ensemble learning, and a and l represent the loss function of the classifier. In the first classifier, the input image is extracted by the semantic feature of the backbone network, and then input into the C-ASPP module to capture the semantic information of different receptive fields, making full use of the output information of the backbone network. In this classifier, the decoder adopts the decoder structure proposed in Deeplabv3plus. In the second classifier, the input image is obtained by the semantic feature of the backbone network and the ASPP module, and then a feature fusion-based decoder is used to fuse the shallow features and deep features by assigning weights to them, so as to make full use of the detail information. In the third classifier, a new loss function is introduced, which uses the real label to guide the network structure to learn the detail information of the shallow layer, and uses the dice loss and the binary cross-entropy loss to optimize the detail information, so as to be beneficial to the segmentation of image details such as edges and shapes.

[0024] In another embodiment, the coordinate attention-based hollow spatial pyramid pooling C-ASPP module is a branch path newly constructed on the basis of the atrous spatial pyramid pooling ASPP, which uses an attention mechanism to enhance the global semantic features, and fuses the output of the branch path with the output of the ASPP.

[0025] For this embodiment, the atrous spatial pyramid pooling (ASPP) module in Deeplabv3plus is composed of atrous convolution, which uses convolution with different atrous rates to obtain multi-scale context information without losing resolution. Although the atrous convolution does not lose information when expanding the receptive field, the zero padding operation makes the input information not fully utilized, and the lost information causes difficulty in segmenting part of the object. In order to make full use of the information output by the backbone network, this method proposes a coordinate attention-based spatial pyramid pooling (C-ASPP) module on the basis of ASPP. The C-ASPP structure is as shown in Figure 3 where Conv represents 3x3 convolution, batch normalization and ReLU.

[0026] In another embodiment, the attention mechanism is a lightweight coordinate attention.

[0027] For this embodiment, in order not to lose information and make full use of the information of the backbone network, while avoiding the attention mechanism from generating more additional overhead, a lightweight attention mechanism, coordinate attention (CoordAttention), is selected. This attention mechanism encodes each channel of the input feature in the horizontal and vertical directions, generating two independent direction perception feature maps, so that it can not only capture cross-channel information, but also capture direction perception and position perception information, helping the network to more accurately locate and identify the target of interest and enhancing the feature representation capability of the network.

[0028] In another embodiment, the coordinate attention mechanism decomposes the two-dimensional global set operation into two one-dimensional encoding to enhance the feature representation.

[0029] In another embodiment, the specific steps of fusing the output of the branch path with the output of the ASPP are as follows:

[0030] S1011: obtaining a multi-scale feature map after passing the output of the backbone network through the ASPP wherein R represents the set of real numbers, C1, H1, W1 represent the channel number, height, and width of the multi-scale feature map O, respectively;

[0031] S1012: simultaneously reducing the channel number of the output information of the backbone network through a 3x3 convolution, and then obtaining a feature map through the coordinate attention mechanism wherein R represents the set of real numbers, C1, H1, W1 represent the channel number, height, and width of the feature map P, respectively;

[0032] S1013: inputting the feature map P and the feature map O into the formula Q = O * (1-sigmoid (P)) to obtain the final feature map and output, wherein * represents matrix point multiplication; wherein Sigmoid() is a nonlinear function, R represents the set of real numbers, C1, H1, W1 represent the channel number, height, and width of the feature map Q, respectively.

[0033] For this embodiment, a multi-scale feature map is obtained after passing the output of the backbone network through the ASPP wherein the parallel hole rates in the ASPP are set to 1, 6, 12, and 18, respectively. Simultaneously, the output information of the backbone network is reduced in channel number through a 3x3 convolution, and then a feature map is obtained through the attention mechanism while capturing long-term dependencies between information and preserving accurate position information. Finally, the relevant operation between the feature map P and O is performed according to the formula Q = O * (1-sigmoid (P)) to obtain the final output where * represents matrix point multiplication. Through the above operation, the output feature map Q makes up for the partial information loss problem caused by the dilated convolution.

[0034] In another embodiment, the feature fusion-based decoder uses the shallow feature with 256 channels after four times down-sampling of the backbone network as input, selects different scale feature maps for weighting, and performs pixel-level summation operation to obtain the fused feature map.

[0035] For this embodiment, in the Deeplabv3plus network structure, a feature map after four times down-sampling in the Xception network backbone is used as a shallow feature, and a 1x1 convolution is used to reduce the channel number of the shallow feature to 48. Finally, the deep feature after up-sampling and the shallow feature are concatenated together to fuse features of different scales. Since it is a shallow feature map after reducing the channel number at this time, the decoder does not fully utilize the shallow feature. In the proposed decoder, the shallow feature with 256 channels after four times down-sampling of the backbone network is used as input, and features of different scales are fused without reducing the channel number. In the proposed decoder, different scale feature maps are selected for weighting, and pixel-level summation operation is performed to obtain the fused result.

[0036] In another embodiment, the specific steps of the feature fusion-based decoder for fusing features of different scales are as follows:

[0037] S1021: First, the deep feature H is up-sampled to the size of the feature map L using the bilinear interpolation method, and the up-sampled deep feature is passed through the coordinate attention mechanism;

[0038] S1022: According to the formula The feature map obtained through the coordinate attention mechanism is used to obtain the weight using the sigmoid function where Atten refers to the coordinate attention mechanism.

[0039] S1023: Obtain the weight Then, according to the formula Different weights are assigned to the shallow feature map L and the up-sampled deep feature map H to obtain new shallow feature map L' and deep feature map H';

[0040] S1024: According to the formula Z = H' + βL', the fused feature map is obtained where β is a positive real number, R represents the set of real numbers, C2, H2, and W2 represent the channel number, height, and width of the feature map Z, respectively.

[0041] In this embodiment, in semantic segmentation tasks, the best results for segmenting different categories of objects often occur on feature maps of different scales. Therefore, multi-scale feature fusion is frequently used to improve segmentation performance. Feature fusion-based decoders (FF-Decoder) are as follows: Figure 4 As shown, S is the sigmoid function, and mul is the matrix dot product. When fusing features at different scales, bilinear interpolation is first used to upsample the input deep features H to the size of the feature map L, and then the upsampled deep features are passed through an attention mechanism. Here, CoordAttention is chosen to capture long-term dependencies while preserving positional information. The feature map obtained through CoordAttention is then weighted using the sigmoid function. As in the formula As shown. After obtaining the weights, follow the formula. Different weights are assigned to the shallow feature map L and the upsampled deep feature map H. Since the shallow features contain the necessary detailed information, satisfying the formula Z = H′ + βL′, the network can learn features from different scales. The fused feature map is obtained using an element-wise addition method. At this point, feature map Z not only possesses rich semantic information but also includes detailed information such as edges and shapes from low-level features. Unlike the decoder in the Deeplabv3plus network structure, shallow and deep features have the same number of channels. Because shallow features do not possess rich semantic information, they are multiplied with the result of deep features after attention. These are shallow features extracted from the backbone network. These are the deep features of the input, representing the result of the backbone network's output information passing through the ASPP module. These are processed shallow features. These are processed deep features.

[0042] In another embodiment, the new loss function includes binary cross-entropy loss and dice loss.

[0043] In this embodiment, due to the irreversibility of convolution operations, the detailed information lost in the encoding part of the network cannot be accurately recovered. To address this issue, a new loss function is introduced in the third classifier to supervise the detailed information, fusing the learned detailed features with the contextual semantic features. `dice` represents the dice loss.

[0044] Binary cross-entropy loss only considers the loss at the microscopic level, leading to coarse results. This method uses binary cross-entropy and Dice loss jointly to optimize detail learning. Dice loss, as a region correlation function, measures the overlap between the predicted map and the ground truth label, indicating that the loss of the current pixel is related not only to the predicted value of the current pixel but also to the values ​​of other pixels. Since it is insensitive to the number of foreground and background pixels, it can alleviate class imbalance in feature maps. For a predicted detail map with height H and width W, the detail loss is L. d (p d g d ) = L dice (p d g d )+L bce (p d g d ), where p d ∈R H×W g represents detailed information such as the boundaries of the network predictions. d ∈R H×W This represents the corresponding actual value. L bce This represents the binary cross-entropy loss, while L... dice This is the Dice loss introduced to alleviate the class imbalance problem, and its formula is: Here, i represents the i-th pixel in the image, and ε is a Laplacian smoothing term. To reduce overfitting, ε is set to 1. Finally, the introduced detail loss is added to the original loss value to obtain the total loss λ, λ = L. d (p d g d )+α, where α refers to the loss function in the Deeplabv3plus network structure.

[0045] Furthermore, the feature map with 256 channels after 4x downsampling in the backbone network is used as the predicted detail feature map, and its corresponding ground truth value is used as a guide for the shallow network to learn detail features, making it have more detailed information than the original feature map.

[0046] In another embodiment, the ensemble learning uses a voting method to determine the final category label, with one of the classifiers as the benchmark for voting, and for each pixel, the class with the highest count is used as the label for that pixel.

[0047] For this embodiment, ensemble learning refers to combining multiple classifiers to achieve better performance than a single classifier. When multiple classifiers are ensembled, a voting method is used to determine the final class label. For each pixel, the class with the highest count is used as the label for that pixel, with one of the classifiers serving as the reference for the vote.

[0048] In another embodiment, the backbone network in each classifier uses a modified Xception model that uses depthwise separable convolution instead of traditional Inception modules.

[0049] For this embodiment, to improve accuracy while reducing computational complexity, the backbone network in each classifier uses a modified Xception model that uses depthwise separable convolution instead of traditional Inception modules. Depthwise separable convolution is a combination of pointwise and depthwise convolution used to extract features, which reduces computational cost and parameter quantity while maintaining performance, achieving complete decoupling of cross-channel correlation and spatial correlation. In addition, the downsampling operation on the image in the Exit flow is canceled and all max pooling operations are replaced with depthwise separable convolution with stride, and the output feature map size of the backbone network is designed to be 1 / 16 of the input image.

[0050] In another embodiment, the method is based on the PyTorch deep learning framework, and experiments are conducted on an NVIDIA GTX1080Ti GPU. For the PASCAL VOC 2012 dataset, the batch size is set to 4 and the input image size is cropped to 512x512; for the Camvid dataset, the batch size is set to 2 and the original image resolution is 960x720. In the experiment, "SGD" is used as the optimizer, the momentum is 0.9, the weight decay is 0.0001, and "cos" is used in the learning strategy, with the maximum learning rate of the model being 0.007. Other settings follow the method proposed in Deeplabv3plus.

[0051] Global accuracy (ACC), average pixel accuracy per class (mPA), and average intersection over union per class (mIoU) are used as evaluation indicators. ACC refers to the ratio of the number of pixels correctly classified to the total number of pixels; mPA is an improvement on accuracy, which is the average of the probability of correct classification of pixels within a class, i.e., the pixel accuracy of each class is calculated and then averaged; mIoU is a standard measure, which calculates the ratio of the intersection and union of two sets. In image segmentation, the intersection over union of the true value and the predicted value of each class is first calculated, and then the average is calculated.

[0052] The proposed multi-classifier based image semantic segmentation algorithm is evaluated on the PASCAL VOC 2012 dataset. The dataset is a general-purpose dataset commonly used in image classification, object detection, image segmentation, and other tasks. The dataset used for semantic segmentation contains 20 object classes and one background class, with 1464 images for training and 1449 images for validation.

[0053] As Figure 5 the visualization results, as can be seen in the third column of the first row, the pixels circled in the figure as a person are segmented as a motorcycle by Deeplabv3plus, while in the fourth column, the segmentation method of the present method corrects this error; in the second row, the baseline algorithm Deeplabv3plus has a fuzzy and discontinuous overall contour segmentation of the ship part circled in the figure, while the fourth example of the segmentation result of the present method successfully segments the object and correctly segments the part of the ship pixels that the baseline algorithm fails to recognize; in the third row, the segmentation method of the present method corrects most of the pixels that are incorrectly segmented with respect to the circled part, and the segmentation method of the present method is significantly better than the baseline algorithm, which incorrectly segments a large area of the chair part circled in the figure as a sofa. In summary, the multi-classifier based image semantic segmentation algorithm not only obtains rich global semantic information to produce correct segmentation results, but also makes the segmentation results more detailed at the details of the image, which shows that the present method is beneficial to obtaining better segmentation results.

[0054] The performance of ensemble learning is significantly higher than that of individual classifiers in terms of global accuracy, class average pixel accuracy, and average intersection over union, as shown in Table 1, where baseline refers to the Deeplabv3plus algorithm. In terms of mIoU evaluation indicators, an 82.42% is ultimately obtained, which is 2.46% higher than the baseline, and ACC and mPA also achieve high performance results of 96.06% and 89.42%. Further proving that the present method can effectively utilize the advantages of different classifiers, correct segmentation errors in the case of accurately segmenting global image information, and produce better segmentation results at details such as corners and boundaries.

[0055]

[0056] Table 1

[0057] The performance of the method and other related algorithms on the data set is shown in Table 2. Compared with the classic network structure of Deeplab series, the network structure of the method obviously achieves a higher mIoU value. And it is 2.75%, 2.02%, 1.82% higher than the network structures of DUpsampling, DANet, HyperSeg respectively in recent years. It can be seen that the network structure based on multi-classifier proposed in this paper is very effective for image semantic segmentation.

[0058]

[0059] Table 2

[0060] In Table 1, it can be observed that the performance of C-ASPP is more excellent than the benchmark algorithm, and the mIoU is improved by 0.87%. The specific class IoU index is shown in Table 2. Figure 6 It can be seen that the IoU value of most classes obtained by using the C-ASPP module is obviously higher than that of the benchmark algorithm. In the visualization results of Figure 7 , it can be seen that the module can obtain more rich semantic features and correct the segmentation errors. For example, in the first row and third column of the picture, the segmentation error of the horse leg, the Deeplabv3plus as the baseline network mistakenly thinks that the fence belongs to the horse leg, while the classifier added with the C-ASPP module can correctly segment the two different objects; in the second row of the visualization results, the incompleteness of the motorcycle tail segmentation of the Deeplabv3plus network is improved; in the third row of the segmentation results, it can be seen that the classifier corrects the chair that is wrongly segmented into a sofa. This shows that the C-ASPP is obviously superior to the ASPP, and can capture more rich global semantic information and make full use of the information output by the backbone network.

[0061] Under the same network structure, Deeplabv3plus based on Xception obtains a performance of 79.17% mIoU on PASCAL VOC 2012 without using the decoder. It can be seen in Table 3 that the obtained mIoU value is higher than 79.17% regardless of the value of β, which shows that the decoder based on feature fusion is effective. Among them, when β = 1.0, the obtained ACC and mIoU are the highest, which are 95.49% and 80.04% respectively; when β = 1.2, the obtained mPA is the highest, which is 88.72%, but at this time the ACC and mIoU begin to decline. Based on the above analysis, for the formula Z = H' + βL', the decoder selects the value of β as 1.0. In Table 1, the ACC of the Deeplabv3plus network structure containing the decoder as the benchmark algorithm is 95.47%, the mPA is 88.32%, and the mIoU is 79.96%. As shown in the results of Table 3 when β is 1.0, compared with the decoder in the benchmark algorithm, the proposed decoder based on feature fusion produces better performance. Deeplabv3plus in Table 3 refers to the network structure without the decoder and other proposed modules.

[0062]

[0063] Table 3

[0064] As shown in Table 1, in the Deeplabvplus architecture with Xception as the backbone network, the introduction of the loss function improves the accuracy and the class average pixel accuracy, and the mIoU is improved by 1.13%, and the specific class IoU is as shown in Table 2. Figure 8 Compared with Deeplabv3plus, the introduction of the new loss function is more conducive to the segmentation of small objects, boundaries and other details. In Figure 9 some segmentation effect diagrams are shown, it can be observed that in the third column in a row, the segmented part of the plant at the edge is not satisfactory, and the background pixels at the edge of the leaf are also predicted as plants, while the fourth column introduces the segmentation result of the loss function, which is obviously more clear at the edge; and in the second row, the corner of the ship circled part, the segmentation result of the loss function after introducing the loss function is also better than the benchmark algorithm; in the third row, compared with the segmentation result of Deeplabv3plus, after introducing the loss function, the fuselage of the circled part in the picture has a thinner edge.

[0065] To verify the effectiveness of the proposed network structure, further evaluation on the CamVid dataset is carried out. The CamVid dataset is a road scene dataset. It is the first video collection with object class semantic labels, containing 701 high-quality pixel-level annotated images extracted from video sequences, of which 367 are used for training, 101 for validation, and 233 for testing. The real semantic labels of the data provide 32 categories, and in the experiment, a subset of 11 popular categories is used.

[0066] As shown in Table 4, compared with the baseline algorithm Deeplabv3plus, the three different structures of the proposed classifier are significantly improved. Among them, the proposed C-ASPP module is added to the classifier 1 as a further component to fully utilize the output information of the backbone network, which increases the ACC to 92.15%, the mPA to 77.29%, and the mIoU to 68.05%. The introduced loss function increases the mIoU to 67.99% and the mPA to 76.42% while improving the global accuracy. In addition, the use of the feature fusion-based decoder achieves better performance than the decoder in the baseline. This shows that the three different classifiers we propose are effective. Finally, the three different classifiers are fused using the ensemble learning method, which finally achieves an accuracy of 92.61% on the Camvid test set, a class average pixel accuracy of 77.51%, and a class average intersection over union of 69.18%. The mIoU value is improved by 1.82% compared with the baseline algorithm.

[0067]

[0068] Table 4

[0069] In Table 5, it can be seen that the network structure of the present method is significantly better than the Deeplab series network structure, and is 3.58% and 0.48% higher than the recent network structure CGNet and BiSeNetV1, respectively. This further illustrates that the proposed network structure based on multiple classifiers is very effective.

[0070]

[0071] Table 5

[0072] Although the embodiments of the present application are described above in conjunction with the drawings, the present application is not limited to the above-mentioned specific embodiments and application fields, and the above-mentioned specific embodiments are only illustrative and guiding, but not limiting. Those skilled in the art can make many forms under the guidance of the present specification and without departing from the scope protected by the claims of the present application, which are all within the protection scope of the present application.

Claims

1. A multi-classifier-based image semantic segmentation method, comprising the following steps: S100: three different classifiers are designed respectively; in the first classifier, the input image is extracted by a backbone network to obtain semantic features, and then input into a coordinate attention-based dilated spatial pyramid pooling (C-ASPP) module to capture semantic information of different receptive fields; in the second classifier, the input image is input into a backbone network and an all spatial pyramid pooling (ASPP) module to obtain semantic features, and then a feature fusion-based decoder is used to fuse the shallow features and the deep features by assigning weights to them respectively; a new loss function is introduced in the third classifier to supervise the detail information; S200: the three different classifiers are fused by an ensemble learning method to obtain a final segmentation result; wherein, the C-ASPP module is a branch path newly constructed on the basis of the ASPP, which uses an attention mechanism to enhance global semantic features, and fuses the output of the branch path with the output of the ASPP; the specific steps of fusing the output of the branch path with the output of the ASPP are as follows: S1011: obtaining a multi-scale feature map after output of the backbone network passes through the ASPP wherein R denotes the set of real numbers, , , respectively represent the number of channels, height, and width of the multi-scale feature map O. S1012: reduce the channel number of the output information of the backbone network through 3x3 convolution, and then obtain a feature map through the coordinate attention mechanism wherein R denotes the set of real numbers, , , respectively represent the channel number, height and width of the feature map P. S1013: input the feature map P and the feature map O into a formula to obtain a final feature map and output, wherein represents matrix point multiplication; wherein, Sigmoid() is a nonlinear function, R represents a real set, 、 、 respectively represent the channel number, height and width of the feature map Q.

2. The method of claim 1, wherein, the attention mechanism is a lightweight coordinate attention.

3. The method of claim 2, wherein, The coordinate attention mechanism enhances the feature representation by decomposing the two-dimensional global set operation into two one-dimensional encodings.

4. The method of claim 1, wherein, The feature fusion-based decoder uses the shallow features with 256 channels obtained by four times down-sampling of the backbone network as input, selects features of different scales for weighting, and performs pixel-level summation operation to obtain the fused feature map.

5. The method of claim 1, wherein, The specific steps of the feature fusion-based decoder to fuse features of different scales are as follows: S1021: first, the input deep features H are up-sampled to the size of the feature map L by using the bilinear interpolation method, and the up-sampled deep features are processed by the coordinate attention mechanism; S1022: According to the formula The feature map obtained by the coordinate attention mechanism is weighted using a sigmoid function wherein Atten refers to the coordinate attention mechanism. S1023: obtain the weight After the formula Different weights are given to the shallow feature map L and the deep feature map H after upsampling, to obtain a new shallow feature map and a deep feature map ; S1024: According to the formula get the fused feature map wherein β is a positive real number, R represents a real set, 、 、 respectively represent the channel number, height and width of the feature map Z.

6. The method of claim 1, wherein, the new loss function includes binary cross-entropy loss and dice loss.

7. The method of claim 1, wherein, The ensemble learning uses a voting method to determine the final class label, and takes one of the classifiers as a reference for voting. For each pixel, the class with the highest count is used as the label of the pixel.

8. The method of claim 1, wherein, The backbone network in each classifier uses a modified Xception model, which uses a depth separable convolution instead of a traditional Inception module.

Images