Lightweight semantic segmentation method based on feature fusion and semantic boundary constraint

Abstract

The application provides a lightweight semantic segmentation method based on feature fusion and semantic boundary constraint, which comprises the following steps: a feature fusion module fuses feature maps from deep layers to shallow layers of an encoder; a joint semantic segmentation loss is calculated according to the feature maps generated in the feature fusion stage; a semantic boundary supervision module performs semantic boundary supervision on intermediate features generated by a decoder; and the network is optimized through the semantic segmentation loss and a semantic boundary loss. The application bridges the semantic gap between different stages by emphasizing the information of interest of the network while suppressing irrelevant information, thereby effectively fusing the feature information of different stages and improving the segmentation precision. In order to further optimize the precision of the semantic boundary, the application proposes a semantic boundary supervision module, which uses the semantic boundary extracted from the real label to constrain the predicted semantic boundary of the network, thereby improving the semantic boundary prediction precision of the network.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, deep learning and semantic segmentation, and particularly relates to a lightweight semantic segmentation method based on feature fusion and semantic boundary constraint. BACKGROUND

[0002] Semantic segmentation is a basic task in computer vision, aiming to accurately predict the label of each pixel in an image. It has been widely applied in many fields, such as autonomous driving, medical image segmentation, video surveillance, etc. With the growing demand for mobile device deployment, real-time semantic segmentation has been a hot research field in the past few years.

[0003] Real-time semantic segmentation is a challenging task that requires balancing segmentation accuracy and inference speed. To balance inference speed and segmentation accuracy, current research mainly focuses on two aspects: one is to design an efficient decoder to process the feature maps from the encoder, and the other is to design an efficient encoder to achieve the balance between inference speed and segmentation accuracy. Although many current methods show impressive performance, they still face the following challenges:

[0004] 1) Most current methods ignore the existence of semantic gap in multi-level feature fusion, resulting in low fusion efficiency. Semantic gap is common between feature maps at different stages. Typically, shallow feature maps encode more spatial information (such as object edges, textures, and corners), while deep feature maps encode more semantic information (such as object characteristics, classification, and scene parsing). Therefore, in a semantic segmentation network, a feature fusion module is usually needed to bridge the semantic gap between feature maps at different stages. Although complex feature fusion modules can effectively bridge the semantic gap and improve the segmentation accuracy of the network, they are usually too computationally intensive to be applied to mobile devices. Simple fusion methods (such as element-wise addition or channel-wise concatenation) are less computationally intensive, but they cannot adaptively bridge the semantic gap, resulting in poor segmentation performance and difficulty in optimization.

[0005] 2) Current methods perform poorly in semantic boundary prediction accuracy. Knowing semantic boundaries is crucial for the accuracy of segmentation, and semantic boundaries belong to the multi-label classification task, while most current works only focus on binary boundary detection, ignoring the semantic information on the boundary, resulting in poor semantic boundary accuracy of the prediction result, among which the prediction accuracy of small objects is most affected. Therefore, further optimizing the prediction accuracy of semantic boundaries can improve the overall semantic segmentation accuracy, especially the semantic segmentation accuracy of small objects. Unfortunately, to our knowledge, few networks focus on the performance of semantic boundaries. SUMMARY

[0006] The purpose of the present application is to provide a lightweight semantic segmentation method based on feature fusion and semantic boundary constraint, which bridges the semantic gap between different stages by emphasizing the information of interest to the network while suppressing irrelevant information, thereby effectively fusing feature information at different stages and improving segmentation accuracy. In order to further optimize the accuracy of semantic boundaries, a semantic boundary supervision module is also proposed, which uses the semantic boundaries extracted from the real labels to constrain the network's predicted semantic boundaries, thereby improving the network's semantic boundary prediction accuracy.

[0007] To achieve the above purpose, the present application provides a lightweight semantic segmentation method based on feature fusion and semantic boundary constraint, which comprises:

[0008] S1, a feature fusion module fuses feature maps from the deep to shallow layers of the encoder;

[0009] S2, according to the feature maps generated in the feature fusion stage, a joint semantic segmentation loss is calculated;

[0010] S3, a semantic boundary supervision module performs semantic boundary supervision on the intermediate features generated by the decoder;

[0011] S4, the network is optimized by the semantic segmentation loss and the semantic boundary loss.

[0012] Further, the step S1 specifically comprises:

[0013] S1.1: The encoder receives the original input image and sequentially processes it through the encoder shallow to deep layers in 4 stages, respectively outputting feature maps with different resolutions in stages 1 to 4, i.e., stage 1 feature map x1, stage 2 feature map x2, stage 3 feature map x3 and stage 4 feature map x4;

[0014] S1.2: The feature fusion module in stage 5 receives the stage 3 feature map x3 and the stage 4 feature map x4 as input, and performs local feature extraction and dimension alignment on the two inputs:

[0015] x3′ = Down(Conv 3×3 (x3))

[0016] x 4′ = Up(Conv 3×3 (x4))

[0017] wherein the resolution of the stage 3 feature map x3 is larger than that of the stage 4 feature map x4, and the number of channel dimensions of the stage 3 feature map x3 is less than that of the stage 4 feature map x4; Conv 3×3 represents a 3*3 convolution operation for adjusting the number of channels of the feature map to make the number of channels of the two input feature maps consistent; Up and Down represent up-sampling operation and down-sampling operation respectively, for making the spatial sizes of the two input feature maps consistent;

[0018] S1.3: performing different dimension context information extraction on the dimension-aligned feature maps:

[0019]

[0020] a3 = A(x 3′ )

[0021]

[0022] a4 = A(x 4′ )

[0023] wherein, A(·) respectively represents spatial pooling operation and channel pooling operation, which respectively extracts spatial dimension context information and channel dimension context information. The spatial pooling is realized by maximum pooling, and the channel pooling is realized by average pooling; a3 and a4 respectively represent the results of performing channel pooling on the dimension-aligned shallow feature x 3′ and deep feature x 4′ ; m3 and m4 respectively represent the results of performing spatial pooling on the dimension-aligned shallow feature x 3′ and deep feature x 4′ ;

[0024] S1.4: cross-fusing the dimension context information of the input to combine a richer context representation:

[0025]

[0026]

[0027] wherein, represents matrix multiplication operation; m 43 represents a context enhanced representation fused with the spatial pooling result of the low resolution feature map and the channel pooling result of the high resolution feature map; m 34a context enhanced representation that combines the high resolution feature map space pooling result and the low resolution feature map channel pooling result, the feature size is consistent with m 43 ;

[0028] S1.5: generate attention weights required for feature fusion by the two context enhanced representations generated by S1.4:

[0029] a5=T(C(m 43 , m 34 ))

[0030] wherein C(·) is a concatenation operation in the channel dimension; T(·) is an attention weight generation module; a5 is the attention weight generated by the context enhanced representation of the stage 3 and stage 4 features through the attention weight generation module, the size of the attention weight is consistent with the feature map size after S1.2 alignment, and the value range is 0 to 1;

[0031] S1.6: fuse the aligned feature maps using the attention weights in a moving average manner:

[0032] x5=x4′⊙a5+x 3′ ⊙(1-a5)

[0033] wherein ⊙ represents Hadamard product operation; x5 is the feature map obtained by fusing the stage 3 feature map x3 and the stage 4 feature map x4, and the size of x5 is consistent with the feature map size after S1.2 alignment;

[0034] S1.7: generate the fusion feature map of stage 6 and stage 7:

[0035]

[0036]

[0037] wherein, the feature fusion module operation, i.e. the operation of S1.1-1.6, the first parameter receives the high resolution feature map from the shallow layer, and the second parameter receives the low resolution feature map from the deep layer; x6 and x7 represent the feature maps generated by stage 6 and stage 7 respectively.

[0038] Further, the convolution operation is used to extract local area features and adjust the number of channels of the feature map, so that the number of channels of the two inputs is consistent.

[0039] Further, the attention weight adaptively emphasizes the information of interest of the network while suppressing irrelevant information according to the input of the feature fusion module.

[0040] Further, the step S2 specifically comprises:

[0041] S2.1: The feature maps x5, x6 and x7 generated according to the feature fusion stages 5, 6 and 7 generate auxiliary semantic segmentation prediction probability maps pred5 and pred6 and main semantic segmentation prediction probability map pred7, respectively:

[0042] pred i = Up(S i (x i ))

[0043] wherein S i (·) represents a segmentation head of the i-th stage, which is used to generate a semantic segmentation prediction probability map and will be discarded during inference; pred i represents a semantic segmentation prediction probability map of the i-th stage, which has a spatial size consistent with that of a real label, and is used to enhance the feature fusion effect of different stages by generating an auxiliary semantic segmentation loss;

[0044] S2.2: According to the semantic segmentation prediction probability map, an OhemCELoss is used to calculate a joint semantic segmentation loss:

[0045]

[0046] wherein gt represents a semantic segmentation label map; represents an OhemCELoss function; l s represents a joint semantic segmentation loss; i represents a stage number.

[0047] Further, the step S3 specifically comprises:

[0048] S3.1: A binary boundary mask is generated from the semantic segmentation label, and an inflation operation is performed on the binary boundary mask:

[0049]

[0050] wherein represents a Laplacian convolution operation, which is used to extract boundary information of the semantic segmentation label; represents a boundary inflation operation, which is used to expand the boundary range; m b represents a binary inflated boundary mask;

[0051] S3.2: A semantic boundary label and a predicted semantic boundary are generated through the binary inflated boundary mask:

[0052] gt b = m b ⊙gt

[0053] pred b = m b ⊙predi

[0054] wherein, gt b denotes semantic boundary label; pred b denotes predicted semantic boundary.

[0055] S3.3: Calculate semantic boundary loss of the predicted semantic boundary by using Focal Loss function through semantic boundary label:

[0056]

[0057] wherein, denotes Focal Loss function; l b denotes loss value of semantic boundary.

[0058] Further, the step S4 specifically comprises: optimizing semantic segmentation accuracy and semantic boundary prediction accuracy of the network by means of weighted summation of loss values:

[0059] l = l s + ε·l b

[0060] wherein, ε is a weight coefficient for balancing semantic segmentation loss and semantic boundary loss; and l is final network loss.

[0061] Further, the specific steps of the feature fusion module for realizing feature fusion comprise:

[0062] The feature maps from different stages with different spatial resolutions are respectively subjected to local region feature extraction and dimension alignment through two 3x3 convolutions with non-shared parameters and two bilinear interpolations;

[0063] Spatial pooling operation and channel pooling operation are used to extract spatial dimension and channel dimension context information of two input feature maps to realize effective feature fusion;

[0064] Hadamard product is used to fuse context information of different dimensions;

[0065] The attention weight generation module generates attention weights by receiving context enhanced representations after concatenation of channel dimensions. The module first compresses the channel dimensions through 1x1 convolution to reduce the amount of calculation; then uses regular batch normalization and ReLU function to adjust the element value range; then uses 1x1 convolution to adjust the number of feature channels to be consistent with the dimension-aligned features; finally, the sigmoid function is used to generate attention weights with element value range of 0 to 1. The attention weights are subjected to Hadamard product operation on the dimension-aligned features through soft selection, and two attention feature maps are obtained.

[0066] The element-wise summation operation is performed on the attention feature map, and a 3*3 convolution is used to aggregate the features and compress the channel dimension of the feature map, so as to obtain a fused feature map, which fuses the feature information from the shallow layer and the deep layer and is transmitted to the next feature fusion module as a low-resolution input.

[0067] Further, the specific steps of implementing the semantic boundary constraint by the semantic boundary supervision module include:

[0068] In the training stage, a binary boundary mask is generated by using Laplacian convolution on the real label, and if it is a boundary, the mask element value is 1, otherwise 0, which is used to extract the boundary information of the real label.

[0069] The binary boundary mask is dilated to obtain a binary boundary dilated mask, which is used to expand the supervision range of the semantic boundary.

[0070] The binary boundary dilated mask is used to perform Hadamard product operation with the real label and the semantic segmentation prediction probability map respectively, so as to obtain the semantic boundary label and the semantic boundary prediction probability map respectively.

[0071] The semantic boundary label is used to constrain the semantic boundary prediction probability map, so as to strengthen the constraint of the network on the semantic boundary during training.

[0072] Further, the encoder includes 3 feature fusion modules, which simultaneously output 4 basic feature maps with different resolutions, and the basic feature maps of different stages are fused in a cascading manner to generate a feature map with rich representation information.

[0073] The beneficial technical effects of the present application are at least the following points:

[0074] (1) Compared with the existing feature fusion method, the feature fusion module generates attention weights by using the context information of the channel dimension and the spatial dimension of the input feature map, retains important information in a "dynamic weighting" manner to enhance the representation ability of the network, and suppresses the interference caused by irrelevant information, thereby effectively bridging the semantic gap and achieving better feature fusion effect. The fusion module proposed in the present application has simple structure, fewer parameters and better fusion effect compared with most existing fusion modules.

[0075] (2) The network decoder is cascaded by a series of feature fusion modules, and the decoder fuses the feature maps from the deep layer to the shallow layer in a top-down manner, and finally outputs a feature map with rich semantic information and spatial information. Compared with most existing decoders, the semantic gap problem that occurs during feature fusion at different stages is paid more attention to, and the decoder is more lightweight and efficient, and can achieve a better balance between semantic segmentation accuracy and inference speed.

[0076] (3) The semantic boundary supervision module proposed in the application optimizes the semantic boundary precision by emphasizing the semantic boundary information through additional constraints on the predicted semantic boundary, and does not bring any additional calculation amount in the reasoning stage, making up for the problem of lack of optimization of the semantic boundary in the existing real-time semantic segmentation network. BRIEF DESCRIPTION OF DRAWINGS

[0077] The application is further described by using the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the application, and other drawings can be obtained by those skilled in the art without creative labor on the premise of not paying any creative labor.

[0078] Figure 1 A flowchart of the lightweight semantic segmentation method based on feature fusion and semantic boundary constraint according to the application.

[0079] Figure 2 A structure diagram of the attention weight generation module according to the first embodiment of the application.

[0080] Figure 3 A segmentation head structure diagram according to the first embodiment of the application.

[0081] Figure 4 A feature fusion flowchart of the feature fusion module according to the first embodiment of the application.

[0082] Figure 5 A semantic boundary constraint flowchart of the semantic boundary supervision module according to the first embodiment of the application.

[0083] Figure 6 A network structure diagram according to the first embodiment of the application. DETAILED DESCRIPTION

[0084] The embodiments of the application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the application, and cannot be understood as a limitation on the application.

[0085] Specific implementation one: as shown in the application, a lightweight semantic segmentation method based on feature fusion and semantic boundary constraint is provided, and the method comprises: Figure 1

[0086] S1, the feature fusion module fuses feature maps from the deep layer to the shallow layer of the encoder.

[0087] Step 1.1: the encoder receives the original input image, and sequentially undergoes the processing of the encoder shallow layer to the deep layer 4 stages, and outputs the feature maps with different resolutions of stages 1-4, i.e., x1, x2, x3 and x4; ​

[0088] Step 1.2: The feature fusion module of stage 5 receives the feature map x3 of stage 3 and the feature map x4 of stage 4 as inputs, performs local feature extraction and dimension alignment on the two inputs:

[0089] x 3′ = Down(Conv 3×3 (x3))

[0090] x 4′ = Up(Conv 3×3 (x4))

[0091] wherein the resolution of the feature map x3 is larger than that of x4, and the number of channel dimensions of x3 is less than that of x4; Conv 3×3 represents a 3*3 convolution operation that extracts local region features and adjusts the number of channel dimensions of the feature map, so that the number of channel dimensions of the two inputs is consistent; Up and Down represent up-sampling operation and down-sampling operation respectively, which aims to make the spatial sizes of the feature maps of the two inputs consistent. This step outputs two feature maps with the same size;

[0092] Step 1.3: Extracting context information of different dimensions from the dimension-aligned feature maps:

[0093]

[0094]

[0095]

[0096] a4 = A(x 4′ )

[0097] wherein, respectively represent spatial pooling operation and channel pooling operation, which respectively extract spatial dimension context information and channel dimension context information. The spatial pooling is realized by maximum pooling, and the channel pooling is realized by average pooling; a3 and a4 respectively represent the results of channel pooling on the dimension-aligned shallow feature x 3′ and deep feature x 4′ ; m3 and m4 respectively represent the results of spatial pooling on the dimension-aligned shallow feature x 3′ and deep feature x 4′ . m i , a i respectively represent the spatial dimension context information and the channel dimension context information extracted after spatial pooling and channel pooling;

[0098] Step 1.4: Cross-fuse the dimension context information of the input to combine into a richer context representation:

[0099]

[0100]

[0101] wherein, denotes matrix multiplication operation; m 43 denotes the context enhanced representation fused with the low resolution feature map spatial pooling result and the high resolution feature map channel pooling result; m 34 denotes the context enhanced representation fused with the high resolution feature map spatial pooling result and the low resolution feature map channel pooling result, whose feature size is consistent with m 43 ;

[0102] Step 1.5: generate the attention weight required for feature fusion through the two context enhanced representations generated by the previous step:

[0103]

[0104] wherein, is the concatenation operation in the channel dimension; is the attention weight generation module, whose structure is shown in Figure 2 ; a5 is the attention weight generated by the context enhanced representation of stage 3 and stage 4 features through the attention weight generation module, which can adaptively emphasize the information of interest of the network while suppressing irrelevant information according to the input of the feature fusion module, whose size is consistent with the feature map size after alignment in step 1.2, and the value range is 0 to 1;

[0105] Step 1.6: fuse the aligned feature map using the attention weight in a moving average manner:

[0106] x5 = x 4′ ⊙ a5 + x 3′ ⊙ (1-a5)

[0107] wherein, denotes Hadamard product operation; x5 is the feature map obtained after fusing stage 3 feature map x3 and stage 4 feature map x4, whose size is consistent with the feature map size after alignment in step 1.2;

[0108] Step 1.7: generate the fusion feature map of stage 6 and stage 7:

[0109]

[0110]

[0111] wherein, the feature fusion module operation, i.e. the operation of steps 1.1 to 1.6, The first parameter receives high-resolution feature maps from the shallow layer, and the second parameter receives low-resolution feature maps from the deep layer; x6 and x7 represent the feature maps generated by stage 6 and stage 7, respectively.

[0112] S2, calculate the joint semantic segmentation loss according to the feature maps generated by the feature fusion stage.

[0113] Step 2.1: generate auxiliary semantic segmentation prediction probability maps pred5 and pred6 according to feature maps x5, x6 and x7 generated by feature fusion stages 5, 6 and 7, respectively, and generate main semantic segmentation prediction probability map pred7:

[0114]

[0115] denotes the segmentation head of the i-th stage, where and is an auxiliary segmentation head used to generate semantic segmentation prediction probability maps, which will be discarded during inference. The segmentation head structure is shown in Figure 3 where the input features are first smoothed by 3x3 convolution, then the element value range is adjusted using batch normalization and ReLU function, and finally the output channel number is adjusted by 1x1 convolution. pred i denotes the semantic segmentation prediction probability map of the i-th stage, which has the same spatial size as the real label size, where pred7 is the main semantic segmentation prediction probability map, and pred5 and pred6 are auxiliary semantic segmentation prediction probability maps, which are used to enhance the feature fusion effect of different stages by generating auxiliary semantic segmentation loss;

[0116] Step 2.2: calculate the joint semantic segmentation loss using OhemCELoss according to the semantic segmentation prediction probability map:

[0117]

[0118] where gt denotes the semantic segmentation label map; denotes the OhemCELoss function (Online hard example mining cross-entropy loss); l s denotes the joint semantic segmentation loss.

[0119] S3, the semantic boundary supervision module performs semantic boundary supervision on the intermediate features generated by the decoder.

[0120] Step 3.1: generate a binary boundary mask from the semantic segmentation label, and perform dilation operation on the binary boundary mask:

[0121]

[0122] wherein, denotes a Laplacian convolution operation for extracting boundary information of semantic segmentation labels; denotes a boundary dilation operation for expanding the boundary range; m b denotes a binary dilated boundary mask;

[0123] Step 3.2: generating semantic boundary labels and predicted semantic boundaries by the binary dilated boundary mask:

[0124] gt b = m b ⊙gt

[0125] pred b = m b ⊙pred i

[0126] wherein, gt b denotes semantic boundary labels; pred b denotes predicted semantic boundaries;

[0127] Step 3.3: calculating semantic boundary loss of predicted semantic boundaries by semantic boundary labels using a Focal Loss function:

[0128]

[0129] wherein, denotes a Focal Loss function, because semantic boundaries are hard samples in semantic segmentation, and there is a classification imbalance problem relative to non-semantic boundaries, therefore, the Focal Loss function is used to calculate semantic boundary loss; l b denotes a loss value of semantic boundaries.

[0130] S4, optimizing the network by semantic segmentation loss and semantic boundary loss.

[0131] Step 4.1: optimizing semantic segmentation accuracy and semantic boundary prediction accuracy of the network by weighted sum of loss values:

[0132] l = l s + ε·l b

[0133] wherein, ε is a weight coefficient for balancing semantic segmentation loss and semantic boundary loss; l is the final network loss.

[0134] As Figure 4 shown, the feature fusion module realizes efficient feature fusion effect in the following way:

[0135] First, local region feature extraction and dimension alignment are performed on feature maps from different stages with different spatial resolutions, which are realized by two non-shared parameter 3x3 convolutions and two bilinear interpolation operations, respectively. Second, to achieve effective feature fusion, spatial pooling and channel pooling operations are used to extract the context information of the spatial and channel dimensions of the two input feature maps. The spatial pooling operation is realized by max pooling, and the channel pooling operation is realized by average pooling. Then, Hadamard product is used to fuse the context information of different dimensions. Then, the attention weight generation module generates attention weights by receiving the context enhanced representation concatenated in the channel dimension. The module first compresses the channel dimension by 1x1 convolution to reduce the amount of calculation; then uses regular batch normalization and ReLU function to adjust the element value range; then uses 1x1 convolution to adjust the number of feature channels to be consistent with the dimension-aligned features; finally, the sigmoid function is used to generate attention weights with element value range of 0 to 1. The weight is used to perform Hadamard product operation on the dimension-aligned features in a soft selection manner, and two attention features are obtained. Finally, element-wise summation operation is performed on the attention feature maps, and 3x3 convolution is used to aggregate the features and compress the channel dimension of the feature map, obtaining the fused feature map, which fuses the feature information from the shallow and deep layers and is passed to the next feature fusion module as a low-resolution input.

[0136] As shown in Figure 5 , the semantic boundary supervision module achieves effective semantic boundary constraints in the following ways:

[0137] First, in the training stage, in order to extract the boundary information of the real label, Laplacian Convolution is used to generate a binary boundary mask for the real label, if it is a boundary, the mask element value is 1, otherwise 0. Secondly, in order to expand the supervision range of the semantic boundary, the binary boundary mask is dilated to obtain a binary boundary dilated mask. Then, the binary boundary dilated mask is used to perform Hadamard product operation with the real label and the semantic segmentation prediction probability graph respectively, and the semantic boundary label and the semantic boundary prediction probability graph are obtained respectively. Finally, the semantic boundary label is used to constrain the semantic boundary prediction probability graph during training, so as to optimize the accuracy of the semantic boundary.

[0138] The common CNN model can be used as the encoder of the network, as shown in the figure, the encoder will output four basic feature maps with different resolutions. The decoder of the network includes three feature fusion modules, which generate feature maps with more rich representation information by fusing basic feature maps of different stages in a cascading manner, that is, the fusion features of the previous stage are further transmitted as input to the feature fusion module of the next stage. In addition, in order to improve the prediction accuracy of the semantic boundary, the semantic boundary supervision module is used to supervise the fusion feature map of the second layer; in order to enhance the fusion effect of different fusion stages, the intermediate supervision is used to supervise the fusion feature map of different stages during training. Figure 6

[0139] In summary, in view of the shortcomings of the prior art, the present application proposes a novel real-time semantic segmentation network, which adopts an encoder-decoder (Encoder-Decoder) structure. The encoder can be any computationally efficient CNN model. In the encoding stage, the convolution operation in the encoder generates feature maps containing different feature information. In the decoding stage, a series of feature fusion modules are combined into a lightweight and efficient decoder in a cascading manner, and the decoder receives feature maps of different stages as input and uses attention mechanism to fuse feature maps of different stages through the context information of the input feature maps. This way bridges the semantic gap between different stages by emphasizing the information of interest to the network while suppressing irrelevant information, effectively fusing feature information of different stages and improving segmentation accuracy. In order to further optimize the accuracy of the semantic boundary, the present application proposes a semantic boundary supervision module, which uses the semantic boundary extracted from the ground truth to constrain the predicted semantic boundary of the network, thereby improving the semantic boundary prediction accuracy of the network.

[0140] ​Although embodiments of the present application have been shown and described, it would be apparent to those skilled in the art that many more modifications, substitutions, replacements and changes can be made to these embodiments without departing from the principles and the scope of the present application, the scope of which is defined by the claims and their equivalents.

Claims

1. A lightweight semantic segmentation method based on feature fusion and semantic boundary constraint, characterized in that, The method comprises: S1, a feature fusion module fuses feature maps from the deep to the shallow layers of an encoder; S2, a joint semantic segmentation loss is calculated according to the feature maps generated in the feature fusion stage; S3, a semantic boundary supervision module performs semantic boundary supervision on the intermediate features generated by the decoder; S4, the network is optimized through a semantic segmentation loss and a semantic boundary loss; The specific steps of S1 include: S1.1: The encoder receives the original input image, and sequentially processes the image through the four stages of the encoder shallow layer to deep layer, and outputs the feature maps with different resolutions of stages 1 to 4, i.e., stage 1 feature map , stage 2 feature map , stage 3 feature map and stage 4 feature map ; S1.2: The feature fusion module of stage 5 receives the stage 3 feature maps and the stage 4 feature maps As input, local feature extraction and dimension alignment are performed on both inputs: wherein the stage 3 feature map has a resolution larger than the stage 4 feature map and a number of channel dimensions less than the stage 4 feature map ; represents a 3*3 convolution operation for adjusting the number of channels of the feature map to make the number of channels of the two input feature maps consistent; and respectively represent an up-sampling operation and a down-sampling operation for making the spatial sizes of the two input feature maps consistent; S1.3, context information of different dimensions is extracted from the dimension-aligned feature maps; wherein, , respectively represent a spatial pooling operation and a channel pooling operation, which respectively extract spatial dimension context information and channel dimension context information, the spatial pooling is realized by maximum pooling, and the channel pooling is realized by average pooling; and respectively represent results of performing channel pooling on the dimension-aligned shallow layer feature and deep layer feature ; and respectively represent results of performing spatial pooling on the dimension-aligned shallow layer feature and deep layer feature ; S1.4, the context information of each dimension of the input is cross-fused to form a more rich context representation; wherein, denotes a matrix multiplication operation; denotes a context enhanced representation that fuses the low resolution feature map spatial pooling result and the high resolution feature map channel pooling result; denotes a context enhanced representation that fuses the high resolution feature map spatial pooling result and the low resolution feature map channel pooling result, with feature dimensions consistent with ; S1.5, the attention weight required for feature fusion is generated through the two context-enhanced representations generated in S1.4; wherein, is a serial operation for the channel dimension; is an attention weight generation module; is a context enhanced representation of stage 3 and stage 4 features, which is generated by the attention weight generation module, the attention weight is consistent with the feature map size after S1.2 alignment, and the value range is 0 to 1; S1.6, the attention weight is used to fuse the dimension-aligned feature maps through moving average; wherein, ⊙ represents Hadamard product operation; For stage 3 feature map And stage 4 feature map The feature map obtained after fusion, The size of the feature map after alignment with S1.2 is consistent with the size of the feature map. S1.7, the fusion feature maps of stages 6 and 7 are generated; wherein, The feature fusion module operates, i.e., the operations of S1.1~1.6, The first parameter receives a high-resolution feature map from a shallow layer, and the second parameter receives a low-resolution feature map from a deep layer; and respectively represent the feature maps generated in stages 6 and 7.

2. The lightweight semantic segmentation method based on feature fusion and semantic boundary constraint according to claim 1, characterized in that, The convolution operation is used to extract local region features and adjust the channel number of the feature maps so that the channel numbers of the two inputs are consistent. 3.The lightweight semantic segmentation method based on feature fusion and semantic boundary constraint of claim 1, wherein, The attention weight adaptively emphasizes the information of interest of the network and suppresses irrelevant information according to the input of the feature fusion module. 4.The lightweight semantic segmentation method based on feature fusion and semantic boundary constraint of claim 1, wherein, The specific steps of S2 include: S2.1: feature maps generated according to features fusion stages 5, 6 and 7 , and auxiliary semantic segmentation prediction probability maps and main semantic segmentation prediction probability maps : wherein, denotes the segmentation head of the stage, which is used to generate a semantic segmentation prediction probability map and will be discarded during inference; denotes the semantic segmentation prediction probability map of the stage, which has the same spatial size as the real label and is used to enhance the feature fusion effect of different stages by generating auxiliary semantic segmentation loss; S2.2, OhemCELoss is used to calculate the joint semantic segmentation loss according to the semantic segmentation prediction probability map: wherein, represents a semantic segmentation label map; represents an OhemCELoss function; represents a joint semantic segmentation loss; i represents a stage number.

5. The lightweight semantic segmentation method based on feature fusion and semantic boundary constraint according to claim 4, characterized in that, The specific steps of S3 include: S3.1, a binary boundary mask is generated from the semantic segmentation label, and an inflation operation is performed on the binary boundary mask; wherein, represents a Laplacian convolution operation for extracting boundary information of the semantic segmentation label; represents a boundary dilation operation for expanding the boundary range; represents a binary dilated boundary mask; S3.2, a semantic boundary label and a predicted semantic boundary are generated through the binary inflation boundary mask; wherein, represents a semantic boundary tag; represents a semantic boundary; S3.3, the Focal Loss function is used to calculate the semantic boundary loss of the predicted semantic boundary through the semantic boundary label: wherein, represents a Focal Loss function; represents a loss value of a semantic boundary.

6. The lightweight semantic segmentation method based on feature fusion and semantic boundary constraint according to claim 5, characterized in that, The specific steps of S4 include: the semantic segmentation accuracy and the semantic boundary prediction accuracy of the network are optimized through weighted summation of the loss values: wherein, is a weight coefficient for balancing the semantic segmentation loss and the semantic boundary loss; is the final network loss.

7. The method of claim 1, wherein the method further comprises: The specific steps of the feature fusion module for realizing feature fusion include: The feature maps from different stages with different spatial resolutions are respectively passed through two non-shared parameters convolution and two bilinear interpolations for feature extraction and dimension alignment of local regions; Spatial pooling and channel pooling are used to extract the spatial dimension and channel dimension context information of the two input feature maps to realize effective feature fusion; Hadamard product is used to fuse context information of different dimensions; The attention weight generation module generates the attention weight by receiving the context-enhanced representation after concatenation of the channel dimension, which first compresses the channel dimension through 1x1 convolution to reduce the calculation amount; then uses regular batch normalization and ReLU function to adjust the element value range; then uses 1x1 convolution to adjust the feature channel number to be consistent with the dimension-aligned feature; finally, the sigmoid function is used to generate the attention weight with an element value range of 0 to 1, which performs Hadamard product operation on the dimension-aligned feature through soft selection to obtain two attention feature maps; Perform element-wise summation on the attention feature map, and use... Convolution is used to aggregate features and compress the channel dimension of the feature map to obtain a fused feature map. This feature map combines feature information from shallow and deep layers and is passed as a low-resolution input to the next feature fusion module. 8.The lightweight semantic segmentation method based on feature fusion and semantic boundary constraint of claim 1, wherein, The specific steps of the semantic boundary supervision module for realizing semantic boundary constraint include: In the training stage, a binary boundary mask is generated from the real label using Laplacian convolution, and if it is a boundary, the mask element value is 1, otherwise it is 0, which is used to extract the boundary information of the real label; The binary boundary mask is dilated to obtain a binary boundary dilated mask, which is used to expand the supervision range of the semantic boundary; The binary boundary dilated mask is subjected to Hadamard product operation with the real label and the semantic segmentation prediction probability map respectively to obtain a semantic boundary label and a semantic boundary prediction probability map respectively; The semantic boundary label is used to constrain the semantic boundary prediction probability map to strengthen the constraint of the network on the semantic boundary during training. 9.The lightweight semantic segmentation method based on feature fusion and semantic boundary constraint of claim 1, wherein, The encoder includes three feature fusion modules, and four basic feature maps of different resolutions are output simultaneously, and the feature maps of rich representation information are generated by fusing the basic feature maps of different stages in a cascading manner.

Images