Edge auxiliary feature calibration method for real-time semantic segmentation

Abstract

The invention provides an edge auxiliary feature calibration method for real-time semantic segmentation, the edge auxiliary feature calibration method is realized based on a real-time semantic segmentation network, and the real-time semantic segmentation network comprises a semantic branch and an edge branch; the edge auxiliary feature calibration method comprises the following steps: S1, inputting an image into an encoder for feature extraction; s2, extracting edge information through edge branches; s3, a decoder decodes the semantic features and the edge information output by the encoder; and S4, enabling the output of the decoder to pass through an edge segmentation head, and obtaining a feature map after semantic segmentation. According to the method, the problem that detail information is easy to submerge is solved by optimizing a feature fusion strategy. Features with different resolutions are extracted from the encoder part to serve as edge branches for reserving spatial information lost due to down-sampling; a GAPPM module is introduced at the tail end of the encoder to enhance the context feature aggregation capability of the network; an EGRFM module is provided to enhance the detailed information of the network submerged during feature fusion.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and computer vision, and in particular to an edge auxiliary feature calibration method for real-time semantic segmentation. Background Art

[0002] Semantic segmentation is a basic task in visual scene parsing, and its goal is to assign each pixel in the input image to a specific category label. With the development of artificial intelligence, the demand for intelligence is increasing, and semantic segmentation has become a basic requirement for applications such as autonomous driving, medical image analysis, and remote sensing image processing. With the development of technology, deep learning has gradually taken a dominant position in the field of semantic segmentation. Although these models achieve high segmentation accuracy, they require huge computational costs and are difficult to apply to application fields with high real-time requirements, such as autonomous driving and robotic surgery. This is obviously an insurmountable gap for large-scale models used for segmentation.

[0003] Real-time semantic segmentation requires that each pixel in the image be classified at real-time speed (FPS ≥ 30) to achieve pixel-level semantic segmentation. In order to meet real-time requirements, researchers have proposed many efficient semantic segmentation models in the past. Many methods try to design lightweight networks or efficient decoders to achieve real-time purposes. Although these methods reduce the amount of computation and speed up the inference speed, there is still a problem of information loss leading to mismatch of input features in lightweight networks. At the same time, for different inputs, the context modeling method in the decoder lacks robustness, resulting in mismatch of context information. These problems not only limit the real-time performance of segmentation, but also affect the accuracy of segmentation, limiting the practical application of semantic segmentation technology.

[0004] With the continuous advancement of real-time semantic segmentation methods, multi-branch network architectures have increasingly become the dominant approach to address these challenges. PIDNet (a PID controller inspired network) introduced a new three-branch network architecture, demonstrated the effectiveness of edge information in segmentation, and provided valuable insights to the field. Recent studies have improved the network's ability to capture fine-grained details by using edge detection algorithms as an auxiliary mechanism. Considering the important impact of edge information on network performance, our goal is to integrate edge information into the network with minimal computational cost to improve the overall efficiency of the network. However, these works mainly focus on the effective extraction of edge information, while ignoring the importance of its proper integration in the network. Summary of the invention

[0005] The present invention aims to at least solve the technical problems existing in the prior art, and in particular innovatively proposes an edge-assisted feature calibration method for real-time semantic segmentation.

[0006] In order to achieve the above-mentioned object of the present invention, the present invention provides an edge auxiliary feature calibration method for real-time semantic segmentation, wherein the edge auxiliary feature calibration method is implemented based on a real-time semantic segmentation network, wherein the real-time semantic segmentation network adopts a dual-branch structure, including a semantic branch and an edge branch; the semantic branch includes an encoder and a decoder; the edge auxiliary feature calibration method comprises the following steps:

[0007] S1, input the image to the encoder to extract features; the encoder is a five-stage downsampling process, and the size of the feature map in each stage is 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image; the feature map of each stage of the encoder contains semantic features, and the final output of the encoder is a 1 / 32 feature map;

[0008] The feature maps of 1 / 4, 1 / 8, and 1 / 16 are used as the input of the three stages in the edge branch respectively; and the feature maps of 1 / 8, 1 / 16, and 1 / 32 are used as the low-level detail spatial features of the third EGRFM module, the second EGRFM module, and the first EGRFM module respectively;

[0009] S2, extract edge information through edge branch, edge branch is a three-stage convolution process, including three convolution blocks, each stage corresponds to a convolution block, and the size of the feature map of each stage is 1 / 8 of the original image; wherein, the input of the first convolution block is the 1 / 4 feature map, which is convolved to obtain a feature map of size 1 / 8; the input of the second convolution block is the 1 / 8 feature map and the 1 / 8 feature map output by the encoder, which is convolved and concat to obtain a feature map of size 1 / 8; the input of the third convolution block is the 1 / 8 feature map and the 1 / 16 feature map output by the encoder, which is convolved and concat to obtain a feature map of size 1 / 8; the 1 / 8 feature map is a feature map containing edge information;

[0010] S3, the decoder decodes the semantic features and edge information output by the encoder; the decoder is composed of three cascaded EGRFM modules (edge-guided resampling fusion modules); the data processing process through the EGRFM module is:

[0011] The input of the EGRFM module consists of three parts: high-level semantic features, low-level detail spatial features, and edge information. At this time, the high-level semantic features of the first EGRFM module are the final output feature map of the encoder, the high-level semantic features of the second EGRFM module are the output feature map of the second EGRFM module, and the high-level semantic features of the third EGRFM module are the output feature map of the second EGRFM module.

[0012] First, the input high-level semantic features and low-level detail spatial features are spliced ​​on the channel, and then the spliced ​​features are upsampled to obtain a 2D offset feature map; then the 2D offset feature map is split and used as a reference to resample the high-level semantic features and low-level detail spatial features; finally, the input edge information is used as a weight guide to fuse the resampled high-level semantic features and low-level detail spatial features to obtain the final output of the EGRFM module;

[0013] S4, passes the output of the decoder through the edge segmentation head to obtain the feature map after semantic segmentation.

[0014] The network's ability to distinguish edges is enhanced by extracting edge information of objects in the image. At the same time, the feature resampling method is used to calibrate high-level semantic features and low-level detail spatial features to alleviate the problem of spatial feature misalignment. The calibrated features are weighted and fused by introducing edge information to maintain a faster inference speed.

[0015] Furthermore, the data processing process of the EGRFM module includes:

[0016] First, the high-level semantic features and low-level detail spatial features are compressed through 1×1 convolution and concatenated in the channel dimension to obtain F concat ; The mathematical formula of the above steps is expressed as:

[0017] F concat =concat(Up(Conv1(F h )),Conv1(F l ))

[0018] Where concat(·) represents the concatenation operation;

[0019] Up(·) indicates upsampling;

[0020] Conv1 represents a 1×1 convolution operation;

[0021] F l Represents low-level detail spatial features;

[0022] F h Represents high-level semantic features;

[0023] Secondly, learn the 2D offset feature map Δ for the concatenated features l , the offset map is divided into two groups Δ1 and Δ2, which are used to resample the high-level semantic features and low-level semantic features; the sampling function It is expressed as:

[0024]

[0025] τ (h,w) =max(0,1-|hh′+Δ1|)*max(0,1-|ww′+Δ2|)

[0026] in Represents the sampling features at the point (h+Δ1, w+Δ2);

[0027] H′ represents the height of the feature offset map;

[0028] W′ represents the width of the feature offset map;

[0029] F (h′,w′) represents the feature at point (h′, w′) in the feature offset graph;

[0030] τ (h,w) represents the weight at the point (h, w);

[0031] h and h′ represent the ordinates of the corrected point and the point on the offset feature map, respectively;

[0032] w and w′ represent the horizontal coordinates of the corrected point and the point on the offset feature map, respectively;

[0033] Δ1 is the high-level semantic feature offset map;

[0034] Δ2 is the low-level semantic feature offset map;

[0035] During the sampling process, we look for the most favorable feature point for the current point to replace the point at the current position to achieve the purpose of calibration; at the same time, each group of offset maps is further divided into two groups to ensure that the calibration process is more precise and accurate;

[0036] Overall, for the calibrated features, its high-level semantic features F h 'Still contains more semantic information, low-level detail spatial features F l ' contains more spatial geometric details, and directly calibrating them and then fusing them does not improve their performance much. Since the edge branch can better preserve the spatial detail information, the edge branch is used as a guide to adaptively fuse the calibrated semantic features F h 'With the detail feature F l ', the output of EGRFM is written as:

[0037] σ=1+tanh(V Edge )Sigmoid(V Edge )

[0038]

[0039] in Represents edge features;

[0040] tanh() represents the tanh activation function;

[0041] Sigmoid() represents the sigmoid activation function;

[0042] Represents matrix multiplication;

[0043] Out EGRFM Represents the output of the EGRFM module;

[0044] F l ′ represents the corrected low-level detail spatial features;

[0045] F h ′ represents the corrected high-level semantic features;

[0046] Represents the weight of edge feature generation.

[0047] Furthermore, there is a GAPPM module (global average pyramid pooling module) between the encoder and the decoder. At this time, the high-level semantic features of the first EGRFM module are the output feature maps of the GAPPM module. The data processing process of the GAPPM module includes:

[0048] First, the features are globally averaged pooled and channel compressed using Conv. Then, the pooled features are pooled with different kernel sizes to obtain features of different scales. The pooled features are added to the channel compressed features respectively, and then concatenated and convolved, and input into the BiAttention module. The BiAttention module processes the input features and the channel compressed features, fuses and multiplies them, and obtains the final result.

[0049] The specific processing flow of the BiAttention module is as follows: receive two inputs, namely the features after channel compression and the features after pooling and splicing; directly add the two features and use Conv to perform channel compression to obtain the weight σ; then multiply the weight σ with the input channel compressed features to obtain feature one, multiply the weight 1-σ with the input pooled and spliced ​​features to obtain feature two, and then add feature one and feature two to obtain the final output.

[0050] The global average pyramid pooling module is designed to aggregate global context information and improve the extraction of semantic information.

[0051] Furthermore, the encoder is a lightweight backbone network STDC.

[0052] Furthermore, the loss function in the real-time semantic segmentation network training process is:

[0053] Loss=λ0·Loss0+λ1·Loss1+λ2·Loss2+λ3·Loss Edge0 +λ4·Loss Edge1

[0054] Among them, λ0, λ1, λ2, λ3, and λ4 are training loss parameters;

[0055] Loss0, Loss1, and Loss2 are the loss functions of the first, second, and third semantic segmentation heads, respectively;

[0056] Loss Edge0 is the marginal loss function;

[0057] Loss Edge1 is the marginal loss regularization function.

[0058] Furthermore, Loss0, Loss1, and Loss2 use cross entropy loss, and the formula is as follows:

[0059]

[0060] Where L CE represents the cross entropy loss;

[0061] N represents the sample size;

[0062] L i represents the cross entropy loss of each sample;

[0063] M represents the number of categories;

[0064] y ic Indicates whether the true category of sample i is equal to c, and the value is 0 or 1;

[0065] p ic It represents the probability that the predicted sample i belongs to category c.

[0066] Furthermore, Loss Edge0 The weighted binary cross entropy loss is used, and the formula is as follows:

[0067]

[0068] Where L_{BCE} represents the weighted binary cross entropy loss;

[0069] N represents the number of samples;

[0070] α i represents the weight of sample i;

[0071] y iIndicates the label of sample i, the value is 0 or 1;

[0072] p i It represents the probability that sample i is the correct sample.

[0073] Furthermore, Loss Edge1 The formula is as follows:

[0074]

[0075] Where L reg represents marginal loss;

[0076] {1:s>thrs} indicates whether s is greater than the set threshold thrs. If so, the value is 1, otherwise 0;

[0077] M represents the number of categories;

[0078] s represents the output of the edge segmentation head when sample i belongs to class c;

[0079] thrs is the threshold value;

[0080] y ic Indicates whether the true category of sample i is equal to c, and the value is 0 or 1;

[0081] p ic It represents the probability that the predicted sample i belongs to category c.

[0082] Furthermore, when the real-time semantic segmentation network is trained, the real-time semantic segmentation network also includes:

[0083] The first semantic segmentation head and the second semantic segmentation head are added after the first EGRFM module and the second EGRFM module respectively; the output feature maps of the first semantic segmentation head and the second semantic segmentation head are upsampled to make them the same size as the training set images, wherein the first semantic segmentation head and the second semantic segmentation head are used to accelerate the convergence of the network model; the loss calculation is performed on the images sampled by the first semantic segmentation head and the second semantic segmentation head and the training set images respectively, and the loss calculation formulas are loss functions loss0 and loss1 respectively, and the network parameters of the real-time semantic segmentation network are adjusted by minimizing the loss function;

[0084] Add an edge segmentation head after the edge branch; calculate the loss of the output image of the edge segmentation head and the training set image respectively, and adjust the network parameters of the real-time semantic segmentation network; the formulas for loss calculation are loss functions Loss-Edge1 and Loss-Edge0 respectively; use the Loss-Edge0 loss function to extract edge information from the Ground Truth image with real labels through the Canny operator and calculate the loss; regularize the results of the edge segmentation head and the semantic segmentation head through the Loss-Edge1 loss function and calculate the loss;

[0085] Furthermore, the upsampling process includes: first upsampling through bilinear interpolation, then through a 3x3 convolution and BatchNorm, and finally through the Relu activation function to obtain the output result.

[0086] The segmented results are enhanced by upsampling after each semantic segmentation head.

[0087] In summary, due to the adoption of the above technical solutions, the edge-guided resampling fusion module (EGRFM) designed in the present invention uses edge information to fuse the resampled and calibrated features to strengthen feature representation. The problem of feature misalignment that occurs when fusing high-resolution and low-resolution features is solved, thereby enhancing the integration of edge information in deep learning models. EGRFM predicts an offset map from high-resolution and low-resolution features, and then resamples the features based on the offset map. Using edge information as a guide to increase the weight of detail features is conducive to the fusion of high-level semantic features with low-level details. This helps the network to segment object edges more effectively. Edge information is extracted from different stages of the main branch and refined using the Canny operator to enhance edge features. In addition, a global average pyramid pooling module (GAPPM) is designed to extract context information through multi-level pooling, which improves segmentation accuracy at a small additional inference time cost; this effectively captures multi-scale context and further enhances the model's ability to handle complex tasks. Based on the above modules, a real-time semantic segmentation network EFCNet is proposed, which achieves a good trade-off between accuracy and speed.

[0088] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0090] Figure 1 It is a schematic diagram of the network structure of EFCNet of the present invention.

[0091] Figure 2 It is a structural schematic diagram of the EGRFM module of the present invention.

[0092] Figure 3 It is a structural schematic diagram of the GAPPM module of the present invention.

[0093] Figure 4 It is the visualization of the intermediate process diagram of the present invention, Figure 4 (a), (b), (c), and (d) are process diagrams of segmentation heads 1, 2, 3, and edge segmentation heads, respectively, where segmentation heads 2 and 3 are auxiliary segmentation heads.

[0094] Figure 5 This is a network prediction effect diagram of the present invention. Figure 5 (a), (b), and (c) are the original image, the segmentation prediction image, and the label image corresponding to the original image, respectively.

[0095] Figure 6 This is a network visualization comparison diagram of the present invention on the Cityscapes validation set. Figure 6 (a), (b), (c), (d), and (e) are the original image, the real label image, the prediction effect of the ppLiteSeg network, the prediction effect of the STDC-Seg network, and the prediction effect of the EFCNet network, respectively. DETAILED DESCRIPTION

[0096] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0097] 1. Related Work

[0098] In recent years, many methods have improved the performance of semantic segmentation in challenging scenarios, including high-precision semantic segmentation and real-time semantic segmentation.

[0099] 1.1 High-precision semantic segmentation

[0100] Early semantic segmentation methods were mainly based on image segmentation and region-based techniques. Among them, the image segmentation (Graph Cuts)-based method has achieved some success in segmentation accuracy, but its computational complexity is high, it requires a long running time, is inefficient for large-scale image processing, and is not suitable for real-time scenarios. Region-based methods rely on manually designed features and heuristic rules, which limits their generalization ability in complex scenarios. In recent years, with the development of deep learning, semantic segmentation methods based on convolutional neural networks (CNNs) have made significant progress.

[0101] The introduction of Fully Convolutional Networks (FCN) makes end-to-end semantic segmentation possible. U-shaped networks and encoder-decoder networks have also been proposed. In the downsampling process of the encoder-decoder network, spatial details are easily ignored. In order to alleviate this problem, dilated convolution is proposed to expand the field of view without reducing the spatial resolution. The DeepLab series of models increase the receptive field by applying convolution kernels with different dilation rates in the network, thereby improving the performance of semantic segmentation tasks. PSPNet introduces a pyramid pooling module (PPM) to parse multi-scale context information. HRNet (see the paper Wang J, Sun K, Cheng T, et al. Deep high-resolution representation learning for visual recognition [J]. IEEE transactions on pattern analysis and machine intelligence, 2020, 43 (10): 3349-3364.) uses multi-path and bilateral connections to learn and fuse representations of different scales. Some works use self-attention mechanisms to dynamically capture the global context information of each pixel. However, under the same resource conditions, they require more computing time and are therefore not applicable in real-time scenarios. ANNN (see Zhu Z, Xu M, Bai S, et al. Asymmetric non-local neural networks for semantic segmentation [C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2019: 593-602.) uses a pyramid sampling module to reduce the burden of self-attention calculation while maintaining sensitivity to global context. However, ANNN still has deficiencies in multi-scale context extraction. SFNet and AlignSeg propose to align multi-level features by learning 2D transformation offsets. However, they ignore the representation differences between features at different levels and sub-features at the same level in this process, which weakens the effectiveness of information propagation.

[0102] 1.2 Real-time semantic segmentation

[0103] Real-time semantic segmentation aims to produce high-quality predictions while maintaining high efficiency. At present, many different works have been proposed to solve the challenge of real-time semantic segmentation. These strategies mainly include lightweight encoder-decoder structure, multi-branch structure, network compression, and changing input size. Below we will elaborate on the classic methods of each strategy.

[0104] 1.2.1 Lightweight Encoder-Decoder Structure

[0105] DFANet introduces a lightweight backbone network to the Xception architecture based on depthwise separable convolutions and reduces the input size for faster inference speed. ShuffleSeg adopts ShuffleNet, which combines channel shuffling and group convolution as its backbone to reduce computational cost. However, these networks still take the form of encoder-decoder architecture, which requires information flow to pass through the deep encoder and then back to the decoder, which introduces too much latency. In addition, since the optimization of depthwise separable convolutions on GPUs is still immature, traditional convolutions are faster while having more FLOPs and parameters. Therefore, we seek more efficient models to avoid convolution decomposition and encoder-decoder architectures.

[0106] 1.2.2 Multi-branch structure

[0107] Effectively fusing global information with local information is an important challenge for achieving accurate semantic segmentation. Encoder-decoder based methods usually address this problem by progressively downsampling feature maps, i.e. extracting higher and higher high-level semantic information at each scale. The main challenge of this approach is to retain the high-resolution details extracted early in the network, so multi-branch architectures are proposed to address this problem by independently extracting features at different scales. For example, the BiSeNet family uses a two-branch structure to encode spatial information and semantic information respectively. DMA-Net recursively fuses features from high-order and low-order branches to produce more robust features. SwiftNet uses lateral connections to fuse features from branches of different resolutions to handle multi-scale issues. STDC-Seg proposes a backbone network designed specifically for semantic segmentation, replacing the spatial path in BiSeNet with a detail aggregation module, which improves efficiency without compromising performance. DDRNet proposes a novel bilateral network with deep dual-resolution branches and multiple bilateral fusions, which results in almost no increase in inference time when performed on low-resolution feature maps. PIDNet links deep convolution with a PID controller and proposes a three-branch network based on the PID controller architecture. DMRNet modifies the ResNet network into a multi-resolution architecture, enabling it to fully extract image information at different resolutions in the downsampling stage.

[0108] 2. EFCNet Architecture

[0109] 2.1 Network Architecture

[0110] The EFCNet network adopts a dual-branch structure and selects the lightweight backbone network STDC as the encoder (see the paper FanM, Lai S, Huang J, et al. Rethinking bisenet for real-time semantic segmentation [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2021: 9716-9725 for details) to extract multi-layer features. Specifically, the proposed EFCNet has two versions, EFCNet-S and EFCNet-B. When STDC1 is the encoder, it is EFCNet-S; when STDC2 is the encoder, it is STDC2. In order to enhance the network's ability to segment the edges of objects, a feature map with a resolution of 1 / 8 of the input image resolution is created through the trunk branch as an edge branch. Afterwards, a global average pyramid pooling module (GAPPM) is designed, which uses pooling kernels of different sizes to extract contextual information of different scales, and fuses multi-scale global contextual information to enrich feature representation. In addition, an edge-guided resampling fusion module (EGRFM) is constructed, which resamples and calibrates the contextual information and spatial geometric information, and uses the edge branch as a guide to fuse the contextual information and spatial geometric information, thereby enhancing the network's ability to distinguish object edges and small targets.

[0111] The overall structure of the EFCNet network is as follows: Figure 1 As shown in the figure: First, a picture is passed in, and the encoder is the semantic branch. Encoding is performed through the encoder, and the encoding process is the process of downsampling. The encoder is divided into five stages, and the size of the feature map of each stage is 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image. We extract features from the three middle stages to form our edge branches, that is, the 1 / 4, 1 / 8, and 1 / 16 parts. The features of this part are directly fused after convolution processing, and are used for loss calculation and as the input of the EGRFM module. The input of the EGRFM module: the input of the previous module, the information of the edge branch, and the features of each stage of the previous encoder part. A semantic segmentation head is added after each EGRFM module. The module parameters of the network are adjusted by calculating the output of the edge segmentation head and the loss of the Ground Truth. At the end of the semantic branch, the global average pyramid pooling module (GAPPM) is introduced to extract spatial feature information of different sizes to improve the network's ability to obtain global information.

[0112] The output of the edge branch is the input of the E-Head. The loss is calculated for the output of the E-Head and the image processed by Canny (the Canny algorithm is used to extract edge features of images with real labels). The loss formula is Loss-Edge0, and then the gradient is back-transferred to update the convolution module parameters of the edge branch.

[0113] The results of the edge segmentation head and the semantic segmentation head are regularized and the loss is calculated through the Loss-Edge1 loss function. The segmentation head corresponding to Loss2 is the final result output segmentation head. The role of the segmentation head corresponding to loss0 and loss1 loss is to accelerate the convergence of the network model. E-up acts after each semantic segmentation head to enhance the segmented results. First, upsampling is performed through bilinear interpolation, then through a 3x3 convolution and BatchNorm, and finally the output result is obtained through the Relu activation function.

[0114] BiSeNetV2 proposed an enhanced training strategy, which is to add additional segmentation heads at the end of each stage of the semantic branch. In order to obtain better results, SFNet adopted a similar strategy called cascaded deep supervised learning. Inspired by BiseNetV2 and SFNet, a semantic segmentation head is placed after the first EGRFM and the second EGRFM modules to generate additional semantic losses Loss0 and Loss1 to better optimize the entire network. The semantic segmentation head placed after the last EGRFM module is the final output result of the network. Its loss Loss2 is CE-Loss as are Loss0 and Loss1. The specific formula can be expressed as:

[0115]

[0116] Where L CE represents the cross entropy loss;

[0117] N represents the sample size;

[0118] L i represents the cross entropy loss of each sample;

[0119] M represents the number of categories;

[0120] y ic Indicates whether the true category of sample i is equal to c, and the value is 0 or 1;

[0121] p ic It represents the probability that the predicted sample i belongs to category c.

[0122] For the edge branch, the weighted binary cross entropy loss Loss-Edge0 is used to solve the problem of sample imbalance in edge detection, because the coarse edge is more suitable for highlighting the edge area and enhancing the characteristics of small targets. The specific formula of Loss-Edge0 can be expressed as:

[0123]

[0124] Where L_{BCE} represents the weighted binary cross entropy loss;

[0125] N represents the number of samples;

[0126] α i represents the weight of sample i.

[0127] y i Indicates the label of sample i, the value is 0 or 1;

[0128] p i Represents the probability that sample i is the correct sample;

[0129] Inspired by Gated-SCNN, we use Loss-Edge1 to regularize the edge loss and use the output of the edge segmentation head to coordinate semantic segmentation and edge detection to ensure the consistency of the edge branch and the semantic branch. The calculation formula of Loss-Edge1 can be written as:

[0130]

[0131] Where L reg Represents the edge loss regularization process;

[0132] {1:s>thrs}(y ic log(p ic )) represents {1:s>thrs} and (y ic log(p ic ))Multiply;

[0133] {1:s>thrs} indicates whether s is greater than the set threshold thrs. If so, the value is 1, otherwise 0;

[0134] M represents the number of categories;

[0135] s represents the output of the edge segmentation head when sample i belongs to class c;

[0136] thrs is the threshold value;

[0137] y ic Indicates whether the true category of sample i is equal to c, and the value is 0 or 1;

[0138] p icIt represents the probability that the predicted sample i belongs to category c.

[0139] Therefore, the final loss of EFCNet is:

[0140] Loss=λ0·Loss0+λ1·Loss1+λ2·Loss2+λ3·Loss Edge0 +λ4·Loss Edge1

[0141] Among them, Loss0, Loss1, and Loss2 are the loss functions of the first semantic segmentation head, the second semantic segmentation head, and the third semantic segmentation head respectively;

[0142] Loss Edge0 is the marginal loss function;

[0143] Loss Edge1 is the marginal loss regularization function;

[0144] According to experience, the training loss parameters of EFCNet are set to λ0=0.4, λ1=0.4, λ2=1, λ3=20, λ4=1, and thrs=0.8.

[0145] 2.2 Edge-Guided Resampling Fusion Module (EGRFM)

[0146] In order to compensate for the loss of spatial details caused by gradual downsampling, cross-layer feature fusion and multi-stage feature fusion methods are used to enhance high-level semantic features and low-level detail spatial features. and high-level semantic features First, the low-level detail spatial features F are upsampled by bilinear interpolation. l Sampling to high-level semantic features F h The size is consistent, and then the high-level semantic features F h and the sampled low-level detail spatial feature F l 'Direct fusion by splicing, addition or cascading.

[0147] It is well known that although high-level features contain rich semantic information, they lack spatial detail information, especially for edge areas. Although low-level features retain spatial detail information well, they only retain part of the semantic information and cannot capture high-level semantics well. To solve this problem, many semantic segmentation methods adopt the strategy of cross-layer feature fusion, combining high-level semantic features with low-level detail spatial features to fully utilize their advantages. When low-level detail information is fused with high-level semantic information, part of the semantic information covered by low-level detail spatial features will confuse high-level semantic features. At the same time, due to the inaccurate restoration of spatial position information during upsampling and the feature redundancy of high-level semantic information, direct fusion will cause excessive erosion of surrounding pixels on the edge of the target, resulting in small targets being easily submerged by large targets, and cannot achieve satisfactory results. In addition, some methods perform unified feature alignment along the channel dimension or spatial dimension, which will also damage certain performance. To alleviate these problems, a new feature fusion module is proposed. We split high-level semantic features and low-level detail spatial features in the channel dimension, perform feature resampling operations on the split sub-features respectively, and adaptively perform cross-layer cascade fusion of high- and low-level feature information combined with the information of edge branches.

[0148] Specific as Figure 2 As shown in the figure: The input of the EGRFM module consists of three parts: high-level semantic features (High-levelFeature), low-level detail spatial features (Low-levelFeature), and edge information (EdgeFeature). First, the incoming high-level semantic features and low-level detail spatial features are spliced ​​on the channel, and then the spliced ​​features are upsampled to obtain a 2D offset feature map (2D Offsets). Then the 2D offset feature map is split, and used as a reference to resample the high-level semantic features and low-level detail spatial features. The sampling principle is explained below. Finally, the sampled features are fused, and the edge information is used as a weight guide to fuse the sampled high-level semantic features and low-level detail spatial features to obtain the final output of this module.

[0149] The detailed process description of the EGRFM module is as follows:

[0150] First, we use 1×1 convolution to convolution high-level semantic features. and low-level detail spatial features Perform channel compression and concatenate in the channel dimension to obtain The mathematical formula of the above steps can be expressed as:

[0151] F concat =concat(Up(Conv1(F h)),Conv1(F l ))

[0152] Where concat(·) represents the concatenation operation;

[0153] Up(·) indicates upsampling;

[0154] Conv1 represents a 1×1 convolution operation;

[0155] F l Represents low-level detail spatial features;

[0156] F h Represents high-level semantic features.

[0157] Secondly, learn the 2D offset feature map for the concatenated features Divide its offset map into two groups and It is used to resample its high-level semantic features and low-level semantic features. Sampling function It can be expressed as:

[0158]

[0159] τ (h,w) =max(0,1-|hh′+Δ1|)*max(0,1-|ww′+Δ2|)

[0160] in Represents the sampling features at the point (h+Δ1, w+Δ2);

[0161] H′ represents the height of the feature offset map;

[0162] W′ represents the width of the feature offset map;

[0163] F (h′,w′) represents the feature at point (h′, w′) in the feature offset graph;

[0164] τ (h,w) represents the weight at the point (h, w);

[0165] h and h′ represent the ordinates of the corrected point and the point on the offset feature map, respectively;

[0166] w and w′ represent the horizontal coordinates of the corrected point and the point on the offset feature map, respectively;

[0167] Δ1 is the high-level semantic feature offset map;

[0168] Δ2 is the low-level semantic feature offset map.

[0169] like Figure 2As shown in the figure, during the sampling process, we look for the most favorable feature point (blue point) for the current point (green point) to replace the point at the current position to achieve the purpose of calibration. At the same time, each group of offset maps is further divided into two groups to ensure that the calibration process is more precise and accurate.

[0170] Overall, for the calibrated features, its high-level semantic features F h 'Still contains more semantic information, low-level detail spatial features F l ' contains more spatial geometric details, and directly calibrating them and then fusing them does not improve their performance much. Since the edge branch can better preserve the spatial detail information, we use the edge branch as a guide to adaptively fuse the calibrated semantic features F h 'With the detail feature F l ', the output of EGRFM can be written as:

[0171] σ=1+tanh(V Edge )Sigmoid(V Edge )

[0172]

[0173] in Represents edge features;

[0174] tanh() represents the tanh activation function;

[0175] Sigmoid() represents the sigmoid activation function;

[0176] Represents matrix multiplication;

[0177] Out EGRFM Represents the output of the EGRFM module;

[0178] F l ′ represents the corrected low-level detail spatial features;

[0179] F h ′ represents the corrected high-level semantic features;

[0180] Represents the weight of edge feature generation. As the weight of edge feature generation, σ has rich edge information, so σ has a more positive guiding significance for low-level detail space features.

[0181] The specific process of the edge-guided resampling fusion module: given features and First, F is transformed through a convolutional layer. l The channels are unified as F hThen, bilinear interpolation is used to calculate F h Upsample to F l The size is the same. To improve efficiency, two 1×1 convolutions are used to compress the channels of the normalized features, and the Concat operation is used to connect the compressed features. After that, we input the connected features into the convolution block to learn the offset map and divide it into two groups Used to resample the features of two levels. At the same time, the edge features As feature F h and F l The fusion weight is used to control the fusion of the two levels of features. In summary, the mathematical formula of the EGRFM module can be expressed as:

[0182]

[0183] Where Up means upsampling;

[0184] Conv1 represents the convolution function with a batch normalization layer;

[0185] tanh() represents the tanh activation function;

[0186] F l Represents low-level detail spatial features;

[0187] F h Represents high-level semantic features;

[0188] The convolution module consists of a convolutional layer with a Relu activation function and a convolution kernel size of 1.

[0189] 2.3 Global Average Pyramid Pooling Module (GAPPM)

[0190] Context information can provide rich scene category priors. In order to better build global scene priors to correct unexpected misclassifications, PSPNet introduces a pyramid pooling module (PPM) to connect multi-scale pooling maps before the convolution layer to form local and global context representations. The pyramid pooling module usually uses pooling layers of different sizes and types to extract image features, and then merges the extracted features in the channel dimension to produce a composite feature map with multi-scale information. Some recent studies have proposed modifying the number and size of pooling layers, as well as merging separate convolutional layers after the pooling layer to enhance the feature extraction process. For example, DeepLabv3+ applies depthwise separable convolutions to the ASPP module to enhance context representation. DenseASPP combines the ASPP module with dense connections in DenseNet, which has a larger receptive field and denser sampling points. These methods all aggregate context within a predefined area and contain too many channels at each scale, which may exceed the representation capacity of a lightweight model and is very time-consuming.

[0191] Therefore, we propose a simple global average pooling pyramid module (GAPPM) with the following module structure: Figure 3 As shown. First, the input features are fused using the pyramid pooling module, which has three global average pooling operations with sizes of 1×1, 2×2, and 4×4. Secondly, convolution and upsampling are performed after the output features, and the number of input channels is reduced by convolution operations with a convolution kernel size of 1×1. In addition, a 1×1 convolution is applied to the input features to retain rich semantic information, and then added to each branch of the pyramid pooling module to enrich the pooled features. Finally, the Concat operation is used to merge various scale features, and then input into the BiAttention module together with the previously retained semantic information to generate the final result.

[0192] 3. Experiment

[0193] To verify the effectiveness of the method, we train and test EFCNet on the Cityscapes and CamVid datasets. Below, we first introduce the datasets and the training details of the network. Then, we conduct an ablation study on the Cityscapes dataset to discuss the effect of the proposed module. Finally, we conduct a detailed comparison with the most advanced (SOTA) works on the two benchmarks to illustrate the effectiveness of the proposed method.

[0194] 3.1 Dataset

[0195] 3.1.1 Cityscapes

[0196] Cityscapes is a semantic understanding image dataset for urban street scenes. The dataset contains a variety of stereo video sequences recorded in street scenes from 50 different cities. In addition to 20,000 weakly annotated frames, it also contains 5,000 frames of high-quality pixel-level annotations. Among them, 2,975 high-resolution images are used for training, 500 for verification, and 1,525 for testing. The image resolution is 2048×1024 pixels, and the annotated images have 30 categories, of which only 19 categories are generally used for image semantic segmentation. The Cityscapes dataset has two sets of evaluation standards, fine and coarse. The former provides 5,000 finely annotated images, and the latter provides 5,000 finely annotated images plus 20,000 coarsely annotated images. In order to make a fair comparison with other networks, we only use these 5,000 high-resolution images for training.

[0197] 3.1.2Camvid

[0198] Camvid Cambridge Driving Labeled Video Database is a small-scale dataset for road scene segmentation, containing 701 high-quality pixel-level annotated car driving scene images in video sequences. Of these 701 images, 367 are used for training, 101 for validation, and 233 for testing. The resolution of these images is 960×720 pixels. The annotated images provide 32 categories, 11 of which are used to perform semantic segmentation tasks.

[0199] 3.2 Experimental Details

[0200] 3.2.1 Training Settings

[0201] Based on experience, we choose stochastic gradient descent as the optimizer to train our network, momentum is set to 0.9, and adopt warm-up strategy and poly learning rate scheduler. For the Cityscapes dataset, the batch size during training is set to 6, the maximum number of iterations is set to 420000, the initial learning rate is 0.005, and the weight decay in the optimizer is 1.25e-4. For the CamVid dataset, the batch size is 6, the maximum number of iterations is 40000, the initial learning rate is 0.0025, and the weight decay is 2.5e-5.

[0202] Data augmentation uses random scaling, random cropping, random horizontal flipping, random color jittering, and normalization. The random scale ranges for Cityscapes and Camvid are [0.125, 1.5] and [0.5, 2.5], respectively. The cropping resolution of Cityscapes is 1024×512, and the lightweight backbone network STDC pre-trained on the ImageNet dataset is used as the backbone for training. The cropping resolution on the Camvid dataset is 960×720. We fine-tune the Cityscapes pre-trained model and stop training when the learning rate is less than 5e-4 to avoid overfitting. All experiments were conducted on PaddlePaddle.

[0203] 3.2.2 Inference Settings

[0204] We use mean intersection over union (mIoU) and frames per second (FPS) to measure accuracy and latency, and use floating point operations (FLOPs) and model parameters (Params) to evaluate model complexity. The test environment platform consists of a single GTX 1080Ti GPU, CUDA 11.6, cuDNN 8.0, and the operating system is Ubuntu 20.04. First, the image is resized to the specified size, then the scaled image is input into the network model and the predicted image is generated, and finally the predicted image is resized to the input image size. The cost of these three steps is counted as inference time. For CamVid, the inference model takes the original image as input with a resolution of 960×720. For Cityscapes, the inference model uses the original resolution size and 0.75 times the original resolution size as input for prediction, that is, 1024x2048 and 768x1536.

[0205] 3.3 Ablation Experiment

[0206] In the ablation experiment, we evaluated each module of the EFCNet network. The other settings during the test remain unchanged, and only the modules that require ablation experiments are modified.

[0207] Table 1Cityscapes val 1536*768

[0208]

[0209] Network structure. In order to verify the effectiveness of the combination of various modules, the baseline network uses the STDC backbone pre-trained on the ImageNet dataset as the encoder part of the EFCNet network, adopts SPPM as the context aggregation module, combines channel attention with spatial attention, and uses addition operations to fuse high-level semantic features with low-level detail spatial features as a feature fusion module. Figure 4The visualization effect diagram of the segmentation head, auxiliary segmentation head and edge path is shown. Table 1 shows the relationship between the segmentation accuracy and inference speed of each module. Edge represents the edge branch, which is mainly used to retain detail information. EGRFM represents the proposed feature fusion module, which enhances the fusion of features through resampling and avoids feature misalignment caused by direct fusion. GAPPM is a context aggregation module, which obtains information of different scales through pooling modules with different receptive field sizes to increase the model's ability to represent small objects. E-up is a simple upsampling module, which represents the result of better learning sampling.

[0210] GAPPM. The pyramid pooling module has the ability to expand the model's perception field and perceive global and local information. It extracts feature representations at different scales to enhance the model's perception of objects of different sizes, thereby improving the accuracy and generalization of semantic segmentation. We compared SPPM with the proposed module. As shown in Table 2, GAPPM improves the network performance from 78.68% to 79.2%, which is better than the SPPM module.

[0211] Table 2 Context aggregation module comparison cityscapes 1536*768val

[0212]

[0213] Table 3 Comparison of feature fusion methods in EGRFM module Cityscapes val 1536*768

[0214] Method QUR Params Channel Concat 77.95 12252498 Resample & Channel Concat 78.189 12305248 Edge & Channel Concat 78.574 13138748 Edge&Resample&Channel Concat 78.72 12832784

[0215] EGRFM. The feature fusion module mainly integrates features from different levels. It is usually located in the middle or end of the network structure to merge feature representations from different levels to capture richer semantic information. This helps ensure that the model can understand higher-level semantic information while retaining details, and improves the model's ability to understand and distinguish targets. We attached the EGRFM module to the back of GAPPM and constructed the fusion module in a cascaded manner. Table 3 lists the experimental results of different methods of the fusion module. Channel Concat means directly combining channel attention with spatial attention for feature fusion. Resample&Channel Concat means sampling first and then channel splicing the sampled features. Edge&Channel Concat means fusing the edge with high and low features through channel attention and spatial attention. Edge&Resample&Channel Concat adds the information of the edge branch to the method Resample&Channel Concat, and fuses the high and low semantic information through the weights generated by the edge branch, resulting in better results. It can be seen that the resampling method increases the accuracy by 0.39% compared with the direct channel fusion method, and adding the edge path here significantly improves the performance by 0.531%.

[0216] At the same time, we tested two weighting methods for the weighted fusion strategy of the processed features, as shown in Table 4. For the network, the parameters and Flops are the same for 1+tanh and sigmoid, but 1+tanh achieves better performance than sigmoid in terms of mIoU and FPS (79.25% vs 79.08%).

[0217] Table 4 Comparison of different activation functions in EGRFM module Cityscapes val 1536*768

[0218] Activation QUR Params Flops FPS 1+tanh 79.25 13406993 97852881792 62.3 Sigmoid 79.08 13406993 97852881792 62.11

[0219] 3.4 Experimental comparison

[0220] 3.4.1Cityscapes

[0221] There are usually two ways to evaluate the Cityscapes test dataset. One method is to train the model on the training dataset, test the model accuracy on the validation set, and then generate the results of the test dataset through the trained model and submit it to the server for evaluation. The other method is to train with the training dataset, evaluate the model by scaling the image size of the validation set, and then use the trained model to predict the test dataset and submit the generated results to the server for evaluation. We use the Cityscapes dataset to compare EFCNet with previous advanced real-time semantic segmentation models. For fair comparison, EFCNet is evaluated at two resolutions, namely 768×1536 and 1024×2048. Table 5 lists the model information, input resolution, FLOPs, GPU used for test speed, FPS, and mIoU of various methods.

[0222] Experimental results show that the proposed EFCNet achieves a good trade-off between accuracy and speed. The resolution of EFCNet-100 is 1024×2048, the mIoU of the validation set reaches 80.01%, and the accuracy of the test set reaches 79.22%, achieving good accuracy. Figure 5 This is a visualization of the EFCNet on the Cityscapes dataset. From left to right are the original image, predicted image, and label. Figure 6 Comparison of visualization effects of STDC network, PP-LiteSeg network and the proposed network.

[0223] Table 5 SOTA comparison of Cityscapes dataset

[0224]

[0225] 3.4.2Camvid

[0226] Experiments are conducted on the Camvid dataset to further demonstrate the performance of EFCNet. Similar to previous work, the resolution of both training and inference is kept the same as the original image size, using the backbone EFCNet network pre-trained on ImageNet and the EFCNet model trained on the Cityscapes dataset as pre-training inputs. As shown in Table 6, EFCNet achieves accuracies of 78.7% and 79.68% on the CamVid test set. Compared with other models, EFCNet achieves the highest accuracy. Overall, EFCNet achieves a good trade-off between accuracy and speed on the Camvid dataset.

[0227] Table 6 Camvid test SOTA comparison

[0228]

[0229]

[0230] The present invention focuses on the problem of unclear segmentation edges and misaligned features. It combines the encoder-decoder structure with the multi-path method and proposes a novel segmentation network EFCNet, which solves the problem of unclear edge distinction and misaligned features while maintaining efficient segmentation. Specifically, in order to solve the problem of unclear segmentation edges, an edge branch is introduced to enhance the network's ability to distinguish edges by extracting edge information of objects in the image. At the same time, a feature resampling method is used to calibrate high-level semantic features and low-level detail spatial features to alleviate the problem of misaligned spatial features. The calibrated features are weighted and fused by introducing edge information to maintain a faster reasoning speed. In addition, a simple pyramid pooling module is designed to aggregate global context information and improve the extraction of semantic information. Experiments are conducted on two challenging datasets, Cityscapes and CamVid. The results show that the proposed EFCNet achieves a good compromise between segmentation accuracy and reasoning speed. In future work, it is planned to improve the reasoning speed of the model by optimizing the number of channels, model compression, etc. At the same time, the proposed method will be applied to more scenarios, including remote sensing image processing and medical image processing.

[0231] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A method for calibrating edge-assisted features for real-time semantic segmentation, wherein the method is implemented based on a real-time semantic segmentation network, wherein the real-time semantic segmentation network adopts a dual-branch structure, including a semantic branch and an edge branch; the semantic branch includes an encoder and a decoder; wherein: The edge-assisted feature calibration method comprises the following steps: S1, input the image to the encoder to extract features; the encoder is a five-stage downsampling process, and the size of the feature map in each stage is 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image; the feature map of each stage of the encoder contains semantic features, and the final output of the encoder is a 1 / 32 feature map; The feature maps of 1 / 4, 1 / 8, and 1 / 16 are used as the input of the three stages in the edge branch respectively; and the feature maps of 1 / 8, 1 / 16, and 1 / 32 are used as the low-level detail spatial features of the third EGRFM module, the second EGRFM module, and the first EGRFM module respectively; S2, extract edge information through edge branch, edge branch is a three-stage convolution process, including three convolution blocks, each stage corresponds to a convolution block, and the size of the feature map of each stage is 1 / 8 of the original image; wherein, the input of the first convolution block is the 1 / 4 feature map, which is convolved to obtain a feature map of size 1 / 8; the input of the second convolution block is the 1 / 8 feature map and the 1 / 8 feature map output by the encoder, which is convolved and concat to obtain a feature map of size 1 / 8; the input of the third convolution block is the 1 / 8 feature map and the 1 / 16 feature map output by the encoder, which is convolved and concat to obtain a feature map of size 1 / 8; the 1 / 8 feature map is a feature map containing edge information; S3, the decoder decodes the semantic features and edge information output by the encoder; the decoder is composed of three cascaded EGRFM modules; the data processing process through the EGRFM module is: The input of the EGRFM module consists of three parts: high-level semantic features, low-level detail spatial features, and edge information. At this time, the high-level semantic features of the first EGRFM module are the final output feature map of the encoder, the high-level semantic features of the second EGRFM module are the output feature map of the first EGRFM module, and the high-level semantic features of the third EGRFM module are the output feature map of the second EGRFM module. First, the input high-level semantic features and low-level detail spatial features are spliced ​​on the channel, and then the spliced ​​features are upsampled to obtain a 2D offset feature map; then the 2D offset feature map is split and used as a reference to resample the high-level semantic features and low-level detail spatial features; finally, the input edge information is used as a weight guide to fuse the resampled high-level semantic features and low-level detail spatial features to obtain the final output of the EGRFM module; S4, passes the output of the decoder through the edge segmentation head to obtain the feature map after semantic segmentation.

2. The edge-assisted feature calibration method for real-time semantic segmentation according to claim 1, characterized in that: The data processing of the EGRFM module includes: First, the high-level semantic features and low-level detail spatial features are compressed through 1×1 convolution and concatenated in the channel dimension to obtain F concat ; The mathematical formula of the above steps is expressed as: F concat =concat(Up(Conv1(F h )),Conv1(F l )) Where concat(·) represents the concatenation operation; Up(·) indicates upsampling; Conv1 represents a 1×1 convolution operation; F l Represents low-level detail spatial features; F h Represents high-level semantic features; Secondly, learn the 2D offset feature map Δ for the concatenated features l , the offset map is divided into two groups Δ1 and Δ2, which are used to resample the high-level semantic features and low-level semantic features; the sampling function It is expressed as: t (h,w) =max(0.1-|hh′+Δ1|)*max(0.1-|ww′+Δ2|) in Represents the sampling features at the point (h+Δ1, w+Δ2); H′ represents the height of the feature offset map; W′ represents the width of the feature offset map; F (h′,w′) represents the feature at point (h′, w′) in the feature offset graph; τ (h,w) represents the weight at the point (h, w); h and h′ represent the ordinates of the corrected point and the point on the offset feature map, respectively; w and w′ represent the horizontal coordinates of the corrected point and the point on the offset feature map, respectively; Δ1 is the high-level semantic feature offset map; Δ2 is the low-level semantic feature offset map; During the sampling process, we look for the most favorable feature point for the current point to replace the point at the current position to achieve the purpose of calibration; at the same time, each group of offset maps is further divided into two groups to ensure that the calibration process is more precise and accurate; The edge branch is used as a guide to adaptively fuse the calibrated semantic features F h 'With the detail feature F l ', the output of EGRFM is written as: σ=1+tanh(V Edge )Sigmoid(V Edge ) Where V Edge Represents edge features; tanh() represents the tanh activation function; Sigmoid() represents the sigmoid activation function; Represents matrix multiplication; Out EGRFM Represents the output of the EGRFM module; F′ l represents the corrected low-level detail spatial features; F′ h Represents the corrected high-level semantic features; σ represents the weight of edge feature generation.

3. The edge-assisted feature calibration method for real-time semantic segmentation according to claim 1, characterized in that: There is also a GAPPM module between the encoder and decoder; At this time, the high-level semantic features of the first EGRFM module are the output feature maps of the GAPPM module; the data processing process of the GAPPM module includes: First, the features are globally averaged pooled and channel compressed using Conv. Then, the pooled features are pooled with different kernel sizes to obtain features of different scales. The pooled features are added to the channel compressed features respectively, and then concatenated and convolved, and input into the BiAttention module. The BiAttention module processes the input features and the channel compressed features, fuses and multiplies them, and obtains the final result. The specific processing flow of the BiAttention module is as follows: receive two inputs, namely the features after channel compression and the features after pooling and splicing; directly add the two features and use Conv to perform channel compression to obtain the weight σ; then multiply the weight σ with the input channel compressed features to obtain feature one, multiply the weight 1-σ with the input pooled and spliced ​​features to obtain feature two, and then add feature one and feature two to obtain the final output.

4. The edge-assisted feature calibration method for real-time semantic segmentation according to claim 1, characterized in that: The encoder is a lightweight backbone network STDC.

5. The edge-assisted feature calibration method for real-time semantic segmentation according to claim 1, characterized in that: The loss function during real-time semantic segmentation network training is: Loss=λ0·Loss0+λ1·Loss1+λ2·Loss2+λ3·Loss Edge0 +λ4·Loss Edge1 Among them, λ0, λ1, λ2, λ3, and λ4 are training loss parameters; Loss0, Loss1, and Loss2 are the loss functions of the first, second, and third semantic segmentation heads, respectively; Loss Edge0 is the marginal loss function; Loss Edge1 is the marginal loss regularization function.

6. The edge-assisted feature calibration method for real-time semantic segmentation according to claim 5, characterized in that: Loss0, Loss1, and Loss2 use cross entropy loss, and the formula is as follows: Where L CE represents the cross entropy loss; N represents the number of samples; L i represents the cross entropy loss of each sample; M represents the number of categories; y ic Indicates whether the true category of sample i is equal to c, and the value is 0 or 1; p ic It represents the probability that the predicted sample i belongs to category c.

7. The edge-assisted feature calibration method for real-time semantic segmentation according to claim 5, characterized in that: Loss Edge0 The weighted binary cross entropy loss is used, and the formula is as follows: Where L_{BCE} represents the weighted binary cross entropy loss; N represents the number of samples; α i represents the weight of sample i; y i Indicates the label of sample i, the value is 0 or 1; p i It represents the probability that sample i is the correct sample.

8. The edge-assisted feature calibration method for real-time semantic segmentation according to claim 5, characterized in that: Loss Edge1 The formula is as follows: Where L reg represents marginal loss; {1:s>thrs} indicates whether s is greater than the set threshold thrs. If so, the value is 1, otherwise 0; M represents the number of categories; s represents the output of the edge segmentation head when sample i belongs to class c; thrs is the threshold value; y ic Indicates whether the true category of sample i is equal to c, and the value is 0 or 1; p ic It represents the probability that the predicted sample i belongs to category c.

9. The edge-assisted feature calibration method for real-time semantic segmentation according to claim 1, characterized in that: When training the real-time semantic segmentation network, the real-time semantic segmentation network also includes: A first semantic segmentation head and a second semantic segmentation head are added after the first EGRFM module and the second EGRFM module, respectively; the output feature maps of the first semantic segmentation head and the second semantic segmentation head are upsampled; the loss is calculated for the images after upsampling on the first semantic segmentation head and the second semantic segmentation head and the training set images, respectively, and the loss calculation formulas are loss functions loss0 and loss1, respectively, and the network parameters of the real-time semantic segmentation network are adjusted by minimizing the loss function; An edge segmentation head is added after the edge branch; the output image of the edge segmentation head and the training set image are respectively calculated for loss, and the network parameters of the real-time semantic segmentation network are adjusted; the formulas for loss calculation are loss functions Loss-Edge1 and Loss-Edge0 respectively; the Ground Truth image with the real label is extracted from the edge information through the Canny operator through the Loss-Edge0 loss function and the loss is calculated; the results of the edge segmentation head and the semantic segmentation head are regularized through the Loss-Edge1 loss function and the loss is calculated.

10. The edge-assisted feature calibration method for real-time semantic segmentation according to claim 9, characterized in that: The upsampling process includes: first upsampling through bilinear interpolation, then through a 3x3 convolution and BatchNorm, and finally through the Relu activation function to obtain the output result.

Images