Edge-assisted feature calibration method for real-time semantic segmentation

Abstract

The present invention proposes an edge-assisted feature calibration method for real-time semantic segmentation, which is implemented based on a real-time semantic segmentation network. The real-time semantic segmentation network includes a semantic branch and an edge branch. The edge-assisted feature calibration method includes the following steps: S1, inputting an image into an encoder to extract features; S2, extracting edge information through an edge branch; S3, a decoder decoding the semantic features and edge information output by the encoder; S4, passing the output of the decoder through an edge segmentation head to obtain a feature map after semantic segmentation. The present invention solves the problem of detail information being easily submerged by optimizing the feature fusion strategy. Features of different resolutions are extracted in the encoder part as edge branches to retain spatial information lost due to downsampling; a GAPPM module is introduced at the end of the encoder to enhance the network's contextual feature aggregation capability; and an EGRFM module is proposed to enhance the detail information that is submerged in the network during feature fusion.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and computer vision, and in particular to an edge-assisted feature calibration method for real-time semantic segmentation. Background Art

[0002] Semantic segmentation is a fundamental task in visual scene parsing. Its goal is to assign each pixel in an input image to a specific category label. With the advancement of artificial intelligence and the increasing demand for intelligent capabilities, semantic segmentation has become a fundamental requirement for applications such as autonomous driving, medical image analysis, and remote sensing image processing. With the advancement of technology, deep learning has gradually taken a dominant position in the field of semantic segmentation. Although these models achieve high segmentation accuracy, they require significant computational overhead, making them difficult to apply to applications with high real-time requirements, such as autonomous driving and robotic surgery. This clearly presents an insurmountable gap for large-scale segmentation models.

[0003] Real-time semantic segmentation requires classifying every pixel in an image at real-time speed (FPS ≥ 30), thereby achieving pixel-level semantic segmentation. To meet real-time requirements, researchers have proposed many efficient semantic segmentation models in the past. Many methods attempt to design lightweight networks or efficient decoders to achieve real-time performance. Although these methods reduce computational complexity and speed up inference, lightweight networks still suffer from information loss leading to mismatches in input features. Furthermore, the context modeling method in the decoder lacks robustness for different inputs, resulting in mismatches in contextual information. These issues not only limit the real-time performance of segmentation, but also affect segmentation accuracy, limiting the practical application of semantic segmentation technology.

[0004] With the continuous advancement of real-time semantic segmentation methods, multi-branch network architectures have increasingly become the main approach to address these challenges. PIDNet (a PID controller-inspired network) introduced a new three-branch network architecture, demonstrated the effectiveness of edge information in segmentation, and provided valuable insights to the field. Recent studies have improved the network's ability to capture fine-grained details by using edge detection algorithms as an auxiliary mechanism. Considering the important impact of edge information on network performance, our goal is to integrate edge information into the network with minimal computational cost to improve the overall efficiency of the network. However, these works mainly focus on effectively extracting edge information, while neglecting the importance of its proper integration in the network. Summary of the Invention

[0005] The present invention aims to at least solve the technical problems existing in the prior art, and in particular innovatively proposes an edge-assisted feature calibration method for real-time semantic segmentation.

[0006] In order to achieve the above-mentioned object of the present invention, the present invention provides an edge-assisted feature calibration method for real-time semantic segmentation, wherein the edge-assisted feature calibration method is implemented based on a real-time semantic segmentation network. The real-time semantic segmentation network adopts a dual-branch structure, including a semantic branch and an edge branch; the semantic branch includes an encoder and a decoder; the edge-assisted feature calibration method comprises the following steps:

[0007] S1: Input the image into the encoder to extract features. The encoder is a five-stage downsampling process, and the size of the feature map in each stage is 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image respectively. The feature map of each stage of the encoder contains semantic features, and the final output of the encoder is a 1 / 32 feature map.

[0008] The feature maps of 1 / 4, 1 / 8, and 1 / 16 are used as the input of the three stages in the edge branch respectively; and the feature maps of 1 / 8, 1 / 16, and 1 / 32 are used as the low-level detail spatial features of the third EGRFM module, the second EGRFM module, and the first EGRFM module respectively;

[0009] S2, extract edge information through edge branch, edge branch is a three-stage convolution process, including three convolution blocks, each stage corresponds to a convolution block, and the feature map size of each stage is 1 / 8 of the original image; wherein, the input of the first convolution block is the 1 / 4 feature map, which is convolved to obtain a feature map of size 1 / 8; the input of the second convolution block is the 1 / 8 feature map and the 1 / 8 feature map output by the encoder, which is convolved and concat to obtain a feature map of size 1 / 8; the input of the third convolution block is the 1 / 8 feature map and the 1 / 16 feature map output by the encoder, which is convolved and concat to obtain a feature map of size 1 / 8; the 1 / 8 feature map is a feature map containing edge information;

[0010] S3, the decoder decodes the semantic features and edge information output by the encoder; the decoder is composed of three cascaded EGRFM modules (edge-guided resampling and fusion modules); the data processing process through the EGRFM module is:

[0011] The input of the EGRFM module consists of three parts: high-level semantic features, low-level detail spatial features, and edge information. At this time, the high-level semantic features of the first EGRFM module are the final output feature map of the encoder, the high-level semantic features of the second EGRFM module are the output feature map of the second EGRFM module, and the high-level semantic features of the third EGRFM module are the output feature map of the second EGRFM module.

[0012] First, the input high-level semantic features and low-level detail spatial features are spliced ​​on the channel. Then, the spliced ​​features are upsampled to obtain a 2D offset feature map. Then, the 2D offset feature map is split and used as a reference to resample the high-level semantic features and low-level detail spatial features. Finally, the input edge information is used as a weight guide to fuse the resampled high-level semantic features and low-level detail spatial features to obtain the final output of the EGRFM module.

[0013] S4, passes the output of the decoder through the edge segmentation head to obtain the feature map after semantic segmentation.

[0014] This method enhances the network's ability to distinguish edges by extracting edge information from objects in the image. It also employs feature resampling to calibrate high-level semantic features with low-level detail spatial features to alleviate spatial feature misalignment. By incorporating edge information, the calibrated features are weighted and fused to maintain a high inference speed.

[0015] Furthermore, the data processing process of the EGRFM module includes:

[0016] First, the high-level semantic features and low-level detail spatial features are channel-compressed by 1×1 convolution and concatenated in the channel dimension to obtain F concat ; The mathematical formula of the above steps is expressed as:

[0017] F concat =concat(Up(Conv1(F h )),Conv1(F l ))

[0018] Where concat(·) represents the concatenation operation;

[0019] Up(·) indicates upsampling;

[0020] Conv1 represents a 1×1 convolution operation;

[0021] F l Represent low-level detail spatial features;

[0022] F h Represents high-level semantic features;

[0023] Secondly, learn the 2D offset feature map Δ for the concatenated features l , the offset map is divided into two groups Δ1 and Δ2, which are used to resample the high-level semantic features and low-level semantic features; the sampling function Expressed as:

[0024]

[0025] τ (h,w) =max(0,1-|hh′+Δ1|)*max(0,1-|ww′+Δ2|)

[0026] in Represents the sampling features at the point (h+Δ1, w+Δ2);

[0027] H′ represents the height of the feature offset map;

[0028] W′ represents the width of the feature offset map;

[0029] F (h′,w′) Represents the feature at point (h′, w′) in the feature offset graph;

[0030] τ (h,w) Represents the weight at point (h, w);

[0031] h and h′ represent the ordinates of the corrected point and the point on the offset feature map, respectively;

[0032] w and w′ represent the horizontal coordinates of the corrected point and the point on the offset feature map, respectively;

[0033] Δ1 is the high-level semantic feature offset map;

[0034] Δ2 is the low-level semantic feature offset map;

[0035] During the sampling process, we search for the most favorable feature point for the current point to replace the point at the current position to achieve the purpose of calibration; at the same time, each set of offset maps is further divided into two groups to ensure a more precise and accurate calibration process;

[0036] Overall, for the calibrated features, the high-level semantic features F h 'Still contains more semantic information, low-level detail spatial features F l ' contains more spatial geometric details, and directly calibrating them and then fusing them does not improve their performance much. Since the edge branch can better preserve spatial detail information, the edge branch is used as a guide to adaptively fuse the calibrated semantic features F h 'With the detail feature F l ', the output of EGRFM is written as:

[0037] σ=1+tanh(V Edge )Sigmoid(V Edge )

[0038]

[0039] in Represents edge features;

[0040] tanh() represents the tanh activation function;

[0041] Sigmoid() represents the sigmoid activation function;

[0042] Represents matrix multiplication;

[0043] Out EGRFM Represents the output of the EGRFM module;

[0044] F l ′ represents the corrected low-level detail spatial features;

[0045] F h ′ represents the corrected high-level semantic features;

[0046] Represents the weight of edge feature generation.

[0047] Furthermore, there is a GAPPM module (global average pyramid pooling module) between the encoder and decoder. In this case, the high-level semantic features of the first EGRFM module are the output feature maps of the GAPPM module. The data processing process of the GAPPM module includes:

[0048] First, the features are globally averaged pooled and channel compressed using Conv. Then, the pooled features are pooled with different kernel sizes to obtain features of different scales. The pooled features are added to the channel compressed features, and then concatenated and convolved before being input to the BiAttention module. The BiAttention module processes the input features and the channel compressed features, fuses and multiplies them, and obtains the final result.

[0049] The specific processing flow of the BiAttention module is as follows: receive two inputs, namely the channel-compressed feature and the pooled-joined feature; directly add the two features and use Conv to perform channel compression to obtain the weight σ; then multiply the weight σ with the input channel-compressed feature to obtain feature 1, multiply the weight 1-σ with the input pooled-joined feature to obtain feature 2, and then add feature 1 and feature 2 to obtain the final output.

[0050] The global average pyramid pooling module is designed to aggregate global context information, improving the extraction of semantic information.

[0051] Furthermore, the encoder is a lightweight backbone network STDC.

[0052] Furthermore, the loss function in the real-time semantic segmentation network training process is:

[0053] Loss=λ0·Loss0+λ1·Loss1+λ2·Loss2+λ3·Loss Edge0 +λ4·Loss Edge1

[0054] Among them, λ0, λ1, λ2, λ3, and λ4 are training loss parameters;

[0055] Loss0, Loss1, and Loss2 are the loss functions of the first, second, and third semantic segmentation heads, respectively;

[0056] Loss Edge0 is the marginal loss function;

[0057] Loss Edge1 is the marginal loss regularization function.

[0058] Furthermore, Loss0, Loss1, and Loss2 use cross entropy loss, and the formula is as follows:

[0059]

[0060] Among them L CE represents the cross entropy loss;

[0061] N represents the number of samples;

[0062] L i represents the cross entropy loss of each sample;

[0063] M represents the number of categories;

[0064] y ic Indicates whether the true category of sample i is equal to c, and the value is 0 or 1;

[0065] p ic It represents the probability that the predicted sample i belongs to category c.

[0066] Furthermore, Loss Edge0 The weighted binary cross entropy loss is adopted, and the formula is as follows:

[0067]

[0068] Where L_{BCE} represents the weighted binary cross entropy loss;

[0069] N represents the number of samples;

[0070] α i represents the weight of sample i;

[0071] y iIndicates the label of sample i, which takes a value of 0 or 1;

[0072] p i It represents the probability that sample i is the correct sample.

[0073] Furthermore, Loss Edge1 The formula is as follows:

[0074]

[0075] Among them L reg represents edge loss;

[0076] {1:s>thrs} indicates whether s is greater than the set threshold thrs. If so, the value is 1, otherwise 0;

[0077] M represents the number of categories;

[0078] s represents the output of the edge segmentation head when sample i belongs to class c;

[0079] thrs is the threshold value;

[0080] y ic Indicates whether the true category of sample i is equal to c, and the value is 0 or 1;

[0081] p ic It represents the probability that the predicted sample i belongs to category c.

[0082] Furthermore, when the real-time semantic segmentation network is trained, the real-time semantic segmentation network also includes:

[0083] The first and second semantic segmentation heads are added after the first and second EGRFM modules, respectively. The output feature maps of the first and second semantic segmentation heads are upsampled to the same size as the training set images. The first and second semantic segmentation heads are used to accelerate the convergence of the network model. Losses are calculated for the images sampled by the first and second semantic segmentation heads and the training set images, respectively. The loss calculation formulas are loss functions loss0 and loss1, respectively. The network parameters of the real-time semantic segmentation network are adjusted by minimizing the loss functions.

[0084] Add an edge segmentation head after the edge branch; calculate the loss of the output image of the edge segmentation head and the training set image respectively, and adjust the network parameters of the real-time semantic segmentation network; the loss calculation formulas are loss functions Loss-Edge1 and Loss-Edge0 respectively; use the Loss-Edge0 loss function to extract edge information from the ground truth image with real labels through the Canny operator and calculate the loss; use the Loss-Edge1 loss function to regularize the results of the edge segmentation head and semantic segmentation head and calculate the loss;

[0085] Furthermore, the upsampling process includes: first upsampling through bilinear interpolation, then through a 3x3 convolution and BatchNorm, and finally through the ReLU activation function to obtain the output result.

[0086] The segmentation results are enhanced by upsampling after each semantic segmentation head.

[0087] In summary, thanks to the aforementioned technical solutions, the edge-guided resampling fusion module (EGRFM) designed in this paper utilizes edge information to fuse resampled and calibrated features to enhance feature representation. This addresses the feature misalignment problem that occurs when fusing high-resolution and low-resolution features, thereby enhancing the integration of edge information in deep learning models. The EGRFM predicts an offset map from high- and low-resolution features and then resamples features based on this offset map. Using edge information as a guide, it increases the weight of detail features, facilitating the fusion of high-level semantic features with low-level details. This helps the network more effectively segment object edges. Edge information is extracted from different stages of the main branch and refined using the Canny operator to enhance edge features. Furthermore, a global average pyramid pooling module (GAPPM) is designed to extract contextual information through multi-level pooling, improving segmentation accuracy at a minimal cost in inference time. This effectively captures multi-scale context and further enhances the model's ability to handle complex tasks. Based on these modules, a real-time semantic segmentation network (EFCNet) is proposed, achieving a good trade-off between accuracy and speed.

[0088] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0090] Figure 1 It is a schematic diagram of the network structure of EFCNet of the present invention.

[0091] Figure 2 Schematic diagram of the structure of the EGRFM module of the present invention.

[0092] Figure 3 It is a structural diagram of the GAPPM module of the present invention.

[0093] Figure 4 It is the visualization of the intermediate process diagram of the present invention. Figure 4 (a), (b), (c), and (d) are process diagrams of segmentation heads 1, 2, 3, and edge segmentation heads, respectively, where segmentation heads 2 and 3 are auxiliary segmentation heads.

[0094] Figure 5 This is the network prediction effect diagram of the present invention. Figure 5 (a), (b), and (c) are the original image, the segmentation prediction image, and the label image corresponding to the original image, respectively.

[0095] Figure 6 This is a comparison diagram of the network visualization of the present invention on the Cityscapes validation set. Figure 6 (a), (b), (c), (d), and (e) are the original image, the real label image, the prediction effect image of the ppLiteSeg network, the prediction effect image of the STDC-Seg network, and the prediction effect image of the EFCNet network, respectively. DETAILED DESCRIPTION

[0096] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0097] 1. Related Work

[0098] In recent years, many methods have improved the performance of semantic segmentation in challenging scenarios, including high-precision semantic segmentation and real-time semantic segmentation.

[0099] 1.1 High-precision semantic segmentation

[0100] Early semantic segmentation methods were primarily based on image segmentation and region-based techniques. Graph Cuts-based methods have achieved some success in segmentation accuracy, but they suffer from high computational complexity, require long runtimes, are inefficient for large-scale image processing, and are unsuitable for real-time scenarios. Region-based methods rely on handcrafted features and heuristic rules, limiting their generalization capabilities in complex scenarios. In recent years, with the development of deep learning, semantic segmentation methods based on convolutional neural networks (CNNs) have made significant progress.

[0101] The introduction of Fully Convolutional Networks (FCN) has made end-to-end semantic segmentation possible, and U-shaped networks and encoder-decoder networks have also been proposed. Among them, spatial details are easily ignored during the downsampling process of the encoder-decoder network. To alleviate this problem, dilated convolution is proposed to expand the field of view without reducing spatial resolution. The DeepLab series of models increases the receptive field by applying convolution kernels with different dilation rates in the network, thereby improving the performance of semantic segmentation tasks. PSPNet introduces a pyramid pooling module (PPM) to parse multi-scale context information. HRNet (see the paper Wang J, Sun K, Cheng T, et al. Deep high-resolution representation learning for visual recognition [J]. IEEE transactions on pattern analysis and machine intelligence, 2020, 43 (10): 3349-3364.) uses multi-path and bilateral connections to learn and fuse representations of different scales. Some works use self-attention mechanisms to dynamically capture the global context information of each pixel. However, under the same resource conditions, they require more computing time and are therefore not applicable in real-time scenarios. ANNN (see the paper Zhu Z, Xu M, Bai S, et al. Asymmetric non-local neural networks for semantic segmentation [C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2019: 593-602.) uses a pyramid sampling module to reduce the burden of self-attention calculation while maintaining sensitivity to global context. However, ANNN still lacks in multi-scale context extraction. SFNet and AlignSeg propose to align multi-level features by learning 2D transformation offsets. However, they ignore the representation differences between features at different levels and sub-features at the same level in this process, which weakens the effectiveness of information propagation.

[0102] 1.2 Real-time Semantic Segmentation

[0103] Real-time semantic segmentation aims to produce high-quality predictions while maintaining high efficiency. Numerous different approaches have been proposed to address this challenge, including lightweight encoder-decoder architectures, multi-branch architectures, network compression, and input size scaling. Below, we will detail the classic approaches to each strategy.

[0104] 1.2.1 Lightweight Encoder-Decoder Structure

[0105] DFANet introduces a lightweight backbone to the Xception architecture based on depthwise separable convolutions and reduces the input size for faster inference. ShuffleSeg adopts ShuffleNet, which combines channel shuffling and group convolution as its backbone to reduce computational cost. However, these networks still adopt the form of encoder-decoder architecture, which requires the information flow to pass through the deep encoder and then back to the decoder, which introduces too much latency. In addition, since the optimization of depthwise separable convolutions on GPUs is still immature, traditional convolutions are faster while having more FLOPs and parameters. Therefore, we seek more efficient models to avoid convolution decomposition and encoder-decoder architectures.

[0106] 1.2.2 Multi-branch structure

[0107] Effectively fusing global and local information is a key challenge in achieving accurate semantic segmentation. Encoder-decoder-based methods typically address this issue by progressively downsampling feature maps, extracting increasingly higher-level semantic information at each scale. The primary challenge with this approach is preserving the high-resolution details extracted early in the network. Therefore, multi-branch architectures have been proposed to address this issue by independently extracting features at different scales. For example, the BiSeNet family utilizes a two-branch structure to encode spatial and semantic information respectively. DMA-Net recursively fuses features from high- and low-order branches to produce more robust features. SwiftNet utilizes lateral connections to fuse features from branches at different resolutions to handle multi-scale issues. STDC-Seg proposes a backbone network specifically designed for semantic segmentation, replacing the spatial path in BiSeNet with a detail aggregation module, improving efficiency without compromising performance. DDRNet proposes a novel bilateral network with deep dual-resolution branches and multiple bilateral fusions, which results in minimal increase in inference time when executed on low-resolution feature maps. PIDNet combines deep convolution with a PID controller, proposing a three-branch network based on the PID controller architecture. DMRNet modifies the ResNet network into a multi-resolution architecture, enabling it to fully extract image information at different resolutions in the downsampling stage.

[0108] 2. EFCNet Architecture

[0109] 2.1 Network Architecture

[0110] The EFCNet network adopts a dual-branch structure and selects the lightweight backbone network STDC as the encoder (see the paper FanM, Lai S, Huang J, et al. Rethinking bisenet for real-time semantic segmentation [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2021: 9716-9725 for details) to extract multi-layer features. Specifically, the proposed EFCNet has two versions, EFCNet-S and EFCNet-B. When STDC1 is the encoder, it is EFCNet-S; when STDC2 is the encoder, it is STDC2. In order to enhance the network's ability to segment the edges of objects, a feature map with a resolution of 1 / 8 of the input image resolution is created through the trunk branch as an edge branch. Afterwards, a global average pyramid pooling module (GAPPM) is designed, which uses pooling kernels of different sizes to extract contextual information of different scales, and fuses multi-scale global contextual information to enrich feature representation. In addition, an edge-guided resampling fusion module (EGRFM) was constructed, which resamples and calibrates the contextual information and spatial geometric information, and uses the edge branch as a guide to fuse the contextual information and spatial geometric information, thereby enhancing the network's ability to distinguish object edges and small targets.

[0111] The overall structure of the EFCNet network is as follows: Figure 1 As shown in the figure, an image is first passed in, and the encoder is the semantic branch. Encoding is performed through the encoder, which is a downsampling process. The encoder is divided into five stages, with the feature map size of each stage being 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image, respectively. Features are extracted from the three middle stages to form our edge branches, namely the 1 / 4, 1 / 8, and 1 / 16 portions. The features of these portions are directly fused after convolution and used for loss calculation and as input to the EGRFM module. The EGRFM module inputs: the input of the previous module, information about the edge branch, and features from the previous encoder stages. A semantic segmentation head is added after each EGRFM module. The network's module parameters are adjusted by calculating the output of the edge segmentation head and the ground truth loss. A global average pyramid pooling module (GAPPM) is introduced at the end of the semantic branch to extract spatial features of different scales, improving the network's ability to acquire global information.

[0112] The output of the edge branch is the input of the E-Head. The loss is calculated for the output of the E-Head and the image processed by Canny (the Canny algorithm is used to extract edge features of images with real labels). The loss formula is Loss-Edge0, and then the gradient is backpropagated to update the convolution module parameters of the edge branch.

[0113] The results of the edge segmentation head and semantic segmentation head are regularized and loss is calculated using the Loss-Edge1 loss function. The segmentation head corresponding to Loss2 is the final output segmentation head. The segmentation heads corresponding to Loss0 and Loss1 are used to accelerate network model convergence. E-up is applied after each semantic segmentation head to enhance the segmented results. First, upsampling is performed using bilinear interpolation, followed by a 3x3 convolution and BatchNorm, and finally the output is obtained using the Relu activation function.

[0114] BiSeNetV2 proposed an enhanced training strategy, which is to add an additional segmentation head at the end of each stage of the semantic branch. To achieve better results, SFNet adopted a similar strategy called cascaded deep supervised learning. Inspired by BiseNetV2 and SFNet, a semantic segmentation head is placed after the first EGRFM and the second EGRFM modules to generate additional semantic losses Loss0 and Loss1 to better optimize the entire network. The semantic segmentation head placed after the last EGRFM module is the final output of the network. Its loss Loss2 is CE-Loss as are Loss0 and Loss1. The specific formula can be expressed as:

[0115]

[0116] Among them L CE represents the cross entropy loss;

[0117] N represents the number of samples;

[0118] L i represents the cross entropy loss of each sample;

[0119] M represents the number of categories;

[0120] y ic Indicates whether the true category of sample i is equal to c, and the value is 0 or 1;

[0121] p ic It represents the probability that the predicted sample i belongs to category c.

[0122] For the edge branch, the weighted binary cross entropy loss Loss-Edge0 is used to solve the sample imbalance problem in edge detection, because coarse edges are more suitable for highlighting edge areas and enhancing the features of small targets. The specific formula of Loss-Edge0 can be expressed as:

[0123]

[0124] Where L_{BCE} represents the weighted binary cross entropy loss;

[0125] N represents the number of samples;

[0126] α i represents the weight of sample i.

[0127] y i Indicates the label of sample i, which takes a value of 0 or 1;

[0128] p i Indicates the probability that sample i is the correct sample;

[0129] Inspired by Gated-SCNN, we use Loss-Edge1 to regularize the edge loss and use the output of the edge segmentation head to coordinate semantic segmentation and edge detection to ensure the consistency of the edge branch and the semantic branch. The calculation formula of Loss-Edge1 can be written as:

[0130]

[0131] Among them L reg Indicates edge loss regularization processing;

[0132] {1:s>thrs}(y ic log(p ic )) means {1:s>thrs} and (y ic log(p ic ))Multiply;

[0133] {1:s>thrs} indicates whether s is greater than the set threshold thrs. If so, the value is 1, otherwise 0;

[0134] M represents the number of categories;

[0135] s represents the output of the edge segmentation head when sample i belongs to class c;

[0136] thrs is the threshold value;

[0137] y ic Indicates whether the true category of sample i is equal to c, and the value is 0 or 1;

[0138] p icIt represents the probability that the predicted sample i belongs to category c.

[0139] Therefore, the final loss of EFCNet is:

[0140] Loss=λ0·Loss0+λ1·Loss1+λ2·Loss2+λ3·Loss Edge0 +λ4·Loss Edge1

[0141] Among them, Loss0, Loss1, and Loss2 are the loss functions of the first semantic segmentation head, the second semantic segmentation head, and the third semantic segmentation head respectively;

[0142] Loss Edge0 is the marginal loss function;

[0143] Loss Edge1 is the marginal loss regularization function;

[0144] According to experience, the training loss parameters of EFCNet are set to λ0 = 0.4, λ1 = 0.4, λ2 = 1, λ3 = 20, λ4 = 1, and thrs = 0.8.

[0145] 2.2 Edge-Guided Resampling Fusion Module (EGRFM)

[0146] In order to compensate for the loss of spatial details caused by gradual downsampling, the method adopts cross-layer feature fusion and multi-stage feature fusion to enhance high-level semantic features and low-level detail spatial features. and high-level semantic features First, the low-level detail spatial features F are upsampled by bilinear interpolation. l Sampling to high-level semantic features F h The same size, then the high-level semantic features F h and the low-level detail spatial features F after sampling l 'Direct fusion by splicing, addition or cascading.

[0147] It is well known that although high-level features contain rich semantic information, they lack spatial detail, especially at edges. While low-level features retain spatial detail well, they only retain a portion of the semantic information and fail to capture high-level semantics well. To address this issue, many semantic segmentation methods have adopted a cross-layer feature fusion strategy, combining high-level semantic features with low-level spatial features to leverage their strengths. However, when fusing low-level and high-level semantic information, some of the semantic information contained in the low-level spatial features can obscure the high-level semantic features. Furthermore, due to inaccurate spatial position restoration during upsampling and feature redundancy in high-level semantic information, direct fusion can over-erode object edges due to surrounding pixels, causing small objects to be easily submerged by larger ones, thus achieving unsatisfactory results. Furthermore, some methods perform uniform feature alignment along the channel or spatial dimensions, which can also compromise performance. To alleviate these issues, a new feature fusion module is proposed. High-level semantic features and low-level spatial features are split along the channel dimension, feature resampling is performed on each of the split sub-features, and information from edge branches is combined to adaptively fuse high- and low-level feature information across layers in a cascaded fashion.

[0148] Specific as Figure 2 As shown in the figure: The input of the EGRFM module consists of three parts: high-level semantic features (High-level Feature), low-level detail spatial features (Low-level Feature), and edge information (Edge Feature). First, the incoming high-level semantic features and low-level detail spatial features are spliced ​​on the channel. Then, the spliced ​​features are upsampled to obtain 2D offset feature maps (2D Offsets). The 2D offset feature maps are then split and used as a reference to resample the high-level semantic features and low-level detail spatial features. The sampling principle is explained below. Finally, the sampled features are fused, using edge information as a weight guide to fuse the sampled high-level semantic features and low-level detail spatial features to obtain the final output of this module.

[0149] The detailed process description of the EGRFM module is as follows:

[0150] First, we use 1×1 convolution to transform high-level semantic features. and low-level detail spatial features Perform channel compression and splice in the channel dimension to obtain The mathematical formula of the above steps can be expressed as:

[0151] F concat =concat(Up(Conv1(F h)),Conv1(F l ))

[0152] Where concat(·) represents the concatenation operation;

[0153] Up(·) indicates upsampling;

[0154] Conv1 represents a 1×1 convolution operation;

[0155] F l Represent low-level detail spatial features;

[0156] F h Represents high-level semantic features.

[0157] Secondly, learn the 2D offset feature map of the spliced ​​features Divide its offset map into two groups and It is used to resample its high-level semantic features and low-level semantic features. Sampling function It can be expressed as:

[0158]

[0159] τ (h,w) =max(0,1-|hh′+Δ1|)*max(0,1-|ww′+Δ2|)

[0160] in Represents the sampling features at the point (h+Δ1, w+Δ2);

[0161] H′ represents the height of the feature offset map;

[0162] W′ represents the width of the feature offset map;

[0163] F (h′,w′) Represents the feature at point (h′, w′) in the feature offset graph;

[0164] τ (h,w) Represents the weight at point (h, w);

[0165] h and h′ represent the ordinates of the corrected point and the point on the offset feature map, respectively;

[0166] w and w′ represent the horizontal coordinates of the corrected point and the point on the offset feature map, respectively;

[0167] Δ1 is the high-level semantic feature offset map;

[0168] Δ2 is the low-level semantic feature offset map.

[0169] like Figure 2As shown in the figure, during the sampling process, we search for the most favorable feature point (blue point) for the current point (green point) to replace the point at the current position to achieve the purpose of calibration. At the same time, each set of offset maps is further divided into two groups to ensure a more precise and accurate calibration process.

[0170] Overall, for the calibrated features, the high-level semantic features F h 'Still contains more semantic information, low-level detail spatial features F l ' contains more spatial geometric details, and directly calibrating them and then fusing them does not improve their performance much. Since the edge branch can better preserve spatial detail information, we use the edge branch as a guide to adaptively fuse the calibrated semantic features F h 'With the detail feature F l ', the output of EGRFM can be written as:

[0171] σ=1+tanh(V Edge )Sigmoid(V Edge )

[0172]

[0173] in Represents edge features;

[0174] tanh() represents the tanh activation function;

[0175] Sigmoid() represents the sigmoid activation function;

[0176] Represents matrix multiplication;

[0177] Out EGRFM Represents the output of the EGRFM module;

[0178] F l ′ represents the corrected low-level detail spatial features;

[0179] F h ′ represents the corrected high-level semantic features;

[0180] Represents the weight of edge feature generation. As the weight of edge feature generation, σ has rich edge information, so σ has a more positive guiding significance for low-level detail spatial features.

[0181] The specific process of the edge-guided resampling fusion module: given features and First, F is transformed into l The channels are unified as F hThen, bilinear interpolation is used to calculate F h Upsample to F l The size is the same. To improve efficiency, two 1×1 convolutions are used to compress the normalized features and the Concat operation is used to connect the compressed features. After that, we input the connected features into the convolution block to learn the offset map. and divide them into two groups Used to resample the features of two levels. At the same time, the edge features As feature F h and F l The fusion weight is used to control the fusion of the two levels of features. In summary, the mathematical formula of the EGRFM module can be expressed as:

[0182]

[0183] Where Up means upsampling;

[0184] Conv1 represents the convolution function with batch normalization layer;

[0185] tanh() represents the tanh activation function;

[0186] F l Represent low-level detail spatial features;

[0187] F h Represents high-level semantic features;

[0188] The convolution module consists of a Relu activation function and a convolution layer with a convolution kernel size of 1.

[0189] 2.3 Global Average Pyramid Pooling Module (GAPPM)

[0190] Contextual information can provide rich scene category priors. In order to better construct global scene priors to correct unexpected misclassifications, PSPNet introduces a pyramid pooling module (PPM) to connect multi-scale pooling maps before the convolutional layer to form local and global context representations. The pyramid pooling module usually uses pooling layers of different sizes and types to extract image features, and then merges the extracted features in the channel dimension to produce a composite feature map with multi-scale information. Some recent studies have proposed modifying the number and size of pooling layers, as well as merging separate convolutional layers after the pooling layer to enhance the feature extraction process. For example, DeepLabv3+ applies depthwise separable convolution to the ASPP module to enhance context representation. DenseASPP combines the ASPP module with

[0191] Combined with the dense connections in DenseNet, this results in a larger receptive field and denser sampling points. These methods aggregate context within a predefined area and contain too many channels at each scale, which may exceed the representation capacity of a lightweight model and is very time-consuming.

[0192] Therefore, we propose a simple global average pooling pyramid module (GAPPM) with the following module structure: Figure 3 As shown in the figure. First, the input features are fused using the pyramid pooling module, which has three global average pooling operations with sizes of 1×1, 2×2, and 4×4. Secondly, convolution and upsampling are performed after the output features, and the number of input channels is reduced by convolution operations with a convolution kernel size of 1×1. In addition, a 1×1 convolution is applied to the input features to retain rich semantic information, and then added to each branch of the pyramid pooling module to enrich the pooled features. Finally, the Concat operation is used to merge features of various scales, and then input into the BiAttention module together with the previously retained semantic information to generate the final result.

[0193] 3. Experiment

[0194] To validate the effectiveness of our approach, we trained and tested EFCNet on the Cityscapes and CamVid datasets. Below, we first introduce the datasets and network training details. We then conduct an ablation study on the Cityscapes dataset to discuss the effectiveness of our proposed module. Finally, we provide a detailed comparison with state-of-the-art (SOTA) works on both benchmarks to demonstrate the effectiveness of our proposed approach.

[0195] 3.1 Dataset

[0196] 3.1.1 Cityscapes

[0197] Cityscapes is a dataset of images for semantic understanding of urban street scenes. It contains a variety of stereo video sequences recorded from street scenes in 50 different cities. In addition to 20,000 weakly annotated frames, it also includes 5,000 frames with high-quality pixel-level annotations. Of these, 2,975 high-resolution images are used for training, 500 for validation, and 1,525 for testing. The image resolution is 2048 × 1024 pixels, and the annotated images fall into 30 categories, of which only 19 are typically used for semantic segmentation. The Cityscapes dataset has two evaluation criteria: fine and coarse. The former provides 5,000 finely annotated images, while the latter provides 5,000 finely annotated images plus 20,000 coarsely annotated images. To ensure fair comparison with other networks, we only use these 5,000 high-resolution images for training.

[0198] 3.1.2Camvid

[0199] The Camvid Cambridge Driving Labeled Video Database is a small-scale dataset for road scene segmentation. It contains 701 high-quality, pixel-level annotated images of cars driving in video sequences. Of these 701 images, 367 are used for training, 101 for validation, and 233 for testing. The images have a resolution of 960 × 720 pixels. The annotated images are classified into 32 categories, 11 of which are used for semantic segmentation.

[0200] 3.2 Experimental details

[0201] 3.2.1 Training Settings

[0202] Based on experience, we chose stochastic gradient descent as the optimizer to train our network, with momentum set to 0.9, a warm-up strategy, and a poly learning rate scheduler. For the Cityscapes dataset, we trained with a batch size of 6, a maximum number of iterations of 420,000, an initial learning rate of 0.005, and a weight decay of 1.25e-4 in the optimizer. For the CamVid dataset, we used a batch size of 6, a maximum number of iterations of 40,000, an initial learning rate of 0.0025, and a weight decay of 2.5e-5.

[0203] Data augmentation uses random scaling, random cropping, random horizontal flipping, random color jittering, and normalization. The random scale ranges for Cityscapes and Camvid are [0.125, 1.5] and [0.5, 2.5], respectively. The crop resolution for Cityscapes is 1024×512, and training is performed using the lightweight backbone network STDC pre-trained on the ImageNet dataset. The crop resolution for the Camvid dataset is 960×720. We fine-tune the Cityscapes pre-trained model, stopping training when the learning rate is less than 5e-4 to avoid overfitting. All experiments were conducted on PaddlePaddle.

[0204] 3.2.2 Inference Settings

[0205] We use mean intersection over Union (mIoU) and frames per second (FPS) to measure accuracy and latency, and use floating-point operations (FLOPs) and model parameters (Params) to evaluate model complexity. The test environment platform consists of a single GTX 1080Ti GPU, CUDA11.6, cuDNN 8.0, and the operating system is Ubuntu 20.04. First, resize the image to the specified size, then input the scaled image into the network model and generate a predicted image, and finally resize the predicted image to the input image size. The cost of these three steps is counted as inference time. For

[0206] For CamVid, the inference model takes the original image as input with a resolution of 960 × 720. For Cityscapes, the inference model uses the original resolution size and 0.75 times the original resolution size as input for prediction, i.e., 1024x2048 and 768x1536.

[0207] 3.3 Ablation Experiment

[0208] In the ablation experiment, we evaluated each module of the EFCNet network. Other settings during the test remain unchanged, and only the modules that require ablation experiments are modified.

[0209] Table 1cityscapes val 1536*768

[0210]

[0211] To verify the effectiveness of the combination of various modules, the baseline network uses the STDC backbone pre-trained on the ImageNet dataset as the encoder part of the EFCNet network, adopts SPPM as the context aggregation module, combines channel attention with spatial attention, and uses addition operations to fuse high-level semantic features with low-level detailed spatial features as the feature fusion module. Figure 4 The visualization of the segmentation head, auxiliary segmentation head, and edge path is shown. Table 1 shows the relationship between the segmentation accuracy and inference speed of each module. Edge represents the edge branch, which is mainly used to retain detail information. EGRFM represents the proposed feature fusion module, which enhances feature fusion through resampling and avoids feature misalignment caused by direct fusion. GAPPM is a context aggregation module that obtains information of different scales through pooling modules with different receptive field sizes, thereby increasing the model's ability to represent small objects. E-up is a simple upsampling module that represents the result of better learning sampling.

[0212] GAPPM. The pyramid pooling module expands the model's perceptual domain and enables the perception of both global and local information. It extracts feature representations at different scales to enhance the model's perception of objects of varying sizes, thereby improving semantic segmentation accuracy and generalization. We compared SPPM with the proposed module. As shown in Table 2, GAPPM improves network performance from 78.68% to 79.2%, achieving superior results compared to the SPPM module.

[0213] Table 2 Context aggregation module comparison cityscapes 1536*768val

[0214]

[0215] Table 3 Comparison of feature fusion methods in EGRFM module cityscapes val 1536*768

[0216] Method mIoU Params Channel Concat 77.95 12252498 Resample & Channel Concat 78.189 12305248 Edge & Channel Concat 78.574 13138748 Edge&Resample&Channel Concat 78.72 12832784

[0217] EGRFM. The feature fusion module primarily integrates features from different layers and is typically located in the middle or end of the network structure. It is used to merge feature representations from different layers to capture richer semantic information. This helps ensure that the model preserves details while also understanding higher-level semantic information, improving the model's ability to understand and distinguish objects. We append the EGRFM module to the GAPPM module in a cascaded manner to construct the fusion module. Table 3 lists the experimental results of different fusion module methods. Channel Concat directly combines channel attention and spatial attention for feature fusion. Resample & Channel Concat first performs sampling and then concatenates the sampled features. Edge & Channel Concat fuses edges and high- and low-level features together using channel attention and spatial attention. Edge & Resample & Channel Concat builds on the Resample & Channel Concat method by adding information from the edge branch. The weights generated by the edge branch are used to fuse high- and low-level semantic information, achieving better results. As can be seen, the resampling method improves accuracy by 0.39% compared to the direct channel fusion method, and the addition of the edge path significantly improves performance by 0.531%.

[0218] We also tested two weighted fusion strategies for the processed features, as shown in Table 4. For the networks using 1+tanh and sigmoid, the parameters and FLOPS are the same, but 1+tanh achieves better performance than sigmoid in terms of mIoU and FPS (79.25% vs 79.08%).

[0219] Table 4 Comparison of different activation functions in EGRFM module Cityscapes val 1536*768

[0220] Activation mIoU Params Flops FPS 1+tanh 79.25 13406993 97852881792 62.3 Sigmoid 79.08 13406993 97852881792 62.11

[0221] 3.4 Experimental comparison

[0222] 3.4.1Cityscapes

[0223] There are generally two approaches to evaluating the Cityscapes test dataset. One approach is to train the model on the training dataset, test the model accuracy on the validation set, then generate results on the test dataset using the trained model and submit them to the server for evaluation. The other approach is to train using the training dataset, evaluate the model by scaling the validation set images, then use the trained model to predict the test dataset and submit the generated results to the server for evaluation. We compared EFCNet with previous state-of-the-art real-time semantic segmentation models using the Cityscapes dataset. For a fair comparison, EFCNet was evaluated at two resolutions: 768×1536 and 1024×2048. Table 5 lists the model information, input resolution, FLOPs, test speed GPU used, FPS, and mIoU for various methods.

[0224] Experimental results show that the proposed EFCNet achieves a good trade-off between accuracy and speed. The resolution of EFCNet-100 is 1024×2048, and the mIoU of the validation set reaches 80.01%, and the accuracy of the test set reaches 79.22%, achieving good accuracy. Figure 5 This is the visualization effect of EFCNet on the Cityscapes dataset, from left to right are the original image, predicted image, and label. Figure 6 Comparison of visualization effects of STDC network, PP-LiteSeg network and the proposed network.

[0225] Table 5 SOTA comparison of Cityscapes dataset

[0226]

[0227] 3.4.2Camvid

[0228] Experiments on the CamVid dataset further demonstrate the performance of EFCNet. Similar to previous work, the training and inference resolutions were kept the same as the original image size. EFCNet networks pre-trained on ImageNet and EFCNet models trained on the Cityscapes dataset were used as pre-training input, respectively. As shown in Table 6, EFCNet achieved accuracies of 78.7% and 79.68% on the CamVid test set. Compared to other models, EFCNet achieved the highest accuracy. Overall, EFCNet achieved a good trade-off between accuracy and speed on the CamVid dataset.

[0229] Table 6 Camvid test SOTA comparison

[0230]

[0231]

[0232] This paper addresses the issues of unclear segmentation edges and misaligned features. By combining an encoder-decoder architecture with a multi-path approach, we propose a novel segmentation network, EFCNet, that addresses these issues while maintaining efficient segmentation. Specifically, to address these issues, an edge branch is introduced to enhance the network's edge differentiation by extracting edge information from objects in the image. Feature resampling is also employed to align high-level semantic features with low-level spatial features to mitigate the spatial feature misalignment. The calibrated features are weighted and fused by incorporating edge information to maintain fast inference speed. Furthermore, a simple pyramid pooling module is designed to aggregate global contextual information and improve semantic extraction. Experiments on two challenging datasets, Cityscapes and CamVid, demonstrate that the proposed EFCNet achieves a good compromise between segmentation accuracy and inference speed. Future work plans to improve inference speed by optimizing the number of channels and performing model compression. The proposed method will also be applied to more scenarios, including remote sensing and medical image processing.

[0233] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A method for calibrating edge-assisted features for real-time semantic segmentation, wherein the method is implemented based on a real-time semantic segmentation network. The real-time semantic segmentation network adopts a dual-branch structure, including a semantic branch and an edge branch; the semantic branch includes an encoder and a decoder; and the method is characterized in that: The edge-assisted feature calibration method includes the following steps: S1: Input the image into the encoder to extract features. The encoder is a five-stage downsampling process, and the size of the feature map in each stage is 1 / 2, 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image respectively. The feature map of each stage of the encoder contains semantic features, and the final output of the encoder is a 1 / 32 feature map. The feature maps of 1 / 4, 1 / 8, and 1 / 16 are used as the input of the three stages in the edge branch respectively; and the feature maps of 1 / 8, 1 / 16, and 1 / 32 are used as the low-level detail spatial features of the third EGRFM module, the second EGRFM module, and the first EGRFM module respectively; S2, extract edge information through edge branch, edge branch is a three-stage convolution process, including three convolution blocks, each stage corresponds to a convolution block, and the feature map size of each stage is 1 / 8 of the original image; wherein, the input of the first convolution block is the 1 / 4 feature map, which is convolved to obtain a feature map of size 1 / 8; the input of the second convolution block is the 1 / 8 feature map and the 1 / 8 feature map output by the encoder, which is convolved and concat to obtain a feature map of size 1 / 8; the input of the third convolution block is the 1 / 8 feature map and the 1 / 16 feature map output by the encoder, which is convolved and concat to obtain a feature map of size 1 / 8; the 1 / 8 feature map is a feature map containing edge information; S3, the decoder decodes the semantic features and edge information output by the encoder; the decoder is composed of three cascaded EGRFM modules; the data processing process through the EGRFM module is: The input of the EGRFM module consists of three parts: high-level semantic features, low-level detail spatial features, and edge information. At this time, the high-level semantic features of the first EGRFM module are the final output feature map of the encoder, the high-level semantic features of the second EGRFM module are the output feature map of the first EGRFM module, and the high-level semantic features of the third EGRFM module are the output feature map of the second EGRFM module. First, the input high-level semantic features and low-level detail spatial features are spliced ​​on the channel. Then, the spliced ​​features are upsampled to obtain a 2D offset feature map. Then, the 2D offset feature map is split and used as a reference to resample the high-level semantic features and low-level detail spatial features. Finally, the input edge information is used as a weight guide to fuse the resampled high-level semantic features and low-level detail spatial features to obtain the final output of the EGRFM module. S4, passes the output of the decoder through the edge segmentation head to obtain the feature map after semantic segmentation.

2. The edge-assisted feature calibration method for real-time semantic segmentation according to claim 1, characterized in that: The data processing process of the EGRFM module includes: First, the high-level semantic features and low-level detail spatial features are channel-compressed by 1×1 convolution and concatenated in the channel dimension to obtain F concat ; The mathematical formula of the above steps is expressed as: F concat =concat(Up(Conv1(F h )),Conv1(F l )) Where concat(·) represents the concatenation operation; Up(·) indicates upsampling; Conv1 represents a 1×1 convolution operation; F l Represent low-level detail spatial features; F h Represents high-level semantic features; Secondly, learn the 2D offset feature map Δ for the concatenated features l , the offset map is divided into two groups Δ1 and Δ2, which are used to resample the high-level semantic features and low-level semantic features; the sampling function Expressed as: in Represents the sampling features at the point (h+Δ1, w+Δ2); H′ represents the height of the feature offset map; W′ represents the width of the feature offset map; F (h′,w′) Represents the feature at point (h′, w′) in the feature offset graph; τ (h,w) Represents the weight at point (h, w); h and h′ represent the ordinates of the corrected point and the point on the offset feature map, respectively; w and w′ represent the horizontal coordinates of the corrected point and the point on the offset feature map, respectively; Δ1 is the high-level semantic feature offset map; Δ2 is the low-level semantic feature offset map; During the sampling process, the most favorable feature point for the current point is found to replace the point at the current position to achieve the purpose of calibration; at the same time, each set of offset maps is further divided into two groups to ensure a more precise and accurate calibration process; The edge branch is used as a guide to adaptively fuse the calibrated semantic features F h 'With the detail feature F l ', the output of EGRFM is written as: σ=1+tanh(V Edge )Sigmoid(V Edge ) Where V Edge Represents edge features; tanh() represents the tanh activation function; Sigmoid() represents the sigmoid activation function; Represents matrix multiplication; Out EGRFM Represents the output of the EGRFM module; F l ′ represents the corrected low-level detail spatial features; F h ′ represents the corrected high-level semantic features; σ represents the weight of edge feature generation.

3. The edge-assisted feature calibration method for real-time semantic segmentation according to claim 1, characterized in that: There is also a GAPPM module between the encoder and decoder; At this point, the high-level semantic features of the first EGRFM module are the output feature maps of the GAPPM module; the data processing process of the GAPPM module includes: First, the features are globally averaged pooled and channel compressed using Conv. Then, the pooled features are pooled with different kernel sizes to obtain features of different scales. The pooled features are added to the channel compressed features, and then concatenated and convolved before being input to the BiAttention module. The BiAttention module processes the input features and the channel compressed features, fuses and multiplies them, and obtains the final result. The specific processing flow of the BiAttention module is as follows: receive two inputs, namely the channel-compressed feature and the pooled-joined feature; directly add the two features and use Conv to perform channel compression to obtain the weight σ; then multiply the weight σ with the input channel-compressed feature to obtain feature 1, multiply the weight 1-σ with the input pooled-joined feature to obtain feature 2, and then add feature 1 and feature 2 to obtain the final output.

4. The edge-assisted feature calibration method for real-time semantic segmentation according to claim 1, characterized in that: The encoder is a lightweight backbone network STDC.

5. The edge-assisted feature calibration method for real-time semantic segmentation according to claim 1, characterized in that: The loss function during real-time semantic segmentation network training is: Loss=λ0·Loss0+λ1·Loss1+λ2·Loss2+λ3·Loss Edge0 +λ4·Loss Edge1 Among them, λ0, λ1, λ2, λ3, and λ4 are training loss parameters; Loss0, Loss1, and Loss2 are the loss functions of the first, second, and third semantic segmentation heads, respectively; Loss Edge0 is the marginal loss function; Loss Edge1 is the marginal loss regularization function.

6. The edge-assisted feature calibration method for real-time semantic segmentation according to claim 5, characterized in that: Loss0, Loss1, and Loss2 use cross entropy loss, and the formula is as follows: Among them L CE represents the cross entropy loss; N represents the number of samples; L i represents the cross entropy loss of each sample; M represents the number of categories; y ic Indicates whether the true category of sample i is equal to c, and the value is 0 or 1; p ic It represents the probability that the predicted sample i belongs to category c.

7. The edge-assisted feature calibration method for real-time semantic segmentation according to claim 5, characterized in that: Loss Edge0 The weighted binary cross entropy loss is adopted, and the formula is as follows: Where L_{BCE} represents the weighted binary cross entropy loss; N represents the number of samples; α i represents the weight of sample i; y i Indicates the label of sample i, which takes a value of 0 or 1; p i It represents the probability that sample i is the correct sample.

8. The edge-assisted feature calibration method for real-time semantic segmentation according to claim 5, characterized in that: Loss Edge1 The formula is as follows: Among them L reg represents edge loss; {1:s>thrs} indicates whether s is greater than the set threshold thrs. If so, the value is 1, otherwise 0; M represents the number of categories; s represents the output of the edge segmentation head when sample i belongs to class c; thrs is the threshold value; y ic Indicates whether the true category of sample i is equal to c, and the value is 0 or 1; p ic It represents the probability that the predicted sample i belongs to category c.

9. The edge-assisted feature calibration method for real-time semantic segmentation according to claim 1, characterized in that: When training the real-time semantic segmentation network, the real-time semantic segmentation network also includes: The first and second semantic segmentation heads are added after the first and second EGRFM modules, respectively. The output feature maps of the first and second semantic segmentation heads are upsampled. Losses are calculated for the images upsampled by the first and second semantic segmentation heads and the training set images, using the loss functions loss0 and loss1, respectively. The network parameters of the real-time semantic segmentation network are adjusted by minimizing the loss functions. An edge segmentation head is added after the edge branch; the output image of the edge segmentation head and the training set image are respectively calculated for loss, and the network parameters of the real-time semantic segmentation network are adjusted; the loss calculation formulas are loss functions Loss-Edge1 and Loss-Edge0 respectively; the ground truth image with the real label is extracted from the edge information through the Canny operator through the Loss-Edge0 loss function and the loss is calculated; the results of the edge segmentation head and the semantic segmentation head are regularized through the Loss-Edge1 loss function and the loss is calculated.

10. The edge-assisted feature calibration method for real-time semantic segmentation according to claim 9, characterized in that: The upsampling process includes: first upsampling through bilinear interpolation, then through a 3x3 convolution and BatchNorm, and finally through the ReLU activation function to obtain the output result.

Images