A pyramid cross-layer fusion decoder based on semantic segmentation

Abstract

The present invention discloses an image semantic segmentation method based on a pyramid cross-layer fusion decoder. By optimizing the decoder structure and utilizing RCE, rich contextual information can be generated from the feature pyramid to enhance the representation capability of the model. The cross-layer fusion extension in the ViT Adapter encoder is applied to the decoder through the Fusion Block, realizing the interactive fusion of contextual information and spatial information, thereby improving the effect of semantic segmentation.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a pyramid cross-layer fusion decoder based on semantic segmentation. Background Technology

[0002] According to the classification standards for things in daily life (i.e., semantic standards), each pixel of the input image is classified, and each pixel is assigned a color according to its category, i.e., coloring the image. Since pixels of the same category are labeled with the same color, it appears as if the image has been separated (or segmented) from the input image; hence, this technique is called semantic segmentation. How to generate semantically segmented images is the task of the semantic segmentation model. Standard semantic segmentation models have an encoder-decoder architecture, where the encoder is used for feature representation learning, and the decoder is used for pixel-level classification of the feature representations generated by the encoder. Existing semantic segmentation models can be divided into two categories: CNN-based and Transformer-based semantic segmentation models.

[0003] Semantic segmentation models based on CNNs: From the perspective of convolution characteristics, segmentation models based on CNNs can be divided into two categories: those based on dilated convolutions and those based on ordinary convolutions.

[0004] Among them, models based on dilated convolution include: PSPNet

[13] Perform regular convolutions on the pyramid layers to capture multi-scale semantic information; DeepLab series [3-6] Parallel dilated convolutions with different dilation rates are employed (different dilation rates capture contextual information at different scales). Recent work [17-20] Several extended decoders have been proposed, such as DenseASPP.

[14] Using dilated convolution with a larger dilation rate to cover a larger receptive field, and other studies... [6,18] Construct an encoder-decoder structure that utilizes multi-resolution features as multi-scale context. DANet [2] and OCNet

[17] These works enhance the representation of each pixel by aggregating the representations of context pixels, where the context consists of all pixels, unlike the global context. They also consider self-attention-based schemes.

[27] The relationships or similarities between pixels are analyzed, and weighted aggregation is performed using similarity as the weight. Furthermore, dilated convolution is still used to obtain a larger receptive field and fuse semantic information.

[0005] Models based on ordinary convolution include: FCN [1] FPN [8] and UpperNet [7]Among them, FCN is the pioneering work of semantic segmentation model, which achieves feature fusion between layers through upsampling and concatenation operations between pyramid feature maps; FPN achieves feature fusion between layers through upsampling and linear addition of features between pyramid feature maps; and UpperNet improves the model's representation ability by achieving adaptive aggregation of features through pyramid pooling modules.

[0006] Transformer-based segmentation models have revolutionized neural language processing and have been very successful in computer vision. ViT

[26] It is the first end-to-end visual transformer for image classification, which transforms the input image into a sequence and appends it to a class label. DeiT

[18] A teacher-student training strategy is introduced through distillation to improve the training efficiency of ViT. In addition to the sequence-to-sequence model structure, PVT...

[19] and SwinTransformer

[11] The emergence of Vision Transformer (VIT) has sparked interest in it. VIT has also been applied to solve downstream tasks and intensive prediction problems, particularly demonstrating good performance in parallel semantic segmentation driven by VIT.

[21] ViT is used as an encoder, and the output patch embedding is upsampled to classify pixels. Unlike SETR, Swin Transformer and ViT Adapter... [9] It applies the ideas of CNNs to Transformers (the core of the model is still the Transformer); SwinTransformer retains the pyramid structure of the output feature maps of traditional convolutional neural network encoders. This pyramid structure allows for integration with traditional neural network decoders, enabling downstream visual tasks based on Transformers. The ViT Adapter, a fusion of convolutional neural networks and ViT Transformers, is used to bridge the performance gap between ViT and vision-specific transformers. Without altering the ViT structure, multi-scale feature information is extracted through the design of a Spatial Prior Module and two feature interaction modules (Spatial FeatureInjector and Multi-Scale Feature Extractor).

[0007] In CNN-based semantic segmentation models, the use of dilated convolutions increases the size of the decoder feature map, leading to increased computational complexity for subsequent attention mechanisms. In models based on ordinary convolutions, FCN and FPN suffer from differences in feature information between deep and shallow layers; simple continuous upsampling fails to achieve effective fusion of deep and shallow features, and this fusion lacks global feature information due to the absence of an attention mechanism. Furthermore, the complexity of UpperNet is limited by the feature channels of the encoder's feature pyramid, resulting in increased computational and floating-point operations.

[0008] In Transformer-based semantic segmentation models, many architectures based on dilated convolutions cannot be used with Transformers because CNNs and Transformers are two different model structures. Since the ViT Transformer focuses on the similarity between features and lacks prior knowledge of spatial continuity, its representational power is reduced. While the Swin Transformer and ViT Adapter incorporate prior knowledge of spatial continuity, their feature pyramids have excessively high feature dimensions, leading to an increase in the number of parameters and floating-point computations.

[0009] References:

[0010] [1] J.Long, E.Shelhamer and T.Darrell, "Fully convolutional networks forsemantic segmentation," 2015 IEEE Conference on Computer Vision and PatternRecognition (CVPR), 2015, pp.3431-3440, doi:10.1109 / CVPR.2015.7298965.

[0011] [2]J.Fu et al., "Dual Attention Network for Scene Segmentation," 2019IEEE / CVF Conference on Computer Vision and Pattern Recognition(CVPR), 2019, pp.3141-3149, doi:10.1109 / CVPR.2019.00326.

[0012] [3]Chen,L.,Papandreou,G.,Kokkinos,I.,Murphy,K.P.,&Yuille,A.L.(2015).Semantic Image Segmentation with Deep Convolutional Nets and Fully ConnectedCRFs.CoRR,abs / 1412.7062.

[0013] [4]Chen,L.,Papandreou,G.,Kokkinos,I.,Murphy,K.P.,&Yuille,A.L.(2018).DeepLab:Semantic Image Segmentation with Deep Convolutional Nets,AtrousConvolution,and Fully Connected CRFs.IEEE Transactions on Pattern Analysisand Machine Intelligence,40,834-848.

[0014] [5]Chen,L.,Papandreou,G.,Schroff,F.,&Adam,H.(2017).Rethinking AtrousConvolution for Semantic Image Segmentation.ArXiv,abs / 1706.05587.

[0015] [6]Chen,L.,Zhu,Y.,Papandreou,G.,Schroff,F.,&Adam,H.(2018).Encoder-Decoder with Atrous Separable Convolution for Semantic ImageSegmentation.ECCV.

[0016] [7]Xiao,T.,Liu,Y.,Zhou,B.,Jiang,Y.,Sun,J.(2018).Unified PerceptualParsing for Scene Understanding.In:Ferrari,V.,Hebert,M.,Sminchisescu,C.,Weiss,Y.(eds)Computer Vision–ECCV 2018.ECCV 2018.Lecture Notes in ComputerScience(),vol 11209.Springer,Cham.https: / / doi.org / 10.1007 / 978-3-030-01228-1_26

[0017] [8]Lin,T.,Dollár,P.,Girshick,R.B.,He,K.,Hariharan,B.,&Belongie,S.J.(2017).Feature Pyramid Networks for Object Detection.2017 IEEE Conference onComputer Vision and Pattern Recognition(CVPR),936-944.

[0018] [9]Chen,Z.,Duan,Y.,Wang,W.,He,J.,Lu,T.,Dai,J.,&Qiao,Y.(2022).VisionTransformer Adapter for Dense Predictions.ArXiv,abs / 2205.08534.

[0019]

[10] K.He,X.Zhang,S.Ren and J.Sun,"Deep Residual Learning for ImageRecognition,"2016 IEEE Conference on Computer Vision and Pattern Recognition(CVPR),2016,pp.770-778,doi:10.1109 / CVPR.2016.90.

[0020]

[11] Z.Liu et al.,"Swin Transformer:Hierarchical Vision Transformerusing Shifted Windows,"2021 IEEE / CVF International Conference on ComputerVision(ICCV),2021,pp.9992-10002,doi:10.1109 / ICCV48922.2021.00986.

[0021]

[12] M.Cordts et al.,"The Cityscapes Dataset for Semantic Urban SceneUnderstanding,"2016 IEEE Conference on Computer Vision and PatternRecognition(CVPR),2016,pp.3213-3223,doi:10.1109 / CVPR.2016.350.

[0022]

[13] H.Zhao,J.Shi,X.Qi,X.Wang and J.Jia,"Pyramid Scene ParsingNetwork,"2017 IEEE Conference on Computer Vision and Pattern Recognition(CVPR),2017,pp.6230-6239,doi:10.1109 / CVPR.2017.660.

[0023]

[14] M.Yang,K.Yu,C.Zhang,Z.Li and K.Yang,"DenseASPP for SemanticSegmentation in Street Scenes,"2018 IEEE / CVF Conference on Computer Visionand Pattern Recognition,2018,pp.3684-3692,doi:10.1109 / CVPR.2018.00388.

[0024]

[15] J.He,Z.Deng,L.Zhou,Y.Wang and Y.Qiao,"Adaptive Pyramid ContextNetwork for Semantic Segmentation,"2019IEEE / CVF Conference on Computer Visionand Pattern Recognition(CVPR),2019,pp.7511-7520,doi:10.1109 / CVPR.2019.00770.

[0025]

[16] Z.Zhu,M.Xu,S.Bai,T.Huang and X.Bai,"Asymmetric Non-Local NeuralNetworks for Semantic Segmentation,"2019IEEE / CVF International Conference onComputer Vision(ICCV),2019,pp.593-602,doi:10.1109 / ICCV.2019.00068.

[0026]

[17] Yuan,Y.,&Wang,J.(2018).OCNet:Object Context Network for SceneParsing.ArXiv,abs / 1809.00916.

[0027]

[18] Touvron,H.,Cord,M.,Douze,M.,Massa,F.,Sablayrolles,A.,&J'egou,H.(2021).Training data-efficient image transformers&distillation throughattention.ICML.

[0028]

[19] Wang,W.,Xie,E.,Li,X.,Fan,D.,Song,K.,Liang,D.,Lu,T.,Luo,P.,&Shao,L.(2021).Pyramid Vision Transformer:A Versatile Backbone for Dense Predictionwithout Convolutions.2021 IEEE / CVF International Conference on ComputerVision(ICCV),548-558.

[0029]

[20] Zheng,S.,Lu,J.,Zhao,H.,Zhu,X.,Luo,Z.,Wang,Y.,Fu,Y.,Feng,J.,Xiang,T.,Torr,P.H.,&Zhang,L.(2021).Rethinking Semantic Segmentation from aSequence-to-Sequence Perspective with Transformers.2021 IEEE / CVF Conferenceon Computer Vision and Pattern Recognition(CVPR),6877-6886.

[0030]

[21] J.Fu,J.Liu,J.Jiang,Y.Li,Y.Bao and H.Lu,"Scene Segmentation WithDual Relation-Aware Attention Network,"in IEEE Transactions on NeuralNetworks and Learning Systems,vol.32,no.6,pp.2547-2560,June 2021,doi:10.1109 / TNNLS.2020.3006524.

[0031]

[22] Bousselham,W.,Thibault,G.,Pagano,L.,Machireddy,A.,Gray,J.,&Chang,Y.H.,et al.(2021).Efficient self-ensemble framework for semanticsegmentation.

[0032]

[23] Yuhui Yuan,Xiaokang Chen,Xilin Chen,and JingdongWang.Segmentation transformer:Object-contextual representations for semanticsegmentation,2021.

[0033]

[24] Sixiao Zheng,Jiachen Lu,Hengshuang Zhao,Xiatian Zhu,Zekun Luo,Yabiao Wang,Yanwei Fu,Jianfeng Feng,Tao Xiang,Philip H.S.Torr,and LiZhang.Rethinking semantic segmentation from a sequence-to-sequenceperspective with transformers,2021.

[0034]

[25] Cheng,B.,Misra,I.,Schwing,A.G.,Kirillov,A.,&Girdhar,R.(2021).Masked-attention Mask Transformer for Universal Image Segmentation.2022IEEE / CVF Conference on Computer Vision and Pattern Recognition(CVPR),1280-1289.

[0035]

[26] Dosovitskiy,A.,Beyer,L.,Kolesnikov,A.,Weissenborn,D.,Zhai,X.,Unterthiner,T.,Dehghani,M.,Minderer,M.,Heigold,G.,Gelly,S.,Uszkoreit,J.,&Houlsby,N.(2020).An Image is Worth 16x16 Words:Transformers for ImageRecognition at Scale.ArXiv,abs / 2010.11929.

[0036]

[27] Lin, Zhouhan, Minwei Feng, Cícero Nogueira dos Santos, Mo Yu, BingXiang, Bowen Zhou and YoshuaBengio. "A Structured Self-attentive SentenceEmbedding." ArXiv abs / 1703.03130(2017):n.pag.

[0037]

[28] Raghu, Maithra, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang and Alexey Dosovitskiy. "Do Vision Transformers See Like Convolutional NeuralNetworks?" Neural Information Processing Systems (2021). Summary of the Invention

[0038] To address the shortcomings of existing technologies, this invention aims to provide a pyramid cross-layer fusion decoder based on semantic segmentation.

[0039] To achieve the above objectives, the present invention adopts the following technical solution:

[0040] An image semantic segmentation method based on a pyramid cross-layer fusion decoder, the specific process of which is as follows:

[0041] S1, Input image;

[0042] S2, Data Preprocessing;

[0043] S3. The image processed in step S2 is fed into the encoder to generate the original feature pyramids F1, F2, F3, and F4.

[0044] S4. The original feature pyramid obtained in step S2.4 is fed into the pyramid cross-layer fusion decoder; in the pyramid cross-layer fusion decoder, the original feature pyramid first undergoes enhanced context embedding (RCE) and fusion module Fusionblock to generate feature pyramid F1. * F2 * F3 * F4 * Then F1 * F2 * F3 * F4 *The data is fed into FFPN, which outputs the final semantic segmentation result; FFPN is an UpperNet decoder without a pooling pyramid module; the specific process is as follows:

[0045] S4.1 Constructing Spatial Information: The spatial information is constructed using the spatial features inherent in the encoder itself, namely F2, F3, and F4. Where H i W i These are feature maps F i The height and width of i = 2, 3, 4, and D is the embedding dimension, which is the same as the context information dimension formed by enhanced context embedding (RCE).

[0046] S4.2, Generate context information:

[0047] Directly using the feature map F output by the encoder i For i = (2,3,4), first, a 1×1 convolutional Conv is used to compress the channels; then, F2 and F4 are upsampled and downsampled to the size of F3, respectively, resulting in F'2, F'3, and F'4 having the same size. Then, a 1×1 convolutional Conv is used, and a Flatten operation is performed, thus forming the context information. Where D is the embedding dimension;

[0048] S4.3 The fusion module consists of three parts: syringe, extractor, and cross-window attention module Swin Block. The syringe and extractor are the spatial feature syringe and multi-scale feature extractor in ViT Adapter. The syringe fuses the feature attention of spatial information into the context information, while the extractor assigns the feature attention of context information to spatial information. The cross-window attention module is used to implement the cross-window attention mechanism.

[0049] Spatial information and contextual information are fused together to generate F2. * F3 * F4 * ;

[0050] S4.4, F2 obtained in step S4.3 * F3 * F4 * After passing through the cross-layer fusion module CLGD, F1 is obtained. * ;

[0051] S4.5, Finally, F1 * F2 * F3 * F4 *The input is fed into the UpperNet decoder without the pooling pyramid module, and the final semantic segmentation result is output.

[0052] Furthermore, the specific process of step S2 is as follows:

[0053] S2.1 Normalization: Normalize the values ​​of the three channels F0 in the RGB image, i.e., F0 n =F0 / 255=[F R ,F G ,F B ] / 255, where F R ,F G ,F B The dimensions of all are 1024×2048;

[0054] S2.2 Standardization: Standardize according to the three dimensions of R, G, and B, i.e., F0 ns =(F0) n -mean) / std, where mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225].

[0055] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0056] The present invention also provides a computer device, including a processor and a memory, wherein the memory is used to store a computer program; and the processor is used to execute the computer program to implement the above-described method.

[0057] The beneficial effects of the present invention are as follows: In the method of the present invention, by optimizing the decoder structure, rich contextual information can be generated from the feature pyramid using RCE, thereby improving the model's representation ability. Furthermore, by extending the cross-layer fusion in the ViT Adapter encoder to the decoder through Fusion Block, interactive fusion of contextual information and spatial information is achieved, thereby improving the semantic segmentation effect. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of the overall architecture of the method implemented in Embodiment 1 of the present invention;

[0059] Figure 2 This is a flowchart illustrating the operation and processing of RCE in Embodiment 1 of the present invention;

[0060] Figure 3 This is a schematic diagram of the ViT Adapter encoder and Fusionblock.

[0061] Figure 4 This is a schematic diagram of feature map visualization in Embodiment 2 of the present invention. Detailed Implementation

[0062] The present invention will be further described below with reference to the accompanying drawings. It should be noted that this embodiment is based on the present technical solution and provides detailed implementation methods and specific operation processes, but the protection scope of the present invention is not limited to this embodiment.

[0063] This embodiment provides an image semantic segmentation method based on a pyramid cross-layer fusion decoder, such as... Figure 1 As shown, the specific process is as follows:

[0064] S1. Input Image: The image is an RGB image F0, and the size and shape of F0 digitally represented are 3×1024×2048; where 3 represents the three channels R (red), G (green), and B (blue), and 1024×2048 represents the width and height of the image;

[0065] S2, Data Preprocessing:

[0066] S2.1 Normalization: To facilitate feature extraction by the model, the values ​​of the three channels F0 in the RGB image need to be normalized, i.e., F0... n =F0 / 255=[F R ,F G ,F B ] / 255, where F R ,F G ,F B The dimensions of all are 1024×2048;

[0067] S2.2 Standardization: To accelerate model convergence, this embodiment performs standardization along the three dimensions of R, G, and B, i.e., F0. ns =(F0) n -mean) / std, where mean = [0.485, 0.456, 0.406], std = [0.229, 0.224, 0.225];

[0068] S3. The image processed in step S2 is fed into the encoder to generate the original feature pyramids F1, F2, F3, and F4. In this embodiment, the Swing-Large encoder is used.

[0069] S4. The original feature pyramid obtained in step S2.4 is fed into the pyramid cross-layer fusion decoder; in the pyramid cross-layer fusion decoder, the original feature pyramid first undergoes enhanced context embedding (RCE) and fusion module Fusionblock to generate feature pyramid F1. * F2 * F3 * F4 * Then F1 * F2 * F3 * F4 * The data is fed into FFPN and outputs the final semantic segmentation result; where FFPN is an UpperNet decoder without the Pooling Pyramid Module (PPM).

[0070] Specifically, if the objects in the image scene are divided into N classes, then the output size of the model is O = [O0, O1, ..., O N-1 ], where O i The dimensions are 1024×2048, O i Indicates image location [H] i W i Output the probability of the i-th category. Select [H] i W i The channel index of the highest probability at a given location is used as the category number to which that pixel belongs; pixels belonging to the same category number are all marked with the same color.

[0071] like Figure 1 As shown, reinforced context embedding (RCE) is used to obtain semantic information (F) from the feature map. c The Fusion Block is used to fuse context information and spatial information; CLGD

[20] For semantic information F c * The cross-layer fusion module of F1; FFPN is an UpperNet decoder without the pooling pyramid module PPM.

[0072] In this embodiment, the specific process of step S4 is as follows:

[0073] S4.1 Constructing Spatial Information. The method for constructing spatial information is similar to the Spatial Prior Module in the ViT Adapter. The difference is that this embodiment uses the spatial features inherent in the encoder itself (i.e., F2, F3, and F4) as spatial information. Where H i W i These are feature maps F i The height and width of (i = 2, 3, 4), where D is the embedding dimension, and this value, together with the context information F formed by the enhanced context embedding (RCE), are given. c Same dimensions.

[0074] S4.2 Generate context information.

[0075] Since the model size of the UpperNet decoder is limited by the number of feature channels in the encoder's feature pyramid, the method in this embodiment compresses the features generated by the encoder through the embedding dimension.

[0076] Specifically, to enhance the model's representational ability, the method in this embodiment compresses the original feature pyramid using RCE to form contextual information, such as... Figure 2 As shown. Since traditional convolutional neural network (CNN) structures inherently possess patch embedding capabilities—that is, extracting information at the same scale through convolution and downsampling—the feature outputs of the CNN itself can be directly used as multi-scale patch embeddings. Furthermore, the number of channels is controlled by the embedding dimension.

[0077] More specifically, the specific process of forming contextual information through RCE is as follows: Figure 2 As shown, the feature map F output by the encoder is used directly. i For i = (2,3,4), the channels are first compressed using a 1×1 convolutional Conv. Then, to enable concatenation along the channel dimension, F2 and F4 are upsampled and downsampled to the size of F3, respectively. This results in F'2, F'3, and F'4 having the same size. Finally, a 1×1 convolutional Conv is applied, and a Flatten operation is performed, thus forming the context information. Where D is the embedding dimension.

[0078] S4.3. Context information F has been obtained through steps S4.1 and S4.2. c and spatial information F spNext, we need to implement the interaction between the two pieces of information. In this embodiment, the design concept of the Fusion Block module is derived from the Cross Attention concept in the ViT Adapter. However, unlike the ViT Adapter, as shown in... Figure 3 As shown in (a), the ViT Adapter is based on ViT, but multiple ViT Blocks based on ViT will increase the computational complexity of the model. In order to reduce the complexity of the model, the method of this embodiment no longer uses multiple ViT Blocks when designing the Fusion Block of the decoder, but only uses one ViT Block to correct semantic information. In order to achieve cross-window attention, the Swing Block is used instead of the ViT Block, thereby solving the problem that it is difficult to introduce an attention mechanism on a single scale due to the size problem between layers of the feature pyramid.

[0079] Specifically, the structure of the Fusion Block is as follows: Figure 3 As shown in (b), it includes three parts: Injector, Extractor, and Swin Block. Injector and Extractor are the Spatial Feature Injector and Multi-Scale Feature Extractor in ViT Adapter, respectively. Injector integrates the feature attention of spatial information into the context information, while Extractor assigns the feature attention of context information to spatial information, thereby enabling the deep features of the feature pyramid to better act on the shallow features. Swin Block is used to implement the cross-window attention mechanism.

[0080] Spatial and contextual information are combined by the Fusion Block module to generate F2. * F3 * F4 * ;

[0081] S4.4, F2 obtained in step S4.3 * F3 * F4 * After passing through the cross-layer fusion module CLGD, F1 is obtained. * .

[0082] S4.5, Finally, F1 * F2 * F3 * F4* The input is fed into the UpperNet decoder without the pooling pyramid module, and the final semantic segmentation result is output.

[0083] Example 2

[0084] To demonstrate the effectiveness of the method in Example 1, this example focuses on the Cityscapes dataset.

[12] The following ablation experiment was performed:

[0085] 1. Evaluation criteria for the model

[0086] This embodiment evaluates the model using three criteria: Mean Intersection over Union (Mean IoU), Floating Point Operations (FLOPs), and Parameters.

[0087] ①Mean IoU:

[0088] TP: In fact, it is the number of positive samples, and the model prediction result is also the number of positive samples;

[0089] TN: In fact, it represents negative samples, and the model's prediction result is also the number of negative samples.

[0090] FP: The number of samples that are actually negative samples but the model predicts as positive samples;

[0091] FN: The number of samples that are actually positive but the model predicts as negative; therefore Mean IoU is the mean of the intersection-union ratios of all classes in the dataset. The larger the value, the higher the accuracy of the model.

[0092] ②FLOPs: refers to floating-point operations, which can be understood as computational cost and is used to measure the computational complexity of a model. The larger the value, the greater the computational cost of the model.

[0093] ③Param: refers to the number of parameters, which is the number of learnable parameters in a model and is used to measure the size of the model. The larger the value, the more space the model occupies.

[0094] 2. Ablation test of RCE

[0095] The method in Example 1 obtains rich semantic information through Relational Execution (RCE). To demonstrate the effectiveness of RCE, this example designs fusion strategies for different layers, as shown in Table 1.

[0096] Table 1

[0097]

[0098]

[0099] Table 1 shows the ablation experiment results of RCE on the Cityscapes validation set, where RCE(*) refers to the output layer F of the backbone network participating in RCE. i .

[0100] As shown in Table 1, RCE significantly improves segmentation accuracy. Compared to FCFPN, the accuracy of F2, F3, and F4 as RCE feature inputs ultimately improved by 1.97% (Res50) and 1.91% (Res101), respectively. This demonstrates that using F2, F3, and F4 as embedding layers is the best choice without significantly increasing model computation and parameter count. For semantic segmentation, context information is crucial for feature map representation; feature maps at different scales can capture different contextual information, which is more conducive to improving model segmentation accuracy. Therefore, RCE will default to using F2, F3, and F4 as embedding layers thereafter.

[0101] 3. Ablation experiment of Fusion Block (FB)

[0102] Table 2

[0103]

[0104] Table 2 shows the ablation experiment results of Fusion Block (FB) and RCE on the Cityscapes dataset. Due to the model characteristics of FB itself, that is, the two-dimensional feature map must be serialized through a Patch Embedding, this embodiment selects the deepest feature F4 as the input of Patch Embedding.

[0105] As shown in Table 2, this embodiment still uses FFPN as the baseline. First, in order to determine the impact of FB and RCE on model accuracy, the above experiments were conducted separately. Regarding FB, compared with FFPN, there are improvements of 5.71% and 4.85% respectively under Res50 and Res101 backbones; while using FB and RCE simultaneously, there are improvements of 6.09% and 5.24% respectively, thus illustrating the impact of FB and RCE on model accuracy.

[0106] 4. Model hyperparameter settings

[0107] Because the ViT Adapter designs encoders of different sizes by setting different Embedding Dim and head, this embodiment also sets decoder settings in five different modes to explore the performance of the pyramid cross-layer fusion decoder (hereinafter referred to as PCFD) in different modes. As shown in Table 3 below, PCFD is divided into five modes: tiny, small, base, and large. In the previous ablation experiments, this embodiment used the tiny mode by default.

[0108] Table 3

[0109]

[0110] Table 3 shows the hyperparameter settings for PCFD; Embedding Dim represents the feature dimension (number of feature channels) of contextual and spatial information, and Head(Space,Context,Swin) represents the number of attention heads in the Injector, Extractor, and Swing Block.

[0111] Table 4

[0112]

[0113]

[0114] Table 4 shows the experimental results on the Cityscapes dataset under different modes, where This indicates that the model was trained using OHEM.

[17] Training strategies.

[0115] The experimental results above show that PCFD does not significantly improve model accuracy with the improvement of model mode. Therefore, in subsequent experiments, only the tiny mode will be selected as the default model hyperparameter configuration.

[0116] Example 3

[0117] To demonstrate the advantages of PCFD, this embodiment aims to conduct the following comparative experiment: the experiment is still conducted on the Cityscapes dataset.

[0118] Table 5

[0119]

[0120]

[0121] Table 5 shows the comparison on the Cityscapes dataset, where FLOPs represent the model's floating-point operations for the same input size. * indicates that the model architecture was fine-tuned using other datasets during training. The model was trained using OHEM.

[26] The training strategy. # indicates that the crop size of the model is 896×896 during training. OM indicates Out of Memory.

[0122] Table 5 presents the results of the state-of-the-art methods on Cityscapes. It is divided into two groups: the first group tests with CNN as the backbone, and the second group tests with Tranformer as the backbone. On this dataset, when using standard ResNet as the representative CNN, PCFD outperforms other methods in terms of model parameter count, floating-point computation cost, and model accuracy. However, when using Swin-L and ViT-Adapter-L as the backbone, PCFD lags behind the best model accuracy (mIoU) by 0.8%, but compared to ViT-Adapter-UperNet, PCFD reduces the number of parameters and floating-point computation cost by 23% and 68%, respectively.

[0123] Feature visualization: Due to the different characteristics of different encoders, we divide encoders into three categories: CNN, ViT, and ViT Adapter. PFD plays a slightly different role in these three categories.

[0124] like Figure 4 As shown, this embodiment visualizes the feature maps F of three encoders: ResNet, Swin Transformer, and Vit Adapter. i and their respective feature maps F obtained after passing through RCE and Fusion Block. i * , where i∈[1,4], with ResNet101, Swin Large, and ViT Adapter as examples respectively. As can be seen from the figure, there is a significant difference in the feature map visualizations of ResNet101 and Swin Large. The fundamental reason for this lies in recent work.

[28] As already mentioned.

[0125] However, for F1, after feature fusion, both can clearly map the edges of segmented objects and reduce the texture inside the object's outline.

[0126] In R101, since convolutional networks focus on local spatial features, PCFD enables feature channel compression, the introduction of attention mechanisms, and the fusion of deep and shallow features.

[0127] In the Swin Transformer, since the Transformer itself emphasizes the similarity of global features, PCFD enhances the semantic distinguishability of shallow features by fusing contextual information obtained from RCE with spatial information across layers. Therefore, the role of PCFD in the Swin Transformer is to compress feature channels and fuse deep and shallow features.

[0128] In the ViT Adapter, due to its inherent characteristics (the width and height of the input image must be consistent), this embodiment selects a portion of the original image as the model input and scales the feature map to the same size as the R101-PCFD and Swin-PCFD feature maps. Visualizing the feature map shows that after PCFD, the feature discrimination between different categories increases, and after feature compression, there is no significant loss of features; the shallow feature F2 also incorporates deep contextual information.

[0129] For those skilled in the art, various corresponding changes and modifications can be made based on the above technical solutions and concepts, and all such changes and modifications should be included within the protection scope of the claims of this invention.

Claims

1. An image semantic segmentation method based on a pyramid cross-layer fusion decoder, characterized in that, The specific process is: S1, Input image; S2, Data Preprocessing; S3. The image processed in step S2 is fed into the encoder to generate the original feature pyramids F1, F2, F3, and F4. S4. The resulting original feature pyramid is fed into the pyramid cross-layer fusion decoder; in the pyramid cross-layer fusion decoder, the original feature pyramid first undergoes enhanced context embedding (RCE) and the fusion module (Fusion block) to generate feature pyramid F1. * F2 * F3 * F4 * Then F1 * F2 * F3 * F4 * The data is fed into FFPN, which outputs the final semantic segmentation result; FFPN is an UpperNet decoder without a pooling pyramid module; the specific process is as follows: S4.1 Constructing Spatial Information: The spatial information is constructed using the spatial features inherent in the encoder itself, namely F2, F3, and F4. Where H i W i These are feature maps F i The height and width of i = 2, 3, 4, and D is the embedding dimension, which is the same as the context information dimension formed by enhanced context embedding (RCE). S4.2, Generate context information: Directly using the feature map F output by the encoder i For i = (2,3,4), first, a 1×1 convolutional Conv is used to compress the channels; then, F2 and F4 are upsampled and downsampled to the size of F3, respectively, resulting in F'2, F'3, and F'4 having the same size. Then, a 1×1 convolutional Conv is used, and a Flatten operation is performed, thus forming the context information. Where D is the embedding dimension; S4.3 The fusion module consists of three parts: syringe, extractor, and cross-window attention module Swin Block. The syringe and extractor are the spatial feature syringe and multi-scale feature extractor in ViT Adapter. The syringe fuses the feature attention of spatial information into the context information, while the extractor assigns the feature attention of context information to spatial information. The cross-window attention module is used to implement the cross-window attention mechanism. Spatial information and contextual information are fused together to generate F2. * F3 * F4 * ; S4.4, F2 obtained in step S4.3 * F3 * F4 * After passing through the cross-layer fusion module CLGD, F1 is obtained. * ; S4.5, Finally, F1 * F2 * F3 * F4 * The input is fed into the UpperNet decoder without the pooling pyramid module, and the final semantic segmentation result is output.

2. The method according to claim 1, characterized in that, The specific process of step S2 is as follows: S2.1 Normalization: Normalize the values ​​of the three channels F0 in the RGB image, i.e., F0 n =F0 / 255=[F R ,F G ,F B ] / 255, where F R ,F G ,F B The dimensions of all are 1024×2048; S2.2 Standardization: Standardize according to the three dimensions of R, G, and B, i.e., F0 ns =(F0) n -mean) / std, where mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225].

3. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-2.

4. A computer device, characterized in that, It includes a processor and a memory, the memory being used to store a computer program; the processor being used to execute the computer program to implement the method of any one of claims 1-2.

Images