Urban streetscape image semantic segmentation method based on spatial content feature cross fusion

Abstract

The invention relates to the field of automatic driving semantic segmentation, in particular to an urban streetscape image semantic segmentation method based on spatial content feature cross fusion. The method comprises the following steps: establishing an improved Segmenter model, and training the improved Segmenter model to obtain a trained improved Segmenter model; the trained improved Segmenter model is used to carry out semantic segmentation on the urban street scene image; wherein the improved Segmenter model comprises a trunk part, a space branch, a content branch and a mask decoder; the backbone part is a backbone of the ViT model and is used for extracting basic features of the input image; the spatial branch is used for realizing spatial feature enhancement by utilizing the query Q of each encoder layer of the trunk part; and the content branch is used for collecting the feature of each encoder layer of the trunk part, fusing the multi-level features and extracting the multi-scale features. According to the method, extraction of space details is improved, multi-stage context information is aggregated, features of different scales and levels are effectively integrated, and the effectiveness and efficiency of processing a complex city semantic segmentation task in automatic driving application are improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving semantic segmentation, and in particular to a semantic segmentation method for urban street scene images based on cross-fusion of spatial content features. Background Art

[0002] In recent years, advances in deep learning have significantly driven progress in computer vision, particularly in semantic segmentation. Semantic segmentation aims to predict the class of each pixel in an image and falls under the category of pixel-by-pixel classification. This technology plays a vital role in a variety of fields, including autonomous driving, medical image analysis, and remote sensing image interpretation.

[0003] With the introduction of the Vision Transformer (ViT), leveraging self-attention mechanisms to capture global image information has become a new research trend. However, direct application of ViT to semantic segmentation tasks still faces many challenges, including insufficient interaction between high-level and low-level features and insufficient fusion of scale features. Effective utilization of low-level features is crucial in semantic segmentation tasks, as spatial details are often lost as network depth increases, while semantic features become more prominent.

[0004] Segmenter uses the encoder's output directly as the decoder's input query, but this ignores the information in the middle and bottom layers.

[0005] By analyzing existing Transformer-based semantic segmentation models, we found that their main problems lie in the underutilization of hierarchical features and the lack of scale characteristics, resulting in insufficient semantic information. Furthermore, although ViT introduces spatial information through positional encoding, its efficient use of spatial position is still insufficient, limiting the performance and efficiency of the model.

[0006] Therefore, it is urgent to solve the problems existing in the existing Transformer-based semantic segmentation model. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to overcome the defects of the existing technology and provide a semantic segmentation method for urban street scene images based on the cross-fusion of spatial content features. It improves the extraction of spatial details and aggregates multi-stage contextual information, effectively integrating features of different scales and levels, and can improve the effectiveness and efficiency of processing complex urban semantic segmentation tasks in autonomous driving applications.

[0008] In order to solve the above technical problems, the technical solution of the present invention is: a semantic segmentation method for urban street scene images based on cross-fusion of spatial content features, the method comprising:

[0009] Build an improved Segmenter model and train it to obtain a trained improved Segmenter model;

[0010] Use the trained improved Segmenter model to perform semantic segmentation on urban street scene images;

[0011] Improve the Segmenter model to include the backbone, spatial branch, content branch, and mask decoder;

[0012] The main body is divided into the backbone of the ViT model, which is used to extract the basic features of the input image;

[0013] The spatial branch is used to achieve spatial feature enhancement using the query Q of each encoder layer of the backbone;

[0014] The content branch is used to collect features from each encoder layer of the backbone, fuse multi-level features, and extract multi-scale features. The output of the multi-scale features is fused with the patch token features output by the backbone to obtain the fused patch token features.

[0015] The mask decoder is used to take the fused Patch token features and the regularly initialized Class Token as input, and fuse the output result Patch token with the output result Class Token by dot product. The fusion result is added to the output of the spatial branch to obtain the predicted mask, and the predicted mask is upsampled to achieve segmentation.

[0016] Furthermore, the query Q of each encoder layer in the backbone is used to enhance spatial features. The specific steps are as follows:

[0017] The query Q of each encoder layer is stacked in the first dimension, and then the mean is calculated using mean. The mean feature is divided into N subgroups in the channel dimension. The dot product similarity between the feature of each subgroup and its global pooled feature is calculated to generate a preliminary attention mask;

[0018] The preliminary attention mask is normalized and then nonlinearly transformed by the sigmoid function to obtain the final attention mask.

[0019] Furthermore, the features of each encoder layer in the backbone are collected and fused with multi-level features. The specific steps are as follows:

[0020] The features of the first encoder layer output F1 are used as the key K and value V, and the outputs F2, F3 and F4 of the second, third and fourth encoder layers are used as the query Q respectively. Attention calculation is performed one by one, and linear transformation is performed after splicing to achieve fusion.

[0021] Furthermore, the features of each encoder layer of the backbone are collected and fused with multi-level features. The calculation formula is:

[0022] f i =Pool i (F1)

[0023]

[0024] FC=LinearFusion(Concat(X2,X3,X4))

[0025] i=2,3,4;Pool i Represents the pooling operation, for different F i , the pooling ratio is different; X i Represents the output of the corresponding layer after attention; LinearFusion represents the linear transformation operation; FC represents the fused output; Concat(X2,X3,X4) represents the X i Perform concatenation on the channel dimension.

[0026] Furthermore, multi-scale features are extracted. The specific steps are as follows:

[0027] Use multi-level max pooling operations to reduce the size of the input feature map, and apply convolution kernels with different expansion rates to extract contextual information from different receptive fields layer by layer;

[0028] The reduced feature map is fused with the context feature through channel splicing, and then the channel is compressed using 1×1 convolution and residually connected with the input feature map to obtain the output feature.

[0029] The present invention also relates to a device for semantic segmentation of urban street scene images based on cross-fusion of spatial content features, comprising:

[0030] Building modules for building improved Segmenter models;

[0031] The training module is used to train the improved Segmenter model to obtain the trained improved ViT model;

[0032] The segmentation module is used to perform semantic segmentation on urban street scene images using the trained improved ViT model;

[0033] Improve the Segmenter model to include the backbone, spatial branch, content branch, and mask decoder;

[0034] The main body is divided into the backbone of the ViT model, which is used to extract the basic features of the input image;

[0035] The spatial branch is used to enhance spatial features using the query Q of each encoder layer in the backbone.

[0036] The content branch is used to collect features from each encoder layer of the backbone, fuse multi-level features, and extract multi-scale features. The multi-scale features are fused with the patch token features output by the backbone to obtain the fused patch token features.

[0037] The mask decoder is used to take the fused Patch token features and the Class Token output by the backbone as input, and the output result is fused with the output of the spatial branch by dot multiplication to obtain the predicted mask. The predicted mask is upsampled to achieve segmentation.

[0038] The present invention also relates to a device comprising:

[0039] memory for storing computer programs;

[0040] The processor is used to execute the computer program, which, when executed by the processor, implements the steps of the urban street scene image semantic segmentation method based on cross-fusion of spatial content features.

[0041] The present invention also relates to a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a method for semantic segmentation of urban street scene images based on cross-fusion of spatial content features.

[0042] The present invention also relates to a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of a method for semantic segmentation of urban street scene images based on cross-fusion of spatial content features.

[0043] By adopting the above technical solutions, our improved Segmenter model achieves an average performance improvement of 1.6% on the ADE20K dataset and a 2.5% performance improvement on the ViT-tiny configuration compared to Segmenter, while only slightly increasing the number of parameters by 2M and computational speed by 0.6GFLOPs. These results demonstrate the effectiveness and efficiency of our approach in handling complex urban semantic segmentation tasks in autonomous driving applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flow chart of the method for semantic segmentation of urban street scene images based on cross-fusion of spatial content features of the present invention;

[0045] Figure 2 This is a framework diagram of the improved Segmenter model of the present invention;

[0046] Figure 3This is a structural diagram of the spatial feature enhancement module of the present invention;

[0047] Figure 4 This is a structural diagram of the multi-stage attention module and dense continuous spatial pyramid pooling module of the present invention;

[0048] Figure 5 This is a comparison chart of the segmentation results of the traditional Segmenter model and the improved Segmenter model of the present invention on the ADE20K dataset. DETAILED DESCRIPTION

[0049] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments in conjunction with the accompanying drawings.

[0050] Example 1

[0051] like Figures 1 to 4 As shown in FIG, a semantic segmentation method for urban street scene images based on cross-fusion of spatial content features includes:

[0052] Step S1, build an improved Segmenter model, which includes a trunk, a spatial branch, a content branch, and a mask decoder; the trunk is divided into the backbone of the ViT model, which is used to extract the basic features of the input image; the spatial branch is used to use the query Q of each encoder layer of the trunk to achieve spatial feature enhancement; the content branch is used to collect the features of each encoder layer of the trunk, fuse multi-level features, and extract multi-scale features; wherein the multi-scale features are fused with the Patch token features output by the trunk to obtain the fused Patch token features; the mask decoder is used to take the fused Patch token features and the regularly initialized Class Token as input, and fuse the output result Patch token with the output result Class Token by dot product, add the fusion result to the output of the spatial branch to obtain a prediction mask, and upsample the prediction mask to achieve segmentation;

[0053] Step S2, training the improved Segmenter model to obtain a trained improved Segmenter model;

[0054] In step S3, the trained improved Segmenter model is used to perform semantic segmentation on the city street scene image.

[0055] In this embodiment, if Figure 1 and Figure 2As shown in the figure, the improved Segmenter model designs a spatial feature enhancement (SFEM) module, a multi-stage attention (MStA) module and a dense continuous spatial pyramid pooling (DCASPP) module based on the traditional Segmenter model. The spatial branch uses the query to enhance spatial information through the SFEM module to improve the perception of image details; the content branch uses the MStA module to integrate multi-stage output features and realizes multi-scale information extraction through DCASPP, thereby enhancing the overall semantic expression ability of the model.

[0056] (1) A detailed introduction to the spatial feature enhancement (SFEM) module.

[0057] The spatial feature enhancement module is a key component designed to enhance the model's ability to capture image details, such as Figure 3 As shown in Figure 2, the ViT model's backbone lacks spatial information support in deep layers, and its utilization of spatial information is relatively primitive, simply by adding learnable position embeddings of equal size, which limits its efficiency. Our proposed Spatial Feature Enhancement (SFEM) module aims to enhance the model's ability to capture and express spatial information.

[0058] In many existing models, the queries generated by the encoder are often overlooked. The SFEM module collects query vectors from each encoder layer. To reduce feature redundancy, we stack and average these queries along the first dimension (batch), focusing on extracting key spatial information, which is then used as the input to the SFEM module. The core mechanism of SFEM enhances features in the spatial dimension, more accurately capturing details in the image, and thus improving the accuracy of the segmentation mask output by the decoder.

[0059] The specific operation process of the SFEM module is as follows Figure 3 As shown in Figure 1, the query Q of each encoder layer is stacked in the first dimension, and then the mean is calculated using mean. The mean features are divided into N subgroups in the channel dimension, and the dot product similarity between the features of each subgroup and its global pooled features is calculated to generate a preliminary attention mask.

[0060] The preliminary attention mask is normalized and then nonlinearly transformed by the sigmoid function to obtain the final attention mask.

[0061] Through this method, the SFEM module can enhance the features of specific regions in a targeted manner, improving the model's perception capabilities at different spatial scales. This multi-level feature processing strategy effectively improves the segmentation performance of the improved Segmenter model in complex scenes, enabling it to more accurately process details and global information in the image.

[0062] (2) A detailed introduction to the multi-stage attention (MStA) module.

[0063] In the backbone of Vision Transformer (ViT), as the number of layers increases, although the semantic features become more obvious, the detailed features gradually decrease. These high-resolution detailed features are crucial to improving the accuracy of the model. In order to solve the problem of high-level and low-level feature interaction and missing detailed features in the backbone network, we introduced the Multi-Stage Attention (MStA) module, such as Figure 4 As shown, this module is designed to remedy this shortcoming.

[0064] The core idea of the MStA module is to make full use of the outputs generated by the model at different stages to improve its understanding of global and local information. We extract features from the four stages of the model, and the features of each stage represent different semantic depths. In this process, we specifically use the output features of the first stage as keys and values. These features contain basic global information and serve as the core of subsequent operations. On this basis, we use the output features of the second, third, and fourth stages as queries, respectively. These features represent deeper semantic information. By performing attention calculations on these queries and keys one by one, we effectively integrate feature information at different levels, enhance the model's ability to understand complex scenes, and capture more key details, effectively improving the accuracy and robustness of segmentation.

[0065] In order to effectively control the amount of computation, we pooled the output features of the first stage to reduce the spatial dimension of the features and reduce the computational complexity. In addition, although we integrated information from multiple feature levels, the entire module only required one attention operation, which greatly saved computational resources while maintaining the model's efficient processing capability for complex image data. The calculation formula can be: i =Pool i (F1)

[0066]

[0067] FC=LinearFusion(Concat(X2,X3,X4))

[0068] F1 represents the output of the first encoder layer; F2, F3, and F4 represent the outputs of the second, third, and fourth encoder layers, respectively; i = 2, 3, 4; Pool i Represents the pooling operation, for different F i , the pooling ratio is different; X iRepresents the output of the corresponding layer after attention; LinearFusion represents the linear transformation operation; FC represents the fused output; Concat(X2,X3,X4) represents the X i Perform concatenation on the channel dimension.

[0069] (3) A detailed introduction to the Dense Continuous Spatial Pyramid Pooling (DCASPP) module.

[0070] We propose a new module, Dense Convolution Atrous Spatial PyramidPooling (DCASPP), to further extract multi-scale information from the feature maps output by the MStA module, aiming to further enhance the model's understanding of global and local contextual information.

[0071] We give the internal structure of Dense Convolution Atrous Spatial Pyramid Pooling (DCASPP), as shown in Figure 4 Traditional pyramid pooling (such as SPP and ASPP) usually extracts features through pooling windows of different sizes or fixed convolution expansion rates. However, when processing complex scenes, the choice of receptive field is relatively limited and it often cannot effectively adapt to the diversity of targets.

[0072] The DCASPP module combines dense dilated convolution with multi-scale pooling to not only expand the receptive field but also capture the features of objects of different sizes in a more flexible way. The DCASPP module first reduces the size of the input feature map through multi-level maximum pooling operations, and at the same time applies convolution kernels with different dilation rates to extract contextual information from different receptive fields layer by layer ( Figure 4 In the DCASPP module, Maxpool2d represents the maximum pooling module, and the green Conv represents dense convolutions with different dilation rates. The Concat module performs concatenation along the channel dimension. These features are ultimately fused through channel concatenation, then compressed using 1×1 convolutions. A residual connection is then made with the input feature map, and finally fused with the Patchtoken input into the decoder. The DCASPP module not only effectively improves the model's performance in complex scenarios, but also ensures a low computational cost through reasonable computational optimization, achieving a balance between accuracy and efficiency.

[0073] Through the detailed introduction of each of the above modules, the design ideas of the improved Segmenter model in this embodiment are summarized as follows:

[0074] First, to enhance spatial information and query utilization, we designed a spatial feature enhancement (SFEM) module. This module collects queries from the encoding phase, performs spatial enhancement on them, and then fuses these enhanced features with the mask output by the model decoder. This design significantly improves the spatial representation of the mask, enriching and refining the feature representation.

[0075] Secondly, in order to overcome the shortcomings of the traditional Vision Transformer (ViT) model in terms of hierarchical feature expression and multi-scale information capture, we introduced the Multi-Stage Attention (MStA) module and the Dense Continuous Spatial Pyramid Pooling (DCASPP) module. MStA divides the ViT model into four stages, using the output of the first stage as the key and value, and the output of the subsequent three stages as the query, so that high-level features can receive more relevant low-level feature information. At the same time, DCASPP performs spatial pooling of input features at different scales and uses convolutions with different expansion rates to effectively extract multi-scale information. This design not only deepens the understanding of low-level information, but also enhances the ability to capture global and local information without increasing a lot of computational overhead.

[0076] Furthermore, the model design of this embodiment fully utilizes existing backbone features. SFEM requires only limited convolution operations, MStA requires only a single attention mechanism, and DCASPP utilizes convolution and pooling operations to ensure computational efficiency. We demonstrate the superiority of the proposed method in terms of model size, computational complexity, and performance on two public datasets. Specific contributions are as follows:

[0077] A. We propose a spatial enhancement module (SFEM) that makes better use of the encoder's query and compensates for the lack of spatial information in the encoder's later stages. Since it leverages the existing query in the VIT and performs limited convolution operations, the computational cost is also kept low.

[0078] B. We propose a new method called Multi-stage Attention (MStA), which effectively promotes the interaction between high- and low-level features within the encoder, ensuring the fusion of high-level semantic information and low-level structural information. This method can be achieved with a single attention process, significantly improving the performance of semantic segmentation tasks while maintaining low computational cost.

[0079] C. We designed the Dense Continuous Atrous Spatial Pyramid Pooling (DCASPP) multi-scale feature extraction module. The DCASPP module combines convolution operations with different dilation rates and maximum pooling operations. This not only maintains a low computational cost but also significantly enhances the model's ability to detect objects of different sizes.

[0080] D. Our proposed model achieves excellent performance on the ADE20K and Pascal Context datasets. On the challenging ADE20K dataset, our model improves mean intersection-over-union (mIoU) by 1.6% compared to the original method, and improves it by 2.5% on the ViT-tiny scale, with almost no increase in computational effort and parameters.

[0081] Example 2

[0082] A device for semantic segmentation of urban street scene images based on cross-fusion of spatial content features, comprising:

[0083] Building modules for building improved Segmenter models;

[0084] The training module is used to train the improved Segmenter model to obtain the trained improved ViT model;

[0085] The segmentation module is used to perform semantic segmentation on urban street scene images using the trained improved ViT model;

[0086] Improve the Segmenter model to include the backbone, spatial branch, content branch, and mask decoder;

[0087] The main body is divided into the backbone of the ViT model, which is used to extract the basic features of the input image;

[0088] The spatial branch is used to achieve spatial feature enhancement using the query Q of each encoder layer of the backbone;

[0089] The content branch is used to collect features from each encoder layer of the backbone, fuse multi-level features, and extract multi-scale features. The multi-scale features are fused with the patch token features output by the backbone to obtain the fused patch token features.

[0090] The mask decoder is used to take the fused Patch token features and the Class Token output by the backbone as input, and the output result is fused with the output of the spatial branch by dot multiplication to obtain the predicted mask. The predicted mask is upsampled to achieve segmentation.

[0091] Example 3

[0092] A device comprising:

[0093] memory for storing computer programs;

[0094] The processor is configured to execute the computer program, wherein when the computer program is executed by the processor, the steps of the method for semantic segmentation of urban street scene images based on cross-fusion of spatial content features as described in the above embodiment are implemented.

[0095] Example 4

[0096] A readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the urban street scene image semantic segmentation method based on cross-fusion of spatial content features as described in the above embodiment.

[0097] Example 6

[0098] A computer program product includes a computer program, which, when executed by a processor, implements the steps of the urban street scene image semantic segmentation method based on cross-fusion of spatial content features as described in the above embodiment.

[0099] The improved Segmenter model involved in the above embodiment is verified through specific experiments below.

[0100] (1) Datasets and evaluation metrics

[0101] The ADE20K dataset, consisting of complex scenes with fine-grained labels, is one of the most challenging datasets in the field of semantic segmentation. Its training set consists of 20,210 images covering 150 semantic categories. The validation and test sets contain 2,000 and 3,352 images, respectively.

[0102] The training set of the Pascal Context dataset contains 4,996 images, including 59 semantic categories and one background category, and the validation set contains 5,104 images.

[0103] In the evaluation criteria for model performance, we used mean intersection over union (mIoU) as the main metric.

[0104] Mean Interference over Union (mIoU) is calculated by averaging the ratio of the intersection-over-union of predicted results and the true labels across all categories. This metric is one of the most popular evaluation methods in current semantic segmentation research. "ssmIoU" and "msmIoU" represent mIoU calculated using single-scale and multi-scale inference, respectively. FLOPs (G) and Params represent the computational overhead and parameter count of the model, respectively, and are key metrics for measuring model complexity and size.

[0105] (2) Comparison with the most advanced models

[0106] Table 1 shows the results on the Pascal-Context dataset. We follow HRNet to evaluate our method and report results on 59 classes (without context) and 60 classes (with context). SCFI-ESeg achieves mIoU of 65.4% and 60.0%, respectively, outperforming other state-of-the-art methods using the ViT-L backbone, such as SegViT and APPNet.

[0107] Table 1: Experimental results on Pascal-Context val. Following published methods, we report results using multi-scale inference (denoted by “ms”). msmIoU59 indicates the average mIoU over 59 classes (without background). msmIoU60 indicates the average mIoU over 60 classes (59 classes plus background).

[0108]

[0109] Table 2 compares several models using the ViT backbone with state-of-the-art methods on the ADE20K validation set. SCFI-ESeg achieves state-of-the-art performance with slightly increased complexity using SFEM, MStA, and the DCASPP module. Our method achieves 55.0% mIoU on the ViT-Large backbone. This represents a 0.8% improvement over the recent StructToken [] using the same backbone and a 1.6% increase over the original Segmenter [] model.

[0110] Table 2: Experimental results on ADE20k val. Indicates that the model uses the backbone weights pre-trained by AugReg. "*" indicates that the model is copied with the same settings as the official version.

[0111]

[0112] Figure 5 The qualitative comparison of segmentation maps generated by SCFI-ESeg-T and Segmenter-T using the ADE20K dataset is shown. Clearly, SCFI-ESeg-T is able to produce better predictions than Segmenter-T. Figure 5 In the visualization results in the first row, unlike Segmenter, SCFI-ESeg maintains clear segmentation boundaries and avoids misclassifying furniture like the TV and cabinets as part of other furniture. In the second row, SCFI-ESeg better segments the regions, avoiding misclassification within them. In the third row, the pillows have clear segmentation boundaries.

[0113] (3) Ablation experiment

[0114] We will conduct an ablation study to demonstrate the effectiveness of our proposed method. Secondly, SCFI-ESeg is mainly based on the Segmenter [ ] improvement, so the experiment defaults to comparing the Segmenter model. We use the mask transformer [ ] as the main prediction decoder by default, and unless otherwise specified, we default to using single-scale inference on the ADE20k dataset to report results.

[0115] In Table 3, we study the impact of transformer size on performance by experimenting with four different ViT variants: Tiny, Small, Base, and Large. Our method continues to improve on the baseline variant, and the improvement on the ADE20K dataset is impressive, with an average performance improvement of 1.6% (the average performance here refers to the improvement at each scale. The average performance of the model at each scale under the condition of ss.mIoU is 1.6%), and the maximum performance improvement is 2.5%. This is because SCFI-ESeg has been enhanced from the spatial and context aspects, making up for the shortcomings of the original Segmenter. The number of parameters and computational speed have only increased slightly by 2M (actually 1.7M, rounded to 2M) and 0.6GFLOP.

[0116] Table 3: Comparative experiments between SCFI-ESeg and Segmenter. We compare Seg [ ] and SCFI-ESeg models of all four scales on the ADE20K val and Pascal Context datasets. #P(M) denotes the number of model parameters, #F(G) denotes the model computational overhead, and mIoU(ss) and mIoU(ss) denote single-scale and multi-scale reasoning, respectively.

[0117]

[0118] Table 4 shows the performance of the SFEM module. We used the original Segmenter method as the baseline, meaning no feature enhancement was performed. We used various enhancement methods, including spatial attention [ ] and CBAM [ ], but the results were not very satisfactory. The CBAM enhancement method even resulted in performance degradation, while the SFEM module effectively improved model performance. In addition, we set the variable λ to control the influence of the SFEM model. Experiments show that when λ is 1, the best results are achieved.

[0119] Table 4: Comparison of different SFEM methods and the impact of λ on the model. SA and CBAM represent different augmentation methods, Origin represents the original experimental results of Segmenter, and λ represents the influence of SFEM on the model. Experiments were conducted using the ADE20K dataset on the ViT-Base backbone network.

[0120]

[0121] Table 5 shows the effective performance of querying, using the original Segmenter as a comparison. We clearly see that querying improves model performance compared to the original Segmenter baseline. We then used different query processing methods, including summing, adding learnable weights to each query, and taking the mean of all queries. Based on experience, we chose to take the mean of all queries, which achieved the best performance.

[0122] Table 5: Comparison of experimental results using different query processing methods. Accumulation represents the accumulation of all queries, Learnable weight represents the addition of learnable weights to each query, and Mean of query represents the mean of all accumulated queries. The experiment was conducted using the ADE20K dataset on the ViT-Small backbone network.

[0123]

[0124] Table 6 demonstrates the effectiveness of using different stage output features as MStA input. To determine which stage output performs best as the key and value, we experimented with varying the key and value from F1 to F4. The results show that F1 performs best as both the key and value. Based on this observation, we use F1 as the key and value for MStA.

[0125] Table 6: Model performance using different stage output features as keys and values. K and V represent key and value. F1-F4 represent the outputs of different stages. Experiments were conducted on the ViT-tiny backbone network.

[0126]

[0127] Table 7 compares DCASPP with other multi-scale feature extraction modules, including Spatial Pyramid Pooling (SPP), Spatial Pyramid Pooling-Fast (SPPF), and Atrous Spatial Pyramid Pooling (ASPP). As can be seen from Table 7, the proposed DCASPP significantly improves upon other multi-scale feature extraction modules, achieving a 1.1% improvement over ASPP while only increasing the number of parameters by 0.4M.

[0128] Table 7: Comparison of DCASPP and other multi-scale modules. The experiments used the ViT-tiny backbone network.

[0129]

[0130] Table 8 investigates the effectiveness of SFEM, MStA, and DCASPP on the ADE20k validation set using ViT-tiny as the backbone. The results are shown in Table 9. Integrating SFEM into Segmenter improves ssmIoU and msmIoU performance on the ADE20K validation set by 0.8% and 0.9%, respectively, with virtually no additional parameters or computational cost. MStA improves ssmIoU and msmIoU performance by 0.6% and 1.0%, respectively, based on SFEM, with only a small increase in parameters. Finally, a model combining SFEM, MStA, and DCASPP achieves performance improvements of up to 1.9% and 2.5% on single-scale and multi-mIoU, respectively, demonstrating the superiority of our SCFI-ESeg.

[0131] Table 8: Effectiveness of the proposed components in the context of the ADE20K dataset and the ViT-tiny backbone network. #param(M) represents the total number of parameters of the model. #F(G) represents the computational cost in FLOPs.

[0132]

[0133]

[0134] (4) Conclusion

[0135] In response to the shortcomings of existing models in utilizing spatial information and capturing global multi-level and multi-scale information, this paper proposes an innovative semantic segmentation framework SCFI-ESeg, which aims to improve the accuracy of semantic segmentation by integrating spatial information and content information. To this end, we propose the Spatial Feature Enhancement Module (SFEM), Multi-Stage Attention (MStA), and Dense Continuous Spatial Pyramid Pooling (DCASPP) modules respectively. The spatial information enhancement branch uses the query to enhance spatial information through the SFEM module to improve the perception of image details; the content information enhancement branch uses the attention mechanism to integrate multi-stage output features and realizes the fusion of multi-scale information through DCASPP, thereby enhancing the overall semantic expression ability of the model. We trained and evaluated on the Segmenter-based backbone network and achieved significant performance improvements on the ADE20K and Pascal Context datasets, especially on the ADE20K dataset, with a performance improvement of up to 2.5%. Importantly, because the model primarily uses existing backbone features, query processing involves only limited convolution operations, and the multi-stage attention mechanism requires only a single Attention operation, SCFI-ESeg not only improves performance while maintaining low computational cost, but also proves its feasibility and efficiency in practical applications.

[0136] With the above-described preferred embodiments of the present invention as a guide, and with reference to the above description, relevant personnel are fully capable of making various changes and modifications without departing from the technical scope of this invention. The technical scope of this invention is not limited to the contents of the specification and must be determined according to the scope of the claims.

Claims

1. A semantic segmentation method for urban street scene images based on cross-fusion of spatial content features, characterized by: Methods include: Build an improved Segmenter model and train it to obtain a trained improved Segmenter model; Use the trained improved Segmenter model to perform semantic segmentation on urban street scene images. The improved Segmenter model includes a backbone, a spatial branch, a content branch, and a mask decoder. The main body is divided into the backbone of the ViT model, which is used to extract the basic features of the input image; The spatial branch is used to achieve spatial feature enhancement using the query Q of each encoder layer of the backbone; The content branch is used to collect features from each encoder layer of the backbone, fuse multi-level features, and extract multi-scale features. The multi-scale features are fused with the patch token features output by the backbone to obtain the fused patch token features. The mask decoder is used to take the fused Patch token features and the regularly initialized Class Token as input, and fuse the output result Patch token with the output result Class Token by dot product. The fusion result is added to the output of the spatial branch to obtain the predicted mask, and the predicted mask is upsampled to achieve segmentation.

2. The urban street scene image semantic segmentation method based on cross-fusion of spatial content features according to claim 1 is characterized in that: The query Q of each encoder layer in the backbone is used to enhance spatial features. The specific steps are as follows: The query Q of each encoder layer is stacked in the first dimension, and then the mean is calculated using mean. The mean feature is divided into N subgroups in the channel dimension. The dot product similarity between the feature of each subgroup and its global pooled feature is calculated to generate a preliminary attention mask; The preliminary attention mask is normalized and then nonlinearly transformed by the sigmoid function to obtain the final attention mask.

3. The urban street scene image semantic segmentation method based on cross-fusion of spatial content features according to claim 1 is characterized in that: Collect the features of each encoder layer of the backbone and fuse the features of multiple levels. The specific steps are: The features of the first encoder layer output F1 are used as the key K and value V, and the outputs F2, F3 and F4 of the second, third and fourth encoder layers are used as the query Q respectively. Attention calculation is performed one by one, and linear transformation is performed after splicing to achieve fusion.

4. The urban street scene image semantic segmentation method based on cross-fusion of spatial content features according to claim 3 is characterized in that: Collect the features of each encoder layer of the backbone and fuse the features of multiple levels. The calculation formula is: f i =Pool i (F1) FC=LinearFusion(Concat(X2,X3,X4)) i=2,3,4;Pool i Represents the pooling operation, for different F i , the pooling ratio is different; X i Represents the output of the corresponding layer after attention; LinearFusion represents the linear transformation operation; FC represents the fused output; Concat(X2,X3,X4) represents the X i Perform concatenation on the channel dimension.

5. The urban street scene image semantic segmentation method based on cross-fusion of spatial content features according to claim 1 is characterized in that: Extract multi-scale features. The specific steps are: Use multi-level max pooling operations to reduce the size of the input feature map, and apply convolution kernels with different expansion rates to extract contextual information from different receptive fields layer by layer; The reduced feature map is fused with the context feature through channel splicing, and then the channel is compressed using 1×1 convolution and residually connected with the input feature map to obtain the output feature.

6. A device for semantic segmentation of urban street scene images based on cross-fusion of spatial content features, characterized in that: include: Building modules for building improved Segmenter models; The training module is used to train the improved Segmenter model to obtain the trained improved ViT model; The segmentation module is used to perform semantic segmentation on urban street scene images using the trained improved ViT model; Improve the Segmenter model to include the backbone, spatial branch, content branch, and mask decoder; The main body is divided into the backbone of the ViT model, which is used to extract the basic features of the input image; The spatial branch is used to achieve spatial feature enhancement using the query Q of each encoder layer of the backbone; The content branch is used to collect features from each encoder layer of the backbone, fuse multi-level features, and extract multi-scale features. The multi-scale features are fused with the patch token features output by the backbone to obtain the fused patch token features. The mask decoder is used to take the fused Patch token features and the Class Token output by the backbone as input, and the output result is fused with the output of the spatial branch by dot multiplication to obtain the predicted mask. The predicted mask is upsampled to achieve segmentation.

7. A device, characterized in that include: memory for storing computer programs; A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of the urban street scene image semantic segmentation method based on cross-fusion of spatial content features according to any one of claims 1 to 5 are implemented.

8. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the urban street scene image semantic segmentation method based on cross-fusion of spatial content features are implemented as described in any one of claims 1 to 5.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for semantic segmentation of urban street scene images based on cross-fusion of spatial content features are implemented as described in any one of claims 1 to 5.

Images