An ultra-high resolution image semantic segmentation method based on UAF and BACA

Abstract

This application relates to the field of computer vision technology, and particularly to a semantic segmentation method for ultra-high resolution images based on User-Agent Forwarding (UAF) and Basic Association for Convolutional Mapping (BACA). The method includes: extracting features from a given ultra-high resolution image using a global branch feature encoder and a local branch feature encoder, respectively, to obtain global semantic features and local detail features; inputting the global semantic features and local detail features into a UAF module for two-stage fusion to obtain UAF fused features; inputting the UAF fused features into a BACA module for boundary structure restoration to obtain boundary-aware enhanced features; and inputting the boundary-aware enhanced features into a segmentation output module to obtain the category prediction result for each pixel of the ultra-high resolution image. This method achieves region-adaptive feature fusion by introducing the UAF mechanism and keeps boundary structure information active during the inference stage by introducing the BACA mechanism, effectively improving the accuracy of semantic segmentation of ultra-high resolution images.

Description

Technical Field

[0001] The embodiments of this application relate to the field of computer vision technology, and in particular to a method for semantic segmentation of ultra-high resolution images based on UAF and BACA. Background Technology

[0002] Ultra-high resolution (UHR) image semantic segmentation is a crucial task in computer vision, with wide applications in remote sensing image analysis, medical image processing, and urban planning. With advancements in imaging technology, satellite and aerial imagery have reached resolutions of tens of millions of pixels, posing a significant challenge to semantic segmentation algorithms. Due to the memory limitations of GPUs (Graphics Processing Units), ultra-high resolution images cannot be directly input into deep neural networks for end-to-end processing. Downsampling the image to accommodate GPU memory constraints, while capturing global semantic context, results in severe loss of spatial details and boundary information. Local cropping at the original resolution limits the receptive field, making it difficult to capture sufficient global contextual information. This contradiction between global semantics and local details is the core challenge of ultra-high resolution image semantic segmentation.

[0003] To resolve the contradiction between global and local contexts, researchers both domestically and internationally have proposed various strategies. Among these, a dual-branch architecture is currently the mainstream solution, where the global branch extracts semantic features, the local branch extracts detailed features, and finally, the features from the two branches are fused. For example, Guo et al. proposed a bidirectional pyramid Transformer method, which achieves global context transfer through a bidirectional inter-block communication mechanism, achieving good results on datasets such as DeepGlobe. Guo et al. also proposed a high-resolution remote sensing semantic segmentation method based on latent diffusion, which selects blocks requiring refinement for diffusion processing through uncertainty estimation.

[0004] However, these methods still have significant shortcomings in two key aspects: feature fusion and boundary restoration.

[0005] The first issue is feature fusion. Most proposed two-branch methods employ spatially uniform or content-independent fusion strategies. Whether through channel modulation, direct feature combination, or context-dependent fusion, none estimate local prediction confidence to adaptively adjust fusion weights. Although some methods introduce uncertainty estimation, it is merely used as a binary block selection criterion rather than continuously guiding the fusion process. This leads to improper handling of high-uncertainty regions and poor fusion results.

[0006] The second issue is boundary restoration. Current methods mainly rely on implicit reinforcement strategies, such as auxiliary supervision that is discarded during the inference phase, and fail to encode persistent structural priors that explicitly guide boundary restoration during deployment. This results in a lack of explicit structural priors, making it difficult to reconstruct complex boundary topologies. Summary of the Invention

[0007] In view of this, embodiments of this application propose an ultra-high resolution image semantic segmentation method based on UAF and BACA. By introducing the UAF (Uncertainty-guided Adaptive Fusion) mechanism, the fusion weights are dynamically adjusted according to the feature reliability of each region to achieve region-adaptive feature fusion. By introducing the BACA (Boundary Aware Cross Attention) mechanism, the boundary prior is encoded into learnable boundary labels, so that the boundary structure information remains active in the inference stage, thereby effectively improving the accuracy of ultra-high resolution image semantic segmentation.

[0008] To achieve the above objectives, embodiments of this application propose a semantic segmentation method for ultra-high resolution images based on UAF and BACA. This method is implemented using a semantic segmentation model composed of a global branch feature encoder, a local branch feature encoder, a UAF module, a BACA module, and a segmentation output module. The method includes: extracting features from a given ultra-high resolution image using the global branch feature encoder and the local branch feature encoder to obtain global semantic features and local detail features; inputting the global semantic features and local detail features into the UAF module for two-stage fusion, first predicting the uncertainty map of the global semantic features and local detail features using an uncertainty estimator, and then performing weighted fusion based on confidence weights calculated according to the uncertainty; then applying selective cross-attention to high uncertainty regions for targeted processing, finally obtaining the UAF fused features; inputting the UAF fused features into the BACA module for boundary structure restoration, first predicting the boundary probability map using a boundary header and generating learnable boundary labels, then achieving bidirectional interaction between features and labels through patch-to-label attention and label-to-patch attention, finally obtaining boundary-aware enhanced features; and inputting the boundary-aware enhanced features into the segmentation output module to obtain the category prediction result for each pixel of the ultra-high resolution image.

[0009] To achieve the above objectives, embodiments of this application also propose an electronic device, including a processor and a memory, wherein the memory stores instructions executable by the processor, and the processor is configured to execute the instructions such that the electronic device can implement the ultra-high resolution image semantic segmentation method based on UAF and BACA as described above.

[0010] To achieve the above objectives, embodiments of this application also propose a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the ultra-high resolution image semantic segmentation method based on UAF and BACA as described above.

[0011] In some embodiments, the backbone network of the global branch feature encoder is a Swing Transformer, and the given ultra-high resolution image is denoted as... , , and They are respectively Height and width, Representing the RGB three channels, a global branch feature encoder is used to extract features from a given ultra-high resolution image to obtain global semantic features, including: Will The input is fed into the global branch feature encoder and downsampled by 4 times to obtain... , ; Will The input is fed into the Swin Transformer for feature extraction, yielding global semantic features that include global contextual information. , , This represents the number of feature channels.

[0012] In some embodiments, the backbone network of the local branch feature encoder is STDC (Short Term Dense Concatenate), which is used to extract features from a given ultra-high resolution image to obtain local detail features, including: Will The input is fed into a local branch feature encoder, and Laplacian pyramid decomposition is applied to extract the first layer of high-frequency residuals. Second layer high frequency residual , , ; Will After upsampling and By splicing, the splicing residual is obtained. , ; Will The input is fed into STDC for feature extraction to obtain the first local detail features. Second local detail features , , .

[0013] In some embodiments, global semantic features and local detail features are input into the UAF module for two-stage fusion. First, an uncertainty estimator predicts the uncertainty map of global semantic features and local detail features, and then performs weighted fusion based on confidence weights calculated according to the uncertainty. Next, selective cross-attention is applied to high uncertainty regions for targeted processing, and finally, the UAF fused features are obtained, including: Will , and The input is fed into the UAF module for two-stage fusion. The input for the first stage of fusion is... and The output is the first final fused feature. The input for the second stage of fusion is and The output is the second final fused feature. This refers to the UAF fusion feature; In the first stage of fusion, an uncertainty estimator is first used to predict... and Their respective uncertainty diagrams and ,based on and Calculate the first confidence weight Based on right and Weighted fusion is performed to obtain the first basic fusion feature. ; exist and Based on this, soft gating is used to identify regions exhibiting high uncertainty in both global and local branches. Choose the one with the highest uncertainty Applying multi-head cross-attention at each position yields the first sparse attention. Based on , and Obtain the first intermediate fusion feature ; based on and Calculate first spatial attention ,right Perform oversuppression scaling to obtain the scaled first spatial attention. ,based on Calculate the first adaptive weights Depth-separable convolution pairs Enhancement is performed to obtain the first enhanced local detail features. Finally based on , , and Perform the final fusion to obtain the first final fusion feature. ; In the second stage of fusion, the uncertainty estimator is first used to predict... and Their respective uncertainty diagrams and ,based on and Calculate the second confidence weight Based on right and Weighted fusion is performed to obtain the second basic fusion feature. ; exist and Based on this, soft gating is used to identify regions exhibiting high uncertainty in both global and local branches. Choose the one with the highest uncertainty Applying multi-head cross-attention at each position yields a second sparse attention. Based on , and Obtain the first intermediate fusion feature ; based on and Calculating second spatial attention ,right By performing anti-oversuppression scaling, we obtain the scaled second spatial attention. ,based on Calculate the second adaptive weights Depth-separable convolution pairs Enhancement is performed to obtain the first enhanced local detail features. Finally based on , , and The final fusion is performed to obtain the UAF fusion features. .

[0014] In some embodiments, This can be expressed by the formula: ; ; , ; ; ; ; ; , ; ; ; , ; ; ; ; ; , ; in, for The spatial mean, For learnable basic weights, For the Sigmoid function, For convolution operations, To compute spatial attention operations, Indicates will Mapping back to the original spatial location, For gating sharpness, For uncertainty threshold, It is a very small positive number used to prevent division by zero, and is generally taken as a value of , For uncertainty estimator, for The spatial mean, Indicates will Map back to the original spatial location.

[0015] In some embodiments, the UAF fused features are input into the BACA module for boundary structure recovery. First, a boundary probability map is predicted using the boundary header, and learnable boundary labels are generated. Then, bidirectional interaction between features and labels is achieved through patch-to-label attention and label-to-patch attention. Finally, boundary-aware enhanced features are obtained, including: Will The data is input into the BACA module for boundary structure recovery. First, the boundary probability map is predicted using the boundary header. After sequentially concatenating 1×1 convolutions and the Softmax function, Projected to Channel obtains soft allocation graph , Based on and Perform boundary-weighted spatial pooling calculations to generate a result containing A set of boundary markers. , Different boundary markers capture different boundary patterns; based on and A patch-to-boundary-label lookup is performed, enabling each location within an image patch to retrieve the boundary structure information stored in the boundary label, resulting in patch-to-label attention-enhanced features. ; based on and A reverse lookup of boundary markers to image patches is performed, enabling the boundary markers to obtain contextual information from the image patches and be updated, resulting in enriched boundary markers. , as a regularization loss monitor; Using feedforward networks Enhancement is performed to obtain boundary awareness enhanced features. .

[0016] In some embodiments, This can be expressed by the formula: ; ; ; ; ; in, It is a feedforward network. As a multi-head attention mechanism, , , These are the query vector, key vector, and value vector, respectively, with subscripts in the lower right corner. For spatial location index, It is a very small positive number, used to prevent division by zero. For 1×1 convolution, For the Softmax function, For the boundary header, This is the Sigmoid function.

[0017] In some embodiments, the total composite loss function used when training the semantic segmentation model is... It consists of segmentation loss, boundary loss, marker localization loss, and marker diversity loss; This can be expressed by the formula: ; ; ; ; in, Main segmentation loss, To assist in segmentation loss, For boundary loss, For labeling localization loss, To label the loss of diversity, , , and They are respectively , , and The corresponding weights and Both use cross-entropy loss with Trimap weighting. For binary cross-entropy loss, The boundary truth values ​​extracted from the segmentation truth values. For Dice's loss, and They are respectively and The corresponding weights To obtain the maximum value, and These are the average distribution values ​​inside and outside the boundary area, respectively. For margin, After mean centering and row normalization , for The identity matrix, To obtain the Frobenius norm, the upper right subscript is used. This is for taking the transpose operation.

[0018] This application proposes a semantic segmentation method for ultra-high resolution images based on UAF and BACA. By introducing the UAF mechanism, the fusion weights are dynamically adjusted according to the feature reliability of each region, achieving dynamic and adaptive fusion of global semantic features and local detail features. This effectively improves the quality and efficiency of feature fusion in ultra-high resolution images and avoids feature redundancy or loss of details. By introducing the BACA mechanism, boundary priors are encoded into learnable boundary labels. Combined with patch-to-label attention and label-to-patch attention mechanisms, boundary structure information remains active during the inference stage, achieving accurate perception and recovery of image boundary structures, and significantly improving the segmentation accuracy and coherence of complex boundaries. The overall design of the semantic segmentation model balances segmentation accuracy and computational efficiency, enabling efficient processing of ultra-high resolution images without significantly increasing computational overhead, breaking through the memory and speed bottlenecks of traditional methods when processing large images. Through the collaborative optimization of multi-dimensional loss functions, the model's generalization ability and robustness are enhanced, enabling it to achieve excellent performance in ultra-high resolution image semantic segmentation tasks in various complex scenarios, and showing good prospects for industrial applications. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies of this application will be briefly introduced below. Obviously, the following drawings are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. The drawings described herein are only used to explain this application and are not intended to limit this application.

[0020] Figure 1 This is a flowchart of an ultra-high resolution image semantic segmentation method based on UAF and BACA provided in one embodiment of this application; Figure 2 This is a structural diagram of a semantic segmentation model provided in one embodiment of this application; Figure 3 This is a structural diagram of a UAF module provided in one embodiment of this application; Figure 4 This is a structural diagram of the BACA module provided in one embodiment of this application; Figure 5 This is a graph showing the performance comparison of this method with other methods on the DeepGlobe dataset, provided in one embodiment of this application. Figure 6 This is a graph showing the performance comparison of this method with other methods on the URUR dataset, provided in one embodiment of this application. Figure 7This is a graph showing the performance comparison of this method with other methods on the ISIC2018 dataset, provided in one embodiment of this application. Figure 8 This is a result diagram of an ablation experiment provided in one embodiment of this application; Figure 9 This is a structural diagram of an electronic device provided in another embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. Those skilled in the art will understand that many technical details have been provided in the embodiments of this application to facilitate better understanding. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments. The division of the following embodiments is for ease of description and should not constitute any limitation on the specific implementation of this application. The following embodiments can be combined with and referenced by each other without contradiction.

[0022] One embodiment of this application proposes an ultra-high resolution image semantic segmentation method based on UAF and BACA. It is implemented based on a semantic segmentation model consisting of a global branch feature encoder, a local branch feature encoder, a UAF module, a BACA module, and a segmentation output module. The implementation details of the ultra-high resolution image semantic segmentation method based on UAF and BACA proposed in this embodiment are described below. The following implementation details are provided for ease of understanding and are not necessary for implementing this solution.

[0023] The specific process of the ultra-high resolution image semantic segmentation method based on UAF and BACA proposed in this embodiment can be described as follows: Figure 1 As shown, the structure of the semantic segmentation model is as follows: Figure 2 As shown, the method includes: S101, using a global branch feature encoder and a local branch feature encoder to extract features from a given ultra-high resolution image, respectively, to obtain global semantic features and local detail features.

[0024] In practice, the first step in semantic segmentation of ultra-high resolution images is to use a dual-branch feature encoder (global branch feature encoder and local branch feature encoder) to extract features from the given ultra-high resolution image. The global branch feature encoder extracts global semantic features, and the local branch feature encoder extracts local detail features.

[0025] In one example, let the given super-resolution image be denoted as , , and They are respectively Height and width, It represents the RGB three-channel.

[0026] The backbone network of the global branch feature encoder is the Swing Transformer. After being input into the global branch feature encoder, it is first downsampled by 4 times to obtain Then The input is fed into the Swin Transformer for feature extraction, yielding global semantic features that include global contextual information. .

[0027] in, , , This is the number of feature channels (typically set to 128). SwinTransformer parameter settings include: embedding dimension set to 96, depth set to... The number of attention heads is set to The window size is set to 7, and the MLP ratio is set to 4.

[0028] The backbone network of the local branch feature encoder is the lightweight STDC. After being input into the local branch feature encoder, the Laplacian pyramid decomposition (a multi-scale decomposition method for extracting high-frequency information of an image through Gaussian blurring and difference operations) is applied to extract the first layer of high-frequency residuals. Second layer high frequency residual Then After upsampling and By splicing, the splicing residual is obtained. Finally The input is fed into STDC for feature extraction to obtain the first local detail features. Second local detail features .

[0029] in, , , , and These are two local detail features at different scales. For local detail features at 1 / 16 resolution, , For local detail features at 1 / 8 resolution, .

[0030] S102, global semantic features and local detail features are input into the UAF module for two-stage fusion. First, the uncertainty map of global semantic features and local detail features is predicted by the uncertainty estimator, and the confidence weight is calculated based on the uncertainty for weighted fusion. Then, selective cross attention is applied to high uncertainty regions for targeted processing, and finally the UAF fused features are obtained.

[0031] In the specific implementation, after obtaining the global semantic features and local detail features, the global semantic features and local detail features can be input into the UAF module for two-stage fusion. First, the uncertainty map of the global semantic features and local detail features is predicted by the uncertainty estimator, and the confidence weight is calculated based on the uncertainty for weighted fusion. Then, selective cross attention is applied to the high uncertainty region for targeted processing, and finally the UAF fused features are obtained.

[0032] In one example, the specific structure of the UAF module is as follows: Figure 3 As shown. Global semantic features First local detail features Second local detail features The input is fed into the UAF module for two-stage fusion. The input for the first stage of fusion is... and The output is the first final fused feature. The input for the second stage of fusion is and The output is the second final fused feature. This refers to the UAF fusion feature.

[0033] In the first stage of fusion, an uncertainty estimator is first used to predict... and Their respective uncertainty diagrams and ,based on and Calculate the first confidence weight Based on right and Weighted fusion is performed to obtain the first basic fusion feature. Traditional fusion methods cannot accurately assess the reliability of features. Therefore, this embodiment selects an uncertainty estimator with a small number of parameters to predict the uncertainty of bi-branch features.

[0034] and The calculation process is as follows: , .

[0035] in, and Uncertainty diagrams and Height and width, The uncertainty estimator consists of a 3×3 depthwise convolution, BN normalization, ReLU activation function, 1×1 pointwise convolution, BN normalization, ReLU activation function, 1×1 pointwise convolution and Sigmoid activation function. The higher the uncertainty value, the lower the reliability of the feature.

[0036] Subsequently, and Based on this, soft gating is used to identify regions exhibiting high uncertainty in both global and local branches. Choose the one with the highest uncertainty Applying multi-head cross-attention at each position yields the first sparse attention. Based on , and Obtain the first intermediate fusion feature This embodiment converts uncertainty into confidence level and uses the confidence level as the fusion weight to achieve spatial adaptive fusion. It effectively solves the problem of poor fusion effect caused by uniform weight in traditional methods. It achieves more sufficient feature interaction through sparse cross attention and overcomes the limitations of simple weighted fusion.

[0037] , , , , The calculation process is as follows: ; ; ; ; ; in, It is a very small positive number, used to prevent division by zero. For the Sigmoid function, Gated sharpness (usually set to 8.0). This is the uncertainty threshold (generally set to 0.5). As a multi-head attention mechanism, , , These are the query vector, key vector, and value vector, respectively. , Indicates the highest level of uncertainty One location, Indicates will Map back to the original spatial location.

[0038] Finally, based on and Calculate first spatial attention ,right Perform anti-oversuppression scaling (i.e.) Rescale to Within the range), thus obtaining the scaled first spatial attention. Based on Calculate the first adaptive weights Depth-separable convolution pairs Enhancement is performed to obtain the first enhanced local detail features. Finally, based on , , and Perform the final fusion to obtain the first final fusion feature. This embodiment enhances local detail features through a spatial attention mechanism, further strengthening the ability to express details of the fused features.

[0039] , , , The calculation process is as follows: ; ; , ; in, To compute spatial attention operations, The learnable base weights (usually set to 3.0). For convolution operations, for The spatial mean (used for normalization).

[0040] In the second stage of fusion, the uncertainty estimator is first used to predict... and Their respective uncertainty diagrams and ,based on and Calculate the second confidence weight Based on right and Weighted fusion is performed to obtain the second basic fusion feature. .

[0041] and The calculation process is as follows: , .

[0042] In obtaining After that, and Based on this, soft gating is used to identify regions exhibiting high uncertainty in both global and local branches. Choose the one with the highest uncertainty Multi-head cross-attention is applied to each of the following locations (default is 20%, i.e., a minimum of 64 patches) to obtain the second sparse attention. Based on , and Obtain the first intermediate fusion feature .

[0043] , , , , The calculation process is as follows: ; ; ; ; ; in, , Indicates the highest level of uncertainty One location, Indicates will Map back to the original spatial location.

[0044] Finally, based on and Calculating second spatial attention ,right By performing anti-oversuppression scaling, we obtain the scaled second spatial attention. ,based on Calculate the second adaptive weights Depth-separable convolution pairs Enhancement is performed to obtain the first enhanced local detail features. Finally based on , , and The final fusion is performed to obtain the UAF fusion features. .

[0045] , , , The calculation process is as follows: ; ; , ; in, for Spatial mean.

[0046] S103, the UAF fused features are input into the BACA module for boundary structure recovery. First, the boundary probability map is predicted through the boundary head and a learnable boundary label is generated. Then, the bidirectional interaction between features and labels is achieved through patch-to-label attention and label-to-patch attention. Finally, the boundary-aware enhanced features are obtained.

[0047] In the specific implementation, after obtaining the UAF fusion features, the UAF fusion features can be input into the BACA module for boundary structure recovery. First, the boundary probability map is predicted through the boundary head and a learnable boundary label is generated. Then, the bidirectional interaction between features and labels is achieved through patch-to-label attention and label-to-patch attention, and finally, the boundary-aware enhanced features are obtained.

[0048] In one example, the specific structure of the BACA module is as follows: Figure 4 As shown, the BACA module provides explicit structural priors for the semantic segmentation model through learnable boundary markers, which can effectively overcome the problems of traditional methods relying on implicit enhancements and insufficient boundary recovery capabilities.

[0049] The BACA module consists of four parts: boundary marker generation, patch-to-marker attention processing, mark-to-patch attention processing, and feedforward network enhancement.

[0050] When generating boundary markers, the UAF fusion features are used. The input is fed into the BACA module for boundary structure recovery. First, a lightweight boundary header (two 3×3 convolutions) is used to predict the boundary probability map. Then, through 1×1 convolution and the Softmax function, Projected to Channels (typically set to 32) are used to obtain a soft allocation map. , Based on and Perform boundary-weighted spatial pooling calculations to generate a result containing A set of boundary markers. , Different boundary markers capture different boundary patterns.

[0051] , , The calculation process is as follows: ; ; ; in, For the boundary header, For the Sigmoid function, For 1×1 convolution, For the Softmax function, It is a very small positive number, used to prevent division by zero, indicated by a subscript. For spatial location index.

[0052] Patch-to-tag attention processing is based on and A patch-to-boundary-label lookup is performed, enabling each location within an image patch to retrieve the boundary structure information stored in the boundary label, resulting in patch-to-label attention-enhanced features. .

[0053] The calculation process is as follows: ; in, As a multi-head attention mechanism, , , These are the query vector, key vector, and value vector, respectively.

[0054] Mark-to-patch attention processing is based on and A reverse lookup of boundary markers to image patches is performed, enabling the boundary markers to obtain contextual information from the image patches and be updated, resulting in enriched boundary markers. , as a regularization loss monitor.

[0055] The calculation process is as follows: ; in, As a multi-head attention mechanism, , , These are the query vector, key vector, and value vector, respectively.

[0056] Feedforward network enhancement is coming soon The input is fed into a feedforward network for enhancement, ultimately yielding boundary-aware enhanced features. . .

[0057] S104: Input the boundary-aware enhancement features into the segmentation output module to obtain the category prediction result for each pixel of the ultra-high resolution image.

[0058] In the specific implementation, after obtaining the boundary-aware enhanced features, these features can be input into the segmentation output module to obtain the category prediction result for each pixel of the ultra-high resolution image. .

[0059] In one example, the segmentation output module consists of a 3×3 convolution, BN normalization, ReLU activation function, and a 1×1 classification convolution. .

[0060] In one example, the total composite loss function used when training a semantic segmentation model. It consists of segmentation loss, boundary loss, label localization loss, and label diversity loss.

[0061] This can be expressed by the formula: ; ; ; ; in, Main segmentation loss, To assist in segmentation loss, For boundary loss, For labeling localization loss, To label the loss of diversity, , , and They are respectively , , and The corresponding weights and Both use cross-entropy loss with Trimap weighting. For binary cross-entropy loss, The boundary truth values ​​extracted from the segmentation truth values. For Dice's loss, and They are respectively and The corresponding weights To obtain the maximum value, and These are the average distribution values ​​inside and outside the boundary area, respectively. For margin, After mean centering and row normalization , for The identity matrix, To obtain the Frobenius norm, the upper right subscript is used. This is for taking the transpose operation.

[0062] In one example , , , .

[0063] In one example .

[0064] The segmentation loss consists of the main segmentation loss. and auxiliary segmentation loss Both are composed of components that use cross-entropy loss with Trimap weighting, giving higher weight to boundary regions.

[0065] Boundary loss consists of two parts: binary cross-entropy loss and Dice loss. Binary cross-entropy loss is primarily used to evaluate the accuracy of boundary prediction pixel-by-pixel, prompting the model to learn the classification features of boundary and non-boundary regions. Dice loss, on the other hand, effectively alleviates the class imbalance problem caused by the small proportion of boundary pixels in ultra-high-resolution images by calculating the overlap between the predicted boundary and the ground truth boundary. The combination of these two methods ensures that the model has both good localization accuracy and maintains boundary coherence when extracting boundaries of complex topologies.

[0066] It is important to note that boundary markers remain active during the inference phase, providing continuous structural guidance, rather than being discarded as in traditional auxiliary supervision methods.

[0067] This embodiment proposes an ultra-high resolution image semantic segmentation method based on UAF and BACA. By introducing the UAF mechanism, the fusion weights are dynamically adjusted according to the feature reliability of each region, achieving dynamic and adaptive fusion of global semantic features and local detail features. This effectively improves the quality and efficiency of ultra-high resolution image feature fusion and avoids feature redundancy or loss of details. By introducing the BACA mechanism, boundary priors are encoded into learnable boundary labels. Combined with patch-to-label attention and label-to-patch attention mechanisms, boundary structure information remains active during the inference stage, achieving accurate perception and recovery of image boundary structures, and significantly improving the segmentation accuracy and coherence of complex boundaries. The overall design of the semantic segmentation model balances segmentation accuracy and computational efficiency, enabling efficient processing of ultra-high resolution images without significantly increasing computational overhead, breaking through the memory and speed bottlenecks of traditional methods when processing large images. Through the collaborative optimization of multi-dimensional composite loss functions, the model's generalization ability and robustness are enhanced, enabling it to achieve excellent performance in ultra-high resolution image semantic segmentation tasks in various complex scenarios, and showing good prospects for industrial applications.

[0068] The steps described above are merely for clarity in describing the technical solution. In actual implementation, they can be combined into one step, or certain steps can be broken down into multiple steps, as long as they involve the same logical relationship, they are all within the scope of protection of this application. Any insignificant modifications or designs added to the algorithm or process, as long as they do not change the core of the algorithm or process, are also within the scope of protection of this application.

[0069] In one embodiment, to verify the effectiveness of the proposed ultra-high resolution image semantic segmentation method based on UAF and BACA (hereinafter referred to as this method or AFBANet), we used three ultra-high resolution image datasets for training and testing, and compared it with other methods. The DeepGlobe dataset is a satellite remote sensing dataset containing 803 images (2448px × 2448px), labeled with 7 landscape categories, with 455, 207, and 142 images used for training, validation, and testing, respectively. The URUR dataset is currently the largest ultra-high resolution segmentation benchmark, containing 3008 images (5120px × 5120px) from 63 cities, labeled with 8 landscape categories. The ISIC2018 dataset is a medical dermoscopy dataset, containing 2596 images for skin lesion segmentation, with the largest image size reaching 6748px × 4499px.

[0070] The performance comparison results of this method with other methods on the DeepGlobe dataset, URUR dataset, and ISIC2018 dataset are as follows: Figure 5 , Figure 6 , Figure 7 As shown in the figure. The evaluation metric we chose for the semantic segmentation task is mIoU (mean Intersection over Union), used to measure the performance of the semantic segmentation model. Our method shows competitive results compared to other methods, achieving 78.1% mIoU on the DeepGlobe dataset, 47.2% mIoU on the URUR dataset, and 83.7% mIoU on the ISIC2018 dataset.

[0071] The effectiveness of this method was verified through analysis of combinations of different components and loss functions. This analysis is as follows: Figure 8 As shown. Component incremental analysis: Based on the baseline model, introducing UAF can effectively repair high uncertainty regions, improving the performance by 0.6%. Introducing boundary supervision and BACA further demonstrates the gain of boundary labeling and structural prior on semantic segmentation, ultimately achieving a 3.6% performance improvement in the semantic segmentation model. Loss function collaborative analysis: Experimental data shows that... and The combined effect is optimal, with their combined contribution exceeding the sum of the improvements from individual loss functions. This demonstrates the collaborative ability of these two loss functions in parameter updates.

[0072] Another embodiment of this application provides an electronic device, such as Figure 9 As shown, it includes: a processor M201 and a memory M202. The memory M202 stores instructions that the processor M201 can execute. When the processor M201 is configured to execute the instructions, the electronic device can implement an ultra-high resolution image semantic segmentation method based on UAF and BACA as described in the above method embodiment.

[0073] The memory and processor are connected via a bus, which includes any number of interconnecting buses and bridges. The bus can connect various circuits of one or more processors and memories, as well as other circuits such as peripherals, voltage regulators, and power management circuits—all well-known in the art and therefore not described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor is transmitted over the wireless medium via an antenna, which also receives and transmits data to the processor.

[0074] The processor manages the bus and handles general processing, providing various functions, including but not limited to timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory, on the other hand, is used to store data used by the processor during operation.

[0075] Another embodiment of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, can implement an ultra-high resolution image semantic segmentation method based on UAF and BACA as described in the above method embodiments.

[0076] That is, those skilled in the art will understand that all or part of the steps in the above method embodiments can be implemented by a program instructing related hardware. The program is stored in a storage medium and includes several instructions to cause a device (such as a microcontroller, chip, etc.) or processor to execute all or part of the steps of the method described in the method embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0077] It will be understood by those skilled in the art that the above embodiments are specific implementations of this application, and various changes in form and detail can be made in practical applications without departing from the spirit and scope of this application. For those skilled in the art, several improvements and modifications can be made without departing from the principles of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A semantic segmentation method for ultra-high resolution images based on UAF and BACA, implemented using a semantic segmentation model composed of a global branch feature encoder, a local branch feature encoder, a UAF module, a BACA module, and a segmentation output module, characterized in that... The method includes: The global branch feature encoder and the local branch feature encoder are used to extract features from the given ultra-high resolution image to obtain global semantic features and local detail features. Global semantic features and local detail features are input into the UAF module for two-stage fusion. First, the uncertainty map of global semantic features and local detail features is predicted by the uncertainty estimator, and the confidence weight is calculated based on the uncertainty for weighted fusion. Then, selective cross attention is applied to high uncertainty regions for targeted processing, and finally the UAF fused features are obtained. The UAF fused features are input into the BACA module for boundary structure recovery. First, the boundary probability map is predicted through the boundary head and a learnable boundary label is generated. Then, the bidirectional interaction between features and labels is achieved through patch-to-label attention and label-to-patch attention. Finally, the boundary-aware enhanced features are obtained. The boundary-aware enhanced features are input into the segmentation output module to obtain the category prediction result for each pixel of the ultra-high resolution image.

2. The ultra-high resolution image semantic segmentation method based on UAF and BACA according to claim 1, characterized in that, The backbone network of the global branch feature encoder is the Swing Transformer, and the given super-resolution image is denoted as... , , and They are respectively Height and width, Representing the RGB three channels, a global branch feature encoder is used to extract features from a given ultra-high resolution image to obtain global semantic features, including: Will The input is fed into the global branch feature encoder and downsampled by 4 times to obtain... , ; Will The input is fed into the Swin Transformer for feature extraction, yielding global semantic features that include global contextual information. , , This represents the number of feature channels.

3. The ultra-high resolution image semantic segmentation method based on UAF and BACA according to claim 2, characterized in that, The backbone network of the Local Branch Feature Encoder (LSDE) is STDC. It extracts features from a given ultra-high resolution image to obtain local detail features, including: Will The input is fed into a local branch feature encoder, and Laplacian pyramid decomposition is applied to extract the first layer of high-frequency residuals. Second layer high frequency residual , , ; Will After upsampling and By splicing, the splicing residual is obtained. , ; Will The input is fed into STDC for feature extraction to obtain the first local detail features. Second local detail features , , .

4. The ultra-high resolution image semantic segmentation method based on UAF and BACA according to claim 3, characterized in that, Global semantic features and local detail features are input into the UAF module for two-stage fusion. First, an uncertainty estimator predicts the uncertainty map of global semantic features and local detail features, and then performs weighted fusion based on confidence weights calculated according to the uncertainty. Next, selective cross-attention is applied to high uncertainty regions for targeted processing. Finally, the UAF fused features are obtained, including: Will , and The input is fed into the UAF module for two-stage fusion. The input for the first stage of fusion is... and The output is the first final fused feature. The input for the second stage of fusion is and The output is the second final fused feature. This refers to the UAF fusion feature; In the first stage of fusion, an uncertainty estimator is first used to predict... and Their respective uncertainty diagrams and ,based on and Calculate the first confidence weight Based on right and Weighted fusion is performed to obtain the first basic fusion feature. ; exist and Based on this, soft gating is used to identify regions exhibiting high uncertainty in both global and local branches. Choose the one with the highest uncertainty Applying multi-head cross-attention at each position yields the first sparse attention. Based on , and Obtain the first intermediate fusion feature ; based on and Calculate first spatial attention ,right Perform oversuppression scaling to obtain the scaled first spatial attention. ,based on Calculate the first adaptive weights Depth-separable convolution pairs Enhancement is performed to obtain the first enhanced local detail features. Finally based on , , and Perform the final fusion to obtain the first final fusion feature. ; In the second stage of fusion, the uncertainty estimator is first used to predict... and Their respective uncertainty diagrams and ,based on and Calculate the second confidence weight Based on right and Weighted fusion is performed to obtain the second basic fusion feature. ; exist and Based on this, soft gating is used to identify regions exhibiting high uncertainty in both global and local branches. Choose the one with the highest uncertainty Applying multi-head cross-attention at each position yields a second sparse attention. Based on , and Obtain the first intermediate fusion feature ; based on and Calculating second spatial attention ,right By performing anti-oversuppression scaling, we obtain the scaled second spatial attention. ,based on Calculate the second adaptive weights Depth-separable convolution pairs Enhancement is performed to obtain the first enhanced local detail features. Finally based on , , and The final fusion is performed to obtain the UAF fusion features. .

5. The ultra-high resolution image semantic segmentation method based on UAF and BACA according to claim 4, characterized in that, This can be expressed by the formula: ； ； ， ； ； ； ； ； ， ； ； ； ， ； ； ； ； ； ， ； in, for The spatial mean, For learnable basic weights, For the Sigmoid function, For convolution operations, To compute spatial attention operations, Indicates will Mapping back to the original spatial location, For gating sharpness, For uncertainty threshold, It is a very small positive number, used to prevent division by zero. For uncertainty estimator, for The spatial mean, Indicates will Map back to the original spatial location.

6. The ultra-high resolution image semantic segmentation method based on UAF and BACA according to claim 4, characterized in that, The UAF fused features are input into the BACA module for boundary structure recovery. First, the boundary probability map is predicted using the boundary header, and learnable boundary labels are generated. Then, bidirectional interaction between features and labels is achieved through patch-to-label attention and label-to-patch attention. Finally, boundary-aware enhanced features are obtained, including: Will The data is input into the BACA module for boundary structure recovery. First, the boundary probability map is predicted using the boundary header. After sequentially concatenating 1×1 convolutions and the Softmax function, Projected to Channel obtains soft allocation graph , Based on and Perform boundary-weighted spatial pooling calculations to generate a result containing A set of boundary markers. , Different boundary markers capture different boundary patterns; based on and A patch-to-boundary-label lookup is performed, enabling each location within an image patch to retrieve the boundary structure information stored in the boundary label, resulting in patch-to-label attention-enhanced features. ; based on and A reverse lookup of boundary markers to image patches is performed, enabling the boundary markers to obtain contextual information from the image patches and be updated, resulting in enriched boundary markers. , as a regularization loss monitor; Using feedforward networks Enhancement is performed to obtain boundary awareness enhanced features. .

7. The ultra-high resolution image semantic segmentation method based on UAF and BACA according to claim 6, characterized in that, This can be expressed by the formula: ； ； ； ； ； in, It is a feedforward network. As a multi-head attention mechanism, , , These are the query vector, key vector, and value vector, respectively, with subscripts in the lower right corner. For spatial location index, It is a very small positive number, used to prevent division by zero. For 1×1 convolution, For the Softmax function, For the boundary header, This is the Sigmoid function.

8. The ultra-high resolution image semantic segmentation method based on UAF and BACA according to claim 6, characterized in that, The total composite loss function used when training the semantic segmentation model It consists of segmentation loss, boundary loss, marker localization loss, and marker diversity loss; This can be expressed by the formula: ； ； ； ； in, Main segmentation loss, To assist in segmentation loss, For boundary loss, For labeling localization loss, To label the loss of diversity, , , and They are respectively , , and The corresponding weights and Both use cross-entropy loss with Trimap weighting. For binary cross-entropy loss, The boundary truth values ​​extracted from the segmentation truth values. For Dice's loss, and They are respectively and The corresponding weights To obtain the maximum value, and These are the average distribution values ​​inside and outside the boundary area, respectively. For margin, After mean centering and row normalization , for The identity matrix, To obtain the Frobenius norm, the upper right subscript is used. This is for taking the transpose operation.

9. An electronic device, characterized in that, include: The processor and memory, wherein the memory stores instructions executable by the processor, and the processor is configured to, when executing the instructions, enable the electronic device to implement a super-resolution image semantic segmentation method based on UAF and BACA as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it can implement a method for ultra-high resolution image semantic segmentation based on UAF and BACA as described in any one of claims 1 to 8.

Images