Image segmentation method based on global sparse attention and cross-scale interaction

Abstract

The invention discloses an image segmentation method based on global sparse attention and cross-scale interaction. The method comprises the following steps: constructing and preprocessing image data and a mask thereof; generating a multi-scale label according to the mask; inputting the image data and the multi-scale label into an end-to-end segmentation network; coding branches of the end-to-end segmentation network are combined with global sparse self-attention for enhancing long-range dependency expression to extract multi-scale features; a decoding branch of the end-to-end segmentation network is combined with global sparse cross attention to generate a multi-scale prediction and main segmentation result; and performing multi-scale supervision and training optimization according to the multi-scale prediction and the main segmentation result. According to the method, economic global sparse self-attention is introduced into an encoder layer, so that efficient long-range dependence modeling is realized; designing a cross-scale selective interaction mechanism, and accurately aligning and injecting multi-scale features; the system integrates sub-pixel up-sampling, channel re-calibration and multi-scale depth supervision, and comprehensively improves the segmentation performance and training stability of a complex structure and a small target while controlling the calculation overhead.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning, and in particular to an image segmentation method based on global sparse attention and cross-scale interaction. Background Technology

[0002] Image segmentation is a fundamental task in computer vision, aiming to assign semantic or instance labels to each pixel in an image. It is widely used in fields such as medical imaging, autonomous driving, and industrial inspection. Current mainstream methods are based on encoder-decoder architectures, extracting features through convolution and downsampling, and then restoring details by combining skip connections. However, they generally suffer from problems such as insufficient modeling of long-range dependencies and inadequate cross-scale fusion.

[0003] Current image segmentation techniques primarily employ an encoder-decoder architecture, exemplified by U-Net, combined with convolutional neural networks (CNNs) for feature extraction and reconstruction. The encoder aggregates high-level semantic information through progressive downsampling, while the decoder fuses shallow location information with deep semantics via skip connections, gradually restoring spatial resolution. To alleviate the locality limitation of convolution, some methods introduce self-attention mechanisms to enhance global modeling; however, directly applying these mechanisms to high-resolution features incurs enormous computational overhead, hindering practical application. Therefore, improved solutions often employ sparse attention, pyramid key-value downsampling, or block-based computation to balance performance and efficiency. Regarding upsampling, traditional transposed convolutions are prone to checkerboard artifacts, while bilinear interpolation loses details. In recent years, pixel-shuffle convolution has gradually become the preferred choice due to its ability to balance resolution enhancement and artifact suppression. To address channel redundancy, channel attention mechanisms such as Squeeze-and-Excitation (SE) enhance effective features and suppress noise by recalibrating channel weights. During training, multi-scale deep supervision accelerates convergence and improves stability by imposing constraints on the decoding layer at each level. Combined with gradient pruning and adaptive learning rate scheduling, it further optimizes the training process.

[0004] Existing technologies suffer from three key shortcomings: First, long-range dependency modeling is inefficient. The locality of convolution leads to fragmented segmentation of elongated structures and cross-regional targets. While global self-attention can compensate for this deficiency, its computational cost is high in multi-scale, high-resolution scenes, making widespread deployment difficult. Second, the cross-scale feature fusion mechanism is crude. Skip connections directly concatenate encoder and decoder features without considering their semantic and statistical distribution differences, resulting in information conflicts and loss of detail. It lacks a lightweight and accurate cross-scale interaction and injection mechanism. Third, upsampling quality, channel optimization, and training supervision fail to coordinate systematically. Although subpixel convolution reduces artifacts, it is not effectively combined with channel recalibration and deep supervision, limiting further improvements in boundary continuity and small target segmentation accuracy. Summary of the Invention

[0005] In view of the above-mentioned deficiencies of the prior art, the present invention provides an image segmentation method based on global sparse attention and cross-scale interaction. It introduces an economical global sparse self-attention in the encoder layer to achieve efficient long-range dependency modeling; designs a cross-scale selective interaction mechanism to accurately align and inject multi-scale features; and integrates sub-pixel upsampling, channel recalibration and multi-scale depth supervision to comprehensively improve the segmentation performance and training stability of complex structures and small targets while controlling computational overhead.

[0006] To achieve the above objectives, the following technical solution is adopted:

[0007] Image segmentation methods based on global sparse attention and cross-scale interaction include the following steps:

[0008] S1. Construct and preprocess image data and its mask; generate multi-scale labels based on the mask;

[0009] S2. Input the image data and the multi-scale labels into an end-to-end segmentation network; the encoding branch of the end-to-end segmentation network is combined with global sparse self-attention to enhance long-range dependency representation to extract multi-scale features; the decoding branch of the end-to-end segmentation network is combined with global sparse cross-attention to generate multi-scale prediction and main segmentation results.

[0010] S3. Perform multi-scale supervision and training optimization based on the multi-scale prediction and the main segmentation results.

[0011] Preferably, the preprocessing includes:

[0012] The original color image is subjected to intensity cropping and normalization; the mask is aligned to the same size using nearest neighbor interpolation; the grayscale image or multimodal image is linearly normalized and aligned for each channel.

[0013] Remove blurry, occluded, abnormally exposed, or obviously missing image samples; perform CLAHE or light gamma correction on image data; unify color space and record imaging protocol metadata for cross-device / cross-domain data.

[0014] Preferably, S1 also includes generating boundary auxiliary labels for enhancing fine-grained supervision and contour continuity constraints.

[0015] Preferably, S2 includes:

[0016] S21. After performing consistency enhancement on the image data and the mask, the multi-scale label is generated; the image data and the multi-scale label are input into the end-to-end segmentation network;

[0017] S22, The encoding branch is an encoder with ResNet34 as the backbone; the image data and the multi-scale labels are input into the encoder; global sparse self-attention is introduced after the multi-scale features of each layer except the highest layer features; a transformer block is set in the highest layer features to perform global feature reshaping;

[0018] S23. Introduce global sparse cross-attention at each decoding scale of the decoding branch; the decoding branch recovers the resolution step by step from top to bottom and fuses shallow texture information to generate multi-scale prediction.

[0019] Preferably, S22 includes:

[0020] The key / value branches of the attention mechanism are spatially sparsely downsampled according to the scale ratio to establish long-range dependencies on a low-density grid; multi-frequency locations are superimposed on two-dimensional features; the output of the attention branch is back-injected into the main convolutional branch in the form of residuals, and the injection intensity is adaptively controlled by learnable coefficients.

[0021] As a preferred option, S23 also includes:

[0022] The first decoding layer first performs a light upsampling of the highest-level features and stabilizes the features with a decoding refinement block, and then sets a transformer block again on the highest-level features to alleviate semantic breaks in the early stage of upsampling; the decoding branch uses a pixel rearrangement upsampling combined with a skip connection stepwise strategy to restore resolution.

[0023] Preferably, S23 further includes: the decoding refinement block includes two layers of batch-normalized 3×3 residual convolution, SE channel recalibration and 3×3 shaping convolution; it also includes a strip convolution used to expand the anisotropic receptive field and located between the 3×3 residual convolution and the 3×3 shaping convolution.

[0024] Preferably, in S23, the strip convolution order is (1×3), (3×1), (3×3); the first group of strip convolutions is set with a stride of (2,1) or (1,2), and alignment compensation is performed in the subsequent pixel rearrangement upsampling and 1×1 convolution.

[0025] In pixel reordering upsampling, the spatial resolution is magnified by 2 times. Before pixel reordering, a 3×3 convolution is used to compress the stitched channels to 1 / 4, and then a 3×3 convolution is used to expand them to 4 times the target number of channels. After reordering, ReLU activation and a 3×3 shaping convolution are used to smooth local artifacts.

[0026] SE sub-blocks with a compression ratio of 16 are used; SE weights are fused with the main decoding branch in a residual manner.

[0027] Preferably, in S3, the main segmentation result and the multi-scale prediction are optimized by deep supervision using a combination of label-smoothed cross-entropy and Dice loss; the loss weights of each multi-scale prediction decrease progressively; and weighted supervision is applied to the output of the decoder at each scale to generate the total loss.

[0028] Compared with the prior art, the beneficial effects of the present invention are reflected in:

[0029] To address the shortcomings of existing segmentation models, such as insufficient long-range dependency modeling, inadequate cross-scale fusion, and upsampling artifacts, this invention proposes an end-to-end segmentation framework based on global sparse attention and cross-scale interaction. This framework fully leverages the complementarity between global semantics at the encoder end and fine-grained texture at the decoder end, and organically combines Global Sparse Self-Attention (GSA) and Global Sparse Cross-Attention (GSCA) to significantly improve segmentation performance for complex boundaries. To effectively filter redundant features and suppress artifacts, a collaborative strategy of pixel rearrangement upsampling + SE decoding blocks is proposed. Furthermore, a transformer renormalization module is introduced at the bottleneck and the first decoding layer to unify the semantic space. Combined with multi-scale deep supervision and stable training strategies, this effectively alleviates overfitting and improves model robustness and deployability. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the method flow of Embodiment 1 of the present invention;

[0031] Figure 2 This is a flowchart of the data preprocessing process in Embodiment 1 of the present invention;

[0032] Figure 3 This is a schematic diagram of the overall network structure of Embodiment 1 of the present invention;

[0033] Figure 4 This is a schematic diagram of the GSA structure and sparse sampling in Embodiment 1 of the present invention;

[0034] Figure 5 This is a schematic diagram of the cross-scale interaction of GSCA in Embodiment 1 of the present invention;

[0035] Figure 6 This is a schematic diagram of the PSU pixel rearrangement upsampling and decoding refinement process in Embodiment 1 of the present invention;

[0036] Figure 7 This is a schematic diagram of the multi-scale supervision head and loss summary in Embodiment 1 of the present invention. Detailed Implementation

[0037] To make the technical means, inventive features, objectives, and effects of the invention readily understandable, the invention is further described below with reference to specific illustrations. However, the invention is not limited to the embodiments described below.

[0038] It should be noted that the structures, proportions, sizes, etc., illustrated in the accompanying drawings of this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0039] To improve the accuracy and robustness of pixel-level segmentation in complex scenes, effectively combine global semantics at the encoding end with fine-grained boundary information at the decoding end, and reduce redundant feature interactions without significantly increasing computational load, this invention aims to provide an end-to-end segmentation framework based on global sparse attention and cross-scale interaction for image segmentation. This framework consists of five key components: a Global Sparse Self-Attention (GSA) module deployed in each encoding layer, a Transformer Reorganization (T-Block) module for bottlenecks and the first decoding layer, an Upsampling Unit (PSU) employing pixel rearrangement, a Decoding Block with Channel Recalibration (SE-Decoder), and a Global Sparse Cross-Attention Block (GSCA) for decoding-encoding alignment. Specifically, GSA models long-range dependencies of multi-scale features and fuses them with learnable coefficients and convolutional branch residuals; T-Block unifies the semantic space at higher levels and alleviates semantic breaks in the early stages of upsampling; PSU and SE-Decoder enhance boundary representation while progressively restoring resolution; and GSCA uses decoded features as queries and encoded features as keys / values ​​to achieve selective cross-scale injection, suppressing redundancy and semantic drift. Finally, the segmentation results are output by setting up a supervisory head (MS-Head) at multiple scales, and joint training is performed through combined loss to achieve simultaneous improvement in segmentation accuracy and model robustness.

[0040] Example 1:

[0041] like Figure 1 The image segmentation method shown includes the following steps: Global sparse attention and cross-scale interaction-based approach.

[0042] S1. Construct and preprocess image data and its mask; generate multi-scale labels based on the mask;

[0043] Datasets are constructed and organized based on target segmentation tasks (medical images, remote sensing, industrial defects, or semantic segmentation of natural scenes). Each input image and its corresponding pixel-level labeled mask are processed synchronously to ensure consistency in scale, geometry, and semantics. Figure 2 As shown, the preprocessing includes:

[0044] Normalization and size registration: First, the original color image is intensity-cropped and normalized (mean subtraction and variance division by channel) to uniformly adjust the image to 352×352; the mask is aligned to the same size using nearest neighbor interpolation, ensuring that the category boundaries are not smoothed. For grayscale or multimodal images (such as MRI sequences and remote sensing multispectral images), linear normalization and alignment are performed separately for each channel.

[0045] Quality control and optional enhancements: Remove severely blurred, occluded, abnormally exposed, or obviously missing labeled samples; to improve the recognition of weak contrast and narrow structures, optional CLAHE (Contrast Limited Adaptive Histogram Equalization) or mild gamma correction can be performed; for cross-device / cross-domain data, the color space is unified and imaging protocol metadata is recorded to facilitate subsequent domain evaluation.

[0046] Data augmentation and partitioning: Consistent scaling and random cropping (size aligned to multiples of 32) are performed using scale sets {0.75, 1.0, 1.25}, along with random horizontal flipping, ±10° small-angle rotation, and slight color perturbations (brightness / contrast / saturation). All geometric transformations are performed synchronously on the image and mask. Training / validation / test set partitioning and caching are completed, and class proportions and sample sources are recorded to facilitate class imbalance and cross-domain robustness handling.

[0047] Multi-scale labels of 176×176, 88×88, 44×44, 22×22, and 11×11 are generated based on high-resolution masks and aligned step-by-step with the output of each decoding layer. Optional boundary auxiliary labels (based on morphology or Sobel / Laplacian) can be generated to enhance fine-grained supervision and contour continuity constraints.

[0048] S2, such as Figure 3 As shown, image data and multi-scale labels are input into an end-to-end segmentation network; the encoding branch of the end-to-end segmentation network is combined with global sparse self-attention to enhance long-range dependency expressions to extract multi-scale features; the decoding branch of the end-to-end segmentation network is combined with global sparse cross-attention to generate multi-scale predictions and main segmentation results.

[0049] The preprocessed image and mask from S1 are used to generate multi-scale labels after multi-scale consistency enhancement: an encoder with ResNet34 as the backbone extracts multi-scale features at different levels step by step, and global sparse self-attention is introduced in each layer to enhance long-range dependency expression, such as... Figure 4As shown, a transformer block is introduced into the highest-level features for global feature reshaping; in the decoding stage, pixel rearrangement upsampling and skip connections are used to restore the resolution step by step, while residual convolutional blocks and channel attention (SE) are combined to enhance boundary information and fine-grained feature expression, so as to obtain multi-scale prediction (out1~out5) and main segmentation results.

[0050] S21. After performing consistency enhancement on the image data and the mask, multi-scale labels are generated; the image data and multi-scale labels are input into the end-to-end segmentation network.

[0051] To balance scale robustness and boundary accuracy, consistency enhancement and pyramid label generation are performed on the image-mask pair preprocessed by S1. Each sampling outputs a fixed 352×352 image and a five-level label aligned with it for subsequent multi-scale supervision.

[0052] S22. The coding branch extracts multi-scale semantic features from bottom to top;

[0053] The encoding branch is an encoder with ResNet34 as the backbone; image data and multi-scale labels are input into the encoder; global sparse self-attention is introduced after the multi-scale features of each layer except the highest layer features; a transformer block is set in the highest layer features to perform global feature reshaping;

[0054] In this embodiment, the input image data first enters the encoder with ResNet34 as the backbone, and sequentially obtains multi-scale features e1 (64×176×176), e2 (64×88×88), e3 (128×44×44), e4 (256×22×22) and bottleneck feature e5 (512×11×11).

[0055] Global sparse self-attention is introduced after the outputs of layers e1 to e4: its key / value branches are spatially sparsely downsampled according to the scale ratio (in this embodiment, the scale ratios are: e1=4, e2=4, e3=2, e4=1, and the attention and feedforward branches are configured with Dropout=0.1), establishing long-range dependencies on a low-density grid to reduce computational cost; multi-frequency positional cues are superimposed on two-dimensional features to enhance spatial coordinate awareness. The output of the attention branch is back-injected into the main convolutional branch in the form of residuals, and the injection intensity is adaptively controlled by learnable coefficients to avoid semantic drift or shallow noise amplification caused by excessive injection.

[0056] A transformer block (T-Block) is set at the highest-level feature e5 position to unify the high-level semantic space, and then the decoding stage begins.

[0057] S23. The decoding branch restores the resolution step by step from top to bottom and fuses shallow texture information, and finally gives the prediction results (out1 to out5) at multiple scales.

[0058] The first decoding layer performs a light upsampling of e5 and stabilizes the features using a decoding refinement block. Then, a transformer block (T-Block) is inserted again at this layer to alleviate semantic fragmentation in the initial upsampling phase. Step-by-step decoding employs a fusion of pixel-shuffle upsampling and skip connections.

[0059] d4=Up(d5,e4)→DecoderBlock(256);

[0060] d3=Up(d4,e3)→DecoderBlock(128);

[0061] d2=Up(d3,e2)→DecoderBlock(64);

[0062] d1=Up(d2,e1)→DecoderBlock(64).

[0063] The Up (PSU) unit achieves spatial ×2 magnification and reduces checkerboard artifacts in the order of "stitching → small convolution to 1 / 4 channel → small convolution to 4× target channel → PixelShuffle(2) → ReLU"; the DecoderBlock (SE-Decoder) consists of two layers of 3×3 residual convolution with batch normalization, SE channel recalibration (r=16) and 3×3 shaping convolution, which are used to enhance boundaries and details. For slender targets, (1×3, 3×1) strip convolutions can be inserted between the two 3×3 layers to expand the anisotropic receptive field and improve continuity.

[0064] The decoding refinement and upsampling strategies are as follows: Strip convolution enhancement: In the DecoderBlock, strip convolutions are inserted between two 3×3 residual convolutions to model anisotropic structures, with the preferred order being (1×3)→(3×1)→(3×3); where strip convolutions are used to extend the horizontal / vertical receptive field, and the padding method of (1×3) and (3×1) maintains the output resolution unchanged. Optional stride setting: To quickly aggregate long-range connections of fine line structures in shallow layers, a stride of (2,1) or (1,2) (selected according to the target principal direction) can be set on the first set of strip convolutions, and alignment compensation is performed in the subsequent PixelShuffle upsampling and 1×1 convolutions. Upsampling and Channel Shaping: The upsampling unit uses PixelShuffle (rearrangement factor 2). Before pixel rearrangement, a 3×3 convolution is used to compress the stitched channels to 1 / 4, and then a 3×3 convolution is used to expand them to 4 times the target number of channels. After rearrangement, ReLU activation and a 3×3 shaping convolution are applied to smooth local artifacts. Channel Recalibration: The SE sub-block compression ratio is preferably r=16 to improve the response of significant channels and suppress redundant channels. The SE weights and the decoding main branch are fused in a residual manner to stabilize training and enhance boundary continuity.

[0065] To achieve alignment and selective injection of encoded semantics and decoding details, such as Figure 5 As shown, global sparse cross-attention is introduced at each decoding scale: using the current decoding feature as the query (Q) and the encoded features aligned to the same resolution as the key / value pair (K / V), sparse downsampling is performed on the K / V branch (suggested: d5=8, d4=18, d3=18, d2=18, d1=36) to efficiently compute cross-scale correlations in the low-resolution domain. The fusion result, after linear mapping and normalization, is used to back-inject residuals into the main decoding branch. The injection strength is adjusted by learnable coefficients to avoid overwriting boundary details already learned in this layer. Finally, after d5→d1, the outputs out5→out1 are generated through the side branch prediction heads, as shown below. Figure 6 As shown, out1 (352×352) is the main segmentation result.

[0066] S3, Multi-scale Supervision and Training Optimization

[0067] Multi-scale predictions are jointly trained with corresponding pyramid labels: Global sparse cross-attention is used in each decoding layer to achieve interactive fusion of encoded and decoded features at corresponding scales, and learnable coefficients are used to constrain the injection strength, such as... Figure 7 As shown, the main output and auxiliary output are respectively optimized using a label smoothing strategy-based cross-entropy and Dice (or IoU) combined loss for deep supervision, thereby improving segmentation accuracy and model robustness. Specifically, this includes:

[0068] Loss function and supervision strategy: The main output (main segmentation result) uses a combination of label-smoothed cross-entropy and Dice / IoU to balance class discrimination and stability of small objects / boundaries; the auxiliary output (multi-scale prediction) uses the same combination loss but with progressively decreasing weights to accelerate convergence and constrain early decoding layers. For datasets with highly imbalanced class distributions, class proportions can be recorded at the data layer and class weights can be introduced into the loss; if boundary auxiliary labels are used, a lightweight boundary loss can be added to the main output to further improve contour sharpness.

[0069] The single-scale loss uses a combination of label-smoothed cross-entropy and Dice:

[0070]

[0071] The single-scale loss corresponding to the i-th scale is a combination of cross-entropy and Dice.

[0072] λ∈[0,1]: The trade-off coefficient between the two losses. The larger λ is, the more emphasis is placed on cross-entropy; the larger 1-λ is, the more emphasis is placed on Dice (usually adjusted between 0.3 and 0.7).

[0073] Label smoothing cross-entropy loss. Its core is to replace strict one-hot labels with a "smoothed target distribution" to reduce overfitting and overconfidence.

[0074] O i : Network prediction (logits or probability) at the i-th scale.

[0075] Y i : Truth labels aligned with the O_i scale.

[0076] Label smoothing cross-entropy: Smooth the true labels from {0,1} to {ε,1-ε}, then...

[0077]

[0078] L CE-sm (O,Y): Label smoothing cross-entropy loss (for binary classification / foreground-background case), calculated pixel by pixel / voxel and then averaged or summed in space and batch.

[0079] O: Logistic odds of the network output.

[0080] Y: Binary truth label, taking values ​​in {0,1}; during label smoothing, 1 is replaced with 1-ε, and 0 is replaced with ε.

[0081] ε∈[0,0.5): Label smoothing coefficient; when ε=0, it degenerates into standard binary cross-entropy.

[0082] The probability of predicting a prospect is given, where σ(·) is the Sigmoid function σ(x) = 1 / (1+e^{-x}).

[0083] Dice loss (avoiding under-segmentation of small targets):

[0084]

[0085] L Dice (O,Y): Soft Dice loss, equal to 1 - Dice coefficient, the smaller the better.

[0086] The foreground probability map is obtained by passing logits through Sigmoid, with values ​​∈ [0,1].

[0087] Y: Binary truth mask, with values ​​in {0,1}.

[0088] The summation of element-wise products in the spatial dimension (either by sample or by batch) represents the total number of "soft" true positives. The L1 norm of Y, which is the sum of probabilities of all pixels / voxels, represents the predicted foreground volume. |Y|1: The L1 norm of Y, which is the number of foreground pixels / voxels, represents the ground truth foreground volume.

[0089] ε: A very small smoothing constant (e.g., 1e-6) to prevent the denominator from being 0 and to improve numerical stability.

[0090] Multi-scale supervision (main and auxiliary combined) is as follows:

[0091] Weighted supervision is applied to the output {O} at each decoding scale (from low resolution to high resolution, with the main output being O), and the total loss is...

[0092]

[0093] L ds Total loss (deep supervision / multi-scale weighted loss).

[0094] N: The number of scales (branches) output by the decoder.

[0095] L i The single-scale loss corresponding to the i-th scale is O i Calculated with annotation

[0096] w i The weights of the loss at the i-th scale are non-negative and sum to 1; they decrease in importance as follows: w1>w2>…>w N .

[0097] Optimization and Regularization: AdamW or similar adaptive optimizers are used, along with weight decay and learning rate scheduling (linear warming + cosine annealing); attention and feedforward branches are set with appropriate random deactivation to reduce overfitting; mixed precision and gradient clipping are enabled during training to stabilize values. An early stopping strategy is used on the validation set, and exponential moving average is applied at the end to improve generalization.

[0098] Inference and Deployment: The default output is out1; for precision-sensitive scenarios, horizontal flipping / multi-scale testing can be enabled for lightweight integration to improve robustness; for latency-sensitive edge devices, shallow GSA can be disabled or sparse sampling intensity can be reduced to retain bottleneck and mid-to-high-level global modeling capabilities. The model can be exported to a general static graph format and supports half-precision or integer quantization.

[0099] Tables 1, 2, and 3 show the comparison results of each method on the public polyp segmentation datasets Kvasir-SEG, CVC-300, and ColonDB, respectively. It can be seen that the proposed method is competitive in terms of metrics such as Dice and IoUd, and performs stably, especially in boundary details and small target scenarios.

[0100] Table 1: Comparison results on the Kvasir-SEG dataset

[0101]

[0102] Table 2: Comparison results on the CVC-300 dataset

[0103] Table 3: Comparison results on the ColonDB dataset

[0104]

Claims

1. An image segmentation method based on global sparse attention and cross-scale interaction, characterized in that, The method comprises the following steps: S1, constructing and preprocessing image data and its mask; generating multi-scale labels according to the mask; S2, inputting the image data and the multi-scale labels into an end-to-end segmentation network; the encoding branch of the end-to-end segmentation network extracts multi-scale features by combining global sparse self-attention for enhancing long-range dependency expression; the decoding branch of the end-to-end segmentation network generates multi-scale predictions and main segmentation results by combining global sparse cross-attention; S3, performing multi-scale supervision and training optimization according to the multi-scale predictions and the main segmentation results.

2. The image segmentation method based on global sparse attention and cross-scale interaction according to claim 1, characterized in that, The preprocessing comprises: Intensity clipping and normalization are performed on the original color image; the mask is aligned to the same size by nearest neighbor interpolation; linear normalization and alignment are performed on the gray image or multi-modal image according to the channel; Image samples with blur, occlusion, abnormal exposure or obvious annotation missing are removed; CLAHE or slight gamma correction is performed on the image data; for cross-device / cross-domain data, the color space is unified and the imaging protocol meta information is recorded.

3. The image segmentation method based on global sparse attention and cross-scale interaction according to claim 1, characterized in that, S1 further comprises generating boundary auxiliary labels for strengthening fine-grained supervision and contour continuity constraints.

4. The image segmentation method based on global sparse attention and cross-scale interaction according to claim 1, wherein S2 The method comprises: S21, generating the multi-scale labels after performing consistency enhancement on the image data and the mask; inputting the image data and the multi-scale labels into the end-to-end segmentation network; S22, the encoding branch is an encoder with ResNet34 as the backbone; the image data and the multi-scale labels are input into the encoder; global sparse self-attention is introduced after each layer of multi-scale features except the highest layer of features; a transformer block is set at the highest layer of features to reorganize global features; S23, global sparse cross-attention is introduced at each decoding scale of the decoding branch; the decoding branch recovers resolution and fuses shallow texture information step by step from top to bottom, generating multi-scale predictions.

5. The image segmentation method based on global sparse attention and cross-scale interaction according to claim 4, characterized in that, S22 comprises: The key / value branch of the attention mechanism is spatially sparsely down-sampled according to the scale ratio, establishing long-range dependencies on a low-density grid; multi-frequency positions are superimposed on two-dimensional features; the output of the attention branch is fed back to the main convolution branch in a residual form, and the injection strength is adaptively controlled by a learnable coefficient.

6. The image segmentation method based on global sparse attention and cross-scale interaction according to claim 4, characterized in that, S23 further comprises: The first decoding layer first performs light up-sampling on the highest layer of features and uses a decoding refinement block to stabilize the features, and then sets a transformer block again on the highest layer of features to relieve semantic fragmentation in the early stage of up-sampling; the decoding branch recovers resolution by using pixel rearrangement up-sampling combined with a skip connection step-by-step strategy.

7. The image segmentation method based on global sparse attention and cross-scale interaction according to claim 6, characterized in that, S23 further comprises: the decoding refinement block comprises two layers of 3×3 residual convolution with batch normalization, SE channel re-labeling and 3×3 reshaping convolution; and further comprises a strip convolution for expanding anisotropic receptive field and arranged between the 3×3 residual convolution and the 3×3 reshaping convolution.

8. The image segmentation method based on global sparse attention and cross-scale interaction according to claim 7, characterized in that, In S23, the strip convolution is sequentially arranged in the order of (1×3), (3×1) and (3×3); a step (2, 1) or (1, 2) is set on the first group of strip convolutions, and alignment compensation is performed in the pixel rearrangement up-sampling and 1×1 convolution immediately following the first group of strip convolutions; In the pixel rearrangement up-sampling, the spatial resolution is enlarged by 2 times, and before the pixel rearrangement, the channels after splicing are compressed to 1 / 4 by 3*3 convolution, and then expanded to 4 times of the target channel number by 3*3 convolution; after the rearrangement, ReLU activation and 3*3 reshaping convolution are used to smooth local artifacts; The SE sub-block with a compression ratio of 16 is used; the SE weight is combined with the decoding main branch in a residual manner.

9. The image segmentation method based on global sparse attention and cross-scale interaction according to claim 1, characterized in that, In S3, the main segmentation result and the multi-scale prediction are supervised and optimized by a combined loss of cross-entropy and Dice with label smoothing; the loss weight of each multi-scale prediction is gradually reduced; and the total loss is generated by applying weighted supervision on each scale output of the decoder.

Images