TransUNet-based medical image segmentation method

Abstract

The invention discloses a medical image segmentation method based on TransUNet, and belongs to the technical field of medical image segmentation. The method comprises the steps of firstly performing data preprocessing on an original image to obtain preprocessed data; and a DCA attention module is used at a jump joint, so that the problem that a semantic gap exists between characteristics of an encoder and a decoder due to the fact that a simple jump connection scheme is difficult to capture a multi-scale context is solved. The semantic difference leads to redundancy between low-level and high-level features, and finally the segmentation performance is limited. Secondly, a multi-scale boundary sensing module is added to the top layer of the encoder, so that the neural network can better segment the boundary of the target image in the training process; and inputting the preprocessed data into the improved TransUNet model to train the medical image, and outputting an image segmentation result.

Description

Technical Field

[0001] This invention belongs to the field of medical image segmentation, specifically relating to a medical image segmentation method based on the deep learning model TransUNet, which is applicable to image analysis in medical diagnostic auxiliary systems. Background Technology

[0002] Abnormal areas in brain images often exhibit characteristics such as sudden onset, rapid progression, and complex structure. Timely identification and subsequent processing are crucial for safeguarding individual health. Currently, brain imaging analysis primarily relies on medical imaging techniques such as CT or MRI. These techniques can directly reflect changes in brain tissue structure and are important bases for determining the location, extent, and characteristics of abnormalities.

[0003] Current mainstream image segmentation models, such as U-Net and its variants, typically employ a symmetrical encoder-decoder structure and use skip connections to pass shallow features to the decoder to mitigate spatial information loss. However, due to the differences in representation levels between low-level features, such as edges and textures, and high-level semantic features, direct fusion may lead to semantic inconsistencies, thus affecting the final segmentation accuracy. Although the TransUNet model combines the global modeling capabilities of the Transformer with the local feature extraction advantages of U-Net to some extent, it still suffers from insufficient detail recognition and an imperfect overall optimization strategy when processing complex brain structure images. Therefore, there is an urgent need for an improved image segmentation method that can more effectively fuse multi-level features, improve boundary awareness and sensitivity to small-scale regions, thereby achieving more accurate and robust automatic segmentation results. Summary of the Invention

[0004] The purpose of this invention is to provide a medical image segmentation method based on an improved TransUNet network to improve the segmentation accuracy of target regions in images. This method is based on the TransUNet network architecture and embeds a dual cross-attention module (DCA) in the encoder and skip connection parts to enhance the model's feature extraction capabilities. This effectively alleviates problems such as insufficient information utilization and inadequate feature fusion in medical images, achieving more comprehensive and detailed modeling of image information, thereby obtaining more convincing and accurate segmentation results.

[0005] This invention constructs a medical image segmentation network model that integrates the dual cross-attention mechanism DCA and the Transformer architecture, and completes model training. The image segmentation network model includes an encoder module, a Transformer module, a decoder module, and a segmentation output module. The specific structure is as follows:

[0006] The encoder module employs a multi-level convolutional neural network (CNN) structure to extract features from the input image layer by layer, outputting multi-scale feature maps F1, F2, and F3, corresponding to different feature levels such as the detail layer, structural layer, and semantic layer, respectively. These feature maps F1, F2, and F3 are sequentially input into the DCA attention mechanism module for context enhancement processing to improve feature representation capabilities. F3, as a highly semantically expressive feature, is further input into a global modeling sub-network composed of a linear mapping module and a Transformer module to extract long-range dependencies and global context, generating a hidden feature map F4. The linear mapping module converts the 2D feature map F3 into a 1D token sequence suitable for Transformer processing. This process includes the following steps: First, F3 is input into the DCA module for channel and spatial dimension attention enhancement. Then, the linear projection module flattens and compresses the vectors, generating P token vectors of length C (where P is the number of tokens and C is the number of channels). A positional encoding module is introduced to inject spatial location information, forming a position-aware input sequence. This sequence is fed into a multi-layer Transformer encoder, each layer consisting of a multi-head self-attention network and a feedforward network, used to model global dependencies and extract deep features. Finally, the dimensionality reconstruction module maps the sequence back to a 2D spatial structure, generating a hidden feature map F4, which is then fed into the decoder module. The decoder module consists of multiple cascaded upsampling units, progressively restoring the spatial resolution of the feature map. Each upsampling module receives the feature map from the previous stage and the feature map from the corresponding layer of the encoder, achieving multi-layer information fusion through a skip connection mechanism to preserve shallow structure and details. The fused features are processed by 3×3 convolutions and the ReLU activation function, sequentially generating feature maps F5, F6, and F7. The DCA attention mechanism module enhances the encoder's ability to model spatial and channel information in feature maps. It comprises two parts: the Channel Cross-Attention (CCA) module and the Spatial Cross-Attention (SCA) module. The input feature map, after being flattened, is fed into both the CCA and SCA modules. The CCA module focuses on the contextual dependencies between channel dimensions, using a weighted modeling approach constructed with a channel attention matrix. The SCA module models long-distance relationships between spatial locations, enhancing the model's spatial awareness through a positional attention mechanism. The outputs of both modules are concatenated and processed using normalization and the GELU activation function to generate an enhanced feature representation that incorporates contextual semantics. The DCA module can be embedded into multiple encoder layers, acting on different semantic layers such as F1, F2, and F3, significantly improving the expression and integration capabilities of multi-scale features. Finally, the feature map F7 output from the decoder is fed into the segmentation head module. This module uses a 1×1 convolutional structure with the number of channels set to the number of classification categories. It converts the final feature map into a semantic segmentation map S, outputting a pixel-level mask map corresponding to the input image to indicate regions of interest with structural features in the image.

[0007] The upsampling module consists of upsampling operations, a 3×3 convolutional layer (Conv, stride=1, padding=1), and a ReLU activation function. It progressively enlarges the feature map size while fusing high-order semantic information with low-order detail information. The feature concatenation module concatenates the upsampled feature map with the corresponding encoder feature map along the channel dimension and unifies the feature space through convolutional fusion operations, improving feature integrity, segmentation accuracy, and spatial consistency. The Segmentation Head is a shallow convolutional structure that uses 1×1 convolutions to achieve dimensionality compression, generating a pixel-level segmentation map consistent with the number of task categories, facilitating supervised training and performance evaluation with the label map. Attached image description:

[0008] Figure 1 A model framework diagram of a medical image segmentation method based on an improved TransUnet network.

[0009] Figure 2 Network structure diagram of the dual cross-attention module (DCA).

[0010] Figure 3 Network structure diagram of the Channel Cross Attention (CCA) module in the dual cross attention module.

[0011] Figure 4 Network structure diagram of Spatial Cross Attention Module (SCA) of the dual cross attention module. Detailed Implementation

[0012] Based on the above description, the following is a specific implementation process, but the scope of protection of this patent is not limited to this implementation process.

[0013] Step 1: Input Preprocessing Stage

[0014] The medical image segmentation system of the present invention first receives an input medical image X∈R H×W×C Where H and W are the height and width of the image, respectively, and C represents the number of channels. For each pixel value x in the image, a linear mapping is used to map it from the original pixel value range [0,255] to the standardized interval [0,1]. The mapping formula is:

[0015]

[0016] Where, x norm ∈R H×W×C This represents the normalized pixel value. After this processing, the pixel values ​​of all channels are compressed to a uniform range. Data augmentation operations are applied to the image to enrich the diversity of training samples, including random rotation, translation, and flipping. The normalized image X norm ∈R H×W×CIt serves as the input for the subsequent convolutional neural network encoder module.

[0017] Step 2: CNN encoder extracts local features

[0018] The CNN encoder consists of four convolutional and pooling layers. Each layer extracts local texture and structural features through convolution, and then uses max pooling to compress spatial dimensions and expand the receptive field. (Normalized medical image X) norm ∈R H×W×C As input to the encoder module, C represents the number of channels in the image, and H and W represent the height and width of the image, respectively. In layer 1, the original image F (0) =X norm First, a convolution operation is performed using C1 3×3 convolution kernels, and the convolution result is then supplemented with a bias term b. (1) Then, after processing with the ReLU activation function, the first layer of convolutional feature maps is obtained:

[0019] F (1) =σ(Conv (1) (F (0) )+b (1) )

[0020] Where Conv (l) (·) indicates the l-th layer convolution operation, using multiple 3×3 convolution kernels; b (l) σ is the bias parameter; σ is the ReLU activation function.

[0021] Next, regarding F (1) Perform a 2×2 max pooling operation to reduce the spatial resolution and obtain the pooled output of layer 1:

[0022]

[0023] In the second layer, input Perform C2 = 2 × C1 3 × 3 convolution operations, plus the bias term b. (2) After processing with an activation function, the second convolutional feature map is obtained:

[0024]

[0025] Subsequently, regarding F (2) Perform max pooling and output the feature map as follows:

[0026]

[0027] In the 3rd layer Perform convolution processing using C3 = 2 × C2 3×3 convolution kernels to obtain the convolution result:

[0028]

[0029] Next, the max pooling operation is applied to F. (3) Dimension reduction yields:

[0030]

[0031] Enter in the 4th layer We use C4 = 2 × C3 3×3 convolutional kernels to extract deeper semantic features of image structure. The convolution calculation formula is as follows:

[0032]

[0033] Finally, by max pooling F (4) Spatial compression is performed to obtain the final output feature map of the CNN encoder module:

[0034]

[0035] The output feature map F cnn It integrates multi-level local structural information and has stronger representation capabilities.

[0036] Step 3: Transformer Encoder

[0037] After completing the local feature extraction of the CNN encoder, the obtained multi-scale encoder output feature map is as follows: The feature map is then input into the Transformer encoder module.

[0038] First, the spatial two-dimensional feature map F cnn Reshape into a sequence. Set the size to [missing information]. The three-dimensional tensor is flattened into a one-dimensional vector sequence. in C4 represents the length of the sequence and C4 represents the feature dimension at each position. This operation converts the spatial feature map into a sequence of vectors that can be processed by the Transformer, while preserving the high-dimensional semantic representation at each position.

[0039] To preserve the relative order and structural relationships of various spatial locations in the image, in the input sequence X... seq Based on this, positional encoding is introduced, and the position vector of each position is weighted and fused with its corresponding feature vector to obtain an embedding sequence containing spatial positional information:

[0040] Z (0) =X seq +E pos

[0041] in This represents the position encoding matrix.

[0042] Next, the position-encoded sequence is input into a Transformer encoder module consisting of m=12 stacked layers. Each Transformer layer contains two sub-layers: a multi-head self-attention mechanism and a feedforward neural network, supplemented by residual connections and layer normalization.

[0043] The forward propagation process of the m-th layer Transformer encoder can be expressed in the following form:

[0044]

[0045] Among them, MHSA(·) represents a multi-head self-attention mechanism, which is used to model long-distance dependencies between different locations; FFN(·) represents a feedforward fully connected network, which is used to improve representation capabilities; and LayerNorm(·) represents a layer normalization operation.

[0046] After stacking m layers of Transformer encoders, the final sequence output features are obtained. It integrates global contextual dependencies and significantly enhances semantic connectivity and global structure awareness between image regions. Finally, the sequence feature Z... ( M) is transformed back into a two-dimensional spatial structure, that is, it is reshaped to a size of Feature map:

[0047]

[0048] The feature map F trans It not only preserves the local texture and structural information extracted by the CNN encoder, but also integrates the global contextual information between distant regions in the image learned by the Transformer encoder.

[0049] Step 4: Dual Cross-Attention Module (DCA)

[0050] The Dual Cross-Attention (DCA) mechanism fuses local texture details extracted by the CNN encoder with global contextual semantic information captured by the Transformer encoder. This module consists of two complementary sub-modules: Spatial Cross-Attention (SCA) and Channel Cross-Attention (CCA). SCA focuses on joint modeling of salient regions in the spatial dimension, while CCA emphasizes the dynamic enhancement of channel-level semantic responses.

[0051] Step 4.1 Spatial Cross-Attention Module (SCA)

[0052] Let F be the local feature map from the CNN encoder path. cnn ∈R H′×W′×CThe global feature map from the Transformer path is F. trans ∈R H′×W′×C Both have the same spatial dimension. First, the Transformer output feature F trans As the main branch, it constructs the query in the attention mechanism, while the features F extracted by the CNN are... cnn It is used to generate both keys and values. The specific process is as follows:

[0053] Q = W Q ·F trans ,

[0054] K = W K ·F cnn ,

[0055] V = W V ·F cnn ,

[0056] Among them W Q W K W V These are trainable 1×1 convolution weights. Next, the spatial attention matrix A is calculated. spatial :

[0057]

[0058] Where d k The dimension of the key vector is used for scaling. This attention weight is then used to weight the value vector to obtain the enhanced spatial attention feature map:

[0059] Fsca=Aspatial·VF sca =A spatial ·V

[0060] The final output F sca It achieves fine-tuning of Transformer spatial attention under the guidance of CNN local structural information, which helps to highlight discriminative target regions.

[0061] Step 4.2 Channel Cross-Attention Module (CCA)

[0062] To enhance the model's feature representation capabilities in the semantic channel dimension, the CCA module introduces an element-level interaction mechanism and a channel modeling structure.

[0063] First, the local feature map F cnn With global feature map F trans Element-wise multiplication along the channel dimension generates the fused response map: F mul =F cnn ⊙F transSubsequently, global average pooling is performed on the fused feature map to generate channel description vectors:

[0064]

[0065] The vector z∈R C This characterizes the overall response of the fused features in each channel. Next, z is input to a two-layer fully connected network to form the channel attention weights:

[0066] s=σ(W2·δ(W1·z+b1)+b2)

[0067] Where W1 and W2 are the weights of the fully connected layer, δ(·) is the ReLU activation function, σ(·) is the Sigmoid function, and the output s∈R c This is the channel attention vector. Finally, for F... trans

[0068] Perform channel weighting to generate the output feature map:

[0069] F cca (i,j,c)=s(c)·F trans (i,j,c)

[0070] Step 4.3 Feature Fusion Output

[0071] Spatial attention feature map F sca With channel attention feature map F cca The fused features F are obtained by concatenating the data along the channel dimension. cat :

[0072] F cat =Concat(F sca ,F cca )

[0073] Feature integration is then achieved through 1×1 convolution:

[0074] F fused =Conv 1×1 (F cat )

[0075] and the original output F of the Transformer trans Adding them together creates residual connections, enhancing stability and semantic consistency.

[0076] F dca =F fused +F trans

[0077] Step 5: Decoder Upsampling and Skip Connections

[0078] Fusion features As the initial input to the decoder, it is upsampled layer by layer using transposed convolution or bilinear interpolation. In each upsampling stage, the output size is gradually restored, from... Restore to Until the original image resolution H×W is returned.

[0079] To compensate for the spatial information loss during downsampling, a skip connection mechanism is adopted to integrate the shallow feature maps of the corresponding scale in the encoder. The features are concatenated to the upsampled feature map and then fused using a convolution operation:

[0080]

[0081] Step 6: Output Layer

[0082] The decoder outputs a feature map of size H×W×C′, where C′ is the set number of channels. This map is then mapped to the number of classes K using a 1×1 convolution.

[0083] F cls =Conv 1×1 (F up )∈R H×W×K

[0084] Then, the class probability of each pixel is obtained through the pixel-wise Softmax function:

[0085]

[0086] The final pixel-level segmentation prediction map with dimensions H×W×K is obtained.

[0087] Step 6.1: Candidate Box Generation and Non-Maximum Suppression (NMS)

[0088] The task requirements include object detection functionality, and further, generating candidate bounding boxes based on the segmentation results. First, a set of candidate bounding boxes is generated based on the category probability map. Each candidate box B i Accompanied by a confidence score S i To remove redundant bounding boxes, the Non-Maximum Suppression (NMS) algorithm is used:

[0089] Set the IoU threshold τ = 0.7 and initialize the retain set. Based on the confidence score S i Sort the candidate box list I in descending order; repeat the process until I is empty.

[0090] Take the index i of the box with the highest confidence, and then... i join in Calculate B i Based on the IoU value of the remaining boxes, delete candidate boxes with IoU > τ; output As the final set of candidate boxes for deduplication.

[0091] Step 7: Loss Function and Training Optimization

[0092] The model training uses a combined loss function. Supervise and guide the model to simultaneously optimize region coverage and class discrimination capabilities:

[0093]

[0094] Where λ1 = 0.7 is the Dice loss. The weights, λ² = 0.3, are the cross-entropy loss. The weights, used to measure the overlap between the predicted and actual regions, are defined as follows:

[0095]

[0096] Cross-entropy loss is used for pixel-level class prediction.

[0097]

[0098] Where p i,k G represents the probability that the i-th pixel is predicted to be of class k. i,k This is a real label;

[0099] The ReduceLROnPlateau strategy is used to adaptively adjust the learning rate; when the validation set loss does not decrease within 15 consecutive epochs, the current learning rate η is updated.

[0100] η′=η×γ

[0101] Where γ = 0.5 is the learning rate decay factor, and the minimum learning rate η is set. min =1×10 -6 To ensure that training does not stagnate; if η = η min If the validation performance no longer improves, the model is considered to have converged, and training is terminated.

Claims

1. An image segmentation method based on TransUNet network, characterized in that: First, a lightweight image region detection model based on convolutional neural networks is constructed and pre-trained on an image dataset; the model uses a dual-channel attention module DCA to extract deep features of the target region in the input image; The DCA module consists of two sub-modules: the Channel Context Aggregation Module (CCA) and the Spatial Context Aggregation Module (SCA). CCA is used to model the global semantic dependencies between different channels, enhancing the model's ability to perceive important semantic channels; SCA focuses on modeling the contextual relationships between spatial locations in the image. A refined image segmentation model combining convolutional neural networks and Transformer structures is constructed. In this model, DCA modules and Transformer layers are fused at different levels according to the design structure. The shallow stage is used to extract texture and edges, while the deep stage is used to capture global context and high semantic features. The model extracts features in parallel at multiple scales and integrates multi-scale information by feature concatenation. In addition, downsampling and upsampling modules are introduced to improve the model’s recognition accuracy for small targets and large-scale structures while maintaining the feature hierarchy. During the training phase, the model is trained on a self-built image dataset and its generalization ability is improved through a variety of data augmentation strategies. The data augmentation methods employed include random image flipping, cropping, focusing enhancement of local target areas, and adjustment of image brightness and contrast parameters to simulate different imaging conditions. During the inference phase, a lightweight detection model is first used to quickly filter the input images, removing redundant image frames that do not contain the target region and retaining only key frames that may contain the target. Subsequently, the keyframes are input into the fine segmentation model to obtain pixel-level target region prediction results. The prediction mask is output through the segmentation head. In the post-processing stage, only the target results with high confidence and accurate localization after non-maximum suppression NMS processing with pixel-level confidence greater than 0.8 and IoU threshold = 0.7 are retained.

2. The detection method according to claim 1, characterized in that: Step 1: Input Preprocessing Stage First, receive the input medical image X∈R. H×W×C Where H and W are the height and width of the image, respectively, and C represents the number of channels; for each pixel value x in the image, a linear mapping is used to map it from the original pixel value range [0,255] to the standardized interval [0,1]. The mapping formula is: Where, x norm ∈R H×W×C This represents the normalized pixel value; after this processing, the pixel values ​​of all channels are compressed to a uniform range; data augmentation operations are applied to the image to enrich the diversity of training samples; the normalized image X norm ∈R H ×W×C As input to the subsequent convolutional neural network encoder module; Step 2: CNN encoder extracts local features The CNN encoder consists of four convolutional and pooling units. Each layer extracts local texture and structural features through convolutional operations, and then compresses spatial dimensions and expands the receptive field through max pooling operations; the normalized medical image X norm ∈R H×W×C As input to the encoder module, where C represents the number of channels in the image, and H and W represent the height and width of the image, respectively; in layer 1, the original image F (0) =X norm First, a convolution operation is performed using C1 3×3 convolution kernels, and the convolution result is then supplemented with a bias term b. (1) Then, after processing with the ReLU activation function, the first layer of convolutional feature maps is obtained: F (1) =σ(Conv (1) (F (0) )+b (1) ) Where Conv (l) (·) indicates the l-th layer convolution operation, using multiple 3×3 convolution kernels; b (l) σ is the bias parameter; σ is the ReLU activation function; Next, regarding F (1) Perform a 2×2 max pooling operation to reduce the spatial resolution and obtain the pooled output of layer 1: In the second layer, input Perform C2 = 2 × C1 3 × 3 convolution operations, plus the bias term b. (2) After processing with an activation function, the second convolutional feature map is obtained: Subsequently, regarding F (2) Perform max pooling and output the feature map as follows: In the 3rd layer Perform convolution processing using C3 = 2 × C2 3×3 convolution kernels to obtain the convolution result: Next, the max pooling operation is applied to F. (3) Dimension reduction yields: Enter in the 4th layer C4 = 2 × C3 3×3 convolutional kernels are used to extract deeper semantic features of image structure; the convolution calculation formula is as follows: Finally, by max pooling F (4) Spatial compression is performed to obtain the final output feature map of the CNN encoder module: Step 3: Transformer Encoder After completing the local feature extraction of the CNN encoder, the obtained multi-scale encoder output feature map is as follows: The feature map is input into the Transformer encoder module; First, the spatial two-dimensional feature map F cnn Reshape into a sequence; change the size to The three-dimensional tensor is flattened into a one-dimensional vector sequence. in C4 represents the length of the sequence and C4 represents the feature dimension at each position. This operation converts the spatial feature map into a vector sequence that can be processed by the Transformer, while preserving the high-dimensional semantic representation at each position. To preserve the relative order and structural relationships of various spatial locations in the image, in the input sequence X seq Based on this, positional encoding is introduced, and the position vector of each position is weighted and fused with its corresponding feature vector to obtain an embedding sequence containing spatial positional information: Z (0) =X seq +E pos in Represents the position encoding matrix; Next, the position-encoded sequence is input into a Transformer encoder module consisting of m=12 stacked layers; each Transformer layer contains two sub-layers: a multi-head self-attention mechanism and a feedforward neural network, supplemented by residual connections and layer normalization; The forward propagation process of the m-th layer Transformer encoder can be expressed in the following form: Wherein, MHSA(·) represents a multi-head self-attention mechanism, used to model long-distance dependencies between different locations; FFN(·) represents a feedforward fully connected network, used to improve representation capabilities; and LayerNorm(·) represents a layer normalization operation. After stacking m layers of Transformer encoders, the final sequence output features are obtained. It integrates global contextual dependencies and significantly enhances semantic connectivity and global structure awareness between image regions; finally, it applies the sequence feature Z... ( M) is transformed back into a two-dimensional spatial structure, that is, it is reshaped to a size of Feature map: The feature map F trans It not only preserves the local texture and structural information extracted by the CNN encoder, but also integrates the global contextual information between distant regions in the image learned by the Transformer encoder; Step 4: Dual Cross-Attention Module (DCA) The dual cross-attention mechanism DCA fuses the local texture details extracted by the CNN encoder with the global contextual semantic information captured by the Transformer encoder. This module consists of two complementary sub-modules: spatial cross-attention (SCA) and channel cross-attention (CCA). SCA focuses on joint modeling of salient regions in the spatial dimension, while CCA focuses on the dynamic enhancement of channel-level semantic responses. Step 4.1 Spatial Cross-Attention Module (SCA) Let F be the local feature map from the CNN encoder path. cnn ∈R H′×W′×C The global feature map from the Transformer path is F. trans ∈R H′×W′×C Both have the same spatial dimension; first, the Transformer output feature F trans As the main branch, it constructs the query in the attention mechanism, while the features F extracted by the CNN are... cnn It is also used to generate both keys and values; the specific process is as follows: Q=W Q ·F trans , K=W K ·F cnn , V=W V ·F cnn , Among them W Q W K W V The weights are trainable 1×1 convolutional weights; next, the spatial attention matrix A is calculated. spatial : Where d k The dimension of the key vector is used for scaling; then, this attention weight is used to weight the value vector to obtain the enhanced spatial attention feature map: Fsca=Aspatial·VF sca =A spatial ·In Step 4.2 Channel Cross-Attention Module (CCA) First, the local feature map F cnn With global feature map F trans Element-wise multiplication along the channel dimension generates the fused response map: F mul =F cnn ⊙F trans Subsequently, global average pooling is performed on the fused feature map to generate channel description vectors: The vector z∈R C The fused features are characterized to represent the overall response in each channel; next, z is input to a two-layer fully connected network to form channel attention weights: s=σ(W2·δ(W1W+b1)+b2) Where W1 and W2 are the weights of the fully connected layer, δ(·) is the ReLU activation function, σ(·) is the Sigmoid function, and the output s∈R C For the channel attention vector; finally, for F trans Perform channel weighting to generate the output feature map: F cca (i,j,c)=s(c)·F trans (i,j,c) Step 4.3 Feature Fusion Output Spatial attention feature map F sca With channel attention feature map F cca The fused features F are obtained by concatenating the data along the channel dimension. cat : F cat =Concat(F sca ,F cca ) Feature integration is then achieved through 1×1 convolution: F fused =Conv 1×1 (F cat ) and the original output F of the Transformer trans Adding them together creates residual connections, enhancing stability and semantic consistency. F dca =F fused +F trans Step 5: Decoder Upsampling and Skip Connections Fusion features As the initial input to the decoder, it is upsampled layer by layer using transposed convolution or bilinear interpolation; in each upsampling stage, the output size is gradually restored, from... Restore to Until the image returns to its original resolution of H×W; To compensate for the spatial information loss during downsampling, a skip connection mechanism is adopted to integrate the shallow feature maps of the corresponding scale in the encoder. The features are concatenated to the upsampled feature map and then fused using a convolution operation: Step 6: Output Layer The decoder outputs a feature map of size H×W×C′, where C′ is the set number of channels; this map is then mapped to the number of classes K using a 1×1 convolution. F cls =Conv 1×1 (F up )∈R H×W×K Then, the class probability of each pixel is obtained through the pixel-wise Softmax function: The final pixel-level segmentation prediction map with dimensions H×W×K is obtained; Step 6.1: Candidate Box Generation and Non-Maximum Suppression (NMS) The task requirements include object detection functionality, and further, generating candidate bounding boxes based on the segmentation results; firstly, a set of candidate bounding boxes is generated based on the category probability map. Each candidate box B i Accompanied by a confidence score S i To remove redundant bounding boxes, the Non-Maximum Suppression (NMS) algorithm is used. Set the IoU threshold τ = 0.7 and initialize the retain set. Based on the confidence score S i Sort the candidate box list I in descending order; loop through the list until I is empty. Take the index i of the box with the highest confidence, and then... i join in Calculate B i Based on the IoU value of the remaining boxes, delete candidate boxes with IoU > τ; output As the final set of candidate boxes for deduplication; Step 7: Loss Function and Training Optimization The model training uses a combined loss function. Supervise and guide the model to simultaneously optimize region coverage and class discrimination capabilities: Where λ1 = 0.7 is the Dice loss. The weights, λ² = 0.3, are the cross-entropy loss. The weights, used to measure the overlap between the predicted and actual regions, are defined as follows: Cross-entropy loss is used for pixel-level class prediction. Where p i,k G represents the probability that the i-th pixel is predicted to be of class k. i,k This is a real label; The ReduceLROnPlateau strategy is used to adaptively adjust the learning rate; when the validation set loss does not decrease within 15 consecutive epochs, the current learning rate η is updated. η′=η×γ Where γ = 0.5 is the learning rate decay factor, and the minimum learning rate η is set. min =1×10 -6 To ensure that training does not stagnate; if η = η min If the validation performance no longer improves, the model is considered to have converged, and training is terminated.

Images