A semantic image segmentation method based on Transformer architecture

Abstract

This invention relates to an image semantic segmentation method based on a Transformer architecture, belonging to the field of medical image segmentation technology. The invention provides a method for extracting image contextual features using a hierarchical Swing Transformer encoder with a moving window. First, image patches are input into a Transformer-based encoder-decoder architecture, where the moving window mechanism ensures more comprehensive feature information. Furthermore, the nested and dense skip connections in the UNet++ model effectively extract contextual features, resulting in more thorough feature fusion.

Description

Technical Field

[0001] This invention belongs to the field of medical image segmentation technology, specifically relating to an image semantic segmentation method based on the Transformer architecture. Background Technology

[0002] Semantic segmentation is pixel-level image understanding, which involves labeling the category of each pixel in an image. The task is to segment an image into several meaningful objects and assign a specific type label to each object. Labeling the category of each pixel in an image is a key task in scene parsing and action recognition in the field of computer vision. Therefore, obtaining a practical and accurate image semantic segmentation model is a very challenging task.

[0003] Unlike classification, semantic segmentation requires determining the category of each pixel in an image for precise segmentation. Because CNNs lose image details during convolution and pooling (i.e., the feature map size gradually decreases), they cannot accurately identify the specific outlines of objects or which object each pixel belongs to, thus failing to achieve precise segmentation. To address this problem, FCNs use deconvolution to upsample the feature map of the last convolutional layer, restoring it to the same size as the input image, thereby generating predictions for each pixel—that is, pixel-level classification. However, FCNs still lack sufficient detail because they do not adequately consider the spatial relationships between pixels. To address this, UNet employs an encoder-decoder structure, recovering edge features lost during downsampling through feature concatenation during upsampling. UNet's elegant structural design has led to significant success in various medical imaging applications. Following this approach, many algorithms, such as Res-UNet, UNet++, and UNet3+, have been developed for image and object segmentation in various medical imaging methods. The excellent performance of these methods in heart segmentation, organ segmentation, and lesion segmentation demonstrates the strong feature learning ability of CNNs. To address the issue of excessively large hierarchical information span in the UNet model, the UNet++ model designs an architecture with nested and densely skip connections. This aims to reduce the semantic gap between the encoder and decoder, resulting in more complete feature fusion and more comprehensive splicing.

[0004] On the other hand, the evolution of network architectures in Natural Language Processing (NLP) has taken a different path, with the Transformer being the currently popular architecture. Designed for sequence modeling and transformation tasks, the Transformer's notable feature is its ability to model long-range dependencies in data. Its tremendous success in the language domain has prompted researchers to investigate its adaptability in computer vision, where it has recently shown promising results in certain tasks, particularly image classification and joint visual-language modeling. Summary of the Invention

[0005] The technical problem this invention aims to solve is to provide a method for extracting image contextual features using a hierarchical Swing Transformer encoder with a moving window. First, image patches are input into a Transformer-based encoder-decoder architecture, where the moving window mechanism ensures that the features contain more comprehensive information. Furthermore, the nested and dense skip connections in the UNet++ model effectively extract contextual features, resulting in more thorough feature fusion.

[0006] Swin-UNet++ is a framework method for medical image segmentation based on UNet++. This paper attempts to extend the applicability of Transformer to make it a general backbone for computer vision. The significant challenge of transferring its high performance in the language domain to the vision domain can be explained by the difference between the two models, one of which is scale. To reduce sequence length, images are divided into windows to address the challenge of varying object sizes within the same image. This invention applies a general Transformer backbone called the Swin Transformer, which constructs hierarchical feature maps and has computational complexity linear with image size. The Swin Transformer constructs hierarchical representations by starting with small image patches and gradually merging adjacent patches in deeper Transformer layers. Through these hierarchical feature maps, the Swin Transformer can easily perform dense predictions based on models such as UNet. Using the Swin Transformer with a moving window mechanism, self-attention is locally computed within non-overlapping windows of the image, better capturing contextual semantic information in the image, and the number of patches in each window is fixed, thus the complexity is linear with image size. These advantages make the Swin Transformer suitable as a general backbone for a variety of vision tasks.

[0007] Feasibility Analysis of this Invention: Commonly used convolutions can extract features through local perception, extracting multi-scale features using convolution kernels of different sizes. SwinTransformer, however, uses a local window self-attention mechanism to extract features from sub-regions of the image, exhibiting computational complexity linear with the image size. As the window size changes, self-attention is computed within a small local window to construct hierarchical feature maps; coupled with a moving window mechanism—specifically, it moves window partitions between consecutive self-attention layers—it allows pixels in different windows to interact with each other, thereby better understanding the context and significantly enhancing modeling capabilities, achieving global modeling. Simultaneously, in constructing the overall structure of SwinTransformer, feature fusion at each stage can achieve the same hierarchical feature changes as UNet during downsampling.

[0008] Based on the above analysis, this invention adopts the following technical solution: Inspired by the Swin Transformer, the Swin-UNet++ method utilizes Transformer for 2D medical image segmentation in this work. Swin-UNet is the first purely Transformer-based U-shaped architecture, consisting of an encoder, bottleneck, decoder, and skip connections. The encoder, bottleneck, and decoder are all built based on the Swin Transformer module. The input medical image is segmented into non-overlapping image patches, each patch being treated as a unit and input into the Transformer-based encoder to learn a deep feature representation. Then, the decoder with a patch extension layer upsamples the extracted contextual features. During the upsampling process, the features are fused with the multi-scale features of the encoder through skip connections to restore the spatial resolution of the feature map for further segmentation prediction. Extensive experiments on multi-organ and heart segmentation datasets demonstrate that this method has good segmentation accuracy and robust generalization ability. Based on this idea, the Swin Transformer module is applied to the UNet++ model, which exhibits faster gradient descent and higher accuracy compared to Swin-UNet.

[0009] Reference Figure 1 Swin-UNet++ includes the following steps:

[0010] The network consists of a preprocessing module, a Swin-UNet++ module, and a recovery module.

[0011] Step 1: The preprocessing module mainly includes Patch Partition and Linear Embedding.

[0012] The initial input image is scaled, and the adjusted image size is denoted as W×H, where W represents the image width, H represents the image height, and W×H describes the image resolution, i.e., the number of pixels. Next, data augmentation is performed on the input image to increase data diversity. The main function of Patch Partition is to compress the image size. Linear Embedding then uses linear embedding layers to transform the projected feature dimension into an arbitrary dimension C.

[0013] Step 2: The input data for the Swin-UNet++ module is the feature map output from Step 1. It primarily extracts image features through multiple downsampling processes, and during upsampling, it leverages dense skip connections to fully fuse features from multiple levels.

[0014] This approach combines the UNet++ architecture with Transformers to fully leverage the advantages of both. Feature learning is performed using SwingTransformer blocks, downsampling is achieved using patch merging layers and Swing Transformer blocks, and upsampling is achieved using patch expansion layers and Swing Transformer blocks. During downsampling, hierarchical feature representations are generated through multiple SwingTransformer blocks and patch merging layers. The upsampling process includes multiple SwingTransformer blocks and patch expanding layers. Simultaneously, skip connections are used to fuse extracted contextual features with multi-scale features from the encoder to compensate for the spatial information loss caused by downsampling.

[0015] Step 3: The recovery module performs pixel-level predictions on the feature maps output by the Swin-UNet++ module. This includes a final patch expansion layer and linear projection, ultimately yielding the labeled image output. The patch expansion layer restores the image feature size, restoring the feature mapping resolution to the input resolution W×H. Then, these upsampled features are linearly projected onto a specific dimension to obtain pixel-level classification prediction results.

[0016] Compared with existing technologies, this method has the following significant advantages:

[0017] (1) Based on the SwinTransformer block, a symmetric encoder-decoder architecture with dense skip connections was constructed. Feature extraction at each level was implemented in the encoder; in the decoder, the features were upsampled to the input resolution and corresponding pixel-level segmentation prediction was performed.

[0018] (2) A patch extension layer was developed to achieve upsampling without using convolution and interpolation operations.

[0019] (3) Experiments showed that skip connections are also effective for Transformers. Therefore, a U-shaped encoder-decoder architecture with nested and dense skip connections based on Transformers was finally constructed and named Swin-UNet++. Under the nested feature concatenation structure of UNet++, the local self-attention moving window mechanism can better extract contextual features and achieve global modeling. Attached Figure Description

[0020] Figure 1 This is the overall flowchart of this method;

[0021] Figure 2 This is the original UNet++ structure diagram;

[0022] Figure 3 Here is a diagram of the Swin-UNet++ architecture;

[0023] Figure 4 A schematic diagram showing the feature dimension changes of the encoder's main unit;

[0024] Figure 5 This is a structural diagram of the SwingTransformer block module. Detailed Implementation

[0025] The method will be further described in detail below with reference to specific embodiments and the accompanying drawings.

[0026] The hardware used includes one PC; the auxiliary tool is PyCharm.

[0027] Step 1, refer to Figure 1 The preprocessing module mainly includes Patch Partition and Linear Embedding.

[0028] The preprocessing module is structured as follows: image preprocessing → Patch Partition layer → Linear Embedding layer → output feature map.

[0029] Step 1.1 Image Preprocessing

[0030] The preprocessing module takes the original image as input and scales it to W×H, where H represents the height of the image and W represents the width. Furthermore, to avoid overfitting due to a small sample size, data augmentation is used to expand the data; specifically, the image is randomly flipped and rotated with a certain probability.

[0031] The specific steps of random flipping include:

[0032] 1. First, rotate the image randomly by 90 degrees, 180 degrees, or 270 degrees;

[0033] 2. Randomly flip the rotated image vertically or horizontally. Vertical flipping treats the rows of the image array as a whole and reverses their order; horizontal flipping treats the columns of the image array as a whole and reverses their order, thus expanding the data.

[0034] Random rotation: Randomly select an angle between -20 degrees and +20 degrees to rotate the image.

[0035] Step 1.2 Image segmentation

[0036] To transform the input information into sequence embeddings, the Patch Partition layer primarily compresses the size of the preprocessed image, thereby reducing the sequence length. The pixels of the medical image are divided into N×N non-overlapping patches. In this partitioning method, each patch window is treated as a "token," and its feature is set to the concatenation of the original pixel's RGB values, i.e., 3. Therefore, the feature dimension of each patch becomes N×N×3. Specifically, N is set to 4, PatchPartition performs a convolution operation with a 4×4 kernel size and a stride of 4.

[0037] Step 1.3 Linear Embedding Layer

[0038] By using a linear embedding layer for feature mapping, the feature dimension of the projection is transformed into an arbitrary dimension C, thereby obtaining a feature map.

[0039] Step 2: The Swin-UNet++ module extracts image features through multiple downsampling and upsampling processes and fuses features at multiple levels. The input data for this module is the feature map output from Step 1, with a size of W / 4×H / 4×C; the output of this module is the extracted feature map, with a size of W / 4×H / 4×C.

[0040] This module performs hierarchical feature fusion on images based on the UNet++ model. UNet++ consists of an encoder and a decoder, connected by a series of nested dense convolutional blocks. The main idea of ​​UNet++ is to bridge the semantic gap between the feature maps of the encoder and decoder before feature fusion. (See reference...) Figure 2 We use X i,j Represents convolutional layer X i,j The output of is where i indexes the downsampling layers in the encoder, and j indexes the convolutional layers of dense blocks along the skip paths. Features are concatenated before the convolutional layers on each skip path, so that all prior feature maps are accumulated and integrated into the current node.

[0041] Building upon this, the encoder in Swin-UNet++ is primarily used to extract features from various levels. The encoder consists of alternating SwinBlock layers and patch merging layers. (See reference...) Figure 4 The encoder's input data is the feature map W / 4×H / 4×C output from step 1; the encoder's output is the extracted feature map W / 32×H / 32×8C.

[0042] The encoder structure is as follows: Input data → First Swing Block layer → Patch merging layer → Second Swing Block layer → Patch merging layer → Third Swing Block layer → Patch merging layer → Bottleneck layer

[0043] The Swing Block layer consists of interconnected individual Swing Transformer modules. The patch merging layer selects elements at intervals along the rows and columns of the patch, concatenates these elements into a tensor, and then expands it. Channel normalization and a fully connected layer are then used to adjust the channel dimension. This process reduces feature resolution and increases feature dimension, thus achieving downsampling. The last patch merging layer in the encoder is connected to the first Swing Transformer Block in the bottleneck layer. The bottleneck layer consists of two Swing Transformer Blocks connected sequentially. These two Swing Transformer Blocks are connected to the output of the last patch merging layer in the encoder and the input of the bottom-most patch expansion layer in the upsampling stage, respectively, using channel normalization and a fully connected layer. In other words, the bottom-most patch expansion layer in the upsampling stage is connected to the second Swing Transformer Block in the bottleneck layer.

[0044] Building upon the encoder, layer-by-layer upsampling and feature fusion are employed to fully understand the contextual information of the image. Feature fusion is achieved through multiple skip connections and patch expansion layers. Skip connections fuse the output of one SwinBlock layer in the same dense block with the upsampled output of the next dense block. The patch expansion layer first expands the number of channels through a fully connected layer, then uses a rearrange operation to reassemble adjacent feature maps into a larger feature map. This process increases feature resolution and reduces feature dimensionality. Finally, LayerNorm channel normalization is performed, thus achieving upsampling.

[0045] Use B i,j Indicates that it passes through the SwinBlock layer B i,jThe output of feature learning is used, where i indexes the downsampling layer of the encoder, j indexes the SwinBlock layer along the jump path, and each Swin Block layer contains two Swin TransformerBlocks. Figure 3 The steps for feature fusion in the Swin-UNet++ architecture are as follows:

[0046] 1. Place B 0,0 With B 1,0 The corresponding upsampled outputs are fused to obtain B. 0,1 ;

[0047] 2. After 1, place B 1,0 and B 2,0 The corresponding upsampled output is fused to obtain B 1,1 B 1,1 The upsampled output and B 0,0 B 0,1 B is obtained through skip connection fusion. 0,2 ;

[0048] 3. Similarly, through hierarchical upsampling and feature fusion, B can be obtained sequentially. 2,1 B 1,2 Finally, we obtained B. 0,3 B 0,3 For B 1,2 upsampled output and B 0,0 B 0,1 B 0,2 The result of fusion.

[0049] The Swin TransformerBlock is responsible for feature representation learning, while maintaining the feature dimension and resolution. To avoid excessive complexity caused by global self-attention computation, the image is divided into non-overlapping windows. Specifically, for efficient modeling, self-attention is computed within local windows. To introduce cross-window connections while maintaining efficient computation of non-overlapping windows, a shifted window partitioning method is proposed, alternating between regular window partitioning and shifted window partitioning within consecutive Swin Transformer blocks, thereby enabling the interaction of spatial information between different partitions.

[0050] The W-MSA submodule (window multi-head self-attention layer) primarily employs a regular window partitioning strategy. Starting from the top-left pixel, it evenly divides the feature map into windows, each containing a certain number of patches. It captures the dependencies between features in different subspaces by calculating attention vectors in each subspace. Specifically, it performs dot product operations to calculate attention vectors in all subspaces, and finally concatenates the attention vectors calculated in all subspaces, mapping them back to the original input space to obtain the final attention vector as the output. This approach better captures both local and global semantic information.

[0051] The SW-MSA submodule (Shifted Window Multi-Head Self-Attention Layer) performs a pixel shift operation of half the size of a window on the image features before the W-MSA submodule operation. The window partitions are shifted, creating new window partitions and introducing connections between adjacent non-overlapping windows. That is, the self-attention computation of the new window crosses the boundaries of the previous window partitions, providing connections between them.

[0052] Reference Figure 5 The structure of a single SwingTransformerBlock is as follows: feature map input data → LN regularization layer → W-MSA submodule or SW-MSA submodule → residual connection layer → LN regularization layer → feedforward network MLP → residual connection layer → output feature map. This SwingTransformerBlock structure loops twice, with odd-numbered layers alternately connected to even-numbered layers. Odd-numbered layers use the W-MSA submodule, and even-numbered layers use the SW-MSA submodule.

[0053] Using the shifted window partitioning method, consecutive SwingTransformerBlocks are calculated as follows:

[0054]

[0055]

[0056]

[0057]

[0058] Where z l-1 This represents the input features of the Swing Transformer block. and z l These represent the output characteristics of the W-MSA module and the MLP module in the l-th SwinTransformer block, respectively. and z l+1Let represent the output features of the SW-MSA module and the MLP module in the (l+1)th SwinTransformer block, respectively. W-MSA and SW-MSA use rule-based and shifted window partitioning to calculate self-attention in each window: first, the query vector Q and the key vector K are multiplied by a dot product, then divided by the square root of the dimension of the key vector K to obtain the score matrix of the query vector Q, and the relative position deviation is added. Finally, the score matrix is ​​normalized by the Softmax function to obtain the weight matrix, and then multiplied by the value vector V to obtain the attention vector of a subspace.

[0059]

[0060] Where Q, K, and V represent the query matrix, key matrix, and value matrix, respectively, and d represents the dimension of the matrix. When calculating similarity, a relative positional bias is introduced, where B represents the bias matrix and SoftMax is the activation function. Specifically, after a LayerNorm regularization layer, W-MSA first uses a fully connected layer to obtain the three matrices Q, K, and V. Based on the above formula, self-attention is calculated, and then dropout is performed. Finally, the resulting output is compared with z... l-1 Perform residual connection to obtain After passing through the LN layer and MLP layer in sequence, dropout is performed, and then... Perform residual connection to obtain z l z l After LayerNorm regularization, the image feature map is shifted by half a window size using SW-MSA. This is repeated cyclically, shifting it upwards and leftwards by half a window size. The window is then divided according to W-MSA, and after the process, the resulting feature map is cyclically shifted downwards and rightwards by half a window size, restoring it to its original position. Based on the above formula, self-attention is calculated, and dropout is then applied, followed by... l Perform residual connection to obtain After passing through the LN layer and MLP layer in sequence, dropout is performed, and then... Perform residual connection to obtain z l+1 The main function of the LN regularization layer is to perform LN regularization, normalizing the input data to a range of 0 to 1, thus ensuring a uniform data distribution. The residual connection layer's main function is to perform residual connections, directly adding the input and output to supplement the feature information lost during convolution, preventing overfitting, improving model performance, and overcoming network degradation. The MLP layer consists of two fully connected layers. The input features pass through a fully connected layer to obtain intermediate features with a dimension set to four times the dimension of the input features, then pass through the GELU activation function, and finally through another fully connected layer to obtain the output features.

[0061] Step 3: The main function of the recovery module is to restore the image size and obtain pixel-level prediction results.

[0062] Reference Figure 1 The structure of the recovery module is as follows: feature map output by the Swin-UNet++ module → patch extension layer → Linear Projection layer → pixel-level prediction result of the output image.

[0063] Step 3.1 Upsamples the feature map using the last patch expansion layer to restore the resolution of the feature map output by the Swin-UNet++ module to the input resolution W×H. This patch expansion layer first expands the number of channels through a fully connected layer. Here, the dimension of the output tensor is set to 16 times that of the input. Then, a rearrange operation is used to reassemble the feature maps of adjacent dimensions into a larger feature map. Finally, LayerNorm channel normalization is performed to achieve upsampling.

[0064] Step 3.2 Perform linear projection on these upsampled features and output pixel-level segmentation prediction results W×H×Class, where Class is the number of pixel-level classification categories, H represents the height of the image, and W represents the width of the image.

[0065] This completes the process of inputting the image into the defined Swin-UNet++ network and obtaining pixel-level prediction results. The DSC (Dice similarity coefficient) is then calculated based on the ground truth labels and the prediction results. The DSC parameters for each subclass of the segmented image and the overall segmented image are recorded to evaluate the segmentation accuracy. The DSC (Dice similarity coefficient) measures the similarity between two sets, with a value ranging from [0,1]. A larger value indicates greater similarity between the two sets.

[0066] The above embodiments are merely exemplary embodiments of the present invention and are not intended to limit the present invention. The scope of protection of the present invention is defined by the claims. Those skilled in the art can make various modifications or equivalent substitutions to the present invention within its spirit and scope of protection, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of the present invention.

Claims

1. A method for image semantic segmentation based on a Transformer architecture, characterized in that: Step 1, a preprocessing module; scale the initial input image, and the adjusted image size is denoted as WxH, where W represents the width of the image, H represents the height of the image, and WxH describes the resolution of the image, i.e., the number of pixel points; then perform data augmentation on the input image to increase data diversity; Patch Partition compresses the size of the image; Linear Embedding uses a linear embedding layer to convert the projected feature dimension to any dimension C; Step 2, a Swin-UNet++ module; learn the feature map output by step 1 based on a SwinTransformer block; use a patch merging layer and a Swin Transformer block to complete downsampling, and use a patch expansion layer and a Swin Transformer block to complete upsampling; in the downsampling process, a hierarchical feature representation is generated through multiple Swin Transformer blocks and patch merging layers; The upsampling process includes multiple SwinTransformer blocks and patch expansion layers; the extracted context features are fused with the multi-scale features of the encoder through a skip connection to make up for the loss of spatial information caused by downsampling; Step 3, a recovery module for pixel-level prediction of the image The structure of the recovery module is as follows: feature map output by the Swin-UNet++ module → patch expansion layer → LinearProjection layer → output image pixel-level prediction result Step 3.1 uses the last patch expansion layer to upsample the feature map, restoring the resolution of the feature map output by the Swin-UNet++ module to the input resolution WxH; this patch expansion layer first expands the channel number through a fully connected layer, sets the dimension of the output tensor to 16 times the input here, and uses a rearrange operation to reorganize the feature maps of adjacent dimensions into larger feature maps, and then performs LayerNorm channel normalization to achieve upsampling; Step 3.2 linearly projects these upsampled features to output pixel-level segmentation prediction results WxHxClass, where Class is the number of pixel-level classification categories, H represents the height of the image, and W represents the width of the image; At this point, the image is input to the defined Swin-UNet++ network and the pixel-level prediction result is obtained, the DSC (Dice Similarity Coefficient) is calculated according to the true label and the prediction result; the DSC parameters of each sub-class of the segmentation image and the DSC parameters of the entire segmentation image are recorded to evaluate the accuracy of the segmentation; where DSC (Dice Similarity Coefficient) is used to measure the similarity of two sets, with a value range of [0, 1], and the larger the value, the more similar the two sets.

2. The method for image semantic segmentation based on a Transformer architecture according to claim 1, characterized in that: Step 1.1 image preprocessing The input of the preprocessing module is the original image, which is scaled in size to W x H; wherein H represents the height of the image, and W represents the width of the image; the image is randomly flipped and rotated; Step 1.2 image division In order to convert the input information into sequence embedding, the Patch Partition layer is used to compress the size of the preprocessed image, thereby reducing the length of the sequence; the pixels of the medical image are divided into non-overlapping patches with a size of N x N; through this division method, each patch window is regarded as a "token", The feature is set to the splicing of the original pixel RGB value, that is, 3, so the feature dimension of each patch becomes N x N x 3; N is set to 4, the Patch Partition is a convolution operation, the convolution kernel size is 4 x 4, and the convolution step size is 4; Step 1.3, the feature mapping is performed by using the linear embedding layer, the projected feature dimension is converted to any dimension C, thereby obtaining the feature map; Step 2, the Swin-UNet++ module extracts image features through multiple down-sampling processes and up-sampling processes, and fuses features at multiple levels; the input data of the module is the feature map output by step 1, and the size is W / 4 x H / 4 x C; the output of the module is the extracted feature map, and the size is W / 4 x H / 4 x C; The down-sampling process is realized by an encoder; the encoder is connected alternately by a SwinBlock layer and a patch merging layer; the input data of the encoder is the feature map W / 4 x H / 4 x C output by step 1; the output of the encoder is the extracted feature map W / 32 x H / 32 x 8C; The encoder structure is: input data→first Swin Block layer→patch merging layer→second Swin Block layer→patch merging layer→third Swin Block layer→patch merging layer→bottleneck layer The Swin Block layer is composed of a single Swin Transformer module connection; the patch merging layer selects elements at a certain distance in the row and column directions of the patch, splices each part together as a tensor, and finally expands; at this time, the channel dimension is adjusted through channel normalization and a fully connected layer; through this processing, the feature resolution is reduced and the feature dimension is increased, thereby realizing down-sampling; the last patch merging layer in the encoder is connected with the first Swin Transformer Block in the bottleneck layer; the bottleneck layer is composed of two Swin Transformer Blocks connected in turn, and the two Swin Transformer Blocks in the bottleneck layer are connected with the output of the last patch merging layer in the encoder and the input of the bottommost patch expansion layer in the up-sampling stage through channel normalization operation and fully connected layer; that is, the bottommost patch expansion layer in the up-sampling stage is connected with the second Swin Transformer Block in the bottleneck layer; On the basis of the encoder, the context information of the image is fully understood through layer-by-layer upsampling and feature fusion; the feature fusion is realized by multiple skip connections and patch expansion layers; the output of the same dense block of the previous SwinBlock layer is fused with the up-sampling output corresponding to the next dense block through the skip connection; the patch expansion layer first expands the channel number through a fully connected layer, rearranges the feature maps of adjacent dimensions into larger feature maps through the rearrange operation, and then performs channel normalization processing through the LayerNorm, so as to realize the up-sampling; B i,j denotes the SwinBlock layer B i,j output of feature learning, where i indexes the down-sampling layers of the encoder, j indexes the SwinBlock layers along the skip path, and it is set that each Swin Block layer contains two Swin Transformer Blocks; The steps of the Swin-UNet++ architecture for feature fusion are as follows: 1). B 0,0 corresponding to B 1,0 up-sampling output fusion, B 0,1 ; 2). After 1), B 1,0 and B 2,0 corresponding up-sampling output fusion B 1,1 ; B 1,1 up-sampling output and B 0,0 , B 0 ,1 fusion through the jump connection B 0,2 ; 3). Through layer-by-layer upsampling and feature fusion, B 2,1 , B 1,2 is obtained in turn 0,3 ; B 0,3 is the upsampling output of B 1,2 and the fusion result of B 0,0 , B 0,1 , B 0,2 ​ The structure of a single SwinTransformerBlock is as follows: feature map input data→LN regularization layer→W-MSA sub-module or SW-MSA sub-module→residual connection layer→LN regularization layer→feedforward network MLP→residual connection layer→output feature map; the structure of the SwinTransformerBlock in the Swin Block layer is cycled twice, and the odd layers and the even layers are connected in turn with an interval, wherein the odd layers use the W-MSA sub-module and the even layers use the SW-MSA sub-module; The W-MSA sub-module adopts a regular window division strategy, and the feature map is uniformly divided into windows from the top-left pixel, and each window contains a patch; the dependence of the features in different subspaces is captured by calculating the attention vector in the subspace, that is, the attention vector is calculated by point multiplication operation in all subspaces, and finally the attention vectors calculated in all subspaces are spliced and mapped to the original input space to obtain the final attention vector as the output, so as to better obtain local and global semantic information; The SW-MSA sub-module performs a pixel displacement operation on the image features before inputting the image features, and then performs the W-MSA sub-module operation; the window partition is moved, generating a new window partition, and introducing the connection between adjacent non-overlapping windows; Using the shift window partition method, the continuous SwinTransformerBlock calculation is as follows: wherein z l-1 denote the input feature of the Swin Transformer block, and z l denote the output feature of the W-MSA module and the MLP module in the l-th SwinTransformer block, respectively; and z l+1 denote the output feature of the SW-MSA module and the MLP module in the l+1-th SwinTransformer block, respectively; W-MSA and SW-MSA respectively use the rule and the shift window partition to calculate the self-attention in each window: first multiply the query vector Q and the key vector K, then divide by the square root of the dimension of the key vector K to get the score matrix of the query vector Q, and add the relative position bias, finally normalize the score matrix by the SoftMax function to get the weight matrix, and then multiply by the value vector V to get the attention vector of a subspace; wherein Q, K, V represent query matrix, key matrix and value matrix respectively, d represents the dimension of the matrix; when calculating the similarity, a relative position deviation is introduced, B represents the bias matrix, and SoftMax is an activation function; specifically, after passing through a LayerNorm regularization layer, three matrices Q, K and V are obtained in W-MSA by using a full connection layer, and the self-attention is calculated based on the above formula, and then dropout is performed, and then the output obtained is added to z l-1 to obtain After sequentially passing through the LN layer and the MLP layer, dropout is performed, and then z is obtained by residual connection l ; z l After passing through a LayerNorm regularization layer, the image feature map is shifted by half the window size through SW-MSA, and each cycle is shifted up and left by half the window size, then the window is divided according to W-MSA, and after running, the obtained feature map is shifted down and right by half the window size, and is restored to the original position; after calculating the self-attention based on the above formula, dropout is performed, and then z l is obtained by residual connection After sequentially passing through the LN layer and the MLP layer, dropout is performed, and then z is obtained by residual connection l+1 ; and the LN regularization layer is to perform LN regularization processing on the input data, and to make it between 0 and 1; the residual connection layer is to perform residual connection, using addition operation to directly add the input and the output; the MLP layer is composed of two full connection layers, the input feature passes through a full connection layer to obtain an intermediate feature, and the dimension thereof is set to 4 times the dimension of the input feature, then passes through an activation function GELU, and then passes through a full connection layer to obtain an output feature.

Images