Melanoma image segmentation method based on Transformer-based feature supplementation

Abstract

The present invention discloses a Transformer-based feature-supplemented melanoma image segmentation method, comprising: extracting feature maps layer0 to layer4 of different scales from a melanoma image using a ResNet network; positionally embedding the feature maps layer0 to layer4, then feeding layer1 to layer4 into the Transformer of a segmentation model to obtain feature maps m1 to m4 of different scales; processing the feature maps m1 to m4 sequentially through the reconstruction feature module of the segmentation model; feeding shallow features layer0 of the encoder end and decoder features into the feature pyramid module of the segmentation model for merging and performing feature lateral connection; then training the segmentation model; feeding the melanoma image to be segmented into the trained segmentation model to obtain corresponding segmentation result maps and corresponding indicators. This method effectively extracts global visual information, retains shallow information to establish a relationship between the encoder and decoder, and compensates for the spatial information loss problem caused by downsampling and upsampling.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image segmentation, and in particular to a Transformer-based feature-supplemented melanoma image segmentation method. Background Art

[0002] Melanoma is a malignant tumor that typically appears on the skin, eyes, and mucous membranes, accounting for approximately 3% of all tumors. In the early stages, non-invasive imaging techniques, such as dermatoscopy, are often used to screen for melanoma. However, manual screening by the physician is subjective, time-consuming, and requires high skill and experience. Furthermore, the inherent visual similarities between different skin conditions, such as melanomas, nevi, and seborrheic keratoses, make it difficult even for expert dermatologists to distinguish them. Melanoma image segmentation technology can help physicians more accurately identify and define lesions, leading to more effective diagnosis and treatment. It can also provide physicians with more detailed information, helping them better assess the condition and achieve more effective treatment.

[0003] Although most currently proposed melanoma skin lesion image segmentation models are based on Transformer networks, which use a self-attention mechanism to model long-range connections and dependencies between features, they can overcome the local nature of convolutional neural networks and capture a wider range of contextual information. However, their high graphics card requirements make them unsuitable for small tasks, and they lack the ability to capture detailed information or fuse and filter features acquired from channels at different scales. To address the shortcomings of existing melanoma skin lesion segmentation models, the Transformer structure is used to capture long-range information. On this basis, convolutional layers are combined to design feature supplementation and feature fusion modules. For example, Swin-Unet replaces convolutional blocks with Transformer modules, extracting image feature representations through multiple layers of grouped convolutions and SwinTransformer blocks. These are then combined with the encoder's feature representations for feature fusion, ultimately outputting a segmentation mask. UCTransNet leverages the Transformer's advantages in long-range dependency modeling to fuse multi-scale encoder features, addressing semantic gaps and enabling automatic medical image segmentation.

[0004] The aforementioned studies are all based on Transformer design, ensuring the model's ability to capture long-range information. However, these studies ignore the intrinsic pixel-level structural features within each image block and fail to fully capture local feature information. This results in blurred lesion boundary segmentation, or even fusion with the surrounding skin. Summary of the Invention

[0005] The purpose of this paper is to propose a Transformer-based feature-supplemented melanoma image segmentation method, FAuNet. This method effectively extracts global visual information, retains shallow information, establishes a connection between the encoder and decoder, and compensates for the spatial information loss caused by downsampling and upsampling. It also enriches feature information through lateral connections, improving the robustness of the model.

[0006] To achieve the above objectives, this application proposes a Transformer-based feature-supplemented melanoma image segmentation method, including:

[0007] Using the ResNet network as the backbone network of the segmentation model, we extracted feature maps of different scales, layer0 to layer4, from the melanoma image.

[0008] Position embedding is performed on the feature maps layer0 to layer4, and then layer1 to layer4 are fed into the Transformer of the segmentation model to dynamically obtain the relative importance of each position relative to other positions and integrate them to obtain feature maps m1 to m4 of different scales;

[0009] The feature maps m1 to m4 are processed in sequence by the reconstructed feature (Refa) module of the segmentation model;

[0010] The shallow features layer0 of the encoder side and the decoder features are fed into the feature pyramid (iFPN) module of the segmentation model for merging and horizontal feature connection; then the segmentation model is trained;

[0011] The melanoma image to be segmented is sent to the trained segmentation model to obtain the corresponding segmentation result map and corresponding indicators.

[0012] Furthermore, the position embedding method for the feature maps layer0 to layer4 is as follows:

[0013] Feature Map i=1,2,3,4Tokenization is used for position embedding, where H represents height, W represents width, and C represents the number of channels; specifically: the features are re-divided into 2D image blocks, so that these image block sequences are mapped to the same area of ​​the encoder features at four scales; in this process, the original channel size is maintained;

[0014] The four layers of Token T i , i=1,2,3,4, Connect, T Σ =Concat(T1, T2, T3, T4) as key and value.

[0015] Furthermore, the token is fed into the Transformer of the segmentation model to dynamically obtain the relative importance of each position relative to other positions and integrate them to obtain feature maps m1 to m4 of different scales, specifically:

[0016] The token is sent to the Transformer to obtain global context information. The Transformer includes a multi-head channel cross attention module and a multi-layer perceptron with a residual structure; the channel relationship and feature dependency are obtained by equations (1) and (2);

[0017] The multi-head channel cross attention module has 5 inputs and contains 4 layers of T i and a T Σ As key and value;

[0018]

[0019]

[0020] in, It is generated by the multi-channel cross attention module in Transformer; is the weight of different inputs, d is the sequence length, C i ,i=1,2,3,4 are the channel sizes of the four skip connection layers; in the case of N-head attention, the output of the multi-head channel cross attention module is the average value of the attention obtained by applying simple MLP and residual operations;

[0021] The Transformer’s self-attention mechanism is used to model the long-distance connections and dependencies between features to obtain feature maps m1 to m4.

[0022] Furthermore, the feature maps are processed in sequence by the reconstruction feature module of the segmentation model, specifically:

[0023] The feature map m4 is processed by the reconstruction feature module. The core component of the reconstruction feature module, PixelShuffle_ICNR, uses inverse convolution and pixel shuffling technology to achieve image upsampling: first, the feature maps m1 to m4 are divided into multiple small blocks, and the size of each small block is (w, h, c1), where w and h represent the width and height of the small block respectively, and c1 represents the number of channels in each small block;

[0024] The number of channels c1 in each small block is divided into two parts, namely c2 and c3; then, the pixels in each small block are reconstructed into a new block with a size of (w×s,h×s,c2+c3), where s is the magnification factor; the pixels of each small block are evenly distributed to the pixels of the high-resolution image;

[0025] The pixels in the new block are rearranged, and the tensor with smaller number of channels is upsampled while the number of channels is increased.

[0026] Furthermore, the specific method of PixelShuffle_ICNR upsampling is as follows:

[0027] Reconstruct the image:

[0028] y=W*x+b (3)

[0029] Where W is a learnable weight matrix, x is the input image, b is a bias, and y is the reconstructed image;

[0030] PixelShuffle Operation:

[0031] y=PixelShuffle(A) (4)

[0032] Where A is the reconstructed image and y is the image after the PixelShuffle operation.

[0033] Furthermore, the feature map sampled by PixelShuffle_ICNR is concatenated with the feature map m3 of the previous layer to form a higher-dimensional feature, and then processed using Dropout2d.

[0034] Then use an activation function Relu to adjust the data, and then process it through two convolutional layers and Dropout2d layers to obtain the feature map dec4;

[0035] Sequentially, m3, m2, and m1 are processed by the reconstruction feature module to extract feature information with stronger discrimination ability and richer classification features; finally, feature maps dec3 to dec1 are obtained.

[0036] Furthermore, the shallow features layer0 of the encoder side are fed into the feature pyramid module of the segmentation model and merged with the decoder features for horizontal feature connection, specifically:

[0037] Upsample the feature maps dec3 to dec1 respectively so that their feature map sizes are the same as dec4, which is 128×128.

[0038] The upsampled feature maps are filtered through a convolutional layer, a Dropout2D layer, a Relu layer, a BatchNorm2D layer, and another convolutional layer to uniformly adjust them to a feature map with 16 channels.

[0039] Concatenate the four feature maps and the shallow feature map layer0 on the encoder side into a feature map with 96 channels;

[0040] A convolution layer and an upsampling layer are used to fuse the features of the concatenated feature map.

[0041] Compared with the prior art, the above technical solution adopted by the present invention has the following advantages: the present invention uses Transformer to obtain global information, while retaining the shallowest information to establish the relationship between the encoder and the decoder, compensating for the spatial information loss problem caused by the downsampling and upsampling process. It helps the model learn more edge information and texture information, thereby enhancing the model's ability to extract image details and helping to improve the performance of the model. In addition, the reconstruction feature module is used to gradually restore the features, thereby improving the model's ability to express the output data, enabling it to better understand the semantic information in the input data and improving the model's segmentation and reconstruction capabilities. The iFPN module is combined to perform horizontal feature connection to enrich the feature information, effectively improving the segmentation performance of the entire model. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 This is a diagram of the Transformer-based feature-supplemented melanoma image segmentation network framework of the present invention;

[0044] Figure 2 This is a structural diagram of the reconstruction feature module ReFa of the present invention;

[0045] Figure 3 This is the structure diagram of the enhanced feature pyramid module iFPN of the present invention;

[0046] Figure 4 Segmentation diagram on the ISIC2018 dataset using different comparison networks. Specific implementation methods

[0047] In order to make the purpose, technical solutions and advantages of this application more clearly understood, this application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application. That is, the embodiments described are only part of the embodiments of this application, not all of them.

[0048] This embodiment provides a Transformer-based feature-supplemented melanoma image segmentation method, including:

[0049] S1. Using the ResNet network as the backbone network of the segmentation model, we extract feature maps of different scales, layer0 to layer4, from the melanoma image.

[0050] Specifically, the melanoma image was cropped to a size of 256×256, and then flipped and rotated as the input of the ResNet network.

[0051] S2. Position embedding is performed on the feature maps layer0 to layer4, and then layers1 to layer4 are fed into the Transformer of the segmentation model to dynamically obtain the relative importance of each position relative to other positions and integrate them to obtain feature maps m1 to m4 of different scales. Specifically,

[0052] Step 2.1: First, the feature map (i=1,2,3,4) Tokenization is used for position embedding, where H represents height, W represents width, and C represents the number of channels. The specific approach is: the features are re-divided into The 2D image patches are mapped to the same region of the encoder features at four scales. In this process, the original channel size is maintained.

[0053] Step 2.2: Connect the Token T of the four layers i (i=1,2,3,4), As key, T Σ =Concat(T1,T2,T3,T4) as value.

[0054] Step 2.3: Send the token into the Transformer to obtain global context information. It uses the Transformer structure in UCTransNet, which includes a multi-head channel cross attention module and a multi-layer perceptron with a residual structure. The channel relationship and feature dependency are obtained by equations (1) and (2). The multi-head channel cross attention module has 5 inputs and contains 4 layers of T i As a query, a T ΣAs key and value:

[0055]

[0056]

[0057] in, It is generated by the multi-channel cross-attention module in Transformer. is the weight of different inputs, d is the sequence length, C i (i = 1, 2, 3, 4) are the channel sizes of the four skip connection layers. In our implementation, C1 = 64, C2 = 128, C3 = 256, and C4 = 512. In the case of N-head attention, the output of the multi-head channel cross-attention module is the average of the attention values ​​calculated using a simple MLP and residual operation.

[0058] Step 2.4: The Transformer’s self-attention mechanism is used to model the long-distance connections and dependencies between features to obtain feature maps m1 to m4.

[0059] S3. Process the feature maps m1 to m4 sequentially through the reconstruction feature module of the segmentation model;

[0060] Step 3.1: The feature map m4 is processed by the reconstruction feature module. The core component of this module, PixelShuffle_ICNR, uses inverse convolution and pixel shuffling technology to achieve image upsampling: first, the feature maps m1 to m4 are divided into multiple small blocks, and the size of each small block is (w, h, c1), where w and h represent the width and height of the small block respectively, and c1 represents the number of channels in each small block.

[0061] Step 3.2: Split the number of channels c1 in each patch into two parts, c2 and c3. Then, reconstruct the pixels in each patch into a new patch of size (w×s,h×s,c2+c3), where s is the magnification factor. The pixels in each patch are evenly distributed among the pixels in the high-resolution image, thus avoiding angularity.

[0062] Step 3.3: Rearrange the pixels in the new block so that they can be combined into a larger high-resolution image. This can upsample the tensor with a smaller number of channels while increasing the number of channels.

[0063] The specific formula for PixelShuffle_ICNR upsampling is as follows:

[0064] Reconstruct the image:

[0065] y=W*x+b (3)

[0066] Where W is a learnable weight matrix, x is the input image, b is a bias, and y is the reconstructed image.

[0067] PixelShuffle Operation:

[0068] y=PixelShuffle(A) (4)

[0069] Where A is the reconstructed image and y is the image after the PixelShuffle operation.

[0070] Step 3.4: Concatenate the feature map sampled by PixelShuffle_ICNR with the feature map m3 of the previous layer to form a higher-dimensional feature. Use Dropout2d to reduce the number of parameters in the neural network and the complexity of the network, thereby shortening the training time and increasing the speed of the network. At the same time, it prevents the neural network from relying on certain specific neurons during training, thereby improving the generalization ability of the network.

[0071] Step 3.5: Next, we use an activation function (ReLU) to adjust the data to avoid the vanishing gradient problem. Then we pass it through two convolutional layers and a Dropout2d layer.

[0072] Step 3.6: In turn, m3, m2, and m1 are processed by the reconstruction feature module to extract feature information with stronger discrimination ability and rich classification features. Finally, feature maps dec4 to dec1 are obtained, and their sizes are 256×16 respectively. 2 , 128×32 2 , 64×64 2 and 32×128 2 .

[0073] S4. The shallow features layer0 of the encoder side and the decoder features are fed into the feature pyramid module of the segmentation model for merging and horizontal feature connection. The segmentation model is then trained. Specifically:

[0074] Step 4.1: Upsample the feature maps dec3 to dec1 respectively so that their feature map size is the same as dec4, which is 128×128.

[0075] Step 4.2: The feature maps described in 4.1 are filtered through a convolutional layer, a Dropout2D layer, a Relu layer, a BatchNorm2D layer, and another convolutional layer, and are uniformly adjusted to feature maps with 16 channels.

[0076] Step 4.3: Concatenate the above four feature maps and the shallow feature map layer0 on the encoder side into a feature map with 96 channels to help the model learn more edge information and texture information, thereby enhancing the network's ability to extract image details.

[0077] Step 4.4: Use a convolutional layer and an upsampling layer to fuse the features of the concatenated feature map.

[0078] S5. Send the melanoma image to be segmented into the trained segmentation model to obtain the corresponding segmentation result map and corresponding indicators.

[0079] Specifically, the obtained segmented image is compared with the original image to calculate the accuracy, Jaccard index and Dice coefficient.

[0080] This experiment was conducted on an NVIDIA GeForce RTX 2060 graphics card using Windows as the development environment and PyCharm as the development platform. The main components included Python 3.7, CUDA 11.1, cuDNN 8.0.5, and PyTorch 1.8. To avoid overfitting, data augmentation techniques were used, including transformations such as flipping and rotation. The model was trained using the Adam optimizer with a batch size of 4. The initial learning rate was set to 6e×10 -4 The learning rate is reduced by a factor of 10 for every 20 epochs, which becomes 6e×10 -5 , thereby smoothly adjusting the learning rate. To improve training stability, cross entropy loss and Dice loss are selected as the training loss functions. The datasets used are ISIC2017 and ISIC2018. The evaluation metrics used include accuracy (AC), Jaccard index (JA), and Dice coefficient (DI).

[0081] The specific expressions are as follows:

[0082]

[0083]

[0084]

[0085] Among them, TP (True Positive) represents true positive examples, TN (True Negative) represents true negative examples, FP (False Positive) represents false positive examples, and FN (False Negative) represents false negative examples.

[0086] Following the above steps, the proposed method was compared with the U-Net model, SwinUNet model, UCTransNet model, TransUNet model, UTNetV2 model, UNeXt-S model, MALUNet model, and TransFuse model. Bold numbers indicate the best performance in each column. The numbers in the table are expressed as percentages of the segmentation metrics. As can be seen in Table 1, the proposed method achieves substantially better accuracy than other methods on both common datasets.

[0087] Table 1: Comparison of semantic image synthesis results of different models

[0088]

[0089] at the same time Figure 4 The melanoma segmentation effect diagram of the present invention is shown. It can be seen that the segmentation result of the present invention is closer to the real boundary and the details are better processed.

[0090] In this application, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed or that are inherent to such process, method, article, or apparatus.

[0091] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A Transformer-based feature-supplemented melanoma image segmentation method, characterized by: include: Using the ResNet network as the backbone network of the segmentation model, we extracted feature maps of different scales, layer0 to layer4, from the melanoma image. Position embedding is performed on the feature maps layer0 to layer4, and then layer1 to layer4 are fed into the Transformer of the segmentation model to dynamically obtain the relative importance of each position relative to other positions and integrate them to obtain feature maps m1 to m4 of different scales; The feature maps m1 to m4 are processed sequentially through the reconstruction feature module of the segmentation model; The shallow features layer0 of the encoder side and the decoder features are fed into the feature pyramid module of the segmentation model for merging and horizontal feature connection; then the segmentation model is trained; The melanoma image to be segmented is sent to the trained segmentation model to obtain the corresponding segmentation result map and corresponding indicators.

2. The feature-supplemented melanoma image segmentation method based on Transformer according to claim 1, characterized in that: The position embedding method for the feature maps layer0 to layer4 is: Feature Map i=1,2,3,4Tokenization is used for position embedding, where H represents height, W represents width, and C represents the number of channels; specifically: the features are re-divided into 2D image blocks, so that these image block sequences are mapped to the same area of ​​the encoder features at four scales; in this process, the original channel size is maintained; The four layers of Token T i , i=1,2,3,4, Connect, T Σ =Concat(T1, T2, T3, T4) as key and value.

3. The feature-supplemented melanoma image segmentation method based on Transformer according to claim 2, characterized in that: The token is fed into the Transformer of the segmentation model, which dynamically obtains the relative importance of each position relative to other positions and integrates them to obtain feature maps m1 to m4 of different scales, specifically: The token is sent to the Transformer to obtain global context information. The Transformer includes a multi-head channel cross attention module and a multi-layer perceptron with a residual structure; the channel relationship and feature dependency are obtained by equations (1) and (2); The multi-head channel cross attention module has 5 inputs and contains 4 layers of T i and a T Σ As key and value; Q i =T i W Qi ,K=T Σ W K ,V=T Σ W V (2) in, It is generated by the multi-channel cross attention module in Transformer; is the weight of different inputs, d is the sequence length, C i ,i=1,2,3,4 are the channel sizes of the 4 skip connection layers; The Transformer’s self-attention mechanism is used to model the long-distance connections and dependencies between features to obtain feature maps m1 to m4.

4. The feature-supplemented melanoma image segmentation method based on Transformer according to claim 3, characterized in that: The feature maps are processed in sequence by the reconstruction feature module of the segmentation model, specifically: The feature map m4 is processed by the reconstruction feature module. The core component of the reconstruction feature module, PixelShuffle_ICNR, uses inverse convolution and pixel shuffling technology to achieve image upsampling: first, the feature maps m1 to m4 are divided into multiple small blocks, and the size of each small block is (w, h, c1), where w and h represent the width and height of the small block respectively, and c1 represents the number of channels in each small block; The number of channels c1 in each small block is divided into two parts, namely c2 and c3; then, the pixels in each small block are reconstructed into a new block with a size of (w×s,h×s,c2+c3), where s is the magnification factor; the pixels of each small block are evenly distributed to the pixels of the high-resolution image; The pixels in the new block are rearranged, and the tensor with smaller number of channels is upsampled while the number of channels is increased.

5. The feature-supplemented melanoma image segmentation method based on Transformer according to claim 4, characterized in that: The specific method of PixelShuffle_ICNR upsampling is as follows: Reconstruct the image: y=W*x+b(3) Where W is a learnable weight matrix, x is the input image, b is a bias, and y is the reconstructed image; PixelShuffle Operation: y=PixelShuffle(A) (4) Where A is the reconstructed image and y is the image after the PixelShuffle operation.

6. The feature-supplemented melanoma image segmentation method based on Transformer according to claim 5, characterized in that: The feature map sampled by PixelShuffle_ICNR is concatenated with the feature map m3 of the previous layer to form a higher-dimensional feature, and then processed by Dropout2d; Then use an activation function Relu to adjust the data, and then process it through two convolutional layers and Dropout2d layers to obtain the feature map dec4; Sequentially, m3, m2, and m1 are processed by the reconstruction feature module to extract feature information with stronger discrimination ability and richer classification features; finally, feature maps dec3 to dec1 are obtained.

7. The feature-supplemented melanoma image segmentation method based on Transformer according to claim 1, characterized in that: The shallow features layer0 of the encoder side and the decoder features are fed into the feature pyramid module of the segmentation model for merging and performing horizontal feature connection. Specifically: Upsample the feature maps dec3 to dec1 respectively so that their feature map sizes are the same as dec4, which is 128×128. The upsampled feature maps are filtered through a convolutional layer, a Dropout2D layer, a Relu layer, a BatchNorm2D layer, and another convolutional layer to uniformly adjust them to a feature map with 16 channels. Concatenate the four feature maps and the shallow feature map layer0 on the encoder side into a feature map with 96 channels; A convolution layer and an upsampling layer are used to fuse the features of the concatenated feature map.

Images