A melanoma image segmentation method

Abstract

This invention discloses a method for segmenting melanoma images, belonging to the field of medical artificial intelligence. The method includes the following steps: S1, acquiring image data and preprocessing the image; S2, extracting features from the preprocessed image using an encoding module; S3, controlling the contribution of feature information at each stage through a gate structure, fusing deep and shallow features; S4, concatenating the output of the gate structure with the output of the decoding module by channel, and upsampling to obtain a multi-channel feature map; S5, sending the data to a segmentation head and outputting a binary black-and-white segmentation image. This invention uses a feature fusion gate structure to filter and fuse the feature information output by the encoding module, avoiding feature information redundancy caused by simple feature fusion. It adopts a bottom-up, deep-to-shallow fusion order, which ensures mutual compensation of effective information while reducing computational complexity, offering the advantages of semantically rich feature fusion and high efficiency.

Description

Technical Field

[0001] This invention belongs to the field of medical artificial intelligence, and specifically relates to a method for image segmentation of melanoma. Background Technology

[0002] Melanoma is a highly malignant skin cancer with an increasing incidence rate. Early diagnosis and treatment are crucial for improving patient survival rates. The development of image segmentation technology for melanoma has gone through three main stages: traditional image processing methods, machine learning methods, and deep learning methods.

[0003] In early studies of melanoma image segmentation, researchers primarily employed traditional image processing techniques such as thresholding and edge detection. While these methods are simple and easy to use, they struggle to achieve accurate segmentation for complex images. As research progressed, machine learning methods were introduced into melanoma image segmentation. Methods like random forests and support vector machines (SVMs) can automatically learn image features and produce relatively good segmentation results. However, these methods require appropriate feature engineering and the involvement of medical experts, and they also have difficulty handling high-dimensional image data.

[0004] In recent years, the rapid development of computer vision and deep learning technologies has brought significant breakthroughs to melanoma image segmentation. Convolutional Neural Networks (CNNs), in particular, have demonstrated outstanding performance in image recognition and segmentation tasks. A neural network model has been trained using a dataset labeled by multiple medical experts. This trained model is then used to segment input images, and the segmentation result, i.e., the model's output, is a segmentation mask corresponding to the input image, providing new possibilities for the automatic identification and segmentation of melanoma. In the task of segmenting melanoma images, the U-Net network is a typical and efficient convolutional neural network architecture. Proposed by Ronneberger et al. in 2015, the U-Net network aims to solve the problem of medical image segmentation. Based on a U-shaped structure, the network consists of two parts: a contracting path and an expanding path. The contracting path contains multiple convolutional layers, activation functions, and pooling layers to extract local features of the input image. The expanding path contains multiple upsampling layers, convolutional layers, and activation functions to recover the spatial information of the image. Skip connections fuse feature maps from different levels, improving the fineness of the segmentation results. Later studies found that, for the scale variations, shape irregularities, and diversity of skin lesions, the traditional U-Net network cannot demonstrate robustness and segmentation accuracy in analyzing targets at different scales.

[0005] To improve the robustness of the network, its depth and width can be increased. Considering the high cost of labeling medical images and the limited number of corresponding datasets, data augmentation can be used while reducing the network's trainable parameters to improve convergence speed and robustness. Simple fusion of feature maps can generate redundant information. It is necessary to ensure both high-level semantic information and the contextual information of the features. Therefore, a fusion mechanism between feature maps needs to be designed to adjust the contribution of fused information and improve the accuracy of the network segmentation results.

[0006] In summary, U-Net, as an efficient and scalable convolutional neural network, has certain advantages in the application of skin melanoma image segmentation. However, there is still room for improvement in its robustness to the analysis of skin melanomas with multiple scales and diverse shapes. By improving and optimizing this network, the accuracy of automatic identification and segmentation of skin melanoma can be further enhanced, thus promoting the level of diagnosis and treatment in the field of skin cancer. Summary of the Invention

[0007] In view of the above-mentioned problems in the existing technology, the purpose of the present invention is to provide a method for image segmentation of melanoma.

[0008] To solve the above problems, the technical solution adopted by the present invention is as follows:

[0009] A method for segmenting melanoma images includes the following steps:

[0010] S1. Acquire image data and preprocess the image;

[0011] S2. Feature extraction is performed on the preprocessed image through the encoding module;

[0012] S3. By controlling the contribution of feature information at each stage through gate structure, deep features are fused with shallow features;

[0013] S4. The output of the gate structure and the output of the decoding module are concatenated by channel, and upsampling is performed to obtain a multi-channel feature map;

[0014] S5. Input the segmentation head and output a binary black and white segmentation image.

[0015] As a preferred embodiment of the present invention, image preprocessing is performed in step S1 according to the following method:

[0016] S101. Convert the RGB medical image to a grayscale image;

[0017] S102. Use the Resize operation to crop the image. Size.

[0018] In a preferred embodiment of the present invention, the encoding module consists of Improved-Inceptionv4 and downsampling, and obtains feature outputs at multiple scales through the encoding module. The proposed Improved-Inceptionv4 structure can provide more spatial context information for the fusion structure and compensate for the increased number of parameters caused by the introduction of the fusion structure.

[0019] As a preferred embodiment of the present invention, the Improved-Inceptionv4 structure consists of five branches: Branch 1: 1×1 convolution, 3×3 convolution, 3×3 convolution; Branch 2: 1×1 convolution, 3×3 convolution; Branch 3: 1×5 convolution, 5×1 convolution; Branch 4: 1×3 convolution, 3×1 convolution; Branch 5: 3×3 depthwise convolution, 1×1 pointwise convolution. Batch normalization and ReLU activation functions are used between the convolution modules of Branch 1 to Branch 4.

[0020] As a preferred embodiment of the present invention, the encoding module obtains feature maps of multiple scales through multiple Improved-Inceptionv4 convolution and downsampling operations. , arrive The feature level increases progressively, the number of channels doubles progressively, the size is halved progressively, and the semantic information is enriched progressively.

[0021] As a preferred embodiment of the present invention, the feature fusion process fuses deep features and shallow features from bottom to top, specifically referring to the following method for fusing the extracted features: Combined with all fused output values ​​below this layer Obtained through feature fusion gate structure Underlying features Skip the feature fusion gate structure to obtain directly First output Then output And so on until the output is reached. The entire feature fusion process adopts a bottom-up fusion order, which can ensure mutual compensation of effective information and reduce computational complexity.

[0022] As a preferred embodiment of the present invention, bilinear interpolation is used for upsampling in step S4.

[0023] As a preferred embodiment of the present invention, the segmentation head outputs 2 channels, converts the obtained multi-channel feature map into a 2-channel feature map, and the output result is a black and white binary image, where the black area represents the background and the white area represents the target object.

[0024] As a preferred embodiment of the present invention, the model constructed in steps S1 to S5 is trained using the ISIC2018 dataset and a binary cross-entropy loss function (BCEWithLogitsLoss) before use. During the training process, the network weights are continuously updated and optimized, and the best network weight file is saved.

[0025] As a preferred embodiment of the present invention, the binary classification cross-entropy loss function is expressed as follows:

[0026] ;

[0027] in, Indicates sample The true label (0 or 1). This represents the score predicted by the model. For the Sigmoid function, This represents the number of samples.

[0028] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0029] 1. This invention uses a feature fusion gate structure to filter and fuse the feature information output by the encoding module, avoiding feature information redundancy caused by simple feature fusion, so that the output of the shallow encoding module has both rich spatial context information and rich high-level semantic information.

[0030] 2. In the feature fusion process, the present invention fuses deep features and shallow features in a bottom-up order. The bottom-up fusion order of the entire feature fusion process can ensure mutual compensation of effective information and reduce computational complexity.

[0031] 3. This invention upsamples the fused output and the output of the decoding module by splicing them together, so that the output of the decoding module, which is close to the segmentation head, contains richer effective spatial context information and high-level semantic information.

[0032] 4. The present invention uses the designed Improved-Inceptionv4 structure for feature extraction, which can provide more spatial context information for the fusion structure and make up for the problem of increased parameters caused by the introduction of the fusion structure. Furthermore, the improved Inception structure—Improved-Inceptionv4—widens the width of the original convolutional module to enhance the network's performance and robustness, while reducing the number of network parameters. Attached Figure Description

[0033] Figure 1 This describes the segmentation process for the U-Net network based on the feature fusion gate structure and the Improved-Inceptionv4 structure.

[0034] Figure 2 This is a diagram showing the overall architecture of the U-Net network based on the feature fusion gate structure and the Improved-Inceptionv4 structure.

[0035] Figure 3 This is a diagram of the Improved-Inceptionv4 architecture.

[0036] Figure 4 This is a structural diagram of the feature fusion gate structure. Detailed Implementation

[0037] The present invention will be further described below with reference to specific embodiments.

[0038] This paper proposes a U-Net segmentation network based on a feature fusion gate structure and an Improved-Inceptionv4 structure. The method uses U-Net as the main network and employs a feature fusion gate structure to filter and fuse the feature information output from the encoding module. A bottom-up fusion order ensures effective fusion of feature information at each stage while avoiding repeated back-and-forth fusion, reducing computational costs and preventing feature redundancy caused by simple feature fusion. This results in the output of the shallow encoding module containing rich spatial context and high-level semantic information. The fused output is then concatenated with the output of the decoding module and upsampled, ensuring that the decoder output is closer to the segmentation head and contains richer effective spatial context and high-level semantic information. Furthermore, to further enhance network performance, an improved Inception structure—Improved-Inceptionv4—is used. This effectively addresses the increased number of trainable parameters caused by the feature fusion result and provides richer spatial context information for subsequent feature fusion, widening the original convolutional module and enhancing network performance and robustness. These two structures complement and promote each other, improving network performance. The proposed method addresses the problems of traditional segmentation networks being insensitive to scale variations and shape irregularities in skin lesion areas, the imbalance between contextual and high-level semantic information in feature extraction, and information redundancy caused by simple feature fusion.

[0039] Example 1

[0040] The present invention is achieved through the following technical solution:

[0041] First, the training data is preprocessed appropriately and used as input to the network for model training. Then, the specific network structure is built, hyperparameters are set, and the network is trained. Finally, the trained model is tested on test data to observe the network's segmentation performance. Specifically, the following steps are included:

[0042] Step 1: Dataset Preprocessing

[0043] Obtaining the ISIC2018 dataset and preprocessing the medical images is beneficial for the network's rapid convergence and improves its training performance. The ISIC2018 dataset consists of RGB images. Preprocessing operations include converting RGB images to grayscale and cropping them to the appropriate size. This processing can convert the 3 channels of an image into 1 channel, effectively reducing the data dimensionality and helping the model to process and analyze the image in the future. It also greatly reduces the computational complexity and time cost of the network. Moreover, medical images pay more attention to texture and shape features, and grayscale images can highlight the texture and shape features of medical images, which helps the model to obtain the feature information of the image.

[0044] Step 2: Building the U-Net model based on the feature fusion gate structure and the Improved-Inceptionv4 structure

[0045] 1. Design features integrate door structure

[0046] In computer vision applications, contextual information helps to better understand and interpret image content, aiding algorithms in determining the boundaries and locations of target objects. Rich spatial contextual information enables models to achieve stable segmentation results even under noise and interference, improving the algorithm's robustness. High-level semantic information can effectively guide segmentation algorithms, improving their accuracy and stability. Typically, segmentation algorithms process images at the pixel level alone, facing many limitations and challenges. For example, similar colors and textures can blur the boundaries between different objects, making it difficult to correctly distinguish object shapes and boundaries. However, high-level semantic information can provide richer semantic context for images, guiding segmentation algorithms to better understand the image and its underlying semantics. Higher-level feature extraction yields more high-level semantic information but less spatial contextual information. This paper designs a feature fusion gate structure that fuses high-level and low-level features, compensating for the lack of high-level semantic information in shallow low-level features. Furthermore, the gate structure selectively controls feature fusion, avoiding information redundancy and effectively enhancing the model's sensitivity to multi-scale, irregular targets. Figure 4 As shown, where to Original features to The fusion result of the feature gate structure, the four-layer encoding outputs are as follows: , , and , arrive The number of channels is doubled sequentially, while the size is halved. This is done through a dimensionality reduction branch and a gated branch. The dimensionality reduction branch consists of a 1×1 convolution, batch normalization, and a ReLU activation function, while the gated branch consists of a 1×1 convolution and a sigmoid activation function. The outputs of the two branches are then summed to obtain the final gate structure output. , , and Then, through center pruning, deep features are fused with shallow features sequentially from bottom to top, thus taking into account both spatial context information and high-level semantic information. This avoids the problems of information redundancy and feature duplication caused by simple fusion, greatly reducing the computational complexity of the model. The feature map contains richer effective feature information, effectively enhancing the model's sensitivity to multi-scale and irregular targets, which is conducive to improving the model's segmentation performance and effect. By fusing deep features with shallow features in a bottom-up order, the entire feature fusion process can ensure mutual compensation of effective information while reducing computational complexity.

[0047] 2. Design the Improved-Inceptionv4 architecture

[0048] Network performance can be improved in terms of both width and depth. Inception is a typical structure that improves network performance by expanding the network width, and it also significantly reduces the number of parameters in the original convolutional modules, saving computational time. Using the Inception structure for feature extraction can provide more spatial context information to the fusion structure and compensate for the increased number of parameters caused by the introduction of the fusion structure. In this embodiment, an improved Inception structure—the Improved-Inceptionv4 structure—is designed. Figure 3 As shown, the improved-Inceptionv4 structure consists of five branches: Branch 1: 1×1 convolution, 3×3 convolution, 3×3 convolution; Branch 2: 1×1 convolution, 3×3 convolution; Branch 3: 1×5 convolution, 5×1 convolution; Branch 4: 1×3 convolution, 3×1 convolution; Branch 5: 3×3 depthwise convolution, 1×1 pointwise convolution. Batch normalization and ReLU activation functions are used between the convolutional modules in branches 1 to 4 to accelerate model convergence, avoid gradient vanishing or exploding, enhance nonlinear expressiveness, increase the nonlinear decision boundary of the model, and thus increase the model's generalization ability. Branch 5, through separable convolution, increases the receptive field of the feature map, captures larger contextual information, extracts more comprehensive features, and also reduces the number of model parameters. By replacing the original convolutional modules with the improved-Inceptionv4 structure, more contextual information is captured, the number of parameters is reduced, and the network performance is improved.

[0049] Step 3: Construct a U-Net network based on the feature fusion gate structure and the Improved-Inceptionv4 structure.

[0050] The U-Net network, based on the feature fusion gate structure and the Improved-Inceptionv4 structure, maintains a U-shaped encoder-decoder structure, as shown in the diagram. Figure 1 As shown, a medical image of melanoma is used as network input. It first passes through a four-layer encoding structure on the left. The encoding module consists of Improved-Inceptionv4 and downsampling, resulting in feature outputs at four different scales. Then, it passes through a feature fusion gate structure for effective feature map fusion. In this embodiment, the bottom-layer features... Skip the feature fusion gate structure to obtain directly ,feature and Obtained through feature fusion gate structure ,feature and , obtained through feature fusion gate structure ,feature and , , Obtained through feature fusion gate structure This feature fusion method takes into account both spatial context information and high-level semantic information. The data is then fed into the decoding module, which consists of Improved-Inceptionv4 and bilinear interpolation upsampling. The output of the gate structure is concatenated with the output of the decoding module for upsampling, and finally sent to the segmentation head to output a binary black and white segmentation image.

[0051] Step 4: Set model hyperparameters

[0052] This experiment was conducted on an NVIDIA GeForce RTX 1080ti using the PyTorch architecture. The RMSprop optimizer was selected as the stochastic gradient descent optimizer. For the ISIC2018 dataset used in the experiment, the learning rate was set to 0.00001, momentum to 0.9, weight decay to 1e-8, batch size to 4, and training epochs to 40. The binary cross-entropy loss function was used.

[0053] ;

[0054] in, Indicates sample The true label (0 or 1). This represents the score predicted by the model. For the Sigmoid function, This represents the number of samples.

[0055] As you can see, It consists of two parts, each corresponding to the difference between the model's predicted values ​​and the true labels. The first part is the positive samples ( The first part is the loss value of 1, which represents the penalty for the model to predict a negative sample as a positive sample; the second part is the loss value of the negative sample ( The loss value is 0, which represents the penalty for the model to predict a positive sample as a negative sample. This design allows the model to focus more on the mis-segmented samples, strengthen the learning of the features of mis-segmented samples, reduce the loss value of the loss function, and gradually optimize the model's segmentation performance during training.

[0056] The optimal network weight file is loaded, and the trained model is validated on validation data to quantify the model's performance. In this paper, Accuracy, Precision, and Sensitivity are selected as quantitative evaluation metrics, and the evaluation metric formulas are expressed as follows:

[0057] Accuracy=(TP+TN) / (TP+TN+FP+FN);

[0058] Precision = TP / (TP + FP);

[0059] Sensitivity = TP / (TP + FN).

[0060] Step 5: Load the optimal network weight file, test the trained model on the test dataset, and obtain the segmented image.

[0061] In this embodiment, feature extraction using the Inception structure provides more spatial context information to the fusion structure and compensates for the increased number of parameters caused by the introduction of the fusion structure. Furthermore, the improved Inception structure—Improved-Inceptionv4—broadens the width of the original convolutional modules, enhancing network performance and robustness while reducing network parameters. During feature fusion, the feature fusion gate structure in this embodiment adopts a bottom-up fusion order, ensuring mutual compensation of effective information, avoiding redundant feature fusion, and reducing computational complexity. The feature fusion gate structure filters and fuses the feature information output by the encoding module, avoiding feature redundancy caused by simple feature fusion. This results in the output of the shallow encoding module containing both rich spatial context information and rich high-level semantic information. The fused output is then concatenated with the output of the decoding module and upsampled, making the output of the decoding module closer to the output layer, thus containing richer effective spatial context information and high-level semantic information. The U-Net model built based on the feature fusion gate structure and the Improved-Inceptionv4 structure exhibits better image processing efficiency and segmentation results.

[0062] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A melanoma image segmentation method, characterized by, Comprise the following steps: S1, obtain image data and pre-process the image; S2, feature extraction is performed on the pre-processed image by an encoding module; S3, the contribution degree of each stage feature information is controlled by a gate structure, and deep features and shallow features are fused from bottom to top; S4, the output of the gate structure is spliced with the output of the decoding module according to the channel, and up-sampling is performed to obtain a multi-channel feature map; S5, input into a segmentation head, and output a binary black and white segmentation map; The encoding module is composed of Improved-Inceptionv4 and down-sampling, and a multi-layer feature output of different scales is obtained through the encoding module; The Improved-Inceptionv4 structure is composed of five branch structures, branch one: 1x1 convolution, 3x3 convolution, 3x3 convolution; branch two: 1x1 convolution, 3x3 convolution; branch three: 1x5 convolution, 5x1 convolution; branch four: 1x3 convolution, 3x1 convolution; branch five: 3x3 deep convolution, 1x1 point convolution, batch normalization and ReLU activation function are used between the convolution modules of branch one to branch four; The encoding module obtains feature maps of multiple scales through multiple Improved-Inceptionv4 convolution and down-sampling operations , to The feature level is gradually increased, the number of channels is doubled, the size is gradually reduced, and the semantic information is gradually enriched. In the feature fusion process, the deep features and the shallow features are fused from bottom to top. The extracted features are fused by referring to the following method: feature all the fusion output values below the layer obtained through the feature fusion gate structure ; Underlying features Directly obtained by skipping feature fusion gate structure ; Then enter the decoding module, the decoding module is composed of Improved-Inceptionv4 and bilinear interpolation up-sampling, the output of the gate structure is spliced with the output of the decoding module for up-sampling, finally input into the segmentation head, and output a binary black and white segmentation map.

2. The melanoma image segmentation method of claim 1, wherein, In step S1, the image is pre-processed as follows: S101, convert the RGB medical image into a gray image; S102, crop the picture to the size of using Resize operation.

3. The melanoma image segmentation method of claim 1, wherein, The output channel number of the segmentation head is 2, the obtained multi-channel feature map is converted into a 2-channel feature map, and the output result is a black and white binary image.

4. The melanoma image segmentation method of claim 1, wherein, The model composed of steps S1 to S5 is trained by ISIC2018 dataset and two-class cross-entropy loss function before being used, and the network weight is updated and optimized during the training process, and the best network weight file is saved.

5. The melanoma image segmentation method of claim 4, wherein, The two-class cross-entropy loss function is shown as follows: ； wherein, denotes the true label of a sample , which is either 0 or 1, denotes the score predicted by the model, is a Sigmoid function, is the number of samples.

Images