A method for segmenting lesions in dermoscopic images

Abstract

A method for segmenting lesions in dermoscopic images belongs to the field of image processing and segmentation, and includes preprocessing the collected original dermoscopic images and corresponding segmentation labels; dividing the preprocessed dermoscopic images and corresponding segmentation labels into a training set, a validation set, and a test set, and performing data enhancement processing on the training set; constructing a segmentation network for segmenting lesions in dermoscopic images; training the segmentation network for segmenting lesions in dermoscopic images; and generating segmentation results. Feature reuse and deep supervision are achieved through the backbone network; high-level semantic features and spatial location information of lesions are extracted through the cascade fusion module; the extraction of channel and space-related information is enhanced through the spatial channel attention module, reducing erroneous information; and the fusion of features is enhanced through the mutual embedding module. The present invention has outstanding segmentation performance for lesions of different sizes, irregular shapes, hair occlusion, and fuzzy boundaries, strong adaptability, high robustness, and high segmentation accuracy.

Description

Technical Field

[0001] The present invention relates to a method for segmenting lesions in a dermatoscope image, and belongs to the technical field of image processing and segmentation. Background Art

[0002] Early research on dermoscopic images primarily relied on classic digital image methods for lesion segmentation. These methods can generally be categorized into four main categories: thresholding, region-based methods, boundary methods, and active contour methods. The robustness of these early digital image processing-based lesion segmentation methods needs to be improved, and they struggle to adapt to highly variable samples in practical applications. In particular, they were unable to effectively address the irregularities and low contrast of lesion regions in dermoscopic images, making it difficult for these early segmentation algorithms to achieve satisfactory segmentation results. Since the emergence of deep learning, deep learning methods based on convolutional neural networks (CNNs) have achieved tremendous success in image segmentation, image classification, and object detection. Similarly, CNNs have revolutionized the field of dermoscopic image segmentation and recognition, enabling end-to-end training and prediction. While CNN-based dermoscopic image segmentation methods have proven effective, many challenges remain in the deep modeling process. Most existing dermoscopic image segmentation models suffer from insufficient feature extraction, limited preservation of edge information, and blurred edges in the segmentation results. For example, the Chinese patent publication number CN107203999A discloses "A method for automatic segmentation of dermatoscopic images based on a fully convolutional neural network" and the Chinese patent publication number CN109886986A discloses "A method for segmentation of dermatoscopic images based on a multi-branch convolutional neural network". Both of them are dermatoscopic image segmentation methods implemented using a segmentation method based on a convolutional neural network. Both have the problems of being unable to effectively handle blurred boundaries, insufficient feature information extraction, and less edge information retained in the segmentation results. Summary of the Invention

[0003] The purpose of the present invention is to provide a method for segmenting lesions in dermoscopic images to address the problems of existing dermoscopic image segmentation models, such as the inability to effectively handle blurred boundaries, insufficient feature information extraction, and less edge information retained in the segmentation results.

[0004] The technical solutions adopted by the present invention to solve the technical problems are as follows:

[0005] A method for segmenting lesions in a dermatoscopic image of the present invention comprises the following steps:

[0006] Step S1: preprocessing the collected original dermoscopic images and corresponding segmentation labels;

[0007] Step S2: Divide the preprocessed dermoscopic images and the corresponding segmentation labels into a training set, a validation set, and a test set, and perform data augmentation on the training set;

[0008] Step S3: constructing a segmentation network for segmenting lesions in dermoscopic images;

[0009] Step S4: training a segmentation network for segmenting lesions in dermoscopic images to obtain an optimal segmentation model;

[0010] Step S5: Generate segmentation results using the optimal segmentation model.

[0011] Furthermore, the specific operation process of step S1 is as follows:

[0012] S1.1 Collect original dermoscopic images, which are derived from the International Skin Open Challenge dataset ISIC2016 and the publicly available dermoscopic image dataset PH2;

[0013] S1.2 uses the cubic interpolation method to scale the resolution of the original dermoscopic image to 512×512, uses the nearest neighbor method to scale the resolution of the corresponding segmentation label to 512×512, and converts the scaled dermoscopic image and the corresponding segmentation label into .npy format.

[0014] Furthermore, the specific operation process of step S2 is as follows:

[0015] S2.1 Dataset Division

[0016] The pre-processed dermoscopic images and corresponding segmentation labels from the International Skin Open Challenge dataset ISIC2016 were divided into a training set and a validation set. The training set contained 900 samples, and the validation set contained 379 samples. The pre-processed dermoscopic images and corresponding segmentation labels from the dermoscopic image dataset PH2 were divided into a test set. The test set contained 200 samples.

[0017] S2.2 Training set data enhancement processing;

[0018] Data augmentation was performed on the dermoscopic images and corresponding segmentation labels in the training set, including horizontal flipping, vertical flipping, and random rotation.

[0019] Furthermore, in step S3, the segmentation network includes: a backbone network, a cascade fusion module, a spatial channel attention module, an inter-embedding module, a sixth 1×1 convolutional layer, a seventh 1×1 convolutional layer, an eighth 1×1 convolutional layer, a ninth 1×1 convolutional layer, and a tenth 1×1 convolutional layer; the backbone network is used to extract features from the original dermoscopic image and output four multi-scale feature maps, one of which is processed by the spatial channel attention module to enhance the channel and extract spatial related information, and the other three feature maps are processed by the sixth 1×1 convolutional layer, the seventh 1×1 convolutional layer, and the ninth 1×1 convolutional layer. 1 convolution layer and the eighth 1×1 convolution layer are processed by the cascade fusion module for feature interaction operation; the features processed by the cascade fusion module and the features processed by the spatial channel attention module are fused through the mutual embedding module, and the results are processed by the tenth 1×1 convolution layer and the corresponding feature map is output; the features processed by the cascade fusion module are processed by the ninth 1×1 convolution layer and the corresponding feature map is output; the feature map output by the tenth 1×1 convolution layer and the feature map output by the ninth 1×1 convolution layer are element-wise added to obtain the final segmentation result.

[0020] Furthermore, the backbone network includes: the first 8-layer harmonic dense block to the fifth 8-layer harmonic dense block, the first 3×3 convolution layer, the second 3×3 convolution layer, the first 1×1 convolution layer to the fifth 1×1 convolution layer, the first maximum pooling layer to the fourth maximum pooling layer; the original dermoscopic image is imported into the backbone network, and is processed in sequence by the first 3×3 convolution layer, the second 3×3 convolution layer, the first maximum pooling layer, the first 8-layer harmonic dense block, and the first 1×1 convolution layer. The results are respectively imported into the spatial channel attention module and the second maximum pooling layer for processing, and the results after the second maximum pooling layer are processed in sequence. After being processed by the second 8-layer harmonic dense block, the second 1×1 convolution layer, the third 8-layer harmonic dense block, and the third 1×1 convolution layer, the results are respectively imported into the sixth 1×1 convolution layer and the third maximum pooling layer for processing. The results after being imported into the third maximum pooling layer are processed by the fourth 8-layer harmonic dense block and the fourth 1×1 convolution layer in sequence, and the results are respectively imported into the seventh 1×1 convolution layer and the fourth maximum pooling layer for processing. The results after being imported into the fourth maximum pooling layer are processed by the fifth 8-layer harmonic dense block and the fifth 1×1 convolution layer in sequence, and the results are imported into the eighth 1×1 convolution layer for processing.

[0021] Furthermore, the cascade fusion module includes: a first feature pyramid module to a fifth feature pyramid module, a third 3×3 convolutional layer, and a fourth 3×3 convolutional layer; the processing result of the eighth 1×1 convolutional layer is upsampled by 2 times, and the obtained results are respectively introduced into the first feature pyramid module and the second feature pyramid module, and the processing result of the first feature pyramid module is multiplied by the processing result of the seventh 1×1 convolutional layer, and the obtained result is connected in series with the processing result of the second feature pyramid module, and finally processed by the third 3×3 convolutional layer to obtain the connected features; the processing result of the seventh 1×1 convolutional layer is upsampled by 2 times, and the obtained result is introduced into the third feature pyramid module, and the processing result of the eighth 1×1 convolutional layer is upsampled by 4 times, and the obtained result is introduced into the fourth feature pyramid module, and the processing result of the third feature pyramid module, the processing result of the fourth feature pyramid module and the processing result of the sixth 1×1 convolutional layer are multiplied, and the obtained mapping is connected in series with the processing result of the fifth feature pyramid module, and finally the obtained feature map is introduced into the fourth 3×3 convolutional layer for dimensionality reduction processing to obtain the connected features.

[0022] Furthermore, each feature pyramid module includes: the eleventh 1×1 convolution layer, the twelfth 1×1 convolution layer, the first depth expansion convolution module to the fourth depth expansion convolution module; the initial features imported into the feature pyramid module are first processed by the eleventh 1×1 convolution layer, and the results are evenly divided into 4 equal block features along the channel. The first block feature is imported into the first depth expansion convolution module for processing, and the processing result is element-wise added with the second block feature. The result is then imported into the second depth expansion convolution module for processing, and the processing result is element-wise added with the third block feature. The addition operation is performed, and the result is then imported into the third depth expansion convolution module for processing. The processing result is element-wise added with the fourth block feature, and the result is imported into the fourth depth expansion convolution module for processing; the processing result of the first depth expansion convolution module, the processing result of the second depth expansion convolution module, the processing result of the third depth expansion convolution module and the processing result of the fourth depth expansion convolution module are connected in series, and the number of channels is restored by the twelfth 1×1 convolution layer on the connection result, and the result is element-wise added with the initial features imported into the feature pyramid module.

[0023] Furthermore, the spatial channel attention module includes: a spatial attention module and a channel attention module; the spatial attention module includes: a first along-channel average pooling layer, a along-channel maximum pooling layer, a 7×7 convolution layer and a first Sigmoid function; the channel attention module includes: a first average pooling layer, a fifth maximum pooling layer, a thirteenth 1×1 convolution layer to a sixteenth 1×1 convolution layer, a ReLU function and a second Sigmoid function; the processing results of the first 1×1 convolution layer are respectively imported into the first along-channel average pooling layer and the along-channel maximum pooling layer, and the results obtained by the two pooling layers are connected in series, and the features after connection are then processed by the 7×7 convolution layer and the first Sigmoid function, and the results are multiplied with the original input of the spatial channel attention module; the processing results of the first 1×1 convolution layer are respectively imported into the first average pooling layer and The fifth maximum pooling layer is used, and the result obtained by the first average pooling layer is processed by the thirteenth 1×1 convolution layer, and the result obtained by the fifth maximum pooling layer is processed by the fourteenth 1×1 convolution layer; the processing result of the thirteenth 1×1 convolution layer is processed by the ReLU function and the fifteenth 1×1 convolution layer to restore the feature to the original number of channels, and at the same time, the processing result of the fourteenth 1×1 convolution layer is processed by the ReLU function and the sixteenth 1×1 convolution layer to restore the feature to the original number of channels, the processing results of the fifteenth 1×1 convolution layer and the sixteenth 1×1 convolution layer are element-wise added and processed by the second Sigmoid function, and the result is multiplied with the original input of the spatial channel attention module; finally, the features processed by the spatial attention module and the channel attention module are element-wise added to obtain low-level features containing rich spatial information.

[0024] Furthermore, the mutual embedding module includes: a second channel average pooling layer, a second average pooling layer, a seventeenth 1×1 convolution layer to a twentieth 1×1 convolution layer; in the first part, the features from the spatial channel attention module are imported into the seventeenth 1×1 convolution layer for processing, the features from the cascade fusion module are imported into the nineteenth 1×1 convolution layer for processing, and then imported into the second average pooling layer and the twentieth 1×1 convolution layer for processing, and the processing results of the seventeenth 1×1 convolution layer are element-wise multiplied with the processing results of the twentieth 1×1 convolution layer; in the second part, the features from the spatial channel attention module are imported into the seventeenth 1×1 convolution layer for processing, and the features from the cascade fusion module are imported into the nineteenth 1×1 convolution layer for processing, and then imported into the second average pooling layer and the twentieth 1×1 convolution layer for processing, and the processing results of the seventeenth 1×1 convolution layer are element-wise multiplied with the processing results of the twentieth 1×1 convolution layer; In the second part, the features from the cascade fusion module are imported into the nineteenth 1×1 convolutional layer for processing, and the features from the spatial channel attention module are imported into the seventeenth 1×1 convolutional layer for processing and then imported into the second along-channel average pooling layer and the eighteenth 1×1 convolutional layer for processing. The processing results of the eighteenth 1×1 convolutional layer and the nineteenth 1×1 convolutional layer are element-wise multiplied; the results of the first part and the results of the second part are element-wise added to fuse the features and obtain a feature map containing high-level semantic information and spatial position information.

[0025] Furthermore, the specific operation process of step S4 and step S5 is as follows:

[0026] Step S4: The data-enhanced training set is imported into the segmentation network in batches. The loss function adopts a combination of a binary cross entropy loss function and an intersection-over-union loss function. The features processed by the backbone network are imported into the cascade fusion module and the mutual embedding module to generate a high-level feature map. The corresponding feature loss is calculated with the corresponding segmentation label through the loss function; the feature loss from the cascade fusion module is added to the feature loss from the mutual embedding module to obtain the total loss, and then the parameters of the segmentation network are optimized by the AdamW optimizer and the total loss back propagation. After each epoch of complete training, the training effect of the segmentation network is verified using the validation set. The optimal segmentation model is obtained when the Jaccard coefficient of the validation set is the largest. During the training process of the segmentation network, the initial learning rate is set to 0.0001, and the cosine annealing strategy is introduced to adjust the learning rate. The batch size is set to 8, and 200 epochs are trained.

[0027] Step S5: Import the dermoscopic images in the test set into the trained optimal segmentation model to obtain a segmentation result map corresponding to the dermoscopic images.

[0028] The beneficial effects of the present invention are:

[0029] A method for segmenting lesions in dermoscopic images of the present invention collects original dermoscopic images containing segmentation labels and performs image preprocessing; then the original dermoscopic images are divided into a data set, specifically into a training set, a validation set and a test set, and the training set is subjected to data enhancement processing; a segmentation network for segmenting lesions in dermoscopic images is constructed, the segmentation network uses HarDNet (Harmonic DenseNet) as the backbone network, and introduces a cascaded fusion module CFM (Cascaded Fusion Module) and a spatial channel attention module SCAM (Spatial Channel Attention Module), and then the mutual embedding module MEM (Mutual Embedding Module) is responsible for feature fusion; the training set and the validation set are imported into the segmentation network for training, and the optimal segmentation model for segmentation is obtained after training; the test set is predicted in the optimal segmentation model to obtain the segmentation result.

[0030] Compared with the prior art, the present invention has the following advantages:

[0031] 1. This paper adopts HarDNet as the backbone network of the optimal segmentation model, extracts more powerful and critical features, realizes feature reuse and deep supervision, and effectively reduces memory usage.

[0032] 2. Based on the backbone network HarDNet, this paper designs three components: a cascade fusion module (CFM), a spatial channel attention module (SCAM), and a mutual embedding module (MEM). The feature pyramid module and progressive approach within the cascade fusion module effectively extract high-level semantic features and spatial location information of the lesion. Simultaneously, the spatial channel attention module enhances the extraction of channel and spatial-related information, obtains detailed information about the lesion, and effectively reduces erroneous information in low-level features. The mutual embedding module effectively enhances feature fusion and supplements the structural information of the lesion.

[0033] 3. The present invention has outstanding segmentation performance for lesions of different sizes, irregular shapes, hair occlusion and blurred boundaries, and has the advantages of strong adaptability, high robustness and high segmentation accuracy, and has good application prospects.

[0034] 4. The present invention proposes a new solution to the problems of existing dermoscopic image segmentation models, such as the inability to effectively handle blurred boundaries, insufficient feature information extraction, and less edge information retained in the segmentation results. The present invention improves the accuracy and robustness of segmentation and can accurately locate the lesion boundaries even in extreme cases. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a flow chart of a method for segmenting lesions in dermatoscopic images according to the present invention.

[0036] Figure 2 Schematic diagram of the structure of the segmentation network used to segment lesions in dermoscopic images.

[0037] Figure 3 Figure 2 is a structural diagram of the cascade fusion module CFM.

[0038] Figure 4 Schematic diagram of the structure of the feature pyramid module FPM.

[0039] Figure 5 Schematic diagram of the structure of the deep dilated convolution module DDCM.

[0040] Figure 6 Schematic diagram of the structure of the spatial channel attention module SCAM.

[0041] Figure 7 Schematic diagram of the structure of the mutual embedding module MEM.

[0042] Figure 8 Schematic diagram of an application example of a method for segmenting lesions in dermatoscopic images according to the present invention. DETAILED DESCRIPTION

[0043] The present invention will be further described in detail below with reference to the accompanying drawings.

[0044] A method for segmenting lesions in a dermatoscopic image of the present invention comprises the following steps:

[0045] Step S1: preprocessing the collected original dermoscopic images and corresponding segmentation labels;

[0046] Step S2: Divide the preprocessed dermoscopic images and the corresponding segmentation labels into a training set, a validation set, and a test set, and perform data augmentation on the training set;

[0047] Step S3: constructing a segmentation network for segmenting lesions in dermoscopic images;

[0048] Step S4: training a segmentation network for segmenting lesions in dermoscopic images to obtain an optimal segmentation model;

[0049] Step S5: Generate segmentation results using the optimal segmentation model.

[0050] The present invention provides a method for segmenting lesions in dermatoscopic images, and the specific operating steps are as follows:

[0051] Step S1: collecting original dermoscopic images and preprocessing the collected original dermoscopic images and corresponding segmentation labels;

[0052] S1.1 The original dermoscopic images are derived from the International Skin Open Challenge dataset ISIC2016 and the publicly available dermoscopic image dataset PH2. The original dermoscopic image dataset is formed by collecting original dermoscopic images, and each original dermoscopic image is accompanied by a corresponding segmentation label.

[0053] S1.2 Preprocessing: To facilitate the training of the segmentation network, the resolution of the original dermoscopic image was scaled to 512 × 512 using the cubic interpolation method. At the same time, the resolution of the corresponding segmentation label was scaled to 512 × 512 using the nearest neighbor method. The scaled dermoscopic image and the corresponding segmentation label were converted into .npy format.

[0054] Step S2: dividing the preprocessed dermoscopic images and the corresponding segmentation labels into a training set, a validation set, and a test set, and performing data augmentation on the training set;

[0055] S2.1 Dataset Division

[0056] The International Skin Open Challenge dataset ISIC2016 and the dermoscopic image dataset PH2 were used together. The pre-processed dermoscopic images from the International Skin Open Challenge dataset ISIC2016 and the corresponding segmentation labels were divided into a training set and a validation set. The training set contained 900 samples, and the validation set contained 379 samples. The pre-processed dermoscopic images from the dermoscopic image dataset PH2 and the corresponding segmentation labels were divided into a test set. The test set contained 200 samples.

[0057] S2.2 Training set data enhancement processing;

[0058] Data augmentation was performed on the dermoscopic images and corresponding segmentation labels in the training set, including horizontal flipping, vertical flipping, and random rotation.

[0059] Step S3: constructing a segmentation network for segmenting lesions in dermoscopic images;

[0060] like Figure 2 As shown in the figure, the constructed segmentation network is an end-to-end structure, mainly consisting of the Harmonic DenseNet (HarDNet) backbone network, the Cascaded Fusion Module (CFM), the Spatial Channel Attention Module (SCAM), the Mutual Embedding Module (MEM), and the sixth 1×1 convolutional layer (Conv8) to the tenth 1×1 convolutional layer (Conv12). The HarDNet backbone network's structure is derived by improving the dense blocks in the Densely Connected Convolutional Network (DenseNet), achieving efficient inference speed and high-precision segmentation performance. The Cascaded Fusion Module (CFM) progressively aggregates features at different scales, effectively acquiring semantic information about lesions from the HarDNet backbone network. The Spatial Channel Attention Module (SCAM) enhances the extraction and modeling of spatial and channel-related information. The Mutual Embedding Module (MEM) efficiently aggregates the semantic information from the Cascaded Fusion Module (CFM) and the spatial information from the Spatial Channel Attention Module (SCAM).

[0061] The overall operation process of the constructed segmentation network is as follows:

[0062] The original dermoscopic image (such as Figure 2The network is imported into the backbone network HarDNet, which performs feature extraction and outputs four multi-scale feature maps, wherein the feature map from the first 1×1 convolution layer Conv3 is processed by the spatial channel attention module SCAM to enhance the channel and extract spatial related information, and the three feature maps from the third 1×1 convolution layer Conv5, the fourth 1×1 convolution layer Conv6, and the fifth 1×1 convolution layer Conv7 are processed by the sixth 1×1 convolution layer Conv8, the seventh 1×1 convolution layer Conv9, and the eighth 1×1 convolution layer Conv10 respectively, and then subjected to feature interaction operation by the cascade fusion module CFM; the features processed by the cascade fusion module CFM are fused with the features processed by the spatial channel attention module SCAM through the mutual embedding module MEM, and the results are processed by the tenth 1×1 convolution layer Conv12 to output the corresponding feature map (as shown in FIG. Figure 2 The features processed by the cascade fusion module CFM are processed by the ninth 1×1 convolution layer Conv11 and the corresponding feature map is output (as shown in B). Figure 2 D in the middle); the feature map output by the tenth 1×1 convolutional layer Conv12 (as shown in Figure 2 B) and the feature map output by the ninth 1×1 convolutional layer Conv11 (as shown in Figure 2 D in the figure) to perform element addition operation to obtain the final segmentation result (as shown in Figure 2 (shown as C in the middle)

[0063] like Figure 2 As shown, the backbone network HarDNet serves as the backbone for feature extraction, and specifically HardNet68 can be used. The HarDNet68 is mainly composed of five 8-layer harmonic dense blocks HarDBlk (HarmonicDenseBlock) (the first 8-layer harmonic dense block HarDBlk1 to the fifth 8-layer harmonic dense block HarDBlk5), two 3×3 convolutional layers (the first 3×3 convolutional layer Conv1 and the second 3×3 convolutional layer Conv2), five 1×1 convolutional layers (the first 1×1 convolutional layer Conv3 to the fifth 1×1 convolutional layer Conv7) and four maximum pooling layers (the first maximum pooling layer MaxPool1 to the fourth maximum pooling layer MaxPool4). A 1×1 convolutional layer is set after each harmonic dense block HarDBlk to adjust the number of channels. Among them, the feature map scales output by the second 8-layer harmonic dense block HarDBlk2 and the third 8-layer harmonic dense block HarDBlk3 are the same, and the feature map scales output by the fourth 8-layer harmonic dense block HarDBlk4 and the fifth 8-layer harmonic dense block HarDBlk5 are halved successively.

[0064] The original dermoscopic image (such as Figure 2As shown in A in the figure, the backbone network HarDNet is imported, and the backbone network HarDNet performs feature extraction and outputs four multi-scale feature maps. The specific operations are as follows:

[0065] The original dermoscopic image (such as Figure 2 As shown in Figure 1, the backbone network HarDNet is imported, and the results are processed by the first 3×3 convolution layer Conv1, the second 3×3 convolution layer Conv2, the first maximum pooling layer MaxPool1, the first 8-layer harmonic dense block HarDBlk1, and the first 1×1 convolution layer Conv3. The results are respectively imported into the spatial channel attention module SCAM and the second maximum pooling layer MaxPool2 for processing. Among them, the result after importing the second maximum pooling layer MaxPool2 is processed by the second 8-layer harmonic dense block HarDBlk2, the second 1×1 convolution layer Conv4, the third 8-layer harmonic dense block HarDBlk3, and the third 1×1 convolution layer Conv5, and the results are respectively imported into the sixth 1×1 convolution layer Conv8 and the third maximum pooling layer MaxP ool3 for processing, wherein the result of the third maximum pooling layer MaxPool3 is processed and then processed by the fourth 8-layer harmonic dense block HarDBlk4 and the fourth 1×1 convolution layer Conv6 in sequence, and the results are respectively imported into the seventh 1×1 convolution layer Conv9 and the fourth maximum pooling layer MaxPool4 for processing, wherein the result after importing the fourth maximum pooling layer MaxPool4 is processed by the fifth 8-layer harmonic dense block HarDBlk5 and the fifth 1×1 convolution layer Conv7 in sequence, and the result is imported into the eighth 1×1 convolution layer Conv10 for processing; the results after the sixth 1×1 convolution layer Conv8, the seventh 1×1 convolution layer Conv9, and the eighth 1×1 convolution layer Conv10 are imported into the cascade fusion module CFM for feature interaction operation.

[0066] In order to make full use of information at different scales, the multi-scale features of the first 8-layer harmonic dense block HarDBlk1, the third 8-layer harmonic dense block HarDBlk3, the fourth 8-layer harmonic dense block HarDBlk4 and the fifth 8-layer harmonic dense block HarDBlk5 in the backbone network HarDNet are extracted; among them, the features of the first 8-layer harmonic dense block HarDBlk1 in the backbone network HarDNet are processed by the first 1×1 convolutional layer Conv3 and then imported into the spatial channel attention module SCAM for processing; at the same time, an effective feature extraction structure, namely the cascade fusion module CFM, is used to perform feature interaction operations on the features of the third 8-layer harmonic dense block HarDBlk3, the fourth 8-layer harmonic dense block HarDBlk4 and the fifth 8-layer harmonic dense block HarDBlk5.

[0067] like Figure 3As shown in the figure, the cascade fusion module CFM is mainly composed of five feature pyramid modules FPM (FeaturePyramidModule) (the first feature pyramid module FPM1-the fifth feature pyramid module FPM5) and two 3×3 convolutional layers (the third 3×3 convolutional layer Conv13 and the fourth 3×3 convolutional layer Conv14), and adopts batch normalization and ReLU function.

[0068] The features of the third 8-layer harmonic dense block HarDBlk3 in the backbone network HarDNet are processed by the third 1×1 convolution layer Conv5 and the sixth 1×1 convolution layer Conv8 and then imported into the cascade fusion module CFM. The features of the fourth 8-layer harmonic dense block HarDBlk4 are processed by the fourth 1×1 convolution layer Conv6 and the seventh 1×1 convolution layer Conv9 and then imported into the cascade fusion module CFM. The features of the fifth 8-layer harmonic dense block HarDBlk5 are processed by the fifth 1×1 convolution layer Conv7 and the eighth 1×1 convolution layer Conv10 and then imported into the cascade fusion module CFM. The cascade fusion module CFM mainly consists of two cascade parts, such as Figure 3As shown in FIG, in the first cascade part, the features of the fifth 8-layer harmonic dense block HarDBlk5 are processed by the fifth 1×1 convolution layer Conv7 and the eighth 1×1 convolution layer Conv10 and then upsampled by 2 times (up×2). Then, the sampling results are respectively imported into the first feature pyramid module FPM1 and the second feature pyramid module FPM2. The features obtained after processing by the first feature pyramid module FPM1 and the features of the fourth harmonic dense block HarDBlk4 processed by the fourth 1×1 convolution layer Conv6 and the seventh 1×1 convolution layer Conv9 are multiplied. The obtained results are connected in series with the results processed by the second feature pyramid module FPM2. Finally, the third 3×3 convolution layer Conv13 is used to process the connected features. In the second cascade part, the features of the fourth harmonic dense block HarDBlk4 are connected by the fourth 1×1 convolution layer Co After processing by nv6 and the seventh 1×1 convolution layer Conv9, the features are upsampled by 2 times (up×2) and then imported into the third feature pyramid module FPM3. The features of the fifth harmonic dense block HarDBlk5 are processed by the fifth 1×1 convolution layer Conv7 and the eighth 1×1 convolution layer Conv10 and upsampled by 4 times (up×4) and then imported into the fourth feature pyramid module FPM4. The features processed by the third feature pyramid module FPM3 and the fourth feature pyramid module FPM4 are multiplied with the features of the third harmonic dense block HarDBlk3 processed by the third 1×1 convolution layer Conv5 and the sixth 1×1 convolution layer Conv8. The obtained mapping is connected in series with the features processed by the fifth feature pyramid module FPM5. Finally, the obtained feature map is introduced into the fourth 3×3 convolution layer Conv14 for dimensionality reduction to obtain the connected features.

[0069] The feature pyramid module FPM is introduced into the cascade fusion module CFM to encode multi-scale features and extract key features more effectively. The feature pyramid module FPM is a module based on the extended reverse bottleneck convolution, such as Figure 4As shown, each feature pyramid module FPM consists of two 1×1 convolutional layers (the eleventh 1×1 convolutional layer Conv15 and the twelfth 1×1 convolutional layer Conv16) and four depthwise dilated convolution modules DDCM (Depthwise Dilated Convolutions Module) (the first depthwise dilated convolution module DDCM1-the fourth depthwise dilated convolution module DDCM4), and adopts residual connection. The initial features imported into the feature pyramid module FPM are first processed by the eleventh 1×1 convolutional layer Conv15, and then the obtained results are divided into 4 equal block features along the channel. Among them, the first equal block feature is imported into the first depthwise dilated convolution module DDCM1 for processing, and the obtained processing result is element-wise added with the second equal block feature. The obtained result is imported into the second depthwise dilated convolution module DDCM2 for processing, and the obtained processing result is element-wise added with the third equal block feature. The obtained result is imported into the third depthwise dilated convolution module DDCM3 for processing, and the obtained processing result is element-wise added with the fourth equal block feature. The result is imported into the fourth depth dilated convolution module DDCM4 for processing; the features processed by the first depth dilated convolution module DDCM1, the features processed by the second depth dilated convolution module DDCM2, the features processed by the third depth dilated convolution module DDCM3 and the features processed by the fourth depth dilated convolution module DDCM4 are connected in series, and the number of channels is restored by the twelfth 1×1 convolution layer Conv16. The obtained result is element-wise added with the initial features imported into the feature pyramid module FPM, and then imported into the ninth 1×1 convolution layer Conv11 for processing and output.

[0070] like Figure 5 As shown in Figure 1, each depthwise dilated convolutional module (DDCM) consists of a 1×D convolutional layer (Conv1×D) with padding set to (0, d), a D×1 convolutional layer (ConvD×1) with padding set to (d, 0), and a 3×3 dilated convolutional layer (DConv3×3) with a dilation rate of D and padding set to (D, D), where d∈{0,1,2,3} and D∈{1,3,5,7}. The depthwise dilated convolutional module (DDCM) increases the receptive field without reducing the feature map resolution, reducing computational complexity.

[0071] like Figure 6As shown in the figure, the spatial channel attention module SCAM is mainly composed of a spatial attention module and a channel attention module. Among them, the spatial attention module is mainly composed of the first channel average pooling layer Avgpoolc1, the channel maximum pooling layer Maxpoolc, the 7×7 convolution layer Conv17 and the first Sigmoid function; the channel attention module is mainly composed of the first average pooling layer Avgpool1, the fifth maximum pooling layer Maxpool5, four 1×1 convolution layers (the thirteenth 1×1 convolution layer Conv18 to the sixteenth 1×1 convolution layer Conv21), the ReLU function and the second Sigmoid function. The processing of input features by the spatial channel attention module SCAM mainly includes two parts. The first part is that in the spatial attention module, the features processed by the first 1×1 convolution layer Conv3 are respectively imported into the first channel average pooling layer Avgpoolc1 and the channel maximum pooling layer Maxpoolc, and the results obtained by the two pooling layers are connected in series. The connected features are then processed by the 7×7 convolution layer Conv17 and the first Sigmoid function, and then the processed features are multiplied with the original input of the spatial channel attention module SCAM (the original input refers to the features of the first 8-layer harmonic dense block HarDBlk1 after processing by the first 1×1 convolution layer Conv3); the second part is that in the channel attention module, the features processed by the first 1×1 convolution layer Conv3 are respectively imported into the first average pooling layer Avgpool1 and the fifth maximum pooling layer Maxpool5, and the results obtained by the first average pooling layer Avgpool1 are processed by the thirteenth 1×1 convolution layer Conv18, and the fifth maximum pooling layer Maxpool5 is processed by the thirteenth 1×1 convolution layer Conv19. The result obtained by the large pooling layer Maxpool5 is processed by the fourteenth 1×1 convolutional layer Conv19, thereby reducing the number of channels by 16 times; then the result of the processing of the thirteenth 1×1 convolutional layer Conv18 is processed by the ReLU function and the fifteenth 1×1 convolutional layer Conv20 to restore the features to the original number of channels. At the same time, the result of the processing of the fourteenth 1×1 convolutional layer Conv19 is processed by the ReLU function and the sixteenth 1×1 convolutional layer Conv21 to restore the features to the original number of channels. The results of the processing of the fifteenth 1×1 convolutional layer Conv20 and the sixteenth 1×1 convolutional layer Conv21 are element-wise added and processed by the second Sigmoid function. The result is multiplied with the original input of the spatial channel attention module SCAM (the original input refers to the features of the first 8-layer harmonic dense block HarDBlk1 after processing by the first 1×1 convolutional layer Conv3); finally, the features processed by the spatial attention module and the channel attention module are element-wise added, and finally a low-level feature containing rich spatial information is obtained.In the present invention, the spatial channel attention module (SCAM) is used to enhance the extraction of channel and spatial related information and effectively identify the details of skin lesions.

[0072] like Figure 7 As shown, the mutual embedding module MEM is mainly composed of the second channel average pooling layer Avgpoolc2, the second average pooling layer Avgpool2 and four 1×1 convolutional layers (the seventeenth 1×1 convolutional layer Conv22-the twentieth 1×1 convolutional layer Conv25). The mutual embedding module MEM processing can be divided into two parts. In the first part, the features from the spatial channel attention module SCAM are imported into the seventeenth 1×1 convolutional layer Conv22 for processing, and the features from the cascade fusion module CFM are imported into the nineteenth 1×1 convolutional layer Conv24 for processing and then imported into the second average pooling layer Avgpool2 and the twentieth 1×1 convolutional layer Conv25 for processing. The processing results of the seventeenth 1×1 convolutional layer Conv22 and the processing results of the twentieth 1×1 convolutional layer Conv25 are element-wise multiplied. In the second part, the features from the cascade fusion module CFM are imported into the The nineteenth 1×1 convolutional layer Conv24 is used for processing, and the features from the spatial channel attention module SCAM are imported into the seventeenth 1×1 convolutional layer Conv22 for processing, and then imported into the second channel average pooling layer Avgpoolc2 and the eighteenth 1×1 convolutional layer Conv23 for processing. The processing results of the eighteenth 1×1 convolutional layer Conv23 and the processing results of the nineteenth 1×1 convolutional layer Conv24 are element-wise multiplied; the result of the first part is element-wise added to the result of the second part to fuse the features obtained from the two parts to obtain a feature map containing high-level semantic information and spatial position information.

[0073] The mutual embedding module MEM embeds the high-level features processed by the cascade fusion module CFM into the context information, and at the same time embeds the low-level features processed by the spatial channel attention module SCAM into the spatial details, effectively enhancing the fusion of features.

[0074] Step S4: training a segmentation network for segmenting lesions in dermoscopic images to obtain an optimal segmentation model;

[0075] First, the data augmented training set is imported into the segmentation network in batches. The loss function adopts a combination of binary cross entropy loss function and intersection-over-union loss function. The features processed by the backbone network HarDNet are imported into the cascade fusion module CFM and the mutual embedding module MEM to generate high-level feature maps, and the corresponding feature loss is obtained by calculating the loss function with the corresponding segmentation labels; then the feature loss from the cascade fusion module CFM and the feature loss from the mutual embedding module MEM are added to obtain the total loss, and then the parameters of the segmentation network are optimized by AdamW optimizer and total loss back propagation, and complete training is performed for each epoch; the validation set is imported and used to verify the training effect of the segmentation network. The optimal segmentation model is obtained when the Jaccard coefficient of the validation set is the largest; wherein, during the training process of the segmentation network, the initial learning rate is set to 0.0001, and the cosine annealing strategy is introduced to adjust the learning rate. The batch size is set to 8, and 200 epochs are trained.

[0076] Step S5: Network testing, using the optimal segmentation model to generate the final segmentation result map;

[0077] The dermoscopic images in the test set are imported into the trained optimal segmentation model to complete the test set test, and finally the segmentation result map corresponding to the dermoscopic images is obtained.

[0078] The feasibility of the method provided by the present invention is verified by specific tests below.

[0079] 1. Working conditions

[0080] This experiment is implemented under the Pytorch deep learning framework, using NVIDIA Geforce 3090 graphics card and Linux operating system.

[0081] 2. Experimental content and results analysis

[0082] like Figure 8 As shown, Figure 8 The four images in the first row are all original dermatoscopic images. Figure 8 The four figures in the second row are respectively the segmentation results of the lesions in the above four original dermoscopic images predicted by using a segmentation method for lesions in dermoscopic images of the present invention. Figure 8 The four images in the third row show the segmentation labels corresponding to the four original dermoscopic images. Comparison reveals that the segmentation method of the present invention achieves excellent lesion segmentation results, and the obtained segmentation results are very close to the segmentation labels, demonstrating that the present invention can accurately segment lesions in dermoscopic images.

[0083] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for segmenting lesions in dermatoscopic images, characterized in that: The following steps are involved: Step S1: preprocessing the collected original dermoscopic images and corresponding segmentation labels; Step S2: Divide the preprocessed dermoscopic images and the corresponding segmentation labels into a training set, a validation set, and a test set, and perform data augmentation on the training set; Step S3: constructing a segmentation network for segmenting lesions in dermoscopic images; The segmentation network uses HarDNet as the backbone network, and introduces the cascade fusion module CFM and the spatial channel attention module SCAM, and then the mutual embedding module MEM is responsible for feature fusion; The feature pyramid module and progressive method within the cascade fusion module are used to extract high-level semantic features and spatial location information of the lesion. At the same time, the spatial channel attention module enhances the extraction of channel and spatial related information and obtains detailed information about the lesion. The mutual embedding module effectively enhances feature fusion and supplements the structural information of the lesion. Step S4: training a segmentation network for segmenting lesions in dermoscopic images to obtain an optimal segmentation model; Step S5: Generate segmentation results using the optimal segmentation model.

2. The method for segmenting lesions in dermatoscopic images according to claim 1, characterized in that: The specific operation process of step S1 is as follows: S1.1 Collect original dermoscopic images, which are derived from the International Skin Open Challenge dataset ISIC2016 and the publicly available dermoscopic image dataset PH2; S1.2 uses the cubic interpolation method to scale the resolution of the original dermoscopic image to 512×512, uses the nearest neighbor method to scale the resolution of the corresponding segmentation label to 512×512, and converts the scaled dermoscopic image and the corresponding segmentation label into .npy format.

3. The method for segmenting lesions in dermatoscopic images according to claim 1, characterized in that: The specific operation process of step S2 is as follows: S2.1 Dataset Division The pre-processed dermoscopic images and corresponding segmentation labels from the International Skin Open Challenge dataset (ISIC 2016) were divided into a training set and a validation set. The training set contained 900 samples, and the validation set contained 379 samples. The pre-processed dermoscopic images and corresponding segmentation labels from the dermoscopic image dataset PH2 are divided into a test set containing 200 samples; S2.2 Training set data enhancement processing; Data augmentation was performed on the dermoscopic images and corresponding segmentation labels in the training set, including horizontal flipping, vertical flipping, and random rotation.

4. The method for segmenting lesions in dermatoscopic images according to claim 1, characterized in that: In step S3, the segmentation network includes: a backbone network, a cascade fusion module, a spatial channel attention module, an inter-embedding module, a sixth 1×1 convolutional layer, a seventh 1×1 convolutional layer, an eighth 1×1 convolutional layer, a ninth 1×1 convolutional layer, and a tenth 1×1 convolutional layer; the backbone network is used to extract features from the original dermatoscope image and output four multi-scale feature maps, one of which is processed by the spatial channel attention module to enhance the channel and extract spatial related information, and the other three feature maps are processed by the sixth 1×1 convolutional layer, the seventh 1×1 convolutional layer, and the ninth 1×1 convolutional layer. After processing by the eighth 1×1 convolution layer, the cascade fusion module performs feature interaction operation; the features processed by the cascade fusion module and the features processed by the spatial channel attention module are fused through the mutual embedding module, and the results are processed by the tenth 1×1 convolution layer and the corresponding feature map is output; the features processed by the cascade fusion module are processed by the ninth 1×1 convolution layer and the corresponding feature map is output; the feature map output by the tenth 1×1 convolution layer and the feature map output by the ninth 1×1 convolution layer are element-wise added to obtain the final segmentation result.

5. The method for segmenting lesions in dermatoscopic images according to claim 4, characterized in that: The backbone network includes: the first 8-layer harmonic dense block to the fifth 8-layer harmonic dense block, the first 3×3 convolution layer, the second 3×3 convolution layer, the first 1×1 convolution layer to the fifth 1×1 convolution layer, and the first maximum pooling layer to the fourth maximum pooling layer; the original dermatoscope image is imported into the backbone network, and is processed in sequence by the first 3×3 convolution layer, the second 3×3 convolution layer, the first maximum pooling layer, the first 8-layer harmonic dense block, and the first 1×1 convolution layer. The results are respectively imported into the spatial channel attention module and the second maximum pooling layer for processing, and the results after the second maximum pooling layer are processed in sequence by the second maximum pooling layer. After the 8-layer harmonic dense block, the second 1×1 convolution layer, the third 8-layer harmonic dense block, and the third 1×1 convolution layer are processed, the results are respectively imported into the sixth 1×1 convolution layer and the third maximum pooling layer for processing. The results after the third maximum pooling layer are processed by the fourth 8-layer harmonic dense block and the fourth 1×1 convolution layer in sequence, and the results are respectively imported into the seventh 1×1 convolution layer and the fourth maximum pooling layer for processing. The result after the fourth maximum pooling layer is processed by the fifth 8-layer harmonic dense block and the fifth 1×1 convolution layer in sequence, and the result is imported into the eighth 1×1 convolution layer for processing.

6. The method for segmenting lesions in dermoscopic images according to claim 5, characterized in that: The cascade fusion module includes: a first feature pyramid module to a fifth feature pyramid module, a third 3×3 convolutional layer, and a fourth 3×3 convolutional layer; performing a 2-fold upsampling on the processing result of the eighth 1×1 convolutional layer, and importing the obtained result into the first feature pyramid module and the second feature pyramid module respectively, multiplying the processing result of the first feature pyramid module with the processing result of the seventh 1×1 convolutional layer, and performing a series operation on the obtained result and the processing result of the second feature pyramid module, and finally processing the result through the third 3×3 convolutional layer to obtain the series-connected features; performing a 2-fold upsampling on the processing result of the seventh 1×1 convolutional layer, and importing the obtained result into the third feature pyramid module, performing a 4-fold upsampling on the processing result of the eighth 1×1 convolutional layer, and importing the obtained result into the fourth feature pyramid module, multiplying the processing result of the third feature pyramid module, the processing result of the fourth feature pyramid module, and the processing result of the sixth 1×1 convolutional layer, and performing a series operation on the obtained mapping and the processing result of the fifth feature pyramid module, and finally introducing the obtained feature map into the fourth 3×3 convolutional layer for dimensionality reduction processing to obtain the series-connected features.

7. The method for segmenting lesions in dermoscopic images according to claim 6, characterized in that: Each feature pyramid module includes: an eleventh 1×1 convolution layer, a twelfth 1×1 convolution layer, a first depth-expanded convolution module to a fourth depth-expanded convolution module; the initial features imported into the feature pyramid module are first processed by the eleventh 1×1 convolution layer, and the obtained results are evenly divided into four equal block features along the channel. The first equal block feature is imported into the first depth-expanded convolution module for processing, and the obtained processing result is element-wise added with the second equal block feature. The obtained result is then imported into the second depth-expanded convolution module for processing, and the obtained processing result is element-wise added with the third equal block feature. The obtained result is then imported into the third depth-expanded convolution module for processing, and the obtained processing result is element-wise added with the fourth equal block feature. The obtained result is then imported into the fourth depth-expanded convolution module for processing; the processing results of the first depth-expanded convolution module, the processing results of the second depth-expanded convolution module, the processing results of the third depth-expanded convolution module, and the processing results of the fourth depth-expanded convolution module are concatenated, and the concatenated result uses the twelfth 1×1 convolution layer to restore the number of channels, and the obtained result is element-wise added with the initial features imported into the feature pyramid module.

8. The method for segmenting lesions in dermoscopic images according to claim 7, characterized in that: The spatial channel attention module includes: a spatial attention module and a channel attention module; the spatial attention module includes: a first along-channel average pooling layer, a along-channel maximum pooling layer, a 7×7 convolution layer and a first Sigmoid function; the channel attention module includes: a first average pooling layer, a fifth maximum pooling layer, a thirteenth 1×1 convolution layer to a sixteenth 1×1 convolution layer, a ReLU function and a second Sigmoid function; the processing results of the first 1×1 convolution layer are respectively imported into the first along-channel average pooling layer and the along-channel maximum pooling layer, and the results obtained by the two pooling layers are connected in series, and the features after connection are then processed by the 7×7 convolution layer and the first Sigmoid function, and the results are multiplied with the original input of the spatial channel attention module; the processing results of the first 1×1 convolution layer are respectively imported into the first average pooling layer and the fifth maximum pooling layer. The large pooling layer is performed, and the result obtained by the first average pooling layer is processed by the thirteenth 1×1 convolution layer, and the result obtained by the fifth maximum pooling layer is processed by the fourteenth 1×1 convolution layer; the processing result of the thirteenth 1×1 convolution layer is processed by the ReLU function and the fifteenth 1×1 convolution layer to restore the feature to the original number of channels, and the processing result of the fourteenth 1×1 convolution layer is processed by the ReLU function and the sixteenth 1×1 convolution layer to restore the feature to the original number of channels, the processing results of the fifteenth 1×1 convolution layer and the sixteenth 1×1 convolution layer are element-wise added and processed by the second Sigmoid function, and the result is multiplied with the original input of the spatial channel attention module; finally, the features processed by the spatial attention module and the channel attention module are element-wise added to obtain low-level features containing rich spatial information.

9. The method for segmenting lesions in dermoscopic images according to claim 8, characterized in that: The mutual embedding module includes: a second along-channel average pooling layer, a second average pooling layer, a seventeenth 1×1 convolution layer to a twentieth 1×1 convolution layer; in the first part, features from the spatial channel attention module are imported into the seventeenth 1×1 convolution layer for processing, features from the cascade fusion module are imported into the nineteenth 1×1 convolution layer for processing, and then imported into the second average pooling layer and the twentieth 1×1 convolution layer for processing, and the processing result of the seventeenth 1×1 convolution layer is element-wise multiplied with the processing result of the twentieth 1×1 convolution layer; in the second part, features from the cascade fusion module are imported into the nineteenth 1×1 convolution layer for processing, features from the spatial channel attention module are imported into the seventeenth 1×1 convolution layer for processing, and then imported into the second along-channel average pooling layer and the eighteenth 1×1 convolution layer for processing, and the processing result of the eighteenth 1×1 convolution layer is element-wise multiplied with the processing result of the nineteenth 1×1 convolution layer; the result of the first part is element-wise added with the result of the second part to fuse the features and obtain a feature map containing high-level semantic information and spatial position information.

10. The method for segmenting lesions in dermatoscopic images according to claim 9, characterized in that: The specific operation process of step S4 and step S5 is as follows: Step S4: The data-enhanced training set is imported into the segmentation network in batches. The loss function adopts a combination of a binary cross entropy loss function and an intersection-over-union loss function. The features processed by the backbone network are imported into the cascade fusion module and the mutual embedding module to generate a high-level feature map. The corresponding feature loss is calculated with the corresponding segmentation label through the loss function; the feature loss from the cascade fusion module is added to the feature loss from the mutual embedding module to obtain the total loss, and then the parameters of the segmentation network are optimized by the Adam W optimizer and the total loss back propagation. After each epoch of complete training, the training effect of the segmentation network is verified using the validation set. The optimal segmentation model is obtained when the Jaccard coefficient of the validation set is the largest. During the training process of the segmentation network, the initial learning rate is set to 0.0001, and the cosine annealing strategy is introduced to adjust the learning rate. The batch size is set to 8, and 200 epochs are trained. Step S5: Import the dermoscopic images in the test set into the trained optimal segmentation model to obtain a segmentation result map corresponding to the dermoscopic images.

Images