A Skin Lesion Image Segmentation Method, Computer Device, and Medium

Abstract

The present invention discloses a method for skin lesion image segmentation, a computer device and a medium, which relate to the field of image segmentation. The method includes inputting a target lesion image into a trained image segmentation model to obtain a skin lesion image segmentation result. The image segmentation model adopts three multi-scale feature extraction modules to fuse features of different scales extracted during the encoding process of the feature encoder, so as to integrate shallow detail information and deep semantic information. The multi-stage context information after fusion is extracted again through the multi-field perception guidance module and the feature decoder, providing rich context information, improving the segmentation performance, and making the obtained skin lesion image segmentation result more accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of image segmentation, and particularly to a method for segmenting skin lesion images, a computer device, and a medium. Background Art

[0002] Due to the rapid increase in the number of skin cancer patients, skin cancer is listed as one of the fastest-growing cancers globally. Skin cancer is divided into two categories, namely non-melanoma and melanoma. Non-melanoma is a more common type of skin cancer, while melanoma is a less common type of skin cancer. However, melanoma is the most dangerous. Although it accounts for only 5% of skin malignancies, the mortality rate is as high as over 75%. Skin cancer can spread to other parts of the body and is usually incurable; early detection can prevent the spread because it is curable. Due to the nature of skin cancer, it is very important to detect it timely and accurately. Dermoscopy, as a non-invasive imaging technology, plays a key role in the diagnosis process of skin cancer. However, manually examining dermoscopy images is a time-consuming task that requires professional experience. To meet these needs, a computer-aided diagnosis system can be introduced to assist dermatologists in more efficient lesion detection and analysis. In a computer-aided diagnosis system, the primary task is to segment the lesion area, and the accuracy and reliability of this step greatly affect the performance of the entire system. Recognizing the important impact of lesion area segmentation on the performance of a computer-aided diagnosis system, a large number of studies have been devoted to the automatic segmentation of the lesion area, including classical machine learning and deep learning methods. Machine learning mainly relies on feature extraction methods, and appropriate feature extraction techniques need to be designed and selected for specific tasks, which may lead to poor performance in complex scenarios. In contrast, deep learning methods can automatically extract features without much human intervention and are superior to classical machine learning methods in the field of skin lesion segmentation, so they dominate in this field.

[0003] In recent years, the feature encoder-feature decoder structure represented by U-Net has been widely used in medical image segmentation tasks. This structure extracts high-level semantic features by gradually stacking convolutional layers and downsampling layers, and restores spatial information layer by layer in the top-down feature decoder. At the same time, by designing multiple skip connections to fuse shallow detail information and deep semantic information, the problem of detail information loss caused by downsampling is solved. Jha D et al. designed multiple global pyramid guidance modules between the feature encoder and the feature decoder based on the U-shaped structure, aiming to provide different levels of global context information for the feature decoder by reconstructing the skip connections. In addition, a scale-aware pyramid fusion module was designed to dynamically fuse multi-scale context information in high-order features. These two pyramid modules can gradually mine and fuse rich context information. However, the above method still has deficiencies in multi-scale fusion, which limits the extraction of global context information. This method proposed pyramid modules to improve the fusion of multi-scale context information, but the pyramid modules only perform multi-scale fusion of deep features to shallow features, ignoring the contribution of detail information in shallower features to network segmentation. Summary of the Invention

[0004] The purpose of the present invention is to provide a skin lesion image segmentation method, a computer device and a medium, which can improve the segmentation performance.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A skin lesion image segmentation method, the method comprising:

[0007] Obtain a target lesion image;

[0008] Input the target lesion image into a trained image segmentation model to obtain a skin lesion image segmentation result; the image segmentation model includes a feature encoder, a multi-scale feature extraction module, a multi-field perception guidance module, a feature decoder and a multi-feature fusion module;

[0009] Wherein, the feature encoder is used to extract multi-scale features from the target lesion image to obtain a first encoded feature, a second encoded feature, a third encoded feature, a fourth encoded feature and a fifth encoded feature;

[0010] The multi-scale feature extraction module includes a first multi-scale feature extraction unit, a second multi-scale feature extraction unit and a third multi-scale feature extraction unit;

[0011] The first multi-scale feature extraction unit is used to fuse the first encoded feature, the second encoded feature and the third encoded feature to obtain a first multi-scale feature;

[0012] The second multi-scale feature extraction unit is used to fuse the second encoded feature, the third encoded feature, and the fourth encoded feature to obtain a second multi-scale feature;

[0013] The third multi-scale feature extraction unit is used to fuse the third encoded feature, the fourth encoded feature, and the fifth encoded feature to obtain a third multi-scale feature;

[0014] The multi-field perception guidance module is used to extract features from the fifth encoded feature by using a first dilated convolution with multiple different dilation rates, and fuse the features extracted by each of the first dilated convolutions to obtain a multi-field fusion feature;

[0015] The feature decoder is used to perform decoding processing on the first encoded feature, the second encoded feature, the third encoded feature, the fourth encoded feature, the fifth encoded feature, the first multi-scale feature, the second multi-scale feature, the third multi-scale feature, and the multi-field fusion feature to obtain a decoded feature;

[0016] The multi-feature fusion module is used to fuse the decoded feature, the first decoded feature, and the second decoded feature to obtain a skin lesion image segmentation result; the first decoded feature and the second decoded feature are respectively the penultimate layer feature and the antepenultimate layer feature decoded by the feature decoder during the decoding process.

[0017] Optionally, the image segmentation model further includes a sub-path module;

[0018] The sub-path module is used for:

[0019] Performing an addition operation on the second encoded feature and the first multi-scale feature to obtain a first addition feature;

[0020] Performing an addition operation on the first addition feature and the first encoded feature to obtain a second addition feature;

[0021] Performing multiple upsamplings on the second addition feature to obtain a boundary feature; the boundary feature is used to be input into the multi-feature fusion module to be fused with the decoded feature, the first decoded feature, and the second decoded feature to obtain a final skin lesion image segmentation result.

[0022] Optionally, the multi-feature fusion module is used for:

[0023] Concatenating the boundary feature, the decoded feature, the first decoded feature, and the second decoded feature to obtain a concatenated feature;

[0024] Performing a spatial attention operation on the concatenated feature to obtain a first attention feature;

[0025] Perform channel attention operation on the splicing feature to obtain a second attention feature;

[0026] Perform channel attention operation and spatial attention operation on the splicing feature in sequence to obtain a third attention feature;

[0027] Fuse the first attention feature, the second attention feature and the third attention feature to obtain the final skin lesion image segmentation result.

[0028] Optionally, the feature encoder is an improved Resnet34 network, and the improved Resnet34 network is obtained by deleting the last average pooling layer and fully connected layer from the Resnet34 network.

[0029] Optionally, the image segmentation model further includes a first preprocessing module, and the first preprocessing module is used for:

[0030] Perform max pooling operation and depthwise separable convolution processing on the first encoded feature in sequence to obtain a first preprocessed feature;

[0031] Perform upsampling operation and depthwise separable convolution processing on the third encoded feature in sequence to obtain a third preprocessed feature.

[0032] Optionally, the first multi-scale feature extraction unit is used for:

[0033] Splice the first preprocessed feature and the third preprocessed feature in the channel dimension to obtain a first channel feature;

[0034] Extract features from the first channel feature by using multiple second atrous convolutions with different atrous rates, and splice the features extracted by each second atrous convolution with the second preprocessed feature in the channel dimension to obtain a second channel feature; the second preprocessed feature is the second encoded feature;

[0035] Extract features from the second channel feature by using multiple third atrous convolutions with different atrous rates, and splice the features extracted by each third atrous convolution in the channel dimension to obtain a third channel feature;

[0036] Perform multiple convolution operations on the third channel feature to obtain a first convolution feature; the first convolution feature is the first multi-scale feature.

[0037] Optionally, the image segmentation model further includes a second preprocessing module, and the second preprocessing module is used for:

[0038] Perform upsampling operation and depthwise separable convolution operation on the first convolution feature in sequence to obtain a second convolution feature;

[0039] Perform a max pooling operation and a depthwise separable convolution operation on the first convolution feature in sequence to obtain a third convolution feature; the first convolution feature, the second convolution feature, and the third convolution feature constitute the first multi-scale feature; the first convolution feature and the second convolution feature are used as inputs to the feature decoder.

[0040] Optionally, the multi-field of view perception guidance module is used for:

[0041] Extract features from the fifth encoded feature using three first atrous convolutions with different atrous rates to obtain a first atrous convolution feature, a second atrous convolution feature, and a third atrous convolution feature respectively;

[0042] Fuse the first atrous convolution feature and the second atrous convolution feature to obtain a fused feature;

[0043] Fuse the fused feature, the third atrous convolution feature, and the fifth encoded feature to obtain a multi-field of view fused feature.

[0044] The present invention also provides a computer device, including a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor is caused to execute the steps of the above skin lesion image segmentation method.

[0045] The present invention also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded and executed by a processor to perform the above skin lesion image segmentation method.

[0046] According to the specific embodiments provided by the present invention, the following technical effects are disclosed: The present invention provides a method for segmenting skin lesion images, a computer device, and a medium. The method includes: obtaining a target lesion image; inputting the target lesion image into a trained image segmentation model to obtain a skin lesion image segmentation result; the image segmentation model includes a feature encoder, a multi-scale feature extraction module, a multi-field perception guidance module, a feature decoder, and a multi-feature fusion module; wherein, the feature encoder is used to extract features of multiple scales from the target lesion image to obtain a first encoded feature, a second encoded feature, a third encoded feature, a fourth encoded feature, and a fifth encoded feature; the multi-scale feature extraction module includes a first multi-scale feature extraction unit, a second multi-scale feature extraction unit, and a third multi-scale feature extraction unit; the first multi-scale feature extraction unit is used to fuse the first encoded feature, the second encoded feature, and the third encoded feature to obtain a first multi-scale feature; the second multi-scale feature extraction unit is used to fuse the second encoded feature, the third encoded feature, and the fourth encoded feature to obtain a second multi-scale feature; the third multi-scale feature extraction unit is used to fuse the third encoded feature, the fourth encoded feature, and the fifth encoded feature to obtain a third multi-scale feature; the multi-field perception guidance module is used to extract features from the fifth encoded feature by using a first dilated convolution with multiple different dilation rates, and fuse the features extracted by each first dilated convolution to obtain a multi-field fusion feature; the feature decoder is used to perform decoding processing on the first encoded feature, the second encoded feature, the third encoded feature, the fourth encoded feature, the fifth encoded feature, the first multi-scale feature, the second multi-scale feature, the third multi-scale feature, and the multi-field fusion feature to obtain a decoded feature; the multi-feature fusion module is used to fuse the decoded feature, a first decoded feature, and a second decoded feature to obtain a skin lesion image segmentation result; the first decoded feature and the second decoded feature are respectively the penultimate layer feature and the antepenultimate layer feature decoded by the feature decoder during the decoding process. The present invention adopts three multi-scale feature extraction modules to fuse features of different scales extracted during the encoding process of the feature encoder, so as to integrate shallow detail information and deep semantic information. The multi-field perception guidance module and the feature decoder extract features from the fused multi-stage context information (i.e., multi-scale features) again, providing rich context information, improving the segmentation performance, and obtaining a more accurate skin lesion image segmentation result. Description of the Drawings

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0048] Figure 1 Schematic diagram of the skin lesion image segmentation method provided by the embodiment of the present invention;

[0049] Figure 2 Schematic diagram of the structure of the image segmentation model provided by the embodiment of the present invention;

[0050] Figure 3 Schematic diagram of the structure of the first multi-scale feature extraction unit provided by the embodiment of the present invention;

[0051] Figure 4 Schematic diagram of the structure of the multi-view perception guidance module provided by the embodiment of the present invention;

[0052] Figure 5 Schematic diagram of the structure of the multi-feature fusion module provided by the embodiment of the present invention;

[0053] Figure 6 Schematic diagram of the structure of the feature encoder provided by the embodiment of the present invention;

[0054] Figure 7 Schematic diagram of the target lesion image provided by the embodiment of the present invention;

[0055] Figure 8 Schematic diagram of the skin lesion image segmentation result provided by the embodiment of the present invention;

[0056] Figure 9 Schematic diagram of the training process of the image segmentation model provided by the embodiment of the present invention;

[0057] Figure 10 Schematic diagram of the structure of a computer device provided by the embodiment of the present invention. Detailed implementation manners

[0058] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0059] CN115530766A proposes an improved MultiResUNet model. This invention retains part of the structure in the MultiResUNet model and improves the segmentation performance of the model by integrating channel and spatial attention mechanism structures. CN116205888A proposes to use a dense encoding module for feature extraction in the encoding part, propose a Gaussian pooling layer according to the degree of correlation of each feature point in the feature map, and design an efficient hybrid attention mechanism to extract features in both the spatial and channel dimensions.

[0060] However, neither of these two methods takes into account the problems of poor segmentation results caused by the variable size, irregular shape, and blurred boundary of the skin lesion area. To overcome the deficiencies of the above methods, the present invention proposes a sub-path specifically used to extract boundary information, which aims to enhance the network's ability to extract boundary features, thereby improving the fineness of skin lesion boundary segmentation. This helps to solve the problems of variable size, irregular shape, and blurred boundary of the skin lesion area, and further improves the accuracy and detail expression ability of skin lesion segmentation.

[0061] The object of the present invention is to provide a skin lesion image segmentation method, a computer device, and a medium. A dual-path network model (i.e., an image segmentation model) that fuses multi-features and multi-fields of view is constructed and trained. The preprocessed image is input into the trained network model, and the segmentation result of the input image is obtained in an end-to-end manner. The dual-path network model that fuses multi-features and multi-fields of view includes a feature encoder, a multi-scale feature extraction module, a multi-field-of-view perception guidance module, a sub-path, and a multi-feature fusion feature decoder. These components work together to provide more comprehensive feature expression and more accurate segmentation results. The feature encoder is used to extract low-level and high-level features of the image. The multi-scale feature extraction module is used to extract features from different scales. The multi-field-of-view perception guidance module is used to guide the network to focus on features in different fields of view. The sub-path is specifically used to process boundary information, and the multi-feature fusion feature decoder fuses the features of each path to generate the final segmentation result. By using the dual-path network model that fuses multi-features and multi-fields of view, the present invention can improve the accuracy and robustness of skin lesion image segmentation, and provides an effective solution for research and application in the field of medical image analysis. Three multi-scale feature extraction modules are adopted to fuse the features of layers E1, E2, E3, E4, and E5 respectively to integrate shallow detail information and deep semantic information. The fused multi-stage context information is extracted again, and rich context information is provided for the feature decoder through skip connections to further improve the segmentation performance. It solves the problem that in the previous methods, due to the limited single-stage feature extraction ability during the upsampling process of the feature decoder, the effective information is lost during the upsampling process of the global context information captured in the deep layer. The design of the dual-path enhances the network's ability to segment the boundary of the lesion area, effectively improving the network's extraction ability and the accuracy of lesion area segmentation.

[0062] To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0063] As Figure 1 shown, the present invention provides a skin lesion image segmentation method, and the method includes:

[0064] S1: Obtain the target lesion image.

[0065] S2: Input the target lesion image into the trained image segmentation model to obtain the skin lesion image segmentation result; the structure of the image segmentation model is as Figure 2 shown, and the image segmentation model includes a feature encoder, a multi-scale feature extraction module, a multi-field perception guidance module, a feature decoder, and a multi-feature fusion module.

[0066] Among them, the feature encoder is used to extract features of multiple scales from the target lesion image to obtain the first encoded feature, the second encoded feature, the third encoded feature, the fourth encoded feature, and the fifth encoded feature.

[0067] The multi-scale feature extraction module includes a first multi-scale feature extraction unit, a second multi-scale feature extraction unit, and a third multi-scale feature extraction unit.

[0068] The first multi-scale feature extraction unit is used to fuse the first encoded feature, the second encoded feature, and the third encoded feature to obtain the first multi-scale feature. The second multi-scale feature extraction unit is used to fuse the second encoded feature, the third encoded feature, and the fourth encoded feature to obtain the second multi-scale feature; the third multi-scale feature extraction unit is used to fuse the third encoded feature, the fourth encoded feature, and the fifth encoded feature to obtain the third multi-scale feature.

[0069] The multi-field perception guidance module is used to extract features from the fifth encoded feature by using a first dilated convolution with multiple different dilation rates, and fuse the features extracted by each of the first dilated convolutions to obtain a multi-field fusion feature.

[0070] The feature decoder is used to perform decoding processing on the first encoded feature, the second encoded feature, the third encoded feature, the fourth encoded feature, the fifth encoded feature, the first multi-scale feature, the second multi-scale feature, the third multi-scale feature, and the multi-field fusion feature to obtain a decoded feature.

[0071] The multi-feature fusion module is used to fuse the decoded feature, the first decoded feature, and the second decoded feature to obtain the skin lesion image segmentation result; the first decoded feature and the second decoded feature are the penultimate layer feature and the third-to-last layer feature decoded by the feature decoder during the decoding process, respectively.

[0072] Such as Figure 6The network structure within the shown box is the actual network structure of the feature encoder. The feature encoder is an improved Resnet34 network, which is obtained by deleting the last average pooling layer and fully connected layer from the Resnet34 network. The feature encoder adopts the transfer learning method and uses five stages in the ResNet34 network that has been trained on the ImageNet dataset as the E1, E2, E3, E4, and E5 layers of the feature encoder, namely the first encoded feature, the second encoded feature, the third encoded feature, the fourth encoded feature, and the fifth encoded feature respectively. The difference between the feature encoder and the ResNet34 network is that it deletes the last average pooling layer (avgpool) and fully connected layer (fc) of the ResNet34 network, and extracts feature information of different scales layer by layer through downsampling operations; every adjacent three layers of features are used as the input of the multi-scale feature extraction module; meanwhile, the E5 layer is also used as the input feature of the multi-field perception guidance module; through the above design, the feature encoder can effectively extract rich multi-scale feature information and provide rich feature inputs for subsequent paths and modules.

[0073] In this embodiment, the image segmentation model further includes a first preprocessing module, and the first preprocessing module is used for:

[0074] Performing a max pooling operation and depthwise separable convolution processing on the first encoded feature in sequence to obtain a first preprocessed feature; performing an upsampling operation and depthwise separable convolution processing on the third encoded feature in sequence to obtain a third preprocessed feature.

[0075] Fusing the first encoded feature, the second encoded feature, and the third encoded feature to obtain a first multi-scale feature, which specifically includes:

[0076] Concatenating the first preprocessed feature and the third preprocessed feature in the channel dimension to obtain a first channel feature; using multiple second atrous convolutions with different atrous rates to extract features from the first channel feature, and concatenating the features extracted by each second atrous convolution with the second preprocessed feature in the channel dimension to obtain a second channel feature; the second preprocessed feature is the second encoded feature; using multiple third atrous convolutions with different atrous rates to extract features from the second channel feature, and concatenating the features extracted by each third atrous convolution in the channel dimension to obtain a third channel feature; performing multiple convolution operations on the third channel feature to obtain a first convolution feature; the first convolution feature is the first multi-scale feature.

[0077] The image segmentation model further includes a second preprocessing module, and the second preprocessing module is used for:

[0078] Perform an upsampling operation and a depthwise separable convolution operation on the first convolution feature in sequence to obtain a second convolution feature; perform a max pooling operation and a depthwise separable convolution operation on the first convolution feature in sequence to obtain a third convolution feature; the first convolution feature, the second convolution feature, and the third convolution feature constitute the first multi-scale feature; the first convolution feature and the second convolution feature are used as inputs to the feature decoder.

[0079] As Figure 3 shown, the multi-scale feature extraction module takes three adjacent layers of features ((E1, E2, E3), (E2, E3, E4), (E3, E4, E5)) in the feature encoder as inputs. Taking the first multi-scale feature extraction unit in the multi-scale feature extraction module of the image segmentation model as an example for illustration, the specific steps are as follows:

[0080] S201: Perform a max pooling operation on the feature E1 in the feature encoder, and use a depthwise separable convolution to adjust the number of channels of the feature E1 to 1 / 2 of the number of channels of the feature E2, to obtain a first preprocessed feature Y1.

[0081] S202: Perform an upsampling operation on the feature E3, and then use a depthwise separable convolution to adjust the number of channels of the feature E3 to 1 / 2 of the number of channels of the feature E2, to obtain a third preprocessed feature Y3.

[0082] S203: Directly use the feature E2 as the input feature Y2, and together with the first preprocessed Y1 and the third preprocessed feature Y3 as the inputs to the multi-scale feature extraction module.

[0083] S204: Concatenate the input features Y1 and Y3 in the channel dimension to generate a first channel feature Z1, and then perform atrous convolution operations with atrous rates of 1, 2, and 4 on the first channel feature Z1 respectively. Among them, the number of channels of the feature generated by the second atrous convolution with an atrous rate of 1 is 1 / 2 of the number of channels of the first channel feature Z1, and the number of channels of the features generated by the atrous convolutions with atrous rates of 2 and 4 is 1 / 4 of the number of channels of the first channel feature Z1. This process formula can be expressed as:

[0084] Z1 = Concat(Y1, Y3).

[0085] where Concat represents feature concatenation in the channel dimension, D i represents the atrous convolution operation with an atrous rate of i.

[0086] S205: Concatenate the new features generated after dilated convolution with Feature Y2 in the channel dimension to generate the second-channel feature Z2. Similarly, perform dilated convolutions with dilation rates of 1, 2, and 4 on the second-channel feature Z2 respectively. Among them, the number of feature channels generated by the dilated convolution operation with a dilation rate of 1 is 1 / 2 of the number of channels of the second-channel feature Z2, and the number of feature channels generated by the dilated convolution operations with dilation rates of 2 and 4 is 1 / 4 of the number of channels of the second-channel feature Z2. The formula for this process can be expressed as:

[0087] Z2 = Concat(D i (Z1), Y2), i ∈ 1, 2, 4.

[0088] S206: Concatenate the new features generated after dilated convolution in the channel dimension to generate the third-channel feature Z3. Perform two 3×3 convolutions on the third-channel feature Z3 and reduce the number of channels to 1 / 2 of the original to obtain the first convolutional feature Z4 after multi-scale feature fusion. The formula for generating the third-channel feature Z3 can be expressed as:

[0089] Z3 = Concat(D i (Z2)), i ∈ 1, 2, 4.

[0090] S207: Perform upsampling and depthwise separable convolution operations on the first convolutional feature Z4 to obtain the second convolutional feature Y1 extracted after multi-scale fusion. The feature Z4 is directly used as the re-extracted feature Y2. Perform max pooling and depthwise separable convolution operations on the feature Z4 to obtain the third convolutional feature Y3 extracted after multi-scale fusion. The re-extracted Y1, Y2, and Y3 are input to the decoder through skip connections.

[0091] The above steps show the operation process of the multi-scale feature extraction module. By utilizing features at different levels and performing processing and fusion, the extraction of multi-scale context information is achieved, providing richer feature inputs for the decoder.

[0092] The multi-field-of-view perception guidance module is used for:

[0093] Extract features from the fifth encoded feature using three first dilated convolutions with different dilation rates to obtain a first dilated convolutional feature, a second dilated convolutional feature, and a third dilated convolutional feature respectively; fuse the first dilated convolutional feature and the second dilated convolutional feature to obtain a fused feature; fuse the fused feature, the third dilated convolutional feature, and the fifth encoded feature to obtain a multi-field-of-view fused feature.

[0094] As Figure 4As shown, the multi-field perception guidance module consists of three parallel dilated convolutions and two cascaded guidance fusion modules. The E5 layer features (with a size of 512×6×8) in the feature encoder are used as the input of this module. The multi-field perception guidance module is embedded between the feature encoder and the multi-feature fusion decoder, and can dynamically select receptive fields of different scales through self-learning to better fuse multi-scale context information. The specific process of this module includes:

[0095] S301: First, use three parallel first dilated convolutions with dilation rates of 1, 2, and 3 respectively to capture the feature information of the input feature E5 at different scales, and generate new features, namely the first dilated convolution feature M1, the second dilated convolution feature M2, and the third dilated convolution feature M3 with a size of 1024×6×8 through the dilated convolution operation. The formula can be expressed as:

[0096] M i =D i (E5), i ∈ 1, 2, 3.

[0097] Among them, D i represents the dilated convolution operation with a dilation rate of i, and M i represents the new feature generated after the first dilated convolution operation with a dilation rate of i.

[0098] S302: Perform a 1×1 convolution operation on the first dilated convolution feature M1 to generate features with a size of 512×6×8, and merge the last two dimensions of the features to generate features with a size of 512×48. Also use a 1×1 convolution operation and the last two dimensions merging operation on the feature M2, and then swap the 1st and 2nd dimensions to generate features with a size of 48×512. Finally, perform matrix multiplication on the processed first dilated convolution feature M1 and the feature M2, and use the Sigmoid activation function on the last dimension of the matrix to generate a weight matrix with a size of 48×48.

[0099] S303: Again, perform a 1×1 convolution operation, the last two dimensions merging operation, and the 1st and 2nd dimensions swapping operation on the second dilated convolution feature M2 in sequence to generate features with a size of 48×512, then perform matrix multiplication on the 48×48 weight matrix and the 48×512 features to generate features with a size of 48×512, perform the 1st and 2nd dimensions swapping operation on the features and split the last dimension into 6×8 operations, and finally generate new features with a size of 512×6×8 obtained by guiding the feature M2 with the feature M1.

[0100] S304: Use a 1×1 convolution on the new features generated in S303 to change them into features with a size of 1024×6×8, and then add the features to the feature M1 to obtain the intermediate guidance feature out1 (i.e., the fusion feature) of the multi-field perception guidance module. The formula for this process can be expressed as:

[0101]

[0102] Among them, Conv1 represents a 1×1 convolution operation, the superscript T represents the transpose operation on the matrix, and σ represents the use of the softmax activation function on the last dimension of the matrix. represents matrix multiplication.

[0103] S305: Use a 1×1 convolution operation on the feature out1 to generate a feature of size 512×6×8, then merge the last two dimensions of the feature to generate a feature of 512×48. Also use a 1×1 convolution operation and the last two dimensions merging operation on the feature M3, and then swap the first and second dimensions to generate a feature of 48×512. Then perform matrix multiplication on the processed feature out1 and the feature M3, and use the Sigmoid activation function on the last dimension of the matrix to generate a 48×48 weight matrix.

[0104] S306: Again, perform a 1×1 convolution operation, the last two dimensions merging operation, and the first and second dimensions swapping operation on the feature M3 in sequence to generate a feature of 48×512. Then perform matrix multiplication on the 48×48 weight matrix and the 48×512 feature to generate a feature of 48×512. Perform the first and second dimensions swapping and the last dimension splitting into 6×8 operations on the feature, and finally generate a new feature of size 512×6×8 obtained by guiding the feature M3 with the feature out1.

[0105] S307: Use a 1×1 convolution on the newly generated feature to change it into a feature of 1024×6×8, add the feature to the feature out1 and then perform a 3×3 convolution operation to generate a new feature of 512×6×8. Finally, add the new feature to the feature E5 to obtain the final output feature out of the multi - view perception guiding module (i.e., the multi - view fusion feature). The formula can be expressed as:

[0106]

[0107] Among them, Conv3 represents a 3×3 convolution operation, Conv1 represents a 1×1 convolution operation, the superscript T represents the transpose operation on the matrix, and σ represents the use of the softmax activation function on the last dimension of the matrix. represents matrix multiplication.

[0108] Embed the multi - view perception guiding module between the feature encoder and the feature decoder. By using three parallel dilated convolutions and two cascaded guiding fusion modules, it can self - learn to dynamically select receptive fields of different scales and better fuse multi - scale context information.

[0109] In this embodiment, the image segmentation model further includes a sub - path module; the sub - path module is used for:

[0110] An addition operation is performed on the second encoded feature and the first multi-scale feature to obtain a first addition feature; an addition operation is performed on the first addition feature and the first encoded feature to obtain a second addition feature; the second addition feature is subjected to multiple upsampling operations to obtain a boundary feature; the boundary feature is used to be input into the multi-feature fusion module to be fused with the decoded feature, the first decoded feature, and the second decoded feature to obtain the final skin lesion image segmentation result.

[0111] Specifically: The sub-path is used to extract the boundary information (boundary feature) of the lesion area to enhance the network's ability to extract boundary features. The sub-path takes the first multi-scale feature extraction module in the network model as the input, and sequentially takes the features of the E1 and E2 layers of the feature encoder as skip connection inputs, and is composed of two upsampling operations.

[0112] The features of the D1 and D2 layers in the feature decoder are input into the multi-feature fusion module after upsampling and depthwise separable convolution.

[0113] The multi-feature fusion module is used for:

[0114] The boundary feature, the decoded feature, the first decoded feature, and the second decoded feature are concatenated to obtain a concatenated feature; a spatial attention operation is performed on the concatenated feature to obtain a first attention feature; a channel attention operation is performed on the concatenated feature to obtain a second attention feature; a channel attention operation and a spatial attention operation are sequentially performed on the concatenated feature to obtain a third attention feature; the first attention feature, the second attention feature, and the third attention feature are fused to obtain the final skin lesion image segmentation result.

[0115] As Figure 5 shown, the multi-feature fusion module takes the preprocessed sub-path boundary segmentation and the preprocessed last three layers of the decoder for overall segmentation as input features. The preprocessing operation first performs upsampling and depthwise separable convolution operations on the features of the D2 (size 64×48×64) and D1 (size 64×96×128) layers in the decoder, and secondly performs 3×3 convolution operations on the boundary segmentation predicted by the sub-path and the overall segmentation predicted by the main path to generate four features of size 16×192×256. The specific process of the multi-feature fusion module includes:

[0116] S31. Concatenate the four input features of size 16×192×256 in the channel dimension to generate a concatenated feature W of size 64×192×256, and input the feature W into three parallel branches for processing.

[0117] S32. The first branch performs spatial attention operation on the splicing feature W to obtain the first attention feature. The formula can be expressed as:

[0118] branch1 = δ(Conv7(Concat(AvgPool(W), MaxPool(W)))) · W.

[0119] Where branch1 is the first attention feature, δ represents the sigmoid activation function, Conv7 represents a 7×7 convolution operation, Concat represents feature splicing in the channel dimension, AvgPool represents the average pooling operation, MaxPool represents the max pooling operation, and · represents matrix dot multiplication.

[0120] S33. The second branch performs channel attention operation on the splicing feature W to obtain the second attention feature. The formula can be expressed as:

[0121] branch2 = δ(MLP(AvgPool(W)) + MLP(MaxPool(W)) + MLP(SoftPool(W))) ·· W

[0122] Where branch2 is the second attention feature, δ represents the sigmoid activation function, MLP represents a shared multi-layer perceptron, AvgPool represents the average pooling operation, MaxPool represents the max pooling operation, SoftPool represents the soft pooling operation, and · represents matrix dot multiplication.

[0123] S34. The third branch first performs channel attention operation on the splicing feature W, and then performs spatial attention operation to obtain the third attention feature.

[0124] S34. Add the features obtained by the three branches to obtain the final output of the multi-feature fusion module.

[0125] Through the above steps, the multi-feature fusion module can make full use of the sub-path boundary segmentation and the overall segmentation of different scales of the decoder, and extract the spatial and channel-related information of the features through spatial attention and channel attention operations. Finally, add the features obtained by the three branches to achieve the fusion of multi-features, so as to obtain the final segmentation result of the dual-path skin lesion image segmentation method based on multi-feature and multi-field-of-view fusion.

[0126] Before inputting the target lesion image into the trained image segmentation model, it is also necessary to train the image segmentation model. As Figure 9 shown, the training process of the image segmentation model includes the following steps:

[0127] S101: Obtain the ISIC 2018 dataset of skin lesion images, which contains 2,594 skin lesion images with different resolutions. To unify the image resolution, all images in the dataset are scaled to 192×256 pixels, and then the dataset is divided into a training set, a validation set, and a test set for model training, parameter selection, and performance evaluation. Specifically, 1,816 images are randomly selected from the dataset as the training set, 260 images as the validation set, and the remaining 518 images as the test set.

[0128] S102: Input the preprocessed training set images into the image segmentation model. The sub-path generates the boundary segmentation of the lesion area, and then the multi-feature fusion module fuses the features of the last three layers of the main-path decoder and the sub-path features to generate the overall segmentation of the lesion area.

[0129] Use the training set data to train the network model. Before training, to increase the diversity of skin lesion images, preprocessing is performed during the training phase. The preprocessing includes: using three strategies of vertical flipping with a 50% probability, horizontal flipping with a 50% probability, and random flipping between -30 degrees and +30 degrees to enhance the diversity of the images. Data augmentation strategies are not used during the validation and test phases. In the training process of the dual-path skin lesion image segmentation method network with multi-feature and multi-field fusion of the present invention, a five-fold cross-validation method is adopted, and the average value of the evaluation metrics is used as the final performance.

[0130] S103: Use the loss function to calculate the losses of the boundary segmentation and the overall segmentation respectively. After adding the two losses, perform backpropagation to update the parameters of the network model; use the validation set images to select the model parameters, and finally use the test set images to evaluate the performance of the segmentation method.

[0131] Specifically: The present invention is implemented on an NVIDIA GeForce RTX 3080Ti using the PyTorch framework. The Stochastic Gradient Descent (SGD) optimization model with a momentum of 0.9 and a weight decay of 1e-4 is used. To dynamically adjust the learning rate, a polynomial learning rate decay strategy is adopted, where the initial learning rate and batch size are set to 0.1 and 16 respectively. The polynomial formula can be expressed as:

[0132]

[0133] where, lr i represents the learning rate in the i-th epoch, and lr i-1 represents the learning rate in the (i - 1)-th epoch. In the present invention, epoch is 150 and exp is 0.9.

[0134] During the training process of the dual-path skin lesion image segmentation method based on multi-feature and multi-field fusion of the present invention, in order to solve the problem of class imbalance, a joint loss function is adopted. The joint loss function is composed of cross-entropy loss and Dice loss, and the formula can be expressed as:

[0135]

[0136]

[0137] L = L BCE + L Dice 。

[0138] Among them, L is the sum of the two losses added together, L BCE is the cross-entropy loss function, and L Dice is the Dice loss function. g i ∈ {0, 1}, p i ∈ {0, 1} respectively represent the true segmentation result (groundtruth) and the predicted segmentation result of the image segmentation model. ε ∈ R ensures numerical stability to avoid a denominator of 0. N = W × H is the number of pixels.

[0139] In this embodiment, four metrics, namely the Jaccard index (Jaccard similarity coefficient) (JI), Dice coefficient (Dice Coefficient), accuracy (ACC), and precision (Precision), are used to evaluate the performance of different methods. The formulas are as follows:

[0140]

[0141]

[0142]

[0143]

[0144] Among them, TP, TN, FP, and FN are true positive, true negative, false positive, and false negative, respectively.

[0145] S104: After each round of training of the image segmentation model on the training set, the validation set data is input into the network and predicted using the parameters of this round of training. The prediction result is used as an evaluation of the image segmentation model. A total of 150 iterations are performed, and the model parameters with the best Jaccard index performance on the validation set are automatically selected as the final model parameters of the dual-path skin lesion image segmentation method based on multi-feature and multi-field fusion.

[0146] S105: After the model is trained, use the final model parameters to perform segmentation prediction on the images in the test set, and evaluate the image segmentation model using four metrics: Dice coefficient (Dice), Jaccard index (JI), accuracy (ACC), and precision (Precision).

[0147] Through this training process, the dual-path network model with multi-feature and multi-field fusion can be gradually optimized to enable more accurate boundary segmentation and overall segmentation of skin lesion areas; at the same time, the use of the validation set and the test set also ensures the generalization ability and stability of the model.

[0148] After obtaining the trained image segmentation model, input the target lesion image into the trained image segmentation model. The output of the image segmentation model is an 8-bit PNG image of the same size as the input target lesion image, that is, the skin lesion image segmentation result. The target lesion image is as Figure 7 shown, and the skin lesion image segmentation result is as Figure 8 shown.

[0149] Through experiments, the comparison results of the segmentation effects of the image segmentation model of the present invention and other existing models are obtained. Table 1 shows the performance of different methods on the ISIC 2018 dataset. It can be clearly seen from the evaluation metrics that the segmentation effect of the image segmentation model of the present invention is significantly better than that of other models. The Jaccard index of the image segmentation model of the present invention is increased by 1.89%, 1.59%, 0.70%, 0.55%, and 0.68% compared with the U-Net, CA-Net, CE-Net, CPF-Net, and Ms-RED models, respectively.

[0150] Table 1 Performance of Different Methods on the ISIC 2018 Dataset

[0151] Model JI(%) Dice(%) ACC(%) Precision(%) U-Net 81.73±0.93 88.73±0.72 95.68±0.20 90.74±1.17 CA-Net 82.03±0.66 88.98±0.55 95.76±0.05 91.25±0.26 CE-Net 82.92±0.61 89.67±0.49 96.10±0.13 91.20±1.19 CPF-Net 83.07±0.62 89.70±0.49 96.06±0.16 91.76±1.46 Ms-RED 82.94±0.67 89.64±0.53 96.05±0.17 91.73±0.69 The present invention 83.62±0.59 90.16±0.46 96.27±0.13 92.08±0.68

[0152] The present invention also provides a computer device, including a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor is caused to execute the steps of the above skin lesion image segmentation method.

[0153] Please refer to Figure 10 , Figure 10 which is a schematic structural diagram of a computer device provided by the present application. As Figure 10As shown in the figure, the computer device 1000 may include: a processor 1001, a network interface 1004, and a memory 1005. In addition, the computer device 1000 may further include: a user interface 1003 and at least one communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. Among them, the user interface 1003 may include a display screen (Display) and a keyboard (Keyboard). Optionally, the user interface 1003 may further include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. Optionally, the memory 1005 may further be at least one storage device located far from the aforementioned processor 1001. As Figure 10 shown, the memory 1005, as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a device control application program.

[0154] In Figure 10 the computer device 1000 shown in the figure, the network interface 1004 can provide network communication functions; while the user interface 1003 is mainly used to provide an input interface for users; and the processor 1001 can be used to call the device control application program stored in the memory 1005 to implement the skin lesion image segmentation method described in the above embodiments, which will not be elaborated here.

[0155] The present invention also provides a computer-readable storage medium storing a computer program, which is suitable for being loaded and executed by a processor to implement the skin lesion image segmentation method described in the above embodiments, which will not be elaborated here.

[0156] The above program can be deployed to be executed on a computer device, or on multiple computer devices located at one place, or on multiple computer devices distributed at multiple places and interconnected through a communication network. The multiple computer devices distributed at multiple places and interconnected through a communication network can form a blockchain network.

[0157] The above computer-readable storage medium may be an internal storage unit of the above computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device. Further, the computer-readable storage medium may also include both the internal storage unit and the external storage device of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store the data that has been output or is to be output.

[0158] The various embodiments in this specification are described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple. For the relevant parts, reference can be made to the description in the method part.

[0159] Specific examples are used in this article to elaborate on the principles and implementation manners of the present invention. The descriptions of the above embodiments are only used to help understand the method of the present invention and its core idea. At the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A method for skin lesion image segmentation, characterized in that The method includes: Obtaining a target lesion image; Inputting the target lesion image into a trained image segmentation model to obtain a skin lesion image segmentation result; the image segmentation model includes a feature encoder, a multi-scale feature extraction module, a multi-field perception guidance module, a feature decoder, and a multi-feature fusion module; Wherein, the feature encoder is used to perform multi-scale feature extraction on the target lesion image to obtain a first encoded feature, a second encoded feature, a third encoded feature, a fourth encoded feature, and a fifth encoded feature; The multi-scale feature extraction module includes a first multi-scale feature extraction unit, a second multi-scale feature extraction unit, and a third multi-scale feature extraction unit; The image segmentation model further includes a first preprocessing module, and the first preprocessing module is used to: perform a max pooling operation and a depthwise separable convolution process on the first encoded feature in sequence to obtain a first preprocessed feature; perform an upsampling operation and a depthwise separable convolution process on the third encoded feature in sequence to obtain a third preprocessed feature; The first multi-scale feature extraction unit is used to fuse the first encoded feature, the second encoded feature, and the third encoded feature to obtain a first multi-scale feature; the first multi-scale feature extraction unit is used to: Concatenate the first preprocessed feature and the third preprocessed feature in the channel dimension to obtain a first channel feature; Use a plurality of second atrous convolutions with different atrous rates to extract features from the first channel feature, and concatenate the features extracted by each of the second atrous convolutions with a second preprocessed feature in the channel dimension to obtain a second channel feature; the second preprocessed feature is the second encoded feature; Use a plurality of third atrous convolutions with different atrous rates to extract features from the second channel feature, and concatenate the features extracted by each of the third atrous convolutions in the channel dimension to obtain a third channel feature; Perform multiple convolution operations on the third channel feature to obtain a first convolution feature; the first convolution feature is the first multi-scale feature; The second multi-scale feature extraction unit is used to fuse the second encoded feature, the third encoded feature, and the fourth encoded feature to obtain a second multi-scale feature; The third multi-scale feature extraction unit is used to fuse the third encoded feature, the fourth encoded feature, and the fifth encoded feature to obtain a third multi-scale feature; The multi-field perception guidance module is used to use a plurality of first atrous convolutions with different atrous rates to extract features from the fifth encoded feature, and fuse the features extracted by each of the first atrous convolutions to obtain a multi-field fusion feature; The feature decoder is used to perform decoding processing on the first encoded feature, the second encoded feature, the third encoded feature, the fourth encoded feature, the fifth encoded feature, the first multi-scale feature, the second multi-scale feature, the third multi-scale feature, and the multi-field fusion feature to obtain a decoded feature; The multi-feature fusion module is used to fuse the decoded feature, the first decoded feature, and the second decoded feature to obtain the skin lesion image segmentation result; the first decoded feature and the second decoded feature are respectively the penultimate layer feature and the antepenultimate layer feature decoded by the feature decoder during the decoding process.

2. The skin lesion image segmentation method according to claim 1, wherein The image segmentation model further includes a sub-path module; the sub-path module is used for: Performing an addition operation on the second encoded feature and the first multi-scale feature to obtain a first added feature; Performing an addition operation on the first added feature and the first encoded feature to obtain a second added feature; Performing multiple upsampling operations on the second added feature to obtain a boundary feature; the boundary feature is used to be input into the multi-feature fusion module to be fused with the decoded feature, the first decoded feature, and the second decoded feature to obtain the final skin lesion image segmentation result.

3. The skin lesion image segmentation method according to claim 2, characterized in that, The multi-feature fusion module is used for: Concatenating the boundary feature, the decoded feature, the first decoded feature, and the second decoded feature to obtain a concatenated feature; Performing a spatial attention operation on the concatenated feature to obtain a first attention feature; Performing a channel attention operation on the concatenated feature to obtain a second attention feature; Performing a channel attention operation and a spatial attention operation on the concatenated feature in sequence to obtain a third attention feature; Fusing the first attention feature, the second attention feature, and the third attention feature to obtain the final skin lesion image segmentation result.

4. The skin lesion image segmentation method according to claim 1, characterized in that, The feature encoder is an improved Resnet34 network, and the improved Resnet34 network is obtained by deleting the last average pooling layer and fully connected layer from the Resnet34 network.

5. The skin lesion image segmentation method according to claim 1, wherein, The image segmentation model further includes a second preprocessing module, and the second preprocessing module is used for: Performing an upsampling operation and a depthwise separable convolution operation on the first convolutional feature in sequence to obtain a second convolutional feature; Performing a max pooling operation and a depthwise separable convolution operation on the first convolutional feature in sequence to obtain a third convolutional feature; the first convolutional feature, the second convolutional feature, and the third convolutional feature constitute the first multi-scale feature; the first convolutional feature and the second convolutional feature are used to be input into the feature decoder.

6. The skin lesion image segmentation method according to claim 1, wherein The multi-field perception guidance module is used for: Using three first atrous convolutions with different atrous rates to extract features from the fifth encoded feature, respectively obtaining a first atrous convolution feature, a second atrous convolution feature, and a third atrous convolution feature; Fusing the first atrous convolution feature and the second atrous convolution feature to obtain a fused feature; Fusing the fused feature, the third atrous convolution feature, and the fifth encoded feature to obtain a multi-field fused feature.

7. A computer device, characterized in that, Comprising a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded and executed by a processor to execute the method according to any one of claims 1-6.

Images