Multi-scale feature fusion interaction skin anomaly segmentation method
Through the multi-scale feature fusion interaction method, the problem of large model size and high computational cost in medical image segmentation is solved, efficient and accurate segmentation of skin abnormal areas is achieved, and diagnostic accuracy and model applicability are improved.
Patent Information
- Application Number
- CN202411075284.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-22
AI Technical Summary
The existing medical image segmentation methods have problems such as large model size, high computational cost and poor edge details capture in abnormal skin segmentation, which affects the quality of segmentation and clinical diagnostic accuracy.
Using the multi-scale feature fusion interaction method, the multi-scale feature extraction module, feature fusion enhancement module and feature extraction fusion module are combined with the encoder and the decoder to achieve efficient and accurate segmentation of skin abnormal areas. The specific steps include preprocessing the image, performing multi-scale feature extraction and dimensionality reduction, extracting global features and edge detail features, performing fusion enhancement and interaction, and finally generating a segmentation mask through the feature extraction fusion module.
Effectively solve the problem of model parameters and calculation cost in abnormal skin segmentation, ensure the complete retention and accurate extraction of edge details, improve the segmentation quality and diagnostic accuracy, reduce the hardware resource requirements, and improve the model in actual medical scenarios. applicability and efficiency.
Smart Images

Figure CN119006493A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image segmentation, and in particular relates to a skin abnormality segmentation method that integrates and interacts multi-scale features. Background Art
[0002] With the significant progress of deep learning technology in the field of medical image analysis, especially the innovation of skin imaging, a series of advanced skin abnormality segmentation algorithms have emerged, which greatly improve the efficiency and accuracy of dermatologists' clinical decision-making. These algorithms distinguish the lesion area from normal skin tissue with high precision, which not only promotes the precise positioning of the lesion, but also provides strong support for the classification of abnormal types and subsequent diagnosis.
[0003] In the field of medical image segmentation in recent years, deep learning, especially convolutional neural networks (CNNs), has demonstrated outstanding capabilities and significantly improved the accuracy and efficiency of image segmentation. Since the pioneering work of fully convolutional neural networks (FCNs) published by Shelhamer et al. in 2017, the technology has leapt from semantic-level image classification to pixel-level image segmentation, achieving fine-grained image analysis. Subsequently, Ronneberger et al.'s U-Net architecture, as a derivative of FCN, effectively improved the accuracy of medical image segmentation by introducing an encoding-decoding framework and skip connections, especially in complex scenarios.
[0004] In order to further improve the segmentation performance, researchers continue to explore and optimize the model structure. Attention U-Net embeds the attention mechanism in the jump connection to enhance the model's learning of key features and reduce the impact of irrelevant information; U-Net++ improves the data fusion method in the jump connection, using dense convolutional layers to replace traditional cropping and splicing operations to reduce information loss and enrich feature representation; DoubleU-Net and CACDU-Net both use a dual U-Net structure. The former stacks two U-Nets to capture more details, while the latter introduces asymmetric convolutional spatial pyramid pooling between encoding and decoding to enhance multi-scale feature extraction capabilities; EIU-Net innovatively uses inverted residual blocks and efficient pyramid squeezing attention mechanisms to optimize feature extraction and cross-layer information transmission, especially in the task of skin abnormality segmentation.
[0005] Although recently proposed medical image segmentation methods have achieved remarkable results in improving segmentation accuracy, they are often accompanied by the expansion of model parameters and the increase of computing resource requirements, which to some extent limits their application and deployment in resource-constrained environments. In addition, existing methods generally lack specialized strategies to deal with the inherent challenges of skin images, such as interference from hairy areas, unclear lesion boundaries, and insufficient expression of abnormal features. The existence of these problems makes it difficult for the segmentation network to accurately capture and retain the key details of abnormal skin areas, which in turn affects the final segmentation quality and the accuracy of clinical diagnosis. Summary of the invention
[0006] The purpose of the embodiment of the present invention is to provide a skin abnormality segmentation method with multi-scale feature fusion interaction, which solves the problems of large model volume, high computational cost and poor edge detail capture in skin abnormality segmentation, and realizes efficient and accurate abnormal area segmentation;
[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is a skin abnormality segmentation method with multi-scale feature fusion interaction, the method comprising:
[0008] S1. Preprocess the skin abnormal area image into an image with a size of 256×256;
[0009] S2, extract multi-scale features from the preprocessed image and then perform dimensionality reduction;
[0010] S3, extracting the global features of the abnormal area from the results of S2 while enhancing the edge detail features of the abnormal skin area, fusing and enhancing the extracted global features and the enhanced edge detail features, interactively fusing the enhanced feature information with the unenhanced features to obtain two different features, then extracting, enhancing and interactively fusing the two features again, and performing dimensionality reduction on the image features after each enhancement;
[0011] S4, perform three feature extractions on the features after dimensionality reduction in S3; in the first two times, downsampling dimensionality reduction is used after each feature extraction, and after the last feature extraction, the three extracted features are fused;
[0012] Then, five feature extractions are performed. After each feature extraction, upsampling is used to increase the dimension, and feature information at different levels is concatenated and fused.
[0013] S5. Map the final spliced and fused feature map to the final segmentation category, complete the segmentation of the abnormal skin area, and output the segmentation result of the abnormal skin area.
[0014] Furthermore, in S2 to S5, feature extraction and fusion are achieved through an encoder and a decoder, multi-scale feature extraction is performed through one encoding stage, and then the multi-scale features are extracted through five encoding stages; then six decoding stages are used to fuse the low-level features in the encoder and reorganize the features into segmentation masks; among them, S2 is the first encoding stage; S3 is the second to third encoding stages; S4 is the fourth to sixth encoding stages and the first to fifth decoding stages; S5 is the sixth decoding stage.
[0015] Furthermore, S2 is specifically implemented by a multi-scale feature extraction module, and the multi-scale feature extraction module includes three convolution kernels of different sizes and three downsampling blocks, and the sizes of the three convolution kernels are 3×3, 5×5 and 7×7 respectively;
[0016] The convolution kernel is used to extract characteristic information of the image of the skin abnormality area;
[0017] The downsampling block is used to reduce the dimension of feature information.
[0018] Furthermore, the S3 is implemented by a feature fusion enhancement module, which includes interactive fusion of 5 RH-ZQ modules and 4 downsampling blocks;
[0019] The RH-ZQ module includes a CH-LW module, an ER-MS module and a FF-AM module;
[0020] The CH-LW module uses Channel split to split channels, 1×1 convolution to expand the number of channels, introduces the depth-separable hole convolution of the Dropout layer to extract global features, uses concat to merge splicing features, Channel shuffle to shuffle channels to mix information between different channels, and uses 1×1 convolution for residual connection;
[0021] The ER-MS module uses 1×1 convolution to reduce the input feature volume to a single channel map, the pow function enhances the gradient size of the edge area, references the calculation method of the Sobel operator to detect the edge to locate the detection edge, and uses the Leaky ReLU activation function for activation, and 1×1 convolution for residual connection;
[0022] The FF-AM module uses 1×1 convolution to expand the number of channels for the single-channel edge features extracted by the ER-MS module, and extracts compressed edge features by layer-by-layer convolution with a convolution kernel size of 3×3; uses 1×1 convolution to extract compressed features for the global features extracted by the CH-LW module; extracts compressed features by layer-by-layer convolution and 1×1 convolution respectively for the fusion enhancement features of edge detail features and global features; uses 1×1 convolution to expand the compressed feature extraction, and uses the Sigmoid activation function to activate the feature and multiply it with the input feature map.
[0023] Furthermore, the S4 is implemented by a feature extraction fusion module, which includes 11 CH-LW modules, 4 downsampling blocks, 5 upsampling blocks and a splicing operation.
[0024] Furthermore, in S3, the implementation process of the RH-ZQ module enhanced feature is as follows:
[0025] F QL =G1(X) F BY =G2(X)
[0026] F ZY =G3(F QL ,F BY ) F H =F ZY
[0027] Among them, X represents the input feature, G1 represents the lightweight module, and F QL represents the features extracted by the lightweight module, G2 represents the edge enhancement module, and F BY represents the edge features enhanced by the edge enhancement module, G3 represents the feature fusion attention module, and F ZY represents the features enhanced by the feature fusion attention module, F H Represents the features enhanced by the feature fusion enhancement module;
[0028] The implementation process of the ER-MS module to enhance the edge detail features of abnormal skin areas is as follows:
[0029] Y1=LR(B(Conv_y(P(C1(X)))))
[0030] Y2=LR(B(Conv_x(P(C1(X)))))
[0031]
[0032] F BY =Y3+C1(X)
[0033] Among them, X represents the input feature, C1 represents 1×1 convolution, P represents the pow function operation, Conv_y represents the horizontal convolution kernel, Conv_x represents the vertical convolution kernel, B represents the BN layer, LR represents the Leaky ReLU activation function, Y1 represents the horizontal gradient value, Y2 represents the vertical gradient value, Y3 represents the detected target edge gradient value, and F BY Represents the edge features enhanced by the edge enhancement module;
[0034] The implementation process of the FF-AM module to fuse and enhance the extracted global features and enhanced edge detail features is as follows:
[0035] Z1=S(C1(DW3(C1(F BY ))))×C1(F BY )
[0036] Z2=S(C1(DW3(C1(F BY )+F QL ))+C1(C1(C1(F BY )+F QL )))×(C1(F BY )+F QL )
[0037] Z3=S(C1(C1(F QL )))×F QL
[0038] F ZY =Z1+Z2+Z2
[0039] Among them, C1 represents 1×1 convolution, DW3 represents 3×3 layer-by-layer convolution, S represents sigmoid activation function, Z1 represents enhanced edge features, Z2 represents edge and global fusion enhanced features, Z3 represents enhanced global features, and F ZY Represents the features enhanced by the feature fusion attention module.
[0040] Furthermore, the implementation process of the CH-LW module to extract the global features of the abnormal area is as follows:
[0041] X1,X2=Channel split(X)
[0042] Q1=C 3(r=1) (C1(X1))+C 3(r=2) (C1(X2))
[0043] Q2=C 3(r=4) (C 3(r=2) (C1(X2)))+Q1
[0044] F QL=Channel shuffle(C(Q1,Q2))+C1(X)
[0045] Among them, X represents the input feature, Channel split represents channel segmentation, X1, X2 represent the features after the segmentation channel, C1 represents 1×1 convolution, C 3(r=i) represents the CH-LW module, i = 1, 2, 4, where r represents the hole rate in the convolution, i represents the size of the hole rate, C represents concatenation, Channel shuffle represents channel shuffle, Q1 and Q2 represent the features extracted from the two subgroup channels respectively, and F QL Represents the features extracted by the lightweight module.
[0046] The beneficial effects of the present invention are as follows: the present invention provides a systematic solution to the limitations of current medical image segmentation technology. The multi-scale feature extraction mechanism of the present invention can comprehensively capture the multi-level information of abnormal skin areas, ensuring that even when the lesion edges are blurred or there is interference from multiple hairs, the features can be fully retained and accurately extracted. Furthermore, the present invention introduces the RH-ZQ module, which aims to enhance the edge details and global features of abnormal skin areas, thereby effectively overcoming the segmentation challenges faced by traditional models in complex scenarios. At the same time, the innovative application of the CH-LW module achieves a significant reduction in the parameter scale and computational cost of the model while ensuring segmentation performance, which not only reduces the demand for hardware resources, but also greatly improves the applicability and efficiency of the model in actual medical scenarios.
[0047] In summary, the LDSD-Net model of the present invention not only ensures high-precision segmentation, but also effectively solves the key technical problems in skin abnormality segmentation through lightweight design and multi-scale feature fusion interaction, providing a more accurate, efficient and economical image analysis method for the diagnosis of dermatological diseases, showing broad application prospects and clinical value. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0049] Figure 1 It is a structural schematic diagram of a lightweight skin abnormality segmentation system with multi-scale feature fusion interaction according to an embodiment of the present invention.
[0050] Figure 2 It is a schematic diagram of the structure of the feature fusion enhancement module RH-ZQ according to an embodiment of the present invention;
[0051] Figure 3 2 is a schematic diagram of the structure of a lightweight module CH-LW according to an embodiment of the present invention;
[0052] Figure 4 is a schematic diagram of the structure of an edge enhancement module ER-MS according to an embodiment of the present invention;
[0053] Figure 5 It is a schematic diagram of the structure of the feature fusion attention module FF-AM of an embodiment of the present invention;
[0054] Figure 6 This is a comparison chart of the segmentation effects of the segmentation method of an embodiment of the present invention and the Unet, Attention U-Net, UNet++, DoubleU-Net, CACDU-Net and EIU-Net segmentation algorithms on abnormal skin areas. DETAILED DESCRIPTION
[0055] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0056] like Figure 1 ,This embodiment provides a skin abnormality segmentation method with multi-scale feature fusion interaction, which is implemented by a skin abnormality segmentation system with multi-scale feature fusion interaction, wherein the segmentation system includes four modules: a multi-scale feature extraction module, a feature fusion enhancement module, a feature extraction fusion module, and a prediction output module;
[0057] In some possible implementations, the multi-scale feature extraction module includes three convolution kernels of different sizes and three downsampling layers, and the sizes of the three convolution kernels are 3×3, 5×5, and 7×7, respectively. By using the above three convolution kernels, the feature information of the target is extracted respectively, and different features extracted by three different convolution kernels are obtained, aiming to extract richer abnormal area features, and then the extracted features are respectively downsampled to increase the receptive field of subsequent convolution layers, so as to learn more global information.
[0058] In some possible implementations, the feature fusion enhancement module includes 5 RH-ZQ modules and 4 downsampling blocks, and the RH-ZQ is composed of 1 lightweight module (CH-LW), 1 edge enhancement module (ER-MS) and 1 feature fusion attention module (FF-AM); in the shallow layer of the network, the edge information of the abnormal skin area is less lost, and the RH-ZQ module can realize feature extraction and enhance the edge features of the abnormal area. The features extracted by the multi-scale feature extraction module are further extracted and the edge features are enhanced using 3 RH-ZQ modules, and then two different features are obtained after interactive fusion and downsampling, and then 2 RH-ZQ modules are used to further extract and enhance the edge features of the fused features, and then downsampling is performed. The edge information of the abnormal skin area is lost more, and continuing to use RH-ZQ cannot achieve better results.
[0059] In some specific implementations, the CH-LW module uses Channel split to split the channel and divide the input channel into two subgroups, which greatly reduces the number of parameters and calculations compared to processing all channels at the same time. 1×1 convolution expands the number of channels, and introduces a deep separable hole convolution containing a Dropout layer to prevent overfitting to extract global features. Concat is used to merge and splice features to maintain the integrity of the original data and obtain more comprehensive features. Channel shuffle is used to shuffle channels to mix information between different channels and promote information exchange between different channels. This helps to extract richer and more useful feature representations, and uses 1×1 convolution for residual connection to more flexibly reuse features of previous layers, which helps the model better capture the complex features of the data and improve the generalization ability of the model.
[0060] The ER-MS module uses 1×1 convolution to reduce the input feature volume to a single-channel mapping, and then strengthens the gradient amplitude of the edge area through the pow function. The Sobel operator is used to accurately locate the edge information. Specifically, the Sobel operator performs edge detection by calculating the weighted difference between each pixel and its adjacent pixels (upper, lower, left, and right). This implementation method uses the Sobel operator, which performs better when processing images with large grayscale gradients and high noise levels. Its simple and intuitive mathematical operations reduce the required computing resources. The ER-MS module further uses Leaky ReLU as an activation function for activation. This design allows negative gradients to pass with a smaller slope, so that the network can learn richer feature representations and effectively reduce the loss of edge features. The use of Leaky ReLU enhances the nonlinear expression ability of the network, which is crucial to improving the overall performance. In addition, the ER-MS module also uses 1×1 convolution to implement residual connections, which not only allows the features of the previous layer to be reused more flexibly, but also helps the model to more effectively capture the complex structural features in the data, thereby improving the generalization ability of the model.
[0061] The FF-AM module uses different weighted enhancement methods for different extracted features. For the single-channel edge features extracted by the ER-MS module, 1×1 convolution is used to expand the number of channels. Since the edge features are irregular in shape and have high spatial variability, layer-by-layer convolution DWConv with a convolution kernel size of 3×3 is used to extract compressed edge features. While taking into account the size of the parameters, it can capture the edge features of the abnormal area more carefully and better focus on important edge features. For the global features extracted by the CH-LW module, 1×1 convolution is used to extract compressed features, which can reduce the consumption of computing resources and achieve good compression of global features. For the fusion features of edge features and global features, layer-by-layer convolution and 1×1 convolution are used to extract compressed features respectively. In order to effectively help the network extract important information in the input feature map and the contextual information relationship within the input feature map, 1×1 convolution is used to expand the compressed feature extraction. The Sigmoid activation function is used to activate the feature and multiply it with the input feature map.
[0062] The feature extraction and fusion module consists of 11 lightweight modules CH-LW, 4 downsampling blocks, 5 upsampling blocks and splicing operations;
[0063] The prediction output module uses a 1×1 convolutional layer to map the fused feature map to the final segmentation category.
[0064] In some specific embodiments, the skin abnormality segmentation system with multi-scale feature fusion interaction includes an encoder and a decoder. The encoder is responsible for extracting features from the input image, while the decoder is responsible for reorganizing these features into a segmentation mask. More specifically, the system is composed of 6 encoding stages (encode stage 1 to 6) and 6 decoding stages (decode stage 1 to 6). The traditional segmentation network adopts a 5-encoding and decoding stage structure. The system adds an additional multi-scale feature extraction stage during feature extraction to extract multi-scale features in order to obtain richer abnormal area features. In order to better process the extracted features, the extracted multi-scale features are further extracted using a 5-stage encoding structure. The encoding has 6 stages, and correspondingly there are 6 decoding stages to fuse the low-level features in the encoder and reorganize the features into a segmentation mask. During the decoding process, the upsampled feature map is subjected to a CH-LW operation to extract higher-level features. These features contain both the spatial information restored by upsampling and the semantic information extracted by the encoder stage. The 6 decoding stages are each composed of a CH-LW module.
[0065] The multi-scale feature extraction module consists of the encoder stage1;
[0066] The feature fusion enhancement module consists of encoding stage 2 to 3;
[0067] The feature extraction and fusion module consists of encoding stage 4 to 6 and decoding stage 1 to 5;
[0068] The prediction output module consists of the decode stage6.
[0069] like Figure 2 In some possible implementations, the RH-ZQ module uses the CH-LW module to extract global features of the abnormal area while using the ER-MS module to enhance the edge detail features of the abnormal skin area, and then uses the FF-AM module to fuse and enhance the global features extracted by the CH-LW module and the edge detail features enhanced by the ER-MS module to obtain more expressive features.
[0070] like Figure 3In some possible implementations, the CH-LW module divides the input channels into two subgroups through channel splitting. On each subgroup channel, 1×1 convolution is used to expand the number of channels and enrich the input feature data. Then, LW-Block of the depth-separable dilated convolution and Dropout layer is used to extract features. On the two subgroups, one subgroup uses the depth-separable dilated convolution with dilation rate r=1 to extract features, and the other subgroup uses two depth-separable dilated convolutions with dilation rates r=2 and r=4 to extract features. Then, the features of different subgroup levels are added, and the obtained features are concat-merged. Channel shuffle is performed on the concatenated feature channels, and finally residual connection is realized through 1×1 convolution.
[0071] like Figure 4 In some possible implementations, the ER-MS module uses 1×1 convolution to reduce the feature volume to a single-channel map, then uses the pow function to enhance the edge gradient size of the abnormal area, and then uses the calculation method of the Sobel operator to detect the edge, and calculates the horizontal and vertical gradient values of the detection target through the horizontal convolution kernel conv_x and the vertical convolution kernel conv_y respectively; then uses the batch normalization BN operation and the Leaky ReLU activation function to activate; the activated horizontal and vertical gradient values are squared and then squared to obtain the edge gradient value. Finally, a residual connection is performed with the single-channel map to obtain a feature map that highlights the edge information of the detection target.
[0072] like Figure 5 In some possible implementations, the FF-AM module has two input features, one is the single-channel edge feature extracted by the ER-MS module, and the number of channels is expanded by 1×1 convolution, and the other is the global feature extracted by the CH-LW module. The edge feature is subjected to layer-by-layer convolution (DWConv) with a convolution kernel size of 3×3 to extract compressed edge features, and then the channel is expanded by 1×1 convolution, and after the Sigmoid activation function, the attention map is multiplied with the input edge feature map; the global feature is subjected to 1×1 convolution to extract compressed features, and then the channel is expanded by 1×1 convolution, and after the Sigmoid activation function, the attention map is multiplied with the input global feature map; for the fusion features of the edge feature and the global feature, the compressed features are extracted by layer-by-layer convolution and 1×1 convolution respectively; the channel is expanded by 1×1 convolution, and after the Sigmoid activation function, the attention map is multiplied with the input fusion feature map; finally, the enhanced features of the three parts are fused.
[0073] Based on the above-mentioned skin abnormality segmentation system with multi-scale feature fusion interaction, this embodiment further provides a skin abnormality segmentation method with multi-scale feature fusion interaction, and the specific process is as follows:
[0074] S1. Preprocess the skin abnormal area image to convert the image size to 256×256, input the processed abnormal area image into the multi-scale feature extraction, complete the abnormal area feature extraction, and then input the extracted multi-scale features into the feature fusion enhancement module for enhancement, and then input the enhanced features into the feature extraction fusion module, and finally output the feature prediction;
[0075] The multi-scale feature extraction extracts the characteristic information of the skin abnormality area image through three convolution kernels of different sizes: 3×3, 5×5 and 7×7, and then performs dimensionality reduction by downsampling;
[0076] S2, the feature fusion enhancement module uses three RH-ZQ modules to enhance the extracted different features, that is, CH-LW extracts the global features of the abnormal area while ER-MS enhances the edge detail features of the abnormal skin area, FF-AM fuses and enhances the extracted global features and the enhanced edge detail features, and the enhanced feature information is interactively fused with the unenhanced features to obtain two different features, and then two RH-ZQ modules are used again to further enhance the feature extraction and interactively fuse, and the image features after each enhancement are reduced in dimension by downsampling;
[0077] S3, the feature extraction and fusion module uses the CH-LW module three times to extract rich and effective features for the two features extracted by the feature fusion enhancement module. In the first two times, downsampling and dimensionality reduction are used after each feature extraction. After the last feature extraction, the features are fused and the CH-LW module is used 5 times to further extract features. Upsampling and dimensionality increase are used after each feature extraction, and feature information of different levels are concatenated at the same time;
[0078] S4. The predicted output uses 1×1 convolution to map the feature map output by the feature extraction and fusion module to the final segmentation category, completes the segmentation of the abnormal skin area, and outputs the segmentation result of the abnormal skin area.
[0079] The realization process of the RH-ZQ enhancement feature of S2 is as follows:
[0080] F QL =G1(X) F BY =G2(X) (1)
[0081] F ZY =G3(F QL ,F BY ) F H =F ZY(2)
[0082] Among them, X represents the input feature, G1 represents the lightweight module, and F QL represents the features extracted by the lightweight module, G2 represents the edge enhancement module, and F BY represents the edge features enhanced by the edge enhancement module, G3 represents the feature fusion attention module, and F ZY represents the features enhanced by the feature fusion attention module, F H Represents the features enhanced by the feature fusion enhancement module;
[0083] The implementation process of CH-LW for S2 and S3 to extract global features of abnormal regions is as follows:
[0084] X1,X2=Channel split(X) (3)
[0085] Q1=C 3(r=1) (C1(X1))+C 3(r=2) (C1(X2)) (4)
[0086] Q2=C 3(r=4) (C 3(r=2) (C1(X2)))+Q1 (5)
[0087] F QL =Channel shuffle(C(Q1,Q2))+C1(X) (6)
[0088] Among them, X represents the input feature, Channel split represents channel segmentation, X1, X2 represent the features after the segmentation channel, C1 represents 1×1 convolution, C 3(r=i) (i=1,2,4) represents the CH-LW module, C represents concatenation, Channelshuffle represents channel shuffle, Q1 and Q2 represent the features extracted from the two subgroup channels respectively, and F QL Represents the features extracted by the lightweight module;
[0089] The implementation process of S2's ER-MS to enhance the edge detail features of abnormal skin areas is as follows:
[0090] Y1=LR(B(Conv_y(P(C1(X))))) (7)
[0091] Y2=LR(B(Conv_x(P(C1(X)))))) (8)
[0092]
[0093] F BY =Y3+C1(X) (10)
[0094] Among them, X represents the input feature, C1 represents 1×1 convolution, P represents the pow function operation, Conv_y represents the horizontal convolution kernel, Conv_x represents the vertical convolution kernel, B represents the BN layer, LR represents the Leaky ReLU activation function, Y1 represents the horizontal gradient value, Y2 represents the vertical gradient value, Y3 represents the detected target edge gradient value, and F BY Represents the edge features enhanced by the edge enhancement module;
[0095] The implementation process of S2's FF-AM to fuse and enhance the extracted global features and enhanced edge detail features is as follows:
[0096] Z1=S(C1(DW3(C1(F BY ))))×C1(F BY ) (11)
[0097] Z2=S(C1(DW3(C1(F BY )+F QL ))+C1(C1(C1(F BY )+F QL )))×(C1(F BY )+F QL ) (12)
[0098] Z3=S(C1(C1(F QL )))×F QL (13)
[0099] F ZY =Z1+Z2+Z2 (14)
[0100] Among them, F BY represents the edge features enhanced by the edge enhancement module, F QL represents the features extracted by the lightweight module, C1 represents 1×1 convolution, DW3 represents 3×3 layer-by-layer convolution, S represents the sigmoid activation function, Z1 represents the enhanced edge features, Z2 represents the features enhanced by edge and global fusion, Z3 represents the enhanced global features, and F ZY Represents the features enhanced by the feature fusion attention module.
[0101] In order to verify the segmentation effect of this embodiment, a comparative experiment was carried out. This embodiment experiment uses the public dataset ISIC-2018 to train and test the model. The dataset contains 2594 skin microscope images with segmentation mask labels. In the experiment, the dataset is randomly divided into training set, validation set and test set in a ratio of 7:1:2. Compared with Unet, AttentionU-Net, UNet++, DoubleU-Net, CACDU-Net, EIU-Net, 6 image segmentation models, the results are shown in Table 1:
[0102] Table 1 Comparative test of different segmentation models
[0103] Methods Params(M) GFLOPs Acc IoU DSE Spe Unet 7.76 13.75 0.950 0.756 0.853 0.976 Attention U-Net 8.73 16.89 0.950 0.760 0.856 0.970 UNet++ 9.16 34.90 0.951 0.756 0.853 0.974 DoubleU-Net 29.28 53.96 0.954 0.774 0.866 0.978 CACDU-Net 37.55 22.98 0.957 0.793 0.880 0.977 EIU-Net 13.38 18.91 0.956 0.783 0.873 0.978 The present invention 0.36 0.86 0.960 0.801 0.885 0.980
[0104] As shown in Table 1, compared with other models, the number of parameters and floating-point operations of LDSD-Net and Unet decreased by 95.4% and 93.7% respectively, and compared with CACDU-Net, the number of parameters and floating-point operations decreased by 99% and 96.3% respectively, while the accuracy increased by 1%, 1%, 0.9%, 0.6%, 0.3% and 0.4% respectively compared with Unet, Attention U-Net, UNet++, DoubleU-Net, CACDU-Net and EIU-Net. From the above data, it can be seen that LDSD-Net effectively solves the problems of large number of parameters, complex calculation and difficulty in accurately segmenting skin abnormal areas in existing segmentation algorithms, and has achieved very good results in the segmentation of skin abnormal areas.
[0105] In order to more comprehensively demonstrate the performance of the LDSD-Net network model in skin abnormality area segmentation, this paper Figure 6 A detailed comparative analysis was conducted. Figure 6 The results of different segmentation algorithms for abnormal skin areas are shown. From the segmentation results in the figure, it can be seen that the segmentation effects of the classic medical segmentation networks UNet, Attention UNet and UNet++ are not ideal. For abnormal skin areas with blurred edges and hair interference, there is over-segmentation, and some background areas are identified as abnormal areas. For unclear abnormal areas, there is insufficient segmentation, and some abnormal skin areas are identified as background areas. The segmentation effects of DoubleU-Net and CACDU-Net networks have improved, but there is still over-segmentation for unclear and hair-interfered abnormal skin areas. The segmentation effect of the EIU-Net segmentation network for unclear abnormal skin areas needs to be improved. The LDSD-Net segmentation algorithm has a better segmentation effect on abnormal skin areas, especially for abnormal areas with blurred edges, unclear edges and hair interference, and the segmentation effect is significantly better than other algorithms.
[0106] It can be seen that the LDSD-Net network proposed in the present invention has a better segmentation effect on complex skin abnormality areas, and can meet the application of mobile devices in terms of model lightweight.
[0107] If the multi-scale feature fusion interactive lightweight skin abnormality segmentation model and its segmentation method described in the embodiment of the present invention are implemented in the form of a software function module and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the image segmentation method described in the embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, ROM, RAM, disk or CD-ROM and other media that can store program codes.
[0108] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0109] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.
Claims
1. A skin abnormality segmentation method with multi-scale feature fusion interaction, characterized in that: The method comprises: S1. Preprocess the skin abnormal area image into an image with a size of 256×256; S2, extract multi-scale features from the preprocessed image and then perform dimensionality reduction; S3, extracting the global features of the abnormal area from the results of S2 while enhancing the edge detail features of the abnormal skin area, fusing and enhancing the extracted global features and the enhanced edge detail features, interactively fusing the enhanced feature information with the unenhanced features to obtain two different features, then extracting, enhancing and interactively fusing the two features again, and performing dimensionality reduction on the image features after each enhancement; S4, perform three feature extractions on the features after dimensionality reduction in S3; in the first two times, downsampling dimensionality reduction is used after each feature extraction, and after the last feature extraction, the three extracted features are fused; Then, five feature extractions are performed. After each feature extraction, upsampling is used to increase the dimension, and feature information at different levels is concatenated and fused. S5. Map the final spliced and fused feature map to the final segmentation category, complete the segmentation of the abnormal skin area, and output the segmentation result of the abnormal skin area.
2. The skin abnormality segmentation method based on multi-scale feature fusion interaction according to claim 1, characterized in that: In S2 to S5, the extraction and fusion of features are realized by an encoder and a decoder, multi-scale feature extraction is performed through one encoding stage, and then the multi-scale features are extracted through five encoding stages; Six decoding stages are then used to fuse the low-level features from the encoder and reorganize the features into segmentation masks; Among them, S2 is the first encoding stage; S3 is the second to third encoding stages; S4 is the fourth to sixth encoding stages and the first to fifth decoding stages; S5 is the sixth decoding stage.
3. The skin abnormality segmentation method based on multi-scale feature fusion interaction according to claim 1, characterized in that: The S2 is specifically implemented by a multi-scale feature extraction module, which includes three convolution kernels of different sizes and three downsampling blocks, and the sizes of the three convolution kernels are 3×3, 5×5 and 7×7 respectively; The convolution kernel is used to extract characteristic information of the image of the skin abnormality area; The downsampling block is used to reduce the dimension of feature information.
4. The skin abnormality segmentation method based on multi-scale feature fusion interaction according to claim 1, characterized in that: The S3 is implemented by a feature fusion enhancement module, which includes 5 RH-ZQ modules interactively fused and 4 downsampling blocks; The RH-ZQ module includes a CH-LW module, an ER-MS module and a FF-AM module; The CH-LW module uses Channel split to split channels, 1×1 convolution to expand the number of channels, introduces the depth-separable hole convolution of the Dropout layer to extract global features, uses concat to merge splicing features, Channel shuffle to shuffle channels to mix information between different channels, and uses 1×1 convolution for residual connection; The ER-MS module uses 1×1 convolution to reduce the input feature volume to a single channel map, the pow function enhances the gradient size of the edge area, references the calculation method of the Sobel operator to detect the edge, and uses the Leak y ReLU activation function is used for activation, and 1×1 convolution is used for residual connection; The FF-AM module uses 1×1 convolution to expand the number of channels for the single-channel edge features extracted by the ER-MS module, and uses layer-by-layer convolution with a convolution kernel size of 3×3 to extract compressed edge features; for the global features extracted by the CH-LW module, 1×1 convolution is used to extract compressed features; For the fusion enhancement of edge detail features and global features, layer-by-layer convolution and 1×1 convolution are used to extract compression features respectively; Use 1×1 convolution to expand the compressed feature extraction, and use the Sigmoid activation function to activate the feature and multiply it with the input feature map.
5. The skin abnormality segmentation method based on multi-scale feature fusion interaction according to claim 1, characterized in that: The S4 is implemented by a feature extraction fusion module, which includes 11 CH-LW modules, 4 downsampling blocks, 5 upsampling blocks and a splicing operation.
6. The skin abnormality segmentation method based on multi-scale feature fusion interaction according to claim 4, characterized in that: In S3, the implementation process of the RH-ZQ module enhanced features is: F QL =G1(X) F BY =G2(X) F ZY =G3(F QL ,F BY ) F H =F ZY Among them, X represents the input feature, G1 represents the lightweight module, and F QL represents the features extracted by the lightweight module, G2 represents the edge enhancement module, and F BY represents the edge features enhanced by the edge enhancement module, G3 represents the feature fusion attention module, and F ZY represents the features enhanced by the feature fusion attention module, F H Represents the features enhanced by the feature fusion enhancement module; The implementation process of the ER-MS module to enhance the edge detail features of abnormal skin areas is as follows: Y1=LR(B(Conv_y(P(C1(X))))) Y2=LR(B(Conv_x(P(C1(X))))) F BY =Y3+C1(X) Among them, X represents the input feature, C1 represents 1×1 convolution, P represents the pow function operation, Conv_y represents the horizontal convolution kernel, Conv_x represents the vertical convolution kernel, B represents the BN layer, LR represents the Leaky ReLU activation function, Y1 represents the horizontal gradient value, Y2 represents the vertical gradient value, Y3 represents the detected target edge gradient value, and F BY Represents the edge features enhanced by the edge enhancement module; The implementation process of the FF-AM module to fuse and enhance the extracted global features and enhanced edge detail features is as follows: Z1=S(C1(DW3(C1(F BY ))))×C1(F BY ) Z2=S(C1(DW3(C1(F BY )+F QL ))+C1(C1(C1(F BY )+F QL )))×(C1(F BY )+F QL ) Z3=S(C1(C1(F QL )))×F QL F ZY =Z1+Z2+Z2 Among them, C1 represents 1×1 convolution, DW3 represents 3×3 layer-by-layer convolution, S represents sigmoid activation function, Z1 represents enhanced edge features, Z2 represents edge and global fusion enhanced features, Z3 represents enhanced global features, and F ZY Represents the features enhanced by the feature fusion attention module.
7. The skin abnormality segmentation method based on multi-scale feature fusion interaction according to claim 4 is characterized in that: The implementation process of CH-LW module to extract global features of abnormal areas is as follows: X1, X2 = Channel split (X) Q1=C 3(r=1) (C1(X1))+C 3(r=2) (C1(X2)) Q2=C 3(r=4) (C 3(r=2) (C1(X2)))+Q1 F QL =Channel shuffle(C(Q1,Q2))+C1(X) Among them, X represents the input feature, Channel split represents channel segmentation, X1 and X2 represent the features after the segmentation channel, C1 represents 1×1 convolution, and C 3(r=i) represents the CH-LW module, i = 1, 2, 4, where r represents the hole rate in the convolution, i represents the size of the hole rate, C represents concatenation, Channel shuffle represents channel shuffle, Q1 and Q2 represent the features extracted from the two subgroup channels respectively, and F QL Represents the features extracted by the lightweight module.
Citation Information
Cited By
Low-light image enhancement method and system based on edge extraction and feature fusion
CN120807330A
