Skin focus segmentation method and device based on ETMA-Net
Through the edge threshold guided parallel network and multi-level feature aggregation module of ETMA-Net, the problems of low feature extraction rate and poor migration performance in skin lesion segmentation are solved, and efficient skin lesion segmentation and good cross-domain adaptability are achieved.
Patent Information
- Application Number
- CN202411123125.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-12-03
AI Technical Summary
The existing skin lesion segmentation method relies on convolutional calculations during feature extraction, resulting in low local and global feature extraction rates, insufficient utilization of feature information, high model complexity and poor migration performance, making it difficult to maintain performance in different pathological image data sets.
A skin lesion segmentation method based on ETMA-Net is proposed, through a multi-level attention parallel network guided by edge threshold, a parallel encoder guided by edge threshold, a grouped multi-scale Hadamard attention module and a jump connection module for multi-level feature aggregation , perform image feature extraction and feature fusion, reduce parameter amount and calculation amount, and improve model segmentation performance.
While maintaining low parameter volume and low computation volume, the model performance of skin lesions segmentation is improved, significantly improving the smoothness and cross-domain generalization ability of segmentation results, and is better than other classical and lightweight models.
Smart Images

Figure CN119067934A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of medical image processing, and in particular to a skin lesion segmentation method and device based on ETMA-Net. Background Art
[0002] Accurately segmenting skin lesions in dermoscopic images is a key factor in the early diagnosis of melanoma. However, this remains a challenging task due to various interfering factors, such as the irregular shape and blurred boundaries of skin lesions. While dermatologists can diagnose malignant melanoma using images generated by a dermatoscope, this visual inspection is time-consuming and tedious, and can easily lead to visual fatigue and misdiagnosis. Therefore, it is of great significance to automatically and accurately segment skin lesions from dermoscopic images using a computer-aided automatic diagnosis system.
[0003] Segmenting skin lesions in dermoscopic images is a complex task. Many skin factors of the patient will affect whether the labeled pixels can be correctly marked in the lesion area. For example, in the skin lesion area, there may be a part with a pixel value very close to that of normal skin, with no obvious boundary between it and the surrounding skin; or, inside the lesion area, there will be parts with large pixel differences and uneven color distribution; in addition, there will be some human tissues such as hair on the patient's lesion skin, which makes pathological segmentation difficult. In order to solve these difficulties, a large number of researchers have made many contributions. However, although some segmentation methods have been proposed to deal with these problems, the effect of skin lesion segmentation is still not very ideal.
[0004] Traditionally, some researchers have segmented lesion regions by identifying pixel features in pathological images and performing preprocessing on them. While this approach has been effective, it has become increasingly challenging for large and complex image data. With the development of neural network technology, lesion segmentation based on convolutional neural networks has gained widespread application.
[0005] In recent years, in the field of skin lesion segmentation, a large number of scholars have conducted research on the segmentation of lesion areas such as melanoma, and have achieved certain results. However, the following problems still exist: (1) When extracting features, they rely heavily on the effects brought by convolution calculations, and the extraction rate of local features and global features in the original image is not high. (2) When fusing the extracted features, the feature information is not fully utilized, and there is a situation where feature information is lost. (3) The large-scale addition of calculations of modules such as convolution increases the complexity of the model, and the lightweight model cannot guarantee the maintenance of model accuracy while maintaining low parameters and computational complexity. (4) The existing models that are applicable to some pathological segmentation in the medical field have poor migration performance and cannot maintain performance in pathological image datasets of other parts. Summary of the Invention
[0006] In response to the above problems, the purpose of the present invention is to provide a skin lesion segmentation method and device based on ETMA-Net. By constructing a multi-level attention parallel network (ETMA-Net) guided by edge thresholds, the feature extraction method of the image is improved, and the extracted features are cross-utilized at multiple levels to fully retain the original information. It can improve the segmentation performance of the model while maintaining low parameter and computational complexity, and has excellent migration performance.
[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:
[0008] In one aspect, a skin lesion segmentation method based on ETMA-Net is provided, the method comprising the following steps:
[0009] S1. Obtain a dermatoscopic image to be segmented;
[0010] S2. Construct an ETMA-Net model, which includes: an edge threshold-guided parallel encoder, a grouped multi-scale Hadamard attention module, a skip connection module for multi-level feature aggregation, and a deep supervision decoder;
[0011] S3. Training the ETMA-Net model;
[0012] S4. Input the dermoscopic image into the trained ETMA-Net model to obtain the skin lesion segmentation result.
[0013] Optionally, the input of the edge threshold guided parallel encoder is obtained by processing the original dermoscopic image and the original dermoscopic image by the Canny edge detection algorithm and the OSTU algorithm respectively;
[0014] The OSTU algorithm includes: converting the original dermoscopic image into a grayscale image and calculating the pixel frequency histogram of the grayscale image; segmenting the pixels into different regions using a selected threshold and calculating the inter-class variance of the different regions; determining the optimal threshold by linear search, with the criterion being the maximum inter-class variance;
[0015] The Canny edge detection algorithm includes: performing Gaussian filtering and denoising on the original dermoscopic image to obtain a smoothed image; using the Sobel operator to calculate the image gradient on the smoothed image, and calculating the size and direction of the edge gradient through the image gradient; performing non-maximum suppression on the image, as well as double threshold and connectivity tests, to obtain an image processed by the Canny edge detection algorithm.
[0016] Optionally, the edge threshold guided parallel encoder includes an ASPP module and an RFB-s module;
[0017] The ASPP module includes a unit convolution, three dilated convolutions, and a pooling module. The feature map output by the pooling module is unified to the size of the input feature map by upsampling, and all feature maps are merged and connected before output. The expansion rates of the three dilated convolutions are {6, 12, 18}, respectively. The pooling module is an adaptive average pooling layer, and each convolution in the ASPP module is connected to a batch normalization layer and a ReLU activation function.
[0018] The RFB-s module includes a unit convolution, a normal convolution, and four dilated convolutions. The expansion rates of the four dilated convolutions are {1, 3, 3, 5}, respectively. An addition operation is performed before the output, and each convolution in the RFB-s module is connected to a batch normalization layer and a ReLU activation function.
[0019] Optionally, in the edge threshold guided parallel encoder, the input image is in RGB format , after edge threshold processing, becomes Then, it passes through the convolution module, ASPP module and RFB-s module in the parallel encoder for feature extraction:
[0020]
[0021] Among them, Y1(x), Y2(x), and Y3(x) represent the overall calculation functions of the convolution module layer, ASPP module layer, and RFB-s module layer respectively; C(x), A(x), and R(x) represent the calculation functions of the convolution module, ASPP module, and RFB-s module respectively; , δ represent the maximum pooling and Gelu activation function respectively.
[0022] Optionally, the grouped multi-scale Hadamard attention module includes a grouped multi-scale attention convolution submodule and a grouped Hadamard attention convolution submodule;
[0023] In the grouped multi-scale attention convolution submodule, the input tensor is first evenly divided into four groups on the channel through four grouped convolutions of different sizes, with the number of groups being {1, 2, 4, 8} and the convolution kernel sizes being {3, 5, 7, 9}. Then, four channel attention weights are generated through the SE module, and the four groups of tensors and the four weights are merged into one tensor and one weight. The weights are then normalized by the SoftMax function and multiplied with the tensor as the output of the grouped multi-scale attention convolution submodule.
[0024] In the grouped Hadamard attention convolution submodule, the input tensor is first divided into 4 groups, and then 3 tensors are established. The dimensions are expanded to the corresponding input dimensions through convolution operations, and then Hadamard products are performed; the 3 tensors established through training are used to obtain weight matrices corresponding to different dimensions, and the weights are processed separately using depthwise separable convolution to keep the size corresponding to the grouped tensor; finally, the grouped tensors multiplied by the weights are merged as the output of the grouped Hadamard attention convolution submodule.
[0025] Optionally, in the grouped multi-scale Hadamard attention module, the input is respectively passed into the grouped multi-scale attention convolution submodule and the grouped Hadamard attention convolution submodule to obtain outputs, the outputs of the two submodules are merged and then the number of channels is adjusted to the size of the corresponding parallel encoder output through 3×3 convolution; the process of grouped convolution of the grouped multi-scale attention convolution submodule is as follows:
[0026]
[0027] Among them, i∈[0,1,2,3]; k is the convolution kernel size corresponding to the group convolution; g is the number of groups; M i is the output of grouped convolution; the outputs of multiple groups of convolutions are merged as follows:
[0028]
[0029] The attention weight is generated by the SE module and calculated with M in the form of a product to obtain the output, as shown in the following formula:
[0030]
[0031] Among them, W represents the weight generated by the SE module; Att represents the weight vector obtained by the Softmax calculation of the weight generated by the SE module; Out GMA Represents the output of the grouped multi-scale attention convolution submodule; next, it is merged with the grouped Hadamard attention convolution submodule and fused through convolution calculation to obtain the final output, as shown in the following formula:
[0032]
[0033] Among them, Out GHA Represents the output of the grouped Hadamard attention convolution submodule; Out represents the final output of the grouped multi-scale Hadamard attention module.
[0034] Optionally, the jump connection module of the multi-level feature aggregation connects the upper and lower level output features of the backbone encoder, and aggregates the output features of the decoder and the output features of the parallel encoder guided by the edge threshold. Finally, all the information is grouped and fused, and then merged as the output of the jump connection module of the multi-level feature aggregation after convolution calculation.
[0035] Optionally, the loss functions used in the ETMA-Net model training are Dice loss function and BCE loss function.
[0036] On the other hand, a skin lesion segmentation device based on ETMA-Net is provided for implementing any of the above methods, the device comprising:
[0037] An image acquisition module, used for acquiring a dermatoscopic image to be segmented;
[0038] A model building module for building an ETMA-Net model, which includes an edge threshold-guided parallel encoder, a grouped multi-scale Hadamard attention module, a skip connection module for multi-level feature aggregation, and a deep supervision decoder;
[0039] A model training module, used to train the ETMA-Net model;
[0040] The segmentation module is used to input the dermoscopic image into the trained ETMA-Net model to obtain the skin lesion segmentation result.
[0041] In another aspect, an electronic device is provided, comprising:
[0042] processor;
[0043] A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are loaded and executed by the processor, the steps of the skin lesion segmentation method based on ETMA-Net are implemented.
[0044] On the other hand, a computer-readable storage medium is provided, wherein program code is stored in the computer-readable storage medium, and the program code can be called by a processor to execute the steps of the above-mentioned skin lesion segmentation method based on ETMA-Net.
[0045] The beneficial effects brought about by the technical solution provided by the present invention include at least:
[0046] The present invention proposes a skin lesion segmentation method and device based on the lightweight segmentation model ETMA-Net. In the ETMA-Net model, a new edge threshold-guided parallel network framework is proposed. The framework extracts image features in a low-computational manner through the OSTU and Canny algorithms. The extracted local and global features are integrated and distilled through the ASPP module, the RFB-s module, and the grouped multi-scale Hadamard attention module proposed in the present invention, and coarse-grained features are extracted and the edge fine-grained features are deeply understood. In order to improve the model's utilization of high- and low-level features and improve the performance of extracting spatial information, a multi-level feature aggregation form of skip connection module is proposed in the ETMA-Net model. This module integrates the high- and low-level features of the parallel network with the feature map output by the deep supervision decoder, fully utilizes the feature information to improve the segmentation performance, and makes the segmentation results smoother. The skin lesion segmentation method based on the ETMA-Net model proposed in the present invention was tested on a public skin lesion dataset. Compared with other classic and recent lightweight models, the ETMA-Net model demonstrated superior performance. Through cross-domain data testing and polyp medical image segmentation dataset testing, it is demonstrated that the ETMA-Net model has good cross-domain generalization performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0048] Figure 1 This is a flowchart of a skin lesion segmentation method based on ETMA-Net provided by an embodiment of the present invention;
[0049] Figure 2 ETMA-Net model framework diagram provided by an embodiment of the present invention;
[0050] Figure 3 is a schematic diagram of a parallel encoder provided by an embodiment of the present invention;
[0051] Figure 4 is a schematic diagram of a grouped multi-scale Hadamard attention module provided by an embodiment of the present invention;
[0052] Figure 5 is a schematic diagram of a grouped multi-scale attention convolution submodule provided by an embodiment of the present invention;
[0053] Figure 6 is a schematic diagram of a grouped Hadamard attention convolution submodule provided in an embodiment of the present invention;
[0054] Figure 7 is a schematic diagram of a skip connection module for multi-level feature aggregation provided by an embodiment of the present invention;
[0055] Figure 8a and Figure 8b FIG2 is a schematic diagram comparing the complexity and performance of different algorithms provided by embodiments of the present invention on the PH2 dataset;
[0056] Figure 9 Schematic diagram of the structure of a skin lesion segmentation device based on ETMA-Net provided by an embodiment of the present invention;
[0057] Figure 10 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0058] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the described embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0059] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" in the present invention should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner.
[0060] The embodiment of the present invention provides a skin lesion segmentation method based on ETMA-Net, which can be implemented by an electronic device, which can be a terminal or a server. Figure 1As shown, the processing flow of the method may include the following steps:
[0061] S1. Obtain a dermatoscopic image to be segmented.
[0062] S2. Construct a multi-level attention parallel network model guided by edge threshold, namely ETMA-Net model, which includes: edge threshold guided parallel encoder, grouped multi-scale Hadamard attention module, multi-level feature aggregation jump connection module, and deep supervision decoder.
[0063] Figure 2 Figure 1 is a schematic diagram of the ETMA-Net model framework. This framework includes a grouped multi-scale Hadamard attention module primarily composed of CNNs and attention layers, two edge threshold-guided parallel encoders consisting of convolutional layers, ASPP modules, and RFB-s modules, a skip connection module for multi-level feature aggregation, and a decoder with a deep supervised loss calculation method.
[0064] The ETMA-Net model takes as input three images: the original image, the thresholded image, and the edge image. A parallel encoder guided by edge thresholds extracts features from the edge and thresholded images. The extracted features are then incorporated into the skip connection module for feature verification and enhancement in subsequent modules. In the parallel encoder guided by edge thresholds, the different-scale dilated convolutions in the ASPP module give the encoder spatial awareness, enabling it to better capture the spatial context of the image. The different-scale receptive fields in the RFB-s module enhance the encoder's feature extraction capabilities, making the network more flexible in acquiring image information. The RFB-s module also reduces the number of parameters and computational overhead. In addition to the parallel encoder, the original image passes through the grouped multi-scale Hadamard attention module at the network's backbone. This module features low parameters and strong feature extraction capabilities. It extracts as much feature information from the original image as possible through a multi-scale and grouped approach. The skip connection module uses features output by both the high-level and low-level encoders. This approach allows for repeated comparison and correction of regions of high feature significance while maintaining low complexity. In the skip connection module, the multi-scale feature maps used in deep supervision training and the output features of the edge threshold-guided parallel encoder are also used. This method allows the extracted features to be reused repeatedly, ensuring the segmentation performance of the network without wasting computing resources.
[0065] In the overall network architecture, parallel networks replace serial networks, reducing the number of model parameters and computational complexity, making the model lightweight. In the backbone encoder of the ETMA-Net model, the number of output channels at each level is {8, 16, 24, 32, 64, 128}, and the corresponding number of output channels at each decoder level is {64, 32, 24, 16, 8, ClassNum}.
[0066] Furthermore, the input of the edge threshold guided parallel encoder is obtained by processing the original dermoscopic image and the original dermoscopic image through the Canny edge detection algorithm and the OSTU algorithm respectively.
[0067] The OSTU algorithm includes: converting the original dermoscopic image into a grayscale image and calculating the pixel frequency histogram of the grayscale image; dividing the pixels into different regions using a selected threshold and calculating the inter-class variance of the different regions; and determining the optimal threshold through linear search, with the judgment criterion being the maximum inter-class variance.
[0068] Specifically, the core idea of the OSTU algorithm is to maximize the inter-class variance of different category regions. The larger the inter-class variance, the greater the difference between pixels, and thus the better the effect. The steps are to first convert the original image into a grayscale image, and then calculate the pixel frequency histogram of the grayscale image. The formula is as follows:
[0069] (1)
[0070] Among them, p i Indicates the frequency of pixels with pixel value i; I i =i represents the number of pixels with value i. Next, a linear search is used to determine the threshold. The criterion for determining the quality of the threshold is the maximum inter-class variance. To calculate the inter-class variance, we first calculate the number of pixels in the regions segmented by the selected threshold, and then calculate the inter-class variance of each region. The formula is as follows.
[0071] (2)
[0072] Among them, g is the between-class variance; w0 and w1 are the proportions of foreground pixels and background pixels in the entire image, which can be expressed as and ; t is the selected threshold; u0 and u1 are the average grayscale values of the foreground pixel and the background pixel, respectively, which can be expressed as and Finally, the optimal threshold can be obtained through linear search.
[0073] The Canny edge detection algorithm includes: performing Gaussian filtering and denoising on the original dermoscopic image to obtain a smoothed image; using the Sobel operator to calculate the image gradient on the smoothed image, and calculating the size and direction of the edge gradient through the image gradient; performing non-maximum suppression on the image, as well as double threshold and connectivity tests, to obtain an image processed by the Canny edge detection algorithm.
[0074] Specifically, the core idea of the Canny edge detection algorithm is to find the area with the largest change in pixel value gradient, which is the initial edge, and then determine the actual edge pixel location by thresholding. When executing the Canny edge detection algorithm, the original image must first be subjected to Gaussian filtering and denoising. Using the two-dimensional discrete Gaussian kernel matrix calculation formula, a 5×5 Gaussian kernel is calculated, and then convolved with the original image. The output image is the image after Gaussian filtering and denoising. The two-dimensional discrete Gaussian kernel matrix calculation formula is as follows:
[0075] (3)
[0076] Wherein, σ is the variance. In an embodiment of the present invention, the size of the Gaussian kernel is set to 5×5. The image after the Gaussian kernel convolution processing will become smooth. At this time, a single pixel noise point in the original image has almost no effect on the image after the Gaussian filter processing. Next, the Sobel operator is used to calculate the pixel gradient of each pixel on the smoothed image, and the first-order derivatives in the vertical and horizontal directions are obtained by convolving the Sobel operator with the smoothed image, which also corresponds to the image gradient. Then, the edge gradient of the image is obtained by the image gradients in the horizontal and vertical directions.
[0077] (4)
[0078] Among them, G x and G y are the image gradients in the horizontal and vertical directions respectively. In addition, the direction of the edge gradient needs to be calculated through the image gradient, as shown in the following formula.
[0079] (5)
[0080] After determining the magnitude and direction of the image's edge gradient, the image is scanned, performing non-maximum suppression to remove non-edge points. Each pixel is checked to see if its gradient is the largest among all surrounding pixels with the same gradient direction. If so, the point is retained; otherwise, it is removed. Finally, to ensure that the selected pixels represent edge information, a dual threshold and connectivity check, known as hysteresis thresholding, is performed. By setting dual thresholds, eligible edge points are found. If a pixel falls between the two thresholds, its connectivity is determined. If so, it is retained; otherwise, it is removed. Through these steps, the image processed by the Canny edge detection algorithm is obtained.
[0081] After obtaining the images processed by the OSTU and Canny algorithms, the two images are used as inputs to the threshold-guided and edge-guided parallel networks. The inputs to the edge-threshold-guided parallel encoder specifically include: the original dermoscopic image, the image processed by the Canny edge detection algorithm, and the image processed by the OSTU algorithm.
[0082] Further, if Figure 3 As shown, the edge threshold guided parallel encoder includes an ASPP module and an RFB-s module. The ASPP (Atrous Spatial Pyramid Pooling) module is a dilated spatial convolutional pyramid pooling module, and the RFB (Receptive Field Block)-s module is a feature extraction module.
[0083] In the parallel encoder, a standard convolution module performs preliminary feature extraction on the input edge image or threshold image, followed by the ASPP module to enhance feature extraction capabilities. In the ASPP module, dilated convolutions are used to extract features at three different scales from the feature map. The ASPP module primarily consists of a unit convolution, three dilated convolutions with different receptive fields, and a pooling module. The feature map output by the pooling module is unified to the size of the input feature map through upsampling. Finally, all feature maps are merged and concatenated before being output. The dilation rates of the three dilated convolutions are {6, 12, 18}, respectively. The pooling module is an adaptive average pooling layer, and each convolution in the ASPP module is followed by a batch normalization layer and a Reinforced Lu (ReLU) activation function.
[0084] After the ASPP module, a regular convolution module is connected to maintain the same feature map size as the backbone's grouped multi-scale Hadamard attention module, facilitating processing in the skip connection module. Finally, an RFB-s module is connected. In the RFB-s module, by adding regular convolution before the dilated convolution layer, deep encoding of feature information is achieved, enabling better feature extraction from dilated convolutions of different scales. The RFB-s module includes unit convolution, regular convolution, and four dilated convolutions with expansion rates of {1, 3, 3, 5}, respectively. An addition operation is performed before the output, and the output weight of the multi-scale framework is 0.1. In addition, each convolution block in the RFB-s module is connected to a batch normalization layer (BatchNormalization) and a Relu activation function.
[0085] Each convolutional module in the parallel encoder is followed by a max pooling layer to remove redundant feature information and compress knowledge. This framework also facilitates operation in subsequent skip connection modules. To reduce the impact of BatchSize on network training, the present invention connects GroupNorm after the convolutional layer in the ETMA-Net model, where the number of grouped categories is 4.
[0086] The parallel encoder's parameter count and computational complexity may not change significantly compared to other lightweight model branches. This is because the ETMA-Net model uses an edge threshold parallel network to extract as many features as possible from the original image, reducing the difficulty of feature extraction in the backbone network and thereby increasing feature extraction across the entire network structure. While this approach may not significantly reduce the complexity of the parallel network, from the perspective of the network as a whole, this framework significantly increases feature extraction and reduces overall complexity. The parallel network structure is sufficient to extract most edge and internal features, and the inclusion of multiple scales enhances the parallel network's ability to learn knowledge and absorb information from different receptive fields in a human-like manner. This feature extraction capability significantly reduces the burden on the backbone network.
[0087] In the parallel encoder framework, the modules are arranged in such a way that they capture as much information as possible from the input image. Therefore, the ASPP module is used to expand the receptive field and extract a large number of features. In the subsequent feature encoding stage, redundant feature information must be removed and information enhanced. Therefore, the RFB-s module is used to encode and filter the input information, followed by dilated convolution to expand the receptive field and collect information. This sequential network structure maintains the ability to enhance primary and secondary features while extracting and filtering a large number of features.
[0088] Furthermore, in the edge threshold guided parallel encoder, the input image is in RGB format. , after edge threshold processing, becomes Then, it passes through the convolution module, ASPP module and RFB-s module in the parallel encoder for feature extraction:
[0089] (6) (7) (8)
[0090] Among them, Y1(x), Y2(x), and Y3(x) represent the overall calculation functions of the convolution module layer, ASPP module layer, and RFB-s module layer, respectively; C(x), A(x), and R(x) represent the calculation functions of the convolution module, ASPP module, and RFB-s module, respectively; φ and δ represent the maximum pooling and Gelu activation functions, respectively.
[0091] Every time a network layer is passed, the size of the input tensor is reduced. That is, the output size of the corresponding layer H×W is And in the 5 outputs of the parallel network, the corresponding number of channels are {8, 16, 24, 32, 64} respectively.
[0092] Furthermore, in the backbone encoder of the ETMA-Net model, a grouped multi-scale Hadamard attention module is proposed. This module is a lightweight network structure that extracts coarse-grained features and enhances fine-grained features. First, the first three layers of the backbone encoder are ordinary convolutional layers connected to maximum pooling modules, and the last three layers are grouped multi-scale Hadamard attention modules connected to maximum pooling modules. Figure 4 As shown, the grouped multi-scale Hadamard attention module includes two sub-modules, one is the grouped multi-scale attention convolution sub-module, and the structure is as follows: Figure 5 As shown, the other is the grouped Hadamard attention convolution submodule with the following structure: Figure 6 shown.
[0093] The implementation of the grouped multi-scale attention convolution submodule uses a pyramid splitting method. First, the input tensor is evenly divided into four groups on the channel through four grouped convolutions of different sizes. The number of groups is {1, 2, 4, 8}, and the convolution kernel size is {3, 5, 7, 9}. Then, four channel attention weights are generated through the SE module, and the four groups of tensors and four weights are merged into one tensor and one weight. The weights are then normalized by the SoftMax function and multiplied with the tensor as the output of the grouped multi-scale attention convolution submodule. The grouped multi-scale attention convolution submodule integrates contextual information of different scales and generates good pixel-level attention. The weights generated by feature maps of different scales can accelerate the capture of information in pathological areas of the feature map. When grouping, each group performs convolution with different convolution kernel sizes to obtain receptive fields of different scales and extract information of different scales, which helps to balance global and local features.
[0094] In the grouped Hadamard attention convolution submodule, the input tensor is first divided into 4 groups, and then 3 tensors are established. The dimensions are respectively expanded to the corresponding input dimensions through convolution operations, and then Hadamard products are performed; through the training of the 3 tensors established, weight matrices corresponding to different dimensions are obtained, and the weights are processed separately using depthwise separable convolution to keep the size corresponding to the grouped tensor; finally, the grouped tensors multiplied by the weights are merged as the output of the grouped Hadamard attention convolution submodule. In the grouped Hadamard attention convolution submodule, by using trainable tensors to perform Hadamard products as attention weights, the quadratic complexity of the attention generation process of many algorithms is overcome. The features in the three dimensions of channel, height and width are mapped through 3 tensors. After training, the 4 groups of features are merged to integrate feature information from different angles.
[0095] Furthermore, in the grouped multi-scale Hadamard attention module, the input is passed to the grouped multi-scale attention convolution submodule and the grouped Hadamard attention convolution submodule to obtain outputs. The outputs of the two submodules are combined and then a 3×3 convolution is performed to adjust the number of channels to the size of the corresponding parallel encoder output. The grouped convolution process of the grouped multi-scale attention convolution submodule is as follows:
[0096] (9)
[0097] Among them, i∈[0,1,2,3]; k is the convolution kernel size corresponding to the group convolution; g is the number of groups; M i is the output of group convolution. The outputs of multiple groups of convolution are combined as follows:
[0098] (10)
[0099] The attention weight is generated by the SE module and calculated with M in the form of a product to obtain the output, as shown in the following formula:
[0100] (11) (12)
[0101] Among them, W represents the weight generated by the SE module; Att represents the weight vector obtained by the Softmax calculation of the weight generated by the SE module; Out GMA Represents the output of the grouped multi-scale attention convolution submodule. The attention weight is calculated by formula (11), and the final grouped multi-scale attention output is obtained by formula (12). Next, it is merged with the grouped Hadamard attention convolution submodule and fused through convolution calculation to obtain the final output, as shown in the following formula:
[0102] (13)
[0103] Among them, Out GHA Represents the output of the grouped Hadamard attention convolution submodule; Out represents the final output of the grouped multi-scale Hadamard attention module.
[0104] Furthermore, the present invention designs a new multi-level feature aggregation skip connection module between the parallel encoder and decoder of the ETMA-Net model. The multi-level feature aggregation skip connection module connects the backbone encoder and the deep supervision decoder, and is used to group and fuse the output features of the edge threshold-guided parallel encoder, the high- and low-level coding modules in the backbone encoder, the low-level feature aggregation module, and the deep supervision decoder, and then merge them into the output of the multi-level feature aggregation skip connection module after convolution calculation.
[0105] The jump connection module uses the high-level and low-level features in the decoder to make information highly utilized when it is transmitted in the network, while highly retaining the feature information transmission and reducing the consumption of repeated calculations. Figure 7The skip connection module shown uses the high-level feature tensor output by the next-level multi-level aggregation connection. This means that the output of the high-level multi-level aggregation connection is not only passed to the decoder, but also serves as the input to the previous-level multi-level aggregation connection. This approach ensures that when the main features extracted at high depths have deviations, they can be corrected using low-level features. When no deviations exist, this approach can enhance feature extraction and distillation capabilities while reducing computational complexity. The output of the edge threshold-guided parallel encoder is also used in the multi-level feature aggregation skip connection module. This enhances the network's understanding of the image by using the edge threshold information of the original image. Deeply supervised decoder output is also introduced, and this interaction with the decoder enhances the overall coordination of the network.
[0106] In the multi-level feature aggregation skip connection module, all input tensors are divided into four blocks, which are then reassembled into four tensors through cross-grouping. Dilated convolutions with dilation rates of {1, 2, 5, 7} are then performed to absorb feature information from different receptive fields. Finally, the four resulting tensors are concatenated and adjusted to the number of channels in the adaptive network through unit convolution.
[0107] Furthermore, the ETMA-Net model utilizes a deep-supervision decoder to compensate for the lack of visual information understanding caused by reducing the network's computational workload. This deep-supervision decoder comprises six computational modules, the first three of which utilize grouped Hadamard convolutions, while the last three utilize two-dimensional convolutions. During the computational transfer process, the output is first used for deep-supervision calculations and then combined with the output of the skip connection module for backward propagation.
[0108] Furthermore, in the ETMA-Net model, to optimize the network training results on dermoscopic images, the Dice loss function and the BCE loss function are used for model training. The Dice loss function is derived from the Dice Score, which can reduce overfitting and accelerate convergence during model training. The formula is as follows:
[0109] (14)
[0110] Where N is the total number of image pixels; P i and G i are the values of i pixels in the pathological segmentation image predicted by the model and the real labeled image, respectively.
[0111] In addition, the skin lesion segmentation problem is also a pixel-level binary classification problem, so the present invention also uses binary cross entropy loss, the formula is as follows:
[0112] (15)
[0113] Where N is the total number of image pixels; P i and G i are the values of i pixels in the pathological segmentation image predicted by the model and the real labeled image, respectively.
[0114] The loss function used in this invention is the sum of the Dice loss function and the BCE loss function. Since the ETMA-Net model uses a deep supervised training method, there are a total of 6 loss calculations in the model, one of which is the loss of the final output, and the remaining 5 are loss calculations of the decoder output. The formula is as follows.
[0115] (16)
[0116] (17)
[0117] Where L in formula (16) represents the output result of each loss calculation; L in formula (17) P and L i are the loss of the final output and the loss of the i-th output of the decoder respectively; ω i is the sum weight of the output loss of the i-th decoder, and the values of the present invention are {0.5, 0.4, 0.3, 0.2, 0.1} respectively; L Final This is the final calculated loss of the ETMA-Net model.
[0118] S3. Training the ETMA-Net model.
[0119] In the embodiment of the present invention, the ETMA-Net model can be trained using existing dermatoscopic image datasets, skin lesion datasets, etc., which will not be described in detail here.
[0120] S4. Input the dermoscopic image into the trained ETMA-Net model to obtain the skin lesion segmentation result.
[0121] To validate the effectiveness of this invention, the ETMA-Net model was tested for performance and migration using publicly available skin lesion datasets and polyp pathology datasets. The skin lesion datasets included the ISIC 2016, ISIC 2017, and ISIC 2018 datasets provided by the International Skin Imaging Collaboration (ISIC), and the PH2 dataset provided by the Department of Dermatology at Hospital Pedro Hispano in Matosinhos of Portugal. The polyp pathology dataset was the CVC-ClinicDB dataset.
[0122] The skin lesion segmentation task is a pixel-level binary classification task. In order to better evaluate the performance of the segmentation algorithm, the evaluation indicators used in this paper are mean Intersection over Union (mIoU), Dice coefficient (Dice), Accuracy (Acc), Specificity (Spec), Sensitivity (Sens), Precision (Prec), Hausdorff distance (HD) and Average symmetric surface distance (ASSD), and the corresponding formulas are Equations (18) to (25), respectively.
[0123] (18) (19) (20) (twenty one)
[0124] (twenty two)
[0125] (twenty three) (twenty four) (25)
[0126] Where TP, FP, TN, and FN represent true positives, false positives, true negatives, and false negatives, respectively; P and G represent the predicted label and the true label, respectively; and d represents the distance calculation function between two pixels. Furthermore, this paper uses parameters (Params) and floating-point operations (GFLOPs) to evaluate model complexity.
[0127] The ETMA-Net model was implemented in Pytorch using an NVIDIA RTX A6000 GPU. The batch size was set to 8, and training was performed for approximately 75 epochs. The AdamW optimizer was used for testing, with an initial learning rate of 0.001, momentum of 0.9, and weight decay of 0.01. Data augmentation was initially performed on all dataset images, including image normalization to [0, 1], horizontal and vertical flips with a probability of 0.5, random rotations, and resizing to 256×256.
[0128] The proposed ETMA-Net model was tested on publicly available skin lesion and polyp segmentation datasets. The effectiveness and portability of the ETMA-Net model were verified by experimental comparisons with seven other classic algorithms and nine state-of-the-art algorithms. The comparison algorithms included UNet, SegNet, ResUNet, UNet++, Attention UNet, ResUNet++, CENet, UTNet, TransUNet, MFSNet, UNeXt, MALUNet, FatNet, EGE-UNet, MSCANet, and UNet#.
[0129] Taking the PH2 dataset as an example, Table 1 shows the segmentation performance of ETMA-Net and other classic and state-of-the-art segmentation algorithms on the PH2 dataset. From the table, it can be analyzed that the ETMA-Net algorithm has good performance in skin lesion segmentation, and compared with other comparison algorithms, due to the lightweight design of the parallel network, ETMA-Net has fewer parameters. And because ETMA-Net uses many parallel calculations and Hadamard product operations, ETMA-Net has less computational complexity. Figure 8a and Figure 8b It can be further concluded that ETMA-Net has fewer parameters and computational overhead than other models, further verifying the effectiveness of the fusion of edge threshold parallel networks and Hadamard products. Table 2 shows the parameter and computational overhead of ETMA-Net and other comparison algorithms.
[0130] Table 1 Test results of different algorithms on the PH2 dataset. Bold indicates the best result
[0131]
[0132] In Table 1, the larger the values of the indicators in the first three columns, the better the model's segmentation performance, and the smaller the values of the indicators in the last two columns, the better the model's segmentation performance. However, the indicators in columns 4 through 6 alone cannot reflect the model's performance. This is because when the target area is small, the model may classify all parts as non-target areas. In this case, the indicators in columns 4 through 6 will increase, but the segmentation effect will be very poor. Therefore, it is necessary to comprehensively analyze the model's performance using all indicators. In Table 1, the ETMA-Net model has superior values for both segmentation accuracy and similarity of the segmented target areas. It can be concluded that ETMA-Net has excellent performance in skin pathology image segmentation.
[0133] Table 2 Parameters and computational complexity of different algorithms. Bold indicates the optimal result.
[0134]
[0135] As can be seen from Table 2, ETMA-Net has fewer parameters and computational complexity than other classic models or the most advanced models, which not only reflects the lightweight nature of the model but also facilitates its deployment. The ETMA-Net model is more suitable for skin pathology segmentation models trained with large amounts of data. Figure 8a and Figure 8b The figure shows a comparison of the mIoU metric, parameter count, and computational complexity for different models. The horizontal and vertical coordinates in the figure show that models closer to the upper left corner of the image are more lightweight and have higher performance. Therefore, both figures indicate that the ETMA-Net algorithm is lighter and has better performance than the other compared algorithms.
[0136] In order to compare the generalization capabilities of different segmentation models in cross-domain dataset scenarios, the present invention uses cross-dataset testing to evaluate the cross-data generalization capabilities of different models. The evaluation of cross-dataset generalization capabilities is very important. In fact, cross-dataset experiments play a vital role in evaluating the effectiveness of different models in actual scenarios. Table 3 shows the test results of the training model obtained on the PH2 dataset applied to the ISIC 2017 dataset, using mIoU, Dice, Acc, Spec, Sens, Prec, HD95 and ASSD as evaluation indicators. From the overall results in Table 3, it can be analyzed that ETMA-Net has good cross-data generalization capabilities compared with classic segmentation methods and the latest segmentation methods.
[0137] Table 3 Results of cross-dataset testing on the ISIC 2017 dataset using the model trained on the PH2 dataset
[0138]
[0139] In Table 3, the ETMA-Net model improves the mIoU metric by 3.88%-25.00%, the Dice metric by 2.94%-21.79%, and the Acc metric by 1.10%-8.24% compared to other comparison models. This analysis shows that the ETMA-Net model has good cross-dataset generalization capabilities and is adaptable to different datasets, demonstrating its superior cross-dataset application performance.
[0140] The ETMA-Net model consists of a basic backbone network, an edge threshold-guided parallel encoder, a skip connection module, and a decoder. Grouped multi-scale Hadamard attention and grouped Hadamard attention mechanisms are used in the parallel encoder and decoder. To verify the performance of different architectures, we conducted ablation experiments on the ETMA-Net model using the ISIC 2018 dataset. The test results are shown in Table 4.
[0141] Table 4 Ablation experiments on the ISIC 2018 dataset
[0142]
[0143] Table 4 compares the baseline without parallel networks and skip connections, the framework with parallel networks and skip connections without GHA, the framework with parallel networks and skip connections without GMHA, and the framework with parallel networks and skip connections. The results show that the GHA module significantly reduces model parameters and computational complexity, and that adding the GMHA module significantly improves segmentation performance. The results also show that the parallel network architecture and multi-stage skip connections effectively improve segmentation performance.
[0144] To verify the segmentation and transfer performance of the ETMA-Net model on images from other medical fields, we used the CVC-ClinicDB dataset to test the performance of different segmentation algorithms. The results are shown in Table 5. The CVC-ClinicDB dataset is a medical image dataset commonly used for polyp lesion segmentation. The test results show that the ETMA-Net model exhibits certain performance advantages over other comparison algorithms, achieving improvements of 2.54%-30.36% in the mIoU metric, 2%-29.06% in the Dice metric, and 0.08%-4.84% in the Acc metric.
[0145] Table 5 Test results of different algorithms on the CVC-ClinicDB dataset. Bold indicates the best result
[0146]
[0147] In summary, compared with the existing technology, the skin lesion segmentation method based on ETMA-Net proposed in this invention has the following advantages:
[0148] (1) A new feature extraction network structure is proposed in the ETMA-Net model. This structure amplifies the features of the original image by processing the input image, providing better feature information for subsequent networks. The proposed network structure is a parallel network with three inputs: the original image and the image processed by Canny edge detection and OSTU thresholding.
[0149] (2) In order to reduce the number of parameters and computational complexity in the ETMA-Net model, a multi-scale grouped Hadamard attention mechanism is introduced into the backbone network of the framework, replacing multiple convolutional layers. This mechanism not only reduces the number of parameters and computational complexity, but also maintains segmentation accuracy.
[0150] (3) In the ETMA-Net model, a multi-level feature aggregation skip connection module is used as the connection between the encoder and decoder. The multi-level feature fusion structure includes the features extracted by the parallel network and the high- and low-level image features output by the encoder. The decoder output features are also used for regulation. Multi-level feature fusion allows the network to pay more attention to fine-grained features such as skin texture in pathological areas, thereby improving the network's segmentation performance.
[0151] (4) During the training of the ETMA-Net model, a deep-supervised hybrid loss is used to make the network more accurately control the convergence direction during training. The hybrid loss combines the Dice loss and the BCE loss.
[0152] (5) The test results on the dataset show that ETMA-Net has certain advantages over other comparison algorithms. The experimental indicators used are mIoU, DSC, Acc, Params, and GFLOPs.
[0153] (6) In comparative testing, ETMA-Net was tested for its migration performance between different datasets of the same pathological image type using a cross-dataset. Furthermore, ETMA-Net was tested for its migration performance between datasets of different pathological sites using the CVC-ClinicDB dataset. The experimental results indicate that ETMA-Net has good migration capabilities when trained on datasets in the same domain and is also effective in segmenting images of other pathological sites.
[0154] Accordingly, an embodiment of the present invention further provides a skin lesion segmentation device based on ETMA-Net. Figure 9 FIG. 1 is a schematic diagram showing a structure of a skin lesion segmentation device based on ETMA-Net according to an exemplary embodiment. Figure 9 As shown, the device includes:
[0155] An image acquisition module 201 is used to acquire a dermatoscopic image to be segmented;
[0156] A model construction module 202 is used to construct an ETMA-Net model, wherein the ETMA-Net model includes: an edge threshold-guided parallel encoder, a grouped multi-scale Hadamard attention module, a multi-level feature aggregation skip connection module, and a deep supervision decoder;
[0157] A model training module 203 is used to train the ETMA-Net model;
[0158] The segmentation module 204 is used to input the dermoscopic image into the trained ETMA-Net model to obtain a skin lesion segmentation result.
[0159] For ease of explanation, Figure 9 Only the main components of the device are shown. The device of this embodiment can be used to perform Figure 1 The technical solution of the method embodiment shown has similar implementation principles and technical effects, which will not be repeated here.
[0160] In an exemplary embodiment, the present invention further provides an electronic device, comprising:
[0161] processor;
[0162] A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are loaded and executed by the processor, the steps of the skin lesion segmentation method based on ETMA-Net are implemented.
[0163] Figure 10 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention, such as Figure 10 As shown, electronic device 300 may include a processor 3001 and a memory 3002. Optionally, electronic device 300 may further include a transceiver 3003. Processor 3001, memory 3002, and transceiver 3003 may be connected, for example, via a communication bus. Memory 3002 stores computer-readable instructions that, when executed by processor 3001, implement the steps of the aforementioned ETMA-Net-based skin lesion segmentation method.
[0164] In a specific implementation, as an embodiment, the processor 3001 may include one or more CPUs, such as Figure 10 CPU0 and CPU1 are shown in FIG.
[0165] In a specific implementation, as an embodiment, the electronic device 300 may also include multiple processors, such as Figure 103001 and processor 3004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).
[0166] The memory 3002 is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor 3001. The specific implementation method can refer to the above method embodiment and will not be repeated here.
[0167] The transceiver 3003 is used to communicate with a network device or a terminal device.
[0168] Optionally, the transceiver 3003 may include a receiver and a transmitter, wherein the receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.
[0169] Optionally, the transceiver 3003 may be integrated with the processor 3001 or exist independently and be coupled to the processor 3001 through an interface circuit of the electronic device 300 , which is not specifically limited in this embodiment of the present invention.
[0170] It should be noted that Figure 10 The structure of the electronic device 300 shown in the figure does not constitute a limitation on the electronic device. The actual electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. In addition, the technical effects of the electronic device 300 can refer to the technical effects of the above-mentioned method embodiment, and will not be repeated here.
[0171] In an exemplary embodiment, the present invention further provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the steps of the above-described ETMA-Net-based skin lesion segmentation method. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device.
[0172] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements but also other elements not explicitly listed, or also includes elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the element.
[0173] References in the specification to "one embodiment," "an embodiment," "exemplary embodiments," "some embodiments," etc. indicate that the described embodiments may include a particular feature, structure, or characteristic, but not every embodiment necessarily includes that particular feature, structure, or characteristic. In addition, when a particular feature, structure, or characteristic is described in conjunction with an embodiment, it is within the knowledge of persons skilled in the relevant art to implement that feature, structure, or characteristic in conjunction with other embodiments (whether or not explicitly described).
[0174] It should be understood that the term "and / or" as used herein simply describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the related objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.
[0175] In this disclosure, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.
[0176] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0177] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of the device or unit, which can be electrical, mechanical or other forms.
[0178] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of the solution of this embodiment according to actual needs.
[0179] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0180] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical disks.
[0181] The present invention encompasses any alternatives, modifications, equivalents, and solutions that fall within the spirit and scope of the present invention. To provide a thorough understanding of the present invention, specific details are described in detail below in connection with the preferred embodiments of the present invention, but those skilled in the art will be able to fully understand the present invention without these detailed descriptions. Furthermore, to avoid unnecessary confusion regarding the essence of the present invention, well-known methods, processes, procedures, components, and circuits have not been described in detail.
[0182] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A skin lesion segmentation method based on ETMA-Net, characterized in that: The following steps are involved: S1, obtaining a dermatoscopic image to be segmented; S2. Construct an ETMA-Net model, which includes: an edge threshold-guided parallel encoder, a grouped multi-scale Hadamard attention module, a multi-level feature aggregation skip connection module, and a deep supervision decoder; S3, training the ETMA-Net model; S4. Input the dermoscopic image into the trained ETMA-Net model to obtain the skin lesion segmentation result.
2. The skin lesion segmentation method based on ETMA-Net according to claim 1, characterized in that: The input of the edge threshold guided parallel encoder is obtained by processing the original dermoscopic image and the original dermoscopic image by the Canny edge detection algorithm and the OSTU algorithm respectively; The OSTU algorithm includes: converting the original dermoscopic image into a grayscale image, calculating the pixel frequency histogram of the grayscale image; dividing the pixel points into different regions by selecting a threshold value, and calculating the inter-class variance of different regions; determining the optimal threshold by linear search, and the judgment criterion is the maximum inter-class variance; The Canny edge detection algorithm includes: performing Gaussian filtering and denoising on the original dermoscopic image to obtain a smoothed image; using the Sobel operator to calculate the image gradient on the smoothed image, and finding the size and direction of the edge gradient through the image gradient; performing non-maximum suppression, double threshold and connectivity tests on the image to obtain an image processed by the Canny edge detection algorithm.
3. The skin lesion segmentation method based on ETMA-Net according to claim 1, characterized in that: The edge threshold guided parallel encoder includes an ASPP module and an RFB-s module; The ASPP module includes a unit convolution, three dilated convolutions and a pooling module. The size of the feature map output by the pooling module is unified to the size of the input feature map by upsampling, and all feature maps are merged and connected before output; wherein the expansion rates of the three dilated convolutions are {6, 12, 18} respectively, the pooling module is an adaptive average pooling layer, and each convolution in the ASPP module is connected to a batch normalization layer and a Relu activation function; The RFB-s module includes a unit convolution, a normal convolution and four dilated convolutions. The expansion rates of the four dilated convolutions are {1, 3, 3, 5}, respectively. An Addition operation is performed before the output, and each convolution in the RFB-s module is connected to a batch normalization layer and a Relu activation function.
4. The skin lesion segmentation method based on ETMA-Net according to claim 1, characterized in that: In the edge threshold guided parallel encoder, the input image is in RGB format. , after edge threshold processing, becomes Then, it passes through the convolution module, ASPP module and RFB-s module in the parallel encoder for feature extraction: ; Among them, Y1(x), Y2(x), and Y3(x) represent the overall calculation functions of the convolution module layer, ASPP module layer, and RFB-s module layer respectively; C(x), A(x), and R(x) represent the calculation functions of the convolution module, ASPP module, and RFB-s module respectively; φ and δ represent the maximum pooling and Gelu activation functions respectively.
5. The skin lesion segmentation method based on ETMA-Net according to claim 1, characterized in that: The grouped multi-scale Hadamard attention module includes a grouped multi-scale attention convolution submodule and a grouped Hadamard attention convolution submodule; In the grouped multi-scale attention convolution submodule, the input tensor is first evenly divided into 4 groups on the channel through 4 grouped convolutions of different sizes, the number of groups is {1, 2, 4, 8}, and the convolution kernel sizes are {3, 5, 7, 9}; then, 4 channel attention weights are generated through the SE module, and the 4 groups of tensors and 4 weights are merged into 1 tensor and 1 weight, and then the weight is normalized by the SoftMax function and multiplied with the tensor as the output of the grouped multi-scale attention convolution submodule; In the grouped Hadamard attention convolution submodule, the input tensor is first divided into 4 groups, and then 3 tensors are established. The dimensions are expanded to the corresponding input dimensions through convolution operations, and then Hadamard products are performed; the 3 tensors established through training are used to obtain weight matrices corresponding to different dimensions, and the weights are processed separately using deep separable convolutions to keep the size of the tensors corresponding to the grouping; finally, the grouped tensors multiplied by the weights are merged as the output of the grouped Hadamard attention convolution submodule.
6. The skin lesion segmentation method based on ETMA-Net according to claim 1, characterized in that: In the grouped multi-scale Hadamard attention module, the input is respectively passed into the grouped multi-scale attention convolution submodule and the grouped Hadamard attention convolution submodule to obtain the output, and the outputs of the two submodules are combined and adjusted to the size of the corresponding parallel encoder output through 3×3 convolution; the process of grouped convolution of the grouped multi-scale attention convolution submodule is as follows: ; Among them, i∈[0,1,2,3]; k is the convolution kernel size corresponding to the grouped convolution; g is the number of groups; M i is the output of grouped convolution; the outputs of multiple groups of convolutions are merged as follows: ; The attention weight is generated by the SE module and calculated with M in the form of a product to obtain the output, as shown in the following formula: ; Among them, W represents the weight generated by the SE module; Att represents the weight vector obtained by the Softmax calculation of the weight generated by the SE module; Out GMA represents the output of the grouped multi-scale attention convolution submodule; next, it is merged with the grouped Hadamard attention convolution submodule and fused through convolution calculation to obtain the final output, as shown in the following formula: ; Among them, Out GHA Represents the output of the grouped Hadamard attention convolution submodule; Out represents the final output of the grouped multi-scale Hadamard attention module.
7. The skin lesion segmentation method based on ETMA-Net according to claim 1, characterized in that: The jump connection module of the multi-level feature aggregation connects the backbone encoder and the deep supervision decoder, and is used to group and fuse the output features of the edge threshold-guided parallel encoder, the high-level and low-level coding modules in the backbone encoder, the low-level feature aggregation module and the deep supervision decoder, and then merge them as the output of the jump connection module of the multi-level feature aggregation after convolution calculation.
8. The skin lesion segmentation method based on ETMA-Net according to claim 1, characterized in that: The loss functions used in the ETMA-Net model training are the Dice loss function and the BCE loss function.
9. A skin lesion segmentation device based on ETMA-Net, the device is used to implement the method according to any one of claims 1 to 8, characterized in that: The device comprises: An image acquisition module, used for acquiring a dermatoscopic image to be segmented; A model building module, used to build an ETMA-Net model, wherein the ETMA-Net model includes: an edge threshold-guided parallel encoder, a grouped multi-scale Hadamard attention module, a multi-level feature aggregation skip connection module, and a deep supervision decoder; A model training module, used for training the ETMA-Net model; The segmentation module is used to input the dermoscopic image into the trained ETMA-Net model to obtain the skin lesion segmentation result.
10. An electronic device, characterized in that: The electronic device comprises: processor; A memory having computer-readable instructions stored thereon, wherein the computer-readable instructions, when loaded and executed by the processor, implement the method according to any one of claims 1 to 8.
Citation Information
Patent Citations
Field video image real-time segmentation system and method based on FPGA
CN110717852A
Skin lesion image segmentation method based on deep learning
CN113256641A
Deep learning-based skin lesion image segmentation method and device, and storage medium
CN114066904A
Skin cancer lesion image region segmentation method and system
CN117788487A
Global and local feature reconstruction network-based medical image segmentation method
US20230274531A1
Cited By
Rapid detection and segmentation method for skin lesion area
CN119963831A
A method for rapid detection and segmentation of skin lesion area
CN119963831B
