Skin disease image segmentation method and system based on joint attention convolutional neural network

Abstract

This invention relates to a method and system for skin disease image segmentation based on a joint attention convolutional neural network, belonging to the field of image processing technology. The method includes: acquiring a skin disease image to be segmented; inputting the skin disease image to be segmented into a skin disease image segmentation model, and outputting the skin disease image segmentation result; the skin disease image segmentation model is a trained joint attention convolutional neural network; the joint attention convolutional neural network is a U-Net-based neural network, in which a transfer learning method is used to use a trained ResNet-34 as the encoder in U-Net, and a spatial attention module is used as a skip connection between the symmetric encoder and decoder in U-Net; in the decoder, each feature decoding block uses a pyramid channel attention module, and a multi-scale fusion attention module is used at the decoder output. This invention improves the accuracy and reliability of skin lesion segmentation in skin disease images.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a skin image segmentation method and system based on a joint attention convolutional neural network. Background Technology

[0002] Traditional image segmentation methods typically rely on optimal thresholding, region growing, active contouring, supervised methods, and edge detection algorithms. However, these methods often require manual intervention or extensive hyperparameter fine-tuning, leading to poor performance in complex scenarios. In contrast, deep learning algorithms can automatically extract features, effectively overcoming the shortcomings of traditional dermatological segmentation algorithms. Furthermore, they can be rapidly extended to different task scenarios through transfer learning. With the development of deep convolutional neural networks (CNNs), the U-Net network has been widely used in medical image segmentation. Inspired by the U-Net network structure, improved versions of U-Net have been widely applied to segment different tissues, organs, or lesions in various medical images, including AttU-Net, CE-Net, CA-Net, CPF-Net, MSU-Net, and FAT-Net.

[0003] AttU-Net: Based on the U-Net network, it introduces an attention mechanism, reconstructs the skip connections of the U-Net network, enhances the network's ability to extract spatial information features, and suppresses the learning of noise and irrelevant information.

[0004] CE-Net combines dense dilated convolution (DAC) and residual multi-kernel pooling (RMP) modules with an encoder-decoder structure to capture more abstract features and retain more spatial information to improve the performance of medical image segmentation.

[0005] CA-Net: Based on the U-Net network, spatial attention, channel attention, and scale attention mechanisms are introduced to improve the network's interpretability and segmentation performance.

[0006] CPF-Net combines the Global Pyramid Guidance (GPG) module and the Scale-Aware Pyramid Fusion (SAPF) module to fuse global / multi-scale contextual information.

[0007] MSU-Net combines multiple convolutional sequences and convolutional kernels with different receptive domains to construct multi-scale blocks, extract more semantic features, and capture detailed multi-scale spatial features to make the features more diverse.

[0008] FAT-Net: Employs a dual encoder that integrates a convolutional neural network (CNN) and a transformer branch to simultaneously capture local features, long-range dependencies, and global contextual information.

[0009] Currently, deep learning-based skin image segmentation algorithms have the following main drawbacks: (1) The network feature extraction capability is limited, features are easily lost, accuracy is low and interpretability is poor, resulting in poor lesion segmentation. (2) The segmentation results for lesions with low contrast (not obvious contrast between foreground and background), occlusion by hair or artifacts, large pixel changes inside the lesion, blurred boundaries, large size changes and irregular shapes are limited.

[0010] The reasons for the above shortcomings are: (1) Insufficient extraction of global context information, insufficient dense prediction of detailed spatial information, and neglect of scale feature fusion at different decoding stages, resulting in the inability to accurately segment irregularly shaped lesions. (2) The network structure is too shallow, and it cannot extract complete local features, resulting in discontinuous edge contours. (3) Neglecting the fusion of scale features at different decoding stages, it cannot accurately predict lesion regions with irregular shapes. (4) Neglecting continuous pooling and mixing inevitably leads to inaccurate lesion boundaries due to limited context information and insufficient discriminative feature mapping. Summary of the Invention

[0011] The purpose of this invention is to provide a method and system for skin disease image segmentation based on joint attention convolutional neural networks, which improves the accuracy and reliability of skin lesion segmentation in skin disease images.

[0012] To achieve the above objectives, the present invention provides the following solution:

[0013] A method for skin disease image segmentation based on joint attention convolutional neural networks includes:

[0014] Obtain the image of the skin disease to be segmented;

[0015] The skin disease image to be segmented is input into the skin disease image segmentation model, and the skin disease image segmentation result is output; the skin disease image segmentation model is a trained joint attention convolutional neural network.

[0016] The joint attention convolutional neural network is a U-Net-based neural network. In this network, a transfer learning method is used to use a trained ResNet-34 as the encoder in the U-Net, and a spatial attention module is used as a skip connection between the symmetric encoder and decoder in the U-Net. The joint attention convolutional neural network also includes a pyramid channel attention module and a multi-scale fusion attention module. In the decoder, each feature decoding block uses the pyramid channel attention module. The input of the multi-scale fusion attention module is the feature map after upsampling the output of each feature decoding block, and the output of the multi-scale fusion attention module is the skin disease image segmentation result.

[0017] Optionally, the encoder in the joint attention convolutional neural network includes sequentially connected encoding layers 1 to N, and sequentially connected decoding layers 1 to N-1, where N is a positive integer;

[0018] The Nth encoding layer and the 1st decoding layer are connected;

[0019] The spatial attention module includes N-2 spatial attention modules, from the first spatial attention module to the (N-2)th spatial attention module. The first input of the nth spatial attention module is the output feature of the nth encoding layer, the second input of the nth spatial attention module is the output feature of the (Nn-1)th decoding layer, and the output of the nth spatial attention module is connected to the Nnth decoding layer. The value of n ranges from 1 to N-2.

[0020] Optionally, a nonlocal operation is used between the (N-1)th encoding layer and the first decoding layer.

[0021] Optionally, each spatial attention module utilizes the formula Obtain spatial attention feature maps;

[0022] Where, x l Let x represent the output feature of the nth coding layer. h This represents the output feature of the (Nn-1)th decoding layer. Represent the spatial attention feature map, α1∈[0,1] H×W Let α2 represent the first attention coefficient, α2∈[0,1]. H×W ReLU represents the second attention coefficient, and ReLU represents the ReLU activation function. This represents the first attention feature map. This represents the second attention feature map. Indicates channel connection; Φ C This represents a 1×1 convolution with C output channels and batch normalization, where C is the product of x and x. l The same value for the number of channels.

[0023] Optionally, the decoding layer is used to add the first feature map and the second feature map, and is also used to decode the addition result using the feature decoding block. The feature decoding block includes an input layer, a first convolutional layer, a second convolutional layer and a pyramid channel attention module connected in sequence. The feature decoding block also includes a third convolutional layer. The input of the third convolutional layer is connected to the input layer. The output of the third convolutional layer is accumulated with the output of the pyramid channel attention module and then output after passing through the ReLU activation function.

[0024] The first and second convolutional layers each include a convolution operation with a 3×3 kernel, and the third convolutional layer includes a convolution operation with a 1×1 kernel.

[0025] Optionally, the pyramid channel attention module includes a pyramid-shaped multi-scale feature extraction block, a channel attention weight extraction unit, and a multi-scale feature extraction unit;

[0026] The pyramid-shaped multi-scale feature extraction block is used to divide the channels of the input feature map into four groups. Each group of channels is convolved using convolution kernels of different sizes. The four convolution results are then concatenated along the channel dimension to obtain the first multi-scale feature map.

[0027] The channel attention weight extraction unit is used to embed the global spatial information of the first multi-scale feature map into the channel descriptor using global average pooling to obtain aggregated features, and then to perform a one-dimensional convolution with a kernel size of 3 on the aggregated features to obtain multi-scale channel attention weights. The multi-scale channel attention weights are calibrated using the activation function Sigmoid.

[0028] The multi-scale feature extraction unit is used to perform element-wise multiplication of the calibrated multi-scale channel attention weights with the first multi-scale feature map to obtain the second multi-scale feature map.

[0029] Optionally, the multi-scale fusion attention module includes a convolutional unit, an upsampling unit, a channel attention unit, and a pixel normalization unit connected in sequence;

[0030] The convolutional unit is used to unify the number of channels in the feature map after upsampling each of the second multi-scale feature maps;

[0031] The upsampling unit is used to unify the feature map output by the convolution unit to a set size and perform channel concatenation to obtain a channel concatenated feature map.

[0032] The channel attention unit is used to extract channel attention features from the channel splicing feature map;

[0033] The pixel normalization unit is used to extract spatial attention features from the channel stitching feature map based on the channel attention features, perform a Softmax activation operation on the spatial attention features to obtain an attention feature map, and perform a convolution operation on the attention feature map to obtain the skin disease image segmentation result.

[0034] Optionally, the training process of the joint attention convolutional neural network includes:

[0035] Obtain a dataset of images of skin lesions;

[0036] Adjust each skin lesion image in the dataset to a first preset size;

[0037] The skin lesion image of the first preset size is randomly cropped to the second preset size;

[0038] Data augmentation is performed on the skin lesion image of the second set size to obtain the data-augmented dataset;

[0039] The joint attention convolutional neural network was trained using the data augmented dataset, and the trained joint attention convolutional neural network was used as a skin disease image segmentation model.

[0040] Optionally, the joint attention convolutional neural network is trained using a hybrid loss function, which includes three levels of loss: map-level loss, patch-level loss, and pixel-level loss.

[0041] This invention discloses a skin disease image segmentation system based on a joint attention convolutional neural network, comprising:

[0042] The module for acquiring skin disease images to be segmented is used to acquire skin disease images to be segmented.

[0043] The skin disease image segmentation module is used to input the skin disease image to be segmented into the skin disease image segmentation model and output the skin disease image segmentation result; the skin disease image segmentation model is a trained joint attention convolutional neural network.

[0044] The joint attention convolutional neural network is a U-Net-based neural network. In this network, a transfer learning method is used to use a trained ResNet-34 as the encoder in the U-Net, and a spatial attention module is used as a skip connection between the symmetric encoder and decoder in the U-Net. The joint attention convolutional neural network also includes a pyramid channel attention module and a multi-scale fusion attention module. In the decoder, each feature decoding block uses the pyramid channel attention module. The input of the multi-scale fusion attention module is the feature map after upsampling the output of each feature decoding block, and the output of the multi-scale fusion attention module is the skin disease image segmentation result.

[0045] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0046] This invention discloses a method and system for skin disease image segmentation based on a joint attention convolutional neural network. Building upon U-Net, it uses transfer learning to employ a trained ResNet-34 as a feature extractor. A spatial attention module, combined with nonlocal operations, establishes long-term dependencies between pixels, extracting effective local features and global contextual information. A pyramid channel attention module fully extracts multi-scale features at different levels during the decoding stage, capturing long-term dependencies between channels and extracting spatial information at different scales in each channel. A multi-scale fusion attention module fuses multi-scale information from different levels during the decoding stage. A novel hybrid loss function based on region prominence features and boundary quality effectively guides the network to learn at the map, patch, and pixel levels, accurately predicting lesion regions with clear boundaries, thus improving the accuracy and reliability of skin lesion segmentation. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a schematic diagram of the process of a skin disease image segmentation method based on a joint attention convolutional neural network according to the present invention;

[0049] Figure 2 This is a schematic diagram of the joint attention convolutional neural network structure of the present invention;

[0050] Figure 3 This is a schematic diagram of the spatial attention module structure of the present invention;

[0051] Figure 4 This is a schematic diagram of the feature decoding block structure of the present invention;

[0052] Figure 5 This is a schematic diagram of the pyramid channel attention module structure of the present invention;

[0053] Figure 6 This is a schematic diagram of the multi-scale fusion attention module structure of the present invention;

[0054] Figure 7 A schematic diagram illustrating the segmentation effect of skin lesions on different networks;

[0055] Figure 8 This is a schematic diagram of the structure of a skin disease image segmentation system based on a joint attention convolutional neural network according to the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] The purpose of this invention is to provide a method and system for skin disease image segmentation based on joint attention convolutional neural networks, which improves the accuracy and reliability of skin lesion segmentation in skin disease images.

[0058] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] Figure 1 This is a schematic diagram of the process of a skin disease image segmentation method based on a joint attention convolutional neural network according to the present invention, as shown below. Figure 1 As shown, a method for skin disease image segmentation based on a joint attention convolutional neural network includes the following steps:

[0060] Step 101: Obtain the image of the skin disease to be segmented.

[0061] Step 102: Input the skin disease image to be segmented into the skin disease image segmentation model and output the skin disease image segmentation result; the skin disease image segmentation model is a trained joint attention convolutional neural network.

[0062] The joint attention convolutional neural network is a U-Net-based neural network. In this network, a transfer learning method is used to use a trained ResNet-34 as the encoder in the U-Net, and a spatial attention module is used as a skip connection between the symmetric encoder and decoder in the U-Net. The joint attention convolutional neural network also includes a pyramid channel attention module and a multi-scale fusion attention module. In the decoder, each feature decoding block uses the pyramid channel attention module. The input of the multi-scale fusion attention module is the feature map after upsampling the output of each feature decoding block, and the output of the multi-scale fusion attention module is the skin disease image segmentation result.

[0063] The training process of the joint attention convolutional neural network includes:

[0064] Obtain a dataset of images of skin lesions.

[0065] The datasets include the ISIC-2018 and ISIC-2017 datasets of skin lesion images. The ISIC-2018 dataset contains 2594 RGB images of skin lesions, which are randomly divided into 1816 images for training, 260 images for validation, and 518 images for testing. The ISIC-2017 dataset provides 2000 images for training, 150 images for validation, and 600 images for testing.

[0066] Adjusting each skin lesion image in the dataset to a first set size specifically includes: the original size of the skin lesion images in the dataset is 720×540-6708×4439, the first set size is 256×342, adjusting each skin lesion image to 256×342, and standardizing it using the mean and standard deviation.

[0067] The first preset size skin lesion image is randomly cropped to a second preset size, specifically including the standardized skin lesion image being randomly cropped to a second preset size of 256×256.

[0068] Data augmentation is performed on the skin lesion image of the second set size to obtain a data-augmented dataset. Specifically, this includes horizontal rotation, vertical rotation, and random rotation of the skin lesion image of the second set size by an angle of (-π / 6, π / 6) to obtain a data-augmented dataset, thereby increasing the diversity of image data samples during training.

[0069] The joint attention convolutional neural network was trained using the data augmented dataset, and the trained joint attention convolutional neural network was used as a skin disease image segmentation model.

[0070] In the training process of the joint attention convolutional neural network of this invention, a five-fold cross-validation method is used, and the average result of five experiments is taken as the final performance.

[0071] This invention is based on the PyTorch platform and trains a joint attention convolutional neural network on an NVIDIA GeForce RTX 3070 GPU with 8GB of VRAM. A "poly" learning rate strategy is employed, with the initial learning rate and power value set to 0.01 and 0.9, respectively. The batch size is set to 12, and the number of iterations is 300. Stochastic gradient descent (SGD) is used to optimize the model, with momentum (the momentum coefficient in the SGD optimizer) and weight decay set to 0.9 and 0.0001, respectively.

[0072] The performance of different methods was evaluated using six metrics: average surface distance (ASSD), Dice coefficient (Dice), overlap (IoU), sensitivity (SE), specificity (SP), and accuracy (ACC).

[0073]

[0074]

[0075]

[0076]

[0077]

[0078]

[0079] Where P is the predicted value and G is the true value. TP, TN, FP, and FN represent true positive, true negative, false positive, and false negative, respectively. a Let S represent the set of boundary points a. b The set representing the ground truth value b, Point The minimum Euclidean distance to all points.

[0080] In the training process of the joint attention convolutional neural network of this invention, a hybrid loss function based on region salience features and boundary quality is employed to effectively highlight the target region and accurately predict the fine structure of clear boundaries. The hybrid loss function L... total =L softDice +λ(L BCE +L SSIM The network is trained under supervised supervision at three levels: map level, patch level, and pixel level, to obtain high-confidence saliency maps and clear boundaries. Among them, L... softDice L SSIM and L BCE These are the soft dice coefficient loss, structural similarity loss (SSIM), and binary cross entropy loss (BCE), respectively. softDice For map-level loss, L SSIM For patch-level loss, L BCE This represents pixel-level loss. λ is the weighting coefficient between loss functions, set to 0.5.

[0081] SoftDice is widely used to evaluate the performance of segmentation based on overlap, a metric at the map level. BCE is a pixel-level loss, regardless of the neighborhood label, and assigns equal weights to foreground and background pixels, which contributes to convergence across all pixels and performs well in binary classification and segmentation. The specific calculations of binary cross-entropy loss and SoftDice loss are as follows:

[0082]

[0083]

[0084] L softDice =1-L Dice

[0085] Among them, L Dice Let A represent the dice loss, B represent the feature map of the network segmentation in this invention, and C represent the ground truth (mask).

[0086] SSIM is a local region for each pixel, assigning a relatively high weight to the boundaries so that the predicted probability of the boundaries is the same as that of the foreground. It encourages predictions to respect the architecture of the original map and highlight the structural features of the object's ground truth. Let x = {x i :i=1,…,N 2} and y = {y i :i=1,…,N 2 The pixel values ​​of the two corresponding patches in the predicted probability map S and the mask G are calculated as follows:

[0087]

[0088] Where μ x μ y , σ x , σ y , σ xy These are the mean, standard deviation, and covariance of the x and y graphs, respectively, with C1 = 0.01. 2 C2 = 0.03 2 It is used to avoid division by zero.

[0089] The Joint Attention Convolutional Neural Network (JAC-Net) extracts features from input skin lesion images, captures spatial attention features, channel attention features, multi-scale features, and fuses features from different levels during the decoding stage. The JAC-Net is based on U-Net and uses a ResNet-34 network instead of the encoder through transfer learning to increase network depth and extract rich feature information. A symmetric encoder-decoder structure with a spatial attention module (DAG) as the network backbone reconstructs skip connections and combines nonlocal operations to capture long-range dependencies between pixels while extracting rich spatial feature information. At the lowest resolution level, the spatial attention module uses nonlocal blocks to capture interactions between all pixels, better extracting overall contextual information. An efficient pyramid channel attention module (EPCA) establishes long-range dependencies between channels, preserving detailed boundary information and multi-scale features while capturing channel attention features. A multi-scale fusion attention module (MSFA) fuses multi-scale features from different levels during the decoding stage. A novel hybrid loss algorithm is used for supervised training at the pixel, patch, and map levels to obtain high-confidence saliency maps and clear boundaries. The technical advantages of the joint attention convolutional neural network of the present invention are as follows: (1) the method of transfer learning is used to reduce network training overhead and save time costs; (2) the spatial attention module DAG is used to reconstruct skip links, suppress irrelevant clutter, reduce noise in the attention map, and avoid misclassification of pixels; (3) an efficient pyramid channel attention module EPCA is designed in the decoding stage, and a multi-scale fusion attention module MSFA is designed at the end of the decoder to extract rich multi-scale feature information.

[0090] The ResNet-34 network is a pre-trained feature extractor. For compatibility reasons, the first four feature extraction blocks are retained while the fully connected layers and average pooling layers are removed. This gives ResNet a fast mechanism, speeds up network convergence, and avoids gradient vanishing.

[0091] like Figure 2 As shown, the encoder in the joint attention convolutional neural network includes a first encoding layer to an Nth encoding layer connected in sequence, and also includes a first decoding layer to an (N-1)th decoding layer connected in sequence, where N is a positive integer.

[0092] The Nth encoding layer and the 1st decoding layer are connected.

[0093] The spatial attention module includes N-2 spatial attention modules, from the first spatial attention module to the (N-2)th spatial attention module. The first input of the nth spatial attention module is the output feature of the nth encoding layer, the second input of the nth spatial attention module is the output feature of the (Nn-1)th decoding layer, and the output of the nth spatial attention module is connected to the Nnth decoding layer. The value of n ranges from 1 to N-2.

[0094] Nonlocal operations are used between the (N-1)th encoding layer and the 1st decoding layer.

[0095] The output of the Nth encoding layer is connected to the 1st decoding layer using a nonlocal operation.

[0096] like Figure 3 As shown, the spatial attention module consists of two parallel attention-gated networks (AGs). Each AG has two branches, and each branch contains a convolutional layer. Figure 3 In the context ψ: 1×1, C, where 1×1 represents 1×1 convolution) and upsampling units ( Figure 3 In the middle (up), convolutional layers are used to capture spatial information, and upsampling units are used to upsample the high-level features of the decoder. h Upsampling and low-level features x from the encoder l Maintain the same shape, where the input feature x l Scaled using attention coefficients α1 and α2 calculated in AG, and analyzed by examining the gating signals x collected in the decoder. h The activation and context information provided are used to select spatial regions, and the grid resampling of attention coefficients is accomplished using bilinear interpolation to obtain spatial features.

[0097] Each spatial attention module uses the formula Obtain spatial attention feature maps;

[0098] Where, x l Let x represent the output feature of the nth coding layer. h This represents the output feature of the (Nn-1)th decoding layer. Represent the spatial attention feature map, α1∈[0,1] H×W [0,1] H×W In the matrix, H×W represents the size of the attention coefficient matrix, α1 represents the first attention coefficient, and α2∈[0,1] H×W α2 represents the second attention coefficient, α1 and α2 are both pixel-level attention coefficients, and ReLU represents the ReLU activation function. This represents the first attention feature map. This represents the second attention feature map. Indicates channel connection; Φ C ( Figure 3In this context, Φ(C) represents a 1×1 convolution with C output channels and batch normalization (BN), where C is the product of x and y. l The same value for the number of channels. Figure 3 middle This represents element-wise multiplication.

[0099] In the DAG spatial attention module, each spatial attention module is composed of two AG modules connected in parallel and a residual structure. By using two AGs to simultaneously focus on the features of the region of interest, the ability to highlight salient features is improved, while noise introduced into low-level information is reduced, thus improving the robustness of segmentation. The main purpose of the residual connection is to weight the elements of the input and the salient feature map, avoiding the loss of some detailed feature information.

[0100] Figure 3 In the middle, due to x l and x h The resolutions are different, and in order to maintain consistency with x l For the same size, x h An upsampling operation was performed. l and the upsampled x h The input signals are first subjected to a 1×1 convolution and then compressed through the output channel C. The results are then summed and passed through a ReLU activation function to obtain a feature map. This feature map is then subjected to a single-channel 1×1 convolution and fed into a Sigmoid function. Finally, a bilinear interpolation upsampling is applied to obtain pixel-level attention coefficients α1∈[0,1]. H×W Similarly, let x be... l and x h Another pixel attention coefficient α2∈[0,1] is obtained by following the same path. H×W Then x l The attention feature maps obtained by multiplying by attention coefficients α1 and α2 respectively are denoted as follows: and The two attention map channels are then concatenated, followed by a 1×1 convolution operation and batch normalization, and then input into the ReLU activation function and combined with x. l Residual connections are used to obtain spatial attention feature maps.

[0101] The decoding layer is used to add the first feature map and the second feature map, and also to decode the addition result using the feature decoding block. The first feature map is the feature map output by the encoding layer that is skipped to the current decoding layer, and the second feature map is the feature map output by the layer preceding the current decoding layer. For the first decoding layer, the second feature map is the feature map output by the Nth encoding layer; for the second to (N-1)th decoding layers, the second feature map is the feature map output by the previous decoding layer. For example, for the second decoding layer, the second feature map is the feature map output by the first decoding layer.

[0102] like Figure 4 As shown, the feature decoding block includes an input layer, a first convolutional layer (conv, 3×3), a second convolutional layer (conv, 3×3), and a pyramid channel attention module (EPCA module) connected in sequence. The feature decoding block also includes a third convolutional layer (conv, 1×1C). The input of the third convolutional layer is connected to the input layer. The output of the third convolutional layer is accumulated with the output of the pyramid channel attention module and then output after passing through the ReLU activation function.

[0103] The first and second convolutional layers each include a convolution operation with a 3×3 kernel, and the third convolutional layer includes a convolution operation with a 1×1 kernel. The output channels of the third convolutional layer are C.

[0104] like Figure 5 As shown, each pyramid channel attention module includes a pyramid-shaped multi-scale feature extraction block. Figure 5 (a) and Channel Attention Weight Extraction Unit ( Figure 5 (b) and multi-scale feature extraction unit ( Figure 5 (c) The output of the pyramid-shaped multi-scale feature extraction block is connected to the input of the channel attention block, and the output of the channel attention block serves as the output of the corresponding decoding layer. Figure 5 middle, This indicates a one-dimensional convolution operation with a 3×3 kernel.

[0105] The pyramid-shaped multi-scale feature extraction block is used to extract the input feature map x∈R. C×H×W The channels are divided into four groups, namely X0, X1, X2 and X3, where H represents the image height and W represents the image width. Each group of channels is convolved using a convolution kernel of a different size (based on the principle that multi-scale convolution kernels in a pyramid structure can generate different spatial resolutions and depths). Then, each group is further convolved (the number of groups G = 1, 4, 8, 16), and the four convolution results are concatenated along the channel dimension to obtain the first multi-scale feature map Y.

[0106] Using formula Determine the first multi-scale feature map Y;

[0107] Where, k i ×k i k represents the kernel size. i =3, 5, 7, 9, group size G i =1,4,8,16 Y i This represents the convolution result of the i-th group, where Cat represents concatenation along the channel dimension and conv represents the convolution operation.

[0108] The channel attention weight extraction unit is used to embed the global spatial information of the first multi-scale feature map into the channel descriptor using global average pooling to obtain aggregated features, and then to perform a one-dimensional convolution with a kernel size of 3 on the aggregated features to obtain multi-scale channel attention weights. The multi-scale channel attention weights are calibrated using the activation function Sigmoid to obtain cross-channel attention.

[0109] The multi-scale feature extraction unit is used to perform element-wise multiplication of the calibrated multi-scale channel attention weights (to obtain cross-channel attention) with the first multi-scale feature map to obtain the second multi-scale feature map.

[0110] Using formula Obtain the second multi-scale feature map.

[0111] Among them, g c Indicates global average pooling. σ represents a fast 1D convolution with a kernel size of 3, σ represents the activation function Sigmoid, δ represents the activation function Softmax, and output represents the second multi-scale feature map.

[0112] like Figure 6 As shown, the input of the multi-scale fusion attention module is connected to the output of each pyramid channel attention module (d1, d2, d3, d4); the multi-scale fusion attention module includes a convolution unit, an upsampling unit, a channel attention unit (BN), and a pixel normalization unit connected in sequence.

[0113] The convolutional unit is used to unify the number of channels in the feature maps after upsampling each of the second multi-scale feature maps. The convolutional unit performs convolution operations on d1, d2, d3, and d4 with a kernel of 1×1 and an output channel of 4.

[0114] The upsampling unit is used to uniformly size the feature maps output by the convolution unit and perform channel concatenation to obtain a channel-concatenated feature map. The set size is 256×256. Figure 6 In this process, the feature maps output by the convolutional units are uniformly sized by using upsampling methods with different ratios.

[0115] The channel attention unit is used to extract channel attention features from the channel splicing feature map.

[0116] The pixel normalization unit is used to extract the spatial attention features of the channel stitching feature map based on the channel attention features, perform a Softmax activation operation on the channel attention features to obtain an attention feature map, and perform a convolution operation on the attention feature map to obtain the skin disease image segmentation result.

[0117] Figure 6 In this process, feature maps d1, d2, d3, and d4 are unified to the same size. The feature map size in layer d4 is 256×16×16, the feature map size in layer d3 is 128×32×32, the feature map size in layer d2 is 64×64×64, and the feature map size in layer d1 is 64×128×128. First, a convolution operation with a 1×1 kernel and 4 output channels is used to transform d4, d3, d2, and d1 into feature maps with 4 channels. Then, bilinear interpolation upsampling is used to resample d4, d3, d2, and d1 after the convolution operation into feature maps of size 4×256×256. These feature maps are then concatenated along the channel dimension to form a 16×256×256 feature map, denoted as F. Figure 6 In this context, γ0, γ1, γ2, and γ3 represent the scaling factors for each channel, ω0, ω1, ω2, and ω3 represent the weights for each channel, and θ0, θ1, θ2, and θ3 represent the spatial attention scaling factors.

[0118] Batch normalization and weight allocation are performed on the stitched feature map F to extract spatial attention features. Pixel normalization units and sigmoid activation operations are used to extract channel attention features. Then, softmax activation is performed on the feature map to obtain the attention feature map. The attention feature map is then convolved with a kernel of 1×1 and two output channels. The output feature map has a size of 256×256, which represents the segmentation result.

[0119] The decoder module uses two 3×3 convolution operations to gradually fuse global context information, and then embeds an efficient pyramid channel attention module (EPCA) to highlight the multi-scale spatial features and boundary information of the channel feature map, so as to obtain a new feature map with rich multi-scale information. Then, the new feature map with rich multi-scale information is residually linked with the input feature map to quickly and efficiently recover the high-resolution feature map, retain multi-scale feature information, improve decoding performance, and alleviate degradation problems during network training.

[0120] The experiments are presented in Tables 1 and 2, showing the performance of different models on the ISIC 2018 and ISIC 2017 datasets, respectively. As shown in Tables 1 and 2, the model of this invention is significantly superior to other comparative methods in terms of segmentation metrics. The Dice of the model JAC-Net of this invention is improved by 4.04%, 3.85%, 2.58%, 1.4%, 0.25%, 1.29%, and 1.74% compared to U-Net, AttU-Net, CE-Net, CA-Net, CPFNet, MSU-Net, and FAT-Net models, respectively.

[0121] Table 1 shows the performance of different state-of-the-art methods on the ISIC 2018 dataset. The bolded values ​​indicate the best results.

[0122] Table 1 shows the performance of different methods on the ISIC 2018 dataset.

[0123]

[0124]

[0125] Table 2 shows the performance of different methods on the ISIC 2017 dataset.

[0126] Model Dice (%) IoU (%) ASSD(pix) ACC (%) SE(%) SP (%) U-Net (2015) 84.99 77.76 133.59 91.02 82.83 94.62 AttU-Net(2019) 84.57 77.04 137.38 90.87 81.65 95.06 CE-Net (2019) 87.06 80.34 117.71 92.64 82.08 96.42 CA-Net (2020) 88.48 82.17 105.26 92.84 84.14 96.10 CPFNet(2020) 89.20 83.29 87.40 94.03 84.51 97.23 FAT-Net (2021) 85.08 77.87 108.97 91.95 76.28 96.48 MSU-Net(2021) 88.15 81.69 107.08 92.90 82.66 97.00 Ours (This Invention) 89.69 83.68 90.41 94.10 83.60 97.73

[0127] The bolded values ​​in Table 2 represent the best results compared to the others.

[0128] Figure 7 This section displays the segmentation results of different networks on the ISIC 2018 and ISIC 2017 datasets. The first five rows show the segmentation results of different networks on the ISIC 2018 dataset, and the last three rows show the segmentation results of different networks on the ISIC 2017 dataset. The first column is the input image, the second column is the actual segmentation result, and the third to tenth columns are the segmentation results of different network methods. Figure 7 As shown, compared with different network methods, the skin disease segmentation network method based on joint attention convolutional neural network of the present invention is more effective and accurate than other comparative methods, especially in the case of low contrast, occlusion by hair or artifacts, large pixel changes inside the lesion, blurred boundaries, large size changes and irregular shape, it achieves the best segmentation results.

[0129] This invention provides a skin disease segmentation network based on a joint attention convolutional neural network, which achieves better segmentation results, especially for lesions with low contrast (inconsistent foreground and background contrast), occlusion by hair or artifacts, large pixel variations within the lesion, blurred boundaries, large size variations, and irregular shapes.

[0130] This invention employs transfer learning to reduce the overhead of network training.

[0131] The efficient pyramid channel attention module EPCA of this invention solves the problem of loss of feature diversity caused by fixed convolution kernel size; and the loss of feature information at each scale caused by feature extraction of a single convolution sequence.

[0132] The joint attention convolutional neural network of this invention will be applied to the analysis of skin images in computer-aided diagnostic systems, which will play an important guiding role in subsequent clinical surgery. At the same time, this network can also be used in other medical image segmentation fields.

[0133] Figure 8 This is a schematic diagram of the process of skin disease image segmentation based on a joint attention convolutional neural network according to the present invention, as shown below. Figure 8 As shown, a skin disease image segmentation system based on a joint attention convolutional neural network includes:

[0134] The skin disease image acquisition module 201 is used to acquire the skin disease image to be segmented.

[0135] The skin disease image segmentation module 202 is used to input the skin disease image to be segmented into the skin disease image segmentation model and output the skin disease image segmentation result; the skin disease image segmentation model is a trained joint attention convolutional neural network.

[0136] The joint attention convolutional neural network is a U-Net-based neural network. In this network, a transfer learning method is used to use a trained ResNet-34 as the encoder in the U-Net, and a spatial attention module is used as a skip connection between the symmetric encoder and decoder in the U-Net. The joint attention convolutional neural network also includes a pyramid channel attention module and a multi-scale fusion attention module. In the decoder, each feature decoding block uses the pyramid channel attention module. The input of the multi-scale fusion attention module is the feature map after upsampling the output of each feature decoding block, and the output of the multi-scale fusion attention module is the skin disease image segmentation result.

[0137] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0138] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for skin disease image segmentation based on a joint attention convolutional neural network, characterized in that, include: Obtain the image of the skin disease to be segmented; The skin disease image to be segmented is input into the skin disease image segmentation model, and the skin disease image segmentation result is output; the skin disease image segmentation model is a trained joint attention convolutional neural network. The joint attention convolutional neural network is a U-Net-based neural network. In this network, a transfer learning method is used to employ a trained ResNet-34 as the encoder in the U-Net. A spatial attention module is used as a skip connection between the symmetric encoder and decoder in the U-Net. The joint attention convolutional neural network also includes a pyramid channel attention module and a multi-scale fusion attention module. Each feature decoding block in the decoder uses the pyramid channel attention module. The input to the multi-scale fusion attention module is the upsampled feature map of the output of each feature decoding block. The output of the multi-scale fusion attention module is the skin disease image segmentation result. The encoder in the joint attention convolutional neural network includes sequentially connected encoding layers 1 to N, and also sequentially connected decoding layers 1 to N-1, where N is a positive integer; The Nth encoding layer and the 1st decoding layer are connected; The spatial attention module includes N-2 spatial attention modules, from the first spatial attention module to the (N-2)th spatial attention module. The first input of the nth spatial attention module is the output feature of the nth encoding layer, the second input of the nth spatial attention module is the output feature of the (Nn-1)th decoding layer, and the output of the nth spatial attention module is connected to the Nnth decoding layer. The value of n ranges from 1 to N-2.

2. The method for skin disease image segmentation based on joint attention convolutional neural network according to claim 1, characterized in that, Nonlocal operations are used between the (N-1)th encoding layer and the 1st decoding layer.

3. The method for skin disease image segmentation based on joint attention convolutional neural network according to claim 1, characterized in that, Each spatial attention module uses the formula Obtain spatial attention feature maps; in, This represents the output feature of the nth coding layer. This represents the output feature of the (Nn-1)th decoding layer. Representation of spatial attention feature map, This represents the first attention coefficient. This indicates the second attention coefficient. ReLU express ReLU Activation function This represents the first attention feature map. This represents the second attention feature map. Indicates channel connection; This represents a 1×1 convolution with C as the number of output channels and batch normalization, where C is taken as AND. The same value for the number of channels.

4. The method for skin disease image segmentation based on joint attention convolutional neural network according to claim 1, characterized in that, The decoding layer is used to add the first feature map and the second feature map, and is also used to decode the addition result using the feature decoding block. The feature decoding block includes an input layer, a first convolutional layer, a second convolutional layer and a pyramid channel attention module connected in sequence. The feature decoding block also includes a third convolutional layer. The input of the third convolutional layer is connected to the input layer. The output of the third convolutional layer is accumulated with the output of the pyramid channel attention module and then output after passing through the ReLU activation function. The first and second convolutional layers each include a convolution operation with a 3×3 kernel, and the third convolutional layer includes a convolution operation with a 1×1 kernel.

5. The skin disease image segmentation method based on a joint attention convolutional neural network according to claim 4, characterized in that, The pyramid channel attention module includes a pyramid-shaped multi-scale feature extraction block, a channel attention weight extraction unit, and a multi-scale feature extraction unit. The pyramid-shaped multi-scale feature extraction block is used to divide the channels of the input feature map into four groups. Each group of channels is convolved using convolution kernels of different sizes. The four convolution results are then concatenated along the channel dimension to obtain the first multi-scale feature map. The channel attention weight extraction unit is used to embed the global spatial information of the first multi-scale feature map into the channel descriptor using global average pooling to obtain aggregated features, and then to perform a one-dimensional convolution with a kernel size of 3 on the aggregated features to obtain multi-scale channel attention weights. The multi-scale channel attention weights are calibrated using the activation function Sigmoid. The multi-scale feature extraction unit is used to perform element-wise multiplication of the calibrated multi-scale channel attention weights with the first multi-scale feature map to obtain the second multi-scale feature map.

6. The method for skin disease image segmentation based on a joint attention convolutional neural network according to claim 5, characterized in that, The multi-scale fusion attention module includes a convolutional unit, an upsampling unit, a channel attention unit, and a pixel normalization unit connected in sequence. The convolutional unit is used to unify the number of channels in the feature map after upsampling each of the second multi-scale feature maps; The upsampling unit is used to unify the feature map output by the convolution unit to a set size and perform channel concatenation to obtain a channel concatenated feature map. The channel attention unit is used to extract channel attention features from the channel splicing feature map; The pixel normalization unit is used to extract spatial attention features from the channel stitching feature map based on the channel attention features, perform a Softmax activation operation on the spatial attention features to obtain an attention feature map, and perform a convolution operation on the attention feature map to obtain the skin disease image segmentation result.

7. The method for skin disease image segmentation based on joint attention convolutional neural network according to claim 1, characterized in that, The training process of the joint attention convolutional neural network includes: Obtain a dataset of images of skin lesions; Adjust each skin lesion image in the dataset to a first preset size; The skin lesion image of the first preset size is randomly cropped to the second preset size; Data augmentation is performed on the skin lesion image of the second set size to obtain the data-augmented dataset; The joint attention convolutional neural network was trained using the data augmented dataset, and the trained joint attention convolutional neural network was used as a skin disease image segmentation model.

8. The method for skin disease image segmentation based on joint attention convolutional neural network according to claim 1, characterized in that, The joint attention convolutional neural network is trained using a hybrid loss function, which includes three levels of loss: map-level loss, patch-level loss, and pixel-level loss. The mixed loss function is expressed as: ; in, For the mixed loss function value, For map-level loss, For patch-level loss, For pixel-level loss, These are weighting coefficients; map-level loss is soft dice coefficient loss, patch-level loss is structural similarity loss, and pixel-level loss is binary cross-entropy loss.

9. A skin disease image segmentation system based on a joint attention convolutional neural network, characterized in that, include: The module for acquiring skin disease images to be segmented is used to acquire skin disease images to be segmented. The skin disease image segmentation module is used to input the skin disease image to be segmented into the skin disease image segmentation model and output the skin disease image segmentation result; the skin disease image segmentation model is a trained joint attention convolutional neural network. The joint attention convolutional neural network is a U-Net-based neural network. In this network, a transfer learning method is used to employ a trained ResNet-34 as the encoder in the U-Net. A spatial attention module is used as a skip connection between the symmetric encoder and decoder in the U-Net. The joint attention convolutional neural network also includes a pyramid channel attention module and a multi-scale fusion attention module. Each feature decoding block in the decoder uses the pyramid channel attention module. The input to the multi-scale fusion attention module is the upsampled feature map of the output of each feature decoding block. The output of the multi-scale fusion attention module is the skin disease image segmentation result. The encoder in the joint attention convolutional neural network includes sequentially connected encoding layers 1 to N, and also sequentially connected decoding layers 1 to N-1, where N is a positive integer; The Nth encoding layer and the 1st decoding layer are connected; The spatial attention module includes N-2 spatial attention modules, from the first spatial attention module to the (N-2)th spatial attention module. The first input of the nth spatial attention module is the output feature of the nth encoding layer, the second input of the nth spatial attention module is the output feature of the (Nn-1)th decoding layer, and the output of the nth spatial attention module is connected to the Nnth decoding layer. The value of n ranges from 1 to N-2.

Images