A skin lesion image segmentation method based on feature interaction fusion

Abstract

The application discloses a skin lesion image segmentation method based on feature interaction fusion, and relates to the technical field of medical image segmentation. The method comprises the following steps: firstly, performing size normalization and noise suppression on a dermoscope image, and expanding a data set through random rotation and flipping operations; then inputting training data into a network model for training to obtain model weights after training; finally, inputting a skin lesion image to be segmented into the network model after training to realize automatic segmentation of a skin lesion area. The application designs a focused feature interaction fusion module to realize deep complementary fusion of CNN and Transformer features; a multi-scale context attention module is constructed to realize multi-scale accurate feature extraction by extracting spatial information from local to global scales while maintaining computational efficiency. The application aims to solve the problems of complex shape, variable size and low contrast of the lesion area of a skin lesion image, and improve the segmentation accuracy and robustness of the skin lesion image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image segmentation, and more particularly to a skin lesion image segmentation method based on feature interaction fusion. BACKGROUND

[0002] With the continuous rise of skin cancer and other skin lesions, accurate identification and segmentation of skin lesion areas are of great significance for early detection and treatment of diseases. However, skin lesion images usually have the following challenges: first, the boundary of the lesion area is fuzzy and the shape is complex, which is easy to be confused with the surrounding skin tissue; second, the size of the lesion varies greatly, and traditional segmentation methods are difficult to consider multi-scale features; third, the image contrast is low and the noise is more, which brings great difficulty to automatic segmentation.

[0003] At present, the automatic segmentation method based on convolutional neural network (CNN) performs excellently in extracting local features, but due to its inherent local receptive field limitation, it is difficult to capture the global context information in skin lesion images, resulting in deficiencies in boundary segmentation and semantic understanding. At the same time, the Transformer structure has made remarkable progress in natural language processing and computer vision in recent years, and with its powerful global modeling capability, it provides a new idea for medical image segmentation. However, the Transformer has certain limitations in processing fine-grained features, and it is difficult to accurately depict the local details of the lesion. Therefore, the fusion of the advantages of CNN and Transformer becomes an ideal solution.

[0004] However, the current mainstream fusion strategy mostly adopts a simple feature concatenation or addition method, which fails to fully exploit the complementary relationship and synergistic potential between the two. In addition, the existing attention mechanism, while improving performance, often introduces a large amount of computational overhead, and usually only focuses on a single scale or global context, lacking comprehensive modeling capability for multi-scale features. These technical bottlenecks seriously restrict the segmentation accuracy and clinical application value of the model in complex skin lesion scenarios.

[0005] Therefore, there is an urgent need for a new image segmentation method that can efficiently fuse local and global features, have multi-scale modeling capability, and maintain low computational cost, thereby improving the accuracy and robustness of skin lesion area segmentation. SUMMARY

[0006] Therefore, the present application provides a skin lesion image segmentation method based on feature interaction fusion to solve the problems of low segmentation accuracy and weak scale perception in the prior art, thereby improving the recognition and processing capability of complex lesion areas.

[0007] To achieve the above purpose, the present application adopts the following technical solutions:

[0008] The embodiment of the present application provides a skin lesion image segmentation method based on feature interaction fusion, comprising the following steps:

[0009] Image preprocessing and data enhancement: the dermoscope image is subjected to size normalization and noise suppression, and the data set is expanded through random rotation and flipping operation;

[0010] Model construction: a deep segmentation network combining CNN and Transformer is constructed, comprising: a feature extraction double encoder for extracting local features based on a convolutional neural network and global features based on a Transformer; a focused feature interaction fusion module for realizing complementary interaction and fusion between convolutional neural network features and Transformer features; a multi-scale context attention module for introducing multi-scale spatial perception ability and enhancing feature representation; a decoder: through upsampling to restore the resolution, and channel splicing with the encoder features transmitted through the jump connection, after convolution, batch normalization and activation function processing, the segmentation result is reconstructed layer by layer, and finally the segmentation probability map of the skin lesion area is output;

[0011] Model training: the images of the data set are input into the deep segmentation network, the network weight is optimized by using the joint loss function of Dice loss and binary cross entropy loss, and the early stopping strategy is combined to prevent overfitting;

[0012] Segmentation prediction: the dermoscope image to be segmented is input into the trained deep segmentation network, and the segmentation mask map of the skin lesion area is output.

[0013] In one embodiment, in the step of model construction, the feature extraction double encoder comprises a SwinTransformer encoder and a U-Net encoder;

[0014] The Swin Transformer encoder comprises four layers, each layer is subjected to feature extraction by a Swin Transformer block, and the output feature dimensions are 、 、 、 respectively, for capturing long-distance dependency relationships of the lesion;

[0015] The U-Net encoder comprises five layers, each layer is composed of a convolutional layer and a pooling layer, and the output feature dimensions are 、 、 、 、 respectively, for extracting local detailed features of the lesion.

[0016] In an embodiment, in the step of model construction, the multi-scale context attention module receives the features extracted by the first layer convolution of the U-Net encoder, and performs multi-scale spatial information extraction and fusion.

[0017] The focus feature interaction fusion module receives the features extracted by the second to fifth layers of the U-Net encoder and the features extracted by the first to fourth layers of the Swin-Transformer encoder, and realizes layer-by-layer fusion.

[0018] In an embodiment, the focus feature interaction fusion module performs the following operations:

[0019] Feature focusing: enhance features through the SE module, first perform 1x1 convolution channel matching on the Swin Transformer encoder feature map; and perform channel attention enhancement on the feature map from the U-Net encoder;

[0020] Feature interaction: generate a reverse attention mask, cross-weight to supplement missing information;

[0021] Feature fusion: concatenate the interaction features and the directly added features, and output through a 1x1 convolution.

[0022] In an embodiment, in the step of feature focusing, the Swin Transformer encoder feature map is matched in channel number through a 1x1 convolution layer, and then input to the SE module to enhance its effective features, as follows:

[0023] ;

[0024] For the feature map from the U-Net encoder , it is directly input to the SE module for feature enhancement, and the processing process is as follows:

[0025] ;

[0026] wherein, represents a 1x1 convolution operation, represents an SE module, represents a Sigmoid activation function, represents a channel-wise element multiplication operation.

[0027] In an embodiment, in the step of feature interaction, the SE module enhanced Transformer features and CNN features are respectively input to the Sigmoid function to generate an attention mask; by taking the complement of the attention mask, the corresponding reverse attention mask is obtained; the reverse attention mask is used to cross-weight the heterogeneous features, and the calculation process is as follows:

[0028] ;

[0029] ;

[0030] wherein, and respectively represent the Transformer feature map and the CNN feature map after passing through the channel attention processing; the symbol represents a Sigmoid activation function, J represents an all-one matrix, and the output is subtracted to form a complementary weight; represents an element-wise multiplication operation.

[0031] In an embodiment, in the feature fusion step, the Transformer feature map and the CNN feature map after cross-feature weighting processing are element-wise added and fused, and a 1x1 convolution is performed to generate a preliminary fusion feature as the first group of fusion features;

[0032] The Transformer feature map and the CNN feature map after focus enhancement are also element-wise added to obtain the second group of fusion features;

[0033] The first group of fusion features and the second group of fusion features are spliced, and a 1x1 convolution is performed to adjust the channel number to generate a fusion output feature map ; the expression is as follows:

[0034] ;

[0035] wherein, represents a feature splicing operation.

[0036] In an embodiment, the multi-scale context attention module performs the following operations:

[0037] Feature division: receiving an input feature map , and dividing it into four sub-feature maps , , and in the channel dimension, wherein each sub-feature map has a size of ;

[0038] Multi-scale feature extraction: through four parallel branches, different scales of image features are captured on the four sub-feature maps through different receptive field structures;

[0039] Feature fusion and output: the feature maps output by the four parallel branches are spliced in the channel dimension to form a fused feature map, and finally an output feature map is obtained 、 、 and In the channel dimension, a fused feature map is formed, and finally an output feature map is obtained .

[0040] In one embodiment, in the step of multi-scale feature extraction:

[0041] The first branch uses point-wise convolution to extract fine features from the input features;

[0042] The second branch uses channel-wise convolution and point-wise convolution operations to perform local feature extraction on the input features;

[0043] The third branch uses channel-wise convolution, channel-wise atrous convolution, and point-wise convolution to extract medium-range context information from the input features;

[0044] The fourth branch first down-samples the input features to extract important features while preserving their location information; then, through channel-wise convolution, channel-wise atrous convolution, and point-wise convolution, it realizes large-scale modeling, and finally up-samples the feature map back to the original size through the location information.

[0045] According to the technical solution described above, compared with the prior art, the present application has the following technical advantages:

[0046] Firstly, the present application effectively realizes the deep complementary fusion of CNN and Transformer features by introducing a focus feature interaction fusion module. This module preserves the local details of the image while strengthening the global semantic information, significantly improving the model's performance in handling images with fuzzy boundaries and complex structures. Secondly, the present application uses a multi-scale context attention module, which can accurately capture multi-scale information from small lesions to large-scale structures while maintaining low computational cost, thereby improving the model's adaptability to targets of different sizes. In addition, the entire model structure design focuses on lightweight and efficiency, making it easy to deploy in clinical auxiliary diagnosis systems or embedded devices, and has good practical application feasibility. Finally, through experiments on multiple mainstream skin disease image datasets, the method significantly outperforms existing technologies in segmentation accuracy, fully demonstrating its effectiveness and practical value. BRIEF DESCRIPTION OF DRAWINGS

[0047] ​​​​In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only constitute a part of the embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0048] Figure 1 The skin lesion image segmentation method based on feature interaction fusion provided by the present application is shown in the flow chart.

[0049] Figure 2 The overall network architecture of the skin lesion image segmentation model provided by the present application is shown in the schematic diagram.

[0050] Figure 3 The structure schematic diagram of the focused feature interaction fusion module provided by the present application is shown in the schematic diagram.

[0051] Figure 4 The structure schematic diagram of the multi-scale context attention module provided by the present application is shown in the schematic diagram. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0053] Referring to Figure 1 The embodiments of the present application disclose a skin lesion image segmentation method based on feature interaction fusion, which comprises the following steps S1-S4 for convenience of description:

[0054] S1: image preprocessing and data enhancement: normalizing the size of the dermoscope image and suppressing noise, and expanding the data set through random rotation and flipping operations;

[0055] This step is used to construct a data set. First, the collected original skin lesion image is standardized, including size unification and noise suppression operations, to ensure the consistency of the input data. Then, the data set is expanded using data enhancement techniques to enhance the generalization ability of the model. The enhancement means includes random rotation and flipping to form diversified data samples for robustness improvement in the training phase.

[0056] S2: model construction: constructing a deep segmentation network that combines CNN and Transformer, as shown in Figure 2 The model construction includes:

[0057] The feature extraction double encoder is used for extracting local features based on a convolutional neural network and global features based on a Transformer respectively.

[0058] The focused feature interaction fusion module is used for realizing complementary interaction and fusion between the convolutional neural network features and the Transformer features.

[0059] The multi-scale context attention module is used for introducing multi-scale spatial perception ability and enhancing feature representation.

[0060] The decoder: through upsampling to restore the resolution, and channel splicing with the encoder features transmitted by the jump connection, after convolution, batch normalization and activation function processing, the skin lesion image segmentation result is reconstructed layer by layer, and finally the segmentation probability map of the skin lesion area is output.

[0061] The application designs a deep segmentation network based on a CNN and a Transformer fusion structure, wherein the feature extraction double encoder comprises a Swin Transformer encoder and a U-Net encoder, which respectively extract features.

[0062] The Swin Transformer encoder comprises four layers, each of which extracts features by a Swin Transformer block. The input image first enters a linear embedding layer as an initial processing layer of the Swin Transformer encoder, divides the image into non-overlapping image blocks, and converts the pixel value of each image block into an embedding vector through linear projection. The output feature dimensions of the four-layer structure are respectively , the second layer , the third layer , the fourth layer , which are used for capturing long-distance dependency relationships of the lesion.

[0063] The U-Net encoder comprises five layers, each of which is composed of a convolutional layer and a pooling layer, and the feature dimensions output by the U-Net encoder are respectively , the first layer is , the second layer is , the third layer is , the fourth layer is , and the fifth layer is , which are used for extracting local detailed features of the lesion.

[0064] It is emphasized that the features extracted by the first layer convolution of the U-Net encoder are input into the multi-scale context attention module, and through the extraction and fusion of multi-scale spatial information, the scale perception from local to global is realized under the premise of keeping low computational complexity, and the recognition ability of the model to different size lesion regions is enhanced. Subsequently, the features extracted by the second to fifth layers of the U-Net encoder are input into the focus feature interaction fusion module together with the features extracted by the first to fourth layers of the Swin-Transformer encoder, so as to realize layer-by-layer fusion. The decoder gradually restores the detail resolution of the skin lesion image through upsampling and skip connection. Finally, the network outputs a probability map through the Sigmoid activation function, and a binary skin lesion region segmentation mask is finally generated through threshold processing.

[0065] S3: Model training: input the images of the data set (using dermoscopy images with real masks) into the deep segmentation network, and optimize the network weight by using the joint loss function of Dice loss and binary cross entropy loss, combined with the early stopping strategy to prevent overfitting.

[0066] After completing the model structure design, the preprocessed and enhanced images are input into the constructed network model for training. The model training adopts the Adam optimizer, the initial learning rate is set to 0.00001, the Dice loss and cross entropy loss are jointly optimized to ensure that the model has good discrimination ability at the pixel level. At the same time, the early stopping mechanism is introduced to prevent model overfitting.

[0067] S4: Segmentation prediction: input the dermoscopy image to be segmented into the trained deep segmentation network, and output the segmentation mask of the skin lesion region.

[0068] The trained network model is used for actual dermoscopy image segmentation. After inputting the image to be detected, the model can automatically output the segmentation result of the lesion region. The output is a mask image with the same size as the original image. This process does not require human intervention and can be widely used in skin disease early auxiliary diagnosis systems.

[0069] The focus feature interaction fusion module designed in the application realizes the deep complementary fusion of CNN and Transformer features; the multi-scale context attention module is constructed to realize multi-scale accurate feature extraction by extracting local to global scale spatial information while keeping the computational efficiency. The application can solve the problems of complex shape, variable size and low contrast of skin lesion image lesion regions, and improve the segmentation accuracy and robustness of skin lesion images.

[0070] Further, the focus feature interaction fusion module in step S2 above has the structure as shown in Figure 3As shown, to realize the deep complementary fusion of CNN and Transformer features, it can be described as the following 3 steps:

[0071] 1. Focus on key features. This step is used to receive feature maps from the Transformer encoder and the CNN encoder respectively. First, the feature map output by the Transformer encoder is subjected to 1x1 convolution to match the number of channels, and then both the feature map of the CNN encoder and the feature map of the Transformer encoder are input into the SE module to extract key channel information, thereby obtaining the Transformer feature map and the CNN feature map respectively which are enhanced by channel attention.

[0072] Specifically, to suppress redundant information and highlight key information, the SE module is introduced in the focus feature interaction fusion module of the present application to strengthen the discriminative feature channels in the Transformer and CNN branches, and to realize the extraction of key features. The process equation is as follows:

[0073] (1) ;

[0074] For the feature map from the CNN encoder, it is directly input into the SE module for feature enhancement, and the processing process is as follows:

[0075] (2) ;

[0076] In these two formulas, represents the feature map from the Transformer, represents the feature map output from the CNN branch (U-Net encoder). represents the 1x1 convolution operation, represents the SE module, represents the Sigmoid activation function, represents the element-by-channel multiplication operation.

[0077] 2. Recall the ignored details. This step is used to generate a reverse attention mask, which can be used to highlight important areas that the current encoder has failed to pay attention to. The reverse attention mask is generated using the feature map of the other encoder, and the feature map of the current encoder is cross- weighted based on the reverse attention mask to supplement the key information that the respective encoder has missed in the feature expression process.

[0078] Specifically, to further realize the complementary fusion between features, the focus feature interaction fusion module of the present application proposes a reverse attention mechanism to make full use of the complementarity of Transformer and CNN in local and global feature modeling. The method is as follows: first, the SE module enhanced Transformer features and CNN features are respectively input into the Sigmoid function to generate attention masks. By taking the complement of the attention mask, the corresponding reverse attention mask is obtained. Then, the reverse attention is used to cross-weight the heterogeneous features, thereby realizing the complementary recovery of detailed information, and the calculation process is as follows:

[0079] (3);

[0080] (4);

[0081] wherein, and respectively represent the Transformer feature map and the CNN feature map after channel attention processing; the symbol represents the Sigmoid activation function, J represents the all-one matrix, and the output is subtracted to form a complementary weight; represents the element-wise multiplication operation.

[0082] 3. Fusion of complementary information.

[0083] Specifically, in the phase of fusion of complementary information, first, the Transformer feature map and the CNN feature map which have undergone cross-feature weighting processing are element-wise added and fused, and a 1×1 convolution is performed to generate preliminary fusion features. At the same time, the focus enhanced Transformer feature map and the CNN feature map are also element-wise added to obtain another group of fusion features. Then, the two types of features are spliced, the channel number is adjusted through 1×1 convolution to generate a fusion output feature map . The expression is as follows:

[0084] (5);

[0085] wherein, represents the feature splicing operation.

[0086] Further, the multi-scale context attention module of the above step S2 refers to the structure of Figure 4As shown, it is used to process the input feature map to obtain multi-scale feature information, thereby improving the recognition ability of the model for multi-scale lesion targets. This module can balance the efficiency and representation ability of feature extraction, and is suitable for resource-constrained scenarios. The specific implementation is as follows in steps 1) to 3):

[0087] 1) Feature division:

[0088] First, the input feature map is denoted as , where H, W and C represent the height, width and channel number of the feature map, respectively. The feature map is evenly divided into four sub-feature maps in the channel dimension, i.e. , where each sub-feature map .

[0089] 2) Multi-scale feature extraction: Through four parallel branches, different scales of image features are captured on the four sub-feature maps through different receptive field structures;

[0090] The first branch takes the sub-feature map as input, first uses a set of one-dimensional convolution operations corresponding to Figure 4 pointwise convolution (input channel number = C / 4, output channel number = C, convolution kernel size = 1, step = 1) to expand its channel dimension. Then, batch normalization (BN) and GeLU activation functions are applied in turn to enhance the non-linear feature expression ability. Finally, another set of pointwise convolution operations with parameters (input channel number = C, output channel number = C / 4, convolution kernel size = 1, step = 1) are applied to compress the channels, and the output features are normalized by Sigmoid activation, and then element-wise multiplied with the original input to obtain the final output of the branch .

[0091] The second branch processes the sub-feature map , which specifically includes a per-channel convolution operation to avoid cross-channel calculation overhead, with parameters input channel number = C / 4, output channel number = C / 4, convolution kernel size = 3, step = 1, padding = 1, and group number = C / 4. Next, after further processing by batch normalization (BN) and GeLU activation function, a pointwise convolution (input channel number = C / 4, output channel number = C / 4, convolution kernel size = 1, step = 1) is applied to integrate the information between channels. Finally, after processing by the Sigmoid function, element-wise multiplication is performed with the original input to form the branch output .

[0092] The third branch takes the sub-feature map as input to obtain medium-dimensional feature information. For The following operations are sequentially applied: the first step uses the per-channel convolution parameters (input channel number = C / 4, output channel number = C / 4, convolution kernel size = 5, step = 1, padding = 2, and group number = C / 4); the second step uses the per-channel atrous convolution parameters (input channel number = C / 4, output channel number = C / 4, convolution kernel size = 7, step = 1, padding = 9, group number = C / 4, and dilation coefficient = 3). After the above two steps, point-wise convolution (input channel number = C / 4, output channel number = C / 8, convolution kernel size = 1, and step = 1) is performed in the channel dimension to splice and integrate cross-channel information. Finally, Sigmoid activation is performed, and the output of the branch is obtained by multiplying the input .

[0093] The fourth branch aims to capture sparse global feature relationships. First, the important features are extracted from the by a downsampling operation and are reconstructed into while retaining their original position information for subsequent restoration. The is sequentially subjected to the following operations: the first step uses a group of per-channel convolutions (parameters: input channel number = C / 4, output channel number = C / 4, convolution kernel size = 5, step = 1, padding = 2, and group number = C / 4) to obtain intermediate features ; the second step applies per-channel atrous convolution (input channel number = C / 4, output channel number = C / 4, convolution kernel size = 7, step = 1, padding = 9, group number = C / 4, and dilation coefficient = 3) to obtain ; the third step performs point-wise convolution (input channel number = C / 4, output channel number = C / 8, convolution kernel size = 1, and step = 1) on and respectively, and then splices them in the channel dimension to form a fusion feature; the fourth step up-samples the feature map to the original size through the position information; the fifth step performs Sigmoid activation and then multiplies the original to obtain the final output .

[0094] 3) Feature fusion and output:

[0095] The output features obtained by the above four branches are spliced in the channel dimension to form a fusion feature map, and finally an output feature map is obtained.

[0096] The following network training configuration is adopted in this embodiment: the maximum number of iterations is set to 100, the batch size is set to 16, and the initial learning rate is set to 0.00001. In the design of the loss function, a hybrid form combining binary cross-entropy loss and Dice loss is adopted. In addition, the Adam optimizer is used for back propagation to update the model parameters. During the training process, an early stopping strategy is implemented, and the training is terminated when the performance of the validation set no longer improves, and the current optimal model weight is saved. After iterative optimization, the model gradually learns discriminative feature representations, and finally obtains a skin lesion image segmentation model with optimal performance on the validation set.

[0097] On the ISIC2018 dataset, this embodiment comprehensively compares the performance of the proposed network model with the current mainstream image segmentation models. The models involved in the comparison include: U-Net, U-Net++, Attention U-Net, TransUNet, I2U-Net, and CSWin-UNet. The evaluation indicators include accuracy, precision, recall, Dice coefficient (DSC), and Jaccard Index, to comprehensively measure the performance of each model in the skin lesion image segmentation task. Table 1 shows the comparison results of each model on the ISIC2018 dataset. ISIC2018 is a skin lesion analysis dataset released by the International Skin Imaging Collaboration (ISIC), mainly used to promote the application of machine learning in skin cancer detection.

[0098] Table 1. Skin lesion segmentation performance of different networks on ISIC2018.

[0099] Model Accuracy (%) Precision (%) Recall (%) Dice coefficient (%) Jaccard coefficient (%) U-Net 91.66 85.05 85.15 85.10 74.06 U-Net++ 91.92 84.66 86.96 85.77 75.10 Attention U-Net 92.26 86.48 85.72 86.09 75.59 TransUNet 92.67 87.80 85.89 86.78 76.65 I 2 U-Net 91.71 86.28 83.67 84.95 73.85 CSWin-UNet 92.75 88.91 84.64 86.73 76.56 The invention 93.22 84.73 92.43 88.41 79.22

[0100] As shown in Table 1, the performance of the method of the present application is compared with other methods in terms of performance indicators. It can be clearly found from the table that compared with other methods, the method of the present application generally achieves the best performance on the ISIC2018 skin lesion image dataset. Specifically, the accuracy, precision, recall, Dice coefficient, and Jaccard coefficient of the present application are 93.22%, 84.73%, 92.43%, 88.41%, and 79.22%, respectively.

[0101] To verify the performance improvement effect of the network module of the present application, five groups of ablation experiments are designed in this embodiment: first, a model with only the output feature dimension of the U-Net adjusted is taken as the baseline (subnetwork 1); on this basis, the focus feature interaction fusion module without the feature focusing part (subnetwork 2) and the complete focus feature interaction fusion module (subnetwork 3) are added respectively to evaluate the independent contribution of the interaction structure and the feature focusing mechanism to the performance; then, the multiscale context attention module is integrated on the subnetwork 1 (subnetwork 4) to test the independent effect of the attention mechanism; finally, the focus feature interaction fusion module and the scale-aware attention module are introduced at the same time (subnetwork 5) to form a complete model. By comparing the experimental results of each subnetwork, the contribution of the two modules and their combination to the performance improvement is systematically evaluated, thereby verifying the rationality and practicability of the design of the present application. Table 2 shows the comparison results of segmentation performance on the ISIC2018 dataset.

[0102] Table 2. Ablation study of key components on ISIC2018 dataset.

[0103] Model Accuracy Precision Recall Dice coefficient (%) Jaccard coefficient (%) Subnetwork 1 91.44 84.62 84.89 84.73 73.51 Subnetwork 2 92.52 83.74 91.00 87.19 77.30 Subnetwork 3 92.63 82.97 92.81 87.58 77.91 Subnetwork 4 92.58 87.68 85.49 86.57 76.32 Subnetwork 5 93.22 84.73 92.43 88.41 79.22

[0104] The experimental data in the table clearly show that enhancing a module alone can bring certain performance improvement, and the effective combination of each module also shows the optimal overall performance.

[0105] Finally, the skin lesion image to be processed can be input into the skin lesion image segmentation model, and the segmentation result of the corresponding skin lesion image is output.

[0106] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between each embodiment can be referred to each other. For the device disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0107] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A skin lesion image segmentation method based on feature interaction fusion, characterized in that, The method comprises the following steps: Image preprocessing and data enhancement: normalizing the size and suppressing the noise of the dermoscopy image, and expanding the data set through random rotation and flipping operations; Model construction: constructing a deep segmentation network combining CNN and Transformer, including: a feature extraction double encoder for extracting local features based on a convolutional neural network and global features based on a Transformer; a focused feature interaction fusion module for realizing complementary interaction and fusion between convolutional neural network features and Transformer features; a multi-scale context attention module for introducing multi-scale spatial perception ability and enhancing feature representation; a decoder: restoring the resolution through upsampling, and channel splicing with the encoder features transmitted through the skip connection, and then processing through convolution, batch normalization and activation function to reconstruct the segmentation result layer by layer, and finally outputting the segmentation probability map of the skin lesion area; Model training: inputting the images of the data set into the deep segmentation network, and optimizing the network weights using the joint loss function of Dice loss and binary cross entropy loss, and combining the early stopping strategy to prevent overfitting; Segmentation prediction: inputting the dermoscopy image to be segmented into the trained deep segmentation network to output the segmentation mask of the skin lesion area; The multi-scale context attention module performs the following operations: Feature division: receive input feature map , and divide it into four sub-feature maps on the channel dimension , , and , where the size of each sub-feature map is ; Multi-scale feature extraction: four parallel branches are used to capture image features of different scales on the four sub-feature maps through different receptive field structures; Feature fusion and output: the feature maps output by the four parallel branches are spliced in the channel dimension to form a fused feature map, and finally an output feature map is obtained , , and In the channel dimension, the feature maps output by the four parallel branches are spliced to form a fused feature map, and finally an output feature map is obtained ; In the step of multi-scale feature extraction: The first branch uses point-wise convolution for processing the input features extracting fine features; The second branch is used for local feature extraction on the input features through a channel-wise convolution and a point-wise convolution operation ; The third branch extracts medium-range context information by applying a channel-wise convolution, a channel-wise atrous convolution, and a point-wise convolution on the input features ; The fourth branch first processes the input features Down-sampling extracts important features while preserving their location information; then, large-scale modeling is achieved through channel-wise convolution, channel-wise dilated convolution, and point-wise convolution, and the features are up-sampled back to the original size through location information.

2. The skin lesion image segmentation method based on feature interaction fusion according to claim 1, characterized in that, In the step of model construction, the feature extraction double encoder includes a Swin Transformer encoder and a U-Net encoder; The Swin Transformer encoder comprises a four-layer structure, each layer is subjected to feature extraction by a Swin Transformer block, and the output feature dimensions are , , , for capturing long-distance dependencies of the lesion. The U-Net encoder comprises a five-layer structure, each layer being composed of a convolutional layer and a pooling layer, and the output feature dimensions are 64, 128, 256, 512 and 1024 respectively , , , , for extracting local detailed features of the lesion.

3. The skin lesion image segmentation method based on feature interaction fusion according to claim 2, characterized in that, In the step of model construction, the multi-scale context attention module receives the features extracted by the first layer convolution of the U-Net encoder for multi-scale spatial information extraction and fusion; The focused feature interaction fusion module receives the features extracted by the second to fifth layers of the U-Net encoder and the features extracted by the first to fourth layers of the Swin-Transformer encoder for layer-by-layer fusion.

4. The skin lesion image segmentation method based on feature interaction fusion according to claim 3, characterized in that, The focused feature interaction fusion module performs the following operations: Feature focusing: enhancing the features through an SE module, performing 1x1 convolution channel matching on the Swin Transformer encoder feature map, and performing channel attention enhancement on the feature map from the U-Net encoder; Feature interaction: generating a reverse attention mask to cross-weight and supplement missing information; Feature fusion: concatenating the interaction features and the direct addition features, and outputting through 1x1 convolution.

5. The skin lesion image segmentation method based on feature interaction fusion according to claim 4, characterized in that, In the feature focusing step, the feature map from the Swin Transformer encoder The channel number is matched by a 1x1 convolution layer, and then input into the SE module to enhance the effective features, as follows: ； For feature maps from the U-Net encoder , the SE module is directly inputted for feature enhancement, and the processing process is as follows: ； wherein, denotes a 1 x 1 convolution operation, denotes an SE module, denotes a Sigmoid activation function, denotes an element-wise multiplication operation per channel.

6. The skin lesion image segmentation method based on feature interaction fusion according to claim 5, characterized in that, In the feature interaction step, the Transformer features and CNN features enhanced by the SE module are input into the Sigmoid function to generate an attention mask; the corresponding reverse attention mask is obtained by taking the complement of the attention mask; the reverse attention mask is used to cross-weight the heterogeneous features, and the calculation process is as follows: ； ； wherein, and respectively represent the transformed feature map and the CNN feature map after the channel attention processing; the symbol represents a Sigmoid activation function, J represents an all-one matrix, and the output is subtracted to form complementary weights; represents an element-wise multiplication operation.

7. The skin lesion image segmentation method based on feature interaction fusion according to claim 6, characterized in that, In the feature fusion step, the cross-feature weighted processed Transformer feature map is element-wise added with the CNN feature map to generate a preliminary fusion feature through 1x1 convolution as the first group of fusion features. Focusing enhanced transformer feature map with CNN feature map Similarly, element-wise addition is performed to obtain a second set of fused features; The first group of fusion features and the second group of fusion features are spliced, and a fusion output feature map is generated by adjusting the number of channels through 1*1 convolution ; the expression is as follows: ； wherein represents a feature stitching operation.

Images