A multi-attention encoder-decoder network and system applied to lesion segmentation

Abstract

The application discloses a kind of multi-attention encoder-decoder network and system applied to skin lesion segmentation, comprising: using pyramid visual transformer (PVT) as feature extraction backbone network.Then join multiple attention fusion (MAF) method in the deep layer of network, to further tap the rich semantic information obtained in the encoder stage, and preliminarily locate the lesion area.Between the encoder and decoder, use selective information aggregation (SIG) method instead of traditional skip connection, to alleviate the semantic difference between the encoder and decoder, and obtain more efficient skip connection.Finally, adopt multi-scale cascaded fusion (MSCF) method, dynamically fuses the features of different stages of decoder.The application has the advantages that: improve the accuracy of segmentation results, speed up the network training speed and improve efficiency, with good flexibility, adaptability and universality.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and image processing technology, and in particular to a skin lesion segmentation method using a multi-attention codec network (MASDF-Net) and system with selective and dynamic fusion capabilities. Background Technology

[0002] Skin cancer is one of the most common malignant tumors worldwide, accounting for 1.79% of the global disease burden. Melanoma is considered the deadliest form of skin cancer, accounting for 75% of skin cancer deaths. Fortunately, if this type of skin cancer is detected early and treated effectively, the survival rate can exceed 99%. Dermoscopy is one of the fundamental methods for diagnosing melanoma; however, the manual examination by dermatologists based on dermoscopic images is often time-consuming and requires a high level of skill and attention, making it prone to operator error. To assist dermatologists in addressing these issues, especially when dealing with a large number of patients in a short period, computer-aided diagnostics (CAD) has become an effective tool for dermatologists' decision-making. Automatic segmentation of dermoscopic images is a crucial step in CAD. However, this task is quite challenging due to the complexity and variability of lesion areas. First, lesion areas are often irregular in shape and size, with uneven color distribution. Second, in the early stages of the disease, the contrast between the lesion area and the surrounding skin is low, resulting in blurred boundaries. Furthermore, the lesion area may also be obscured by artifacts such as hair and air bubbles. Therefore, developing an algorithm that can automatically and accurately segment skin lesions is of great significance in assisting dermatologists in clinical diagnosis.

[0003] Traditional skin lesion segmentation algorithms, such as thresholding, region merging, and active contouring, are not only computationally complex but also lack robustness, making them ill-suited for complex skin lesion scenarios. In contrast, deep learning models based on convolutional neural networks (CNNs) can adaptively learn the boundary features of skin lesion regions and outperform traditional solutions in both performance and accuracy. However, classic CNN architectures cannot achieve pixel-level predictions until fully convolutional neural networks (FCNs) were developed. [1] The emergence of DeepLabv3+ replaced the final fully connected layer with convolutional layers, enabling end-to-end, pixel-to-pixel training and achieving better semantic segmentation. Due to the excellent performance of FCN in image segmentation tasks, numerous FCN-based encoder-decoder networks have been proposed. For example, DeepLabv3+... [2] Asymmetric networks, represented by SegNet, and others [3] and U-Net [4]Symmetric networks, represented by U-Net, are the most widely used network in the field of medical image segmentation. Various variants derived from U-Net, such as Attention-UNet, UNet++, and Double-UNet, have attracted widespread attention from researchers.

[0004] Since its release in 2015, U-Net has experienced explosive growth in medical image segmentation applications. The basic structure of U-Net consists of an encoder and a decoder. The encoder, similar to a convolutional network, typically comprises multiple convolutional and pooling layers and is responsible for extracting high-level semantic features from the input image. The decoder is responsible for remapping the abstract semantic features extracted by the encoder back to the input image space, restoring pixel-level detail. Unlike FCNs, U-Net uses skip connections to fuse features from both the encoder and decoder, thus preserving more detail and boundary information.

[0005] Although U-Net-based skin lesion segmentation methods have demonstrated their superiority, most current methods still suffer from the following problems due to the complexity of dermoscopy images:

[0006] (1) Due to the inherent locality of convolution operations, the network cannot capture long-distance dependencies.

[0007] (2) Continuous downsampling operations will lead to the loss of high-level feature location information and the loss of global dependencies between pixels.

[0008] The introduction of skip connections may lead to an overemphasis on low-level features while neglecting higher-level semantic information, thus affecting the performance of the model.

[0009] Definitions of abbreviations and key terms

[0010] CAD (Computer-aided Diagnosis)

[0011] CNN (Convolutional Neural Network)

[0012] PVT (Pyramid Vision Transformer)

[0013] MAF (Multi-attention fusion)

[0014] SIG (Selective Information Gathering)

[0015] MSCF (Multi-scale Cascade Fusion)

[0016] References

[0017] [1] E. Shelhamer, J. Long, T. Darrell, Fully convolutional networks for semantic segmentation, IEEE Transactions on Pattern Analysis and Machine Intelligence 39(4)(2017) 640–651;

[0018] [2] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, H. Adam, Encoder-decoder with atrous separable convolution for semantic image segmentation, in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 833–851;

[0019] [3] V. Badrinarayanan, A. Kendall, R. Cipolla, SegNet: A deep convolutional encoder-decoder architecture for image segmentation, IEEE Transactions on Pattern Analysis and Machine Intelligence 39(12)(2017) 2481–2495;

[0020] [4] O. Ronneberger, P. Fischer, T. Brox, U-Net: Convolutional networks for biomedical image segmentation, in: Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015, 2015, pp. 234–241;

[0021] [5]W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, L. Shao, Pyramid vision transformer: A versatile backbone for dense prediction without convolutions, in: Proceedings of the IEEE / CVF International Conference on Computer Vision, 2021, pp. 568–578;

[0022] [6]Y. Cao, J. Xu, S. Lin, F. Wei, H. Hu, GCNet: Non-local networks meetsqueeze-excitation networks and beyond, in: 2019 IEEE / CVF International Conference on Computer Vision Workshop(ICCVW), 2019, pp. 1971–1980;

[0023] [7]S. Woo, J. Park, J.-Y. Lee, I.S. Kweon, CBAM: Convolutional block attention module, in: Computer Vision–ECCV 2018, 2018, pp. 3–19;

[0024] [8]X. Wang, R. Girshick, A. Gupta, K. He, Non-local neural networks, in: 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 7794–7803;

[0025] [9]Z. Huang, X. Wang, L. Huang, C. Huang, Y. Wei, W. Liu, CCNet: Criss-cross attention for semantic segmentation, in: 2019 IEEE / CVF International Conference on Computer Vision(ICCV), 2019, pp. 603–612. Summary of the Invention

[0026] This invention addresses the shortcomings of existing technologies by providing a multi-attention encoder-decoder network and system for lesion segmentation. It improves upon the traditional U-shaped network. In the encoder stage, MASDF-Net uses a Pyramid Visual Transformer (PVT) as the feature extraction backbone, replacing the convolutional layers in U-Net. This allows for better modeling of long-range dependencies between lesion locations and the extraction of more semantically informative features. In deeper layers, MASDF-Net introduces a Multiple Attention (MAF) mechanism to deeply mine the rich semantic information from the encoder stage and initially locate the lesion region. Because of the semantic differences between low-level features in the encoder stage and high-level features in the decoder stage, traditional skip connections introduce irrelevant noise. To address this, MASDF-Net designs a Selective Information Aggregation (SIG) method based on cross-attention to obtain more efficient skip connections and reduce the impact of irrelevant noise. In the decoder stage, feature maps at different levels contain different contextual information. Low-resolution features contain more semantic information, while high-resolution features contain more location and boundary information. Therefore, MASDF-Net employs a Multi-Scale Cascaded Fusion (MSCF) method to dynamically fuse features from different stages of the decoder to accurately locate the lesion boundaries. With the above improvements, MASDF-Net can segment skin lesions more accurately, improving the accuracy and efficiency of segmentation.

[0027] To achieve the above-mentioned objectives, the technical solution adopted by the present invention is as follows:

[0028] A multi-attention encoder-decoder network for skin lesion segmentation includes:

[0029] A Pyramid Visual Transformer (PVT) was used as the backbone network for feature extraction. Then, a Multiple Attention Fusion (MAF) method was incorporated into the deeper layers of the network to further mine the rich semantic information obtained in the encoder stage and to initially locate the lesion region. Between the encoder and decoder, Selective Information Aggregation (SIG) was used instead of traditional skip connections to alleviate the semantic differences between the encoder and decoder and to obtain more efficient skip connections. Finally, a Multi-Scale Cascaded Fusion (MSCF) method was employed to dynamically fuse features from different stages of the decoder to accurately locate the lesion boundary.

[0030] Furthermore, the MAF method fuses features using a multilayer perceptron (MLP) combined with a spatial attention map to extract richer contextual information. The specific steps are as follows:

[0031] 1) Perform two parallel pooling operations on the input feature X: global average pooling and global max pooling, and use a 1×1 convolution operation to generate a two-dimensional spatial attention map S.a (X). S a (X) can capture the spatial dependencies of input features.

[0032] 2) Transfer the spatial attention map S a Transpose (X) and multiply it with the input feature X to obtain the feature weighted by spatial attention.

[0033] 3) Input weighted features into the MLP to further enhance the expressive power of the features.

[0034] 4) Finally, the output of the MLP is weighted and fused with the input features to obtain the final output feature Y. This enhances global contextual information and suppresses irrelevant information.

[0035] Furthermore, the specific steps of MAF are summarized as follows:

[0036] S a (X)=Softmax((f 1×1 ([P avg (X); P max (X)])) T (1)

[0037]

[0038] Among them, S a (·) represents spatial attention operation, f 1×1 P represents a convolution operation with a kernel size of 1×1. avg (·) and P max (·) represents global average pooling and global max pooling, respectively. represents matrix multiplication, and T represents matrix transpose.

[0039] Furthermore, the SIG method employs a cross-attention mechanism to selectively aggregate information between the encoder and decoder, thereby fully utilizing the positional information of low-level features and the semantic information of high-level features. The specific steps are as follows:

[0040] 1) Given the symmetric feature maps F and D of the encoder and decoder, generate feature map Q on F and feature maps K and V on D using 1×1 convolutions.

[0041] 2) Perform affinity operations on feature maps Q and K to generate an attention weight map. This weight map measures the correlation between each pixel in the encoder and decoder.

[0042] 3) Aggregate the attention weight map with the feature map V to obtain the output of the first stage. In this way, each pixel in the encoder can collect information from the corresponding horizontal and vertical positions in the decoder.

[0043] 4) Two loops are set up so that the encoder can indirectly obtain the complete context information in the decoder.

[0044] 5) Finally, the output of the first stage is spliced ​​with the feature map D of the decoder on the channel to complete the selective information aggregation operation.

[0045] Furthermore, the MSCF method effectively integrates features at different levels through cascaded modules and scale-aware modules. The specific steps are as follows:

[0046] 1) Features of different resolutions are processed through cascaded modules. X1 is a feature with higher resolution and detailed spatial location information. X2, X3, and X4 are features with lower resolution but more semantic information. The number of channels in X2, X3, and X4 is adjusted through a series of convolutional units to obtain the corresponding X2′, X3′, and X4′.

[0047] 2) Use a cascade module to perform feature fusion on X2′, X3′, and X4′. The output of the cascade module is defined as X1′=CM(X2′, X3′, X4′), where CM represents the cascade module.

[0048] 3) To dynamically balance the weights between different scales, a scale-aware module is introduced. X1′ and X are concatenated on the channel, and then spatial pixel-level mappings A and A′ are obtained through convolutional layers and softmax layers.

[0049] 4) Perform a weighted summation of the features at the two different scales to obtain the final output feature map X. fusion Among them, X fusion =X1⊙A+X1′⊙B, where ⊙ represents the Adama product.

[0050] This invention also discloses a skin lesion segmentation system based on the above-mentioned multi-attention encoder-decoder network, comprising:

[0051] Input preprocessing module: preprocesses the input image, including resizing and normalization, so that it can be input into the network for processing.

[0052] Pre-backbone network: A pre-trained Pyramid Visual Transformer (PVT) is used as the pre-backbone network of MASDF-Net to extract image features.

[0053] Multi-scale feature extraction module: Utilizing the features extracted by the pre-backbone network, the multi-scale feature extraction module obtains feature maps with different resolutions and semantics.

[0054] Multi-scale cascaded fusion module: Through the cascaded module and the scale-aware module, features at different scales are fused to obtain a fused output feature map.

[0055] Output prediction module: The fused output feature map is processed using convolutional layers and softmax layers to obtain a pixel-level mapping of skin damage, which is used to represent the location of skin damage.

[0056] Post-processing module: Performs post-processing on the output pixel-level mapping, including thresholding and connected component analysis, to obtain the final skin lesion segmentation result.

[0057] Compared with the prior art, the advantages of the present invention are as follows:

[0058] 1. High accuracy: MASDF-Net employs a multi-scale feature extraction and multi-scale cascade fusion mechanism, which can capture feature information of different scales and semantics, thereby improving the accuracy of segmentation results.

[0059] 2. Strong robustness: Through the design of scale-aware modules and cascaded modules, MASDF-Net can effectively handle skin damage of different scales and shapes, and has strong robustness.

[0060] 3. High efficiency: MASDF-Net uses a pre-trained deep convolutional neural network as the backbone network, which can utilize existing large-scale datasets for transfer learning, thereby accelerating network training and improving efficiency.

[0061] 4. Practicality: MASDF-Net is a network model specifically designed for skin lesion segmentation tasks, and can be widely used in practical applications such as medical fields and the diagnosis and treatment of skin diseases.

[0062] 5. Improved skip connection process: MASDF-Net improves the skip connection process in traditional encoder-decoder networks, effectively reducing the influence of irrelevant noise such as hair and artifacts in dermoscopy images and improving the accuracy of segmentation results.

[0063] 6. Dynamic Feature Fusion: MASDF-Net dynamically fuses features from different levels through the MSCF method, which enables the network to accurately locate boundary information even when faced with complex and ambiguous boundaries, thus improving the accuracy of segmentation.

[0064] 7. Plug and play feature: The MASDF-Net method has a plug and play feature, which can be easily applied to existing codec networks without redesigning the entire network structure, thus improving the flexibility and practicality of the method.

[0065] 8. Strong generalization ability: Experiments on four public skin lesion datasets showed that MASDF-Net achieved the best performance in both performance and generalization, indicating that the method has good adaptability and versatility for different datasets and skin lesion types. Attached Figure Description

[0066] Figure 1 This is the overall structure of MASDF-Net in this embodiment of the invention. In the figure, PVT v2 backbone represents the PVTv2 backbone network, downsampling represents downsampling, Conv represents convolution, upsampling represents upsampling, Feature Flow represents feature flow, Bilinear interpolation represents bilinear interpolation, and Conv Layer represents convolutional layer.

[0067] Figure 2 This is a schematic diagram of the multi-attention fusion method according to an embodiment of the present invention. In the diagram, MaxPool represents max pooling, AvgPool represents average pooling, Reshape represents matrix reshaping, Transpose represents matrix transpose, MLP represents multilayer perceptron, Concatenation represents channel concatenation, Softmax is the normalized exponential function, Matrixmultiplication represents matrix multiplication, and Addition represents matrix addition.

[0068] Figure 3 This is a schematic diagram of the selective information aggregation method according to an embodiment of the present invention. In the diagram, Affinity represents the affinity operation, and aggregation represents the aggregation operation.

[0069] Figure 4 This is a schematic diagram of the multi-scale cascaded fusion method according to an embodiment of the present invention. In the diagram, HadamardProduct represents Hadamard product, Concatenation represents channel concatenation, Upsampling represents upsampling, Conv represents convolution, ReLU represents activation function, and BN represents batch normalization. Cascade module represents cascaded module, and Scale-aware module represents scale-aware module.

[0070] Figure 5 This is a performance comparison chart between the MASDF-Net proposed in this embodiment and state-of-the-art methods. The best results are indicated in bold. Params (M) represents the number of parameters, FLOPs (G) represents the number of floating-point operations, Dataset represents the dataset, Type represents different types of methods, CNN represents methods based on convolutional neural networks, and Trans represents networks that use Transformers.

[0071] Figure 6 This is a cross-dataset test graph of the ISIC 2018 and PH2 datasets in an embodiment of the present invention, where ISIC 2018 is used as the training set and PH2 is used as the test set. The best results are indicated in bold.

[0072] Figure 7 This is a graph showing the quantitative results of ablation experiments conducted on the ISIC 2018 dataset according to an embodiment of the present invention. Best results are indicated in bold. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and examples.

[0074] I. Overall Structure of MASDF-Net

[0075] like Figure 1 As shown, the proposed multi-attention encoder-decoder network structure with selective and dynamic fusion is MASDF-Net: Based on the traditional U-Net, it uses the Pyramid Visual Transformer (PVT) as the backbone network for feature extraction, while adding the Multi-Attention Fusion (MAF) method in the deep layers of the network, and using the Selective Information Aggregation (SIG) method to replace skip connections. Finally, it uses the Multi-Scale Cascaded Fusion (MSCF) method to integrate features from different levels of the decoder stage.

[0076] II. Multiple Attention Fusion Methods

[0077] Attention mechanisms can be used to enhance effective features and suppress irrelevant information. However, existing methods typically focus on only one or two attention mechanisms. To extract more contextual information at deeper network layers, GCNet... [6] and CBAM [7] Inspired by [the previous work], this invention proposes a Multiple Attention Fusion (MAF) method. For example... Figure 2 As shown, the input features are processed through two parallel pooling operations and a 1×1 convolution to generate a two-dimensional spatial attention map. This map is then transposed and multiplied with the input features before being fed into a multilayer perceptron (MLP). This results in a one-dimensional channel attention map that captures long-term dependencies and enhances spatial features. Finally, this map is fused with the input features to enhance global contextual information.

[0078] Set the input feature map to X and the output feature map to Y. The specific steps of MAF can be summarized as follows:

[0079] S a (X)=Softmax((f1×1 ([P avg (X); P max (X)])) T (1)

[0080]

[0081] Where S a (·) represents spatial attention operation, f 1×1 P represents a convolution operation with a kernel size of 1×1. avg (·) and P max (·) represents global average pooling and global max pooling, respectively. represents matrix multiplication, and T represents matrix transpose.

[0082] III. Selective Information Aggregation Methods

[0083] like Figure 3 As shown, in a U-shaped network, the encoder part suffers from the loss of positional information due to continuous downsampling. Skip connections can fuse positional information from low-level features with semantic information from high-level features to enrich spatial details. However, due to the semantic differences between the encoder and decoder, simple skip connections can introduce irrelevant noise and ambiguity. To focus on both positional information in low-level features and rich semantic information in high-level features, this invention designs a selective information aggregation method based on cross-attention, as shown in the figure. It is worth noting that the calculation of attention differs from that of conventional non-local blocks. [8] Unlike others, this invention uses CCNet. [9] Cross-attention is used to improve computational efficiency. Given two symmetric feature maps F and D for the encoding and decoding stages, a 1×1 convolution is used to generate feature map Q on F and feature maps K and V on D. An affinity operation is applied to Q and K to generate attention weight maps, which are then aggregated with V to obtain the output of the first stage. Since cross-attention only allows each pixel in F to collect information from the corresponding horizontal and vertical positions in D, two loops are set up so that F indirectly obtains complete contextual information from D. Finally, it is concatenated with D in channels to complete a more efficient skip connection operation.

[0084] IV. Multi-scale Cascade Fusion Method

[0085] like Figure 4 As shown, in order to more effectively integrate features at different levels, a multi-scale cascade fusion (MSCF) method consisting of cascaded modules and scale-aware modules is proposed.

[0086] Cascade module represents a cascaded module, and Scale-aware module represents a scale-aware module.

[0087] X1 has a higher resolution and contains detailed spatial location information. For X2, X3, and X4, which have lower resolution but contain more semantic information, their channel counts are first adjusted through a series of convolutional units to obtain X2′, X3′, and X4′. Feature fusion of X2′, X3′, and X4′ is then performed using the most popular recent practices. The output of the cascaded module is defined as X1′ = CM(X2′, X3′, X4′). To dynamically balance the weights between different scales, a scale-aware module is introduced. Specifically, X1′ and X are concatenated along their channels and fed into convolutional and softmax layers to obtain spatial pixel-level mappings A and A′. Finally, the features at the two different scales are weighted and summed to obtain the final output.

[0088] X fusion =X1⊙A+X1′⊙B

[0089] Among them, X fusion ⊙ represents the fused output feature map, and ⊙ represents the Adama product.

[0090] V. Experiment

[0091] 5.1 Dataset

[0092] To compare the performance of this invention with existing technologies, extensive experiments were conducted on four public skin lesion datasets, including ISIC 2016, ISIC 2017, ISIC 2018, and PH2. Details of these four datasets are as follows:

[0093] The ISIC 2016 dataset contains 1297 JPEG images of dermoscopic lesions and PNG ground truth images (binary mask images), of which 900 images were used for training and 379 images were used for testing.

[0094] ISIC 2017 is an expanded dataset that provides 2000 training sets, 150 validation sets, and 600 test sets.

[0095] The ISIC 2018 dataset consists of 2594 RGB images and their corresponding ground truth images. In the experiments, it was randomly divided into training (70%), validation (10%), and test sets (20%).

[0096] PH2 is a small dataset containing only 200 dermal images, which we use to evaluate the generalization ability of the model.

[0097] 5.2 Training Configuration in this Example

[0098] This embodiment compares the proposed MASDF-Net with 10 state-of-the-art medical image segmentation networks, including: U-Net, AttU-Net, Deeplabv3+, CE-Net, CPFNet, MSCA-Net, Swin-Unet, TransFuse, UCTransNet, and Polyp-PVT. The first six are pure CNN-based networks, while the latter four incorporate Transformers. To ensure fairness, all networks were retrained under the same experimental conditions. Furthermore, to comply with the ISIC challenge requirements, the Jaccard index (JI) was used as the most important metric for evaluating network segmentation performance, followed by Dice Scoring Coefficient (DSC), Accuracy (ACC), Sensitivity (SE), and Specificity (SP). Figure 5 The proposed MASDF-Net shows the best segmentation performance on all three datasets. Figure 6 It can be seen that the proposed MASDF-Net also performs best in terms of generalization.

[0099] 5.3 Training Configuration in this Example

[0100] The proposed MASDF-Net is implemented in the PyTorch library and utilizes an NVIDIA GeForce RTX 3070 graphics card for computational acceleration. End-to-end training is performed using the Adam optimizer. The learning rate is set to 0.0001, the batch size to 16, and the maximum number of training iterations to 100. The model with the highest Jaccard index score on the validation set is used for evaluation, and its performance on the test set is also tested. Furthermore, all images are scaled to 224×224 and data augmentation operations, such as random rotation, horizontal inversion, and color jitter, are performed before being input into the model.

[0101] 5.3 Ablation Experiment

[0102] To verify the effectiveness of the key method proposed in MASDF-Net of this invention, a series of ablation experiments were conducted on the ISIC 2018 dataset. The following models were mainly compared:

[0103] Baseline: Consists of a U-Net with PVT v2 as the encoder.

[0104] Model 1: Baseline + MAF method.

[0105] Model 2: Baseline + SIG method.

[0106] Model 3: Baseline + MAF method.

[0107] Model 4: Baseline + SIG + MAF method.

[0108] Model 5: Baseline + MSCF + MAF method.

[0109] Model 6: Baseline + SIG + MSCF method.

[0110] Model 7: (MASDF-Net): Baseline + MAF + SIG + MSCF method.

[0111] Figure 7 Quantitative results from ablation experiments are provided. Compared to the baseline, models 1-3 show JI improvements of 1.02%, 1.27%, and 1.38%, respectively, confirming the effectiveness of the MAF, SIG, and MSCF methods in improving segmentation performance. Notably, model 2 achieves a significant performance improvement compared to the baseline, with only an increase of 0.15M parameters and 0.11G FLOPs. This observation reflects the advantages of the SIG module in terms of computational burden and memory consumption. Furthermore, it can be observed that models 4-7 achieve further performance enhancements by incorporating other modules besides models 1-3. This observation highlights the synergistic effect of the MAF, SIG, and MSCF methods, indicating that the strong performance of the proposed MASDF-Net is attributed to the synergistic effect among these modules.

[0112] In another embodiment of the present invention, a skin lesion segmentation system based on a multi-attention encoder-decoder network is provided, comprising:

[0113] Input preprocessing module: preprocesses the input image, including resizing and normalization, so that it can be input into the network for processing.

[0114] Pre-backbone network: A pre-trained Pyramid Visual Transformer (PVT) is used as the pre-backbone network of MASDF-Net to extract image features.

[0115] Multi-scale feature extraction module: Utilizing the features extracted by the pre-backbone network, the multi-scale feature extraction module obtains feature maps with different resolutions and semantics.

[0116] Multi-scale cascaded fusion module: Through the cascaded module and the scale-aware module, features at different scales are fused to obtain a fused output feature map.

[0117] Output prediction module: The fused output feature map is processed using convolutional layers and softmax layers to obtain a pixel-level mapping of skin damage, which is used to represent the location of skin damage.

[0118] Post-processing module: Performs post-processing on the output pixel-level mapping, including thresholding and connected component analysis, to obtain the final skin lesion segmentation result.

[0119] In another embodiment of the present invention, a terminal device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for skin lesion segmentation operations.

[0120] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory). This computer-readable storage medium is a memory device in a terminal device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and extended storage media supported by the terminal device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device.

[0121] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the instructions related to lesion segmentation in the above embodiments.

[0122] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0123] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0124] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0125] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0126] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the implementation methods of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of the present invention.

Claims

1. A lesion segmentation method based on a multi-attention encoder-decoder network, characterized in that, include: The Pyramid Visual Transformer (PVT) is used as the backbone network for feature extraction. Then, a multi-attention fusion (MAF) method is added to the deep layers of the network to deeply mine the rich semantic information obtained in the encoder stage and initially locate the lesion area. Between the encoder and decoder, a selective information aggregation (SIG) method is used to replace skip connections to alleviate the semantic differences between the encoder and decoder and obtain more efficient skip connections. Finally, a multi-scale cascade fusion (MSCF) method is adopted to dynamically fuse features from different stages of the decoder to accurately locate the boundary of the lesion. The MAF method described above fuses features using a multilayer perceptron (MLP) combined with a spatial attention map to extract richer contextual information; the specific steps are as follows: 1) Perform two parallel pooling operations on the input feature X: global average pooling and global max pooling, and use a 1×1 convolution operation to generate a two-dimensional spatial attention map. It can capture the spatial dependencies of input features; 2) Spatial attention map Transpose the input feature X and multiply it by matrix to obtain the spatial attention-weighted feature. 3) Input weighted features into the MLP to further enhance the expressive power of the features; 4) Finally, the output of the MLP is weighted and fused with the input features to obtain the final output feature Y; The SIG method employs a cross-attention mechanism to selectively aggregate information between the encoder and decoder, thereby fully utilizing the positional information of low-level features and the semantic information of high-level features. The specific steps are as follows: 1) Given the symmetric feature maps F and D of the encoder and decoder, generate feature map Q on F and feature maps K and V on D using 1×1 convolutions respectively; 2) Perform affinity operations on feature maps Q and K to generate an attention weight map; this weight map can measure the correlation of each pixel between the encoder and decoder; 3) Aggregate the attention weight map with the feature map V to obtain the output of the first stage; in this way, each pixel in the encoder can collect information from the corresponding horizontal and vertical positions in the decoder; 4) Two loops are set up so that the encoder can indirectly obtain the complete context information from the decoder; 5) Finally, the output of the first stage is spliced ​​with the feature map D of the decoder on the channel to complete the selective information aggregation operation.

2. The lesion segmentation method based on a multi-attention encoder-decoder network according to claim 1, characterized in that: The specific steps of MAF are summarized as follows: (1) (2) in, Represents spatial attention operations. This represents a convolution operation with a kernel size of 1×1. and These represent global average pooling and global max pooling, respectively. Represents matrix multiplication. This represents the matrix transpose operation.

3. The lesion segmentation method based on a multi-attention encoder-decoder network according to claim 1, characterized in that: The MSCF method effectively integrates features at different levels through cascaded modules and scale-aware modules; The specific steps are as follows: 1) Features at different resolutions are processed through cascaded modules; among which, It is characterized by having high resolution and containing detailed spatial location information; , and These are features with lower resolution but containing more semantic information; they are processed through a series of convolutional units. , and The number of channels is adjusted to obtain the corresponding... , and ; 2) Utilize cascading modules to , and Perform feature fusion; the output of the cascaded module is defined as follows: CM stands for cascade module; 3) To dynamically balance the weights between different scales, a scale-aware module is introduced; X1' and X1 are concatenated on the channel, and then a spatial pixel-level mapping is obtained through convolutional layers and softmax layers. and ; 4) Perform a weighted summation of the features at the two different scales to obtain the final output feature map. ;in, ⨀ represents the Adama product.

4. A lesion segmentation system, characterized in that: This lesion segmentation system is used to implement the lesion segmentation method based on a multi-attention encoder-decoder network as described in any one of claims 1 to 3; Specifically, it includes: Input preprocessing module: preprocesses the input image, including resizing and normalization, so that it can be input into the network for processing; Pre-backbone network: A pre-trained deep convolutional neural network is used as the pre-backbone network of MASDF-Net for extracting image features; Multi-scale feature extraction module: Utilizing the features extracted by the pre-backbone network, the multi-scale feature extraction module obtains feature maps with different resolutions and semantics. Multi-scale cascaded fusion module: Through the cascaded module and the scale-aware module, features at different scales are fused to obtain a fused output feature map; Output prediction module: The fused output feature map is processed using convolutional layers and softmax layers to obtain a pixel-level mapping of skin damage, which is used to represent the location of skin damage; Post-processing module: Performs post-processing on the output pixel-level mapping, including thresholding and connected component analysis, to obtain the final skin lesion segmentation result.

Images