Disease diagnosis-oriented medical image segmentation guide classification system

Abstract

The invention discloses a medical image segmentation guide classification system for disease diagnosis, and the system comprises a data importing module which is used for reading and preprocessing disease image data and disease category data, and obtaining an original image, a real semantic segmentation mask image and a real disease category label Label; and the lesion semantic segmentation and disease classification module is used for executing a segmentation task and a classification task on the original image obtained by the data import module by adopting an improved VM-UNet network to obtain a predicted semantic segmentation mask image and a predicted disease category, so that accurate lesion semantic segmentation and accurate disease classification of the medical image are realized. According to the method, the semantic segmentation task and the classification task of disease diagnosis can be optimized at the same time, implicit segmentation guide classification is realized by using the advantage of multi-task learning, and the shape information and texture information of a focus of interest can be enhanced by reusing a prediction semantic segmentation mask image, so that explicit segmentation guide classification is realized, and the classification efficiency is improved. And the method has relatively high calculation efficiency and generalization.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning and medical image processing, and in particular to a medical image segmentation guided classification system for disease diagnosis. Background Art

[0002] In recent years, the advancement of artificial intelligence technology has made automatic disease diagnosis a reality, and deep learning has been widely used to assist doctors in making decisions. Among them, the purpose of semantic segmentation tasks for disease diagnosis based on deep learning is to predict and label the category of each pixel in the disease image in order to accurately locate and analyze the lesion area; while the purpose of disease classification tasks for disease diagnosis based on deep learning is mainly to predict the patient's condition into different disease categories based on medical images, thereby assisting doctors in diagnosing diseases or guiding treatment. Semantic segmentation and classification for disease diagnosis are tasks with strong correlation. The key morphological features such as lesion shape, size, and edge texture provided by semantic segmentation are conducive to improving the accuracy of disease classification.

[0003] There are three common disease classification systems: 1) Single disease classification system: This type of system focuses on extracting effective features from the disease image itself for disease classification, but this type of system does not consider the guidance of segmentation knowledge at all. 2) Multi-task learning system for segmentation and classification: This type of system additionally introduces semantic segmentation mask images as supervisory information to guide lesion feature extraction. Introducing the semantic segmentation task of disease diagnosis on the basis of the disease classification task can enable the learning of detailed information about the lesion from the shared features of multi-task learning, realizing implicit segmentation-guided classification, and promoting the improvement of disease classification accuracy to a certain extent. However, this method lacks direct interaction between the two tasks, and does not directly use the prior knowledge contained in the semantic segmentation mask image to guide the disease classification task. Therefore, this type of system does not make sufficient use of semantic segmentation mask knowledge. 3) Two-stage segmentation-guided classification system: This type of system uses semantic segmentation mask images to preprocess disease images for explicit segmentation-guided classification. The system is simple. Usually, in the first stage, a semantic segmentation network is trained to predict the semantic segmentation mask image. Then, in the second stage, the semantic segmentation mask image obtained in the first stage is used to locate the disease image and crop the lesion area. The cropped image is then adjusted to the initial input size and input into the classification network for disease classification prediction. However, this method does not use the semantic segmentation mask image as supervisory information to participate in network training like multi-task learning, lacks implicit segmentation-guided classification, and their explicit segmentation-guided classification method does not take into account the more valuable information about the lesion shape contained in the semantic segmentation mask image itself, and the utilization rate of the semantic segmentation mask knowledge is still not high.

[0004] In general, most current disease classification systems make relatively limited use of semantic segmentation mask knowledge. They usually only use semantic segmentation mask knowledge in the preprocessing process, or use it as supervisory information to guide lesion feature extraction, which leads to low disease diagnosis accuracy. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and propose a medical image segmentation-guided classification system for disease diagnosis, which can simultaneously optimize the semantic segmentation task and classification task of disease diagnosis, utilize the advantages of multi-task learning to achieve implicit segmentation-guided classification, and reuse the predicted semantic segmentation mask image to enhance the shape information and texture information of the lesion of interest, thereby achieving explicit segmentation-guided classification and having high computational efficiency and generalization.

[0006] To achieve the above-mentioned purpose, the technical solution provided by the present invention is: a medical image segmentation guided classification system for disease diagnosis, comprising:

[0007] The data import module is used to read the disease image data and disease category data, and preprocess the disease image data and disease category data to obtain the original image, the true semantic segmentation mask image and the true disease category label Label, where the true semantic segmentation mask image and the true disease category label Label are used to calculate the loss function of the improved VM-UNet network to perform the segmentation task and the classification task respectively;

[0008] The lesion semantic segmentation and disease classification module uses an improved VM-UNet network to perform segmentation tasks and classification tasks on the original image obtained by the data import module, obtains a predicted semantic segmentation mask image and a predicted disease category, and realizes accurate lesion semantic segmentation and accurate disease classification of medical images; wherein, the improved VM-UNet network adds a feature refinement module, an explicit segmentation-guided classification module, and a feature fusion and disease classification module on the basis of retaining the encoder module, the decoder module, and the jump connection module of the original VM-UNet network; the feature refinement module is inserted into the jump connection module to adaptively identify and retain key information in the encoder module features to realize information refinement; the explicit segmentation-guided classification module is used to extract shape perception features and texture perception features from the predicted semantic segmentation mask image and the original image, and transmit them to the feature fusion and disease classification module; the feature fusion and disease classification module is used to enhance the shape perception features, texture perception features, and the original image features obtained by the encoder module through the channel attention module respectively, and then perform feature fusion, and output the predicted disease category.

[0009] Furthermore, the data import module specifically performs the following operations:

[0010] The disease image data is uniformly adjusted to an image resolution of 256×256, and then normalized and pixel value range is scaled, and then data enhancement is performed to finally obtain the original image and the true semantic segmentation mask image; wherein, the pixel value range is scaled so that each pixel value takes the value ∈ [0,255], and data enhancement includes contrast transformation, rotation, flipping, translation, and adding Gaussian noise;

[0011] Read the disease category data stored in the CSV format file, obtain the disease category corresponding to each disease image data, and map the disease category to the real disease category label Label.

[0012] Furthermore, the improved VM-UNet network specifically performs the following operations:

[0013] 1) The original image is input into the encoder module and the original image features F are obtained through four downsampling stages. original , and then the original image feature F original The input is fed into the decoder module, and the skip connection module with the feature refinement module inserted is used to obtain the semantic segmentation Logits through four upsampling stages, and then the semantic segmentation Logits are post-processed to obtain the predicted semantic segmentation mask image;

[0014] 2) Input the predicted semantic segmentation mask image and the original image obtained in step 1) into the explicit segmentation guided classification module to obtain the shape feature image and the texture feature image; then use the convolution module designed in the explicit segmentation guided classification module to extract the shape perception feature F from the shape feature image and the texture feature image. shape and texture-aware features F texture , used to improve the shape perception and texture perception of the improved VM-UNet network on the lesions contained in the original image;

[0015] 3) The original image feature F obtained in step 1) original The shape perception feature F obtained in step 2) shape and texture-aware features F texture They are input into the feature fusion and disease classification module together. They first pass through the channel attention module designed in the feature fusion and disease classification module to obtain the enhanced feature F' original , F' shape , F' texture , where the channel attention module can assign weights to different channels, enhance important information, and suppress irrelevant information, thereby improving the representation ability of the improved VM-UNet network; then F' original , F' shape , F' textureAfter feature fusion, it is used to obtain the disease classification prediction probability Prob, where feature fusion is conducive to enriching information representation to improve the generalization of the improved VM-UNet network; finally, the predicted disease category is obtained based on the disease classification prediction probability Prob.

[0016] Furthermore, the encoder module obtains the original image feature F original The specific process is as follows:

[0017] The original image is input into the encoder module and is first divided into non-overlapping image blocks of size 4×4 by the image block embedding layer, and then regularized by the normalization layer. After four downsampling stages, the original image feature F is obtained. original , where each of the first three downsampling stages needs to go through two visual state space blocks and image block merging operations in sequence, while the fourth downsampling stage only needs to go through two visual state space blocks, where the image block merging operation is used to reduce the height and width of the features and increase the number of channels.

[0018] Furthermore, the specific process of the decoder module obtaining the predicted semantic segmentation mask image is as follows:

[0019] Original image feature F original First, it goes through four upsampling stages, and then through a final projection layer to restore the size of the features to obtain the semantic segmentation Logits, and then the semantic segmentation Logits are post-processed to obtain the predicted semantic segmentation mask image; among them, the first upsampling stage only needs to go through 2 visual state space blocks, the second and third upsampling stages need to go through image block expansion operations and 2 visual state space blocks in turn, and the fourth upsampling stage needs to go through image block expansion operations and 1 visual state space block, where the image block expansion operation is used to increase the height and width of the features and reduce the number of channels; the post-processing operation applied to the semantic segmentation Logits is specifically as follows: first, the semantic segmentation Logits are passed through the Sigmoid function so that the value of each pixel is scaled to the probability range of 0 to 1, and the predicted semantic segmentation probability image is obtained, and then the category selection is performed with 0.5 as the threshold, and pixels with values ​​greater than 0.5 are assigned to the foreground category, otherwise they are assigned to the background category, so that the predicted semantic segmentation mask image is obtained.

[0020] Furthermore, the specific operation process of the skip connection module into which the feature refinement module is inserted is as follows:

[0021] The features outputted from the third, second and first downsampling stages of the encoder module are respectively refined by the feature refinement module, and then element-wise summed with the features outputted from the first, second and third upsampling stages of the decoder module, so as to obtain the input features accepted by the second, third and fourth upsampling stages of the decoder module respectively;

[0022] Among them, the feature refinement module is improved on the basis of the Gate Unit of the GateNet network. The feature refinement module replaces one of the global average pooling in the Gate Unit with the global maximum pooling. The combination of global average pooling and global maximum pooling can more effectively enhance the selectivity of the feature control process from different focuses. In addition, the feature refinement module also replaces the ordinary convolution layer in the Gate Unit with a depth-separable convolution layer, which can reduce the amount of calculation while ensuring that the improved VM-UNet network has a high accuracy.

[0023] Furthermore, the feature refinement module uses the features X∈R obtained at different downsampling stages of the encoder module C×H×W As input, R represents a set of real numbers, C represents the number of channels, H represents the height, and W represents the width. Through a 3×3 depth-separable convolution operation, the following process is performed: First, the input feature X is divided into two feature maps f 1 and f 2 , so that the feature refinement module can focus on the effective information from different representation subspaces, and then use the Sigmoid function to transform f 1 and f 2 Normalize to the interval [0,1] to get the intermediate feature f' 1 and f' 2 , by the intermediate feature f' 1 and f' 2 Use global average pooling and global maximum pooling to calculate a pair of refined weight values ​​R 1 and R 2 Finally, use R 1 and R 2 Select context information from the input features respectively, fuse the context information by element-by-element summation, and then perform a 3×3 depth-separable convolution operation to obtain the output feature Y∈R C ×H×W The feature refinement module can selectively send features from the encoder module to the decoder module for the element-by-element summation process of the skip connection module, which can solve the problem that the features of the encoder module transmitted to the decoder module through the skip connection module contain redundant or misleading information;

[0024] The formula for the above process is as follows:

[0025]

[0026]

[0027]

[0028] In the formula, GAP represents global average pooling, GMP represents global maximum pooling, and DSConv 3×3 represents a 3×3 depthwise separable convolution operation, represents the Hadamard product, Represents element-by-element addition.

[0029] Furthermore, the explicit segmentation guided classification module specifically performs the following operations:

[0030] After the original image and the predicted semantic segmentation mask image are input into the explicit segmentation guided classification module, since the predicted semantic segmentation mask image directly reflects the approximate shape of the lesion area visually and contains rich lesion shape information, the value of each pixel of the predicted semantic segmentation mask image is multiplied by 255 and then concatenated with the original image in the channel dimension to obtain a shape feature image containing four channels. At the same time, the Hadamard product is calculated using the original image and the predicted semantic segmentation mask image, that is, the pixel value of the background part of the original image corresponding to the predicted semantic segmentation mask image is changed to 0, and the other pixel values ​​remain unchanged, thereby obtaining a texture feature image, in which irrelevant background pixels are shielded, and the texture detail information inside the lesion is highlighted.

[0031] The shape feature image and texture feature image are respectively input into the convolution module designed in the explicit segmentation guided classification module to extract the shape perception feature F shape and texture-aware features F texture ;

[0032] The convolution module consists of five stages, each of which undergoes the following operations in sequence:

[0033] f i+1 =MaxPool(ReLU(BN(Conv 3×3 (f i )))),i=1,2,3,4,5

[0034] In the formula, f i represents the input of the i-th stage of the convolution module, f i+1 Represents the output of the i-th stage of the convolution module, Conv 3×3 represents a 3×3 convolutional layer, BN represents batch normalization, ReLU is the activation function, and MaxPool represents the maximum pooling layer with a pooling kernel size and a stride of 2.

[0035] Furthermore, the original image feature F obtained in the encoder module original The shape-aware features F obtained in the explicit segmentation-guided classification module shape and texture-aware features F texture It will be input into the feature fusion and disease classification module to obtain the predicted disease category, and the following operations are performed:

[0036] 1) The original image feature F original , shape perception feature F shape and texture-aware features F texture The channel attention module in the feature fusion and disease classification module is input together to obtain the enhanced feature F' original , F' shape , F' texture ;

[0037] Among them, the channel attention module is improved on the basis of the squeeze and excitation module of the SE-Net network. The channel attention module further introduces global maximum pooling as another squeeze operation based on the global average pooling selected for the squeeze operation in the squeeze and excitation module of the SE-Net network, and captures the overall feature parameters of each channel representing the feature map from two different angles of global average pooling and global maximum pooling. The specific definition of the channel attention module is as follows:

[0038] p=Flatten(Concat(GAP(F),GMP(F))),F=F original ,F shape ,F texture

[0039] po=View(Sigmoid(Linear(ReLU(Linear(p)))))

[0040]

[0041] Where F∈R C×H×W As input, the value is F original 、F shape or F texture , Concat means concatenation in the channel dimension, Flatten means flattening a multidimensional tensor into one dimension, Sigmoid is the activation function, Linear means the linear layer, and View means adjusting the shape of the tensor to R C×1×1 , po represents the channel attention weight value, represents the Hadamard product, F'∈R C ×H×W As output, the value is F' original , F'shape or F' texture ;

[0042] 2) Let F' original , F' shape , F' texture Feature fusion is achieved by splicing in the channel dimension, and then the fused features are passed through the classification head to obtain the disease classification prediction probability Prob, and then the category with the largest prediction probability value among the disease classification prediction probability Prob is selected as the predicted disease category;

[0043] The formula for obtaining the disease classification prediction probability Prob is as follows:

[0044] F fusion =Concat(F' original ,F' shape ,F' texture )

[0045] Prob=Sigmoid(Linear(ReLU(Linear(Flatten(GAP(F fusion ))))))

[0046] In the formula, F fusion Represents the fused features.

[0047] Furthermore, the loss function L of the improved VM-UNet network for performing segmentation tasks is Seg And the loss function L of the classification task Cls The total loss function L is obtained by linearly combining the dynamic and learnable weights Total , L Total Used to guide parameter optimization to collaboratively improve the accuracy of the improved VM-UNet network in segmentation and classification tasks; the total loss function L Total The calculation process includes the following steps:

[0048] 1) Calculate the loss function L of the segmentation task based on the predicted semantic segmentation probability image obtained in the decoder module and the real semantic segmentation mask image obtained in the data import module Seg , the formula is as follows:

[0049] L Seg =w b ×L BCE +w d ×L Dice

[0050]

[0051]

[0052] Where, L BCE is the binary cross entropy loss function, L Dice is the Dice loss function, w b YesL BCE The weight, w d YesL Dice The weight, w b and w d The value of is 1, N is the total number of pixels, y i is the i-th pixel value in the true semantic segmentation mask image, p i is the pixel value at the corresponding position in the predicted semantic segmentation probability image, and ε is a smoothing term close to 0, which is used to prevent the denominator from being zero and improve numerical stability;

[0053] 2) Calculate the loss function L of the disease classification task based on the disease classification prediction probability Prob obtained in the feature fusion and disease classification module and the actual disease category label Label obtained in the data import module Cls , using the cross entropy loss function, where L Cls is defined as follows:

[0054]

[0055] In the formula, c is the number of categories in the disease classification task, Label i Represents the one-hot encoding value of the i-th category, Prob i Represents the predicted probability of the i-th category;

[0056] 3) Total loss function L Total The weights of the lesion semantic segmentation task and the disease classification task are dynamically adjusted based on homoscedastic uncertainty. The specific definitions are as follows:

[0057]

[0058] In the formula, σ 1 With σ 2 are learnable noise parameters that are automatically adjusted during the training of the improved VM-UNet network. They are used to represent the uncertainty of the two tasks. The larger the uncertainty of the task, the larger the noise parameter, and the smaller the weight value of the loss function of the corresponding task;

[0059] When the improved VM-UNet network is in the training stage, the L calculated above Total It will be further used to calculate the gradient, and its gradient value will be used by the AdamW optimizer to update the parameters of the improved VM-UNet network during the back propagation process, that is, to guide parameter optimization to collaboratively improve the accuracy of the improved VM-UNet network in segmentation and classification tasks. After multiple iterations to LTotal The value of is the smallest, and finally the optimal improved VM-UNet network is obtained.

[0060] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0061] 1. The system proposed in the present invention improves the original VM-UNet network to obtain an improved VM-UNet network. The improved VM-UNet network can achieve good accuracy in both the semantic segmentation task and the disease classification task of disease diagnosis, and output the prediction results of disease classification end-to-end and provide doctors with interpretable predicted semantic segmentation mask images, so as to efficiently and reliably assist doctors in the process of diagnosing and treating diseases, which has clinical promotion significance. The original VM-UNet network can only be used for the prediction of semantic segmentation tasks.

[0062] 2. The present invention introduces the semantic segmentation task of disease diagnosis on the basis of a single disease classification task to realize a multi-task learning system. By additionally introducing a semantic segmentation mask image as supervisory information to guide lesion feature extraction, more local detail information of lesions that is beneficial to disease classification can be learned from the shared features of multi-task learning, thereby realizing implicit segmentation-guided classification to improve the accuracy of disease classification.

[0063] 3. The present invention designs an explicit segmentation-guided classification module, which reuses the predicted semantic segmentation mask image to preprocess the original image, fully mines the shape knowledge and texture knowledge of the lesion contained in the predicted semantic segmentation mask image, guides the improved VM-UNet network to improve the shape perception and texture perception of the lesion, realizes explicit segmentation-guided classification, and further improves the accuracy of disease classification.

[0064] 4. The present invention designs a feature refinement module that adaptively identifies and retains key information in the features of the encoder module to achieve information refinement, thereby solving the problem that the features of the encoder module of the original VM-UNet network transmitted to the decoder module via jump connections contain redundant or misleading information.

[0065] 5. The present invention designs a feature fusion and disease classification module, which uses the channel attention mechanism to assign weights to different channels of the feature map, enhances important information, and suppresses irrelevant information, thereby improving the representation ability of the improved VM-UNet network. Further feature fusion is conducive to enriching the information representation, so that the improved VM-UNet network can simultaneously pay attention to the global information of the original image and the shape details and texture details of the local lesions, thereby improving the robustness of the improved VM-UNet network.

[0066] 6. The present invention uses homoscedastic uncertainty to define the loss function weight coefficients of the semantic segmentation task and the disease classification task of disease diagnosis, so that the loss function weight coefficients can be dynamically learned and adjusted by the improved VM-UNet network, solving the problem of manually specifying the weights of each loss function of multi-task learning, which requires a lot of time to adjust the parameters.

[0067] 7. The improved VM-UNet network proposed in the present invention has a small number of floating-point operations, which means high computational efficiency, and has good generalization, and can be applied to the auxiliary diagnosis of multiple diseases at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 Flow chart of the system of the present invention.

[0069] Figure 2 To improve the architecture of the VM-UNet network; in the figure, FR 1～4 Represents feature refinement modules 1 to 4, CA 1～3 Represents channel attention modules 1 to 3, Conv 1～2 represents convolution modules 1 to 2, Mel represents the disease category as “melanoma”, Nev represents the disease category as “nevus”, and SK represents the disease category as “seborrheic keratosis”.

[0070] Figure 3 This is a structural diagram of the feature refinement module.

[0071] Figure 4 Schematic diagram of the convolution module.

[0072] Figure 5 Schematic diagram of the structure of the channel attention module. DETAILED DESCRIPTION

[0073] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0074] Example 1

[0075] This embodiment is described using the ISIC2017 dataset as an example. The dataset is a standard public dataset for skin lesion analysis and is annotated by professional doctors. There are three disease categories in the dataset, namely, melanoma, nevus, and seborrheic keratosis.

[0076] like Figure 1As shown, this embodiment discloses a medical image segmentation guided classification system for disease diagnosis, including a data import module and a lesion semantic segmentation and disease classification module; the ISIC2017 dataset is first input into the data import module to obtain the ISIC2017 real disease category label, the ISIC2017 original image and the ISIC2017 real semantic segmentation mask image, and then the ISIC2017 original image is input into the lesion semantic segmentation and disease classification module to obtain the ISIC2017 predicted disease category and the ISIC2017 predicted semantic segmentation mask image, thereby assisting doctors in the process of diagnosing and treating diseases; wherein the lesion semantic segmentation and disease classification module is implemented based on the improved VM-UNet network, as shown in FIG. Figure 2 As shown in FIG, the improved VM-UNet network is improved by adding a feature refinement module, an explicit segmentation-guided classification module, a feature fusion and disease classification module on the basis of retaining the encoder module, decoder module and skip connection module of the original VM-UNet network;

[0077] The operation process of the system of the present invention includes the following steps:

[0078] 1) Data import:

[0079] This example uses the ISIC2017 dataset as an example. This dataset is a standard public dataset for skin lesion analysis and is annotated by professional doctors. There are three disease categories in this dataset, namely, melanoma, nevus, and seborrheic keratosis. Figure 1 As shown, the ISIC2017 dataset is first input into the data import module for processing, which includes the following steps:

[0080] The ISIC2017 disease image data in the ISIC2017 dataset is uniformly adjusted to an image resolution of 256×256, and then normalized and pixel value range is scaled, and then data enhancement is performed to finally obtain the ISIC2017 original image and the ISIC2017 true semantic segmentation mask image; wherein, the pixel value range is scaled so that each pixel value takes the value ∈ [0,255], and data enhancement includes contrast transformation, rotation, flipping, translation, and adding Gaussian noise;

[0081] Read the ISIC2017 disease category data of the ISIC2017 dataset stored in a CSV file, obtain the disease category corresponding to each ISIC2017 disease image data, and map the disease category to the real disease category label Label.

[0082] Specifically, the CSV file contains three columns of data. The first column of data represents the file name of each ISIC2017 disease image data excluding the file extension. The second column of data represents whether the ISIC2017 disease image data is of the disease category of melanoma, which has only two values, "0" and "1". The third column of data represents whether the ISIC2017 disease image data is of the disease category of seborrheic keratosis, which also has only two values, "0" and "1". The mapping rule from disease category to the real disease category label Label is that if the disease category of a case is "mole", then the value of its real disease category label Label is 0. If the disease category of a case is "nevus", then the value of its real disease category label Label is 0. If the disease category of a case is "melanoma", the value of its true disease category label Label is 1. If the disease category of a case is "seborrheic keratosis", the value of its true disease category label Label is 2. For example, the first column of data of a case in ISIC2017 is "ISIC_0000002", the value of the second column of data is "1", indicating that the case belongs to the disease category of melanoma, and the value of the third column of data is "0", indicating that the case does not belong to the disease category of seborrheic keratosis. Therefore, the disease category of the case is "melanoma". After mapping, the true disease category label Label of the case is 1.

[0083] 2) Perform segmentation task to obtain original image features F original Image with predicted semantic segmentation mask:

[0084] 2.1) Input the ISIC2017 original image into the encoder module and obtain the ISIC2017 original image feature F through four downsampling stages original ;

[0085] The encoder module obtains the ISIC2017 original image feature F original The specific process is as follows:

[0086] The ISIC2017 original image is input into the encoder module and is first divided into non-overlapping image blocks of size 4×4 by the image block embedding layer, and then regularized by the normalization layer. After four downsampling stages, the ISIC2017 original image feature F is obtained. original , where each of the first three downsampling stages needs to go through two visual state space blocks and image block merging operations in sequence, while the fourth downsampling stage only needs to go through two visual state space blocks, where the image block merging operation is used to reduce the height and width of the feature and increase the number of channels;

[0087] 2.2) Then the ISIC2017 original image feature F originalThe input is fed into the decoder module, and the skip connection module with the feature refinement module inserted is used to obtain the semantic segmentation Logits through four upsampling stages, and then the semantic segmentation Logits are post-processed to obtain the ISIC2017 predicted semantic segmentation mask image;

[0088] The specific process of the decoder module obtaining the predicted semantic segmentation mask image is as follows:

[0089] ISIC2017 original image features F original First, after four upsampling stages, a final projection layer is used to restore the size of the features to obtain the semantic segmentation Logits, and then the semantic segmentation Logits are post-processed to obtain the predicted semantic segmentation mask image; among them, the first upsampling stage only needs to go through 2 visual state space blocks, the second and third upsampling stages need to go through image block expansion operations and 2 visual state space blocks in turn, and the fourth upsampling stage needs to go through image block expansion operations and 1 visual state space block, where the image block expansion operation is used to increase the height and width of the features and reduce the number of channels; the post-processing operation applied to the semantic segmentation Logits is specifically as follows: first, the semantic segmentation Logits are passed through the Sigmoid function so that the value of each pixel is scaled to a probability range of 0 to 1, and the ISIC2017 predicted semantic segmentation probability image is obtained, and then the category selection is performed with 0.5 as the threshold, and the pixels with a pixel value greater than 0.5 are assigned to the foreground category, otherwise they are assigned to the background category, so that the ISIC2017 predicted semantic segmentation mask image is obtained, which can provide doctors with accurate information about the location, shape, size, etc. of the lesion, and provide a certain degree of interpretability for the classification prediction results output synchronously by the system;

[0090] The specific operation process of the skip connection module into which the feature refinement module is inserted is as follows:

[0091] The features outputted from the third, second and first downsampling stages of the encoder module are respectively refined by the feature refinement module, and then element-wise summed with the features outputted from the first, second and third upsampling stages of the decoder module, so as to obtain the input features accepted by the second, third and fourth upsampling stages of the decoder module respectively;

[0092] Among them, the feature refinement module is improved on the basis of the Gate Unit of the GateNet network. The Gate Unit only uses global average pooling to obtain a pair of refined weight values, and the convolution operation in the Gate Unit uses a common convolution layer with a large amount of calculation; and the feature refinement module replaces one of the global average pooling in the Gate Unit with a global maximum pooling, and adopts a combination of global average pooling and global maximum pooling, which can more effectively enhance the selectivity of the feature control process from different focuses; in addition, the feature refinement module also replaces the common convolution layer in the Gate Unit with a depth-separable convolution layer, which can reduce the amount of calculation while ensuring that the improved VM-UNet network has a high accuracy rate;

[0093] Specifically, if Figure 3 As shown, the feature refinement module uses the features X∈R obtained at different downsampling stages of the encoder module C×H×W As input, R represents a set of real numbers, C represents the number of channels, H represents the height, and W represents the width. Through a 3×3 depth-separable convolution operation, the following process is performed: First, the input feature X is divided into two feature maps f 1 and f 2 , so that the feature refinement module can focus on the effective information from different representation subspaces, and then use the Sigmoid function to transform f 1 and f 2 Normalize to the interval [0,1] to get the intermediate feature f' 1 and f' 2 , by the intermediate feature f' 1 and f' 2 Use global average pooling and global maximum pooling to calculate a pair of refined weight values ​​R 1 and R 2 Finally, use R 1 and R 2 Select context information from the input features respectively, fuse the context information by element-by-element summation, and then perform a 3×3 depth-separable convolution operation to obtain the output feature Y∈R C×H×W The feature refinement module can selectively send features from the encoder module to the decoder module for the element-by-element summation process of the skip connection module, which can solve the problem that the features of the encoder module transmitted to the decoder module through the skip connection module contain redundant or misleading information;

[0094] The formula for the above process is as follows:

[0095]

[0096]

[0097]

[0098] In the formula, GAP represents global average pooling, GMP represents global maximum pooling, and DSConv 3×3 represents a 3×3 depthwise separable convolution operation, represents the Hadamard product, Represents element-by-element addition.

[0099] 3) Explicit segmentation guided classification:

[0100] The ISIC2017 predicted semantic segmentation mask image and the ISIC2017 original image obtained in step 2) are input into the designed explicit segmentation guided classification module to obtain the ISIC2017 shape feature image and the ISIC2017 texture feature image; and the convolution module designed in the explicit segmentation guided classification module is further used to extract the ISIC2017 shape feature image and the ISIC2017 texture feature image to obtain the ISIC2017 shape perception feature F shape Compared with ISIC2017 texture perception feature F texture , which can improve the shape perception and texture perception of the lesions contained in the ISIC2017 original image of the improved VM-UNet network, and realize explicit segmentation-guided classification; specifically, the above process includes the following steps:

[0101] 3.1) Reuse the ISIC2017 predicted semantic segmentation mask image and the ISIC2017 original image in the explicit segmentation guided classification module to obtain the ISIC2017 shape-aware feature F shape Compared with ISIC2017 texture perception feature F texture ;

[0102] Specifically, after the ISIC2017 original image and the ISIC2017 predicted semantic segmentation mask image obtained in step 2) are input into the explicit segmentation guided classification module, since the ISIC2017 predicted semantic segmentation mask image directly reflects the approximate shape of the skin lesion area visually and contains rich lesion shape information, the value of each pixel of the ISIC2017 predicted semantic segmentation mask image is multiplied by 255 and then concatenated with the ISIC2017 original image in the channel dimension to obtain an ISIC2017 shape feature image containing four channels; at the same time, the ISIC2017 original image and the ISIC2017 predicted semantic segmentation mask image obtained in step 2) are used to calculate the Hadamard product, that is, the pixel value of the background part of the ISIC2017 original image corresponding to the ISIC2017 predicted semantic segmentation mask image is changed to 0, and the other pixel values ​​remain unchanged, thereby obtaining an ISIC2017 texture feature image, in which irrelevant background pixels are shielded, and the texture detail information inside the skin lesion is highlighted;

[0103] 3.2) Further, the ISIC2017 shape feature image and the ISIC2017 texture feature image are respectively input into the convolution module in the explicit segmentation guided classification module to extract the ISIC2017 shape perception feature F shape Compared with ISIC2017 texture perception feature F texture , the internal implementation details of the convolution module are as follows Figure 4 As shown;

[0104] The convolution module consists of five stages, each of which undergoes the following operations in sequence:

[0105] f i+1 =MaxPool(ReLU(BN(Conv 3×3 (f i )))),i=1,2,3,4,5

[0106] In the formula, f i represents the input of the i-th stage of the convolution module, f i+1 Represents the output of the i-th stage of the convolution module, Conv 3×3 represents a 3×3 convolutional layer, BN represents batch normalization, ReLU is the activation function, and MaxPool represents the maximum pooling layer with a pooling kernel size and a stride of 2;

[0107] also, Figure 2 There are two different convolution modules in the network, namely Conv 1 With Conv 2 , this is because the input images they accept are different, such as Figure 4As shown in the input image area on the left side of the convolution module, they accept the ISIC2017 texture feature image and the ISIC2017 shape feature image as input respectively. Specifically, since Conv 1 The input is the ISIC2017 texture feature image, which has 3 channels, so Conv 1 The number of input channels of the first stage convolutional layer is 3; Conv 2 The input is the ISIC2017 shape feature image, which is the concatenation of the ISIC2017 original image and the ISIC2017 predicted semantic segmentation mask image in the channel dimension. That is, the number of channels of the image is 4, so Conv 2 The number of input channels of the convolutional layer in the first stage is 4;

[0108] Since the convolution kernel size of the convolution layer is fixed to 3×3, and the stride and padding are both 1, the width and height of the input feature map and the output feature map of each convolution layer will not change. In addition, the number of output channels of the convolution layers in the second, third, and fourth stages is twice the number of input channels. For example, the number of channels × width × height of the input feature map of the convolution layer in the second stage is 64×128×128, and the number of channels × width × height of its output feature map is 128×128×128; the first convolution layer is different from the fifth convolution layer. The number of input channels of the first convolution layer is specified as 3 and 4 respectively according to the difference between the input ISIC2017 texture feature image and the ISIC2017 shape feature image, and its output channel number is fixed to 256, while the number of input and output channels of the convolution layer in the fifth stage is 512 and 384 respectively. The dimensions of each feature map after batch normalization and ReLU activation function layer will not change. For example, the number of channels × width × height of the input feature map and output feature map of the first batch normalization and ReLU activation function layer are both 64 × 256 × 256. Since the pooling kernel size and stride of the maximum pooling layer are both 2, the width and height of the feature map after each maximum pooling layer will be halved, but the number of channels remains unchanged. For example, the number of channels × width × height of the input feature map of the fifth maximum pooling layer is 384 × 16 × 16, and the number of channels × width × height of the output feature map is 384 × 8 × 8.

[0109] 4) Feature fusion and disease classification:

[0110] The ISIC2017 original image feature F obtained in step 2) original The ISIC2017 shape perception feature F obtained by step 3) shape Compared with ISIC2017 texture perception feature F textureThey are input into the designed feature fusion and disease classification module together, and they first pass through the channel attention module designed in the feature fusion and disease classification module to obtain the enhanced feature F′ original , F′ shape , F′ texture , the channel attention module can assign weights to different channels, enhance important information, and suppress irrelevant information, thereby improving the representation ability of the improved VM-UNet network; then F′ original , F′ shape , F′ texture After feature fusion, it is used to obtain the disease classification prediction probability Prob. Feature fusion is conducive to enriching information representation and improving the generalization of the improved VM-UNet network. Finally, the predicted disease category is obtained based on the disease classification prediction probability Prob. The following operations are performed specifically:

[0111] 4.1) The ISIC2017 original image feature F original 、ISIC2017 shape perception feature F shape Compared with ISIC2017 texture perception feature F texture The enhanced feature F′ is input into the channel attention module in the designed feature fusion and disease classification module to obtain the enhanced feature original , F′ shape , F′ texture ;

[0112] Among them, the internal implementation details of the channel attention module are as follows Figure 5 As shown in FIG. 1 , it is improved on the basis of the squeeze and excitation module of the SE-Net network. The channel attention module further introduces global maximum pooling as another squeeze operation based on the global average pooling used in the squeeze operation of the squeeze and excitation module of the SE-Net network, and captures the overall feature parameters of each channel representing the feature map from two different perspectives: global average pooling and global maximum pooling. The specific definition of the channel attention module is as follows:

[0113] p=Flatten(Concat(GAP(F),GMP(F))), F=F original , F shape , F texture

[0114] po=View(Sigmoid(Linear(ReLU(Linear(p)))))

[0115]

[0116] Where F∈R C×H×W As input, the value is F original 、Fshape or F texture , GAP and GMP Figure 5 There are two squeezing operations in , Concat means concatenation in the channel dimension, Flatten means flattening the multidimensional tensor into one dimension, Sigmoid is the activation function, Linear means the linear layer, and View means adjusting the tensor shape to R C×1×1 , po represents the channel attention weight value, and the process of obtaining po represented by the second formula above is Figure 5 The specific content of the incentive operation, represents the Hadamard product, F'∈R C×H×W As output, the value is F' original , F' shape or F' texture ;

[0117] 4.2) Let F' original , F' shape , F' texture Feature fusion is achieved by splicing in the channel dimension, and then the fused features are passed through the classification head to obtain the disease classification prediction probability Prob, and then the category with the largest prediction probability value among the disease classification prediction probability Prob is selected as the predicted disease category;

[0118] The formula for obtaining the disease classification prediction probability Prob is as follows:

[0119] F fusion =Concat(F' original ,F' shape ,F' texture )

[0120] Prob=Sigmoid(Linear(ReLU(Linear(Flatten(GAP)F fusion ))))))

[0121] In the formula, F fusion Represents the fused features.

[0122] 5) Calculate the total loss function L Total :

[0123] The improved VM-UNet network performs the loss function L of the segmentation task Seg And the loss function L of the classification task Cls The total loss function L is obtained by linearly combining the dynamic and learnable weights Total , L TotalIt is used to guide parameter optimization to collaboratively improve the accuracy of the improved VM-UNet network in segmentation and classification tasks, thereby taking advantage of multi-task learning to achieve implicit segmentation-guided classification; this dynamic learning method solves the problem of manually specifying the weights of each loss function in multi-task learning, which requires a lot of time to adjust parameters; the total loss function L Total The calculation process includes the following steps:

[0124] 5.1) Calculate the loss function L of the segmentation task based on the ISIC2017 predicted semantic segmentation probability image obtained in the decoder module and the ISIC2017 real semantic segmentation mask image obtained in the data import module Seg , the formula is as follows:

[0125] L Seg =w b =L BCE +w d =L Dice

[0126]

[0127]

[0128] Where, L BCE is the binary cross entropy loss function, L Dice is the Dice loss function, w b YesL BCE The weight, w d YesL Dice The weight, w b and w d The value of is 1, N is the total number of pixels, y i is the i-th pixel value in the true semantic segmentation mask image, p i is the pixel value at the corresponding position in the predicted semantic segmentation probability image, and ε is a smoothing term close to 0, which is used to prevent the denominator from being zero and improve numerical stability;

[0129] 5.2) Calculate the loss function L of the disease classification task based on the disease classification prediction probability Prob obtained in the feature fusion and disease classification module and the actual disease category label Label obtained in the data import module Cls , using the cross entropy loss function;

[0130] Among them, L Cls is defined as follows:

[0131]

[0132] In the formula, c is the number of categories in the disease classification task. For the ISIC2017 dataset in this example, the value of c is 3, because there are three categories: nevus, melanoma, and seborrheic keratosis. i Represents the one-hot encoding value of the i-th category, Prob i Represents the predicted probability of the i-th category;

[0133] 5.3) Total loss function L Total The weights of the lesion semantic segmentation task and the disease classification task are dynamically adjusted based on homoscedastic uncertainty. The specific definitions are as follows:

[0134]

[0135] In the formula, σ 1 With σ 2 are learnable noise parameters that are automatically adjusted during the training of the improved VM-UNet network. They are used to represent the uncertainty of the two tasks. The larger the uncertainty of the task, the larger the noise parameter, and the smaller the weight value of the loss function of the corresponding task;

[0136] When the improved VM-UNet network is in the training stage, the L calculated above Total It will be further used to calculate the gradient, and its gradient value will be used by the AdamW optimizer to update the parameters of the improved VM-UNet network during the back propagation process, that is, to guide parameter optimization to collaboratively improve the accuracy of the improved VM-UNet network in segmentation and classification tasks. After multiple iterations to L Total The value of is the smallest, and finally the optimal improved VM-UNet network is obtained.

[0137] In order to verify the effectiveness of the system of the present invention, this embodiment conducted comparative experiments, ablation experiments, and exploratory experiments on the prediction semantic segmentation mask image reuse strategy on the ISIC2017 dataset. Each experimental data is expressed in the form of mean ± standard deviation; the data in bold indicates the optimal result, and the data underlined below indicates the suboptimal result; among them, the evaluation indicators of the disease classification task include accuracy Acc, recall Recall, precision Precision and area under the curve AUC. The larger the values ​​of these four evaluation indicators, the better the system effect; the evaluation indicators of the lesion semantic segmentation task include dice coefficient Dice, average intersection over union ratio mIoU, 95% asymmetric Hausdorff distance HD95 and average surface distance ASD. The larger the Dice and mIoU, the better the system effect, while the smaller the HD95 and ASD, the better the system effect.

[0138] Among them, Acc represents the proportion of correctly classified samples in all categories to the total number of samples, which measures the overall classification performance; Recall represents the proportion of correctly identified samples in each category, reflecting the sensitivity of the system to the target category; Precision represents the proportion of samples predicted to belong to this category in each category, which measures the reliability of the system's prediction results; AUC represents a comprehensive indicator of the system's ability to distinguish sample categories under different classification thresholds; Dice measures the degree of overlap between the predicted results and the true labels at the pixel level. The closer the value is to 1, the more accurate the prediction; mIoU measures the average value of the intersection over union of all category prediction results and the true labels, reflecting the global semantic segmentation performance; HD95 is the 95% quantile of the maximum error between the predicted results and the true boundary. The smaller the error, the better the boundary match, and the unit is millimeter; ASD represents the average surface distance between the predicted results and the true boundary, which measures the overall deviation of the segmentation result, and the unit is millimeter.

[0139] The comparative experimental results of the disease classification task on the ISIC2017 dataset are shown in Table 1:

[0140] Table 1 Comparative experimental results of disease classification tasks on the ISIC2017 dataset

[0141] system Acc(%) Recall(%) Precision(%) AUC (%) ResNet-50 network 74.73± 64.49±1.05 68.41± 80.35±1.14 DenseNet-121 Network 72.60± 68.25±1.24 63.04± 85.32±0.72 Inception-V4 Network 70.19± 58.41±0.88 58.73± 78.89±1.46 EfficientNet-B3Net 75.57± 70.58±1.41 68.62± 86.34±0.85 MedViT Network 71.83± 61.93±0.88 63.61± 80.29±0.70 MTDUNet Network 76.67± 71.51±0.77 68.13± 85.87±0.91 ACSNet Network <![CDATA[ 77.80± ]]> <![CDATA[ 72.39±1.14 ]]> <![CDATA[ 70.10± ]]> <![CDATA[ 88.14±1.28 ]]> System of the present invention 80.13± 76.92±0.95 74.51± 90.91±1.03

[0142] It can be seen from Table 1 that compared with the comparison systems represented by other networks, the system of the present invention has achieved the best results in the four disease classification indicators of the ISIC2017 dataset, which reflects the superiority of the system of the present invention.

[0143] The comparative experimental results of the lesion semantic segmentation task on the ISIC2017 dataset are shown in Table 2:

[0144] Table 2 Comparative experimental results of lesion semantic segmentation task on ISIC2017 dataset

[0145] system mIoU(%) Dice(%) HD95(mm) ASD(mm) UNet Network 70.17± 82.76±0.36 13.05± 1.62±0.21 FCN Network 69.21± 81.74±0.58 18.47± 4.58±0.17 DeeplabV3+ network 73.98± 85.19±0.39 10.21± 1.51±0.06 UNet++ Network 72.38± 83.79±0.92 11.38± 1.46±0.09 It turns out that VM-UNet 76.11± 86.09±0.51 9.25±0.71 1.60±0.12 MTDUNet Network 75.28± 85.31±0.80 9.58±0.69 1.40±0.11 ACSNet Network <![CDATA[ 77.39± ]]> <![CDATA[ 86.87±0.76 ]]> <![CDATA[ 7.95±0.38 ]]> <![CDATA[ 1.11±0.17 ]]> System of the present invention 78.72± 88.59±0.44 7.20±0.57 1.07±0.08

[0146] It can be seen from Table 2 that compared with the comparison systems represented by other networks, the system of the present invention has achieved the best results in the four lesion semantic segmentation indicators of the ISIC2017 dataset, reflecting the superiority of the system of the present invention.

[0147] The ablation experiment results on the ISIC2017 dataset are shown in Table 3:

[0148] Table 3 Ablation experiment results of ISIC2017 dataset

[0149]

[0150]

[0151] In Table 3, the system represented by the VM-ClsNet network only retains the encoder module, feature fusion and disease classification module, and is only used for disease classification tasks. original Input into the feature fusion and disease classification module, so no feature fusion operation is performed in the feature fusion and disease classification module, but F is directly original Input to the classification head to obtain the disease classification prediction probability Prob; the system represented by VM-CUNet is obtained by further adding a skip connection module and a decoder module on the basis of the system represented by the VM-ClsNet network, that is, the lesion semantic segmentation task is introduced to realize multi-task learning, and lesion segmentation and disease classification can be completed simultaneously; It can be seen from Table 3 that the introduction of the lesion semantic segmentation task to realize multi-task learning and the shape perception guidance in the explicit segmentation-guided classification module have a very obvious effect on the improvement of the system of the present invention; The results shown in Table 3 successfully verify the effectiveness of the system of the present invention introducing the lesion semantic segmentation task to realize multi-task learning, the feature refinement module, the shape perception guidance in the explicit segmentation-guided classification module, and the texture perception guidance in the explicit segmentation-guided classification module.

[0152] The experimental results of exploring the reuse strategy of predicted semantic segmentation mask images on the ISIC2017 dataset are shown in Table 4:

[0153] Table 4 Experimental results of exploring the reuse strategy of predicted semantic segmentation mask images on the ISIC2017 dataset

[0154]

[0155]

[0156] In Table 4, the reuse strategy of element-by-element addition is to first multiply each pixel of the predicted semantic segmentation mask image by 255, then perform element-by-element summation with the original image, and finally divide the result by 2; the reuse strategy of cropping and enlarging is to use the predicted semantic segmentation mask image to generate a bounding box that can just include all foreground pixels, then transfer the coordinates of this bounding box to the original image, and then crop the area in the original image within the bounding box to obtain a new image containing only a small number of background pixels and lesion areas, and then enlarge the resolution of the new image to the resolution of the original image; It can be seen from Table 4 that when element-by-element addition or cropping and enlarging are used alone, the effect is not as good as using the shape perception guidance of the system of the present invention alone or using the texture perception guidance of the system of the present invention alone, and when element-by-element addition or cropping and enlarging are further added on the basis of using shape perception guidance and texture perception guidance at the same time, the effect is actually reduced compared with using shape perception guidance and texture perception guidance at the same time; The results shown in Table 4 successfully verify the effectiveness of the two predicted semantic segmentation mask image reuse strategies of shape perception guidance and texture perception guidance proposed by the system of the present invention.

[0157] In addition, Table 5 also verifies the FLOPs size of the compared systems. FLOPs represents the number of floating-point operations and measures the computing efficiency of the system. The smaller the FLOPs, the higher the computing efficiency of the system. Its unit is GFLOPs:

[0158] Table 5 FLOPs size of the compared systems

[0159]

[0160]

[0161] In Table 5, the systems represented by the ResNet-50 network, DenseNet-121 network, Inception-V4 network, EfficientNet-B3 network and MedViT network can only be used for disease classification tasks, belonging to single-task systems, their FLOPs are low and the computational efficiency is high, while the system of the present invention can simultaneously complete the lesion segmentation task and the disease classification task, belonging to a multi-task system. Under the premise that the system architecture and implementation details are more complex and the implementation difficulty is greater, the FLOPs size of the system of the present invention is very close to those of these single-task systems for disease classification, and is not much higher than them, which shows that the system of the present invention has strong competitiveness in computational efficiency; the systems represented by the UNet network, FCN network, DeeplabV3+ network, UNet++ network and the original VM-UNet network can only be used for disease classification tasks, belonging to single-task systems, and their FLOPs are low and the computational efficiency is high, while the system of the present invention can simultaneously complete the lesion segmentation task and the disease classification task, belonging to a multi-task system. Under the premise that the system architecture and implementation details are more complex and the implementation difficulty is greater, the FLOPs size of the system of the present invention is very close to those of these single-task systems for disease classification, and is not much higher than them, which shows that the system of the present invention has strong competitiveness in computational efficiency. The system used for lesion segmentation task is also a single-task system. However, under the premise that the system architecture and implementation details are more complicated and the implementation difficulty is greater, the FLOPs of the system of the present invention is lower than most of them, and is only slightly higher than the system represented by the original VM-UNet network and is very close, which proves that the computational efficiency of the system of the present invention is superior to these single-task systems for lesion segmentation tasks; the systems represented by the MTDUNet network and the ACSNet network and the system of the present invention are both multi-task learning systems that can simultaneously complete the lesion semantic segmentation task and the disease classification task, and the FLOPs of the system of the present invention is much lower than the systems represented by the MTDUNet network and the ACSNet network, which proves that the computational efficiency of the system of the present invention is superior to other multi-task systems; in summary, the computational efficiency of the system of the present invention is excellent.

[0162] Example 2

[0163] In addition, in order to further verify that the system of the present invention has good generalization, this embodiment conducts comparative experiments and ablation experiments on another dataset called BUSI. The BUSI dataset is a public breast tumor dataset annotated by professional doctors, and the categories in the dataset include benign tumors and malignant tumors. The description of the experimental indicators is consistent with the description in Example 1. The bold data in the table represents the optimal result, and the underlined data represents the suboptimal result.

[0164] The comparative experimental results of the disease classification task on the BUSI dataset are shown in Table 6:

[0165] Table 6 Comparative experimental results of disease classification tasks on the BUSI dataset

[0166] system Acc(%) Recall(%) Precision(%) AUC (%) ResNet-50 network 90.84±1.73 90.55±1.69 89.81±2.05 93.07±0.98 DenseNet-121 90.55±1.68 89.97±2.03 90.22±2.40 93.86±1.18 Inception-V4 Network 88.09±1.89 84.11±1.51 90.59±1.86 92.90±1.04 EfficientNet-B3Net 91.04±1.51 87.25±2.01 92.13±1.95 94.15±0.88 MedViT Network 88.93±1.41 88.11±1.71 86.78±1.93 93.15±0.92 MTDUNet Network 91.58±1.71 90.95±1.43 91.41±1.85 94.78±1.22 ACSNet Network <![CDATA[ 92.60±1.96 ]]> <![CDATA[ 92.22±1.88 ]]> <![CDATA[ 93.12±2.31 ]]> <![CDATA[ 95.98±0.82 ]]> System of the present invention 95.25±1.98 94.09±1.65 94.88±2.01 97.11±0.83

[0167] The comparative experimental results of the lesion semantic segmentation task on the BUSI dataset are shown in Table 7:

[0168] Table 7 Comparative experimental results of lesion segmentation task on BUSI dataset

[0169] system mIoU(%) Dice(%) HD95(mm) ASD(mm) UNet Network 61.52± 75.90±1.44 19.54± 8.19±1.15 FCN Network 59.88± 72.95±1.39 20.33± 8.73±1.02 DeeplabV3+ network 66.54± 79.35±1.03 15.04± 5.13±0.89 UNet++ Network 62.73± 77.02±1.23 18.71± 6.43±1.33 It turns out that VM-UNet 67.68± 80.72±1.21 15.68± 5.56±0.77 MTDUNet Network 67.27± 80.43±1.33 14.95± <![CDATA[ 4.74±0.93 ]]> ACSNet Network <![CDATA[ 70.11± ]]> <![CDATA[ 81.33±1.07 ]]> <![CDATA[ 14.50± ]]> 4.88±0.86 System of the present invention 71.43± 82.91±1.21 13.81± 4.27±0.91

[0170] The ablation experiment results on the BUSI dataset are shown in Table 8:

[0171] Table 8 Ablation experiment results of BUSI dataset

[0172]

[0173]

[0174] From the data in Tables 6 and 7, compared with the compared systems, the system of the present invention also achieved the best effect on the BUSI dataset; the results shown in Table 8 also successfully verified the effectiveness of the system of the present invention introducing the lesion semantic segmentation task to realize multi-task learning, the feature refinement module, the shape perception guidance in the explicit segmentation guided classification module, and the texture perception guidance in the explicit segmentation guided classification module.

[0175] In summary, the system of the present invention can achieve excellent results not only on the skin lesion analysis dataset ISIC2017, but also on the breast tumor dataset BUSI, which proves that the system of the present invention is suitable for disease diagnosis of different types of diseases, has good generalization, and is worthy of promotion.

[0176] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A medical image segmentation guided classification system for disease diagnosis, characterized in that: include: The data import module is used to read the disease image data and disease category data, and preprocess the disease image data and disease category data to obtain the original image, the true semantic segmentation mask image and the true disease category label Label, where the true semantic segmentation mask image and the true disease category label Label are used to calculate the loss function of the improved VM-UNet network to perform the segmentation task and the classification task respectively; The lesion semantic segmentation and disease classification module uses an improved VM-UNet network to perform segmentation tasks and classification tasks on the original image obtained by the data import module, obtains a predicted semantic segmentation mask image and a predicted disease category, and realizes accurate lesion semantic segmentation and accurate disease classification of medical images; wherein, the improved VM-UNet network adds a feature refinement module, an explicit segmentation-guided classification module, and a feature fusion and disease classification module on the basis of retaining the encoder module, decoder module, and jump connection module of the original VM-UNet network; the feature refinement module is inserted into the jump connection module to adaptively identify and retain key information in the encoder module features to achieve information refinement; the explicit segmentation-guided classification module is used to extract shape perception features and texture perception features from the predicted semantic segmentation mask image and the original image, and transmit them to the feature fusion and disease classification module; the feature fusion and disease classification module is used to enhance the shape perception features, texture perception features, and the original image features obtained by the encoder module through the channel attention module, and then perform feature fusion, and output the predicted disease category.

2. A medical image segmentation guided classification system for disease diagnosis according to claim 1, characterized in that: The data import module specifically performs the following operations: The disease image data is uniformly adjusted to an image resolution of 256×256, and then normalized and pixel value range is scaled, and then data enhancement is performed to finally obtain the original image and the true semantic segmentation mask image; wherein, the pixel value range is scaled so that each pixel value takes the value ∈ [0,255], and data enhancement includes contrast transformation, rotation, flipping, translation, and adding Gaussian noise; Read the disease category data stored in the CSV format file, obtain the disease category corresponding to each disease image data, and map the disease category to the real disease category label Label.

3. A medical image segmentation guided classification system for disease diagnosis according to claim 2, characterized in that: The improved VM-UNet network specifically performs the following operations: 1) The original image is input into the encoder module and the original image features F are obtained through four downsampling stages. original , and then the original image feature F original The input is fed into the decoder module, and the skip connection module with the feature refinement module inserted is used to obtain the semantic segmentation Logits through four upsampling stages, and then the semantic segmentation Logits are post-processed to obtain the predicted semantic segmentation mask image; 2) Input the predicted semantic segmentation mask image and the original image obtained in step 1) into the explicit segmentation guided classification module to obtain the shape feature image and the texture feature image; then use the convolution module designed in the explicit segmentation guided classification module to extract the shape perception feature F from the shape feature image and the texture feature image. shape and texture-aware features F texture , used to improve the shape perception and texture perception of the improved VM-UNet network on the lesions contained in the original image; 3) The original image feature F obtained in step 1) original The shape perception feature F obtained in step 2) shape and texture-aware features F texture They are input into the feature fusion and disease classification module together. They first pass through the channel attention module designed in the feature fusion and disease classification module to obtain the enhanced feature F' original 、F' shape 、F' texture , where the channel attention module can assign weights to different channels, enhance important information, and suppress irrelevant information, thereby improving the representation ability of the improved VM-UNet network; then F' original 、F' shape 、F' texture After feature fusion, it is used to obtain the disease classification prediction probability Prob, where feature fusion is conducive to enriching information representation to improve the generalization of the improved VM-UNet network; finally, the predicted disease category is obtained based on the disease classification prediction probability Prob.

4. The medical image segmentation guided classification system for disease diagnosis according to claim 3, characterized in that: The encoder module obtains the original image features F original The specific process is as follows: The original image is input into the encoder module and is first divided into non-overlapping image blocks of size 4×4 by the image block embedding layer, and then regularized by the normalization layer. After four downsampling stages, the original image feature F is obtained. original , where each of the first three downsampling stages needs to go through two visual state space blocks and image block merging operations in sequence, while the fourth downsampling stage only needs to go through two visual state space blocks, where the image block merging operation is used to reduce the height and width of the features and increase the number of channels.

5. A medical image segmentation guided classification system for disease diagnosis according to claim 4, characterized in that: The specific process of the decoder module obtaining the predicted semantic segmentation mask image is as follows: Original image feature F original First, it goes through four upsampling stages, and then through a final projection layer to restore the size of the features to obtain semantic segmentation Logits, and then the semantic segmentation Logits are post-processed to obtain the predicted semantic segmentation mask image; Among them, the first upsampling stage only needs to go through 2 visual state space blocks, the second and third upsampling stages need to go through image block expansion operation and 2 visual state space blocks in turn, and the fourth upsampling stage needs to go through image block expansion operation and 1 visual state space block, where the image block expansion operation is used to increase the height and width of the feature and reduce the number of channels; the post-processing operation applied to the semantic segmentation Logits is specifically as follows: first, the semantic segmentation Logits are passed through the Sigmoid function so that the value of each pixel is scaled to the probability range of 0 to 1, and the predicted semantic segmentation probability image is obtained, and then the category selection is performed with 0.5 as the threshold, and pixels with values ​​greater than 0.5 are assigned to the foreground category, otherwise they are assigned to the background category, so that the predicted semantic segmentation mask image is obtained.

6. A medical image segmentation guided classification system for disease diagnosis according to claim 5, characterized in that: The specific operation process of the skip connection module into which the feature refinement module is inserted is as follows: The features outputted from the third, second and first downsampling stages of the encoder module are respectively refined by the feature refinement module, and then element-wise summed with the features outputted from the first, second and third upsampling stages of the decoder module, so as to obtain the input features accepted by the second, third and fourth upsampling stages of the decoder module respectively; Among them, the feature refinement module is improved on the basis of the Gate Unit of the GateNet network. The feature refinement module replaces one of the global average pooling in the Gate Unit with the global maximum pooling. The combination of global average pooling and global maximum pooling can more effectively enhance the selectivity of the feature control process from different focuses. In addition, the feature refinement module also replaces the ordinary convolution layer in the Gate Unit with a depth-separable convolution layer, which can reduce the amount of calculation while ensuring that the improved VM-UNet network has a high accuracy.

7. The medical image segmentation guided classification system for disease diagnosis according to claim 6, characterized in that: The feature refinement module uses the features X∈R obtained at different downsampling stages of the encoder module. C×H×W As input, R represents a real number set, C represents the number of channels, H represents the height, and W represents the width. Through a 3×3 depth-separable convolution operation, the following process is performed: first, the input feature X is divided into two feature maps f1 and f2, so that the feature refinement module can focus on the effective information from different representation subspaces, and then the Sigmoid function is used to normalize f1 and f2 to the interval [0,1] to obtain intermediate features f'1 and f'2. A pair of refined weight values ​​R1 and R2 are calculated by using global average pooling and global maximum pooling for the intermediate features f'1 and f'2. Finally, R1 and R2 are used to select context information from the input features respectively, and the context information is fused by element-by-element summation and then subjected to a 3×3 depth-separable convolution operation to obtain the output feature Y∈R C×H×W The feature refinement module can selectively send features from the encoder module to the decoder module for the element-by-element summation process of the skip connection module, which can solve the problem that the features of the encoder module transmitted to the decoder module through the skip connection module contain redundant or misleading information; The formula for the above process is as follows: In the formula, GAP represents global average pooling, GMP represents global maximum pooling, and DSConv 3×3 represents a 3×3 depthwise separable convolution operation, represents the Hadamard product, Represents element-by-element addition.

8. The medical image segmentation guided classification system for disease diagnosis according to claim 7, characterized in that: The explicit segmentation guided classification module specifically performs the following operations: After the original image and the predicted semantic segmentation mask image are input into the explicit segmentation guided classification module, since the predicted semantic segmentation mask image directly reflects the approximate shape of the lesion area visually and contains rich lesion shape information, the value of each pixel of the predicted semantic segmentation mask image is multiplied by 255 and then concatenated with the original image in the channel dimension to obtain a shape feature image containing four channels. At the same time, the Hadamard product is calculated using the original image and the predicted semantic segmentation mask image, that is, the pixel value of the background part of the original image corresponding to the predicted semantic segmentation mask image is changed to 0, and the other pixel values ​​remain unchanged, thereby obtaining a texture feature image, in which irrelevant background pixels are shielded, and the texture detail information inside the lesion is highlighted. The shape feature image and texture feature image are respectively input into the convolution module designed in the explicit segmentation guided classification module to extract the shape perception feature F shape and texture-aware features F texture ; The convolution module consists of five stages, each of which undergoes the following operations in sequence: f i+1 =MaxPool(ReLU(BN(Conv 3×3 (f i )))),i=1,2,3,4,5 In the formula, f i represents the input of the i-th stage of the convolution module, f i+1 Represents the output of the i-th stage of the convolution module, Conv 3×3 represents a 3×3 convolutional layer, BN represents batch normalization, ReLU is the activation function, and MaxPool represents the maximum pooling layer with a pooling kernel size and a stride of 2.

9. The medical image segmentation guided classification system for disease diagnosis according to claim 8, characterized in that: The original image feature F obtained in the encoder module original The shape-aware features F obtained in the explicit segmentation-guided classification module shape and texture-aware features F texture It will be input into the feature fusion and disease classification module to obtain the predicted disease category, and the following operations are performed: 1) The original image feature F original , shape perception feature F shape and texture-aware features F texture The channel attention module in the feature fusion and disease classification module is input together to obtain the enhanced feature F' original 、F' shape 、F' texture ; Among them, the channel attention module is improved on the basis of the squeeze and excitation module of the SE-Net network. The channel attention module further introduces global maximum pooling as another squeeze operation based on the global average pooling selected for the squeeze operation in the squeeze and excitation module of the SE-Net network, and captures the overall feature parameters of each channel representing the feature map from two different angles of global average pooling and global maximum pooling. The specific definition of the channel attention module is as follows: p=Flatten(Concat(GAP(F),GMP(F))),F=F original ,F shape ,F texture po=View(Sigmoid(Linear(ReLU(Linear(p))))) F'=F' original ,F' shape ,F' texture Where F∈R C×H×W As input, the value is F original 、F shape or F texture , Concat means concatenation in the channel dimension, Flatten means flattening a multidimensional tensor into one dimension, Sigmoid is the activation function, Linear means the linear layer, and View means adjusting the shape of the tensor to R C×1×1 , po represents the channel attention weight value, represents the Hadamard product, F'∈R C×H×W As output, the value is F' original 、F' shape or F' texture ; 2) Let F' original 、F' shape 、F' texture Feature fusion is achieved by splicing in the channel dimension, and then the fused features are passed through the classification head to obtain the disease classification prediction probability Prob, and then the category with the largest prediction probability value among the disease classification prediction probability Prob is selected as the predicted disease category; The formula for obtaining the disease classification prediction probability Prob is as follows: F fusion =Concat(F' original ,F' shape ,F' texture ) Prob=Sigmoid(Linear(ReLU(Linear(Flatten(GAP(F fusion )))))) In the formula, F fusion Represents the fused features.

10. The medical image segmentation guided classification system for disease diagnosis according to claim 9, characterized in that: The improved VM-UNet network performs the loss function L of the segmentation task Seg And the loss function L of the classification task Cls The total loss function L is obtained by linearly combining the dynamic and learnable weights Total , L Total Used to guide parameter optimization to collaboratively improve the accuracy of the improved VM-UNet network in segmentation and classification tasks; the total loss function L Total The calculation process includes the following steps: 1) Calculate the loss function L of the segmentation task based on the predicted semantic segmentation probability image obtained in the decoder module and the real semantic segmentation mask image obtained in the data import module Seg , the formula is as follows: L Seg =w b ×L BCE +w d ×L Dice Where, L BCE is the binary cross entropy loss function, L Dice is the Dice loss function, w b YesL BCE The weight, w d YesL Dice The weight, w b and w d The value of is 1, N is the total number of pixels, y i is the i-th pixel value in the true semantic segmentation mask image, p i is the pixel value at the corresponding position in the predicted semantic segmentation probability image, and ε is a smoothing term close to 0, which is used to prevent the denominator from being zero and improve numerical stability; 2) Calculate the loss function L of the disease classification task based on the disease classification prediction probability Prob obtained in the feature fusion and disease classification module and the actual disease category label Label obtained in the data import module Cls , using the cross entropy loss function, where L Cls is defined as follows: In the formula, c is the number of categories in the disease classification task, Label i Represents the one-hot encoding value of the i-th category, Prob i Represents the predicted probability of the i-th category; 3) Total loss function L Total The weights of the lesion semantic segmentation task and the disease classification task are dynamically adjusted based on homoscedastic uncertainty. The specific definitions are as follows: In the formula, σ1 and σ2 are learnable noise parameters, which will be automatically adjusted during the training of the improved VM-UNet network. They are used to represent the uncertainty of the two tasks. The larger the uncertainty of the task, the larger the noise parameter, and the smaller the weight value of the loss function of the corresponding task; When the improved VM-UNet network is in the training stage, the L calculated above Total It will be further used to calculate the gradient, and its gradient value will be used by the AdamW optimizer to update the parameters of the improved VM-UNet network during the back propagation process, that is, to guide parameter optimization to collaboratively improve the accuracy of the improved VM-UNet network in segmentation and classification tasks. After multiple iterations to L Total The value of is the smallest, and finally the optimal improved VM-UNet network is obtained.

Images