Medical image classification method based on deep convolution and Swin Transform hybrid SSM model

Abstract

The invention relates to a medical image classification method based on a deep convolution and SwinTransform hybrid SSM model, and aims to realize accurate classification of medical images by effectively combining a deep convolutional neural network (CNN), a state space model (SSM) and a Swinin Transform. According to the method, local detail modeling is carried out on an input image through continuous multi-core depth separable convolution, and deep discrimination features are extracted; thirdly, further enhancing the feature extraction capability through a depth separable convolution and 2D selective scanning module (SS2D), retaining a global receptive field, establishing a remote dependency relationship, and capturing remote detail features; and finally, global feature modeling is carried out through a Swin Transform model, global detail information of the image is focused, and more sufficient image feature extraction is ensured. Experiments on a plurality of public medical data sets and private data sets show that compared with an existing medical image classification method, the method has higher classification precision and better performance, and especially has remarkable advantages in the aspects of medical image fine-grained feature extraction and long-distance dependency relationship modeling.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and medical image processing technology, specifically to a medical image classification method based on a hybrid model of depthwise separable convolution (DSC), a state-space model (SSM), and a SwinTransformer (MambaDSCST). This method is suitable for computer-aided diagnosis (CAD) scenarios and can efficiently handle the precise classification of medical images such as CT, MRI, and X-rays. It is particularly adept at extracting fine-grained local features and establishing long-range global dependencies in complex medical images. Background Art

[0002] Medical image classification is one of the core tasks of clinical diagnosis, but it faces the following technical bottlenecks:

[0003] 1. Insufficient local feature extraction: Traditional convolutional neural networks (CNNs) are limited by the local receptive field and have difficulty capturing subtle pathological features in images (such as tumor edges and vascular lesion textures).

[0004] 2. Low global modeling efficiency: Although Transformer-based models can model long-distance dependencies, the computational complexity of their self-attention mechanism grows quadratically with the image size, making them difficult to apply to high-resolution medical images.

[0005] 3. Noise sensitivity: Medical images often contain interference such as noise and artifacts. Existing models are not robust enough to noise, resulting in limited classification accuracy.

[0006] Limitations of existing solutions:

[0007] 1. Limitations of a single model: CNN models such as ResNet and ConvNeXt excel in local feature extraction but have weak global modeling capabilities. Models such as ViT and Swin Transformer, while capable of global perception, are computationally expensive and lack sensitivity to local details.

[0008] 2. Hybrid architecture defects: Existing hybrid models (such as CNN+Transformer) mostly adopt a serial design, resulting in insufficient interaction between local and global features and failing to effectively solve the problem of computational efficiency.

[0009] Therefore, there is an urgent need for a medical image classification method that can take into account local detail extraction, global efficient modeling and noise robustness. Summary of the Invention

[0010] 1. Purpose of the Invention

[0011] To solve the above problems, the present invention proposes a hybrid model based on SSM (MambaDSCST), which realizes local detail extraction, long-distance dependency modeling and global feature fusion of medical images by fusing deep separable convolution, state-space model and Swin Transformer, thereby improving classification accuracy and computational efficiency.

[0012] 2. Technical solution

[0013] The MambaDSCST model proposed by the method of the present invention includes the following core modules:

[0014] (1) Patch embedding layer:

[0015] The input medical image Divide into non-overlapping image patches of size 4×4, map to the high-dimensional feature space through a 4×4 convolution kernel (step size 4), output feature dimension C=96, and obtain the embedded image features The mathematical expression is:

[0016] x′=Conv2D(x; kernel=4, stride=4, channels=C)

[0017] (2)DSConv-SSM-SwinT module

[0018] This module consists of convolution branch, SSM branch, feature fusion and Swin Transformer module:

[0019] Convolution branch:

[0020] The input features are modeled using continuous multi-kernel depthwise separable convolutions for local feature modeling. The specific steps are as follows:

[0021] 1.3×3 depth convolution: extract local features, the formula is:

[0022] F dw =DepthwiseConv(x;kernel=3)

[0023] 2. Batch normalization and activation: Enhance feature stability. The formula is:

[0024] F bn =ReLU(BN(F dw ))

[0025] 3.1×1 point-by-point convolution: Fusion of channel information, the formula is:

[0026] F conv =PointConv(F bn; kernel = 1)

[0027] SSM Branch:

[0028] The input features are divided into two paths after layer normalization (LayerNorm):

[0029] 1. Linear projection and activation:

[0030] F linear =SiLU(W1·LayerNorm(x))

[0031] 2. Depthwise Separable Convolution and SS2D Module:

[0032] Depthwise separable convolution: extract local features, the formula is:

[0033] F dw =DepthwiseConv(LayerNorm(x);kernel=3)

[0034] 2D Selective Scan Module (SS2D):

[0035] Establishing long-range dependencies through selective scanning mechanisms:

[0036] F SS2D =SS2D(F dw ;scan_direction=[row,col])

[0037] Among them, the discretized state space equation of the SS2D module is:

[0038]

[0039] y(t)=Ch(t)

[0040] 3. Branch fusion: The two outputs are multiplied element by element. The formula is:

[0041] F SSM =F linear ⊙SiLU(F SS2D )

[0042] Feature fusion and global modeling:

[0043] The outputs of the convolution branch and the SSM branch are concatenated and input into the Swin Transformer Block:

[0044] F fusion =SwinT(Concat(F conv , F SSM ))

[0045] Swin Transformer uses a sliding window self-attention mechanism (W-MSA / SW-MSA), and the attention weight is calculated as:

[0046]

[0047] Where Q, K, V are query, key, and value matrices, d k is the dimension scaling factor.

[0048] (3) Hierarchical feature downsampling

[0049] Each level compresses the spatial size (halved) and expands the number of channels (doubled) through a 2×2 patch merging layer:

[0050] x down =PatchMerging(x)=Conv2D(x; kernel=2, stride=2, channels=2C)

[0051] 3.Technological advantages

[0052] Efficient long-distance modeling: The SSM branch establishes global dependencies with linear complexity, overcoming the Transformer computational bottleneck.

[0053] Local-global feature collaboration: Depthwise separable convolution is combined with Swin Transformer to balance local details and global semantics;

[0054] Lightweight design: Depthwise separable convolution and parameter sharing strategies significantly reduce the number of model parameters, making it suitable for high-resolution medical image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Flowchart of the present invention;

[0056] Figure 2 This is a schematic diagram of the overall architecture of MambaDSCST;

[0057] Figure 2 (a): The overall model process, including the patch embedding layer, the 4-level DSConv-SSM-SwinT module and the classification head;

[0058] Figure 2 (b): Dual-branch structure of the DSConv-SSM-SwinT module;

[0059] Figure 2 (c): Internal structure of the SSM branch, showing the SS2D and feature mixing process;

[0060] Figure 2 (d); Deep convolution module structure;

[0061] Figure 2 (e): Swin Transformer’s sliding window self-attention mechanism and layer-wise downsampling design;

[0062] Figure 3 This is an example of the experimental dataset;

[0063] Figure 4(a): Comparison of ablation test results of different models on the ISIC 2018 dataset, verifying the contribution of each module to classification performance based on performance;

[0064] Figure 4(b): Comparison of ablation test results of different models on the Kvasir dataset, verifying the contribution of each module to classification performance based on performance; Figure 4(c): Comparison of ablation experiment results of different models on the PN-CX dataset, verifying the contribution of each module to classification performance based on performance; Figure 4(d): Comparison of ablation test results of different models on the Brain Tumour dataset, verifying the contribution of each module to classification performance based on performance; Figure 4(e): Comparison of ablation test results of different models on the Brain Tumour Plus dataset, verifying the contribution of each module to classification performance based on performance; Figure 4(f): Comparison of ablation test results of different models on the Brain Tumour Plus-6 dataset, verifying the contribution of each module to classification performance based on performance; Figure 5 : Performance comparison of different algorithms on seven-category tasks on the ISIC 2018 dataset; Figure 6 : Performance comparison of different algorithms for eight-category classification tasks on the Kvasir dataset; Figure 7 : Performance comparison of different algorithms for three-category classification tasks on the PN-CX dataset; Figure 8 : Performance comparison of different algorithms for four-category classification tasks on the Brain Tumour dataset; Figure 9 : Performance comparison of different algorithms for four-category classification tasks on the Brain Tumour Plus dataset; Figure 10 : Performance comparison of different algorithms for six-category classification tasks on the Brain Tumour Plus-6 dataset. DETAILED DESCRIPTION

[0065] The present invention will be described in further detail below with reference to the accompanying drawings.

[0066] like Figure 1 The specific steps are as follows:

[0067] Step s1. Data preprocessing

[0068] The input medical image is uniformly scaled to 224×224 pixels and normalized:

[0069] μ=0.5,σ=0.5

[0070] Data augmentation strategy: random horizontal flipping, rotation (±15°) and brightness adjustment (±10%).

[0071] Step s2. Model construction

[0072] like Figure 2 As shown, the overall process of the model is as follows:

[0073] Patch Embedding Layer:

[0074] Input image By mapping to C=96 dimensional feature space through 4×4 convolution kernel (step size 4), the output feature map size is 56×56×96.

[0075] DSConv-SSM-SwinT module:

[0076] Each module consists of the following steps:

[0077] 1. Convolution branch: 3×3 depthwise convolution → batch normalization → ReLU → 1×1 pointwise convolution;

[0078] 2.SSM branch:

[0079] The input features are normalized by the layer and processed in two parallel ways:

[0080] Path 1: Linear projection to C dimension → SiLU activation;

[0081] Path 2: 3×3 depthwise separable convolution → SS2D module → SiLU activation;

[0082] The two outputs are multiplied element by element and then merged;

[0083] 3. Feature fusion: Concatenate the outputs of the convolution branch and the SSM branch and input them into the Swin Transformer Block.

[0084] Layer-wise downsampling: Each level passes through a 2×2 patch merging layer, halving the spatial size and doubling the number of channels.

[0085] Step s3. Training and optimization

[0086] Loss function: Cross entropy loss:

[0087]

[0088] Optimizer: Adam, parameter settings:

[0089] Learning rate = 0.001, β1 = 0.9, β2 = 0.999, weight decay = 10 -4

[0090] Training strategy: batch size 64, training for 500 epochs, and using early stopping strategy to prevent model overfitting.

[0091] Step s4. Quantitative results analysis

[0092] The dataset used in this experiment is Figure 3 Shown, including:

[0093] ISIC 2018 dataset: This dataset is released by the International Skin Imaging Collaboration (ISIC) and contains 10,015 images in 7 different categories, including dermatofibroma (115 cases), vascular lesions (142 cases), actinic keratosis (327 cases), basal cell carcinoma (514 cases), benign keratosis (1099 cases), melanoma (1113 cases) and melanocytic nevus (6705 cases). All images are 650x450 pixels in size. For example, Figure 3 As shown in row 4.

[0094] Kvasir dataset: This dataset was collected using the Vestre Viken endoscopy facility in Norway and annotated and verified by several experienced endoscopists. The dataset is divided into 8 different categories, with 500 images in each category, including gastrointestinal pathologies (esophagitis, polyps, ulcerative colitis), anatomical landmarks (normal cecum, normal pylorus, normal z-line), and two other categories (stained resection margins, stained polyps). The resolution of the dataset ranges from 720x576 to 1920x1072. For example, Figure 3 As shown in row 5.

[0095] Pneumonia-Normal Chest X-Ray Images (PN-CX) dataset: This data is divided into three categories: 1802 normal chest X-rays, 1626 COVID-19 infected chest X-rays, and 1800 viral pneumonia chest X-rays. The COVID-19 infected chest X-rays were collected from multiple public resources such as Github, German Medical School, and SIRM, while the normal chest X-rays and viral pneumonia chest X-rays were collected from Kaggle's "Chest X-Ray Images" database. The pixels of all its images are adjusted to 256x256, for example, Figure 3 As shown in row 1.

[0096] Brain Tumour dataset: This dataset comes from Kaggle's brain tumor classification database, with a total of 3264 MRI images, including 4 different categories, including 926 gliomas, 901 pituitary tumors, 937 meningiomas, and 500 non-tumor images. The dataset resolution is adjusted to 256x256, for example Figure 3 As shown in row 2.

[0097] Brain Tumour Plus Dataset: Radiologists at Jiangsu Provincial People's Hospital have verified that the non-tumor images in the Brain Tumour dataset are from different MRI scan sequences than the other three categories, which may affect classification performance. The Brain Tumour Plus dataset, based on the Brain Tumour dataset, includes additional non-tumor images from the Department of Radiology at Jiangsu Provincial People's Hospital, from the same sequences as the other three categories. The dataset is doubled in size, with 1,852 images of gliomas, 1,802 of pituitary tumors, 1,874 of meningiomas, and 1,000 of non-tumor images. The dataset resolution is adjusted to 256x256.

[0098] Brain Tumour Plus-6 dataset: Based on the Brain Tumour Plus dataset, this dataset also imports acoustic neuroma and lymphoma images from the Department of Radiology of Jiangsu Provincial People's Hospital, which are of the same sequence as the other four categories. There are 1850 acoustic neuroma images and 1850 lymphoma images. The dataset resolution is adjusted to 256x256, as shown in the following example: Figure 3 As shown in row 3.

[0099] Based on the characteristics of medical images, this paper uses accuracy, precision, recall, and F1 score as evaluation indicators for the model. The values ​​of these indicators can reflect the performance of the model. The definitions of the indicators are as follows:

[0100]

[0101] Where TP, FP, TN, and FN represent the number of true positive cases, false positive cases, true negative cases, and false negative cases, respectively. The values ​​of all indicators are in [0, 1], and larger values ​​mean better performance of the evaluation model.

[0102] To demonstrate the effectiveness of the model, several representative benchmark algorithms were selected for comparison, including CNN-based algorithms ResNet, ConvNeXt, and VGG-19, and Transformers-based algorithms ViT-B and Swin-T.

[0103] like Figure 5-8As shown in the figure, the proposed method outperforms the baseline algorithm in image classification tasks on public datasets of different categories. Specifically, on the relatively simple three-category PN-CX dataset and the four-category BrainTumour dataset, the proposed method improves the accuracy, F1 score, precision, and recall by 2.53%, 1.55%, 1.47%, and 1.63%, and by 2.36%, 1.53%, 1.35%, and 1.68%, respectively, compared to the best-performing Swin-T algorithm in the benchmark. On the more complex seven-category ISIC 2018 dataset, the proposed method improves the accuracy, F1 score, precision, and recall by 1.99%, 0.73%, 1.08%, and 0.38%, respectively, compared to the best-performing Swin-T algorithm in the benchmark. In the eight-category Kvasir dataset with the most complex categories, the proposed method still showed excellent performance, and compared with the best performing VGG-19 algorithm in the benchmark, the accuracy, F1 score, precision and recall were improved by 1.94%, 1.32%, 1.37% and 1.29% respectively. When performing image classification tasks on private datasets of different categories, the proposed method still outperformed the benchmark algorithm in terms of performance. Figure 9-10 As shown in the figure, the method of the present invention improves the accuracy, F1 score, precision and recall by 2.17%, 1.54%, 0.94% and 2.15% and 2.09%, 1.46%, 1.53% and 1.39% respectively on the four-category Brain Tumour Plus dataset and the six-category Brain Tumour Plus-6 dataset compared with the best performing Swin-T algorithm in the benchmark.

[0104] The advantages of the method of the present invention are mainly due to the powerful feature extraction capability of MambaDSCST. As a sub-branch of MambaDSCST, it locally models the input features through continuous multi-core depthwise separable convolution, aiming to focus on local details of medical images, extract deep discriminant features, and use parameter sharing to minimize the amount of computation. As another branch of MambaDSCST, SSM uses depthwise separable convolution and 2D selective scanning module SS2D to further extract features, aiming to preserve the global receptive field and establish long-distance dependencies to capture remote detail features. As the final step of MambaDSCST, the Swin Transformer model performs global feature modeling on the mixed features obtained from the two branches, aiming to focus on more global detail information and extract more comprehensive image features.

[0105] contrast Figure 8 and Figure 9It can be found that standardized data formats and abundant data volumes can improve the effectiveness of medical image classification. Specifically, on the four-category Brain Tumour Plus dataset, the accuracy, F1 score, precision, and recall of our method were improved by 2.4%, 1.36%, 0.86%, and 1.88% respectively, compared to the four-category Brain Tumour dataset.

[0106] Step s5. Ablation experiment

[0107] To further validate the reliability of our model, we conducted systematic ablation experiments, as shown in Figure 4. We denoted the new model combining Depthwise Separable Convolution with Swin Transformer as DSC-SwinT (dark red bars); the new model combining SSM with Swin Transformer as SSM-SwinT (yellow bars); the new module combining Depthwise Separable Convolution and SSM in parallel as DSC-SSM (blue bars); and our model as Ours (purple bars). We conducted experiments on six datasets. The results, shown in Figure 4, show that our model performed best in terms of accuracy, F1 score, precision, and recall. The DSC-SSM model ranked second, the SSM-SwinT model ranked third, and the DSC-SwinT model performed the worst. This demonstrates the effectiveness of the SSM model, which can capture detailed remote features by establishing long-range dependencies, thereby improving model performance. Furthermore, combining it with the local modeling capabilities of CNNs can yield even greater advantages.

[0108] This paper proposes a MambaDSCST model based on the SSM and explores its feasibility for classifying different types of medical images. To evaluate the model's performance on medical image classification tasks, this study conducted extensive experiments and compared it with state-of-the-art classification algorithms. The experimental results demonstrate that the proposed model achieves excellent results on classification tasks, opening up a new perspective for image-based CAD systems and further promoting their application.

Claims

1. A medical image classification method based on a hybrid SSM model of deep convolution and Swin Transformer, characterized in that: The following steps are involved: Step S1: Input medical image Where H is the image height, W is the image width, and C is the number of channels. The image is divided into non-overlapping 4×4 image blocks through the patch embedding layer to generate the initial feature map Where D is the embedding dimension; Step S2: Input F0 into two independent branches in parallel: Convolution branch: extracts local detail features through a local feature extraction module composed of continuous multi-core depth-separable convolutions Its operation is defined as: in, is the kth point-by-point convolution kernel, D k represents the depth convolution kernel, is a depth-wise separable convolution operation, K is the number of convolution kernels; SSM branch: extracts global dependency features through a global feature enhancement module consisting of depthwise separable convolution and 2D selective scanning module (SS2D) Establish long-distance dependency, and its discretized state space model is: in, is the diagonal state matrix, and represents the projection parameter. Δ is the learnable time scale parameter, ⊙ represents element-wise multiplication, x t Represented as the t-th position feature of the input sequence, h t is the intermediate implicit state; Step S3: Output the convolution branch F local With SSM branch output F SSM Splice by channel dimension to generate mixed features The formula is: F mix =Concat(F local ,F SSM ) Step S4: F mix Input the Swin Transformer module for multi-scale global modeling, and its sliding window self-attention is calculated as: in: Q=F mix W Q , K=F mix W K , V=F mix W V , W Q , W K , W V is the learnable projection matrix, is the relative position bias matrix, M=7 is the window size; Step S5: Output the prediction result through the classification head, and its probability distribution is: P(y|x)=Softmax(W cls ·GAP(F final )) Among them: GAP(·) is the global average pooling operation, and the output dimension is is the classification weight matrix, and C is the total number of categories.

2. The method according to claim 1, characterized in that The convolution branch of the continuous multi-core depthwise separable convolution in step S2 includes three layers of operations: First layer: The depthwise convolution kernel size is 3×3, the stride is 1, the fill mode is "same", and the number of output channels is D; Second layer: the point-by-point convolution kernel size is 1×1 and the number of output channels is D; The third layer uses batch normalization (BatchNorm) and ReLU activation function, and the formula is expressed as:

3. The method according to claim 1, characterized in that The SSM branch in step S2 adjusts parameters by dynamic path selection: The input feature F0 is linearly projected to generate Δ, A, B, C, and the formula is: [Δ, A, B, C] = Linear(F0) Intermediate implicit state h t Initialization uses zero padding, and the sequence scanning direction is alternating between row priority and column priority.

4. The method according to claim 1, wherein The Swin Transformer module hierarchy in step S4 is designed as follows: Phase 1: Input It is calculated by alternating between local window self-attention (W-MSA) and moving window self-attention (SW-MSA); Phase 2: Reduce the resolution to The number of channels increases to 4D, and the formula is: F stage2 =PatchMerging(F stagel ) PatchMerging concatenates adjacent 2×2 feature blocks and linearly projects them. Phase 3: Resolution reduced to The number of channels increases to 8D, and the global features are finally output 5. The method according to claim 1, wherein The method uses the cross entropy loss function during training:

6. The method according to claim 1, wherein The method adopts the following optimization strategy during training: The optimizer is Adam, the learning rate is η = 0.001, the parameters β1 = 0.9, β2 = 0.999, and the weight decay coefficient λ = 10 -4 , batch size B = 64, and adopt the early stopping strategy: if the validation loss does not decrease for 10 consecutive rounds, the training is terminated to prevent overfitting.

Images