A multimodal medical image classification method based on improved DenseNet

Abstract

This invention discloses a multimodal medical image classification method based on an improved DenseNet. The method constructs a medical image classification network comprising a preliminary feature extraction module, a feature refinement module, and a classification module. The feature refinement module combines the dense inter-layer connectivity of DenseNet with the efficient feature processing capabilities of the Mamba module to extract and fuse complex image features. The classification module, using fully connected layers and a Softmax function, is responsible for converting the rich image feature representations into specific image classification results. By incorporating a transition layer with the Mamba module, the invention enhances the model's ability to capture key image features. This method, while maintaining the original advantages of DenseNet, further improves the classification accuracy of 3D medical images, providing strong technical support for the application of deep learning in medical image processing.

Description

Technical Field

[0001] The present invention relates to the field of 3D medical image classification and deep learning technology, and in particular to a method based on improved Multimodal medical image classification method. Background Art

[0002] With the rapid development of medical information technology and artificial intelligence, the role of medical images in disease diagnosis, treatment planning, and prognosis assessment is becoming increasingly prominent. From X-rays, CT scans, MRIs, to ultrasound images, various medical images provide a wealth of diagnostic information for the medical field. Medical image classification, a core task in medical image analysis, uses automated techniques to classify and interpret images, not only improving diagnostic efficiency but also providing important support for precision medicine. However, despite recent progress in this field, technical bottlenecks and practical challenges remain.

[0003] Medical image classification requires researchers to possess both a background in medical image analysis and digital imaging. Existing model approaches, such as traditional machine learning recognition methods and deep convolutional neural network-based methods, have limited accuracy and generalization due to image scale, data format, and data category distribution. In practice, the availability of labeled medical image data is limited due to the high cost of labeling medical image data and the challenges of collecting case medical images.

[0004] The limited availability of high-quality annotated medical image datasets, particularly for research on specific diseases or rare conditions, limits the training and generalization capabilities of models. This data scarcity makes it difficult for many models to achieve their intended results in real-world applications, especially when processing images from different patients or with different devices.

[0005] Furthermore, most current mainstream medical image classification methods rely solely on single-modality medical data. While they have achieved some success in specific tasks, these methods have significant limitations and struggle to meet the demands of complex clinical scenarios. While single-modality medical images can provide valuable information under certain conditions, they often fail to fully reflect the complexity and diversity of lesions. For example, CT images excel in clearly displaying skeletal structures but are relatively inadequate in capturing the details of soft tissues. MRI, on the other hand, has advantages in soft tissue imaging but is relatively weak in its ability to display skeletal structures. This limitation of single-modality images forces doctors to rely on multiple images for comprehensive judgment when making diagnoses, increasing diagnostic complexity and the risk of error.

[0006] Images from different modalities can provide complementary information, thus providing doctors with a more comprehensive basis for diagnosis. However, fully utilizing multimodal medical image data presents a significant challenge. The richness of multimodal data is often not effectively utilized. This limitation not only reduces the model's ability to identify complex lesions but can also lead to the omission of important information, thus affecting the accuracy of clinical decision-making. Summary of the Invention

[0007] The present invention provides a method based on improved image classification technology to overcome the problems existing in the existing image classification technology in medical image classification. In order to more comprehensively understand the information in multimodal medical images and improve the accuracy and efficiency of image classification, a multimodal medical image classification method was proposed.

[0008] The present invention adopts the following technical solutions to solve the above problems:

[0009] The present invention is based on an improved The multimodal medical image classification method is characterized by comprising the following steps:

[0010] Step 1: Get the image size And the mode number is The multimodal image set of a certain user is processed and image registration and bias field correction are performed to obtain a pre-processed multimodal medical image set of a certain user. = ,in, Indicates the A medical image of a modality, ; represents the width of a single-modality medical image, represents the height of a single-modality medical image, represents the depth of a single-modality medical image; let The true category label is recorded as and 1,2,…, }, Indicates the total number of categories;

[0011] Step 2: Construct a medical image classification network, including: preliminary feature extraction module, feature refinement module and classification module;

[0012] Step 2.1, the preliminary feature extraction module includes a convolutional layer and an activation function;

[0013] Will After cascading medical images of different modalities, we get Cascaded medical images And input into the preliminary feature extraction module for processing to obtain the shallow feature map ,in, is the number of channels of a single-modality medical image;

[0014] Step 2.2: The feature refinement module processes the dense features output by the Nth dense block. ;

[0015] Step 2.3, the classification module includes a fully connected layer and function, and then After processing, the output The predicted class label of ;

[0016] Step 3: Train the medical image classification network;

[0017] Step 3.1: Use formula (1) to construct the cross entropy loss function :

[0018] (1)

[0019] In formula (1), express middle In the The predicted probability of each category; express middle In the The true probability of each category;

[0020] Step 3.2: Use the back propagation algorithm to train the medical image classification network and calculate the cross entropy loss function. To update the network parameters until the cross entropy loss function Until convergence, the optimal medical image classification network model is obtained, which is used to perform classification prediction on any input multimodal medical image.

[0021] The present invention is based on an improved The multimodal medical image classification method is also characterized in that the feature refinement module in step 2.2 includes: dense blocks and with The transition layer of the enhancement module; where the nth dense block contains dense layers; any The dense layers include: two normalization layers, two activation functions, and two convolutional layers;

[0022] Any with The transition layer of the enhancement module contains the convolutional dimension reduction layer and Enhancement layer, where the convolutional dimension reduction layer contains a normalization layer, activation function, a convolution layer and a pooling layer; the Mamba enhancement layer contains a linear mapping layer, a convolution layer, a selective state space model layer and activation function;

[0023] Step 2.2.1, initialize n=1; As the output feature of the n-1th dense block ;

[0024] Step 2.2.2, Initialization =1;

[0025] Step 2.2.3, Input into the nth dense block and pass through the After processing the dense layers, we get the nth dense block The hierarchical features output by the dense layer ;

[0026] Step 2.2.4, and After superposition, assign to ,Will +1 assigned to Then return to step 2.2.3 and execute sequentially until So far, the dense features of the nth dense block output are obtained ={ , ,…, ,…, , };

[0027] Step 2.2.5, Initialization =n; As the first -1 with The fusion features of the transition layer output of the enhancement module ;

[0028] Step 2.2.6, Enter with In the transition layer of the enhancement module, and respectively through the convolutional dimension reduction layer and The processing of the enhancement layer is The extracted features and mapping features, thereby The extracted features and After weighted fusion of the mapping features, we get Fusion features ;

[0029] Step 2.2.7, As , after assigning n+1 to n, return to step 2.2.2 and execute sequentially until n>N, thus obtaining the dense features output by the Nth dense block .

[0030] The electronic device of the present invention includes a memory and a processor, and is characterized in that the memory is used to store a program that supports the processor to execute the multimodal medical image classification method, and the processor is configured to execute the program stored in the memory.

[0031] The present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program executes the steps of the multimodal medical image classification method when executed by a processor.

[0032] Compared with the existing classification methods, the beneficial effects of the present invention are embodied in:

[0033] 1. This invention utilizes 3D multimodal medical image data: 3D images used in this method provide richer spatial information than traditional 2D images, enabling the model to accurately capture complex details of brain structure and early-stage lesions. Combining magnetic resonance imaging (MRI) and positron emission tomography (PET) data captures not only structural information but also functional and metabolic information. Leveraging multimodal images significantly enhances insight into disease characteristics, thereby improving image classification accuracy and model robustness.

[0034] 2. This invention introduces a dropout mechanism: This method incorporates a dropout mechanism. Specifically, after generating new features in a dense layer, dropout randomly discards some feature channels with a certain probability. Its primary benefit is preventing overfitting. By randomly omitting some neurons, the network becomes less reliant on specific paths or features, thereby enhancing the model's generalization capabilities. Furthermore, it significantly improves model performance during validation and testing, particularly when working with complex datasets, helping to prevent overfitting and improving practical application effectiveness.

[0035] 3. The present invention is Customized optimization of the structure: by adjusting The number of dense layers within the dense blocks in the architecture effectively reduces the consumption of computing resources while improving the representation ability of the model, especially showing superiority when processing high-dimensional data.

[0036] 4. The present invention introduces Modules: The module reduces computational requirements by dividing the subsequences and performing independent processing, which helps reduce the number of parameters in the overall model and makes the network more efficient in processing large-scale 3D medical image data. The module enhances the network's ability to process complex image data and optimizes the efficiency and accuracy of feature extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Based on the improvement of the present invention Flowchart of the multimodal medical image classification method;

[0038] Figure 2 It is a specific framework schematic diagram of the present invention, with For example;

[0039] Figure 3 Schematic diagram of the specific structure of the dense layer of the present invention;

[0040] Figure 4 Schematic diagram of dense connection mode of dense layers in dense blocks of the present invention;

[0041] Figure 5 For the present invention Schematic diagram of the specific structure of the module. DETAILED DESCRIPTION

[0042] In this embodiment, taking two different modalities of medical images MRI and PET as examples, the general process is as follows: Figure 1 As shown, based on the improvement The specific network framework of the multimodal medical image classification method is as follows Figure 2 As shown in Figure 2, the addition of dense blocks and Mamba modules significantly improves the model's image classification efficiency and accuracy, providing a reference value for future auxiliary medical treatment applications. Specifically, a method based on improved The multimodal medical image classification method includes the following steps:

[0043] Step 1: Get the image size And the mode number is The multimodal image set of a certain user is processed and image registration and bias field correction are performed to obtain a pre-processed multimodal medical image set of a certain user. = ,in, Indicates the A medical image of a modality, ; represents the width of a single-modality medical image, represents the height of a single-modality medical image, represents the depth of a single-modality medical image; let The true category label is recorded as and 1,2,…, }, Indicates the total number of categories;

[0044] In specific implementation, The dataset comes from the Alzheimer's Disease Neuroimaging Initiative (ADNI), using MRI and PET images from the same user with an interval of no more than one month. All multimodal data are provided in the form of NIfTI files (.nii.gz). The experiment uses 443 sample pairs, namely 443 MRIs and 443 PETs. ​​The training set contains 159 sets of image data pairs of Alzheimer's patients, 130 sets of image data pairs of patients with mild cognitive impairment, and 154 sets of medical image data pairs of normal subjects. Both MRI and PET are grayscale images, and the size of each medical image is ,Right now .

[0045] Both MRI and PET images were processed according to standard procedures. First, skull stripping was performed using the FSL tool to remove non-brain tissue from magnetic resonance imaging (MRI) and positron emission tomography (PET) images to reduce interference from irrelevant imaging features. Next, the images were precisely registered using advanced normalization tools (ANTs). A standard brain template with a 1 mm resolution was used to ensure spatial consistency between different images and eliminate global linear differences (including global translation, scale, and rotation differences). Furthermore, the images were bias-field corrected to eliminate image intensity variations caused by magnetic field inhomogeneities. Finally, to make the image data more suitable for subsequent analysis and processing, maximum and minimum normalization was performed to standardize the image intensity values ​​to a uniform range. This series of meticulous preprocessing steps ensured data quality and provided a solid foundation for subsequent analysis and research.

[0046] Step 2: Construct a medical image classification network, including: preliminary feature extraction module, feature refinement module and classification module;

[0047] Step 2.1, the preliminary feature extraction module contains a convolutional layer and activation function;

[0048] Will After cascading medical images of different modalities, we get Cascaded medical images And input into the preliminary feature extraction module for processing to obtain the shallow feature map ,in, is the number of channels of a single-modality medical image;

[0049] In this implementation case, a convolution layer with a convolution kernel size of 7 is used, and the activation function is Activation function. This larger convolution kernel can extract features over a wider range, increasing the receptive field, thereby capturing richer contextual information and more effectively capturing large-scale structures and patterns in the image. It is particularly suitable for processing input data containing complex and large-scale features. For example, in medical image analysis, a larger convolution kernel can better capture the overall morphology and large-scale spatial distribution of abnormal areas such as tumors, rather than focusing solely on detailed features.

[0050] Step 2.2: Feature Refining Module includes: dense blocks and with The transition layer of the enhancement module; where the nth dense block contains dense layers; any The dense layers include: two normalization layers, two activation functions, and two convolutional layers;

[0051] In specific implementation, , the model structure is as follows Figure 4 As shown in the figure, the four dense blocks contain different numbers of dense layers: the first dense block contains 6 dense layers, the second dense block contains 18 dense layers, the third dense block contains 18 dense layers, and the fourth dense block contains 12 dense layers. Adjusting the number of dense layers in the dense blocks improves the network's feature extraction diversity, optimizes the number of parameters and computational complexity, and improves model efficiency. Furthermore, adjusting the dense layer structure enhances the model's flexibility, better adapting it to medical image classification tasks, and ultimately improving the model's generalization capabilities.

[0052] The dense layers include the first normalization layer, Activation function, the first convolution layer with convolution kernel of 1, the second normalization layer, The activation function and the second convolution layer with a convolution kernel of 3 are as follows: Figure 3 shown.

[0053] The dense layer uses dense connections. Specifically, in the dense block, the The input received by the dense layer of the layer includes both the original features of the initial input of the dense block and the All these features are concatenated together as the output features of all dense layers in the first layer. The output features generated by each layer are appended to the end of the current input features to form new input features, which are then passed to subsequent layers for further processing. This promotes the rapid transmission of information and gradient flow, alleviates the gradient vanishing problem, and facilitates model training. At the same time, the direct reuse of features improves the representation ability and performance of the model. In addition, dense connections make it easier to train deeper networks and improve feature propagation, helping the model better learn complex feature representations.

[0054] Any with The transition layer of the enhancement module contains the convolutional dimension reduction layer and Enhancement layer, where the convolutional dimension reduction layer contains a normalization layer, activation function, a convolution layer and a pooling layer; the Mamba enhancement layer contains a linear mapping layer, a convolution layer, a selective state space model layer and activation function;

[0055] In this embodiment, the convolutional dimensionality reduction layer uses a convolutional layer with a convolution kernel size of 3 and Activation function to further extract features, The specific structure of the reinforcement layer is as follows Figure 5 As shown, the Mamba enhancement layer uses Activation function and depth-wise separable convolution, used in the depth-wise convolution stage The convolution kernel is used in the point-by-point convolution stage. convolution kernel. Depthwise separable convolution significantly reduces the computational complexity and the number of parameters while maintaining high model performance. It greatly improves computational efficiency, reduces memory and video memory usage, and significantly shortens training time while ensuring performance. In addition, the data is deeply processed through the state-space model, and a weight-learnable jump connection strategy is implemented, which effectively maintains the integrity of the original input information and prevents the gradient disappearance problem that is common in multi-layer networks. After completing the state-space processing, the data is again layer-normalized to ensure the consistency of the feature distribution, and finally adjusted to the required output dimension through a linear projection layer. The output data is finally reversed and reshaped to ensure that it is consistent with the input data in spatial structure, providing high-quality feature representation for subsequent processing, so that the model can significantly improve the information utilization and feature expression capabilities when processing data with complex spatial and feature dimensions.

[0056] Step 2.2.1, initialize n=1; As the output feature of the n-1th dense block ;

[0057] Step 2.2.2, Initialization =1;

[0058] Step 2.2.3, Input into the nth dense block and pass through the After processing the dense layers, we get the nth dense block The hierarchical features output by the dense layer ;

[0059] Step 2.2.4, and After superposition, assign to ,Will +1 assigned to Then return to step 2.2.3 and execute sequentially until So far, the dense features of the nth dense block output are obtained ={ , ,…, ,…, , };

[0060] Step 2.2.5, Initialization =n; As the first -1 with The fusion features of the transition layer output of the enhancement module ;

[0061] Step 2.2.6, Enter with In the transition layer of the enhancement module, and respectively through the convolutional dimension reduction layer and The processing of the enhancement layer obtains the refined features and the mapping features accordingly, and then the refined features and the mapping features are weighted and fused to obtain the first Fusion features ;

[0062] Step 2.2.7, As , after assigning n+1 to n, return to step 2.2.2 and execute sequentially until n>N, thus obtaining the dense features output by the Nth dense block ;

[0063] In this example, the outputs of the first three dense blocks are passed as input to the next one with The transition layer of the enhancement block and the output of the fourth dense block are used to perform the final classification task.

[0064] Step 2.3, the classification module contains the fully connected layer and function, and then After processing, the output The predicted class label of ;

[0065] Step 3: Train the medical image classification network;

[0066] Step 3.1: Use formula (1) to construct the cross entropy loss function :

[0067] (1)

[0068] In formula (1), express middle In the The predicted probability of each category; express middle In the The true probability of each category.

[0069] Step 3.2: Use the back propagation algorithm to train the medical image classification network and calculate the cross entropy loss function To update the network parameters until the cross entropy loss function Until convergence, the optimal medical image classification network model is obtained, which is used to classify and predict any input multimodal medical image. In this embodiment, the learning rate during training is set to ,use Optimization, the number of iterations is 200, using The optimizer's total loss Perform optimization and solution; after training, the best image classification network model is obtained.

[0070] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0071] In this embodiment, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are executed.

Claims

1. An improvement based on A multimodal medical image classification method is characterized in that: The steps include: Step 1: Get the image size And the mode number is The multimodal image set of a certain user is processed and image registration and bias field correction are performed to obtain a pre-processed multimodal medical image set of a certain user. = ,in, Indicates the A medical image of a modality, ; represents the width of a single-modality medical image, represents the height of a single-modality medical image, represents the depth of a single-modality medical image; let The true category label is recorded as and 1,2,…, }, Indicates the total number of categories; Step 2: Construct a medical image classification network, including: preliminary feature extraction module, feature refinement module and classification module; Step 2.1, the preliminary feature extraction module includes a convolutional layer and an activation function; Will After cascading medical images of different modalities, we get Cascaded medical images And input into the preliminary feature extraction module for processing to obtain the shallow feature map ,in, is the number of channels of a single-modality medical image; Step 2.2: The feature refinement module processes the dense features output by the Nth dense block. ; Step 2.3, the classification module includes a fully connected layer and function, and then After processing, the output The predicted class label of ; Step 3: Train the medical image classification network; Step 3.1: Use formula (1) to construct the cross entropy loss function : (1) In formula (1), express middle In the The predicted probability of each category; express middle In the The true probability of each category; Step 3.2: Use the back propagation algorithm to train the medical image classification network and calculate the cross entropy loss function. To update the network parameters until the cross entropy loss function Until convergence, the optimal medical image classification network model is obtained, which is used to perform classification prediction on any input multimodal medical image.

2. A method based on improvement according to claim 1 A multimodal medical image classification method is characterized in that: The feature refinement module in step 2.2 includes: dense blocks and with The transition layer of the enhancement module; where the nth dense block contains dense layers; any The dense layers include: two normalization layers, two activation functions, and two convolutional layers; Any with The transition layer of the enhancement module contains the convolutional dimension reduction layer and Enhancement layer, where the convolutional dimension reduction layer contains a normalization layer, activation function, a convolution layer and a pooling layer; the Mamba enhancement layer contains a linear mapping layer, a convolution layer, a selective state space model layer and activation function; Step 2.2.1, initialize n=1; As the output feature of the n-1th dense block ; Step 2.2.2, Initialization =1; Step 2.2.3, Input into the nth dense block and pass through the After processing the dense layers, we get the nth dense block The hierarchical features output by the dense layer ; Step 2.2.4, and After superposition, assign to ,Will +1 assigned to Then return to step 2.2.3 and execute sequentially until So far, the dense features of the nth dense block output are obtained ={ , ,…, ,…, , }; Step 2.2.5, Initialization =n; As the first -1 with The fusion features of the transition layer output of the enhancement module ; Step 2.2.6, Enter with In the transition layer of the enhancement module, and respectively through the convolutional dimension reduction layer and The processing of the enhancement layer is The extracted features and mapping features, thereby The extracted features and After weighted fusion of the mapping features, we get Fusion features ; Step 2.2.7, As , after assigning n+1 to n, return to step 2.2.2 and execute sequentially until n>N, thus obtaining the dense features output by the Nth dense block .

3. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the multimodal medical image classification method according to claim 1 or 2, and the processor is configured to execute the program stored in the memory.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multimodal medical image classification method according to claim 1 or 2 are performed.

Images