Multi-modal medical image classification method based on improved DenseNet

Abstract

The invention discloses a multi-modal medical image classification method based on an improved DenseNet, a medical image classification network is constructed, the network comprises a preliminary feature extraction module, a feature refining module and a classification module, the feature refining module fuses the interlayer dense connection characteristic of the DenseNet and the efficient feature processing capability of a Mamba module, and the classification module is used for classifying the Mamba module. Extraction and fusion of complex image features are realized; and the classification module adopts a full connection layer and a Softmax function and is responsible for converting rich image feature representation into a specific image classification result. According to the invention, through the transition layer with the Mama module, the capturing capability of the model on the key features of the image is enhanced, so that the classification precision of the 3D medical image is further improved on the basis of maintaining the original advantages of the DenseNet, and powerful technical support is provided for the application of deep learning in medical image processing.

Description

Technical Field

[0001] The present invention relates to the field of 3D medical image classification and deep learning technology, and in particular to a method based on improved Multimodal medical image classification method. Background Art

[0002] With the rapid development of medical information technology and artificial intelligence, the role of medical images in disease diagnosis, treatment planning and prognosis assessment has become increasingly prominent. From X-rays, CT, MRI to ultrasound images, various types of medical images provide rich diagnostic information for the medical field. Medical image classification, as the core task of medical image analysis, uses automated technology to classify and interpret images, which not only improves diagnostic efficiency, but also provides important support for precision medicine. However, despite the progress made in this field in recent years, technical bottlenecks and challenges in practical applications still exist.

[0003] Medical image classification requires researchers to have knowledge of both medical image analysis and digital images. Due to the influence of image scale, data format, and data category distribution, existing model methods, such as traditional machine learning recognition methods and methods based on deep convolutional neural networks, have limited recognition accuracy and generalization. In actual scenarios, due to the labeling cost of medical image data and the collection of case medical images, the available labeled medical image data is limited.

[0004] The limited number of high-quality annotated medical image datasets, especially in the study of specific diseases or rare conditions, limits the training and generalization capabilities of models. This lack of data makes it difficult for many models to achieve the expected results in practical applications, especially when processing images acquired by different patients or different devices, the adaptability of the model is particularly insufficient.

[0005] In addition, most of the current mainstream medical image classification methods rely only on medical data of a single modality. Although they have achieved certain success in specific tasks, this method has significant limitations and is difficult to meet the needs of complex clinical scenarios. Although single-modality medical images can provide valuable information under certain conditions, they often cannot fully reflect the complexity and diversity of lesions. For example, CT images are excellent in clearly displaying bone structures, but are relatively insufficient in capturing the details of soft tissues; while MRI has advantages in soft tissue imaging, its ability to display bone structures is relatively weak. This limitation of single-modality images requires doctors to rely on multiple images for comprehensive judgment when making diagnoses, which increases the complexity of diagnosis and the risk of errors.

[0006] Images of different modalities can provide complementary information, thus providing doctors with a more comprehensive basis for diagnosis. However, how to make full use of multimodal medical image data is also a big challenge. The richness of multimodal data has not been effectively utilized. This limitation not only reduces the model's ability to recognize complex lesions, but also may lead to the omission of important information, thereby affecting the accuracy of clinical decision-making. Summary of the invention

[0007] The present invention provides a method for overcoming the problems existing in the existing image classification technology in medical image classification. In order to more comprehensively understand the information in multimodal medical images and improve the accuracy and efficiency of image classification, a multimodal medical image classification method was proposed.

[0008] The present invention adopts the following technical solutions to solve the above problems:

[0009] The present invention is based on the improvement The multimodal medical image classification method is characterized by comprising the following steps:

[0010] Step 1: Get the image size And the mode number is The multimodal image set of a certain user is processed and image registration and bias field correction are performed to obtain a multimodal medical image set of a certain user after preprocessing. = ,in, Indicates A medical image of a modality, ; represents the width of a single-modality medical image, represents the height of a single-modality medical image, represents the depth of a single-modality medical image; let The true category label is recorded as and 1,2,…, }, Indicates the total number of categories;

[0011] Step 2: Construct a medical image classification network, including: a preliminary feature extraction module, a feature refinement module, and a classification module;

[0012] Step 2.1, the preliminary feature extraction module includes a convolution layer and an activation function;

[0013] Will After cascading medical images of different modalities, we get Cascade of medical images And input into the preliminary feature extraction module for processing to obtain the shallow feature map ,in, is the number of channels of a single-modality medical image;

[0014] Step 2.2: The feature refinement module processes the Nth dense block to obtain the dense features output by the Nth dense block. ;

[0015] Step 2.3: The classification module includes a fully connected layer and function, and then After processing, the output The predicted class label of ;

[0016] Step 3: Train the medical image classification network;

[0017] Step 3.1: Use formula (1) to construct the cross entropy loss function :

[0018] (1)

[0019] In formula (1), express middle In the The predicted probability of each category; express middle In the The true probability of each category;

[0020] Step 3.2: Use the back propagation algorithm to train the medical image classification network and calculate the cross entropy loss function. To update the network parameters until the cross entropy loss function Until convergence, the optimal medical image classification network model is obtained, which is used to classify and predict any input multimodal medical image.

[0021] The present invention discloses a method based on improvement The multimodal medical image classification method is also characterized in that the feature refining module in step 2.2 includes: dense blocks and with The transition layer of the enhancement module; where the nth dense block contains dense layers; any The dense layers include: two normalization layers, two activation functions, and two convolutional layers;

[0022] Any with The transition layer of the enhancement module consists of a convolutional dimension reduction layer and Enhancement layer, where the convolutional dimension reduction layer contains a normalization layer, activation function, a convolution layer and a pooling layer; the Mamba enhancement layer contains a linear mapping layer, a convolution layer, a selective state space model layer and an activation function;

[0023] Step 2.2.1, initialize n=1; As the output feature of the n-1th dense block ;

[0024] Step 2.2.2, Initialization =1;

[0025] Step 2.2.3: Input into the nth dense block and pass through the After processing the dense layers, we get the nth dense block. The hierarchical features of the dense layer output ;

[0026] Step 2.2.4: and After superposition, assign to ,Will +1 assigned to Then return to step 2.2.3 and execute sequentially until So far, the dense features output by the nth dense block are obtained ={ , , …, , …, , };

[0027] Step 2.2.5: Initialization =n; As the -1 with The fused features of the transition layer output of the enhancement module ;

[0028] Step 2.2.6: Enter the with In the transition layer of the enhancement module, and then through the convolutional dimension reduction layer and The processing of the enhancement layer is The extracted features and mapping features, thereby The extracted features and After weighted fusion of the mapping features, we get Fusion features ;

[0029] Step 2.2.7: As , after assigning n+1 to n, return to step 2.2.2 and execute sequentially until n>N, thus obtaining the dense features output by the Nth dense block .

[0030] An electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the multimodal medical image classification method, and the processor is configured to execute the program stored in the memory.

[0031] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program executes the steps of the multimodal medical image classification method when executed by a processor.

[0032] Compared with the existing classification methods, the beneficial effects of the present invention are:

[0033] 1. The present invention uses 3D multimodal medical image data: The 3D images used in this method can provide richer spatial information than traditional 2D images, allowing the model to accurately capture the complex details of brain structures and early lesions. Combining the two modalities of magnetic resonance imaging (MRI) and positron emission tomography (PET) data can not only capture structural information, but also obtain functional and metabolic information. The full use of multimodal images significantly improves the insight into disease characteristics, thereby improving the accuracy of image classification and the robustness of the model.

[0034] 2. The present invention introduces the Dropout mechanism: In this method, the Dropout mechanism is introduced. In particular, after generating new features in the dense layer, Dropout will randomly discard some feature channels with a certain probability. Its main benefit is to prevent overfitting. By randomly ignoring some neurons, the network cannot rely on specific paths or features, thereby enhancing the generalization ability of the model. In addition, it can significantly improve the performance of the model in the verification and testing stages, especially when processing complex data sets, which helps to suppress overfitting and improve the effect in practical applications.

[0035] 3. The present invention is Customized optimization of structure: by adjusting The number of dense layers within the dense blocks in the architecture effectively reduces the consumption of computing resources while improving the representation ability of the model, especially showing superiority when processing high-dimensional data.

[0036] 4. The present invention introduces Modules: The module reduces computational requirements by dividing the subsequences and processing them independently, which helps reduce the number of parameters in the overall model and makes the network more efficient in processing large-scale 3D medical image data. The module enhances the network's ability to process complex image data and optimizes the efficiency and accuracy of feature extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Based on the improvement of the present invention Flowchart of multimodal medical image classification method;

[0038] Figure 2 It is a specific framework schematic diagram of the present invention, For example;

[0039] Figure 3 It is a schematic diagram of the specific structure of the dense layer of the present invention;

[0040] Figure 4 A schematic diagram of a dense connection method of a dense layer in a dense block of the present invention;

[0041] Figure 5 For the present invention Schematic diagram of the specific structure of the module. DETAILED DESCRIPTION

[0042] In this embodiment, taking two different modalities of medical images MRI and PET as examples, the general process is as follows: Figure 1 As shown, based on the improvement The specific network framework of the multimodal medical image classification method is as follows Figure 2 As shown in Figure 2, the addition of dense blocks and Mamba modules significantly improves the model's image classification efficiency and accuracy, providing a reference value for future auxiliary medical treatment applications. The multimodal medical image classification method includes the following steps:

[0043] Step 1: Get the image size And the mode number is The multimodal image set of a certain user is processed and image registration and bias field correction are performed to obtain a multimodal medical image set of a certain user after preprocessing. = ,in, Indicates A medical image of a modality, ; represents the width of a single-modality medical image, represents the height of a single-modality medical image, represents the depth of a single-modality medical image; let The true category label is recorded as and 1,2,…, }, Indicates the total number of categories;

[0044] In specific implementation, The dataset comes from the Alzheimer's Disease Neuroimaging Project (ADNI), using MRI and PET images from the same user with an interval of no more than one month. All multimodal data are provided in the form of NIfTI files (.nii.gz). The experiment uses 443 sample pairs, namely 443 MRIs and 443 PETs. ​​The training set contains 159 sets of image data pairs of Alzheimer's patients, 130 sets of image data pairs of mild cognitive impairment patients, and 154 sets of medical image data pairs of normal people. Both MRI and PET are grayscale images, and the size of each medical image is ,Right now .

[0045] Both MRI and PET were processed according to standard procedures. First, the FSL tool was used to perform skull stripping to remove non-brain tissue from magnetic resonance imaging (MRI) and positron emission tomography (PET) images to reduce interference from irrelevant imaging features. Next, the advanced normalization tools (ANTs) were used to accurately align the images, and a standard brain template with a resolution of 1 mm was used to ensure spatial consistency between different images and eliminate global linear differences (including global translation, scale, and rotation differences). In addition, the images were biased and corrected to eliminate image intensity changes caused by magnetic field inhomogeneity. Finally, in order to make the image data more suitable for subsequent analysis and processing, maximum and minimum normalization was performed to standardize the intensity values ​​of the images to a uniform range. This series of sophisticated preprocessing steps ensures data quality and provides a solid foundation for subsequent analysis and research.

[0046] Step 2: Construct a medical image classification network, including: a preliminary feature extraction module, a feature refinement module, and a classification module;

[0047] Step 2.1, the preliminary feature extraction module contains a convolutional layer and activation function;

[0048] Will After cascading medical images of different modalities, we get Cascade of medical images And input into the preliminary feature extraction module for processing to obtain the shallow feature map ,in, is the number of channels of a single-modality medical image;

[0049] In this implementation case, a convolution layer with a convolution kernel size of 7 is used, and the activation function is Activation function. This larger convolution kernel can extract features in a larger range, increase the receptive field, and thus capture richer contextual information. It can more effectively capture large-scale structures and patterns in the image, and is particularly suitable for processing input data containing complex and large-scale features. For example, in medical image analysis, a larger convolution kernel can better capture the overall morphology and large-scale spatial distribution of abnormal areas such as tumors, rather than just focusing on detailed features.

[0050] Step 2.2: Feature refinement module includes: dense blocks and with The transition layer of the enhancement module; where the nth dense block contains dense layers; any The dense layers include: two normalization layers, two activation functions, and two convolutional layers;

[0051] In the specific implementation, , the model structure is as follows Figure 4 As shown in the figure, the four dense blocks contain different numbers of dense layers. The first dense block contains 6 dense layers, the second dense block contains 18 dense layers, the third dense block contains 18 dense layers, and the fourth dense block contains 12 dense layers. The adjustment of the number of dense layers in the dense blocks improves the diversity of feature extraction of the network, optimizes the number of parameters and calculations, and improves the efficiency of the model. At the same time, adjusting the dense layer structure enhances the flexibility of the model, better adapts to the medical image classification task, and ultimately improves the generalization ability of the model.

[0052] The dense layers include the first normalization layer, Activation function, the first convolution layer with convolution kernel 1, the second normalization layer, The activation function and convolution kernel of the second convolution layer are 3. The specific structure is as follows Figure 3 shown.

[0053] The dense layer uses dense connections. Specifically, in the dense block, The input received by the dense layer of the layer includes both the original features of the initial input of the dense block and the previous All these features are concatenated together as the output features of all dense layers in the first The output features generated by each layer are appended to the end of the current input features to form new input features, and are passed to subsequent layers for further processing, which promotes the rapid transmission of information and gradient flow, alleviates the gradient vanishing problem, and is beneficial to model training; at the same time, the direct reuse of features improves the representation ability and performance of the model; in addition, dense connections make it easier to train deeper networks and improve feature propagation, which helps the model better learn complex feature representations.

[0054] Any with The transition layer of the enhancement module consists of a convolutional dimension reduction layer and Enhancement layer, where the convolutional dimension reduction layer contains a normalization layer, activation function, a convolution layer and a pooling layer; the Mamba enhancement layer contains a linear mapping layer, a convolution layer, a selective state space model layer and an activation function;

[0055] In this embodiment, the convolutional dimension reduction layer uses a convolutional layer with a convolution kernel size of 3 and Activation function is used to further extract features. The specific structure of the enhancement layer is as follows: Figure 5 As shown, the Mamba enhancement layer uses Activation function and depth-wise separable convolution, used in the depth-wise convolution stage The convolution kernel is used in the point-by-point convolution stage. convolution kernel. Depthwise separable convolution significantly reduces computational complexity and parameter count while maintaining high model performance. It greatly improves computational efficiency, reduces memory and video memory usage, and significantly shortens training time while ensuring performance. In addition, the data is deeply processed through the state-space model, and a weight-learnable jump connection strategy is implemented, which effectively maintains the integrity of the original input information and prevents the common gradient vanishing problem in multi-layer networks. After state-space processing, the data is again layer-normalized to ensure the consistency of feature distribution, and finally adjusted to the required output dimension through a linear projection layer. The output data is finally reversed and reshaped to ensure that it is consistent with the input data in spatial structure, providing high-quality feature representation for subsequent processing, so that the model can significantly improve the utilization of information and the expression ability of features when processing data with complex spatial and feature dimensions.

[0056] Step 2.2.1, initialize n=1; As the output feature of the n-1th dense block ;

[0057] Step 2.2.2, Initialization =1;

[0058] Step 2.2.3: Input into the nth dense block and pass through the After processing the dense layers, we get the nth dense block. The hierarchical features of the dense layer output ;

[0059] Step 2.2.4: and After superposition, assign to ,Will +1 assigned to Then return to step 2.2.3 and execute sequentially until So far, the dense features output by the nth dense block are obtained ={ , , …, , …, , };

[0060] Step 2.2.5: Initialization =n; As the -1 with The fused features of the transition layer output of the enhancement module ;

[0061] Step 2.2.6: Enter the with In the transition layer of the enhancement module, and then through the convolutional dimension reduction layer and The processing of the enhancement layer obtains the refined features and the mapping features accordingly, so that the refined features and the mapping features are weighted and fused to obtain the first Fusion features ;

[0062] Step 2.2.7: As , after assigning n+1 to n, return to step 2.2.2 and execute sequentially until n>N, thus obtaining the dense features output by the Nth dense block ;

[0063] In this example, the outputs of the first three dense blocks are passed as input to the next one with The transition layer of the enhancement block and the output of the fourth dense block are used to perform the final classification task.

[0064] Step 2.3: The classification module contains a fully connected layer and function, and then After processing, the output The predicted class label of ;

[0065] Step 3: Train the medical image classification network;

[0066] Step 3.1: Use formula (1) to construct the cross entropy loss function :

[0067] (1)

[0068] In formula (1), express middle In the The predicted probability of each category; express middle In the The true probability of each class.

[0069] Step 3.2: Use the back propagation algorithm to train the medical image classification network and calculate the cross entropy loss function To update the network parameters until the cross entropy loss function Until convergence, the optimal medical image classification network model is obtained, which is used to classify and predict any input multimodal medical image. In this embodiment, the learning rate during training is set to ,use Optimization, the number of iterations is 200, using The optimizer has a total loss Perform optimization and solution; after training, the best image classification network model is obtained.

[0070] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0071] In this embodiment, a computer-readable storage medium stores a computer program on the computer-readable storage medium, and the computer program executes the steps of the above method when executed by a processor.

Claims

1. An improvement based on A multimodal medical image classification method is characterized in that: The steps include: Step 1: Get the image size And the mode number is The multimodal image set of a certain user is processed and image registration and bias field correction are performed to obtain a multimodal medical image set of a certain user after preprocessing. = ,in, Indicates A medical image of a modality, ; represents the width of a single-modality medical image, represents the height of a single-modality medical image, represents the depth of a single-modality medical image; let The true category label is recorded as and 1,2,…, }, Indicates the total number of categories; Step 2: Construct a medical image classification network, including: a preliminary feature extraction module, a feature refinement module, and a classification module; Step 2.1, the preliminary feature extraction module includes a convolution layer and an activation function; Will After cascading medical images of different modalities, we get Cascade of medical images And input into the preliminary feature extraction module for processing to obtain the shallow feature map ,in, is the number of channels of a single-modality medical image; Step 2.2: The feature refinement module processes the Nth dense block to obtain the dense features output by the Nth dense block. ; Step 2.3: The classification module includes a fully connected layer and function, and then After processing, the output The predicted class label of ; Step 3: Train the medical image classification network; Step 3.1: Use formula (1) to construct the cross entropy loss function : (1) In formula (1), express middle In the The predicted probability of each category; express middle In the The true probability of each category; Step 3.2: Use the back propagation algorithm to train the medical image classification network and calculate the cross entropy loss function. To update the network parameters until the cross entropy loss function Until convergence, the optimal medical image classification network model is obtained, which is used to classify and predict any input multimodal medical image.

2. A method according to claim 1 based on improvement A multimodal medical image classification method is characterized in that: The feature refinement module in step 2.2 includes: dense blocks and with The transition layer of the enhancement module; where the nth dense block contains dense layers; any The dense layers include: two normalization layers, two activation functions, and two convolutional layers; Any with The transition layer of the enhancement module consists of a convolutional dimension reduction layer and Enhancement layer, where the convolutional dimension reduction layer contains a normalization layer, activation function, a convolution layer and a pooling layer; the Mamba enhancement layer contains a linear mapping layer, a convolution layer, a selective state space model layer and an activation function; Step 2.2.1, initialize n=1; As the output feature of the n-1th dense block ; Step 2.2.2, Initialization =1; Step 2.2.3: Input into the nth dense block and pass through the After processing the dense layers, we get the nth dense block. The hierarchical features of the dense layer output ; Step 2.2.4: and After superposition, assign to ,Will +1 assigned to Then return to step 2.2.3 and execute sequentially until So far, the dense features output by the nth dense block are obtained ={ , , …, , …, , }; Step 2.2.5: Initialization =n; As the -1 with The fused features of the transition layer output of the enhancement module ; Step 2.2.6: Enter the with In the transition layer of the enhancement module, and then through the convolutional dimension reduction layer and The processing of the enhancement layer is The extracted features and mapping features, thereby The extracted features and After weighted fusion of the mapping features, we get Fusion features ; Step 2.2.7: As , after assigning n+1 to n, return to step 2.2.2 and execute sequentially until n>N, thus obtaining the dense features output by the Nth dense block .

3. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the multimodal medical image classification method according to claim 1 or 2, and the processor is configured to execute the program stored in the memory.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multimodal medical image classification method according to claim 1 or 2 are performed.

Images