Multi-modal image fusion method and system and computer equipment

Abstract

The invention relates to a multi-modal image fusion method and system and computer equipment, and the method comprises the steps: obtaining an sMRI image and an FDG PET image of a tested object, and carrying out the preprocessing of the sMRI image and the FDG PET image; calculating image entropies of all axial slices for each sMRI image, extracting a preset number of axial slices with the maximum image entropies as sMRI key slices, and extracting FDG PET key slices of the FDG PET image corresponding to the sMRI key slices; the sMRI key slices and the corresponding FDG PET key slices are fused, a fused image is obtained, and the fused image is used for recognizing the disease category to which the image belongs. According to the method and the device, more efficient and accurate recognition of people in different cognitive states is realized based on complementary information of sMRI and FDG PET modal images.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to multimodal image fusion methods, apparatus, systems, computer devices, and computer-readable storage media. Background Technology

[0002] Medical imaging technology has played a significant role in the research and early diagnosis of neurodegenerative diseases such as Alzheimer's Disease (AD) and Mild Cognitive Impairment (MCI).

[0003] With the development of medical imaging, various imaging techniques, such as structural magnetic resonance imaging (sMRI), diffusion tensor imaging (DTI), functional magnetic resonance imaging (fMRI), and positron emission tomography (PET), are widely used to reveal the pathological characteristics between Alzheimer's disease (AD), mild cognitive impairment (MCI), and normal cognitive state (CN). Among these, sMRI has a natural advantage in reflecting changes in brain structure (such as ventricular enlargement and hippocampal atrophy) and is often used to help determine the degree of neurodegeneration. PET, on the other hand, provides effective functional indicators for the diagnosis of Alzheimer's disease (AD) by revealing abnormalities in brain metabolic function, especially fluorodeoxyglucose (FDG) PET images.

[0004] Currently, researchers have established various computational-aided diagnostic (CAD) models based on these image features and combined with machine learning algorithms such as support vector machines (SVM), sparse coding, and deep neural networks. For example, some methods have improved the accuracy of MCI / AD recognition through multi-template MRI feature extraction and relationship modeling; other studies have utilized metabolic features selected from PET images to effectively alleviate the curse of dimensionality caused by high-dimensional features, maintaining high classification performance while compressing feature dimensions.

[0005] In research aimed at further improving diagnostic accuracy, multimodal medical image fusion has become a popular research direction. However, feature fusion methods typically require constructing feature extraction channels for each modality separately, resulting in complex model structures and a large number of parameters. This not only increases the difficulty of training but also limits the model's generalization ability.

[0006] Therefore, although existing AD diagnosis methods based on multimodal imaging have certain accuracy and application prospects, they still have the following shortcomings: (1) The feature fusion model has a complex structure and a large number of parameters, resulting in low training efficiency and high requirements for computing resources; (2) There are problems such as difficulty in feature alignment between modalities and information redundancy in the process of multimodal feature extraction, which affect the final classification performance. Summary of the Invention

[0007] In a first aspect, embodiments of this application provide a multimodal image fusion method, including: The data acquisition steps involve acquiring sMRI and FDG PET images of the subjects and preprocessing them to reduce inter-individual differences. The subjects include Alzheimer's disease patients, patients with mild cognitive impairment, and people with normal cognitive status. The slice selection step involves calculating the image entropy of all axial slices for each sMRI image and extracting a preset number of axial slices with the largest image entropy as sMRI key slices. FDG PET key slices corresponding to the sMRI key slices are then extracted, specifically three images. The sequence number of the sMRI key slices is determined as an index, and FDG PET key slices are extracted based on this index. Ultimately, 285 images from Alzheimer's patients, 585 images from patients with mild cognitive impairment, and 474 images from individuals with normal cognitive states are selected. The image fusion step involves merging sMRI key sections and their corresponding FDG PET key sections after image registration to obtain a fused image, which is used to identify the disease category to which the image belongs.

[0008] In some embodiments, during the slice selection step, the image entropy of each axial slice is calculated based on the following computational model: ,in, For the first The probability of a gray level appearing, where N is the number of gray levels, i.e., the image entropy is the total information content of the image pixels.

[0009] In some embodiments, the data acquisition step includes: The sMRI image preprocessing steps involve using the CAT12 toolkit to perform skull dissection and tissue segmentation on the sMRI images, and registering and aligning the sMRI images to a standard template space. The FDG PET image preprocessing step involves using the FLIRT module in the FSL software to register each subject's FDG PET image onto its corresponding preprocessed sMRI image, thereby obtaining an FDG PET image with the same spatial orientation, image size, and voxel size as the aforementioned sMRI image.

[0010] In some embodiments, the image fusion step further includes: The image registration step uses an sMRI key slice as the reference image and an FDG PET key slice corresponding to the sMRI key slice as a floating image. A joint histogram of the reference image and the floating image is calculated using a partial volume interpolation algorithm to more accurately simulate the spatial overlap of image grayscale values ​​under different modalities. Based on the joint histogram, the mutual information between the reference image and the floating image is calculated, and this mutual information is used as a similarity metric. An optimization search algorithm is then used to find registration parameters that maximize the mutual information. The floating image is transformed according to the registration parameters to register the floating image and the reference image, thereby registering the floating image to the spatial coordinate system of the reference image and achieving spatial consistency of the multimodal image.

[0011] In some embodiments, the method further includes: The image classification step involves inputting the fused image into a preset classification model to obtain the disease category to which the fused image belongs. The disease categories include: Alzheimer's disease, mild cognitive impairment, and normal cognitive state.

[0012] In some embodiments, in the classification model, the input fused image is processed sequentially by the first to the fifth convolutional blocks and then input into the first fully connected layer and the second fully connected layer, and finally output as a classification probability through the Softmax layer.

[0013] Secondly, embodiments of this application provide a multimodal image fusion system, including: The data acquisition module acquires sMRI and FDG PET images of the subjects and performs preprocessing. The subjects include Alzheimer's disease patients, patients with mild cognitive impairment, and people with normal cognitive status. The slice selection module calculates the image entropy of all axial slices for each sMRI image, extracts a preset number of axial slices with the largest image entropy as sMRI key slices, and extracts the FDG PET key slices of the FDG PET image corresponding to the sMRI key slices. The image fusion module is used to fuse sMRI key sections and their corresponding FDG PET key sections to obtain a fused image, which is used to identify the disease category to which the image belongs.

[0014] In some embodiments, the data acquisition module includes: The sMRI image preprocessing unit uses the CAT12 toolkit to perform skull dissection and tissue segmentation on the sMRI image, and registers and aligns the sMRI image to a standard template space to obtain a whole brain image with a size of 121×145×121. The FDG PET image preprocessing unit uses the FLIRT module in the FSL software to register each subject's FDG PET image onto its corresponding preprocessed sMRI image, thereby obtaining an FDG PET image with the same spatial orientation, image size, and voxel size as the aforementioned sMRI image.

[0015] In some embodiments, the image fusion module further includes: The image registration unit uses an sMRI key slice as the reference image and the corresponding FDG PET key slice as the floating image. It calculates the joint histogram of the reference image and the floating image using a partial volume interpolation algorithm. Based on the joint histogram, it calculates the mutual information between the reference image and the floating image and then uses an optimization search algorithm to find the registration parameters that maximize the mutual information. The floating image is transformed according to the registration parameters to register the floating image with the reference image, thereby registering the floating image to the reference image spatial coordinate system.

[0016] In some embodiments, the system further includes: The image classification module is used to input the fused image into a preset classification model to obtain the disease category to which the fused image belongs. The disease categories include: Alzheimer's disease, mild cognitive impairment, and normal cognitive state.

[0017] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multimodal image fusion method as described in the first aspect above.

[0018] Compared to related technologies, this application proposes a combined use of multimodal medical image fusion and deep learning technologies for AD diagnosis. Specifically, firstly, multimodal medical image fusion technology is used to fuse multimodal images to obtain a fused image. Compared to existing feature fusion methods, multimodal medical image fusion is a more intuitive method, and the resulting fused image possesses rich modal features and strong information representation capabilities. Then, a simple single-input classification model is constructed to classify the fused image. Theoretically, the proposed method will have fewer model parameters, greatly reducing the computational burden while improving the performance of the classification model.

[0019] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of a multimodal image fusion method according to an embodiment of this application; Figure 2 This is a step-by-step flowchart of the multimodal image fusion method according to an embodiment of this application; Figure 3 This is a step-by-step flowchart of the multimodal image fusion method according to an embodiment of this application; Figure 4 This is a schematic diagram illustrating the logical principle of a multimodal image fusion method according to an embodiment of this application; Figure 5 This is a logical schematic diagram of the image registration steps according to an embodiment of this application; Figure 6 This is a structural block diagram of a multimodal image fusion system according to an embodiment of this application; Figure 7 This is a schematic diagram of the classification model according to an embodiment of this application; Figure 8 This is a schematic diagram of the structure of a dual-stream input CNN model according to an embodiment of this application; Figure 9 This is a schematic diagram of the structure of a computer device according to an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0022] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0023] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0024] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0025] The Alzheimer's Disease Neuroimaging Initiative (ADNI) database is a highly influential public database in the field of Alzheimer's disease research.

[0026] The Mini-Mental State Examination (MMSE) is a commonly used clinical tool for screening cognitive function, often used to assess the severity of dementia or cognitive impairment.

[0027] This embodiment provides a multimodal image fusion method. Figures 1 to 3 This is a flowchart of a multimodal image fusion method according to an embodiment of this application. Figure 4 This is a schematic diagram illustrating the logical principle of the multimodal image fusion method according to an embodiment of this application, such as... Figures 1 to 4 As shown, the process includes the following steps: In data acquisition step S1, sMRI and FDG PET images of the subjects were acquired and preprocessed to reduce inter-individual variability. Subjects included Alzheimer's disease patients, patients with mild cognitive impairment (MCI), and individuals with normal cognitive function. In this embodiment, 448 subjects with both sMRI and FDG PET images (fluorodeoxyglucose (FDG)-PET) were selected from the ADNI database for the study. Among them, 95 subjects had Alzheimer's disease (AD), 158 had mild cognitive impairment (MCI), and 155 had normal cognitive function (CN). Demographic information for all subjects is shown in Table 1.

[0028] Table 1 - Demographic information of the subjects

[0029] In slice selection step S2, the image entropy of all axial slices for each sMRI image is calculated, and a preset number of axial slices with the largest image entropy are extracted as sMRI key slices. FDG PET key slices of the FDG PET images corresponding to the sMRI key slices are extracted. Specifically, the preset number is three images. The sequence number of the sMRI key slices is determined as an index, and PET key slices are extracted based on this index. Finally, 285 images from Alzheimer's patients, 585 images from patients with mild cognitive impairment, and 474 images from people with normal cognitive status are selected. In image fusion step S3, the sMRI key slices and their corresponding FDG PET key slices are image registered and fused to obtain a fused image, which is used to identify the disease category to which the image belongs.

[0030] Based on the above steps, this application leverages the complementary information from sMRI and FDG PET images to fully combine the high-resolution advantage of sMRI in anatomical structure representation with the sensitivity of FDG PET in brain metabolic activity, achieving more efficient and accurate identification of individuals in different cognitive states. By utilizing image entropy indicators to select the three most informative axial slices from each sMRI image, redundant information is effectively reduced and processing efficiency is improved. Corresponding FDG PET image slices are simultaneously extracted based on their indices, ensuring spatial consistency across modal slices. A fused image is generated through image fusion technology and input into the subsequent disease classification model to improve recognition accuracy. Image fusion is not a simple superposition; it generates more new and effective information, achieving a 1+1>>2 effect.

[0031] In another embodiment, the preprocessing of the original image can include operations such as standardization, denoising, and brain region segmentation to adapt to different data quality conditions or clinical application needs.

[0032] In some embodiments, during the slice selection step, the image entropy of each axial slice is calculated based on the following computational model: ,in, For the first The probability of a gray level appearing, where N is the number of gray levels, i.e., the image entropy is the total information content of the image pixels.

[0033] Based on the above slice selection steps, compared with the existing random slice selection for training classification models, this application judges the contribution of each slice to the model's classification performance based on the image entropy of the slice. Although image entropy is easily affected by noise, the image preprocessing steps include denoising and standardization operations. Therefore, by sorting the slices in descending order of image entropy, the slice with the highest entropy value can be considered as the image with the most information. Training the classification model based on this can improve the robustness of the model.

[0034] Although all sMRI and FDG PET images had undergone specific image preprocessing steps by the ADNI research group, further preprocessing steps are necessary for both types of images to facilitate subsequent research. (Reference) Figure 2 As shown, the data acquisition step S1 in this embodiment includes: In the sMRI image preprocessing step S11, the CAT12 toolkit is used to perform skull dissection and tissue segmentation on the sMRI image, and the sMRI image is registered and aligned to the standard template space. Skull dissection is used to remove non-brain tissue regions, and tissue segmentation is used to distinguish gray matter, white matter and cerebrospinal fluid. The sMRI image preprocessing step may also include modulation operation, which is used to preserve local volume information changes, so as to finally obtain a three-dimensional whole brain image with consistent structure and uniform resolution, resulting in a whole brain image with a size of 121×145×121. In the FDG PET image preprocessing step S12, the FLIRT module in the FSL software is used to register the FDG PET image of each subject to its corresponding preprocessed sMRI image, thereby obtaining an FDG PET image with the same spatial orientation, image size and voxel size as the aforementioned sMRI image.

[0035] Based on the above steps, the embodiments of this application significantly improve the accuracy and operability of multimodal image fusion through a refined image preprocessing workflow. sMRI images undergo CAT12 processing to achieve structural standardization, which helps reduce cross-individual variability while preserving structural variation information. FDG PET images are registered to the MRI image space, ensuring strict consistency between the two modalities in spatial coordinates, thereby making the selection of key slices and subsequent fusion analysis more logical and accurate. Furthermore, the consistency of voxel size and image size provides a unified format for input to deep learning models, improving the model's learning efficiency and recognition performance.

[0036] In some embodiments, reference Figure 3 , Figure 5 As shown, the image fusion step S3 further includes: In image registration step S31, an sMRI key slice is used as the reference image, and the corresponding FDGPET key slice is used as the floating image. A joint histogram of the reference image and the floating image is calculated using a partial volume (PV) interpolation algorithm. This more accurately simulates the spatial overlap of image grayscale values ​​under different modalities. Based on the joint histogram, the mutual information between the reference image and the floating image is calculated. Using mutual information as a similarity metric, an optimization search algorithm is used to find registration parameters that maximize the mutual information. The floating image is transformed according to the registration parameters to register the floating image and the reference image, thereby registering the floating image to the spatial coordinate system of the reference image and achieving spatial consistency of the multimodal image.

[0037] In the above embodiments, the registration parameters This can be represented by the following computational model: , Where RI (Reference Image) represents the reference image, and FI (Float Image) represents the floating image. Here are the transformation parameters for transforming FI, and these transformation parameters are: The dimensional vector is determined by the specific type of image transformation. This refers to the mutual information.

[0038] Based on the above steps, this application aims to achieve precise spatial alignment between sMRI key slices and their corresponding FDG PET key slices. It introduces an image registration strategy based on mutual information, effectively overcoming the registration difficulties caused by the large differences in grayscale characteristics between sMRI and FDG PET images. A partial volume interpolation algorithm enhances the grayscale alignment accuracy of the joint histogram and improves the discriminative power of mutual information, making the similarity measurement of images from different modalities more accurate during the registration process. An optimization search algorithm automatically finds the optimal registration parameters, improving registration efficiency and accuracy. Overall, this method improves the robustness of key slice registration, provides a precise spatial basis for subsequent image fusion, and significantly enhances the structural integrity and discriminative power of the fused image.

[0039] In another embodiment, the image fusion step employs pixel-level fusion techniques (such as weighted averaging and wavelet transform) to fuse key sections from sMRI and FDG PET into a single image. This preserves both anatomical and metabolic information, thereby enhancing the classification model's ability to identify diseases. The fused image combines the high spatial resolution structural information provided by sMRI with the metabolic activity information revealed by FDG PET, enabling the model to more comprehensively capture pathological changes during training and improving the accuracy of identifying AD, MCI, and NC states.

[0040] In another embodiment, brain template registration (such as MNI spatial normalization) can be further introduced during the image preprocessing stage to unify the spatial reference frame of each image and improve the comparability of images from multiple subjects. The fused images can be used as input to subsequent classification models (such as ResNet, VGG, or Transformer structures) to achieve more refined disease grading and identification.

[0041] In some embodiments, the method further includes: The image classification step involves inputting the fused image into a preset classification model to obtain the disease category to which the fused image belongs. The disease categories include: Alzheimer's disease, mild cognitive impairment, and normal cognitive state.

[0042] refer to Figure 7 As shown, in the classification model of this application embodiment, the input fused image is processed sequentially by the first to the fifth convolutional blocks to obtain a 4×4×64 feature map, which is then input into the first fully connected layer and the second fully connected layer. Finally, the classification probability is output through the Softmax layer. The input fused image contains sMRI and FDG PET information, and spatial and semantic features are extracted layer by layer through five consecutive convolutional blocks.

[0043] This image classification architecture effectively distinguishes between individuals in different cognitive states by fully utilizing structural and metabolic features from fused images. Multiple convolutional blocks enhance the model's ability to extract key image features, while fully connected layers further improve feature discriminative power. The Softmax output provides intuitive class probabilities, which is beneficial for uncertainty analysis in medical decision-making. Furthermore, the model has a relatively compact structure and high computational efficiency, making it suitable for automated processing of large-scale image data and contributing to improved accuracy and real-time performance in early Alzheimer's disease identification.

[0044] The first and second convolutional blocks each include a convolutional layer, a ReLU layer, and a max pooling layer for feature extraction. The convolutional kernel size of the convolutional layer is 5×5, the stride is 1, and the added sides are all 2 to keep the image input and output sizes the same. The max pooling layer has a stride of 2, which is used to adjust the output image size to one-quarter of the input image, thereby reducing the amount of computation and improving the level of feature abstraction.

[0045] The third to fifth convolutional blocks each include convolutional layers and ReLU layers to continue extracting high-level semantic features, but pooling is no longer performed. Each layer is connected to a ReLU activation function to increase the network's non-linear modeling capability. The kernel size of the convolutional layers in the third and fifth convolutional blocks is 11×11 to enhance the receptive field, while the kernel size of the convolutional layers in the fourth convolutional block is 5×5 to extract mesoscale features.

[0046] The first fully connected layer has a dimension of 256, and the second fully connected layer has a dimension of 2. The feature map obtained from the fifth convolutional block is flattened and then input into the 256-dimensional and 2-dimensional first and second fully connected layers, respectively, for feature compression and output mapping. The feature vector output from the 2-dimensional fully connected layer is processed by the Softmax layer and used for classification, outputting the predicted probabilities of three disease states.

[0047] The classification probability output by the Softmax layer can be a three-class probability, such as the probability of Alzheimer's disease, mild cognitive impairment, and normal cognitive state, or it can be a two-class probability, such as the probability of Alzheimer's disease and mild cognitive impairment.

[0048] This implementation improves the model's expressive power and recognition accuracy through the rational design of convolutional kernel size and structural configuration. The first two convolutional blocks help preserve key structural information, and max pooling significantly reduces computational complexity. Large-size convolutional kernels achieve broader aggregation of contextual information at higher levels, aiding in the identification of cross-regional structural features, such as areas of brain atrophy or metabolic abnormalities. 256-dimensional fully connected layers effectively extract key features, while 2-dimensional fully connected layers reduce model parameters and improve training efficiency. The overall structural design adapts to the characteristics of small target regions and subtle differences in medical images, improving the robustness and interpretability of Alzheimer's disease-related classification tasks.

[0049] CNN is the most commonly used neural network in deep learning. It supports images as input to the network model and abstracts the input image data layer by layer through convolution operations, activation function mapping, and pooling operations, automatically learning more representative and distinguishable features.

[0050] This application aims to verify the superiority of the proposed method by comparing it with single-image fusion input methods and feature fusion methods. Figure 8 The diagram shows the structure of a two-stream input CNN model. Each stream of this model consists of the first 13 layers of a single-input CNN, and the depth of each stream's fully connected (FC) layers is concatenated through a fusion FC layer with 512 dimensions. Then, an FC layer with 2 (or 3) dimensions is added after the fusion FC layer to facilitate subsequent classification. Finally, a softmax layer is used for prediction.

[0051] Four independent experiments were conducted in this application: (1) AD classification experiment based on MRI images (hereinafter referred to as MRI image experiment). (2) AD classification experiment based on PET images (hereinafter referred to as PET image experiment). (3) AD classification experiment based on fused images (hereinafter referred to as fused image experiment) and (4) AD classification experiment based on multimodal feature fusion (hereinafter referred to as feature fusion experiment).

[0052] For the first three experiments, this application adopted the above-mentioned... Figure 7 The classification model shown was used for training and prediction, and three sets of binary classification experiments (AD vs. CN, AD vs. MCI, and MCI vs. CN) and one set of multi-class classification experiments (AD vs. MCI vs. CN) were conducted. The performance evaluation metrics for the binary classification task included classification accuracy (ACC), sensitivity (SEN), and specificity (SPE), while the performance evaluation metric for the multi-class task was ACC. The specific formulas for calculating ACC, SEN, and SPE are as follows:

[0053]

[0054]

[0055] Wherein, TP represents a true positive; TN represents a true negative; FP represents a false positive; and FN represents a false negative. For the feature fusion experiment, this application adopted... Figure 8 The dual-stream input CNN model shown was used for training and prediction. Similar to the previous three experiments, three sets of binary classification experiments and one set of multi-class classification experiments were conducted, and the same evaluation parameters were used.

[0056] In these four experiments, the data was divided into training and test sets in an 8:1:1 ratio. For the training options, an adaptive moment estimation (Adam) algorithm was used, with a minimum batch size of 32 and a maximum number of iterations of 100. Notably, the initial learning rate for the first three experiments was 0.00005, and the initial learning rate for the last experiment was also 0.0005. All experiments were independently trained and predicted five times, with the data reshuffled for each training iteration, and the learning rate remained constant throughout the model training process.

[0057] This application compares the classification results of the first three experiments (i.e., MRI image experiment, PET image experiment and fusion image experiment) in Table 2.

[0058] Table 2 Classification results of the classification model

[0059] As shown in Table 2, the performance of classification using fused images is superior to that using single-modal images (MRI or PET) in all classification tasks. Specifically, compared with MRI images, the classification accuracy of the fused image method in this application is improved by 6.49% (AD vs. CN), 7.5% (AD vs. MCI), 11.03% (MCI vs. CN), and 19.12% (AD vs. MCI vs. CN). Compared with PET images, the classification accuracy of the fused image method in this application is improved by 5.71% (AD vs. CN), 5.68% (AD vs. MCI), 2.62% (MCI vs. CN), and 4.42% (AD vs. MCI vs. CN).

[0060] Therefore, the image fusion method proposed in this application has good performance, with sensitivity and specificity of more than 91.86% in the three sets of binary classification experiments. Moreover, compared with MRI images and PET images, most of the sensitivity and specificity have been improved.

[0061] Based on these results, this application concludes that by effectively fusing sMRI and FDG PET, the fused images can simultaneously contain the structural and metabolic features of the brain, providing robustness for the classification results of CNN models.

[0062] Table 3 shows the classification results of the dual-input CNN model. Comparing the results in Tables 2 and 3 reveals that the dual-input CNN model effectively integrates information from different modalities of images, thereby improving the classification accuracy of Alzheimer's disease (AD). Specifically, compared to using only MRI images, this feature fusion method improved classification accuracy by 6.21% (AD vs. CN), 6.54% (AD vs. MCI), 9.63% (MCI vs. CN), and 16.29% (AD vs. MCI vs. CN). Compared to the PET image method, this feature fusion method improved classification accuracy by 5.43% (AD vs. CN), 4.72% (AD vs. MCI), 1.22% (MCI vs. CN), and 1.59% (AD vs. MCI vs. CN).

[0063] However, the image fusion method proposed in this application is superior to the feature fusion method of the dual-stream input CNN model. For the image fusion experiment section in Table 2 and the classification results in Table 3, for AD and CN classification, the former's classification accuracy is 0.28% higher than the latter; for AD and MCI classification, the former's classification accuracy is 0.96% higher than the latter; for MCI and CN classification, the former's classification accuracy is 1.4% higher than the latter; and for AD, MCI, and CN classification, the former's classification accuracy is 2.83% higher than the latter. Based on these results, compared to the feature fusion method, the AD classification method based on fused images in this application has better classification performance.

[0064] Table 3. Classification results of the dual-input CNN model

[0065] It should be noted that the steps shown in the above flowchart or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here. For example, steps S11 and S12 may be executed concurrently.

[0066] This embodiment also provides a multimodal image fusion system for implementing the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0067] Figure 6 This is a structural block diagram of a multimodal image fusion system according to an embodiment of this application, such as... Figure 6 As shown, the system includes: Data acquisition module 1 acquires sMRI and FDG PET images of the subjects and performs preprocessing. The subjects include Alzheimer's disease patients, patients with mild cognitive impairment, and people with normal cognitive status. Slice selection module 2 calculates the image entropy of all axial slices for each sMRI image, extracts a preset number of axial slices with the largest image entropy as sMRI key slices, and extracts the FDG PET key slices of the FDG PET image corresponding to the sMRI key slices. Image fusion module 3 is used to register and fuse key sMRI slices and their corresponding key FDG PET slices to obtain a fused image, which is used to identify the disease category to which the image belongs.

[0068] In some embodiments, the data acquisition module 1 includes: The sMRI image preprocessing unit uses the CAT12 toolkit to perform preprocessing operations such as skull dissection, tissue segmentation, registration, and modulation on the sMRI image to obtain a whole brain image with a size of 121×145×121. The FDG PET image preprocessing unit uses the FLIRT module in the FSL software to register each subject's FDG PET image onto its corresponding preprocessed sMRI image, thereby obtaining an FDG PET image with the same spatial orientation, image size, and voxel size as the aforementioned sMRI image.

[0069] In some embodiments, the image fusion module further includes: The image registration unit uses an sMRI key slice as the reference image and the corresponding FDG PET key slice as the floating image. It calculates the joint histogram of the reference image and the floating image using a partial volume interpolation algorithm. Based on the joint histogram, it calculates the mutual information between the reference image and the floating image and then uses an optimization search algorithm to find the registration parameters that maximize the mutual information. The floating image is transformed according to the registration parameters to register the floating image with the reference image, thereby registering the floating image to the reference image spatial coordinate system.

[0070] In some embodiments, the system further includes: The image classification module is used to input the fused image into a preset classification model to obtain the disease category to which the fused image belongs. The disease categories include: Alzheimer's disease, mild cognitive impairment, and normal cognitive state.

[0071] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0072] In addition, combined Figures 1 to 3 The multimodal image fusion method described in this application can be implemented by a computer device. (See references) Figure 9 As shown, the computer device may include a processor 41 and a memory 42 storing computer program instructions.

[0073] Specifically, the processor 41 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0074] The memory 42 may include a large-capacity memory for data or instructions. For example, and not limitingly, the memory 42 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), flash memory, an optical disk drive, a magneto-optical disk drive, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 42 may include removable or non-removable (or fixed) media. Where appropriate, the memory 42 may be internal or external to a data processing device. In a particular embodiment, the memory 42 is non-volatile memory. In a particular embodiment, the memory 42 includes read-only memory (ROM) and random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), an electrically alterable read-only memory (EAROM), or flash memory, or a combination of two or more of these. Where appropriate, the RAM can be Static Random-Access Memory (SRAM) or Dynamic Random-Access Memory (DRAM). DRAM can be Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), Extended Data Out Dynamic Random-Access Memory (EDODRAM), Synchronous Dynamic Random-Access Memory (SDRAM), etc.

[0075] The memory 42 can be used to store or cache various data files that need to be processed and / or used for communication, as well as possible computer program instructions executed by the processor 41.

[0076] The processor 41 implements any of the multimodal image fusion methods described in the above embodiments by reading and executing computer program instructions stored in the memory 42.

[0077] In some embodiments, the computer device may further include a communication interface 43 and a bus 40. For example, Figure 8 As shown, the processor 41, memory 42, and communication interface 43 are connected through bus 40 and complete communication with each other.

[0078] The communication interface 43 is used to enable communication between the various modules, devices, units, and / or equipment in the embodiments of this application. The communication interface 43 can also enable data communication with other components such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations.

[0079] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0080] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A multimodal image fusion method, characterized in that, include: The data acquisition steps involve acquiring sMRI and FDG PET images of the subjects and performing preprocessing. The subjects include Alzheimer's disease patients, patients with mild cognitive impairment, and people with normal cognitive status. The slice selection step involves calculating the image entropy of all axial slices for each sMRI image, extracting a preset number of axial slices with the largest image entropy as sMRI key slices, and extracting the FDG PET key slices of the FDG PET image corresponding to the sMRI key slices. The image fusion step involves merging sMRI key sections and their corresponding FDG PET key sections after image registration to obtain a fused image, which is used to identify the disease category to which the image belongs.

2. The multimodal image fusion method according to claim 1, characterized in that, The data acquisition steps include: The sMRI image preprocessing steps include using the CAT12 toolkit to perform skull dissection and tissue segmentation on the sMRI images, and registering and aligning the sMRI images to a standard template space. The FDG PET image preprocessing step involves using the FLIRT module in the FSL software to register each subject's FDG PET image onto its corresponding preprocessed sMRI image.

3. The multimodal image fusion method according to claim 1, characterized in that, The image fusion step further includes: The image registration step involves using an sMRI key slice as the reference image and the corresponding FDG PET key slice as the floating image. A joint histogram of the reference image and the floating image is calculated using a partial volume interpolation algorithm. Based on the joint histogram, the mutual information between the reference image and the floating image is calculated. Finally, an optimization search algorithm is used to find registration parameters that maximize the mutual information. The floating image is transformed according to the registration parameters to register the floating image with the reference image.

4. The multimodal image fusion method according to claim 3, characterized in that, Also includes: The image classification step involves inputting the fused image into a preset classification model to obtain the disease category to which the fused image belongs. The disease categories include: Alzheimer's disease, mild cognitive impairment, and normal cognitive state.

5. The multimodal image fusion method according to claim 4, characterized in that, In the classification model, the input fused image is processed sequentially by the first to the fifth convolutional blocks, then input to the first fully connected layer and the second fully connected layer, and finally output as a classification probability through the Softmax layer.

6. A multimodal image fusion system, characterized in that, include: The data acquisition module acquires sMRI and FDG PET images of the subjects and performs preprocessing. The subjects include Alzheimer's disease patients, patients with mild cognitive impairment, and people with normal cognitive status. The slice selection module calculates the image entropy of all axial slices for each sMRI image, extracts a preset number of axial slices with the largest image entropy as sMRI key slices, and extracts the FDG PET key slices of the FDG PET image corresponding to the sMRI key slices. The image fusion module is used to register and fuse key slices from sMRI and their corresponding key slices from FDG PET to obtain a fused image, which is used to identify the disease category to which the image belongs.

7. The multimodal image fusion system according to claim 6, characterized in that, The data acquisition module includes: The sMRI image preprocessing unit uses the CAT12 toolkit to perform skull dissection and tissue segmentation on the sMRI image, and registers and aligns the sMRI image to a standard template space. The FDG PET image preprocessing unit uses the FLIRT module in the FSL software to register each subject's FDG PET image onto its corresponding preprocessed sMRI image.

8. The multimodal image fusion system according to claim 6, characterized in that, The image fusion module further includes: The image registration unit uses an sMRI key slice as the reference image and the corresponding FDG PET key slice as the floating image. It calculates the joint histogram of the reference image and the floating image using a partial volume interpolation algorithm. Based on the joint histogram, it calculates the mutual information between the reference image and the floating image and then uses an optimization search algorithm to find the registration parameters that maximize the mutual information. The floating image is transformed according to the registration parameters.

9. The multimodal image fusion system according to claim 6, characterized in that, Also includes: The image classification module is used to input the fused image into a preset classification model to obtain the disease category to which the fused image belongs. The disease categories include: Alzheimer's disease, mild cognitive impairment, and normal cognitive state.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multimodal image fusion method as described in any one of claims 1 to 5.

Images