A multi-modal medical image fusion method based on multi-scale transformer

Abstract

The application discloses a multi-modal medical image fusion method based on a multi-scale transformer and belongs to the technical field of medical image fusion. The application proposes a novel and efficient fusion model, designs a multi-scale transformer model to introduce a feature extraction network, so that the feature extraction network can effectively extract multi-scale deep features and reserve more meaningful information for a fusion task; in the process of network training, the receptive field and the patch size are self-adapted, and a structural similarity-based optimization objective function is constructed to constrain the image generation quality; convolution calculation is combined with the transformer to provide better visual effects and quantitative results for medical image fusion results.

Description

Technical Field

[0001] This invention relates to the field of medical image fusion technology, and specifically to a multimodal medical image fusion method based on a multiscale transformer. Background Technology

[0002] Image fusion combines different information from different images to create a new image that integrates various complex information from different images. Image fusion has applications in many fields; for example, infrared and visible light images can be used in the military to improve the detection and reconnaissance capabilities of systems. In medical imaging, image fusion combines details from different modalities of medical images. For example, fusing MRI and SPECT images yields an image that simultaneously preserves the functional metabolic information of the SPECT image and the structural soft tissue information of the MRI. Rain tumor segmentation in multimodal magnetic resonance imaging (MRI) scans is fundamental to obtaining key quantitative indicators such as tumor two-dimensional diameter and tumor volume, and has significant clinical implications in disease diagnosis and treatment efficacy evaluation. Since gliomas are the most common primary malignant brain tumors, most brain tumor segmentation research focuses on gliomas. The typical goal of glioma segmentation is to locate various types of pathological regions within the multimodal MRI volume, including edema (ED), necrosis, non-enhancing tumors (NCR / NET), and enhancing tumors (ET), typically using T1-weighted (T1), contrast-enhanced T1-weighted (T1c), and T2-weighted (T2) imaging. Fusing MRI images from different sequences, including images with different modalities, can help doctors with subsequent diagnosis or other procedures.

[0003] Deep learning-based medical image fusion has achieved good fusion results, but there are still many problems: (1) The fusion framework is only used for specific tasks and has low generalization performance. For example, the framework used to fuse PET and SPECT images has requirements for the resolution of the input network images because the resolution of the two images is quite different, making it difficult to use for fusion of other tasks; (2) Existing deep learning-based multimodal medical image fusion methods are all fusions of two modal images, and there is no fusion of images of more than two modalities; (3) Medical image fusion technology is currently lacking in application. The fusion of two images alone cannot play a significant role in medical diagnosis. It is worth trying to see if it can be used to augment datasets; (4) Existing methods are generally based on CNN methods, which can capture local information well, but it is difficult to capture global information. In order to achieve accurate fusion of the characteristics and commonalities of brain glioma medical image lesion regions, a multimodal medical image fusion method based on multiscale transformer is proposed. Summary of the Invention

[0004] The technical problem to be solved by this invention is: how to achieve accurate fusion of the regional characteristics and commonalities of brain glioma medical imaging lesions, so as to overcome the shortcomings of information fusion in existing multimodal medical image fusion processing, and to provide a multimodal medical image fusion method based on multiscale transformer.

[0005] The present invention solves the above-mentioned technical problems through the following technical solution, and the present invention includes the following steps:

[0006] Step S1: Slice the dataset of brain glioma medical images to remove areas without lesions;

[0007] Step S2: Preprocess the data from the brain glioma slides;

[0008] Step S3: Construct the multi-scale transformer module;

[0009] Step S4: Construct a fusion network mechanism to obtain a multi-scale transformer fusion network, and optimize the training of the multi-scale transformer fusion network using data from the preprocessed training set;

[0010] Step S5: Based on structural similarity measurement, construct a loss function to constrain the generation direction of the image;

[0011] Step S6: After processing in steps S1-S5, a medical image fusion model based on a multi-scale transformer fusion network is obtained. The brain glioma medical image to be fused is input into the medical image fusion model for fusion processing to obtain the fusion result.

[0012] Furthermore, in step S1, the dataset of medical images of brain gliomas includes four image sequences: a FLAIR sequence, a T1-weighted sequence, a contrast-enhanced T1-weighted sequence, and a T2-weighted sequence. These four image sequences are processed simultaneously. The dataset is randomly shuffled, and 30% is selected as the validation set, while the remaining 70% is used as the training set. Within the 70% training set, it is randomly divided into training sets x according to a predetermined ratio. training and validation set x test , where x training ,x test ∈X.

[0013] Furthermore, in step S2, the data preprocessing formula is as follows:

[0014] x i_input =normailze(corp(x) i )),x i ∈x training

[0015] Where, x i_input As input to the multi-scale transform fusion network constructed in step S4, corp(·) is the sampling function, and normailze(·) represents the normalization operation on the sampled data.

[0016] Furthermore, in step S3, the multi-scale transformer module includes four sequentially connected blocks, each block being implemented by convolutional computation of different receptive field sizes and transformer hierarchical connections of different patches.

[0017] Furthermore, the receptive field and patch size of each block are determined by the module's input, and the receptive field and patch size are adaptively adjusted according to the input size. i The input to the i-th block is first processed by a convolution with a receptive field of size p×p. Then, the features are divided into h / p patches, and an attention mechanism is applied to each patch. This process is repeated for four blocks until the feature map is extracted. Then, the next step is performed. The calculation formula for each block is as follows:

[0018] V i =conv2(w i ,x i )+b i

[0019]

[0020] FFZ(Z) = max(0, ZW1, +b1)W2 + b2

[0021] Where conv2(·) represents convolution calculation, w i and b i These are the weights and biases generated during the iteration process, respectively. softmax(·) is the softmax activation function, and max(·) represents taking the maximum value among them.

[0022] Furthermore, the multi-scale transformer fusion network includes a feature extraction network, a fusion module, and a feature reconstruction network; wherein, the feature extraction network includes three convolutional modules and three multi-scale transformer modules connected in a staggered manner, with the convolutional modules preceding the multi-scale transformer modules; the feature reconstruction network includes four upsampling modules connected in a staggered manner, with the last multi-scale transformer module connected to the first upsampling module through the fusion module.

[0023] Furthermore, after each convolutional module processes the input image, it is input into a multi-scale transformer module. After passing through one block in the multi-scale transformer module, the feature map changes, and the number of channels is reduced to half of its original value. The transformed feature map is represented by the following formula:

[0024] feature map=FFZ(Attention(conv(x i )))

[0025] Where conv(·) represents a convolution with a stride of 1 and a kernel size of 3, x i It is the input to the multi-scale hierarchical transformer module.

[0026] Furthermore, the calculation formula for the fusion module is as follows:

[0027]

[0028] In this case, cat(·) means concatenating two tensors, where c, p, N, H and W refer to the number of channels, patch size, number of patches, height and width of the source image, respectively.

[0029] Furthermore, in step S5, the formula for the structural similarity metric index SSIM and the structural similarity metric index loss function are as follows:

[0030]

[0031]

[0032] Where μ(·) is the calculated average, σ(·) is the calculated variance, C1 and C2 are numbers that approach infinity, and G(I1,I2) is the result generated by generator G.

[0033] Furthermore, in step S6, after obtaining the medical image fusion model, a validation set x is used. test The medical image fusion model was tested.

[0034] Compared with existing technologies, this invention has the following advantages: This multimodal medical image fusion method based on multi-scale transformer proposes a novel and efficient fusion model. A multi-scale transformer model is designed and introduced into a feature extraction network, enabling the feature extraction network to effectively extract multi-scale deep features and retain more meaningful information for the fusion task. During network training, the receptive field and patch size are adaptively adjusted, and a structural similarity-based optimization objective function is constructed to constrain the image generation quality. By utilizing convolutional computation combined with Transformer, better visual effects and quantitative results are provided for the medical image fusion results. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the training and testing process of the medical image fusion model in an embodiment of the present invention;

[0036] Figure 2 This is a structural framework diagram of the multi-scale transformer fusion network in an embodiment of the present invention;

[0037] Figure 3 This is a structural framework diagram of the multi-scale transformer module in an embodiment of the present invention. Detailed Implementation

[0038] The embodiments of the present invention are described in detail below. These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. However, the scope of protection of the present invention is not limited to the following embodiments.

[0039] This embodiment provides a technical solution: a multimodal medical image fusion method based on multi-scale transformers, comprising the following main steps:

[0040] Step 1: Slice the dataset of brain glioma medical images to remove areas without lesions;

[0041] Step 2: Preprocess the data from the brain glioma slides;

[0042] Step 3: Construct the multi-scale transformer module;

[0043] Step 4: Construct a fusion network mechanism to obtain a multi-scale transformer fusion network, and optimize the training of the multi-scale transformer fusion network using data from the preprocessed training set;

[0044] Step 5: Based on structural similarity measurement, construct a loss function to constrain the generation direction of the image;

[0045] Step 6: After processing in steps 1-5, a medical image fusion model based on a multi-scale transformer fusion network is obtained. The fusion effect of the medical image fusion model is tested using data from the preprocessed test set.

[0046] The detailed steps are as follows:

[0047] Step 1: The brain glioma medical image dataset includes four image sequences: FLAIR sequence, T1-weighted (T1) sequence, contrast-enhanced T1-weighted (T1c) sequence, and T2-weighted (T2) sequence, all in 3D format (240×240×155 pixels). The four image sequences are processed simultaneously. The dataset is randomly shuffled, and 30% is selected as the validation set, while the remaining 70% is used as the training set. Within this 70% training set, it is randomly divided into training and validation sets according to a certain ratio (a secondary partitioning is performed during training, with a portion used for validation to constrain the training results). All 3D image data are sliced ​​and converted into 2D image data of size 160×160 pixels.

[0048] Step 2: Following the processing in Step 1, the brain glioma medical image dataset is divided into two parts: a training set and a validation set. The training set is used to train the model parameters to achieve the expected results, while the validation set is used to verify the generalization ability of the model trained on the training set. Before being input into the multi-scale transformer fusion network, the training set data is divided into training set x... training and validation set x test , where x training ,x test ∈X. Each data point undergoes pruning and normalization to ensure that each feature is treated equally by the classifier. Data preprocessing can be expressed by the following formula:

[0049] x i_input =normailze(corp(x) i )),x i ∈x training

[0050] Where, x i_input As input to the multi-scale transform fusion network constructed in step 4, corp(·) is the sampling function, and normailze(·) represents the normalization operation on the sampled data.

[0051] Step 3: Construct a multi-scale hierarchical transformer module (multi-scale transformer module). This module includes 4 blocks, each consisting of convolutional computations with different receptive field sizes and transformer hierarchical connections with different patches. This module combines the local feature extraction capability of convolutional computation with the contextual long dependency learning of transformer, making the model more interpretable. Furthermore, using 4 different scale computation modes improves the model's ability to extract both local and global features, resulting in the extraction of more image quality features.

[0052] Furthermore, the receptive field and patch size of each block are determined by the module's input, and the receptive field and patch size are adaptively adjusted according to the input size. i The input to the i-th block is first processed by a convolution with a receptive field of size p×p. Then, the features are divided into h / p patches, and an attention mechanism is applied to each patch. This process continues until the high-quality feature map extracted after four blocks is processed, which is then used in the next step. The calculation formula for each block is as follows:

[0053] V i =conv2(w i ,x i )+b i

[0054]

[0055] FFZ(Z) = max(0, ZW1, +b1)W2 + b2

[0056] Where conv2(·) represents convolution calculation, w i and b i These are the weights and biases generated during the iteration process, respectively. softmax(·) refers to the softmax activation function, and max(·) means taking the maximum value among them.

[0057] Step 4: The multi-scale hierarchical transformer module constructed in Step 3 is part of the feature extraction part of the entire network. The multi-scale transformer fusion network consists of three parts: feature extraction, fusion, and feature reconstruction.

[0058] Furthermore, the feature extraction part is implemented through three convolutional calculations, reLU function calculation, and max pooling. The feature map is then fed into the multi-scale hierarchical transformer module constructed in step 3. After each block, the feature map changes, and the number of channels is halved. The transformed feature map can be represented by the following formula:

[0059] feature map=FFZ(Attention(conv(x i )))

[0060] Where conv(·) represents a convolution with a stride of 1 and a kernel size of 3, x i It is the input to the multi-scale hierarchical transformer module;

[0061] Furthermore, the extracted feature maps contain rich information. In this step, the feature maps that need to be fused are fused. Based on the similarity of brightness and texture on the feature maps, the resulting fusion map focuses more on typical targets and details. The fusion map calculation formula is as follows:

[0062]

[0063] In this case, cat(·) means concatenating two tensors, where c, p, N, H and W refer to the number of channels, patch size, number of patches, height and width of the source image, respectively.

[0064] Furthermore, the feature reconstruction part includes four upsampling calculations. The fusion map contains rich feature information, but it is all contour information and lacks semantic information. Feature reconstruction through upsampling calculations generates a fusion image containing rich lesion details from the source image. Each upsampling calculation is implemented using a 3×3 convolution kernel, a convolution with a stride of 1, and a ReLU activation function. The calculation principle of the four blocks (the calculation process of the four upsamplings) is the same, but the size of the resulting feature map is doubled, and a linearization operation and Maxpooling reduction are performed after the four blocks.

[0065] Step 5: The Structural Similarity Metric (SSIM) serves as the primary objective metric for constructing the objective optimization function. Throughout the training process of the multi-scale transformer fusion network, these different loss functions are weighted by the pre-set hyperparameter α, thereby calculating the overall loss function of the multi-scale transformer fusion network. The SSIM constrains the way the similarity between two images is measured. It calculates the content difference between the generated image and the two source images, constraining the visual perception quality of the generated image from the perspectives of brightness, contrast, and structure. The formula for SSIM and the SSIM loss function are shown below:

[0066]

[0067]

[0068] Where μ(·) is the calculated average, σ(·) is the calculated variance, C1 and C2 are numbers that approach infinity, and G(I1,I2) is the network generation result to avoid calculation anomalies.

[0069] Step 6: After data preprocessing in steps 1-5, construction of the multi-scale transformer fusion network, and training and optimization of the objective function, the optimal fusion model is obtained. This model is then used on the validation set x. test The model is used to predict outcomes and verify its objective and subjective effectiveness, performing qualitative and quantitative evaluations. The test results can serve as data preprocessing for other downstream tasks, such as medical image segmentation and classification, and also as an adjunct to medical diagnosis.

[0070] In summary, the multimodal medical image fusion method based on multi-scale transformers described above proposes a novel and efficient fusion model. A multi-scale transformer model is designed and introduced into a feature extraction network, enabling the network to effectively extract multi-scale deep features and retain more meaningful information for the fusion task. During network training, the receptive field and patch size are adaptively adjusted, and a structural similarity-based optimization objective function is constructed to constrain the image generation quality. By utilizing convolutional computation combined with the Transformer, better visual effects and quantitative results are provided for the medical image fusion.

[0071] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A multimodal medical image fusion method based on multi-scale transformer, characterized in that, Includes the following steps: Step S1: Slice the dataset of brain glioma medical images to remove areas without lesions; Step S2: Preprocess the data from the brain glioma slides; Step S3: Construct the multi-scale transformer module; Step S4: Construct a fusion network mechanism to obtain a multi-scale transformer fusion network, and optimize the training of the multi-scale transformer fusion network using data from the preprocessed training set; Step S5: Based on structural similarity measurement, construct a loss function to constrain the generation direction of the image; Step S6: After processing in steps S1-S5, a medical image fusion model based on a multi-scale transformer fusion network is obtained. The brain glioma medical image to be fused is input into the medical image fusion model for fusion processing to obtain the fusion result. In step S3, the multi-scale transformer module includes four sequentially connected blocks, each block being implemented by convolutional computation of different receptive field sizes and transformer hierarchical connections of different patches; The receptive field and patch size of each block are determined by the input to that block, and the receptive field and patch size are adaptively adjusted according to the input size. As the first The input of each block first passes through the receptive field. The convolution calculation is performed on the size of the feature, and then the feature is divided into... The algorithm iterates through four patches, then applies an attention mechanism to each patch, and so on. After processing four blocks, a feature map is extracted, and the next step is performed. The calculation formula for each block is as follows: ； ； ； in, This represents convolution calculation. and These are the weights and biases generated during the iteration process, respectively. The softmax activation function is used. This indicates taking the maximum value among them.

2. The multimodal medical image fusion method based on multi-scale transformer according to claim 1, characterized in that: In step S1, the dataset of brain glioma medical images includes four image sequences: a FLAIR sequence, a T1-weighted sequence, a contrast-enhanced T1-weighted sequence, and a T2-weighted sequence. These four image sequences are processed simultaneously. The dataset is randomly shuffled, and 30% is selected as the validation set, while the remaining 70% is used as the training set. Within the 70% training set, the data is randomly divided into training sets according to a predetermined ratio. and verification set ,in .

3. The multimodal medical image fusion method based on multi-scale transformer according to claim 2, characterized in that: In step S2, the data preprocessing formula is as follows: ； in, As input to the multi-scale transform fusion network constructed in step S4, For sampling function, This indicates that the sampled data is normalized.

4. The multimodal medical image fusion method based on multi-scale transformer according to claim 3, characterized in that: The multi-scale transformer fusion network includes a feature extraction network, a fusion module, and a feature reconstruction network. The feature extraction network includes three convolutional modules and three multi-scale transformer modules connected in a staggered manner, with the convolutional modules preceding the multi-scale transformer modules. The feature reconstruction network includes four upsampling modules connected in a staggered manner, with the last multi-scale transformer module connected to the first upsampling module through the fusion module.

5. The multimodal medical image fusion method based on multi-scale transformer according to claim 4, characterized in that: After each convolutional module processes the input image, it is fed into a multi-scale transformer module. Each time the image passes through one block in the multi-scale transformer module, the feature map changes, and the number of channels is halved. The transformed feature map is represented by the following formula: ； in, This represents a convolution calculation with a stride of 1 and a kernel size of 3. It is the input to the multi-scale hierarchical transformer module.

6. The multimodal medical image fusion method based on multi-scale transformer according to claim 5, characterized in that: The calculation formula for the fusion module is as follows: ； in, This indicates that two tensors are concatenated. c, p, N, H and W refer to the number of channels, patch size, number of patches, height and width of the source image, respectively.

7. The multimodal medical image fusion method based on multi-scale transformer according to claim 6, characterized in that: In step S5, the formula for the structural similarity metric index SSIM and the structural similarity metric index loss function are as follows: ； ； in, To calculate the average, To calculate the variance, and It is a number that approaches infinity. This represents the network generation result.

8. The multimodal medical image fusion method based on multi-scale transformer according to claim 2, characterized in that: In step S6, after obtaining the medical image fusion model, a validation set is used. The medical image fusion model was tested.

Images