A multi-modal medical image fusion method based on multi-scale codec

Abstract

The application discloses a kind of multi-modal medical image fusion methods based on multi-scale codec, comprising:1 data preprocessing;2 multi-scale codec network is constructed, comprising: multi-scale encoder and decoder, wherein, encoder is combined with convolution and visual converter, mainly multi-scale feature extraction is carried out;Decoder is cross-modal fusion module, multi-scale feature fusion module and upsampling module, mainly to the multi-scale feature of encoder extraction is cross-modal fusion and multi-scale interaction, finally upsampling obtains fusion result;3 multi-scale codec network is trained, and the trained multi-scale codec network is used to carry out image fusion.The application can make full use of the complementary information and multi-scale features in different modal medical images to fuse better quality medical images, provide better quality images for human eye observation, so as to promote the execution of subsequent computer vision tasks, provide favorable support for clinical diagnosis and surgical navigation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal medical image fusion, and in particular to a multi-modal medical image fusion method based on multi-scale. BACKGROUND

[0002] Due to the physical limitations of imaging technology, a single imaging mode can only reflect the characteristics of tissues or organs in some aspects. Different medical imaging modes can convey unique and complementary understanding of body structures and tissues. For example, anatomical images have high spatial resolution and can clearly depict the anatomical structure of organs, but cannot reflect the metabolism of the human body. Magnetic resonance imaging (MRI) is a typical anatomical mode. It provides rich soft tissue information and high resolution. In contrast, functional images can reflect functional and metabolic information, but due to their low resolution, they cannot describe the anatomical details of organs. Low-resolution images such as positron emission tomography (PET) and single photon emission tomography (SPECT).

[0003] Considering the limitations of single-mode images, the purpose of multi-modal medical image fusion is to fuse typical and complementary information together to obtain high-quality medical images. The existing methods have the following problems: first, the convolution-based method has the problem of local extraction of convolution features, which cannot effectively extract the global information important for image fusion, affecting the final fusion result; second, the model based on visual transformer can effectively extract global information of the image, but it lacks the ability to extract local features, and in addition, it has the problem of excessive calculation due to the calculation of the square relationship with the size of the image; third, the current multi-modal medical image fusion network lacks effective fusion and interaction of multi-scale features; in addition, the commonly used loss function is mostly based on structural similarity and pixel intensity, which are constraints from the spatial domain, without considering the constraints from the frequency domain, resulting in poor fusion results. SUMMARY

[0004] To overcome the problems of existing image fusion in multi-modal medical image fusion, the present application provides a multi-modal medical image fusion method based on multi-scale encoder-decoder, which can make full use of the complementary and redundant information of different modal medical images to provide better image feature expression and reconstruct higher quality images, thereby providing better quality images for human eye observation and supporting computer vision tasks such as image segmentation and classification.

[0005] The present application adopts the following technical solutions to solve the above problems:

[0006] The application discloses a multi-modal medical image fusion method based on a multi-scale codec.

[0007] Step 1: obtaining two different modal medical image pairs of the same part for gray scale processing to obtain gray scale images under two different modalities; and dividing the gray scale images under two different modalities into T image blocks respectively to obtain image block sets under two different modalities Wherein, denotes the tth pair of image blocks under the structure imaging modality and the functional imaging modality.

[0008] Step 2, constructing a multi-scale codec network, including a multi-scale encoder and a decoder.

[0009] Step 2.1: the multi-scale encoder includes a convolution block A and K encoding modules; each encoding module includes a convolution layer, a gradient layer and a visual converter layer.

[0010] The tth pair of gray scale images are input into the multi-scale encoder and are sequentially processed through the convolution block A and the K encoding modules to obtain multi-scale feature maps Wherein, denotes a pair of feature maps output by the kth encoding module under two different modalities.

[0011] Step 2.2: the decoder network sequentially includes K cross-modal fusion modules, K-2 multi-scale feature interaction modules, K-3 up-sampling modules and one convolution up-sampling module, and the are processed to obtain the tth fusion result I t f .

[0012] Step 3: constructing a total loss function L of the multi-scale codec network by using formula (1) total :

[0013] L total =L spa +λ·L fre (1)

[0014] In formula (1), λ is a weight coefficient of a frequency domain loss, L spa denotes a spatial loss function and is obtained by formula (2), and L fre denotes a frequency domain loss function and is obtained by formula (7).

[0015] L spa =L ssim +α·L int (2)

[0016] In formula (2), L ssimLet L represent the structural similarity loss, which is obtained from equation (3). int The loss is the pixel intensity loss, which is obtained from equation (4), where α is the weight coefficient corresponding to the loss;

[0017]

[0018]

[0019] In equation (3), SSIM represents structural similarity: and The weighting coefficients are for two different modes, and are obtained from equation (5):

[0020]

[0021] In equation (5), and The significance measurements under two different modalities are represented and obtained from equation (6):

[0022]

[0023] In equation (6), H and W represent the length and width of the image block, and x and y represent the horizontal and vertical coordinates of the pixel in the image block; Represents the t-th image patch in the structured imaging mode. The pixel value at (x,y) in the middle. Represents the t-th image patch in the functional imaging modality. The pixel value at (x, y);

[0024]

[0025] In equation (7), β represents the weighting coefficient, and L ffl The introduced focusing frequency domain loss is obtained from equation (8):

[0026]

[0027] In equation (8), It is the t-th image patch in the structured imaging mode. The frequency domain representation of the pixel at (x,y). It is the t-th image patch in the structured imaging mode. The frequency domain representation of the pixel at (x,y). It is the fusion result of the t-th input image patch. The frequency domain representation of the pixel at (x,y);

[0028] Step 4: Train the multi-scale codec network using the backpropagation algorithm and calculate the total loss function L. totalAdjust the network parameters until the set maximum number of iterations is reached, so as to obtain a trained multi-scale codec fusion network for fusing a to-be-processed multi-modal medical image pair, and the obtained fusion image is subjected to color gamut conversion, so as to finally obtain a color fusion image result.

[0029] The multi-modal medical image fusion method based on a multi-scale codec according to the application also has the following characteristics.

[0030] Step 2.2.1: inputting the first cross-modal feature map into the first cross-modal fusion module for processing to obtain the first cross-modal feature map Step 2.2.2: when k = 1, 2, …, K-2, inputting the kth cross-modal feature map into the kth cross-modal fusion module for processing to obtain the kth cross-modal feature map Step 2.2.3: inputting the K-2th cross-modal feature map into the K-3th cross-modal fusion module for processing to obtain the K-3th cross-modal feature map Step 2.2.4: inputting the first up-sampling feature map into the convolution up-sampling module for linear interpolation up-sampling and convolution operation to obtain a convolution feature, and then inputting the convolution feature into the Sigmoid function to obtain a weight matrix, and then inputting the weight matrix into the weighted average operation to obtain the tth fusion result

[0031] Step 2.2.2: when k = 1, 2, …, K-2, inputting the kth cross-modal feature map into the kth cross-modal fusion module for processing to obtain the kth cross-modal feature map Step 2.2.3: inputting the K-2th cross-modal feature map into the K-3th cross-modal fusion module for processing to obtain the K-3th cross-modal feature map Step 2.2.4: inputting the first up-sampling feature map into the convolution up-sampling module for linear interpolation up-sampling and convolution operation to obtain a convolution feature, and then inputting the convolution feature into the Sigmoid function to obtain a weight matrix, and then inputting the weight matrix into the weighted average operation to obtain the tth fusion result

[0032] Step 2.2.2: when k = 1, 2, …, K-2, inputting the kth cross-modal feature map into the kth cross-modal fusion module for processing to obtain the kth cross-modal feature map Step 2.2.3: inputting the K-2th cross-modal feature map into the K-3th cross-modal fusion module for processing to obtain the K-3th cross-modal feature map Step 2.2.4: inputting the first up-sampling feature map into the convolution up-sampling module for linear interpolation up-sampling and convolution operation to obtain a convolution feature, and then inputting the convolution feature into the Sigmoid function to obtain a weight matrix, and then inputting the weight matrix into the weighted average operation to obtain the tth fusion result Step 2.2.2: when k = 1, 2, …, K-2, inputting the kth cross-modal feature map into the kth cross-modal fusion module for processing to obtain the kth cross-modal feature map Step 2.2.3: inputting the K-2th cross-modal feature map into the K-3th cross-modal fusion module for processing to obtain the K-3th cross-modal feature map Step 2.2.4: inputting the first up-sampling feature map into the convolution up-sampling module for linear interpolation up-sampling and convolution operation to obtain a convolution feature, and then inputting the convolution feature into the Sigmoid function to obtain a weight matrix, and then inputting the weight matrix into the weighted average operation to obtain the tth fusion result Step 2.2.2: when k = 1, 2, …, K-2, inputting the kth cross-modal feature map into the kth cross-modal fusion module for processing to obtain the kth cross-modal feature map Step 2.2.3: inputting the K-2th cross-modal feature map into the K-3th cross-modal fusion module for processing to obtain the K-3th cross-modal feature map

[0033] Step 2.2.4: inputting the first up-sampling feature map into the convolution up-sampling module for linear interpolation up-sampling and convolution operation to obtain a convolution feature, and then inputting the convolution feature into the Sigmoid function to obtain a weight matrix, and then inputting the weight matrix into the weighted average operation to obtain the tth fusion result Step 2.2.4: inputting the first up-sampling feature map into the convolution up-sampling module for linear interpolation up-sampling and convolution operation to obtain a convolution feature, and then inputting the convolution feature into the Sigmoid function to obtain a weight matrix, and then inputting the weight matrix into the weighted average operation to obtain the tth fusion result

[0034] The electronic device according to the application comprises a memory and a processor, wherein the memory is configured to store a program supporting the processor to execute the multi-modal medical image fusion method, and the processor is configured to execute the program stored in the memory.

[0035] The computer readable storage medium of the present application, the computer readable storage medium stores a computer program, characterized in that, the computer program is run by the processor to execute the steps of the multi-modal medical image fusion method.

[0036] Compared with the prior art, the beneficial effects of the present application are reflected in:

[0037] 1、The present application proposes a multi-scale codec network to realize the multi-modal medical image fusion task, which fully utilizes the redundancy and complementary information between different modal images, and fuses high-quality images, and the existing method, which obtains competitive results in visual effect and evaluation index.

[0038] 2、The present application designs a multi-scale feature combining CNN and transformer from top to bottom and bottom to top, which effectively extracts local and global features; The model is designed as a multi-scale structure, which can better make each different scale feature have strong semantic information, and comprehensively consider the details of the bottom layer and the semantic information of the high layer, so as to bring better performance to the fusion result.

[0039] 3、The present application designs a decoder based on multi-scale feature interaction, which can effectively fuse multi-scale features, fully utilize deep features, and retain more information of different scales extracted by the codec network to prevent the network from losing shallow features while extracting deeper features, so that the feature information extracted by the network is more comprehensive, and then the multi-scale features obtained by the decoder can be fully utilized to strengthen the image quality of the fusion, in addition, a cross-modal feature fusion module and a multi-scale feature interaction module are designed to process the complementary information between different modalities, so as to obtain better fusion results.

[0040] 4、The present application designs a loss function combining spatial domain and frequency domain to improve the effect of network training and strengthen the image quality of fusion. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 The flow chart of the multi-modal medical image fusion method based on multi-scale codec of the present application;

[0042] Figure 2 The network structure diagram of the present application;

[0043] Figure 3 The encoder module structure diagram of the present application;

[0044] Figure 4 The cross-modal fusion module structure diagram of the present application;

[0045] Figure 5A schematic diagram of a multi-scale fusion module structure of the present application;

[0046] Figure 6 A schematic diagram of a polymeric up-sampling module structure of the present application. DETAILED DESCRIPTION

[0047] In this embodiment, the flow of a multi-modal medical image fusion method based on a multi-scale codec is as shown in Figure 1 , which includes the following steps:

[0048] Step 1: Obtain two different modal medical image pairs of the same part for gray scale processing to obtain gray scale images under two different modalities; and divide the gray scale images under two different modalities into T image blocks respectively to obtain image block sets under two different modalities , wherein the tth pair of image blocks under the structural imaging modality and the functional imaging modality;

[0049] Step 2: Construct a multi-scale codec network as shown in Figure 2 , which includes a multi-scale encoder and a decoder.

[0050] Step 2.1: The multi-scale encoder includes a convolution block A and K encoding modules, each of which is a combined module of a plurality of convolution layers, gradient layers and visual converter layers connected in parallel, and a bidirectional interaction layer of space and channel is added between the visual converter layer and the convolution layer branch, the features obtained by the visual converter layer branch and the convolution layer are spliced along the channel, then passed through a feedforward neural network module, and then added to the features obtained by the gradient layer and transmitted to the next layer, and the specific structure is as shown in Figure 3 . The channel interaction layer is composed of a global pooling layer GAP, a 1x1 convolution layer, a BN layer, a GELU layer, a 1x1 convolution layer and a Sigmoid layer, and the space interaction layer is composed of a 1x1 convolution layer, a BN layer, a GELU layer, a 1x1 convolution layer and a sigmoid layer. In a specific implementation, the number of layers of the overall framework is K=5, the number of combined modules of each layer is {2, 2, 4, 8, 8}, the output channel number of the convolution block A is 8, the output channel number of the first layer encoding module is 16, the output channel number of the second layer encoding module is 32, the output channel number of the third layer encoding module is 64, the output channel number of the fourth layer encoding module is 128, the output channel number of the fifth layer encoding module is 256, and the window size of the visual converter layer in the encoding module is 7x7. The convolution kernel size of the convolution layer is 3x3.

[0051] the tth pair of gray scale images After being processed by the convolution block A, the input is sequentially processed by K Encoders to obtain K groups of corresponding multi-scale feature maps , wherein, a pair of feature maps output by the kth encoding module in two different modalities;

[0052] The decoder network of step 2.2 includes K cross-modal fusion modules, K-2 multi-scale feature interaction modules, K-3 up-sampling modules, and a convolution up-sampling module.

[0053] Step 2.2.1: the feature pair is input into the kth cross-modal fusion module for processing to obtain the kth cross-modal feature map Thus, the feature pair is processed by the K cross-modal fusion modules to obtain cross-modal feature maps of different scales The specific structure of the cross-modal fusion module is shown in Figure 4 In specific implementation, the feature pair and the fusion feature obtained by adding the two are respectively subjected to a 1x1 convolution and a 3x3 convolution, and then two single-modal spatial attention matrices and a cross-modal spatial attention matrix are obtained through a Sigmoid. Then, the two single-modal attention matrices are multiplied by the cross-modal attention matrix and the corresponding input modality, respectively, to obtain two features with complementary information, which are added to the original feature pair, respectively. Then, the feature pairs are spliced along the channel, and finally a 1x1 convolution is performed to obtain the final cross-modal feature K is 5 in the present example.

[0054] Step 2.2.2: when k = 1, 2, …, K-2, the feature pair is input into the kth multi-scale feature interaction module for effective interaction to obtain the kth multi-scale interaction feature map Thus, K-2 multi-scale interaction feature maps are obtained The structure of the multi-scale feature interaction module is shown in Figure 5 In specific implementation, the k+2 layer and the k+1 layer are first up-sampled and down-sampled to keep the spatial size consistent with the k layer, and then residual convolution layers are used to further obtain information of different scales. After splicing along the channel, a 1x1 convolution is used to obtain the final multi-scale interaction feature.

[0055] Step 2.2.3: the K-2 multi-scale interaction feature map and the K-3 multi-scale interaction feature map are processed by the K-3 up-sampling module to obtain the K-3 up-sampled feature map the K-4 multi-scale interaction feature map and the K-3 up-sampled feature map are processed by the K-4 up-sampling module to obtain the K-4 up-sampled feature map Thus, the K-3 up-sampling modules are used to process the first up-sampling feature map to obtain the up-sampling feature map The specific structure of the up-sampling module is shown in Figure 6 .

[0056] Step 2.2.4: The first up-sampling feature map is processed by the convolution up-sampling module to obtain a convolution feature, and then the convolution feature is processed by a Sigmoid function to obtain a weight matrix. After weighted average operation is performed on the weight matrix, the t-th fusion result is obtained

[0057] Step 3: There is a lack of basic facts in multi-modal medical image fusion, which poses a challenge to the training of deep learning models. Existing methods usually define loss functions in the spatial domain, and rarely consider optimization in the frequency domain, which may lead to large differences between the fused image and the source image in the frequency domain. Research has found that narrowing the frequency gap can effectively improve the quality of image reconstruction and synthesis tasks. In this paper, we introduce a recently proposed frequency domain loss called focused frequency loss (FFL) into the study of image fusion. In order to preserve more important information, the invention designs a loss function composed of spatial domain constraints and frequency domain constraints. The proposed fusion model is optimized by minimizing the following loss function. Equation (1) is the total loss function L for constructing a multi-scale encoder-decoder network:

[0058] L total =L spa +λ·L fre (1)

[0059] In equation (1), λ is the weight coefficient of the frequency domain loss, and λ is set to 300 in this example. L spa represents the loss function in the spatial domain, and is obtained from equation (2). L fre represents the loss function in the frequency domain, and is obtained from equation (7)

[0060] L spa =L ssim +α·L int (2)

[0061] In equation (2), L ssim represents the structural similarity loss, and is obtained from equation (3). L int is the loss of pixel intensity, and is obtained from equation (4). α is the weight coefficient of the loss, and is set to 10 in this example.

[0062]

[0063]

[0064] ​In formula (3), SSIM represents structural similarity: and are weight coefficients in two different modalities, and are obtained from formula (5):

[0065]

[0066] In formula (5), and represent the saliency measurement values in two different modalities, and are obtained from formula (6):

[0067]

[0068] In formula (6), H and W represent the length and width of the image block, and x and y represent the horizontal and vertical coordinates of the pixel in the image block; represents the pixel value at (x, y) in the tth image block in the structural imaging modality, represents the pixel value at (x, y) in the tth image block in the functional imaging modality;

[0069]

[0070] In formula (7), β represents a weight coefficient, which is set to 0.1 in the present example, and L ffl is the introduced focus frequency domain loss, and is obtained from formula (8):

[0071]

[0072] In formula (8), is the frequency domain representation of the pixel at (x, y) in the tth image block in the structural imaging modality, is the frequency domain representation of the pixel at (x, y) in the tth image block in the structural imaging modality, is the frequency domain representation of the pixel at (x, y) in the tth image block in the structural imaging modality, is the frequency domain representation of the pixel at (x, y) in the fusion result of the tth pair of input image blocks. Step 4: The multi-scale encoder-decoder network is trained using the back propagation algorithm, and the total loss function L total is calculated to adjust the network parameters until the set maximum number of iterations is reached, thereby obtaining the trained multi-scale encoder-decoder fusion network, which is used to fuse the multi-modal medical image pairs to be processed, and the obtained fusion image is subjected to color gamut conversion, thereby finally obtaining a color fusion image result.

[0073] Step 4: The multi-scale encoder-decoder network is trained using the back propagation algorithm, and the total loss function L total is calculated to adjust the network parameters until the set maximum number of iterations is reached, thereby obtaining the trained multi-scale encoder-decoder fusion network, which is used to fuse the multi-modal medical image pairs to be processed, and the obtained fusion image is subjected to color gamut conversion, thereby finally obtaining a color fusion image result.

[0074] ​In this embodiment, an electronic device includes a memory for storing a program supporting a processor to execute the above method, and the processor configured to execute the program stored in the memory.

[0075] In this embodiment, a computer readable storage medium has a computer program stored thereon, and the computer program is run by a processor to perform the steps of the above method.

Claims

1. A multimodal medical image fusion method based on a multi-scale codec, characterized in that, Includes the following steps: Step 1: Acquire two medical image pairs of the same body part from different modalities and perform grayscale processing to obtain grayscale images in two different modalities; then divide each grayscale image in two different modalities into T image blocks to obtain image block sets for the two different modalities. in, This represents the t-th image patch in both the structural imaging mode and the functional imaging mode; Step 2: Construct a multi-scale encoder-decoder network, including: a multi-scale encoder and a decoder; Step 2.1: The multi-scale encoder includes: a convolutional block A and K encoding modules; each encoding module includes a convolutional layer, a gradient layer, and a visual converter layer; The t-th pair of grayscale images The input is processed by the multi-scale encoder and sequentially through convolutional block A and K encoding modules to obtain the multi-scale feature map. in, This represents a pair of feature maps output by the k-th encoding module under two different modalities; Step 2.2: The decoder network sequentially includes: K cross-modal fusion modules, K-2 multi-scale feature interaction modules, K-3 upsampling modules, and one convolutional upsampling module, and performs... After processing, the t-th fusion result is obtained. Step 3: Construct the total loss function L of the multi-scale codec network using equation (1). total : L total =L spa +λ·L fre (1) In equation (1), λ represents the weighting coefficient of the frequency domain loss, and L spa Let L represent the loss function of the space, which is obtained from equation (2). fre Let the frequency domain loss function be represented, and it is obtained from equation (7); L spa =L ssim +α·L int (2) Equation (2) yields L ssim Let L represent the structural similarity loss, which is obtained from equation (3). int The loss is the pixel intensity loss, which is obtained from equation (4), where α is the weight coefficient corresponding to the loss; In equation (3), SSIM represents structural similarity: and The weighting coefficients are for two different modes, and are obtained from equation (5): In equation (5), and The significance measurements under two different modalities are represented and obtained from equation (6): In equation (6), H and W represent the length and width of the image block, and x and y represent the horizontal and vertical coordinates of the pixel in the image block; Represents the t-th image patch in the structured imaging mode. The pixel value at (x,y) in the middle. Represents the t-th image patch in the functional imaging modality. The pixel value at (x, y); In equation (7), β represents the weighting coefficient, and L ffl The introduced focusing frequency domain loss is obtained from equation (8): In equation (8), It is the t-th image patch in the structured imaging mode. The frequency domain representation of the pixel at (x,y). It is the t-th image patch in the structured imaging mode. The frequency domain representation of the pixel at (x,y). It is the fusion result of the t-th input image patch. The frequency domain representation of the pixel at (x,y); Step 4: Train the multi-scale codec network using the backpropagation algorithm and calculate the total loss function L. total The network parameters are adjusted until the maximum number of iterations is reached, thereby obtaining a trained multi-scale encoder-decoder fusion network, which is used to fuse the multimodal medical image pairs to be processed. The fused image is then subjected to color gamut conversion, thus finally obtaining the color fused image result.

2. The multimodal medical image fusion method based on a multi-scale codec according to claim 1, characterized in that, Step 2.2 includes: Step 2.2.1: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation The k-th cross-modal fusion input is processed to obtain the k-th cross-modal feature map. Thus, K cross-modal fusion modules are used to... The processing yields cross-modal feature maps at different scales. Step 2.2.2: When k = 1, 2, ..., K-2, ... Effective interaction is performed in the k-th multi-scale feature interaction module to obtain the k-th multi-scale interactive feature map. This yields K-2 multi-scale interactive feature maps. Step 2.2.3: Transfer the (K-2)th multi-scale interactive feature map and the (K-3)th multi-scale interactive feature map After processing by the (K-3)th upsampling module, the (K-3)th upsampled feature map is obtained. The (K-4)th multi-scale interactive feature map and the (K-3)th upsampled feature map After processing by the (K-4)th upsampling module, the (K-4)th upsampled feature map is obtained. Thus, K-3 upsampling modules are used for After processing, the upsampled feature map is obtained. Step 2.2.4: The convolutional upsampling module upsamples the first feature map. After performing linear interpolation upsampling and convolution operations, convolutional features are obtained. These features are then processed by the Sigmoid function to obtain a weight matrix. Finally, a weighted average is performed on the weight matrix to obtain the t-th fusion result.

3. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports the processor in executing the multimodal medical image fusion method of claim 1 or 2, and the processor is configured to execute the program stored in the memory.

4. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program, when run by a processor, executes the steps of the multimodal medical image fusion method according to claim 1 or 2.

Images