Multimodal medical image fusion method based on spatial attention and reversible neural network

Abstract

The application discloses a kind of multi-modal medical image fusion methods based on spatial attention and reversible neural network, comprising:1, input two different modal images, respectively through two convolution layers to initialize feature mapping and obtain F1 0 And F2 0 ;2, the initialization feature mapping is input into feature extraction module;3, repeat step 2, the result obtained by each feature extraction module is input into the next feature extraction module;4, all obtained different stage feature information is sent into the aggregation module based on spatial attention mechanism to carry out feature integration;5, the spatial integration feature is sent into reversible convolution network together with the depth feature obtained in the last stage to carry out feature fusion, finally, cross-scale fusion feature information is reconstructed by a series of convolutional neural network Fusion image.The application can capture complementary information of different modalities, improve the fusion effect, and thus assist clinical analysis.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of medical image processing, and discloses a multi-modal medical image fusion method based on spatial attention and reversible neural networks. BACKGROUND

[0002] Magnetic resonance imaging (MRI) is a safe, high spatial resolution, non-invasive imaging method, which is widely used in medical research. By changing the sampling parameters of the magnetic resonance signal, different image sequences, i.e. multi-modal MR images, can be obtained. Different modal images may contain specific structural information, such as tumor location, edema area, lesion details, etc. However, in actual clinical applications, the location, size and other information of the tumor are usually evaluated based on a single modal MR image, which may ignore important features in other modal images. Combining different sequence mode MR images to extract the complementary information contained therein and generating a new fused image is a problem to be solved.

[0003] Image fusion technology utilizes the complementarity of information to make the fused image have a more comprehensive and clear description of the scene, and has attracted great attention in various fields. Due to the different levels of image representation, image fusion can be generally divided into three levels of fusion: pixel-level fusion, feature-level fusion and decision-level fusion. Pixel-level fusion, as the most basic operation among the three levels, is a research hotspot in the field of image fusion. The image fusion algorithm at this level directly operates on the pixels of the source images, and its advantage is that it can preserve the details and most of the original information of the images. Previous research work has tried to explore traditional spatial domain-based methods to calculate the weighted average of the pixel-level significance of two source images, or to transform the source images into the wavelet domain to obtain different frequency components. However, the manually designed feature extraction method may not effectively preserve important information in the source images, resulting in artifacts in the fused image. In addition, most traditional methods are limited to pre-designed fusion rules, which reduces the performance of the algorithm.

[0004] With the development of deep learning, researchers have tried to apply deep learning techniques to image fusion and design end-to-end models without fusion rules. For example, a convolution neural network (CNN)-based image fusion method calculates the Laplacian coefficients using a convolution neural network to obtain a fused image. In addition, a general image fusion framework IFCNN is proposed, which selects appropriate fusion rules for feature maps according to the type of input images, and finally uses two convolution layers to reconstruct the feature maps to obtain the output image.

[0005] Although end-to-end models have made significant progress, most existing methods focus more on the feature map captured in the last stage, ignoring the potential connection between spatial features obtained from different modalities at different stages, which can lead to the loss of some detailed anatomical texture information. In this process, some structural edge details from different models may be lost. In addition, most models may lose information when fusing features, and image fusion algorithms also face the dilemma of lacking unified standards. SUMMARY

[0006] The present application is to solve the problems existing in the prior art, and proposes a multi-modal medical image fusion model based on spatial attention and reversible neural network, so as to reflect the characteristics of different modal MR images in the fused image, thereby improving the fusion accuracy and being beneficial to the subsequent segmentation task, and providing a new method for medical image fusion.

[0007] In order to achieve the above application purposes, the technical scheme adopted by the present application is as follows:

[0008] The multi-modal medical image fusion method based on spatial attention and reversible neural network of the present application is characterized in that the following steps are performed:

[0009] Step 1: Preprocess two magnetic resonance images of different modalities respectively to obtain preprocessed medical images I1 and I2 of the two modalities. Wherein, C, H, W represent the channel number, height and width of the image respectively;

[0010] Step 2: Construct an image fusion network based on spatial attention and reversible convolution, including an initial mapping layer, a feature extraction module, a multi-stage attention module and a fusion module based on a reversible neural network.

[0011] Step 2.1: The initial mapping layer uses a convolution kernel of n1x n1 to process the medical images I1 and I2 of the two modalities to obtain the first modality initial feature And the second modality initial feature

[0012] Step 2.2: The feature extraction module is composed of m feature extraction units, each of which includes 2 residual connection modules and 1 texture separation module, for processing the first modality initial feature And the second modality initial feature To obtain the second modality combined feature at different stages;

[0013] Step 2.3: The multi-stage attention module is composed of a Softmax layer and a series of reshaping operations and transpose operations.

[0014] First mode features Second mode combined features After connecting along the new dimension, the multi-stage total feature is obtained. The input is then fed into the multi-stage attention module, where the multi-stage total feature F is first reshaped and expanded into a feature matrix. Then the feature matrix After performing the transpose operation, we obtain the transpose characteristic matrix. Then the feature matrix With the transpose characteristic matrix After multiplication, the matrix is ​​passed through a Softmax layer and the output is an attention feature weight affinity matrix.

[0015] The multi-stage attention module combines the attention feature weight affinity matrix W with the transposed feature matrix. After multiplication, we obtain the feature matrix F' based on the attention mechanism. att Then for F' att After the reshaping operation, the superimposed multi-stage attention feature is obtained by adding it to the multi-stage total feature F. Ultimately, the multi-stage attention features will be superimposed. Remodeling into spatial attention features And serve as the output of the multi-stage attention module;

[0016] Step 2.4: The fusion module based on the reversible neural network consists of 3 HIN modules and 2 convolutional layers with n5×n5 kernels; each HIN module consists of 2 convolutional layers with n4×n4 kernels.

[0017] Integrate the second mode of the m-th stage with features Spatial attention features F att The input is processed in the fusion module based on the reversible neural network, wherein the second modality of the m-th stage combines features. The input features are obtained by first changing the channel dimension through the first n4×n4 convolutional layer of the first HIN module. Then, it is divided into the first input features by averaging along the channel dimension. Second input features The first input feature F in1 After performing instance regularization, the first input regularized feature F is obtained. nor The second input feature F in2 After performing the flag mapping operation, the second input flag feature F is obtained. ide The first HIN module will use the first input regularization feature F nor Second input flag feature Fide After concatenation along the channel dimension, the aggregated feature F is obtained after passing through the second n4×n4 convolutional layer. agg Finally, the aggregated feature F agg With input features F in The first HIN feature F is obtained after addition. HIN1 ;

[0018] The first HIN feature F HIN1 Spatial attention feature F att The intermediate attention features are obtained by adding them together. The data is then input into the second and third HIN modules in sequence, and after undergoing the same processing, the second HIN feature F is output respectively. HIN2 and the third HIN feature F HIN3 ;

[0019] Second HIN feature F HIN2 After exponential operation, the features are combined with the second mode of the m-th stage. Multiply, then combine with the third HIN feature F HIN3 The intermediate stage binding characteristics are obtained after addition.

[0020] The fusion module will incorporate the intermediate attention features Features combined with intermediate stages After concatenation along the channel dimension, the final output image I is obtained by passing it through two n5×n5 convolutional layers. f ;

[0021] Step 3: Construct the overall loss function L total :

[0022] L total =αL content +βL texture (2)

[0023] In equation (2), L content and I texture Let represent content loss and texture loss respectively, and let α and β be the weights for balancing content loss and texture loss, obtained from equations (3) and (4):

[0024]

[0025]

[0026] In equations (3) and (4), Let λ1 and λ2 be the Laplacian operator, and let λ1 and λ2 be the weighted fusion images I. f The parameters representing the difference between the L2 norms of the input images I1 and I2, where μ1 and μ2 are the weighted fused images I fa parameter of Laplacian norm difference of input images I1, I2;

[0027] Step 4: input the medical image set into the image fusion network based on spatial attention and reversible neural network for training, and calculate the overall loss function L tptal When the number of training iterations reaches the set number of times, the training is stopped, thereby obtaining an optimal multi-modal medical image fusion network for realizing image fusion.

[0028] The multi-modal medical image fusion method based on spatial attention and reversible neural network has the characteristics that the step 2.2 is performed as follows:

[0029] Step 2.2.1: the residual connection module is composed of 4 convolution layers with a kernel of n2x n2, 2 BN layers, 2 ReLu layers, 1 average pooling layer and 1 Sigmoid layer;

[0030] The initial first modality feature is input into the first feature extraction unit, and after being processed by the first n2x n2 convolution layer, the first normalization layer BN and the first ReLu layer in the first residual connection module, the intermediate feature is obtained The intermediate feature is obtained after being processed by the second n2x n2 convolution layer and the second normalization layer BN, respectively The intermediate deep feature is obtained after being processed by the average pooling layer, the third n2x n2 convolution layer, the second ReLu layer, the fourth n2x n2 convolution layer and the Sigmoid layer, respectively, and the channel attention weight a1 is obtained, and the first stage first modality feature output from the first residual connection module is obtained by using formula (1)

[0031]

[0032] The initial second modality feature is input into the second residual connection module of the first feature extraction unit, and after being processed in the same way, the first stage second modality feature is output

[0033] Step 2.2.2: the texture separation module includes 2 n3x n3 convolution layers and a sigmoid layer;

[0034] The first stage first modality feature and the first stage second modality feature are input into the texture separation module, wherein the first stage first modality feature After processing through the first n3*n3 convolution layer and a sigmoid layer, a foreground high-frequency weight matrix is obtained Again, the difference between the foreground high-frequency weight matrix and the all "1" matrix is obtained

[0035] The first stage first modal feature The first stage second modal feature After connecting along the channel dimension, the foreground high-frequency weight matrix And the background low-frequency weight matrix are multiplied to obtain the foreground separation feature F fore And the background separation feature F back .

[0036] The foreground separation feature F fore And the background separation feature F back After connecting along the channel dimension, the first stage second modal feature is added to obtain the first stage second modal combination feature

[0037] The first stage first modal feature And the first stage second modal combination feature Input the second feature extraction unit and after the same processing, output the second stage first modal feature And the second stage second modal combination feature

[0038] The second stage first modal feature After sequentially passing through m feature extraction units, the first modal features of different stages are obtained Among them, The first modal feature of the mth stage is represented.

[0039] The second stage second modal combination feature After sequentially passing through m feature extraction units, the second modal combination features of different stages are obtained Among them, The second modal feature of the mth stage is represented.

[0040] An electronic device, comprising a memory and a processor, characterized in that the memory is used to store a program supporting the processor to execute the multi-modal medical image fusion method, and the processor is configured to execute the program stored in the memory.

[0041] ​The application is a computer readable storage medium, and a computer program is stored on the computer readable storage medium, wherein the computer program is executed by a processor to perform the steps of the multi-modal medical image fusion method.

[0042] Compared with the existing image fusion method, the advantages of the application are reflected in that:

[0043] 1. The application provides an end-to-end model for multi-modal MR images, an MSAIF-Net network, which uses a multi-level attention module to capture long-range context information in each image, assigns attention weights to different stages, and explores the structural details between foreground and background provided by each stage, thereby retaining comprehensive features of different modalities for subsequent fusion, combining the characteristics of images of different modalities to generate a clear MR structure image of a lesion, thereby assisting subsequent medical tasks.

[0044] 2. The application reversibly combines cross-scale features of two modalities through a reversible neural network to reconstruct a fusion image, which is helpful for subsequent semantic segmentation analysis and facilitates the exploration of the effect of different modalities on the segmentation task, thereby making up for the shortcomings of traditional end-to-end fusion network methods and improving the performance of image fusion, which is of great significance for the study of multi-modal medical images. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 A flowchart of the multi-modal medical image fusion method provided by the embodiments of the application is provided.

[0046] Figure 2 A principle framework diagram of the multi-modal medical image fusion method provided by the embodiments of the application is provided.

[0047] Figure 3 A structural schematic diagram of a residual connection module in the feature extraction module provided by the embodiments of the application is provided.

[0048] Figure 4 A structural schematic diagram of a texture separation module in the feature extraction module provided by the embodiments of the application is provided.

[0049] Figure 5 A multi-stage attention module provided by the embodiments of the application is provided.

[0050] Figure 6 A fusion module based on a reversible neural network provided by the embodiments of the application is provided.

[0051] Figure 7 An HIN module in the fusion module provided by the embodiments of the application is provided.

[0052] Figure 8A visual comparison effect diagram on the BraTS2019 dataset provided by the embodiment of the present application. DETAILED DESCRIPTION

[0053] In this embodiment, a multi-modal medical image fusion method based on spatial attention and reversible neural network is named MSAIF-Net, and the overall flow chart is as shown in Figure 1 On the one hand, a series of residual blocks are used to capture feature maps from two source images at different stages, and then the feature maps are connected by channels to obtain remote spatial information through a multi-stage attention module. On the other hand, a feature extraction module is proposed to separate the foreground and the background, and the details and textures extracted in one modality are fused with those in another modality, so that the comprehensive features of different modalities can be retained. The design motivation of the feature fusion module comes from the application of reversible neural network in computer vision, so it is tried to introduce it as a fusion strategy in an information lossless manner.

[0054] The embodiment of the present application explicitly uses a multi-stage attention module to explore the potential correlation between features at different stages, aiming to capture remote context information in each image, and uses a reversible convolutional network to fuse features. In addition, the method provided by the present application can realize end-to-end training and achieve the best performance requirement, and provides a new practical idea for multi-modal medical image fusion. In the embodiment, the MICCAI BraTS2019 multi-modal medical dataset is taken as an example, and the specific implementation is described in combination with the drawings. The method mainly includes:

[0055] Step 1: Preprocess two magnetic resonance images of different modalities respectively to obtain preprocessed medical images I1 and I2 of two modalities. Wherein, C, H, W respectively represent the channel number, height and width of the image; in this embodiment, the public dataset of MICCAI BraTS brain tumor segmentation challenge in 2019 is used, the data includes T1, T2, T1ce, T2Flair and their corresponding brain tumor segmentation images, for each data, the standard selection of the segmentation image higher than a certain pixel is adopted, including 3557 pictures, reading two different modal image pairs, performing cropping and normalization operations to obtain corresponding preprocessed data with a size of 160x160x1, wherein 160 corresponds to the image size, and 1 is the channel number;

[0056] Step 2: Construct an image fusion network based on spatial attention and reversible convolution, as shown in Figure 2 It includes: an initial mapping layer, a feature extraction module, a multi-stage attention module and a fusion module based on a reversible neural network;

[0057] Step 2.1: The initial mapping layer utilizes a convolutional layer with a convolution kernel of n1x n1 to process the medical images I1 and I2 of the two modalities to obtain the initial first modality feature and the initial second modality feature In this embodiment, n1 = 3, that is, the convolution kernel of each convolution in the initial mapping layer is 3x3, the convolution step is 1, the padding is zero, and the channel number of the output feature map is 32.

[0058] Step 2.2: The feature extraction module is composed of m feature extraction units, each of which includes 2 residual connection modules and 1 texture separation module; in this embodiment, m = 4, and the network structure is as shown in Figure 2 .

[0059] Step 2.2.1: The residual connection module is as shown in Figure 3 , which is composed of 4 convolutional layers with a convolution kernel of n2x n2, 2 BN layers, 2 ReLu layers, 1 average pooling layer and 1 Sigmoid layer;

[0060] The initial first modality feature is input into the first feature extraction unit and is processed by the first n2x n2 convolution layer, the first normalization layer BN and the first ReLu layer in the first residual connection module to obtain the intermediate feature . The intermediate feature is further processed by the second n2x n2 convolution layer and the second normalization layer BN to obtain the intermediate deep feature . The intermediate deep feature is sequentially processed by the average pooling layer, the third n2x n2 convolution layer, the second ReLu layer, the fourth n2x n2 convolution layer and the Sigmoid layer to obtain the channel attention weight a1, and finally the first stage first modality feature output from the first residual connection module is obtained by using formula (1) In this embodiment, n2 = 3, that is, the convolution kernel of each convolution is 3x3, the convolution step is 1, the padding is zero, and the channel number of the output feature map is 32.

[0061]

[0062] The initial second modality feature is input into the second residual connection module of the first feature extraction unit and is processed in the same way to output the first stage second modality feature

[0063] Step 2.2.2: The texture separation module is as shown in Figure 4 , which includes 2 n3x n3 convolution layers and a sigmoid layer.

[0064] first stage first modality feature and first stage second modality feature into the input texture separation module, wherein the first stage first modality feature After processing by the first n3x n3 convolution layer and a sigmoid layer, a foreground high-frequency weight matrix is obtained Then, the foreground high-frequency weight matrix is subtracted from the full "1" matrix to obtain a background low-frequency weight matrix

[0065] first stage first modality feature and first stage second modality feature After connecting along the channel dimension, the foreground high-frequency weight matrix and the background low-frequency weight matrix are multiplied, respectively, to obtain foreground separation features F fore and background separation features F back ;

[0066] foreground separation features F fore and background separation features F back After connecting along the channel dimension, the first stage second modality feature is added to obtain a first stage second modality combination feature

[0067] first stage first modality feature and first stage second modality combination feature are input into the second feature extraction unit and undergo the same processing, and the second stage first modality feature and the second stage second modality combination feature After m feature extraction units, the first modality features and the second modality combination features of different stages are obtained: wherein, represents the first modality feature of the mth stage; represents the second modality feature of the mth stage. In this embodiment, n3=3, that is, the convolution kernel of each layer of convolution is 3x3, the convolution step is 1, the padding is zero, and the channel number of the output feature map is 32.

[0068] Step 2.3: The multi-stage attention module is as shown in Figure 5 , which is composed of a Softmax layer and a series of reshaping operations and transpose operations;

[0069] After connecting the modality features of 2m different stages along the new dimension, a multi-stage total feature is obtained ​The multi-stage total feature F is input into the multi-stage attention module. The multi-stage total feature F is first expanded into a feature matrix through a reshaping operation. Feature matrix The transpose characteristic matrix is ​​obtained through the transpose operation. Then, the feature matrix is ​​multiplied by the transposed feature matrix, and the result is passed through a softmax layer to output the attention feature weight affinity matrix. Attention feature weight affinity matrix W and transpose feature matrix Multiplication yields the attention-based feature matrix F' att After the reshaping operation, the multi-stage attention feature is obtained by adding it to the total multi-stage feature F. Ultimately, the multi-stage attention features will be superimposed. Remodeling into spatial attention features As the output of the multi-stage attention module.

[0070] Step 2.4: The fusion module based on the reversible neural network, as shown below. Figure 6 As shown, it consists of three HIN modules and two n5×n5 convolutional layers.

[0071] Step 2.4.1: HIN module as follows Figure 7 As shown, it consists of two n4×n4 convolutional layers.

[0072] Integrate the second mode of the m-th stage with features Spatial attention features F att The input is a fusion module based on a reversible neural network, and the second modality combines features in the m-th stage. First, the input features are obtained by changing the channel dimension through the first n4×n4 convolutional layer of the first HIN module. Then, it is divided into the first input features by averaging along the channel dimension. Second input features First input feature F in1 Perform instance regularization to obtain the first input regularized feature F. nor The second input feature F in2 Perform a flag mapping operation to obtain the second input flag feature F ide The first input regularization feature F nor Second input flag feature F ide The aggregated feature F is obtained by concatenating along the channel dimension and passing it through the second n4×n4 convolutional layer. agg Aggregate feature F agg With input features F in Adding them together yields the first HIN feature F. HIN1In this embodiment, n4 = 3, meaning that the convolution kernels of both convolutional layers are 3×3, the convolution stride is 1, and zero padding is applied. After two outputs, the number of channels in the feature map is 16.

[0073] First HIN feature F HIN1 Spatial attention feature F att Adding them together yields the intermediate attention features. Intermediate attention features The input is fed into the second and third HIN modules, and after undergoing the same processing, the second HIN feature F is output respectively. HIN2 and the third HIN feature F HIN3 .

[0074] Second HIN feature F HIN2 After exponential operation, the features are combined with the second mode of the m-th stage. Multiply, then combine with the third HIN feature F HIN3 Adding them together yields the intermediate stage combination characteristics. Intermediate attention features Features combined with intermediate stages After concatenation along the channel dimension, the final output image I is obtained by passing it through two n5×n5 convolutional layers. f In this embodiment, n4 = 3, meaning that the convolution kernels of both convolutional layers are 3×3, the convolution stride is 1, and zero padding is used. After the first output, the feature map has 32 channels, and after the second output, it has 1 channel.

[0075] In this embodiment of the invention, the feature fusion stage is implemented using a reversible neural network (INN) structure. Based on the reversible neural network framework, it is implemented using the HINblock module structure in HINet, as shown in the figure. In recent years, as an effective scheme for reversible image transformation, the reversible neural network (INN) has attracted widespread attention and has been applied to various image tasks. It has three important properties: 1) the input-output mapping of an INN is bijective; 2) the forward and backward mappings of an INN are efficient and computable; 3) bijective mapping can easily handle the Jacobian determinant to explicitly compute the posterior probability. Due to these properties, INNs have achieved success in many tasks. Among them, pioneering research based on INNs can be found in the papers NICE and RealNVP. INN-based networks can combine input features in a bijective manner without losing information.

[0076] Step 3: Construct the overall loss function L total :

[0077] L total =αL content +βL texture (2)

[0078] In formula (2), L content and L texture respectively represent content loss and texture loss, and α and β are weights for balancing the content loss and the texture loss, in the embodiment, α = β = 1, and formula (3) and formula (4) are obtained:

[0079]

[0080]

[0081] In formula (3) and formula (4), is a Laplacian operator, λ1 and λ2 are parameters for balancing the fusion image I f and the two-norm difference of the input image I1 and I2, μ1 and μ2 are parameters for balancing the fusion image I f and the two-norm difference of the Laplacian operator of the input image I1 and I2. In the embodiment, μ1 = μ2 = λ1 = λ2 = 1.

[0082] Step 4: inputting the multi-modal medical image into the image fusion network based on the spatial attention and the reversible neural network to train and calculate the overall loss function L total When the number of training iterations reaches the set number of times, the training is stopped, and thus the optimal multi-modal medical image fusion network is obtained. In the embodiment, the SGD optimizer is adopted, the learning rate is decreased according to the increase of the number of iterations, and the training is iterated for 50 times. The effect comparison with different methods is shown in FIG. 5, from which it can be seen that the fusion method can better combine the clear structure inside the Flair image lesion and the overall brain structure of the T2 image compared with other methods, and thus is applied to subsequent medical related tasks. Figure 8

[0083] In order to quantitatively evaluate the effect of the application and verify the effectiveness of the application, the method of the application is compared with seven algorithms such as IFCNN. The correlation coefficient (correlation coefficient, CC) and the structure similarity (Structure Similarity Index Measure, SSIM) and other common indicators in the field of image fusion are selected as unsupervised evaluation standards. In addition, the obtained fusion image is input into the segmentation network model U-Net commonly used in the medical field to obtain a segmentation image, and the Dice similarity coefficient (Dice Similarity Coefficient, DSC), PPV and Hausdorff distance (HD, HD95, i.e. 95% HD) are selected as three performance indicators as supervised evaluation standards.

[0084] ​Seven comparison methods and the multi-modal medical image fusion method proposed in the present application are listed in this embodiment. The seven comparison methods are PMEF, SPD-MEF, MEFAW, U2Fusion, IFCNN, PGMI and MMFAL respectively. The corresponding paper titles of the seven comparison methods are Perceptual Quality Assessment for Multi-Exposure Image Fusion, Fast Multi-Scale Structural Patch Decomposition for Multi-Exposure Image Fusion, A Multi-exposure Image Fusion Based On The Adaptive Weights Reflecting The Relative Pixel Intensity And Global Gradient, U2Fusion: A Unified Unsupervised Image Fusion Network, Rethinking the Image Fusion: A Fast Unified Image Fusion Network based on Proportional Maintenance of Gradient and Intensity and Glioma Segmentation-Oriented Multi-modal MR Image Fusion with Adversarial Learning.

[0085] In this embodiment, the indicators for evaluating the fused image include the structure similarity-based indicators CC, SSIM, Q y and Q w ; the image feature-based indicators Q abf and Q p ; the human perception-based indicators Q cb and VIFF.

[0086] In this embodiment, an electronic device includes a memory for storing a program supporting a processor to execute the above multi-modal medical image fusion method, and the processor is configured to execute the program stored in the memory.

[0087] In this embodiment, a computer readable storage medium has a computer program stored thereon, and the computer program is run by a processor to execute the steps of the above multi-modal medical image fusion method.

Claims

1. A multi-modal medical image fusion method based on spatial attention and reversible neural network, characterized in that, is performed as follows: Step 1: preprocessing two different modal magnetic resonance images respectively to obtain preprocessed medical images I1 and I2 of two modalities wherein C, H, W represent the channel number, height and width of the image respectively. Step 2: constructing an image fusion network based on spatial attention and reversible convolution, including: an initial mapping layer, a feature extraction module, a multi-stage attention module, and a fusion module based on a reversible neural network; Step 2.1: The initial mapping layer processes the medical images I1 and I2 of the two modalities using a convolution kernel of a convolution layer with n1x n1 to obtain the first modality initial feature and the second modality initial feature Step 2.2: the feature extraction module is composed of m feature extraction units, each of which includes 2 residual connection modules and 1 texture separation module, for processing the first modality initial features and the second modality initial features to obtain second modality combined features at different stages; Step 2.3: the multi-stage attention module is composed of a Softmax layer and a series of reshaping operations and transpose operations; First modality feature And second modality combined feature After connecting along the new dimension, a multi-stage total feature is obtained The multi-stage attention module is inputted, and the multi-stage total feature F is first reshaped and unfolded into a feature matrix The feature matrix is then multiplied by the transpose feature matrix After the transpose operation, a transpose feature matrix is obtained Then the feature matrix is multiplied by the transpose feature matrix After the transpose operation, a transpose feature matrix is obtained After multiplication, a Softmax layer is passed through and an attention feature weight affinity matrix is outputted The multi-stage attention module multiplies the attention feature weight affinity matrix W with the transpose feature matrix to obtain an attention mechanism-based feature matrix F′ att After reshaping operation on F′ att , the superimposed multi-stage attention feature is obtained by adding the multi-stage total feature F to the superimposed multi-stage attention feature reshaped into a spatial attention feature F att ∈ and is taken as the output result of the multi-stage attention module; Step 2.4: the fusion module based on the reversible neural network is composed of 3 HIN modules and 2 convolution layers with a convolution kernel of n5x n5; each HIN module is composed of 2 convolution layers with a convolution kernel of n4x n4; Integrate the second mode of the m-th stage with features Spatial attention features F att The input is processed in the fusion module based on the reversible neural network, wherein the second modality of the m-th stage combines features. The input features are obtained by first changing the channel dimension through the first n4×n4 convolutional layer of the first HIN module. Then, it is divided into the first input features by averaging along the channel dimension. Second input features The first input feature F in1 After performing instance regularization, the first input regularized feature F is obtained. nor The second input feature F in2 After performing the flag mapping operation, the second input flag feature F is obtained. ide The first HIN module will use the first input regularization feature F nor Second input flag feature F ide After concatenation along the channel dimension, the aggregated feature F is obtained after passing through the second n4×n4 convolutional layer. agg Finally, the aggregated feature F agg With input features F in The first HIN feature F is obtained after addition. HIN1 ; The first HIN feature F HIN1 The spatial attention feature F att The intermediate attention feature F The second HIN feature F HIN2 and the third HIN feature F HIN3 ; Second HIN feature F HIN2 After exponentiation and combination with the mth stage second modality feature Multiplication, then combination with the third HIN feature F HIN3 After addition, the intermediate stage combined feature is obtained The fusion module fuses the intermediate attention features and the intermediate stage combination features After being connected along the channel dimension, the output image I is obtained by sequentially passing through two n5x n5convolution layers f ; Step 3: Utilizing the constructed overall loss function L total : L total = aL content + bL texture , (2) In formula (2), L content and L texture respectively represent the content loss and the texture loss, and a and β are weights for balancing the content loss and the texture loss, and are obtained by formula (3) and formula (4): in formula (3) and formula (4), is the Laplacian operator, λ1, λ2 are parameters to weigh the fused image I f is the parameter of the L2 norm difference of the input images I1, I2, μ1, μ2 are parameters to weigh the fused image I f is the parameter of the L2 norm difference of the Laplacian operator of the input images I1, I2; Step 4: input the medical image set into the image fusion network based on spatial attention and reversible neural network for training, and calculate the overall loss function L total When the number of training iterations reaches the set number of times, the training stops, thereby obtaining an optimal multi-modal medical image fusion network for realizing image fusion.

2. The multi-modal medical image fusion method based on spatial attention and reversible neural network according to claim 1, characterized in that, The step 2.2 is performed as follows: Step 2.2.1: the residual connection module is composed of 4 convolution layers with a convolution kernel of n2x n2, 2 BN layers, 2 ReLu layers, 1 average pooling layer and 1 Sigmoid layer; initial first modal feature The input is input into the first feature extraction unit and processed through the first n2×n2 convolution layer, the first normalization layer BN, and the first ReLu layer in the first residual connection module to obtain an intermediate feature The intermediate feature The intermediate feature is further processed through the second n2×n2 convolution layer and the second normalization layer BN to obtain an intermediate deep feature The intermediate deep feature The channel attention weight a1 is obtained by sequentially processing through the average pooling layer, the third n2×n2 convolution layer, the second ReLu layer, the fourth n2×n2 convolution layer, and the Sigmoid layer, and the first stage first modal feature output from the first residual connection module is obtained by using formula (1) initial second modality feature input into the second residual connection module of the first feature extraction unit and after the same processing, output the first stage second modality feature Step 2.2.2: the texture separation module includes 2 n3x n3 convolution layers and a sigmoid layer; The first-stage first-modal feature and a first-stage second-modal feature The texture separation module is inputted, wherein the first-stage first-modal feature After processing by a first n3x n3 convolutional layer and a sigmoid layer, a foreground high-frequency weight matrix is obtained Then, the foreground high-frequency weight matrix is subtracted from the full "1" matrix to obtain a background low-frequency weight matrix ​ The first stage first modality feature The first stage second modality feature After being connected along the channel dimension, respectively multiplied by the foreground high-frequency weight matrix And the background low-frequency weight matrix To obtain the foreground separation feature F fore And the background separation feature F back ; The foreground separation feature F fore and the background separation feature F back After concatenation along the channel dimension, the first stage second modality feature is added to obtain the first stage second modality combined feature the first stage first modality feature and the first stage second modality combined feature input the second feature extraction unit, and after the same processing, output the second stage first modality feature and the second stage second modality combined feature Second stage first modality feature After sequentially passing through m feature extraction units, different stage first modality features are obtained Wherein, The first modality feature of the mth stage is represented Second stage second modality combined feature After sequentially passing through m feature extraction units, different stage second modality combined features are obtained Wherein, The second modality feature of the mth stage is represented.

3. An electronic device comprising a memory and a processor, characterized in that The memory is used to store a program supporting the processor to execute the multi-modal medical image fusion method of claim 1 or 2, and the processor is configured to execute the program stored in the memory.

4. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to perform the steps of the multi-modal medical image fusion method of claim 1 or 2.

Images