A multi-modal medical image fusion method

Abstract

The application discloses a multi-modal medical image fusion method, which comprises the following steps: acquiring a first medical image and a second medical image to be fused, wherein the first medical image is an MRI image, and the second medical image is a CT image, a SPECT image or a PET image; determining a luminance image and a chroma image according to the second medical image; performing fusion processing on the first medical image and the luminance image by using a multi-modal medical image fusion model to obtain a preliminary fusion image; and determining a final fusion image according to the preliminary fusion image and the chroma image. The application can more reliably capture complex detail information and complementary information in different mode images, thereby effectively improving the fusion quality between different modal images.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a multi-modal medical image fusion method. BACKGROUND

[0002] The implementation purpose of the multi-modal medical image fusion technology is to fuse the complementary information and comprehensive information of multiple different modal images, which plays a crucial role in disease diagnosis. At present, researchers have introduced deep learning algorithms into multi-modal medical image fusion technology, mainly including convolutional neural network (CNN), generative adversarial network (GAN), autoencoder network (AE) and the like, but these networks usually use simple fusion rules such as element addition, element averaging and element multiplication to integrate features from different mode images, which often fails to capture complex detail information and complementary information in different mode images, and mainly focuses on spatial domain feature extraction, fails to fully utilize frequency domain features, and is prone to cause poor image fusion quality. SUMMARY

[0003] The present application provides a multi-modal medical image fusion method to solve one or more technical problems existing in the prior art, at least to provide a beneficial choice or create conditions.

[0004] The present application provides a multi-modal medical image fusion method, which comprises:

[0005] Obtaining a first medical image and a second medical image to be fused, the first medical image being an MRI image, and the second medical image being a CT image or a SPECT image or a PET image;

[0006] According to the second medical image, a luminance image and a chrominance image are determined;

[0007] A multi-modal medical image fusion model is used to fuse the first medical image and the luminance image to obtain a preliminary fusion image;

[0008] According to the preliminary fusion image and the chrominance image, a final fusion image is determined.

[0009] Further, the multi-modal medical image fusion model comprises an encoding network, a fusion network and a decoding network;

[0010] The encoding network is used to extract four first feature maps of different scales from the first medical image and four second feature maps of different scales from the luminance image;

[0011] The fusion network is used for cross fusion of the first feature map and the second feature map of the same scale to obtain an initial fusion feature map, and multi-scale fusion of four initial fusion feature maps of different scales to obtain corresponding four final fusion feature maps.

[0012] The decoding network is used for reconstruction of the four final fusion feature maps of different scales to obtain a preliminary fusion image.

[0013] Further, the encoding network includes four layers of encoders, and each adjacent two layers of encoders are connected through a maximum pooling layer.

[0014] Each layer of the encoder includes a dense convolution block and a first dynamic frequency transformer, and the first dynamic frequency transformer includes a frequency domain attention layer and a first dynamic frequency domain forward network layer.

[0015] When the first dynamic frequency transformer receives the initial feature map output by the dense convolution block, the frequency domain feature enhancement is performed on the initial feature map through the frequency domain attention layer to obtain a first intermediate feature map, the overall feature enhancement is performed on the first intermediate feature map through a residual connection mode to obtain a second intermediate feature map, the frequency domain feature extraction is performed on the second intermediate feature map through the first dynamic frequency domain forward network layer to obtain a third intermediate feature map, and the overall feature enhancement is performed on the third intermediate feature map through a residual connection mode to obtain a final feature map output.

[0016] Further, in the frequency domain attention layer, a first value vector, a first key vector and a first query vector are extracted from the initial feature map, the first key vector and the first query vector are multiplied in the frequency domain to obtain a first attention vector, and the first attention vector and the first value vector are multiplied in the spatial domain and then subjected to convolution operation to obtain the first intermediate feature map.

[0017] Further, in the first dynamic frequency domain forward network layer, the second intermediate feature map is subjected to convolution operation and split flattening to obtain a plurality of first feature vectors, the plurality of first feature vectors are multiplied with learnable weights in the frequency domain to obtain a plurality of second feature vectors, and the plurality of second feature vectors are subjected to image reorganization in the spatial domain and then subjected to residual connection to obtain a third intermediate feature map.

[0018] Further, the fusion network includes four layers of cross dynamic frequency fusioners and four layers of multi-scale fusioners, and each layer of cross dynamic frequency fusioner is connected with each layer of multi-scale fusioner.

[0019] Each layer of the cross dynamic frequency fusioner includes a feature enhancement network and a feature mining network, when the cross dynamic frequency fusioner receives the first feature map and the second feature map corresponding to the scale, the first feature map is channel split to obtain a first feature subgraph and a second feature subgraph, the second feature map is channel split to obtain a third feature subgraph and a fourth feature subgraph, the first feature subgraph and the third feature subgraph are subjected to specific feature enhancement and overall feature fusion by the feature enhancement network to obtain a first intermediate feature subgraph, the second feature subgraph and the fourth feature subgraph are subjected to common feature enhancement and overall feature fusion by the feature mining network to obtain a second intermediate feature subgraph, and the first intermediate feature subgraph and the second intermediate feature subgraph are fused to obtain an initial fusion feature map.

[0020] In each layer of the multi-scale fusioner, after the four initial fusion feature maps are spliced, a final fusion feature map is obtained through multiple convolution operations and single residual connection.

[0021] Further, the feature enhancement network includes a second dynamic frequency transformer and a third dynamic frequency transformer.

[0022] After the first feature subgraph and the third feature subgraph are subtracted, the third intermediate feature subgraph is obtained by processing via the second dynamic frequency transformer, and after the third feature subgraph and the first feature subgraph are subtracted, the fourth intermediate feature subgraph is obtained by processing via the third dynamic frequency transformer, and after the third intermediate feature subgraph and the fourth intermediate feature subgraph are subjected to overall feature enhancement by residual connection respectively, the first intermediate feature subgraph is obtained by splicing.

[0023] Further, the feature mining network includes a first cross dynamic frequency transformer and a second cross dynamic frequency transformer.

[0024] The first cross dynamic frequency transformer includes a first cross frequency domain attention layer and a second dynamic frequency domain forward network layer, based on the common features between the second feature subgraph and the fourth feature subgraph, the common features contained in the second feature subgraph are enhanced by the first cross frequency domain attention layer to obtain a fifth intermediate feature subgraph, after residual connection of the fifth intermediate feature subgraph, frequency domain feature extraction is performed by the second dynamic frequency domain forward network layer to obtain a sixth intermediate feature subgraph, and residual connection of the sixth intermediate feature subgraph obtains a seventh intermediate feature subgraph.

[0025] The second cross dynamic frequency transformer comprises a second cross frequency domain attention layer and a third dynamic frequency domain forward network layer, the common features contained in the fourth feature subgraph are enhanced through the second cross frequency domain attention layer to obtain an eighth intermediate feature subgraph, the eighth intermediate feature subgraph is connected in residual manner and then frequency domain feature extraction is performed through the third dynamic frequency domain forward network layer to obtain a ninth intermediate feature subgraph, and the ninth intermediate feature subgraph is connected in residual manner to obtain a tenth intermediate feature subgraph.

[0026] The seventh intermediate feature subgraph and the tenth intermediate feature subgraph are enhanced in overall feature through residual connection manner respectively, and then the second intermediate feature subgraph is obtained through splicing.

[0027] Further, in the first cross frequency domain attention layer, a second value vector and a second key vector are extracted from the second feature subgraph, a second query vector is extracted from the fourth feature subgraph, the second key vector and the second query vector are multiplied in the frequency domain to obtain a second attention vector, and the second attention vector and the second value vector are multiplied in the spatial domain to obtain a fifth intermediate feature subgraph through convolution operation.

[0028] In the second cross frequency domain attention layer, a third value vector and a third key vector are extracted from the fourth feature subgraph, a third query vector is extracted from the second feature subgraph, the third key vector and the third query vector are multiplied in the frequency domain to obtain a third attention vector, and the third attention vector and the third value vector are multiplied in the spatial domain to obtain an eighth intermediate feature subgraph through convolution operation.

[0029] Further, the decoding network comprises three layers of decoders formed by seven decoders in a connection mode of same layer dense jump and adjacent layer dense transmission, and each decoder comprises a 1x1 convolution layer, a fourth dynamic frequency transformer and a 3x3 convolution layer connected in sequence.

[0030] The present application has at least the following beneficial effects: the multi-modal medical image fusion model is built by introducing the deep learning algorithm and the frequency domain transformation algorithm, and the main performance is that the dynamic frequency transformer architecture is used in the model to fully extract and integrate the frequency domain features of different modal images, that is, the information features with stronger local changes in different modal images, and the cross dynamic frequency fusioner architecture is used in the model to capture the feature correlation between different modal images, so that the complex detail information and complementary information in different mode images can be more reliably captured, thereby effectively improving the fusion quality between different modal images. BRIEF DESCRIPTION OF DRAWINGS

[0031] The accompanying drawings are used to provide a further understanding of the technical solutions of the present application, and constitute a part of the specification, and are used together with the embodiments of the present application to explain the technical solutions of the present application, and do not constitute a limitation on the technical solutions of the present application.

[0032] Figure 1 is a flow diagram of a multi-modal medical image fusion method in an embodiment of the present application;

[0033] Figure 2 is a structural diagram of a multi-modal medical image fusion model in an embodiment of the present application;

[0034] Figure 3 is a structural diagram of a dynamic frequency converter in an embodiment of the present application;

[0035] Figure 4 is a structural diagram of a cross dynamic frequency fusion device in an embodiment of the present application;

[0036] Figure 5 is a structural diagram of a multi-scale fusion device in an embodiment of the present application;

[0037] Figure 6 is a structural diagram of a first cross dynamic frequency converter in an embodiment of the present application;

[0038] Figure 7 is a structural diagram of a second cross dynamic frequency converter in an embodiment of the present application;

[0039] Figure 8 is a diagram showing fusion effects of a plurality of fusion models in three different typical medical image fusion tasks in an embodiment of the present application;

[0040] Figure 9 is a hardware structural diagram of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and do not limit the present application.

[0042] It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in an order different from that in the flowchart. The terms "first", "second", etc. in the specification and claims and the above drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence, and it should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to be limiting of this application.

[0044] Moreover, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided for a thorough understanding of embodiments of the application. However, those of ordinary skill in the relevant art will recognize that the application can be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. In other instances, well-known structures, devices, implementations, or operations are not shown or described in detail in order to avoid obscuring aspects of the application.

[0045] The flowcharts shown in the drawings are only exemplary illustrations, not necessarily including all contents and operations / steps, nor necessarily executed in the order described. For example, some operations / steps can be further decomposed, and some operations / steps can be combined or partially combined, so the actual execution order can be changed according to actual conditions.

[0046] First, some of the terms involved in the present application are explained as follows:

[0047] Dense block (Dense Block) is a key component in DenseNet (DenseNet), which aims to improve the performance of the model and reduce the number of parameters by increasing the connection density of the network; the dense block is stacked by multiple layers, each layer contains 1x1 and 3x3 convolution operations, where 1x1 convolution operation is used for feature compression, and 3x3 convolution operation is used for feature extraction; this structure allows to increase the number of channels while keeping the input image size unchanged, enhancing the model's ability to capture local features.

[0048] Residual connection refers to introducing a direct connection between one or more layers of the network, so that the output of these layers includes not only the features after nonlinear transformation, but also the input features without processing, thereby allowing the network to learn the difference between the input and the output, rather than directly learning a complete mapping.

[0049] Layer normalization (LN, Layer Normalization) refers to normalizing each feature channel of each sample, so that the mean is close to 0 and the variance is close to 1, and it is not operated on the entire batch of samples, and is not affected by the batch size.

[0050] Max pooling refers to dividing the input image into several rectangular regions and selecting the pixel with the maximum value from each region as the output. This operation can reduce the image dimension and preserve the most prominent important features in the image.

[0051] Deep convolution refers to gradually extracting and combining the features of input data by stacking multiple convolution layers, thereby achieving accurate understanding and recognition of complex data.

[0052] Patch unfolding refers to dividing the input image into multiple fixed-size pixel blocks, then flattening (or straightening) each pixel block into a one-dimensional vector, and subsequently connecting multiple one-dimensional vectors corresponding to multiple pixel blocks into a one-dimensional sequence output.

[0053] Patch folding refers to dividing the one-dimensional sequence into multiple fixed-length subsequences, then reshaping (or folding) each subsequence into the original two-dimensional image block, and subsequently combining multiple two-dimensional images corresponding to multiple subsequences into a complete two-dimensional image.

[0054] GEGLU function is a variant of GLU (Gated Linear Unit) activation function, which combines the characteristics of GELU (Gaussian Error Linear Unit) activation function and GLU activation function to enhance the model's ability to process nonlinear features.

[0055] Bottleneck layer (Bottleneck Conv) usually includes 1x1 convolution layer, 3x3 convolution layer and 1x1 convolution layer connected in sequence. The first 1x1 convolution layer is used to reduce the channel number (i.e. dimension) of the input data to reduce the computational complexity of the 3x3 convolution layer, and the second 1x1 convolution layer is used to restore the data processed by the 3x3 convolution layer to the original channel number.

[0056] Please refer to Figure 1 , Figure 1 is a flowchart of a multi-modal medical image fusion method provided by an embodiment of the present application. The method comprises the following steps:

[0057] Step S110, acquiring a first medical image and a second medical image to be fused;

[0058] The first medical image is an MRI (Magnetic Resonance Imaging) image, and the second medical image is a CT (Computed Tomography) image or a SPECT (Single Photon Emission Computed Tomography) image or a PET (Positron Emission Tomography) image; the MRI image and the CT image are both gray-scale images, and the SPECT image and the PET image are both color images; the first medical image and the second medical image should be detection images of the same part of the same user, and are preferably human brain images.

[0059] In step S120, a luminance image and a chrominance image are determined according to the second medical image.

[0060] In step S130, a multimodal medical image fusion model is used to perform fusion processing on the first medical image and the luminance image to obtain a preliminary fusion image.

[0061] In step S140, a final fusion image is determined according to the preliminary fusion image and the chrominance image.

[0062] In some embodiments of the present application, the above step S120 is described in the following two cases:

[0063] In the first case, when the second medical image is a PET image or a SPECT image, the second medical image is converted from an RGB color space to a YCrCb color space for representation to obtain a to-be-detected medical image; the to-be-detected medical image is subjected to channel decomposition to obtain a Y component image (i.e., an image carrying a luminance component), a Cr component image (i.e., an image carrying a red color concentration offset component), and a Cb component image (i.e., an image carrying a blue color concentration offset component); the Y component image is directly defined as the luminance image, and the Cr component image and the Cb component image are uniformly defined as the chrominance image.

[0064] In the second case, when the second medical image is a CT image, the second medical image is directly defined as the luminance image, and there is no chrominance image.

[0065] In some embodiments of the present application, the above step S140 is described in the following two cases:

[0066] In a first case, when the chroma image exists, the preliminary fusion image, the Cr component image and the Cb component image are superimposed in channels to obtain an intermediate fusion image; the intermediate fusion image is converted from the YCrCb color space to the RGB color space for representation to obtain the final fusion image;

[0067] In a second case, when the chroma image does not exist, the preliminary fusion image is directly defined as the final fusion image.

[0068] In some embodiments of the present application, the multi-modal medical image fusion model mentioned in the above step S130 comprises an encoding network, a fusion network and a decoding network connected in sequence, as shown in Figure 2 In actual application, the encoding network is used to extract four first feature maps at different scales from the first medical image and four second feature maps at corresponding different scales from the luminance image; the fusion network is used to cross-fuse the first feature map and the second feature map at the same scale to obtain an initial fusion feature map, and then perform multi-scale fusion processing on the four initial fusion feature maps at different scales to obtain corresponding four final fusion feature maps; and the decoding network is used to reconstruct the four final fusion feature maps at different scales to obtain the preliminary fusion image.

[0069] More specifically, as shown in Figure 2 and Figure 3 The encoding network comprises four layers of encoders arranged at different scales, and each two layers of encoders having an adjacent relationship are connected through a maximum pooling layer; wherein each layer of encoder comprises a first dynamic frequency transformer and a dense convolution block, and the first dynamic frequency transformer comprises a first dynamic frequency domain forward network layer and a frequency domain attention layer.

[0070] In actual application, when the layer of encoder receives an input image, the input image is subjected to feature enhancement processing through the dense convolution block to obtain an initial feature map; after the first dynamic frequency transformer receives the initial feature map, the initial feature map is subjected to frequency domain feature enhancement processing through the frequency domain attention layer to obtain a first intermediate feature map; the first intermediate feature map is subjected to overall feature enhancement processing through a residual connection mode to obtain a second intermediate feature map, that is, the first intermediate feature map and the initial feature map are elementally added to obtain the second intermediate feature map; the second intermediate feature map is subjected to frequency domain feature extraction through the first dynamic frequency domain forward network layer to obtain a third intermediate feature map; and the third intermediate feature map is subjected to overall feature enhancement processing through a residual connection mode to obtain a final feature map output, that is, the third intermediate feature map and the second intermediate feature map are elementally added to obtain the final feature map output.

[0071] It should be noted that the initial feature map should be subjected to layer normalization before being input into the frequency domain attention layer, and the second intermediate feature map should also be subjected to layer normalization before being input into the first dynamic frequency domain forward network layer.

[0072] Further, referring to Figure 3 , the frequency domain attention layer includes a query attention head, a key attention head, a value attention head, and a 1x1 convolution layer. The three attention heads are in parallel and run independently to capture different attention information in different feature representation subspaces. Each attention head includes a 1x1 convolution layer and a 3x3 deep convolution layer connected in sequence.

[0073] In actual application, the initial feature map subjected to layer normalization is processed through the three attention heads to extract the corresponding first query vector Q1, first key vector K1, and first value vector V1. The first query vector Q1 and the first key vector K1 are multiplied element-wise in the frequency domain to obtain a first attention vector, that is, the first query vector Q1 and the first key vector K1 are both converted from the spatial domain to the frequency domain representation by using fast Fourier transform (FFT), and then the Hadamard product is performed to obtain the first attention vector. The first value vector V1 and the first attention vector are multiplied element-wise in the spatial domain, and then the convolution operation is performed to obtain a first intermediate feature map, that is, the first attention vector is converted from the frequency domain to the spatial domain representation by using inverse fast Fourier transform (IFFT), and then the standardization processing is performed. The normalized result and the first value vector V1 are subjected to the Hadamard product, and then the 1x1 convolution layer is processed to obtain the first intermediate feature map.

[0074] Further, referring to Figure 3 , the first dynamic frequency domain forward network layer includes a 1x1 convolution layer, a flattening layer, a folding layer, and a GEGLU function layer.

[0075] In practical applications, the second intermediate feature map after layer normalization is sequentially subjected to convolution operation and segmentation flattening processing to obtain a plurality of first feature vectors, that is, the second intermediate feature map after layer normalization is subjected to standardization processing, and then sequentially subjected to 1x1 convolution layer and flattening layer processing to obtain the plurality of first feature vectors, the learnable weight W and the plurality of first feature vectors are multiplied in the frequency domain to obtain a plurality of corresponding second feature vectors, that is, after the plurality of first feature vectors are converted from the spatial domain to the frequency domain by fast Fourier transform, the plurality of corresponding second feature vectors are obtained by respectively performing Hadamard product with the learnable weight W, and after the plurality of second feature vectors are subjected to image reorganization processing in the spatial domain, a third intermediate feature map is obtained through residual connection, that is, after the plurality of second feature vectors are converted from the frequency domain to the spatial domain by inverse fast Fourier transform, a reorganized feature image is obtained by sequentially passing through a folding layer and a GEGLU function layer, and then the reorganized feature image and the second intermediate feature map after layer normalization are added element by element to obtain the third intermediate feature map.

[0076] More specifically, referring to Figure 2 , the fusion network includes four layers of multi-scale fusioners and four layers of cross dynamic frequency fusioners arranged correspondingly at different scales, and each layer of cross dynamic frequency fusioner and each layer of multi-scale fusioner are connected;

[0077] Referring to Figure 4 , each layer of cross dynamic frequency fusioner includes a feature mining network, a feature enhancement network and a bottleneck layer; in practical applications, when the layer of cross dynamic frequency fusioner receives the first feature map and the second feature map at the corresponding same scale, the first feature map is subjected to channel segmentation to obtain a first feature submap and a second feature submap The second feature map is subjected to channel segmentation to obtain a third feature submap and a fourth feature submap The first feature submap and the third feature submap are sequentially subjected to specific feature enhancement processing and overall feature fusion processing by the feature enhancement network to obtain a first intermediate feature submap, and the second feature submap and the fourth feature submap The common feature enhancement processing and the overall feature fusion processing are sequentially performed to obtain a second intermediate feature subgraph, and the second intermediate feature subgraph and the first intermediate feature subgraph are fused to obtain an initial fusion feature map, that is, the second intermediate feature subgraph and the first intermediate feature subgraph are spliced and then processed by the bottleneck layer to obtain the initial fusion feature map;

[0078] Referring to Figure 5 As shown in the figure, each layer of the multi-scale fusioner includes a 1x1 convolution layer and two 3x3 convolution layers; in actual application, when the layer of the multi-scale fusioner receives the four initial fusion feature maps of different scales, the four initial fusion feature maps are spliced and then sequentially processed by multiple convolution operations and single residual connection to obtain a final fusion feature map of the scale associated with the layer of the multi-scale fusioner, that is, the four initial fusion feature maps are spliced and then processed by the 1x1 convolution layer to obtain a first intermediate fusion feature map, the first intermediate fusion feature map is sequentially processed by the two 3x3 convolution layers to obtain a second intermediate fusion feature map, and the first intermediate fusion feature map and the second intermediate fusion feature map are element-wise added to obtain the final fusion feature map.

[0079] Further, referring to Figure 4 As shown in the figure, the feature enhancement network includes a second dynamic frequency transformer and a third dynamic frequency transformer; in actual application, the first feature subgraph and the third feature subgraph are element-wise subtracted and then processed by the second dynamic frequency transformer to obtain a third intermediate feature subgraph, the third feature subgraph and the first feature subgraph are element-wise subtracted and then processed by the third dynamic frequency transformer to obtain a fourth intermediate feature subgraph, the fourth intermediate feature subgraph and the third intermediate feature subgraph are overall feature enhanced by residual connection, and then spliced to obtain a first intermediate feature subgraph, that is, the third intermediate feature subgraph and the first feature subgraph are element-wise added to obtain an enhanced third intermediate feature subgraph The fourth intermediate feature subgraph and the third feature subgraph are element-wise added to obtain an enhanced fourth intermediate feature subgraph The enhanced third intermediate feature subgraph and the enhanced fourth intermediate feature subgraph are spliced to obtain the first intermediate feature subgraph.

[0080] It should be noted that the data processing process possessed in the second dynamic frequency transformer and the data processing process possessed in the third dynamic frequency transformer are the same as the data processing process possessed in the first dynamic frequency transformer, and will not be described here.

[0081] Further, referring to Figure 4 , the feature mining network includes a first cross dynamic frequency transformer and a second cross dynamic frequency transformer.

[0082] Referring to Figure 6 , the first cross dynamic frequency transformer includes a second dynamic frequency domain forward network layer and a first cross frequency domain attention layer; in actual application, since there are common features between the second feature subgraph and the fourth feature subgraph , the common features contained in the second feature subgraph are enhanced by the first cross frequency domain attention layer to obtain a fifth intermediate feature subgraph, and the fifth intermediate feature subgraph is overall feature enhanced by a residual connection manner to obtain an enhanced fifth intermediate feature subgraph, that is, the fifth intermediate feature subgraph and the second feature subgraph are element-added to obtain the enhanced fifth intermediate feature subgraph, and the enhanced fifth intermediate feature subgraph is frequency domain feature extracted by the second dynamic frequency domain forward network layer to obtain a sixth intermediate feature subgraph, and the sixth intermediate feature subgraph is overall feature enhanced by a residual connection manner to obtain a seventh intermediate feature subgraph, that is, the sixth intermediate feature subgraph and the enhanced fifth intermediate feature subgraph are element-added to obtain the seventh intermediate feature subgraph.

[0083] Referring to Figure 7 , the second cross dynamic frequency transformer includes a third dynamic frequency domain forward network layer and a second cross frequency domain attention layer; in actual application, the common features contained in the fourth feature subgraph are enhanced by the second cross frequency domain attention layer to obtain an eighth intermediate feature subgraph, and the eighth intermediate feature subgraph is overall feature enhanced by a residual connection manner to obtain an enhanced eighth intermediate feature subgraph, that is, the eighth intermediate feature subgraph and the fourth feature subgraph The enhanced eighth intermediate feature subgraph is subjected to frequency domain feature extraction via the third dynamic frequency domain forward network layer to obtain a ninth intermediate feature subgraph, and the ninth intermediate feature subgraph is subjected to overall feature enhancement processing in a residual connection manner to obtain a tenth intermediate feature subgraph, that is, the ninth intermediate feature subgraph and the enhanced eighth intermediate feature subgraph are subjected to element addition to obtain the tenth intermediate feature subgraph;

[0084] Referring to Figure 4 illustrated, the tenth intermediate feature subgraph and the seventh intermediate feature subgraph are subjected to overall feature enhancement processing in a residual connection manner respectively, and then a second intermediate feature subgraph is obtained by splicing, that is, the seventh intermediate feature subgraph and the second feature subgraph are subjected to element addition to obtain an enhanced seventh intermediate feature subgraph The tenth intermediate feature subgraph and the fourth feature subgraph are subjected to element addition to obtain an enhanced tenth intermediate feature subgraph The enhanced seventh intermediate feature subgraph and the enhanced tenth intermediate feature subgraph are spliced to obtain the second intermediate feature subgraph.

[0085] It should be noted that the enhanced fifth intermediate feature subgraph should be subjected to layer normalization processing before being input to the second dynamic frequency domain forward network layer, and the enhanced eighth intermediate feature subgraph should also be subjected to layer normalization processing before being input to the third dynamic frequency domain forward network layer.

[0086] It should be noted that the data processing process possessed by the second dynamic frequency domain forward network layer and the data processing process possessed by the third dynamic frequency domain forward network layer are the same as the data processing process possessed by the first dynamic frequency domain forward network layer, and will not be described here.

[0087] Further, referring to Figure 6 illustrated, the first cross-frequency attention layer also includes a query attention head, a key attention head, a value attention head, and a 1x1 convolution layer. The three attention heads adopt a parallel independent running mode to capture different attention information in different feature representation subspaces, wherein each attention head includes a 1x1 convolution layer and a 3x3 deep convolution layer connected in sequence.

[0088] In actual application, the second feature subgraph and the fourth feature subgraph are subjected to layer normalization processing and then input to the first cross-frequency attention layer, and the second feature subgraph The second key vector K2 and the second value vector V2 are extracted from the fourth feature subgraph after layer normalization via the key attention head and the value attention head. The second query vector Q2 is extracted from the fourth feature subgraph via the query attention head. The second query vector Q2 and the second key vector K2 are element-wise multiplied in the frequency domain to obtain a second attention vector. The second value vector V2 and the second attention vector are element-wise multiplied in the spatial domain to obtain a fifth intermediate feature subgraph via convolution operation. The second attention vector is converted from the spatial domain to the frequency domain by using the inverse fast Fourier transform, and then normalized. The normalized result and the second value vector V2 are multiplied by Hadamard product, and then processed by the 1x1 convolution layer to obtain the fifth intermediate feature subgraph.

[0089] Further, referring to Figure 7 As shown, the second cross-frequency domain attention layer also includes a query attention head, a key attention head, a value attention head, and a 1x1 convolution layer. The three attention heads are independently operated in parallel to capture different attention information in different feature representation subspaces. Each attention head includes a 1x1 convolution layer and a 3x3 deep convolution layer connected in sequence.

[0090] In actual application, the second feature subgraph and the fourth feature subgraph are input to the second cross-frequency domain attention layer after layer normalization. The fourth feature subgraph after layer normalization is processed via the key attention head and the value attention head to extract a corresponding third key vector K3 and a third value vector V3. The second feature subgraph after layer normalization is processed via the query attention head to extract a third query vector Q3. The third query vector Q3 and the third key vector K3 are element-wise multiplied in the frequency domain to obtain a third attention vector. The third value vector V3 and the third attention vector are element-wise multiplied in the spatial domain to obtain an eighth intermediate feature subgraph via convolution operation. The third attention vector is converted from the frequency domain to the spatial domain by using the inverse fast Fourier transform, and then normalized. The normalized result and the third value vector V3 are multiplied by Hadamard product, and then processed by the 1x1 convolution layer to obtain the eighth intermediate feature subgraph.

[0091] More specifically, referring to Figure 2 As shown, the decoding network comprises three layers of decoders arranged at different scales, the first layer of decoders arranges four decoders, the second layer of decoders arranges two decoders, and the third layer of decoders arranges one decoder, wherein each decoder comprises a 1x1 convolution layer, a fourth dynamic frequency transformer and a 3x3 convolution layer connected in sequence, the fourth dynamic frequency transformer has the same data processing process as the first dynamic frequency transformer, which will not be repeated here.

[0092] In the decoding network, the above seven decoders are connected in the manner of adjacent layer dense transmission and same layer dense jump, which is specifically shown as follows:

[0093] In the third layer of decoders, the input end of the only decoder (denoted as the first decoder) is connected with the output end of the third layer of multi-scale fusioners and the output end of the fourth layer of multi-scale fusioners respectively;

[0094] In the second layer of decoders, according to the direction of data flow, the input end of the first decoder (denoted as the second decoder) is connected with the output end of the second layer of multi-scale fusioners and the output end of the third layer of multi-scale fusioners respectively, and the input end of the second decoder (denoted as the third decoder) is connected with the output end of the first decoder, the output end of the second decoder and the output end of the second layer of multi-scale fusioners respectively;

[0095] In the first layer of decoders, according to the direction of data flow, the input end of the first decoder (denoted as the fourth decoder) is connected with the output end of the first layer of multi-scale fusioners and the output end of the second layer of multi-scale fusioners respectively, the input end of the second decoder (denoted as the fifth decoder) is connected with the output end of the second decoder, the output end of the fourth decoder and the output end of the first layer of multi-scale fusioners respectively, the input end of the third decoder (denoted as the sixth decoder) is connected with the output end of the third decoder, the output end of the fourth decoder, the output end of the fifth decoder and the output end of the first layer of multi-scale fusioners respectively, and the input end of the fourth decoder (denoted as the seventh decoder) is connected with the output end of the sixth decoder, and the output end of the seventh decoder is the output end of the multi-modal medical image fusion model.

[0096] In some embodiments of the present application, the multi-modal medical image fusion model is obtained by training according to the following total loss function:

[0097] L total =L int +α1L ssim +α2Lsd + alpha3L grad ,

[0098]

[0099]

[0100]

[0101]

[0102] wherein, L total denotes the total loss, L int denotes the intensity loss function, L ssim denotes the structural similarity loss function, L sd denotes the similarity decomposition loss function, L grad denotes the texture loss function, alpha1, alpha2 and alpha3 are all super parameters for balancing the loss, and are preferably set as alpha1=1, alpha2=1 and alpha3=10 after a large number of experiments, M is the number of network layers and is set as 4 in the present application, H is the length of an image, W is the width of an image, and the length and width of each feature map involved in the whole model processing process are corresponding, is the feature map output by the mth cross dynamic frequency fusioner, is the feature map output by the mth encoder for extracting the first medical image, is the feature map output by the mth encoder for extracting the brightness image, ‖·‖1 denotes the L1 norm (Manhattan norm), I f is the feature map finally output by the decoding network, which can also be understood as the feature map finally output by the multi-modal medical image fusion model, I1 is the first medical image, I2 is the brightness image, SSIM denotes the structural similarity index measure operator, w m is the weighting parameter vector for balancing the loss amplitude, and is set as {1, 10, 100, 1000} in the present application, is the enhancement result of the feature map output by the second dynamic frequency transformer, is the enhancement result of the feature map output by the third dynamic frequency transformer, is the enhancement result of the feature map output by the first cross dynamic frequency transformer, is the enhancement result of the feature map output by the second cross dynamic frequency transformer, and epsilon is a constant and is set as 1.01 in the present application, denotes the Hamiltonian operator.

[0103] To verify the effectiveness of the multimodal medical image fusion model (denoted as SDFormer model) proposed in this invention, it was compared and validated with four other different fusion models: FusionGAN (Fusion Generative Adversarial Network), U2Fusion (Unified Unsupervised Image Fusion Network), EMFusion (Enhanced Medical Image Fusion Network), and SwinFusion (Swin Transformer-based Image Fusion Network). Three sets of experimental operations and their corresponding fusion effect evaluations were conducted, as detailed below:

[0104] The first set of experiments involved acquiring CT and MRI images from a specific brain scan of a user. Five different fusion models were then used to fuse these CT and MRI images, respectively. Figure 8 As shown; further utilizing existing MS-SSIM (Multi Scale Structural Similarity Index Measure), FMI (Feature Mutual Information), and Q... AB / F The four image quality evaluation metrics, namely (a metric used to measure the amount of edge information transferred from the original image to the fused image) and VIF (Visual Information Fidelity), are used to evaluate the fusion effect presented by these five fusion models, as shown in Table 1.

[0105] Table 1 Evaluation of fusion effect in the first group of experiments (CT-MRI fusion)

[0106]

[0107] The second group of experiments involved acquiring PET and MRI images from a specific brain scan of a user. Five different fusion models were then used to fuse these PET and MRI images, respectively. Figure 8 As shown; then utilizing existing MS-SSIM, FMI, Q AB / F The fusion effects of the five fusion models were evaluated using four image quality evaluation metrics, namely, VIF, etc., as shown in Table 2.

[0108] Table 2 Evaluation of fusion effect in the second group of experiments (PET-MRI fusion)

[0109]

[0110]

[0111] The third group of experiments involved acquiring SPECT and MRI images obtained from a specific brain scan of a user. Five different fusion models were then used to fuse these SPECT and MRI images, respectively. Figure 8 As shown; then utilizing existing MS-SSIM, FMI, Q AB / F The fusion effects of the five fusion models are evaluated using four image quality evaluation metrics, namely, VIF, etc., as shown in Table 3.

[0112] Table 3. Evaluation of fusion effect in the third group of experiments (SPECT-MRI fusion)

[0113]

[0114] As shown in Tables 1 to 3, compared with the other four fusion models, the SDFormer model proposed in this invention exhibits superior fusion performance in these three different typical medical image fusion tasks, due to its Q... AB / F The calculated results for the image quality evaluation metrics Q and VIF are better. AB / F A larger VIF value indicates that the final fused image can capture more edge information more effectively. A larger VIF value indicates that the final fused image can exhibit better visual quality and reduce image distortion to a greater extent. Furthermore, its calculation results on the two image quality evaluation metrics, MS-SSIM and FMI, are also significant, resulting in the final fused image having high structural similarity and containing rich feature information.

[0115] In addition, three ablation experiments were conducted on the SDFormer model proposed in this invention. The first ablation experiment replaced all dynamic frequency converters in the SDFormer model with existing converters. The second ablation experiment replaced the fusion network in the SDFormer model with an image fusion method that directly adds two images at the same scale element by element. The third ablation experiment removed only the similarity decomposition loss function L, which was included in the total loss function used by the SDFormer model during training. sd This weakens the model's ability to learn the fusion of complementary and common features;

[0116] Based on the PET image and the MRI image obtained in the second set of experiments, the three fusion models trained based on the SDFormer model for the three ablation experiments are used to fuse the PET image and the MRI image, and the existing MS-SSIM, FMI, Q AB / F and VIF image quality evaluation indexes are used to evaluate the fusion effect of the three fusion models, as shown in Table 4.

[0117] Table 4 Fusion effect evaluation of three ablation experiments (PET-MRI fusion)

[0118]

[0119] According to Table 4, compared with the fusion models used in the three ablation experiments, the SDFormer model proposed in the present application has better fusion effect in the PET-MRI image fusion task, because the calculation results of the four image quality evaluation indexes MS-SSIM, FMI, Q AB / F and VIF are all optimal.

[0120] In the embodiment of the present application, the deep learning algorithm and the frequency domain transformation algorithm are introduced to build a multi-modal medical image fusion model, which mainly uses a dynamic frequency transformer architecture in the model to fully extract and integrate the frequency domain features of different modal images, that is, the information features with stronger local changes in different modal images, and uses a cross dynamic frequency fusioner architecture in the model to capture the feature correlation between different modal images, so that the complex detail information and complementary information in different modal images can be more reliably captured, thereby effectively improving the fusion quality between different modal images.

[0121] In addition, the embodiment of the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the multi-modal medical image fusion method in the above embodiment. The computer readable storage medium includes, but is not limited to, any type of disk (including a floppy disk, a hard disk, an optical disk, a CD-ROM, and a magneto-optical disk), a ROM (Read-Only Memory), a RAM (Random Access Memory), an EPROM (Erasable Programmable Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a flash memory, a magnetic card or an optical card. That is, the storage device includes any medium that stores or transmits information in a readable form by a device (for example, a computer, a mobile phone, etc.), and can be a read-only memory, a magnetic disk or an optical disk, etc.

[0122] In addition, Figure 9 is a hardware structure schematic diagram of a computer device provided by the embodiment of the present application, and the computer device includes a processor 220, a memory 230, an input unit 240, a display unit 250 and the like. Those skilled in the art can understand that, Figure 9 The device structure shown does not constitute a limitation on all devices, and can include more or fewer components than shown, or combine some components. The memory 230 can be used to store the computer program 210 and various functional modules, and the processor 220 runs the computer program 210 stored in the memory 230, so as to execute various functional applications and data processing of the device. The memory can be an internal memory or an external memory, or include an internal memory and an external memory. The internal memory can include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory or a random memory. The external memory can include a hard disk, a floppy disk, a U disk, a magnetic tape, etc. The memory 230 disclosed in the embodiment of the present application includes, but is not limited to, the above-mentioned types of memory. The memory 230 disclosed in the embodiment of the present application is only an example, not a limitation.

[0123] The input unit 240 is configured to receive input of signals and receive a keyword input by a user. The input unit 240 can include a touch panel and other input devices. The touch panel can collect a touch operation of a user thereon or adjacent thereto (such as an operation of the user on or adjacent to the touch panel by using a finger, a stylus or any suitable object or accessory) and drive a corresponding connection device according to a pre-set program; the other input devices can include, but are not limited to, one or more of a physical keyboard, function keys (such as play control buttons, switch buttons, etc.), a trackball, a mouse, a joystick, etc. The display unit 250 can be configured to display information input by a user or information provided to the user and various menus of the terminal device. The display unit 250 can take the form of a liquid crystal display, an organic light-emitting diode, etc. The processor 220 is a control center of the terminal device, connects all parts of the device through various interfaces and lines, performs various functions and processes data by running or executing software programs and / or modules stored in the memory 230 and calling data stored in the memory 230.

[0124] As an embodiment, the computer device comprises a processor 220, a memory 230 and a computer program 210, wherein the computer program 210 is stored in the memory 230 and configured to be executed by the processor 220, and the computer program 210 is configured to perform a multi-modal medical image fusion method in one of the above embodiments.

[0125] Although the description of the present application has been quite detailed and particularly described with respect to several embodiments, it is not intended to be limited to any of these details or embodiments or any special embodiment, but should be considered to effectively cover the intended scope of the present application by referring to the appended claims, taking into account the broadest possible interpretation of these claims in view of the prior art. In addition, the present application is described above in embodiments that the inventors can foresee, and the purpose is to provide a useful description, and non-essential modifications to the present application that have not yet been foreseen can still represent equivalent modifications of the present application.

Claims

1. A multimodal medical image fusion method, characterized in that, The method includes: Acquire a first medical image and a second medical image to be fused, wherein the first medical image is an MRI image and the second medical image is a CT image, a SPECT image, or a PET image; Based on the second medical image, determine the luminance image and chrominance image; The first medical image and the brightness image are fused using a multimodal medical image fusion model to obtain a preliminary fused image; Based on the preliminary fused image and the chroma image, the final fused image is determined; The multimodal medical image fusion model includes an encoding network, a fusion network, and a decoding network. The coding network is used to extract four first feature maps of different scales from the first medical image and four second feature maps of different scales from the brightness image. The fusion network is used to cross-fuse the first feature map and the second feature map at the same scale to obtain an initial fused feature map, and then to perform multi-scale fusion on the four initial fused feature maps at different scales to obtain the corresponding four final fused feature maps. The decoding network is used to reconstruct four final fusion feature maps at different scales to obtain a preliminary fusion image; The fusion network includes a four-layer cross-dynamic frequency fusion unit and a four-layer multi-scale fusion unit, with each layer of cross-dynamic frequency fusion unit connected to each layer of multi-scale fusion unit. Each layer of the cross-dynamic frequency fusion network includes a feature enhancement network and a feature mining network. When the cross-dynamic frequency fusion network receives a first feature map and a second feature map of the corresponding scale, it performs channel segmentation on the first feature map to obtain a first feature sub-map and a second feature sub-map, and performs channel segmentation on the second feature map to obtain a third feature sub-map and a fourth feature sub-map. The feature enhancement network performs specific feature enhancement and overall feature fusion on the first feature sub-map and the third feature sub-map to obtain a first intermediate feature sub-map. The feature mining network performs common feature enhancement and overall feature fusion on the second feature sub-map and the fourth feature sub-map to obtain a second intermediate feature sub-map. The first intermediate feature sub-map and the second intermediate feature sub-map are then fused to obtain an initial fused feature map. In each multi-scale fusion layer, the four initial fusion feature maps are concatenated and then subjected to multiple convolution operations and single residual connections to obtain the final fusion feature map.

2. The multimodal medical image fusion method according to claim 1, characterized in that, The encoding network includes four encoder layers, with each adjacent encoder layer connected through a max pooling layer. Each encoder layer includes dense convolutional blocks and a first dynamic frequency converter, which includes a frequency domain attention layer and a first dynamic frequency domain feedforward network layer. When the first dynamic frequency converter receives the initial feature map output by the dense convolutional block, it performs frequency domain feature enhancement on the initial feature map through the frequency domain attention layer to obtain a first intermediate feature map, performs overall feature enhancement on the first intermediate feature map through residual connection to obtain a second intermediate feature map, performs frequency domain feature extraction on the second intermediate feature map through the first dynamic frequency domain feedforward network layer to obtain a third intermediate feature map, and performs overall feature enhancement on the third intermediate feature map through residual connection to obtain the final feature map output.

3. The multimodal medical image fusion method according to claim 2, characterized in that, In the frequency domain attention layer, a first value vector, a first key vector, and a first query vector are extracted from the initial feature map. The first key vector and the first query vector are multiplied in the frequency domain to obtain a first attention vector. The first attention vector and the first value vector are multiplied in the spatial domain and then subjected to convolution to obtain a first intermediate feature map.

4. The multimodal medical image fusion method according to claim 2, characterized in that, In the first dynamic frequency domain feedforward network layer, the second intermediate feature map is convolved and segmented to obtain multiple first feature vectors. The multiple first feature vectors are multiplied with learnable weights in the frequency domain to obtain multiple second feature vectors. The multiple second feature vectors are then image-reconstructed in the spatial domain and connected via residuals to obtain a third intermediate feature map.

5. The multimodal medical image fusion method according to claim 1, characterized in that, The feature enhancement network includes a second dynamic frequency converter and a third dynamic frequency converter; The first feature sub-image is subtracted from the third feature sub-image and then processed by the second dynamic frequency converter to obtain the third intermediate feature sub-image. The third feature sub-image is subtracted from the first feature sub-image and then processed by the third dynamic frequency converter to obtain the fourth intermediate feature sub-image. The third intermediate feature sub-image and the fourth intermediate feature sub-image are then subjected to overall feature enhancement through residual connection and then spliced ​​to obtain the first intermediate feature sub-image.

6. The multimodal medical image fusion method according to claim 1, characterized in that, The feature mining network includes a first cross-dynamic frequency converter and a second cross-dynamic frequency converter. The first cross-frequency dynamic converter includes a first cross-frequency domain attention layer and a second dynamic frequency domain feedforward network layer. Based on the common features between the second feature sub-map and the fourth feature sub-map, the first cross-frequency domain attention layer enhances the common features contained in the second feature sub-map to obtain a fifth intermediate feature sub-map. After residual connection of the fifth intermediate feature sub-map, frequency domain features are extracted through the second dynamic frequency domain feedforward network layer to obtain a sixth intermediate feature sub-map. After residual connection of the sixth intermediate feature sub-map, a seventh intermediate feature sub-map is obtained. The second cross-frequency dynamic converter includes a second cross-frequency domain attention layer and a third dynamic frequency domain feedforward network layer. The common features contained in the fourth feature sub-map are enhanced by the second cross-frequency domain attention layer to obtain an eighth intermediate feature sub-map. The eighth intermediate feature sub-map is then residual-connected and frequency domain features are extracted by the third dynamic frequency domain feedforward network layer to obtain a ninth intermediate feature sub-map. The ninth intermediate feature sub-map is then residual-connected to obtain a tenth intermediate feature sub-map. The second intermediate feature sub-image is obtained by performing overall feature enhancement on the seventh and tenth intermediate feature sub-images respectively using residual connection and then splicing them together.

7. The multimodal medical image fusion method according to claim 6, characterized in that, In the first cross-frequency domain attention layer, the second value vector and the second key vector are extracted from the second feature sub-map, the second query vector is extracted from the fourth feature sub-map, the second key vector and the second query vector are multiplied in the frequency domain to obtain the second attention vector, and the second attention vector and the second value vector are multiplied in the spatial domain and then subjected to convolution operation to obtain the fifth intermediate feature sub-map. In the second cross-frequency domain attention layer, the third value vector and the third key vector are extracted from the fourth feature sub-map, and the third query vector is extracted from the second feature sub-map. The third key vector and the third query vector are multiplied in the frequency domain to obtain the third attention vector. The third attention vector and the third value vector are multiplied in the spatial domain and then subjected to convolution operation to obtain the eighth intermediate feature sub-map.

8. The multimodal medical image fusion method according to claim 1, characterized in that, The decoding network comprises a three-layer decoder consisting of seven decoders connected in a dense jump pattern within the same layer and a dense pass pattern between adjacent layers. Each decoder includes a sequentially connected 1×1 convolutional layer, a fourth dynamic frequency converter, and a 3×3 convolutional layer.

Images