Multi-modal medical image fusion method, system, device, medium and program product

Abstract

The invention discloses a multi-modal medical image fusion method, system and device, a medium and a program product, and the method comprises the steps: carrying out the feature extraction of two images through a feature extraction module, obtaining a first-grade medical anatomical image feature, a first-grade medical functional image feature and a first-grade common feature, obtaining a second-level medical anatomical image feature, a second-level medical functional image feature and a second-level common feature; the trained medical image fusion network adopts a feature fusion module to fuse the three extracted features of the first level to obtain a first fusion feature; fusing the three extracted features of the second level to obtain a second fused feature; and the trained medical image fusion network adopts an image reconstruction module to carry out image reconstruction on the first fusion feature and the second fusion feature to obtain a final fusion image.

Description

Technical Field

[0001] The present invention relates to the field of image fusion technology, and in particular to a multimodal medical image fusion method, system, device, medium and program product. Background Art

[0002] Due to the limitations of imaging systems and sensor hardware, single-modality medical images have less feature information and cannot fully display the appearance and internal structure of lesions. In clinical diagnosis, doctors need to first observe medical imaging of one modality, form a memory in their brains, and then observe the corresponding imaging of another modality, which brings inconvenience to doctors' diagnosis. Multimodal medical image fusion technology fuses medical images of different modalities so that the fused image contains multiple modal information at the same time, thereby integrating the structure, metabolism and other information of the patient's lesions into one picture, providing high-quality lesion imaging for clinical examination and diagnosis. At present, multimodal medical image fusion has been widely used in disease diagnosis, lesion segmentation, surgical navigation and other fields.

[0003] The inventors found that the deep learning-based method still has some defects that need to be solved:

[0004] (1) In the feature encoding part, dual branches are usually used to encode different modal medical imaging. However, the interaction between the two branches is often relatively simple, resulting in insufficient multimodal feature interaction in the common feature extraction process, which to a certain extent affects the establishment of feature correlation between multimodal imaging.

[0005] (2) Most deep learning image fusion methods do not pay enough attention to the frequency domain information of images.

[0006] (3) Most existing methods use convolutional neural networks (CNN) to reconstruct images, but ordinary convolution cannot capture global features well due to its limited receptive field. Summary of the invention

[0007] In order to solve the deficiencies of the prior art, the present invention provides a multimodal medical image fusion method, system, device, medium and program product;

[0008] On the one hand, a multimodal medical image fusion method is provided, including:

[0009] Obtaining the medical anatomical image and the medical functional image to be fused, inputting the two images into the trained medical image fusion network to obtain the image fusion result;

[0010] The trained medical image fusion network uses a feature extraction module to extract features from the two images respectively, and obtains first-level medical anatomical image features, first-level medical functional image features, and first-level common features, and also obtains second-level medical anatomical image features, second-level medical functional image features, and second-level common features;

[0011] The trained medical image fusion network uses a feature fusion module to fuse the three features of the first level extracted to obtain a first fused feature; and also fuses the three features of the second level extracted to obtain a second fused feature;

[0012] The trained medical image fusion network uses an image reconstruction module to reconstruct the first fusion feature and the second fusion feature to obtain a final fusion image.

[0013] On the other hand, a multimodal medical image fusion system is provided, comprising:

[0014] An image fusion module is configured to: obtain a medical anatomical image and a medical functional image to be fused, input the two images into a trained medical image fusion network, and obtain an image fusion result;

[0015] The trained medical image fusion network uses a feature extraction module to extract features from the two images respectively, and obtains first-level medical anatomical image features, first-level medical functional image features, and first-level common features, and also obtains second-level medical anatomical image features, second-level medical functional image features, and second-level common features;

[0016] The trained medical image fusion network uses a feature fusion module to fuse the three features of the first level extracted to obtain a first fused feature; and also fuses the three features of the second level extracted to obtain a second fused feature;

[0017] The trained medical image fusion network uses an image reconstruction module to reconstruct the first fusion feature and the second fusion feature to obtain a final fusion image.

[0018] On the other hand, an electronic device is provided, comprising:

[0019] a memory for non-transitory storage of computer-readable instructions; and

[0020] a processor for executing the computer readable instructions,

[0021] When the computer-readable instructions are executed by the processor, the method described in the first aspect is executed.

[0022] On the other hand, a storage medium is provided, which non-temporarily stores computer-readable instructions, wherein when the non-temporary computer-readable instructions are executed by a computer, the method described in the first aspect is executed.

[0023] On the other hand, a computer program product is provided, comprising a computer program, wherein the computer program is used to implement the method described in the first aspect when running on one or more processors.

[0024] The above technical solution has the following advantages or beneficial effects:

[0025] In order to fully extract the common features between modalities and the unique features within the modality, the feature extraction module of the present invention adopts three branches. The two side branches only accept single modality input to extract the unique features within the modality; the middle branch integrates the medical image features of different modalities and learns the common feature information from the modalities. In addition, the designed progressive multi-scale image feature extraction module can fully extract the multi-scale information of different modality images through a progressive strategy, and the designed feature extraction module submodule can simultaneously integrate the features of all scale levels.

[0026] In order to learn the representation of images in the frequency domain and enhance their feature interactions in the spatial domain, the present invention designs a feature fusion module, which uses the feature interaction capability of the cross-attention mechanism to enhance the spatial features of different modalities and guide the fusion of high- and low-frequency position information in the frequency domain. The model is deeply trained using image features of different frequencies, so that the fused image can fully integrate multiple frequency information such as texture and details. In addition, the alternating enhanced fusion of the spatial domain and the frequency domain can improve the model fusion capability. At the same time, the present invention uses a variety of gradient operators to further supplement the edge features of the image for the model.

[0027] In deep learning, the global features and local features of medical images refer to the overall or local attribute information of the image, respectively. Global features generally refer to the features calculated by the model based on all pixel values ​​of the image in the spatial domain, while local features refer to the features calculated by the model based on the pixel values ​​of each sub-region of the image in the spatial domain. In actual application scenarios, the global features of medical images reflect the overall structure, shape and other properties of human organs or tissues, while local features express information such as texture details in certain local areas of medical images. In clinical practice, doctors judge the health of the human body based on these feature information combined with medical knowledge.

[0028] In order to effectively utilize global and local feature information for image reconstruction, the present invention designs a global-local feature information reconstruction module based on Swin Transformer (hierarchical vision transformer using shifted windows) and CNN, and utilizes the self-attention mechanism and shifted window mechanism of Swin Transformer to adaptively focus on different regions of the image, establish global feature connections of the image, deeply mine the overall feature information of the image, and flexibly handle the interdependence of features at different positions through cross-window interaction. At the same time, the regional feature information extracted by CNN is combined to effectively establish the global and local feature connections of the image, thereby improving the quality of the reconstructed image.

[0029] In summary, the present invention designs a multi-scale medical image fusion network (PEFI) based on progressive feature extraction, frequency domain information complementation and Swin Transformer combined with CNN reconstruction. This model can effectively extract the multi-scale features of the image and use the high and low frequency position information in the frequency domain to guide the enhanced feature fusion effect, while improving the reconstruction quality of the fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0031] Figure 1 This is a schematic diagram of the overall network structure of Example 1;

[0032] Figure 2 A detailed diagram of the overall network structure of Embodiment 1;

[0033] Figure 3 It is a feature extractor of embodiment 1;

[0034] Figure 4 The image fusion module of the first embodiment;

[0035] Figure 5 This is the image reconstruction module of the first embodiment. DETAILED DESCRIPTION

[0036] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0037] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0038] In this embodiment, all data is obtained in compliance with laws and regulations and based on the user's consent, and is used legally.

[0039] Embodiment 1

[0040] This embodiment provides a multimodal medical image fusion method;

[0041] Multimodal medical image fusion methods, including:

[0042] Obtaining the medical anatomical image and the medical functional image to be fused, inputting the two images into the trained medical image fusion network to obtain the image fusion result;

[0043] The trained medical image fusion network uses a feature extraction module to extract features from the two images respectively, and obtains first-level medical anatomical image features, first-level medical functional image features, and first-level common features, and also obtains second-level medical anatomical image features, second-level medical functional image features, and second-level common features;

[0044] The trained medical image fusion network uses a feature fusion module to fuse the three features of the first level extracted to obtain a first fused feature; and also fuses the three features of the second level extracted to obtain a second fused feature;

[0045] The trained medical image fusion network uses an image reconstruction module to reconstruct the first fusion feature and the second fusion feature to obtain a final fusion image.

[0046] Furthermore, medical anatomical images refer to computed tomography (CT), magnetic resonance imaging (MRI), etc., which can provide clear anatomical structure information but cannot provide information such as human function and metabolism.

[0047] Furthermore, medical functional images refer to single-photon emission computed tomography (SPECT), positron emission tomography (PET), etc., which can reflect function, metabolism and other information but are difficult to provide anatomical structure and positioning information.

[0048] Furthermore, the training process of the trained medical image fusion network includes:

[0049] Constructing a training set, wherein the training set is a medical anatomical image and a medical functional image of a known fusion image;

[0050] The training set is input into the medical image fusion network, and the network is trained. When the total loss function value of the network no longer decreases, the training is stopped to obtain the trained medical image fusion network.

[0051] Furthermore, the total loss function of the network is a weighted sum of loss functions consisting of structural similarity loss, pixel intensity loss, gradient loss, regional mutual information loss and mean square error loss.

[0052] Furthermore, the total loss function of the network is specifically expressed as:

[0053] L=L RMI +ω1L S +ω2L int +ω3L MSE +ω4L grad (19)

[0054] Among them, ω1, ω2, ω3 and ω4 are all hyperparameters, ω1 = 7, ω2 = 2, ω3 = 10, ω4 = 10. L RMI is the regional mutual information loss, L S is the structural similarity loss, L int is the pixel intensity loss, L MSE is the mean square error loss, L grad is the gradient loss.

[0055] In order to better determine the loss weight and make the fusion result biased towards the original image with richer information and more significant grayscale value, the present invention adopts the loss weight based on the measurement of saliency and information entropy:

[0056]

[0057] Among them, m saliency and m abundance are the significance measure and information richness measure, respectively. Subscripts A and B represent functional images and anatomical images, respectively. λ is a hyperparameter, which is set to 3 here. a and b is an intermediate variable, ω a and ω b are the loss weights for functional images and anatomical images, respectively. Saliency measure m saliency for:

[0058]

[0059] Among them, Sign(x) is the sign function, τ is the threshold for screening the prominent area, and are the pixel values ​​of the functional image and the anatomical image at (i, j), respectively, and H and W are the height and width of the image, respectively.

[0060] Information richness is defined as follows:

[0061] m A abundance =EN(I A ),m B abundance =EN(I B ); (13)

[0062] Among them, I A and I B represent functional images and anatomical images respectively, and EN(·) is the information entropy function.

[0063] Furthermore, an excellent image fusion model can fully retain the information of each sub-region of different modality images. Therefore, the present invention introduces the regional mutual information loss L RMI , which enables the fusion result to effectively aggregate the information from the two original images from a region-to-region perspective.

[0064] L RMI =ω a ·RMI(I F ,I A )+ω b ·RMI(I F ,I B )(18)

[0065] Among them, the regional mutual information function Hyperparameters η∈[0,1], I F is a fused image, S represents a functional image or an anatomical image, and L ce (I F ,S) is I F and the cross entropy loss between S, B is the batch size, I l (I F ; S) represents the lower limit of mutual information.

[0066] Furthermore, in order to make the fused image retain the structural features of the original images from different modalities, the structural similarity loss is introduced:

[0067] L S =ω a (1-SSIM(I F ,I A ))+ω b(1-SSIM(I F ,I B )); (14)

[0068] Among them, SSIM(·) is the structural similarity function.

[0069] Furthermore, in order to make the fused image retain the visually significant area of ​​the original image as much as possible, the present invention introduces the pixel intensity loss L int To encourage the model to generate fused images with appropriate intensity:

[0070]

[0071] Among them, ·1 is the L1 criterion, and max(·) means taking the maximum value of the element.

[0072] In addition, if the fusion model only has pixel intensity loss, then the optimal solution of the model's fusion result has been artificially fixed, that is, the pixel maximum value matrix of the functional image and the anatomical image at each spatial position.

[0073] Furthermore, in order to enable the model to break through the optimal solution of artificial limitations and obtain better results, the mean square error loss function L is introduced MSE :

[0074]

[0075] in, is the pixel value of the fused image at (i, j).

[0076] Furthermore, in order to fully preserve the gradient information of images of different modalities and make the fused image have the texture details of both original images, the gradient loss L is introduced. grad :

[0077]

[0078] in, Represents the Sobel gradient operator.

[0079] Furthermore, if Figure 1 As shown, the trained medical image fusion network includes: a feature extraction module, a feature fusion module and an image reconstruction module connected in sequence.

[0080] Furthermore, the feature extraction module includes: three parallel branches;

[0081] The three branches are respectively a first branch, a second branch and a third branch;

[0082] The first branch includes: a first feature extractor, a second feature extractor and a third feature extractor connected in sequence;

[0083] The second branch comprises: a fourth feature extractor, a fifth feature extractor and a sixth feature extractor connected in sequence;

[0084] The third branch comprises: a seventh feature extractor and an eighth feature extractor connected in sequence;

[0085] The input end of the first feature extractor inputs a medical anatomical image; the input end of the fourth feature extractor inputs a medical functional image;

[0086] The output end of the first feature extractor and the output end of the fourth feature extractor are both connected to the input end of the seventh feature extractor.

[0087] The feature extraction module is designed with three branches to extract the common information between modalities and the unique information within each modality, such as Figure 2 The feature extraction module is shown in Figure 2. The feature extraction module operation can be expressed as:

[0088]

[0089] Among them, MSFE(·) is the feature extractor, I A and I B They are medical functional images and medical anatomical images, subscript uniqueA i and uniqueB i It refers to the output of the i-th feature extractor, and Concatenate(·) means concatenation by channel.

[0090] Gradient information plays an important role in the image fusion process, indicating the speed and direction of the gray value change of the image, and can provide the edge features of each medical image for the model. At the same time, a high-quality fused image must fully retain the multi-scale information of the original image. Therefore, the present invention simplifies the process of multi-scale feature extraction and fusion and uses it as a separate feature extraction module. In the feature extraction process, the feature extractor is repeatedly used to progressively extract and fuse the multi-scale features of the image, and the high-resolution features are enhanced with the help of low-resolution features at all depth levels.

[0091] The feature extraction module can effectively extract the multi-scale feature information of medical images of different modalities. Secondly, the feature fusion module can make full use of the feature position information of multi-modal medical images in the frequency domain to guide the more targeted fusion of high- and low-frequency features between modalities. Finally, the image reconstruction module uses the global feature modeling capability of Swin Transformer and the local feature extraction capability of CNN to realize the reconstruction of feature information in the global range of the image.

[0092] Specifically, the feature extraction module is divided into three branches. The first and second branches extract the unique information of different modal images, and the third branch extracts their common information. First, two multi-scale feature extractors guided by gradient information (MSFE) are used to extract the features of the two original images, and the feature maps are cascaded and input into the third branch. The common features of the original images are repeatedly and progressively extracted using MSFE, while the first and second branches continue to progressively extract unique features.

[0093] The common features between modalities extracted by each layer of the feature extraction module and the unique features of two different modalities are input into the progressive fusion module with cross-attention (CAPF) for inter-modal fusion. The fused features are cascaded and input into the decoder, and the image is reconstructed using the image reconstruction module. Finally, they are input into two convolution blocks for channel compression to obtain the final fused image.

[0094] In addition, in order to solve the problem of mismatch between the functional image and the anatomical image channels, in the image preprocessing and postprocessing stages, the present invention adopts RGB-YCrCb color space transformation, converts the RGB functional image color space from RGB to YCrCb, and takes the Y channel for fusion with the anatomical image. The fused image is used as the Y channel to perform an inverse transformation of the YCrCb-RGB color space with the chromaticity components Cr and Cb of the functional image, thereby obtaining the final fused image.

[0095] Furthermore, a feature extraction module is used to extract features from the two images respectively, to obtain first-level medical anatomical image features, first-level medical functional image features and first-level common features, and to obtain second-level medical anatomical image features, second-level medical functional image features and second-level common features, including:

[0096] a second feature extractor, outputting a first-level medical anatomical image feature; a fifth feature extractor, outputting a first-level medical functional image feature; and a seventh feature extractor, outputting a first-level common feature;

[0097] The third feature extractor outputs the second level of medical anatomical image features; the sixth feature extractor outputs the second level of medical functional image features; and the eighth feature extractor outputs the second level of common features.

[0098] Furthermore, the internal structures of the first feature extractor, the second feature extractor, the third feature extractor, the fourth feature extractor, the fifth feature extractor, the sixth feature extractor, the seventh feature extractor and the eighth feature extractor are consistent, such as Figure 3 As shown, the first feature extractor includes: six parallel branches: a fourth branch, a fifth branch, a sixth branch, a seventh branch, an eighth branch and a ninth branch;

[0099] The fourth branch includes: a first convolutional layer, a first batch of normalization layers and a first activation function layer connected in sequence;

[0100] The fifth branch comprises: a first downsampling layer, a second convolutional layer, a second batch normalization layer, a second activation function layer, a first adder, a first multiplier, a second adder and a first connection layer connected in sequence;

[0101] The sixth branch comprises: a second downsampling layer, a third convolutional layer, a third batch normalization layer and a third activation function layer connected in sequence;

[0102] The seventh branch includes: a Sobel operator layer;

[0103] The eighth branch includes: a Laplace operator layer, a second connection layer and a spatial attention layer connected in sequence;

[0104] The ninth branch includes: a Prewitt operator layer;

[0105] The input end of the first convolutional layer, the input end of the first downsampling layer, the input end of the second downsampling layer, the input end of the Sobel operator layer, the input end of the Laplace operator layer, and the input end of the Prewitt operator layer are all connected to the input end of the first feature extractor; the input end of the first feature extractor is also connected to the input end of the first connection layer;

[0106] Wherein, the output end of the first activation function layer and the output end of the third activation function layer are both connected to the input end of the first adder; the output end of the first adder is connected to the input end of the second adder;

[0107] Among them, the output end of the Sobel operator layer and the output end of the Prewitt operator layer are both connected to the input end of the second connection layer; the output end of the spatial attention mechanism layer is connected to the input end of the first multiplier.

[0108] It should be understood that the first connection layer and the second connection layer connect the input values ​​according to the channel dimension.

[0109] It should be understood that the first downsampling layer is implemented by a double downsampling layer, and the double downsampling layer means that the overall number of pixels of the original medical image or feature map is reduced to half of the original number after the downsampling operation.

[0110] The second downsampling layer is implemented by a four-fold downsampling layer, and the four-fold downsampling layer means that the overall number of pixels of the original medical image or feature map is reduced to one-fourth of the original number after the downsampling operation.

[0111] Furthermore, the working processes of the first feature extractor, the second feature extractor, the third feature extractor, the fourth feature extractor, the fifth feature extractor, the sixth feature extractor, the seventh feature extractor and the eighth feature extractor are consistent, such as Figure 3 As shown, the first feature extractor comprises:

[0112] like Figure 3 As shown in the figure, the input feature map is first downsampled by 2 and 4 times, and then the downsampled feature map and the original feature map are input into three 3×3 convolution blocks for multi-scale feature extraction, and the element-by-element addition operation is performed. This process can be expressed as:

[0113]

[0114] Among them, Conv out is the output of the convolutional block, Conv 3×3 (·) is a convolution with a kernel size of 3, BN(-) and PReLu(·) represent batch normalization and parameterized linear activation functions, respectively. Input represents the input feature map, and Down(Input, N) represents downsampling the input by a factor of N.

[0115] At the same time, the gradient information of the input feature map is extracted. Specifically, the gradient information of the image is extracted through the Sobel, Laplace and Prewitt operators respectively. After channel cascading, the spatial position weights are adjusted and fused using the spatial attention mechanism. This process is expressed as:

[0116] F grad =Spa(Concatenate(Sobel(Input),Laplace(Input),Prewitt(Input)))(3)

[0117] Among them, Spa(·) represents the spatial attention mechanism, Sobel(·), Laplace(·) and Prewitt(·) represent the Sobel, Laplace and Prewitt operators respectively.

[0118] Finally, the multi-scale feature map is multiplied and added with the gradient information, and then cascaded with the module input to obtain the fusion result. The final result can be expressed as:

[0119] F out =Concatenate((F conv +(F conv ×F grad ),Input)(4).

[0120] Spatial attention mechanism from Sanghyun Woo, Jongchan Park, Joon-Young Lee and In SoKweon. CBAM: Convolutional block attention module.

[0121] Furthermore, the trained medical image fusion network uses a feature fusion module to fuse the three features of the first level extracted to obtain a first fused feature; and also fuses the three features of the second level extracted to obtain a second fused feature; the feature fusion module includes:

[0122] a first feature fusion unit and a second feature fusion unit;

[0123] The first feature fusion unit fuses the three extracted features of the first level to obtain a first fused feature;

[0124] The second feature fusion unit fuses the three extracted features of the second level to obtain a second fused feature.

[0125] Furthermore, if Figure 4 As shown, the internal structures of the first feature fusion unit and the second feature fusion unit are consistent, and the first feature fusion unit includes:

[0126] the tenth branch, the eleventh branch, and the twelfth branch;

[0127] The tenth branch includes: a first cross attention mechanism layer, a third adder, a fourth convolution layer, a fourth batch normalization layer, a fourth activation function layer and a first Fourier transform layer connected in sequence;

[0128] The eleventh branch includes: a second cross attention mechanism layer, a fourth adder, a fifth convolution layer, a fifth batch normalization layer, a fifth activation function layer, and a second Fourier transform layer connected in sequence;

[0129] The twelfth branch includes: a fifth adder, a sixth convolution layer, a sixth batch normalization layer, a sixth activation function layer, a third Fourier transform layer and a third connection layer connected in sequence;

[0130] The output end of the first Fourier transform layer and the output end of the second Fourier transform layer are both connected to the input end of the fifth adder, and the output end of the third connection layer is the output end of the first feature fusion unit;

[0131] The input end of the third connection layer of the first feature fusion unit is used to input the output value of the seventh feature extractor;

[0132] An input end of the first cross attention mechanism layer of the first feature fusion unit is used to input the output value of the second feature extractor; an input end of the second cross attention mechanism layer of the first feature fusion unit is used to input the output value of the fifth feature extractor;

[0133] The input end of the first cross attention mechanism layer of the first feature fusion unit is also connected to the input end of the third adder; the input end of the first cross attention mechanism layer of the first feature fusion unit is also connected to the input end of the second cross attention mechanism layer;

[0134] The input end of the second cross-attention mechanism layer of the first feature fusion unit is also connected to the input end of the fourth adder; the input end of the second cross-attention mechanism layer of the first feature fusion unit is also connected to the input end of the first cross-attention mechanism layer.

[0135] Furthermore, the input end of the third connection layer of the second feature fusion unit is used to input the output value of the eighth feature extractor; the input end of the first cross-attention mechanism layer of the second feature fusion unit is used to input the output value of the third feature extractor; and the input end of the second cross-attention mechanism layer of the second feature fusion unit is used to input the output value of the sixth feature extractor.

[0136] Furthermore, the third connection layer connects the input values ​​in the channel dimension.

[0137] The cross-attention mechanism can mutually enhance multimodal information in the spatial domain. Through query, key and value operations, the feature sequences of medical images of two different modalities can pay attention to each other, and promote the interactive improvement of features such as structural position and functional metabolism in functional imaging and structural imaging, so as to further enrich the feature representation of the image, provide the reconstruction module with features with a higher degree of image information retention, and further improve the quality of multimodal medical fusion images.

[0138] In order to make the single-modal features complement the features of another modality from a global perspective, the present invention performs cross-attention fusion on the unique information from two different modalities in the spatial domain in the feature fusion part, such as Figure 4 shown.

[0139] Furthermore, the first cross attention mechanism layer includes:

[0140] Given a functional medical image feature F A ∈R H×W and anatomical medical image features F B ∈R H×W , the feature map is transformed from R H×W Zoom to Then, the downsampled functional medical image features and anatomical medical image features are projected into matrices q1, k1, v1 and q2, k2, v2 respectively through linear transformation:

[0141]

[0142] Among them, q, k, and v represent query, key, and value respectively.

[0143] Secondly, the correlation matrix between modes is established through dot product operation, and the feature F A Supplementary feature F B Information: Use feature F A The projected q1 matrix and feature F B The k2 matrix obtained by projection is subjected to dot product operation and the result is normalized using the softmax function to obtain the correlation weight between the functional image and the anatomical image. B The final result is obtained by performing a dot product between the projected v2 matrix and the correlation weight. d is the input feature dimension, and the above process can be expressed as:

[0144]

[0145] Towards feature F B Supplementary feature F A The information process is:

[0146]

[0147] Most deep learning models usually focus on the processing of image features in the spatial domain when fusing features, and do not pay enough attention to the representation information of the image in the frequency domain.

[0148] Furthermore, the first feature fusion unit includes:

[0149] The two feature maps after cross attention fusion are added to the original feature map, and the features are further extracted using a 3×3 convolution block, and the feature map is converted to the frequency domain through Fourier transform;

[0150] The feature maps are added and fused element by element in the frequency domain, and the connection between local features is established through a 3×3 convolution block. The fusion result is transformed to the spatial domain through the inverse Fourier transform.

[0151] Finally, the common features between modalities extracted by the feature extraction module are cascaded; the above process is expressed as:

[0152]

[0153] Among them, Fourier(·) represents the two-dimensional discrete Fourier transform, Inverse(·) represents the inverse Fourier transform, and F j Represents the common features between modalities extracted by the feature extraction module.

[0154] The two-dimensional discrete Fourier transform and inverse transform are defined as follows:

[0155]

[0156] It should be understood that the model aggregates the same frequency features of the image that are dispersed and less connected in the spatial domain through Fourier transform, so that the high- and low-frequency information is in the same position in the feature maps of two different modes, and uses the position information of high and low frequencies in the frequency domain to guide the fusion of modal features, so as to more specifically fuse high-frequency components such as the edges and textures of human organs and low-frequency components with smooth changes in grayscale values ​​such as tissue metabolism, enrich the knowledge learned by the model, and further improve the fusion performance.

[0157] Furthermore, the trained medical image fusion network uses an image reconstruction module to perform image reconstruction on the first fusion feature and the second fusion feature to obtain a final fusion image;

[0158] The image reconstruction module includes: a fourth connection layer, a first reconstruction unit, a second reconstruction unit, a third reconstruction unit, a seventh convolutional layer, a seventh batch normalization layer, a seventh activation function layer, an eighth convolutional layer, an eighth batch normalization layer and an eighth activation function layer connected in sequence.

[0159] Furthermore, the internal structures of the first reconstruction unit, the second reconstruction unit and the third reconstruction unit are consistent, and the first reconstruction unit comprises:

[0160] the Thirteenth, Fourteenth, and Fifteenth Branches;

[0161] The thirteenth branch includes: a ninth convolutional layer, a ninth batch normalization layer, and a ninth activation function layer connected in sequence;

[0162] The fourteenth branch includes: a patch partitioning module, a linear mapping layer, a first-layer normalization module, a window-based multi-head self-attention mechanism layer, a sixth adder, a second-layer normalization module, a first multi-layer perceptron and a seventh adder connected in sequence;

[0163] All modules of the fourteenth branch come from Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. The patch partitioning module divides the image into small blocks of the same size, and the linear mapping layer embeds the patch into a specific channel dimension.

[0164] The fifteenth branch includes: a third normalization module, a shift window self-attention mechanism layer, an eighth adder, a fourth normalization module, a second multi-layer perceptron, a ninth adder and a fifth connection layer connected in sequence;

[0165] All modules in the fifteenth branch are from Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows.

[0166] The input end of the ninth convolutional layer and the input end of the patch division module are both connected to the input end of the first reconstruction unit; the output end of the fifth connection layer is the output end of the first reconstruction unit; the fifth connection layer connects the input values ​​according to the channel dimension;

[0167] The seventh adder is connected to the third layer normalization module; the output end of the ninth activation function layer is connected to the input end of the fifth connection layer;

[0168] The output end of the linear mapping layer is connected to the input end of the sixth adder; the output end of the sixth adder is connected to the input end of the seventh adder;

[0169] The output terminal of the seventh adder is connected to the input terminal of the eighth adder, and the output terminal of the eighth adder is connected to the input terminal of the ninth adder.

[0170] The first reconstruction unit comprises:

[0171] Swin-CNN out =Concatenate(Swin(Input),PReLu(BN(Conv 3×3 (Input)))); (10)

[0172] Swin-CNN outIt is a global-local feature information reconstruction module based on Swin-CNN. Swin(·) is a SwinTransformer network. Source: Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows.

[0173] The core idea of ​​Swin Transformer is window attention and shifting window mechanism. By dividing the image into multiple non-overlapping windows and performing self-attention operations within the window, the local dependencies of the medical image can be effectively established, so that the local features of the medical image can be fully preserved. The shifting window mechanism is used to interact features between different windows to reduce the amount of calculation and fully establish the global connection between the features of each local window, thereby improving the model's ability to pay attention to the global medical image feature sequence and effectively establishing the association between features at different positions.

[0174] Inspired by Swin Transformer, in order to improve the problem of poor global dependency modeling ability of CNN due to limited receptive field, so as to fully extract the global features of multimodal medical images and establish long-range dependencies across windows, the convolutional layer and Swin Transformer are combined in parallel. 3×3 convolutional layers, batch normalization layers and PReLu activation functions (parametric rectified linear unit) are used to extract local information of feature maps. At the same time, Swin Transformer is used to extract global information of feature maps, and then the local and global information are cascaded.

[0175] The feature maps obtained by the three Swin-CNN blocks are sequentially input into two modules consisting of convolution, batch normalization layer and activation function layer to output the final fused image, as shown in Figure 5 To further reduce the amount of calculation, the Swin Transformer in the present invention only uses one base layer in the original Swin Transformer framework.

[0176] The purpose of the present invention is to fully retain the details of the unique features of each modality and effectively integrate the common features between modalities. However, the commonly used dual-branch image coding method currently has problems such as imperfect information exchange between the interdependent and interrelated common features between modalities and insufficient process, which affects the establishment of feature correlation between multimodal imaging. In response to these problems, a multi-scale medical image fusion network based on progressive feature extraction, frequency domain information supplementation and Swin Transformer combined with CNN reconstruction is designed.

[0177] Firstly, a multi-scale feature extraction module guided by gradient information is designed. Its combined three-branch feature extraction architecture fully considers the information interaction between modalities and can progressively extract the common features of the image and the unique features of different modalities. In order to make full use of the guidance of frequency domain information on the modal fusion position and enable the model to more directional fuse the high- and low-frequency parts of medical images, a progressive fusion module is designed in combination with cross-attention, which simultaneously realizes the spatial information interaction enhancement between different modalities and the targeted multi-modal information fusion guided by the high- and low-frequency position information in the frequency domain. Finally, the Swin-CNN reconstruction module is designed to establish the connection between the corresponding features of the global and local regions of medical images by making full use of the Swin Transformer's ability to capture global information such as the overall structure and shape of the image and the CNN's advantages in extracting regional features such as local texture details of the image.

[0178] Extensive experiments on three public fusion datasets show that in MRI-SPECT and MRI-PET fusion tasks, the best results are achieved in eight visual fusion task evaluation indicators compared with eight state-of-the-art methods. Compared with the second-best performing model, mutual information (MI) and discrete cosine transform feature mutual information (FMI) dct ) were significantly improved, with MI increasing by 4.42% and 17.30% respectively, and FMI dct The performance of the proposed model was significantly improved by 5.17% and 11% respectively. In the GFP-PC fusion task, 6 optimal results and 2 suboptimal results were achieved. Compared with the second-best model, MI and VIF (visual information fidelity) were significantly improved by 16.43% and 16.87% respectively.

[0179] The proposed fusion model fully exploits the features of different modal images and progressively integrates multi-scale information through the common and unique feature extraction capabilities of the three-branch architecture. It uses the feature fusion module to guide the model to more specifically fuse high- and low-frequency features, and pays attention to the global and local attribute information of medical images during the reconstruction process, effectively improving the quality of multimodal medical image fusion.

[0180] Embodiment 2

[0181] This embodiment provides a multimodal medical image fusion system, including:

[0182] An image fusion module is configured to: obtain a medical anatomical image and a medical functional image to be fused, input the two images into a trained medical image fusion network, and obtain an image fusion result;

[0183] The trained medical image fusion network uses a feature extraction module to extract features from the two images respectively, and obtains first-level medical anatomical image features, first-level medical functional image features, and first-level common features, and also obtains second-level medical anatomical image features, second-level medical functional image features, and second-level common features;

[0184] The trained medical image fusion network uses a feature fusion module to fuse the three features of the first level extracted to obtain a first fused feature; and also fuses the three features of the second level extracted to obtain a second fused feature;

[0185] The trained medical image fusion network uses an image reconstruction module to reconstruct the first fusion feature and the second fusion feature to obtain a final fusion image.

[0186] It should be noted that the above-mentioned image fusion module corresponds to the step of obtaining the medical anatomical image and the medical functional image to be fused, inputting the two images into the trained medical image fusion network, and obtaining the image fusion result in the first embodiment, and the examples and application scenarios implemented by the above-mentioned module and the corresponding steps are the same, but are not limited to the contents disclosed in the first embodiment. It should be noted that the above-mentioned module as part of the system can be executed in a computer system such as a set of computer executable instructions.

[0187] The description of each embodiment in the above embodiments has different emphases. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0188] The proposed system can be implemented in other ways. For example, the system embodiment described above is only illustrative, and the division of the modules is only a logical function division. In actual implementation, there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0189] Embodiment 3

[0190] This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory so that the electronic device executes the method described in the above embodiment one.

[0191] It should be understood that in this embodiment, the processor may be a central processing unit CPU, and the processor may also be other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0192] The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0193] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software.

[0194] The method in the first embodiment can be directly embodied as a hardware processor, or a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0195] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0196] Embodiment 4

[0197] This embodiment further provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the method described in the first embodiment is completed.

[0198] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multimodal medical image fusion method, characterized in that: include: Obtaining the medical anatomical image and the medical functional image to be fused, inputting the two images into the trained medical image fusion network to obtain the image fusion result; The trained medical image fusion network uses a feature extraction module to extract features from the two images respectively, and obtains first-level medical anatomical image features, first-level medical functional image features, and first-level common features, and also obtains second-level medical anatomical image features, second-level medical functional image features, and second-level common features; The trained medical image fusion network uses a feature fusion module to fuse the three features of the first level extracted to obtain a first fused feature; and also fuses the three features of the second level extracted to obtain a second fused feature; The trained medical image fusion network uses an image reconstruction module to reconstruct the first fusion feature and the second fusion feature to obtain a final fusion image.

2. The multimodal medical image fusion method according to claim 1, characterized in that: The total loss function of the trained medical image fusion network is the weighted sum of the loss functions consisting of structural similarity loss, pixel intensity loss, gradient loss, regional mutual information loss, and mean square error loss. The total loss function of the network is specifically expressed as: L=L RMI +ω1L S +ω2L int +ω3L MSE +ω4L grad (19) Among them, ω1, ω2, ω3 and ω4 are hyperparameters, L RMI is the regional mutual information loss, L S is the structural similarity loss, L int is the pixel intensity loss, L MSE is the mean square error loss, L grad is the gradient loss.

3. The multimodal medical image fusion method according to claim 1, characterized in that: The trained medical image fusion network includes: a feature extraction module, a feature fusion module and an image reconstruction module connected in sequence; the feature extraction module includes: three parallel branches; The three branches are respectively a first branch, a second branch and a third branch; The first branch includes: a first feature extractor, a second feature extractor and a third feature extractor connected in sequence; The second branch comprises: a fourth feature extractor, a fifth feature extractor and a sixth feature extractor connected in sequence; The third branch comprises: a seventh feature extractor and an eighth feature extractor connected in sequence; The input end of the first feature extractor inputs a medical anatomical image; the input end of the fourth feature extractor inputs a medical functional image; The output end of the first feature extractor and the output end of the fourth feature extractor are both connected to the input end of the seventh feature extractor; The feature extraction module operation is expressed as: Among them, MSFE(·) is the feature extractor, I A and I B They are medical functional images and medical anatomical images, subscript uniqueA i and uniqueB i It refers to the output of the i-th feature extractor, and Concatenate(·) means concatenation by channel.

4. The multimodal medical image fusion method according to claim 3, characterized in that: The first feature extractor includes: firstly downsampling the input feature map by 2 times and 4 times respectively, then inputting the downsampled feature map and the original feature map into three 3×3 convolution blocks respectively for multi-scale feature extraction, and performing element-by-element addition operation, which is expressed as: Among them, Conv out is the output of the convolutional block, Conv 3×3 (·) is a convolution with a kernel size of 3, BN(·) and PReLu(·) represent batch normalization and parameterized linear activation functions, respectively; Input represents the input feature map, and Down(Input, N) represents downsampling the input by a factor of N; The gradient information of the image is extracted by Sobel, Laplace and Prewitt operators respectively. After channel cascading, the spatial position weights are adjusted and fused using the spatial attention mechanism. This process is expressed as: F grad =Spa(Concatenate(Sobel(Input),Laplace(Input),Prewitt(Input)))(3) Among them, Spa(·) represents the spatial attention mechanism, Sobel(·), Laplace(·) and Prewitt(·) represent Sobel, Laplace and Prewitt operators respectively; Finally, the multi-scale feature map is multiplied and added with the gradient information, and then cascaded with the module input to obtain the fusion result. The final result can be expressed as: F out =Concatenate((F conv +(F conv ×F grad ),Input)(4)。 5. The multimodal medical image fusion method according to claim 1, characterized in that: The feature fusion module includes: a first feature fusion unit and a second feature fusion unit; The first feature fusion unit fuses the three extracted features of the first level to obtain a first fused feature; The second feature fusion unit fuses the three extracted features of the second level to obtain a second fused feature; The first feature fusion unit includes: The two feature maps after cross attention fusion are added to the original feature map, and the features are further extracted using a 3×3 convolution block, and the feature map is converted to the frequency domain through Fourier transform; The feature maps are added and fused element by element in the frequency domain, and the connection between local features is established through a 3×3 convolution block. The fusion result is transformed to the spatial domain through the inverse Fourier transform. Finally, the common features between modalities extracted by the feature extraction module are cascaded. The above process is expressed as: Among them, Fourier(·) represents the two-dimensional discrete Fourier transform, Inverse(·) represents the inverse Fourier transform, and F j Represents the common features between modalities extracted by the feature extraction module.

6. The multimodal medical image fusion method according to claim 1, characterized in that: The trained medical image fusion network uses an image reconstruction module to reconstruct the first fusion feature and the second fusion feature to obtain a final fusion image; The image reconstruction module comprises: a fourth connection layer, a first reconstruction unit, a second reconstruction unit, a third reconstruction unit, a seventh convolutional layer, a seventh batch normalization layer, a seventh activation function layer, an eighth convolutional layer, an eighth batch normalization layer and an eighth activation function layer connected in sequence; The first reconstruction unit comprises: Swin-CNN out =Concatenate(Swin(Input),PReLu(BN(Conv 3×3 (Input))));(10)。 7. A multimodal medical image fusion system, characterized in that: include: An image fusion module is configured to: obtain a medical anatomical image and a medical functional image to be fused, input the two images into a trained medical image fusion network, and obtain an image fusion result; The trained medical image fusion network uses a feature extraction module to extract features from the two images respectively, and obtains first-level medical anatomical image features, first-level medical functional image features, and first-level common features, and also obtains second-level medical anatomical image features, second-level medical functional image features, and second-level common features; The trained medical image fusion network uses a feature fusion module to fuse the three features of the first level extracted to obtain a first fused feature; and also fuses the three features of the second level extracted to obtain a second fused feature; The trained medical image fusion network uses an image reconstruction module to reconstruct the first fusion feature and the second fusion feature to obtain a final fusion image.

8. An electronic device, comprising: a memory for non-transitory storage of computer readable instructions; as well as a processor for executing the computer readable instructions, Wherein, when the computer-readable instructions are executed by the processor, the method described in any one of claims 1 to 6 is executed.

9. A storage medium, characterized in that: Non-transitory storage of computer-readable instructions, wherein when the non-transitory computer-readable instructions are executed by a computer, the method according to any one of claims 1 to 6 is performed.

10. A computer program product, characterized in that The invention comprises a computer program, which is used to implement the method according to any one of claims 1 to 6 when running on one or more processors.

Images