Medical image fusion method, device, equipment and storage medium

Abstract

The present invention discloses a method, device, equipment and storage medium for medical image fusion based on deep learning, including: S1, obtaining the original medical image of the target object to be fused, and preprocessing it to obtain enhanced medical image data; S2, extracting depth information from the enhanced medical image data to obtain a depth feature information map containing preliminary feature information; S3, extracting multi-scale complementary information from the depth feature information map to obtain a fused feature information map; S4, inputting the fused feature information map into a convolution reconstruction module to obtain a final visualized fused grayscale image. The present invention is used to fuse multiple modality medical images to provide clearer structural and soft tissue medical fusion images, solving the technical problem that a single medical modality image often cannot provide comprehensive diagnostic information.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a medical image fusion method, device, equipment and storage medium based on deep learning. Background Art

[0002] X-ray computed tomography (CT) images reflect the tissue density and X-ray absorption information of an organism, which helps diagnose tumors, vascular diseases, and brain injuries. However, in some cases, CT has relatively limited resolution for subtle soft tissue structures, which may lead to unclear structures. On the other hand, magnetic resonance imaging (MRI) images contain rich soft tissue structure information, usually with higher spatial resolution and contrast, and can identify subtle soft tissue information that CT cannot pay attention to. In actual clinical diagnosis scenarios, doctors usually need to comprehensively consider multiple aspects of human tissue structure, function, metabolism, etc., and a single medical modality image often cannot provide comprehensive diagnostic information.

[0003] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The embodiment of the present invention aims to provide a medical image fusion method, apparatus, device and storage medium based on deep learning, aiming to integrate CT and MRI medical images from different imaging modes to highlight the characteristic information of key tissues and organs.

[0005] As a first aspect of the present invention, in order to achieve the above-mentioned object of the present invention, an embodiment of the present invention provides a medical image fusion method based on deep learning, the method comprising:

[0006] S1. Obtain the original medical image of the target object to be fused, and perform preprocessing to obtain enhanced medical image data;

[0007] S2. extracting depth information from the enhanced medical image data to obtain a depth feature information map containing preliminary feature information;

[0008] S3, extracting multi-scale complementary information from the depth feature information map to obtain a fused feature information map;

[0009] S4. Input the fused feature information graph into a convolutional reconstruction module to obtain a final visualized fused grayscale image.

[0010] Preferably, the extracting of depth information from the enhanced medical image data to obtain a depth feature information map containing preliminary feature information specifically includes:

[0011] Use the convolution module to process the input CT image data I CT and MRI image data I MRI , and get the convolution output feature map:

[0012] X Out =ReLU(BN(AC(X In )))

[0013] Among them, X In Represents the input CT and MRI image tensor information, AC represents adaptive convolution, BN represents batch normalization, and ReLU represents rectified linear unit;

[0014] Using STBs with different numbers of layers, information of different depths is extracted from the convolution output feature map to obtain a deep feature information map:

[0015]

[0016] in, is the convolution output feature map after processing by the convolution module, It represents the deep feature information feature map output by STB processing, LN represents layer normalization, MSA represents multi-head self-attention mechanism operation, and MLP represents multi-layer perceptron.

[0017] Preferably, the extracting multi-scale complementary information from the depth feature information map to obtain a fused feature information map specifically includes:

[0018] Three different dilation factors are used to expand the receptive field of the depth feature information graph to extract features of different scales. The formula of dilated convolution is expressed as:

[0019]

[0020] Among them, RF represents the size of its receptive field, K represents the size of the convolution kernel, d represents the size of the expansion factor, I represents the size of the input, P represents the padding size, S represents the step size, and F out Represents the feature map obtained after dilated convolution;

[0021] The residual pyramid attention network is used to perform inter-scale learning on the feature map obtained after the dilated convolution to obtain a fused feature information map containing information of different scales of medical images of different modalities. The formula is as follows:

[0022] F(d i )=[p 1 (d i )+p 2 (p 1 (d i ))+p 3 (p2 (p 1 (d i )))]+d i

[0023] where d i , i∈[1,3,5] is the feature map obtained after dilation convolution with different dilation factors of {1, 3, 5}, p i , i∈[1,2,3] correspond to the top-down layers of the pyramid attention network, F(d i ), i∈[1,2,3] corresponds to the output of each branch of the pyramid attention network from top to bottom.

[0024] Preferably, the step of inputting the fused feature information graph into a convolution reconstruction module to obtain a final visualized fused grayscale image specifically includes:

[0025] The fusion information feature map is subjected to continuous convolution operation:

[0026] The 64-channel fusion information feature map is converted into a 32-channel feature image through convolution operation;

[0027] Continue to use convolution operation to convert the 32-channel image into 16 channels;

[0028] The convolution operation is continued to be used to convert the 16-channel feature map into a single-channel grayscale image, that is, the visualized fused grayscale image.

[0029] Furthermore, the method further comprises:

[0030] The fused image is optimized based on the original medical image through the mixed loss of the optimizer. The mixed loss of the optimizer can be formulated as:

[0031] L total =L SSIM +L RMI +L PERCEP

[0032] Where L SSIM is the structural loss, L RMI is the area loss, L PERCEP It is a perceived loss;

[0033] L SSIM =α1·(1-SSIM(I F ,I CT )+α2·(1-SSIM(I F ,I MRI )

[0034] where α i, i∈[1,2] is the trade-off factor that controls the balance between the two items;

[0035]

[0036] in, and μ I Respectively represent I F (fused images) and the average intensity of I (CT / MRI images);

[0037] C1 and C2 are used as constants to deal with the current or Close to 0;

[0038] and Respectively represent I F and the variance of I, Indicates I F The covariance of and I;

[0039] L RMI =β1·RMI(I F ,I CT )+β2·RMI(I F ,I MRI )

[0040] Among them, β i , i∈[1,2] is the trade-off factor that controls the balance between the two items;

[0041]

[0042] Among them, λ∈[0,1] is the weight factor, l ce (I F ,I) represents the normal cross entropy loss, B represents the number of images in the mini-batch, represents the lower bound of the mutual information;

[0043] L PERCEP =θ1L CT_percep +θ2L MRI_percep

[0044]

[0045] where θ i , i∈[1,2] is the trade-off factor that controls the balance between the two terms, is the output of the pre-trained VGG16 network at the i-th channel of the n-th layer for the input image I, where C, W, and H are the number of channels, width, and height of the input image, respectively.

[0046] In addition, to achieve the above object, the present invention also proposes a medical image fusion device, the medical image fusion device comprising:

[0047] An input module, used for acquiring an original medical image of a target object to be fused;

[0048] A preprocessing module, used for cropping the original medical image based on a preset cropping strategy to obtain enhanced medical image data;

[0049] A depth information extraction module, used to extract depth information from the enhanced medical image data to obtain a depth feature information map containing preliminary feature information;

[0050] A multi-scale information extraction module, used to extract multi-scale complementary information from the depth feature information map to obtain a fused feature information map;

[0051] The reconstruction module is used to input the fused feature information map into a convolution reconstruction module to obtain a final visualized fused grayscale image.

[0052] Furthermore, the device also includes:

[0053] The optimization output module is used to optimize the fused grayscale image by optimizing through a mixed loss of an optimizer.

[0054] In addition, to achieve the above-mentioned purpose, the present invention also proposes a medical image fusion device, a memory, a processor, and a medical image fusion program stored in the memory and executable on the processor, wherein the medical image fusion program is configured to implement the steps of the medical image fusion method as described above.

[0055] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, on which a deep learning-based medical image fusion program is stored. When the deep learning-based medical image fusion program is executed by a processor, the steps of the medical image fusion method as described above are implemented.

[0056] The embodiment of the present invention obtains the medical image to be fused, and performs data enhancement processing based on a preset image cropping strategy; extracts depth information from the medical image to obtain preliminary feature information; extracts multi-scale complementary information on the basis of fully extracting the preliminary feature information to obtain information of different scales of medical images of different modalities; obtains the final visualized fused image through a convolution reconstruction module, and optimizes the processing of medical images of multiple modalities by constructing a multi-modal medical image fusion model, reduces the difference between the visualized fused image and the original medical image, and obtains the final medical fusion image. In the above manner, medical images of multiple modalities are fused to provide clearer structural and soft tissue medical fusion images, solving the technical problem that a single medical modality image often cannot provide comprehensive diagnostic information. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly describe the embodiments of the present invention and highlight the technical solutions that are different from those in the prior art, the drawings required for use are briefly introduced below. Obviously, the drawings described below only represent some embodiments of the present invention. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0058] Figure 1 It is a flowchart of the medical image fusion method based on deep learning of the present invention;

[0059] Figure 2 A schematic diagram of the sub-step flow of a medical image fusion method based on deep learning provided by an embodiment of the present invention;

[0060] Figure 3 A schematic diagram of the sub-step flow of a medical image fusion method based on deep learning provided by an embodiment of the present invention;

[0061] Figure 4 A schematic diagram of the sub-step flow of a medical image fusion method based on deep learning provided by an embodiment of the present invention;

[0062] Figure 5 The fusion result diagrams of five groups of representative cross-section images of brain patients provided in the embodiment of the present invention based on different methods;

[0063] Figure 6 A schematic diagram showing the numerical results of six objective evaluation indicators provided by an embodiment of the present invention;

[0064] Figure 7 It is a structural block diagram of a medical image fusion device based on deep learning of the present invention;

[0065] Figure 8This is a schematic diagram of the hardware operating environment and equipment structure involved in the medical image fusion based on deep learning of the present invention. DETAILED DESCRIPTION

[0066] The purpose, features and advantages of the present invention will be further described in conjunction with the embodiments and with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0067] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0068] Figure 1 This is a flowchart of an embodiment of the image fusion method based on deep learning of the present invention, see Figure 1 , the image fusion method based on deep learning includes:

[0069] S1. Obtain the original medical image of the target object to be fused, and perform preprocessing to obtain enhanced medical image data;

[0070] The CT and MRI images were preprocessed, including grayscale normalization and image registration, to ensure that they have the same spatial and grayscale ranges, and data augmentation was performed based on a preset image cropping strategy, as follows:

[0071] The acquired 256×256 original medical images are enhanced, and all images in the training set are cropped into block pairs of size 128×128 with a crop size of 16.

[0072] After cropping a pair of paired CT / MRI images (of the same part) with a size of 256×256, 49 pairs of corresponding paired CT / MRI images with a size of 128×128 can be obtained.

[0073] In this step, CT (Computed Tomography) medical images are acquired through a CT scanner, which uses the absorption characteristics of X-rays passing through different tissues in the human body to produce images. During a CT scan, an X-ray source rotates around the patient while a detector records the absorption of the X-rays as they pass through the patient's body. A computer converts this data into cross-sectional images to visualize the internal structures of the human body.

[0074] MRI (magnetic resonance imaging) medical images are acquired using an MRI scanner. MRI uses powerful magnetic fields and harmless radio waves to produce high-resolution images of human tissue. During an MRI scan, the patient is placed in a large magnetic field while a series of radio waves are sent into the patient's body. These radio waves excite hydrogen atoms in the body's tissues to produce signals, which are then captured by a receiver and converted into an image. A computer converts these signals into detailed images showing the tissue structure inside the human body.

[0075] Specifically, Python scripts can be used to set the cropping size of the acquired images, and to crop the CT and MRI images of the same part one by one. For example, first obtain paired (same part) CT / MRI images of 256×256 size, and the cropping size (the step size of the next crop from the previous crop) is 16. It can be cropped seven times horizontally and seven times vertically. Each CT / MRI can be cropped into 49 128×128 images, and 49 pairs of paired medical images are obtained.

[0076] S2. extracting depth information from the enhanced medical image data to obtain a depth feature information map containing preliminary feature information;

[0077] refer to Figure 2 . Figure 2 A schematic flow chart of the S2 sub-step of a deep learning-based medical image fusion method for extracting depth information provided in an embodiment of the present invention. This sub-step obtains the preliminary depth feature information contained in each CT / MRI image and obtains a single-channel 256×256 depth feature information map. This step specifically includes:

[0078] The convolution module (CB) is used to process the input CT image (I CT ) and MRI images (I MRI ),

[0079] refer to Figure 2 , the convolution module consists of three main parts: adaptive convolution (AC), batch normalization (BN) layer, and activation function (LeakyReLU). The corresponding formula of the convolution module is:

[0080]

[0081] in, Represents the input CT and MRI image tensor information. During the training phase, the result after adaptive convolution AC processing is used as the input of BN; the input of ReLU is the output of BN.

[0082] The convolution module (CB) is used to process the input CT image (I CT ) and MRI images (I MRI).

[0083] Specifically, the convolution module consists of three main parts: adaptive convolution (AC), batch normalization (BN) layer, and batch normalized rectified linear unit (ReLU). Adaptive convolution can dynamically adjust the convolution kernel according to the characteristics of the input image, so as to more effectively capture and fuse information from different imaging modalities. The BN layer is designed to improve the stability and efficiency of model training, while the ReLU activation function is used to introduce nonlinearity so that the model can capture more complex features; the processed image information is then processed by the SwinTransformer (STB) block to produce the output feature map. In STB, LN represents layer normalization. MSA stands for the operation of the multi-head self-attention mechanism, which is the core of the Transformer structure and allows the model to consider the relationship between different parts in processing data. MLP stands for multi-layer perceptron, which is usually composed of multiple fully connected layers. In the Transformer structure, it is used to further process the features extracted by the self-attention mechanism. The three are connected through residuals. By adding the input to the output, it can help alleviate the gradient vanishing problem and improve the training efficiency of the model.

[0084] The function of the convolution module is to extract features and perform nonlinear transformation on the input image information. Specifically, each convolution layer can convolve the input image information with the convolution kernel to extract local features in the image. The batch normalization layer can normalize the feature map of each channel, which helps to accelerate the convergence of the network and improve the stability of the model. The activation function introduces nonlinearity, and by setting negative values ​​to zero, the module can learn more complex and rich feature maps.

[0085] Afterwards, SwinTransformer (STB) with different layers is used to extract information of different depths. The specific operation of STB can be expressed as:

[0086]

[0087] in, It is the output feature map after the convolution module is processed, which is used as the input feature of the Swin Transformer (STB) block; It is the deep feature information map output after being processed by the Swin Transformer (STB) block;

[0088] LN stands for layer normalization, MSA stands for the operation of the Multi-head Self-Attention mechanism, which is the core of the Transformer structure and allows the model to consider the relationship between different parts when processing data. MLP stands for Multi-layer Perceptron, which is usually composed of multiple fully connected layers.

[0089] S3, extracting multi-scale complementary information from the depth feature information map to obtain a fused feature information map;

[0090] refer to Figure 3 . Figure 3 A schematic diagram of the sub-step flow chart of a medical image fusion method based on deep learning provided in an embodiment of the present invention.

[0091] First, different dilation factors of {1, 3, 5} are used to perform dilated convolution on the deep feature information graph to expand the receptive field to extract features of different scales. The formula for dilated convolution can be expressed as:

[0092]

[0093] Among them, RF represents the size of its receptive field, K represents the size of the convolution kernel, d represents the size of the expansion factor, I represents the size of the input, P represents the padding size, S represents the step size, and F out Represents the feature map obtained after dilated convolution;

[0094] Then, the residual pyramid attention network is used to perform inter-scale learning on the feature map obtained after the dilated convolution to obtain a fused feature information map containing information of different scales of medical images of different modalities (CT / MRI). The formula is as follows:

[0095] F(d i )=[p 1 (d i )+p 2 (p 1 (d i ))+p 3 (p 2 (p 1 (d i )))]+d i

[0096] Among them, d i (i∈[1,3,5]) is the output feature map after using different expansion factors of {1, 3, 5}, p i (i∈[1,2,3]) correspond to the top-down layers of the pyramid attention network, F(d i)(i∈[1,2,3]) corresponds to the output features of each branch of the pyramid attention network from top to bottom. The output feature map of each layer is added to the feature map of the previous layer by residual connection. Finally, the feature maps of all scales are weighted by element-by-element multiplication to obtain a fused feature information map with 64 channels and a shape of 256×256.

[0097] For the deep feature information map containing preliminary feature information, in order to prevent the loss of local and edge detail information in the image, different expansion factors of {1, 3, 5} are used to extract multi-scale information respectively, instead of directly downsampling. The extracted feature map is input into the residual pyramid attention network for processing.

[0098] The advantage of the residual attention network is that it converges quickly and can cope with the disappearance or explosion of gradients, but it may not be able to extract and learn deep features. Therefore, it is combined with the pyramid attention network to form the residual pyramid attention network. The idea of ​​replacing larger convolution kernels with smaller convolution kernels is adopted. A single 3×3 convolution is used at the top of the pyramid, two 3×3 convolutions are used in the middle to represent a single 5×5 convolution, and three 3×3 convolutions are used at the bottom to represent a single 7×7 convolution. Among them, pooling represents downsampling, upsample represents upsampling, and ReLU represents the activation function. and Represent element-by-element addition and element-by-element multiplication respectively. After the above processing, the fusion information feature map is obtained.

[0099] S4. Input the fused feature information graph into a convolutional reconstruction module to obtain a final visualized fused image.

[0100] refer to Figure 4 , Figure 4 A schematic diagram of the sub-step flow chart of a medical image fusion method based on deep learning provided in an embodiment of the present invention.

[0101] The fused information feature map with 64 channels obtained in the previous step is further subjected to four consecutive convolution operations, where conv represents a 3×3 convolution operation. A 3×3 convolution kernel is used, the step size is 1, and the padding is set to 0 to perform channel compression. The 64-channel feature image is converted into a 32-channel feature image through convolution operations. This method can remove some redundant information while retaining the main features;

[0102] Continue to use the above convolution operation to convert the 32-channel image into 16-channel to further compress the feature map and enhance the model's focus on important information.

[0103] The above convolution operation is continued to be used to convert the 16-channel feature map into a single-channel grayscale image, which is the grayscale image after fusion of CT and MRI obtained by the method.

[0104] This gradual change from 64 to single channel is carefully designed through convolutional layers at each step, aiming to retain key information while reducing the number of channels to obtain the resulting grayscale image. The process of converting 64 channels to 32 channels helps the feature map remove some redundant information while retaining the most critical features for the final result. The conversion of 32 channels to 16 channels further compresses the feature map and strengthens the focus of the feature map on important information. The last step converts the feature map into a single-channel grayscale image, which is the resulting fused grayscale image.

[0105] The final visualized fusion image is obtained through the convolution reconstruction module. Specifically, the convolution reconstruction module contains 4 convolution operations and the LeakyReLU activation function. In each convolution layer, the number of channels of the input feature map is specified by the in_channels parameter, and the number of channels of the output feature map is specified by the out_channels parameter. By adjusting these two parameters, the number of output channels of the convolution operation can be controlled. Its function is to convert the 64-channel feature image into 32 channels, and then further convert it into 16 channels, and finally output a 1-channel grayscale image.

[0106] Therefore, in combination with the description of the above embodiments, the present invention realizes the fusion of multi-modal medical images, so that medical images of multiple modalities (CT and MRI) can be processed.

[0107] In a specific implementation, taking CT and MRI images as an example, in the data preparation stage, 420 pairs of 256×256 pixel CT and MRI images in the mainstream medical image database Harvard may be used.

[0108] For example, 400 pairs of the 420 pairs of brain slice images from 8 disease types were used as training sets (88 pairs of CT / MRI from Acute stroke, 29 pairs of CT / MRI from Hypertensive Encephalopathy; 45 pairs of CT / MRI from Multiple Embolic Infarctions; 42 pairs of CT / MRI from Fatal stroke; 42 pairs of CT / MRI from Metastatic Bronchogenic Carcinoma; 45 pairs of CT / MRI from Meningioma; 55 pairs of CT / MRI from Sarcoma; 54 pairs of CT / MRI from Cerebral Toxoplasmosis).

[0109] The other 20 pairs of images were used as test sets (among them, 2 pairs of CT / MRI were from Acute Stroke; 1 pair of CT / MRI was from Hypertensive Encephalopathy; 3 pairs of CT / MRI were from Fatal Stroke; 3 pairs of CT / MRI were from Metastatic Bronchogenic Carcinoma; 5 pairs of CT / MRI were from Sarcoma; and 6 pairs of CT / MRI were from Cerebral Toxoplasmosis).

[0110] In order to obtain a sufficiently large training set, the original data was enhanced and all images in the training set were cropped into block pairs of size 128×128 with a crop size of 16. In this way, 32,400 pairs of image blocks were obtained for network training.

[0111] In terms of environment configuration, the PyTorch framework can be used to perform network training on a single A30 GPU.

[0112] In terms of model training settings, the Adam optimizer is used in the training phase, the learning rate is initially specified as 0.01, the batch size is 64, and the number of epochs is fixed to 1000. The hyperparameters in the loss function are set to α1=1, α2=1, β1=1, β2=2.5, θ1=0.01, θ2=0.01, aiming to dynamically optimize the gradient and learning rate.

[0113] Each batch of images is optimized by the hybrid loss function in the optimizer, and the fused image is optimized based on the original medical image. The hybrid loss function in the optimizer can be formulated as:

[0114] L total =L SSIM +L RMI +L PERCEP

[0115] Among them, L SSIM is the structural loss, L RMI is the area loss, L PERCEP It is a perceived loss.

[0116] L SSIM =α1·(1-SSIM(I F ,I CT )+α2·(1-SSIM(I F ,I MRI )

[0117] Among them, α i , i∈[1,2] is the trade-off factor that controls the balance between the two terms.

[0118]

[0119] in, and μ I Represent the fused image (I F ) and the average intensity of I (CT / MRI images), C1 and C2 are used as constants to deal with the or When it is close to 0, and Represent the fused image (I F ) and the variance of I(CT / MRI images), σ IFI Represents the fused image (I F ) and the covariance of I(CT / MRI images).

[0120] L RMI =β1·RMI(I F ,I CT )+β2·RMI(I F ,I MRI )

[0121] where β i , i∈[1,2] is the trade-off factor that controls the balance between the two terms.

[0122]

[0123] where λ∈[0,1] is the weight factor, l ce (I F ,I) represents the normal cross entropy loss, B represents the number of images in the mini-batch, Represents the fused image (I F ) and I(CT / MRI images).

[0124] L PERCEP =θ1L CT_percep +θ2L MRI_percep

[0125]

[0126] where θ i (i∈[1,2]) is the trade-off factor that controls the balance between the two terms, is the output of the pre-trained VGG16 network on the i-th channel of the n-th layer of the input image I (CT / MRI image), where C, W, and H are the number of channels, width, and height of the input image, respectively.

[0127] In order to comprehensively evaluate the effectiveness and feasibility of the method proposed in the embodiment of the present invention, the inventors selected medical images (CT and MRI) from the Harvard whole brain dataset to conduct extensive quantitative and qualitative experiments, and compared the proposed method with the prior art methods, such as the following four prior art methods, in terms of evaluation indicators:

[0128] (1)Anatomical-functional image fusion by information of interest in local Laplacian filtering domain.(LLF)

[0129] (2)Medical image fusion with parameter-adaptive pulse coupled neural network in nonsubsampled shearlet transform domain. (NSST-PAPCNN)

[0130] (3)Zero-learning fast medical image fusion.(Zero-Learning)

[0131] (4)Gesenet: A general semantic-guided network with couple maskensemble for medical image fusion. (GeSeNet)

[0132] The evaluation indicators are 6 widely used, including entropy information (EN), feature mutual information (FMI), multi-scale structural similarity index (MS-SSIM), normalized mutual information QMI, gradient-based metric Q G and nonlinear correlation information entropy Q NCIE .

[0133] Among them, entropy information (EN) is an indicator to measure the information content of an image. In the context of image fusion, a higher entropy value means that the fused image contains more information. This indicator evaluates whether the fused image can retain as much information from the source image as possible, which is very critical for the diagnosis of medical images;

[0134] Feature Mutual Information (FMI) is the mutual information between the fused image and the source image, especially at the feature level. A high FMI value indicates that the fused image effectively retains the important features of the source image. Since the goal of medical image fusion is to integrate key features from different modalities, FMI provides an important indicator to measure the degree to which this goal is achieved.

[0135] The multi-scale structural similarity index (MS-SSIM) measures the structural similarity between the fused image and the source image at different scales. A high MS-SSIM value indicates that the fused image is visually consistent with the original image. In medical image fusion, maintaining structural integrity is crucial to ensure the accuracy of diagnostic information.

[0136] Quantized Mutual Information (QMI) is a metric for evaluating image similarity, specifically used to quantify the amount of shared information between two images. Compared with traditional mutual information (MI), QMI is particularly useful when dealing with discrete or quantitative data. It evaluates their similarity by calculating the distribution of common information between images. In the context of image fusion, a high QMI means that the fused image effectively integrates information from the source images, reflecting the success of the fusion process in retaining key information.

[0137] Gradient-based metrics (Q G ,Gradient-based metrics) reflect the clarity and intensity of edges and textures in an image, which is very important for capturing the details and contours of an image. In medical image fusion, edge preservation is the key to evaluating fusion quality because they often contain key information necessary in the diagnosis process.

[0138] Nonlinear correlation information entropy (Q NCIE ,Nonlinear correlation information entropy) measures the information content of nonlinear correlations between images, which is very valuable for capturing and evaluating the complex information dynamics that may be introduced during the fusion process. In medical images, the nonlinear relationship between different imaging modalities may be rich in diagnostic information. Therefore, the introduction of Q NCIE It helps to evaluate how well these relationships are preserved in the fused image. The higher its value means that the more information about the nonlinear relationships between different imaging modalities is preserved.

[0139] Figure 5The qualitative comparison between the method proposed in the embodiment of the present invention and the aforementioned four existing methods on CT and MRI image pairs is demonstrated. In order to make a better intuitive comparison, the red and green 32*32 pixel areas in each image are enlarged as close-ups, and all five methods can obtain relatively satisfactory fusion performance.

[0140] It can be observed that, compared with the traditional method, the image fusion method based on deep learning of the present invention is slightly better in processing the smoothness of the fusion junction. In general, compared with the other four methods, the method proposed in the embodiment of the present invention performs relatively well.

[0141] Specifically, the LLF-based method over-retains the soft tissue information in the MRI image, resulting in scattered artifacts in the fused image. The NSST-PAPCNN-based method reduces the artifacts to a certain extent, but there are still a small number of artifacts that cannot be eliminated. The Zero-Learning-based method can extract structural information from CT images well, but there are some sharp noises on the edges. The GeSeNet-based method has good information preservation capabilities in capturing information in CT and MRI images, but at the same time introduces white artificial artifacts. In general, the method proposed in the embodiment of the present invention exhibits better fusion characteristics in retaining the complementary information of the edge features of the source image.

[0142] The numerical results of the six evaluation indicators are as follows: Figure 6 It can be observed that the method proposed in the embodiment of the present invention is effective in EN, FMI, Q NCIE The highest scores were obtained in all three indicators. The highest entropy value (EN) means that the fused image retains more information from the source images (CT and MRI), which is critical for the diagnosis of medical images. The highest feature mutual information (FMI) value indicates that the fused image effectively retains the important features of the source images (CT and MRI). The optimal nonlinear correlation information entropy (Q NCIE ) means that the fused image generated by the method proposed in this embodiment contains rich nonlinear relationship information between CT and MRI. G , Q MI The three indicators are not much different from the other four methods and have no obvious defects.

[0143] The embodiment of the present invention obtains the medical image to be fused and performs data enhancement based on a preset image cropping strategy; then extracts depth information from the medical image to obtain sufficient depth feature information; extracts multi-scale complementary information based on the fully extracted feature information to obtain fusion feature information of different scales of medical images of different modalities, and finally transforms the fusion feature information graph of medical images from different modalities into a final fused image through a convolution reconstruction module. Through the above method of the present invention, multiple modality medical images are fused to obtain clearer structural and soft tissue medical fusion images, solving the technical problem that a single medical modality image often cannot provide comprehensive diagnostic information.

[0144] Reference Figure 7 , Figure 7 This is a structural block diagram of the first embodiment of the medical image fusion device based on deep learning of the present invention.

[0145] like Figure 7 As shown, the medical image fusion device proposed in the embodiment of the present invention includes:

[0146] The input module 001 is used to obtain the medical image to be fused.

[0147] It should be noted that, in a specific implementation, CT (computed tomography) medical images are acquired through a CT scanner. MRI (magnetic resonance imaging) medical images are acquired through an MRI scanner. The scanning functions of the two instruments may also be integrated into a scanning acquisition device built into the input module, and this embodiment does not impose any limitation on this.

[0148] It should be noted that during the image acquisition process, the images captured by the CT scanner and the MRI scanner are aligned, and the image size is cropped to a specified size (for example, 256×256). For input images that do not meet the specifications, the input module will issue a warning and the user will be required to recheck the image to be fused.

[0149] The preprocessing module 002 is used to adopt a cropping strategy to crop the medical image to obtain enhanced data.

[0150] Furthermore, the preprocessing module 002 runs a Python script with a designed cropping strategy to enhance the original image data. For example, all input 256×256 images are cropped into 128×128 block pairs with a cropping size of 16. In this way, one image is cropped into 49 images so that subsequent modules can perform sufficient image depth feature information learning.

[0151] The depth information extraction module 003 is used to extract enhanced image data to obtain a feature map containing sufficient depth feature information.

[0152] It can be understood that the convolution module (CB) and SwinTransformer are first used to perform preliminary extraction of context information of multiple modality medical image features.

[0153] It should be noted that the convolution module (CB) is first used to process the input CT image (I CT ) and MRI images (I MRI ), and They represent the input and output features of the convolutional module (CB), which consists of three main parts: adaptive convolution (AC), batch normalization (BN) layer, and rectified linear unit (ReLU).

[0154] Next, the processed data is transmitted to the swintransfomer module (STB)

[0155]

[0156] in represents the SwinTransformer (STB) block input features, Represents the output feature map after being processed by the SwinTransformer (STB) block.

[0157] LN stands for layer normalization. MSA stands for the operation of the Multi-head Self-Attention mechanism, which is the core of the Transformer structure and allows the model to consider the relationship between different parts when processing data. MLP stands for Multi-layer Perceptron, which is usually composed of multiple fully connected layers.

[0158] The multi-scale information extraction module 004 is used to extract a feature map containing sufficient depth feature information at multiple scales to obtain a fused feature information map.

[0159] It can be understood that the dilated convolution can effectively represent multi-scale information at a fine-grained level. In order to prevent the loss of local and edge detail information in the image, the multi-scale information extraction module 004 uses different dilation factors of {1, 3, 5} to extract multi-scale information respectively, instead of directly using downsampling. The extracted features are input to the residual pyramid attention network for processing. The implementation process of the dilated convolution can be expressed as:

[0160]

[0161] Among them, RF represents the size of its receptive field, K represents the size of the convolution kernel, d represents the size of the expansion factor; I represents the size of the input, P represents the padding size, S represents the step size, and F out Represents the feature map after dilated convolution.

[0162] It should be noted that using different dilation factors of {1, 3, 5} is a relatively classic dilation convolution method, and the dilation convolution factor of the multi-scale information extraction module 004 can also be readjusted according to the size of the medical image.

[0163] The reconstruction module 005 is used to transform the fused feature information graph into a visual fused image.

[0164] It can be understood that after the above processing, the embodiment obtains a feature map with a channel number of 64. In order to obtain a visual fused image, it is further input into the reconstruction module 005 and four consecutive layers of convolution operations are performed to change the number of channels. This gradual change from 64 to a single channel is carefully designed through the convolution layer at each step, aiming to reduce the number of channels while retaining key information to obtain a visual fused image. The process of converting 64 channels to 32 channels helps remove some redundant information while retaining the most critical features for the final result. The conversion of 32 channels to 16 channels further compresses the feature map and strengthens the focus on important information. The last step is to convert the feature map into a visual fused image.

[0165] It should be noted that the use of four layers of channel convolution is set based on the input original image specifications of the current embodiment. The embodiment does not limit the number of layers of channel convolution used in practice.

[0166] Preferably, the device further comprises:

[0167] The optimization output module 006 is used to reduce the difference between the visualized fused image and the original medical image to obtain the final medical fused image.

[0168] It can be understood that after the above processing, the embodiment obtains a preliminary visual fusion image. In order to synthesize multiple images into a single image while retaining the information and edge detail texture in the input image as much as possible, the optimization output module 006 constructs a hybrid loss to guide the fusion process. The goal of the model is to minimize this hybrid loss, thereby generating a fusion image with high perceptual quality while maintaining the key features of the original image. The optimization output module 006 adopts the back propagation algorithm to incorporate this loss function, and gradually reduces the loss value through the preliminary visual fusion image and the original CT image and MRI image calculation, and finally obtains the optimized visual fusion image.

[0169] It should be noted that the mixed loss consists of three parts: L total =L SSIM+L RMI +L PERCEP , where L SSIM is the structural loss, L RMI is the area loss, L PERCEP It is a perceptual loss. According to the types of different medical modality images to be fused, the optimization output module 006 may also be added or reduced in part of its loss composition, which is not limited in this embodiment.

[0170] This embodiment obtains the medical image to be fused and performs data enhancement processing based on the pre-processed image cropping strategy; performs a module for extracting depth information on the medical image to obtain sufficient feature information; performs a module for extracting multi-scale complementary information on the basis of fully extracting feature information to obtain information of different scales of medical images of different modalities; converts the fused feature information graphs of medical images from different modalities into the final fused image through a convolution reconstruction module; and constructs a multi-modal medical image fusion model to process medical images of multiple modalities. In the above manner, medical images of multiple modalities are fused to provide clearer structural and soft tissue medical fusion images, solving the technical problem that a single medical modality image often cannot provide comprehensive diagnostic information.

[0171] Reference Figure 8 , Figure is a schematic diagram of the hardware operating environment and equipment structure involved in the medical image fusion based on deep learning of the present invention

[0172] like Figure 8As shown, the medical image device may include: a CT image scanner 2001, a processor 2002, an image acquisition device 2003, an image data receiving device 2004, a data bus 2005, an MRI image scanner 2006, a memory 2007, and a network device 2008. Among them, the data bus 2005 is used to connect these components for data transmission and communication, the CT image scanner 2001 may be a multi-slice spiral CT (MDCT), the processor 2002 may be a central processing unit (CPU), the image acquisition device 2003 may register the CT image and the MRI image, the image data receiving device 2004 is used to obtain the registered medical image data, the MRI image scanner 2006 may be an ultra-high field MRI scanner, the memory 2007 may be a flash memory, a tape memory, or a cloud storage, and the network device 2008 includes a series of software and hardware such as a router, a switch, a firewall, and a load balancer. The CT image scanner 2001, the MRI image scanner 2006 and the image acquisition device 2003 are responsible for acquiring the medical image data of the patient. These data are then transmitted to the image data receiving device 2004 and transmitted to the processor 2002 via the data bus 2005 for further processing and analysis. The memory 2007 is used to store the medical image data and processing results to ensure the security and reliability of the data. The network device 2008 enables the medical image data to be remotely transmitted and shared via the network.

[0173] Technicians can understand that Figure 8 The structure shown in is not the only way to construct a medical image fusion device, it is just an example. In fact, the medical image fusion device may include more or fewer components, or some components may be combined, or even different component arrangements may be adopted. In other words, the diagram only shows a possible configuration, rather than a specific limitation on the design of the device. . The above serial numbers are only for the convenience of description and do not mean that one embodiment is better or worse than other embodiments. The above description provides a specific implementation of the present invention, but the scope of protection of the present invention is not limited thereto. Anyone familiar with the technical field can easily think of various equivalent modifications or alternatives within the technical scope disclosed by the present invention, and these modifications or alternatives should be deemed to be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be based on the scope determined by the claims.

Claims

1. A medical image fusion method based on deep learning, characterized in that: The method comprises: S1. Acquire the original medical image of the target object to be fused, wherein the original medical image includes a CT image and an MRI image, and perform preprocessing to obtain enhanced medical image data; S2. Extracting depth information from the enhanced medical image data to obtain a depth feature information map containing preliminary feature information, specifically including: Use the convolution module to process the input CT image data I CT and MRI image data I MRI , and get the convolution output feature map: X Out =ReLU(BN(AC(X In ))) Among them, X In Represents the input CT and MRI image tensor information, AC represents adaptive convolution, BN represents batch normalization, and ReLU represents rectified linear unit; Using STBs with different numbers of layers, information of different depths is extracted from the convolution output feature map to obtain a deep feature information map: in, is the convolution output feature map after processing by the convolution module, It represents the feature map of deep feature information output by STB processing, LN represents layer normalization, MSA represents multi-head self-attention mechanism operation, and MLP represents multi-layer perceptron; S3, extracting multi-scale complementary information from the depth feature information map to obtain a fused feature information map, wherein the extracting multi-scale complementary information from the depth feature information map to obtain a fused feature information map specifically includes: Three different dilation factors are used to expand the receptive field of the depth feature information graph to extract features of different scales. The formula of dilated convolution is expressed as: Among them, RF represents the size of its receptive field, K represents the size of the convolution kernel, d represents the size of the expansion factor, I represents the size of the input, P represents the padding size, S represents the step size, and F out Represents the feature map obtained after dilated convolution; The residual pyramid attention network is used to perform inter-scale learning on the feature map obtained after the dilated convolution to obtain a fused feature information map containing information of different scales of medical images of different modalities. The formula is as follows: F(d i )=[p 1 (d i )+p 2 (p 1 (d i ))+p 3 (p 2 (p 1 (d i )))]+d i where d i , i∈[1,3,5] is the feature map obtained after dilation convolution with different dilation factors of {1, 3, 5}, p i , i∈[1,2,3] correspond to the top-down layers of the pyramid attention network, F(d i ), i∈[1,2,3] corresponds to the output of each branch of the pyramid attention network from top to bottom; S4. Input the fused feature information graph into a convolutional reconstruction module to obtain a final visualized fused grayscale image.

2. The medical image fusion method based on deep learning according to claim 1, characterized in that: The step of inputting the fused feature information graph into a convolution reconstruction module to obtain a final visualized fused grayscale image specifically includes: The fused feature information graph is subjected to continuous convolution operation: The 64-channel fusion information feature map is converted into a 32-channel feature image through convolution operation; Continue to use convolution operation to convert the 32-channel image into 16 channels; The convolution operation is continued to be used to convert the 16-channel feature map into a single-channel grayscale image, that is, the visualized fused grayscale image.

3. The medical image fusion method based on deep learning according to claim 1, characterized in that: The method further comprises: The fused grayscale image is optimized by optimizing the mixed loss of the optimizer, and the mixed loss formula of the optimizer is: L total =L SSIM +L RMI +L PERCEP Among them, L SSIM is the structural loss, L RMI is the area loss, L PERCEP It is a perceived loss; L SSIM =α1·(1-SSIM(I F ,I CT )+α2·(1-SSIM(I F ,I MRI ) where α i , i∈[1,2] is the trade-off factor that controls the balance between the two items; in, and μ I Represent the fused image I F and the average intensity of CT / MRI image I, C1 and C2 are used as constants to deal with the or When it is close to 0, and Represents I F and the variance of I, Indicates I F The covariance of and I; L RMI =β1·RMI(I F ,I CT )+β2·RMI(I F ,I MRI ) where β i , i∈[1,2] is the trade-off factor that controls the balance between the two items; where λ∈[0,1] is the weight factor, represents the normal cross entropy loss, B represents the number of images in the mini-batch, represents the lower bound of mutual information; L PERCEP =θ1L CT_percep +θ2L MRI_percep where θ i , i∈[1,2] is the trade-off factor that controls the balance between the two terms, is the output of the pre-trained VGG16 network at the i-th channel of the n-th layer for the input image I, where C, W, and H are the number of channels, width, and height of the input image, respectively.

4. A medical image fusion device based on deep learning, characterized in that: The medical image fusion device comprises: An input module, used for acquiring an original medical image of a target object to be fused; A preprocessing module, used for cropping the original medical image based on a preset cropping strategy to obtain enhanced medical image data; The depth information extraction module is used to extract the depth information of the enhanced medical image data to obtain a depth feature information map containing preliminary feature information, specifically including: Use the convolution module to process the input CT image data I CT and MRI image data I MRI , and get the convolution output feature map: X Out =ReLU(BN(AC(X In ))) Among them, X In Represents the input CT and MRI image tensor information, AC represents adaptive convolution, BN represents batch normalization, and ReLU represents rectified linear unit; Using STBs with different numbers of layers, information of different depths is extracted from the convolution output feature map to obtain a deep feature information map: in, is the convolution output feature map after processing by the convolution module, It represents the feature map of deep feature information output by STB processing, LN represents layer normalization, MSA represents multi-head self-attention mechanism operation, and MLP represents multi-layer perceptron; The multi-scale information extraction module is used to extract multi-scale complementary information from the depth feature information map to obtain a fused feature information map. The multi-scale complementary information extraction from the depth feature information map to obtain a fused feature information map specifically includes: Three different dilation factors are used to expand the receptive field of the depth feature information graph to extract features of different scales. The formula of dilated convolution is expressed as: Among them, RF represents the size of its receptive field, K represents the size of the convolution kernel, d represents the size of the expansion factor, I represents the size of the input, P represents the padding size, S represents the step size, and F out Represents the feature map obtained after dilated convolution; The residual pyramid attention network is used to perform inter-scale learning on the feature map obtained after the dilated convolution to obtain a fused feature information map containing information of different scales of medical images of different modalities. The formula is as follows: F(d i )=[p 1 (d i )+p 2 (p 1 (d i ))+p 3 (p 2 (p 1 (d i )))]+d i where d i , i∈[1,3,5] is the feature map obtained after dilation convolution with different dilation factors of {1, 3, 5}, p i , i∈[1,2,3] correspond to the top-down layers of the pyramid attention network, F(d i ), i∈[1,2,3] corresponds to the output of each branch of the pyramid attention network from top to bottom; The reconstruction module is used to input the fused feature information map into a convolution reconstruction module to obtain a final visualized fused grayscale image.

5. A medical image fusion device based on deep learning as described in claim 4, characterized in that: The device also includes: The optimization output module is used to optimize the fused grayscale image through an optimizer.

6. A medical image fusion device, characterized in that: The device comprises: a memory, a processor, and a medical image fusion program stored in the memory and executable on the processor, wherein the medical image fusion program is configured to implement the steps of the medical image fusion method according to any one of claims 1 to 3.

Images