Fusion model and method of cerebral arterial thrombosis multi-sequence magnetic resonance images
In the multi-sequence MRI image fusion of ischemic stroke, the progressive learning network guided by expert knowledge solves the problems of insufficient information expression and low operation efficiency, and achieves efficient image fusion and diagnostic efficiency improvement, achieving subjective and objective evaluation unified.
Patent Information
- Application Number
- CN202411009523.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-22
AI Technical Summary
The prior art has insufficient information expression, low operation efficiency and insufficient exploration of loss functions in the multi-sequence MRI image fusion of ischemic stroke, resulting in inconsistent subjective and objective evaluation.
A progressive learning network (EKGFusion) based on expert knowledge guidance is designed, including lesion detection module, progressive feature guidance module and edge detection module. It adopts local pixel comparison and self-attention mechanism, combined with Scharr edge detection algorithm, and uses peak signal. The noise ratio and structural mean square error loss functions are used for image fusion.
It has achieved efficient fusion of multi-sequence MRI images of ischemic stroke, improved diagnostic efficiency, improved model computing efficiency and visual attractiveness, achieved the unity of subjective and objective evaluation, and significantly improved the work efficiency of clinical diagnosis.
Smart Images

Figure CN119006501A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of medical images, and in particular relates to a fusion model and method of multi-sequence magnetic resonance images of ischemic stroke. Background Art
[0002] Medical image fusion aims to fuse medical image information of different modalities, so that a single fused image can fully characterize the lesion area, speed up the diagnosis and treatment process, and relieve the work pressure of doctors. In actual clinical diagnosis, different imaging devices and technologies have their own modality advantages and representation ranges. When facing a certain type of disease, radiologists often need to combine multiple imaging technologies and repeatedly observe images of different modalities of the same part to meet the requirements of subjective diagnosis of the disease. Ischemic stroke has become an important issue affecting human health due to its high morbidity and mortality. Accurately locating and evaluating ischemic lesions and their surrounding areas is very important for the clinical diagnosis and treatment of ischemic stroke patients. Therefore, combining the complementary information of different modalities and sequence images can help doctors develop individualized ischemic stroke treatment plans to meet actual clinical needs and application value.
[0003] In recent years, image fusion methods based on deep neural networks have been widely used. However, when image fusion technology is applied to diseases such as ischemic stroke, it is easy to face the dilemma of inconsistent subjective and objective evaluation. Therefore, to design an efficient medical image fusion model for multi-sequence MRI image fusion of ischemic stroke, it is necessary to focus on solving the following three challenges:
[0004] 1) Accurately locating and evaluating ischemic lesions and their surrounding areas is crucial to promoting the personalized diagnosis and treatment of ischemic stroke patients. However, due to insufficient motivation for fusion tasks, image fusion work in the context of ischemic stroke is still in a blank stage, which is a problem to be solved.
[0005] 2) Most of the existing deep learning-based fusion methods improve fusion performance by increasing the depth and width of the fusion model, resulting in more redundant calculations and lower fusion model operation efficiency. Therefore, how to maintain fusion performance while improving operation efficiency is a challenge that needs to be solved.
[0006] 3) Most researchers tend to use conventional loss functions, such as structural information, mutual information, gradient, space and other information gaps. Researchers lack in-depth exploration of the important role of loss functions from different angles in improving fusion performance, and still lack in-depth exploration and understanding.
[0007] In view of this, the present invention proposes a new ischemic stroke multi-sequence magnetic resonance image fusion model and method based on expert knowledge guided progressive learning network, which effectively overcomes the deficiency of expressing information under single modality images. Summary of the invention
[0008] The purpose of the present invention is to provide a fusion model of multi-sequence magnetic resonance images of ischemic stroke, which effectively overcomes the deficiency of expressing information under single modality images and can be used to help doctors improve diagnostic efficiency and promote precise preoperative navigation.
[0009] In order to achieve the above purpose, the technical solution adopted is:
[0010] A fusion model of multi-sequence magnetic resonance images of ischemic stroke, comprising: a lesion detection module, two progressive feature guidance modules, an edge detection module, and a loss function module;
[0011] The lesion detection module is used to focus on the high signal lesion area in the DWI image;
[0012] The progressive feature guiding module is used to guide the modal image branch to perform feature extraction;
[0013] The edge detection module is used to extract the edge features of the lesion.
[0014] Furthermore, the lesion detection module: combines the median filter and binarization, and uses local pixel contrast to achieve adaptive separation of ischemic lesions and background areas, thereby quickly focusing on high-signal lesion areas in the DWI image.
[0015] Furthermore, the progressive feature guidance module includes: a Restormer module and a CNN module;
[0016] The CNN module is used to obtain cross-modal shallow features;
[0017] The Restormer module extracts global features from high-resolution input images by applying a self-attention mechanism on the feature dimension.
[0018] Furthermore, the edge detection module combines a multi-layer convolution structure with the Scharr edge detection algorithm to generate information-rich edge features of the ischemic core area.
[0019] Furthermore, the edge detection module extracts image features by constructing multiple groups of 3*3 convolutional layers Conv, and after using concatenate and addition operations, uses 1×1 convolution to reduce the feature quantity to a single channel map, and obtains the output features under multiple groups of convolutional layers, using the following formula:
[0020]
[0021]
[0022] At this time, Conv1 is a 5*5 convolution layer and Relu activation function, Conv2 is a 3*3 convolution layer and Relu activation function, Cat is a concatenate operation, and Conv is a 1*1 convolution operation. represents the output features of the i-th layer, ε1(.), ε2(.), ε3(.), ε4(.) and ε(.) are set to the same operation;
[0023] The Scharr gradient filter is used to learn the gradient information in the horizontal and vertical directions, and the gradient information is fed into the 1×1 convolution layer using the following formula:
[0024]
[0025] in, Represents ε i The output features of (.), Scharr_xy represents the Scharr gradient filter operation.
[0026] Furthermore, the loss function module adopts the following formula:
[0027] L total (U,D,F)=αL psnr (U,F)+βL psnr (U,D)+γL SM (U,F)+δL SM (U,D);
[0028] Among them, L total (U, D, F) represents the total loss function, L psnr (x,y) represents the peak signal-to-noise ratio loss function, L SM (x,y) represents the structural loss and mean square error loss.
[0029] Furthermore, the peak signal-to-noise ratio loss function adopts the following formula:
[0030]
[0031] Among them, MSE represents mean square error;
[0032] The structural loss and mean square error loss are expressed as follows:
[0033] L SM (U,D)=L SSIM (U,D)+L MSE (U,D);
[0034] L SM (U,F)=L SSIM (U,F)+LMSE (U,F);
[0035] L SSIM (x,y)=1-SSIM(x,y);
[0036] L MSE (x,y)=MSE(x,y);
[0037] Among them, SSIM(x,y) is a structural similarity measure commonly used in the statistical fusion process to calculate the information loss and image distortion, and calculates the correlation loss, brightness and contrast distortion. MSE(x,y) represents the mean square error.
[0038] Another object of the present invention is to provide a method for fusing multi-sequence magnetic resonance images of ischemic stroke, which effectively overcomes the deficiency of expressing information under a single modality image.
[0039] In order to achieve the above purpose, the technical solution adopted is:
[0040] A method for fusing multi-sequence magnetic resonance images of ischemic stroke comprises: preprocessing the multi-sequence magnetic resonance images of ischemic stroke and inputting them into the above-mentioned fusion model for fusion;
[0041] Furthermore, the ischemic stroke multi-sequence magnetic resonance images include DWI, FLAIR, and MASK images;
[0042] The DWI and FLAIR images are subjected to global and local feature extraction respectively, the DWI image is input into the lesion detection module to generate a MASK image, and then the MASK image is input into the edge detection module for feature extraction;
[0043] The features output by the edge detection module are then used to guide the progressive feature guidance modules of the DWI and FLAIR branches to perform feature extraction.
[0044] Compared with the prior art, the present invention has the following beneficial effects:
[0045] The technical solution of the present invention can be used for multi-sequence MRI image fusion of ischemic stroke. Specifically, EKGFusion realizes adaptive separation of ischemic lesions and background areas by means of local pixel contrast, thereby simulating the process of experts focusing on the disease. Secondly, the lesion area is used to perform progressive feature guidance on DWI and FLAIR images respectively. Under the prompt of the lesion area, EKGFusion can actively guide the feature extraction module to focus on ischemic lesions and surrounding areas. Therefore, thanks to the efficient division of image areas and the small scale of feature extraction modules, redundant calculations are reduced and the operational efficiency of the model is improved. Thirdly, the present invention uses error sensitivity values to autonomously constrain the point-by-point pixel differences between the source image and the fused image, thereby enhancing the subjective visual appeal of the fusion model.
[0046] Specific:
[0047] 1. The technical solution of the present invention is a progressive learning network (EKGFusion) guided by expert knowledge. Different from the previous deep learning fusion methods, this model simulates the injection of professional knowledge of radiology experts to achieve adaptive separation of lesion areas and background areas, and automatically guides the fusion module to progressively focus on ischemic lesions and surrounding areas. The introduction of this model will help further promote the application of multimodal image fusion technology in the medical field.
[0048] 2. The technical solution of the present invention designs an objective function from the perspective of peak signal-to-noise ratio by effectively training EKGFusion, and uses error sensitivity values to constrain and optimize the point-by-point pixel difference between the source image and the fused image, thereby improving the visual appeal of the fused image.
[0049] 3. The technical scheme of the present invention constructs a new ischemic stroke multi-sequence MRI image dataset Ischemic Stroke 2024. Compared with the mainstream method, the comprehensive 8 objective indicators and the subjective scores of 9 experts have highly recognized the EKGFusion fusion results. It not only achieves SOTA in objective indicators, but also has more interpretable subjective fusion effects. It realizes the unification of subjective and objective evaluation of auxiliary diagnosis of ischemic stroke for the first time, effectively solves the clinical problems in the fusion of multi-sequence magnetic resonance images of ischemic stroke, and further promotes the development and application of multimodal image fusion technology in the medical field. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 A progressively guided demonstration of image fusion tasks guided by expert knowledge;
[0051] Figure 2 The overall architecture of the model;
[0052] Figure 3The results of the IschemicStroke 2024 dataset are shown in Figure 1. From left to right, they are: (1) source image / fused image; (2) a magnified image of a local area placed on the original image; (3) a magnified image of a local area; (4) grayscale histogram of the source image / fused image; (5) grayscale histogram of the magnified image of a local area;
[0053] Figure 4 The results of the IschemicStroke 2024 dataset are shown in Figure 1. From left to right, they are: (1) source image / fused image; (2) a magnified image of a local area placed on the original image; (3) a magnified image of a local area; (4) grayscale histogram of the source image / fused image; (5) grayscale histogram of the magnified image of a local area;
[0054] Figure 5 It is a display of 128 pairs of DWI-FLAIR quantitative indicators;
[0055] Figure 6 This is the scoring result of 9 radiologists;
[0056] Figure 7 Comparative display of quantitative experimental results of ablation of each module;
[0057] Figure 8 Comparison of quantitative experimental results for loss function coefficient ablation. DETAILED DESCRIPTION
[0058] In order to further illustrate the fusion model and method of multi-sequence magnetic resonance images of ischemic stroke of the present invention and achieve the intended purpose of the invention, the fusion model and method of multi-sequence magnetic resonance images of ischemic stroke proposed in accordance with the present invention, its specific implementation method, structure, characteristics and efficacy are described in detail below in combination with the preferred embodiments. In the following description, different "one embodiment" or "embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.
[0059] Before elaborating in detail a fusion model and method of multi-sequence magnetic resonance images of ischemic stroke of the present invention, it is necessary to further explain the relevant background mentioned in the present invention to achieve better results.
[0060] 1. Traditional image fusion
[0061] Traditional medical image fusion methods are mainly divided into multi-scale transform (MST) and sparse representation (SR) based methods. Compared with single-scale image fusion methods, MST-based image fusion methods include multi-scale features and different fusion rules to improve fusion performance. Among them, wavelet transform (WT), Laplacian pyramid (LP) transform, gradient pyramid transform, non-subsampled contourlet transform (NSCT) and non-subsampled shearlet transform (NSST) all decompose the source image in different transform domains to achieve feature extraction at different scales. In recent years, MST-based methods have received increasing attention. For example, Yin et al. proposed a new strategy for medical image fusion based on NSST decomposition; Du et al. designed a Laplacian pyramid fusion method combining multiple features; Wang et al. developed a fusion technology combining non-subsampled contourlet transform and CNN. Pan et al. proposed an image fusion method based on saliency measurement improvement and local structure similarity correction for medical image tasks.
[0062] In addition, image fusion methods based on sparse representation (SR) use training algorithms to learn overcomplete dictionaries from source images, design optimization algorithms to obtain sparse coefficients, and then select appropriate fusion rules to generate fused sparse representation coefficients, and finally act on the overcomplete dictionary and fusion coefficients to reconstruct the fused image. It is worth noting that the quality of the overcomplete dictionary and the sparsity of the sparse coefficients will affect the quality of the fused image. In recent years, more SR-based fusion methods have received increasing attention, including convolutional sparsity based morphological component analysis (CS-MCA), convolutional sparse representation (CSR), tensor sparse representation (TSR), and complex sparse representation (ComSR).
[0063] Traditional image fusion methods usually follow the process of decomposition, fusion, and reconstruction. First, a specific image representation model is used to decompose the source image. Then, a predefined fusion strategy is used to merge the coefficients, such as addition, maximum value method, local gradient method, local variance, and weighted average method. Finally, the image is reconstructed through inverse transformation to generate the final fused image. It is worth noting that the above steps and components are often designed manually, which limits the improvement of fusion performance to a certain extent. In addition, using the same transformation and fusion strategy for different modal input source images is a common problem of most traditional methods, which can easily lead to the loss of key modal information and ignore the differences in feature information between modalities.
[0064] 2. Deep learning image fusion method
[0065] At present, image fusion methods based on deep learning are developing rapidly. This type of method aims to improve the limitations of traditional methods. A pioneering work was proposed by Lahoud et al., who used a pre-trained convolutional neural network for feature extraction and used a Gaussian filter to fuse these features. However, this method cannot fully capture the salient features of different types of medical images. Subsequently, many researchers have constructed various fusion architectures through carefully designed model structures and loss functions to achieve the integration of feature extraction, image fusion and image reconstruction. These fusion architectures can significantly improve the fusion performance.
[0066] Specifically, Tang et al. designed a multi-scale fusion architecture, based on the adaptive convolution and adaptive Transformer structures, and constructed a loss function to achieve end-to-end multimodal medical image fusion. Subsequently, Tang et al. successively designed the YDTR, DATFuse and TCCFusion model structures to achieve infrared and visible light image fusion, trying to improve the performance of the fusion model, focusing on the influence of the loss function on the fusion model, and innovatively applied the SF and RMI loss functions to the image fusion task. Wang et al. fully considered the feature interactions and intrinsic characteristics of different scales of different modal images, and then proposed a cross-scale iterative attention adversarial fusion network (CrossFuse) for infrared and visible light image fusion. Zhao et al. paid attention to the training instability and lack of interpretability of the GAN-based generation method, and proposed a denoising diffusion probability model (DDPM), which was applied to infrared-visible light image fusion and medical image fusion respectively. Zhao et al. used an encoder to decompose the multimodal source image into background feature maps and detail feature maps, and used a decoder to generate a fused image. Zhao et al. paid attention to the importance of modality-specific and modality-shared features in multimodal source images, and creatively proposed a correlation-driven feature decomposition and fusion (CDDFuse) network for infrared-visible image fusion and medical image fusion tasks, and used it in downstream tasks to verify its good performance. Li et al. proposed an edge enhancement module and a semantic module to form a flexible semantic-guided architecture (GeSeNet) to improve fusion performance while ensuring time efficiency.
[0067] In recent years, image fusion methods based on deep neural networks have been widely used. However, when image fusion technology is applied to diseases such as ischemic stroke, it is easy to face the dilemma of inconsistent subjective and objective evaluation. Therefore, in order to design an efficient medical image fusion model for multi-sequence MRI image fusion of ischemic stroke, it is necessary to focus on solving three challenges in the background technology.
[0068] In order to alleviate the above limitations, the present invention proposes a fusion model and method for multi-sequence magnetic resonance images of ischemic stroke, which is a progressive learning network (EKGFusion) guided by simulated expert knowledge, used for multi-sequence MRI image fusion of ischemic stroke.
[0069] After understanding the relevant background mentioned in the present invention, the fusion model and method of ischemic stroke multi-sequence magnetic resonance images of the present invention will be further introduced in detail in combination with specific embodiments:
[0070] The fusion model and method of multi-sequence magnetic resonance images of ischemic stroke described in the present invention are a fusion model and method for multi-sequence magnetic resonance images of ischemic stroke based on simulated expert knowledge to guide the progressive learning network. Previous medical image fusion methods mainly pursue fusion images with good visual effects by designing complex fusion models. However, the above method does not take into account the importance of expert professional knowledge. In order to better conduct pathological analysis and scientific research, the present invention injects simulated expert knowledge into the ischemic stroke image fusion model, and proposes a progressive learning network guided by simulated expert knowledge, thereby improving the interpretability of the medical image fusion model. The present invention takes into account that the ischemic core high signal area is often small in size, so too many stacked layers or too large convolutional network modules may cause information loss, thereby affecting the fusion performance. Therefore, the present invention designs a lightweight and efficient ischemic stroke image fusion network, which injects expert knowledge and progressively guides the feature extraction module to focus on the modal information of the ischemic core and surrounding areas. The technical solution of the present invention is:
[0071] A fusion model of multi-sequence magnetic resonance images of ischemic stroke, comprising: a lesion detection module (object detector), two progressive feature guidance modules, an edge detection module, and a loss function module;
[0072] The lesion detection module is used to focus on the high signal lesion area in the DWI image;
[0073] The progressive feature guiding module is used to guide the modal image branch to perform feature extraction;
[0074] The edge detection module is used to extract the edge features of the lesion.
[0075] Preferably, the lesion detection module: combines a median filter and a binarization filter, and uses local pixel contrast to achieve adaptive separation of ischemic lesions and background areas, thereby quickly focusing on high-signal lesion areas in the DWI image.
[0076] Preferably, the progressive feature guidance module comprises: a Restormer module and a CNN module;
[0077] The CNN module is used to obtain cross-modal shallow features;
[0078] The Restormer module extracts global features from high-resolution input images by applying a self-attention mechanism on the feature dimension.
[0079] Preferably, the edge detection module combines a multi-layer convolution structure with a Scharr edge detection algorithm to generate information-rich edge features of the ischemic core area.
[0080] Further preferably, the edge detection module extracts image features by constructing multiple groups of 3*3 convolutional layers Conv, uses concatenate and addition operations, and then uses 1×1 convolution to reduce the feature quantity to a single channel map, and obtains output features under multiple groups of convolutional layers, using the following formula:
[0081]
[0082] At this time, Conv1 is a 5*5 convolution layer and Relu activation function, Conv2 is a 3*3 convolution layer and Relu activation function, Cat is a concatenate operation, and Conv is a 1*1 convolution operation. represents the output features of the i-th layer, ε1(.), ε2(.), ε3(.), ε4(.) and ε(.) are set to the same operation;
[0083] The Scharr gradient filter is used to learn the gradient information in the horizontal and vertical directions, and the gradient information is fed into the 1×1 convolution layer using the following formula:
[0084]
[0085] in, Represents ε i The output features of (.), Scharr_xy represents the Scharr gradient filter operation.
[0086] Preferably, the loss function module is used to train the expert knowledge to guide the progressive learning network, driving the network to focus on edge details and structural information. The following formula is used:
[0087] L total (U,D,F)=αL psnr (U,F)+βL psnr (U,D)+γL SM (U,F)+δL SM(U,D);
[0088] Among them, L total (U, D, F) represents the total loss function, L psnr (x,y) represents the peak signal-to-noise ratio loss function, L SM (x,y) represents the structural loss and mean square error loss.
[0089] Further preferably, the peak signal-to-noise ratio loss function adopts the following formula:
[0090]
[0091] Among them, MSE represents mean square error;
[0092] The structural loss and mean square error loss are expressed as follows:
[0093] L SM (U,D)=L SSIM (U,D)+L MSE (U,D);
[0094] L SM (U,F)=L SSIM (U,F)+L MSE (U,F);
[0095] L SSIM (x,y)=1-SSIM(x,y);
[0096] L MSE (x,y)=MSE(x,u);
[0097] Among them, SSIM(x,y) is a structural similarity measure commonly used in the statistical fusion process to calculate the information loss and image distortion, and calculates the correlation loss, brightness and contrast distortion. MSE(x,y) represents the mean square error.
[0098] A method for fusing multi-sequence magnetic resonance images of ischemic stroke comprises: preprocessing the multi-sequence magnetic resonance images of ischemic stroke and inputting them into the above-mentioned fusion model for fusion;
[0099] Preferably, the ischemic stroke multi-sequence magnetic resonance images include DWI, FLAIR, and MASK images;
[0100] The DWI and FLAIR images are subjected to global and local feature extraction respectively, the DWI image is input into the lesion detection module to generate a MASK image, and then the MASK image is input into the edge detection module for feature extraction;
[0101] The features output by the edge detection module are then used to guide the progressive feature guidance modules of the DWI and FLAIR branches to perform feature extraction.
[0102] The fusion model and method of multi-sequence magnetic resonance images of ischemic stroke described in the present invention are a multi-sequence magnetic resonance image fusion model and method (EKGFusion) for ischemic stroke based on expert knowledge guided progressive learning network, which is used for multi-sequence MRI image fusion of ischemic stroke. Specifically, EKGFusion realizes adaptive separation of ischemic lesions and background areas by local pixel contrast, thereby simulating the process of experts focusing on the disease. Further, the lesion area is used to perform progressive feature guidance on DWI and FLAIR images respectively. Under the prompt of the lesion area, EKGFusion can actively guide the feature extraction module to focus on ischemic lesions and surrounding areas. Thanks to the efficient division of image areas and the small scale of feature extraction modules, redundant calculations are reduced and the operational efficiency of the model is improved. Secondly, in order to explore the importance of loss functions at different angles, the present invention uses error sensitivity values to autonomously constrain the point-by-point pixel differences between the source image and the fused image to enhance the subjective visual appeal of the fusion model. Figure 1 The motivation of simulating ischemic stroke image fusion guided by expert knowledge and the brief overall framework of EKGFusion are described. The specific embodiments are as follows:
[0103] Example 1.
[0104] The following three parts will be described. The embodiment of the present invention first describes the overall framework EKGFusion (A). Secondly, the embodiment of the present invention introduces the progressive feature guidance (B) and the edge detection module (C). Finally, the embodiment of the present invention describes the optimal objective function recommended for construction, namely the loss function (D) and the experimental results analysis (E).
[0105] The specific steps are as follows:
[0106] AOverall Framework:
[0107] Previous medical image fusion methods mainly pursue fusion images with good visual effects by designing complex fusion models. However, the above methods do not take into account the importance of expert expertise. In order to better conduct pathological analysis and scientific research, the present invention injects expert knowledge into the ischemic stroke image fusion model, and proposes a progressive learning network guided by simulated expert knowledge to improve the interpretability of the medical image fusion model. Considering that the ischemic core high signal area is often small in size, too many stacked layers or too large convolutional network modules may lead to information loss, thereby affecting the fusion performance. Therefore, the present invention designs a lightweight and efficient ischemic stroke image fusion network, which injects expert knowledge and progressively guides the feature extraction module to focus on the modal information of the ischemic core and surrounding areas.
[0108] B Injecting expert knowledge and progressive feature guidance:
[0109] The overall network structure of the EKGFusion fusion model proposed in this invention is as follows: Figure 2 The construction process of the EKGFusion fusion model includes two steps: injecting expert knowledge and progressive feature guidance.
[0110] In the step of injecting expert knowledge, the present invention designs a lesion detection module, combines the median filter and binarization, and uses local pixel contrast to achieve adaptive separation of ischemic lesions and background areas, thereby simulating the process of experts focusing on the disease and quickly focusing on the high-signal lesion area in the DWI image. The input images DWI, FLAIR, MASK and fused images are all grayscale images with a size of h×w×1.
[0111] In the progressive feature-guided step, the Restormer module and the CNN module are combined to extract global and local features of the image. The CNN module is used to obtain shallow features across modalities without adding too much computational burden, and enhances the feature expression ability of ischemic stroke infarcts. This design strategy improves the performance and accuracy of the model while maintaining computational efficiency. The Restormer module is a method for extracting global features from high-resolution input images by applying a self-attention mechanism on the feature dimension. CNN, on the other hand, focuses on local features of image regions by using convolution kernels of different sizes.
[0112] C edge detection module:
[0113] In the ischemic stroke image fusion task, an important issue is how to guide the model to focus on the core area of cerebral ischemia to help radiologists and researchers conduct case analysis and research. However, the ischemic core area is often small in size and there is vasogenic edema in the surrounding area. If only the features of the last convolutional layer are considered and the features of the intermediate convolutional layers are ignored, it is very easy to cause edge information loss.
[0114] In order to extract more abundant lesion edge features, the present invention introduces the Scharr image edge detection operator and designs an edge detection module in combination with a multi-layer convolutional network. The edge detection modules of the three layers of DWI, FLAIR, and MASK are shown in Figure 2 As shown in εx(.)(x∈[1,4]) and ε(.). The design of the EKGFusion model is based on the MASK image features, and the lesion area is used to perform progressive feature guidance on the DWI and FLAIR images respectively. This design idea can better simulate the attention allocation ratio of radiologists when observing stroke patients and increase the attention to the ischemic core. Therefore, the present invention proposes an edge detection module and introduces the Scharr edge detection algorithm based on the multi-layer convolution structure to generate information-rich edge features of the ischemic core area. Specifically, as Figure 2 As shown:
[0115] By constructing multiple groups of 3*3 convolutional layers Conv and activation function Relu to extract image features, concatenate and add operations are used to avoid the loss of shallow convolution features, and finally 1×1 convolution is used to reduce the feature quantity to a single channel map to obtain the output features under multiple groups of convolutional layers. The specific formula is as follows:
[0116]
[0117] At this time, Conv1 is a 5*5 convolution layer and Relu activation function, Conv2 is a 3*3 convolution layer and Relu activation function, Cat is a concatenate operation, and Conv is a 1*1 convolution operation. Represents the output feature of the i-th layer. It is worth noting that in this embodiment, ε1(.), ε2(.), ε3(.), ε4(.) and ε(.) are set to the same operation.
[0118] Secondly, a novel Scharr gradient filter is introduced to learn the gradient information in the horizontal and vertical directions, and the gradient information is fed into the 1×1 convolution layer to reduce the difference in channel dimensions, so that the edge area information of the ischemic core can be captured more accurately, thereby improving the positioning accuracy and recognition ability of the ischemic core and surrounding areas of ischemic stroke. The body formula is as follows:
[0119]
[0120] in, Represents ε i The output features of (.), Scharr_xy represents the Scharr gradient filter operation.
[0121] D loss function construction
[0122] The loss function has an important influence on the improvement of the fusion model performance. It is used to train expert knowledge to guide the progressive learning network and drive the network to focus on edge details and structural information. Therefore, this embodiment attempts to fully retain the complementary information of the multimodal source images from multiple perspectives and designs a total loss function to control the transmission of model features. The formula is as follows:
[0123] L total (U,D,F)=αL psnr (U,F)+βL psnr (U,D)+γL SM (U,F)+δL SM (U,D);
[0124] Among them, U, D and F represent the fused image, DWI source image and FLAIR source image respectively, and L topal (U, D, F) represents the total loss function, L psnr (x,y) represents the peak signal-to-noise ratio loss function, L SM (x,y) represents the structural loss and mean square error loss, see the following formula.
[0125] L SM (U,D)=L SSIM (U,D)+L MSE (U,D);
[0126] L SM (U,F)=L SSIM (U,F)+L MSE (U,F);
[0127] L SSIM (x,y)=1-SSIM(x,y);
[0128] L MSE (x,y)=MSE(x,y);
[0129] Among them, SSIM(x,y) is a structural similarity measure commonly used in the statistical fusion process to calculate the information loss and image distortion, and calculates the correlation loss, brightness and contrast distortion. MSE(x,y) represents the mean square error.
[0130] The peak signal-to-noise ratio loss function can use the error sensitivity value to constrain the pixel-by-pixel difference between the source image and the fused image, measure the image distortion at the pixel level, and characterize the ratio of peak power to noise power in the fused image. The specific formula is:
[0131]
[0132] Here, MSE stands for mean square error.
[0133] The structural loss and mean square error loss are formulated as follows:
[0134] L SM (U,D)=L SSIM (U,D)+L MSE (U,D);
[0135] L SM (U,F)=L SSIM (U,F)+L MSE (U,F);
[0136] L SSIM (x,y)=1-SSIM(x,y);
[0137] L MSE (x,y)=MSE(x,u);
[0138] Among them, SSIM(x,y) is a structural similarity measure commonly used in the statistical fusion process to calculate the information loss and image distortion, and calculates the correlation loss, brightness and contrast distortion. MSE(x,y) represents the mean square error.
[0139] E. Experimental results analysis:
[0140] This example introduces the dataset used and the training details. Secondly, this example shows the qualitative and quantitative comparison and corresponding analysis of the EKGFusion method and other advanced methods on the DWI-FLAIR fusion task. Thirdly, the subjective evaluation of 9 doctors is presented. Finally, this example conducts a large number of ablation experiments on the model structure, the setting of the trade-off parameters in the loss function, and the composition of the objective function to verify the effectiveness of the specific design of the EKGFusion model.
[0141] (1) Dataset collection and preprocessing
[0142] 1) Data Collection
[0143] The embodiment of the present invention constructs a representative ischemic stroke multi-sequence MRI image dataset IschemicStroke 2024 for training and evaluation. Among them, the inclusion criteria are: ① Magnetic resonance imaging confirmed as acute cerebral infarction; ② Magnetic resonance imaging examination within 3 days after the onset of symptoms; ③ Patients with the first onset. The exclusion criteria are: ① The presence of brain tumors, brain trauma or surgical history; ② Receiving medication or treatment for other organ diseases; ③ Severe image artifacts affect lesion observation. All samples were obtained from the First Affiliated Hospital of Xinjiang Medical University, and the approval report of the Medical Ethics Committee of the First Affiliated Hospital of Xinjiang Medical University was obtained (ethical approval number: K202302-29).
[0144] Fusion of modal information under DWI and FLAIR sequences can help doctors accurately locate and evaluate ischemic lesions and their surrounding areas, obtain tissue variation and vascular status of patients, and promote individualized diagnosis and treatment as soon as possible. Therefore, this embodiment incorporates DWI and FLAIR into the fusion study.
[0145] 2) Data preprocessing
[0146] First, load the original DICOM file and convert it to NIFTI format while deleting irrelevant information. Then use the bet2 command in FMRIB Software Library (FSL) to remove the skull of the FLAIR sequence images. Subsequently, use a voxel size of 2mm×1mm×1mm to isotropically resample the image. Finally, select 128mm×256mm×256mm as the image normalization size. And through padding and center cropping operations, the image is unified to a size of 128mm*256mm*256mm. For the normalized image, this embodiment uses the flirt command in FSL to perform rigid registration with 6 degrees of freedom, and uses deviation normalization to normalize the image pixel data to [0,1] and save the image. So far, this embodiment has collected 239 patients with ischemic stroke, randomly selected 5-30 images for each patient, and collected a total of 4165 DWI-FLAIR images, and also collected clinical data for each patient.
[0147] In the training process, this embodiment uses the Adam optimizer, the original learning rate is 1e-3, the epoch is set to 10, and the batch size is 32. The convolution and activation functions used in this embodiment are shown in Figure 2As shown. In order to calculate the value of the loss function, the hyperparameters α, β, γ, and δ are preset to 0.005, 0.005, 1.5, and 1.5, respectively. According to the survey, the control variable method is particularly commonly used in selecting appropriate hyperparameters. Therefore, after the model optimization process, the hyperparameters α, β, γ, and δ are finally set to 0.05, 0.005, 1.5, and 1.5. This embodiment is deployed in the PyTorch framework, RTX 3090 (24GB), and the memory is 80GB.
[0148] (2) Comparison of algorithms and evaluation indicators
[0149] This example compares EKGFusion with seventeen representative traditional methods and deep learning, including CDDFuse, CNN, CNP, COF, CrossFuse, DATFuse, FGF, GeSeNet, HID, IFCNN, Li, LEGEF, MATR, MSMG-PCNN-EA, S-ADE, SwinFusion, UMFusion, ETEM, and MDHU. In addition, the above methods are tested using the codes published in the paper.
[0150] In order to objectively demonstrate the performance of the fusion results, this embodiment selects eight quantitative evaluation indicators, namely SD, SF, VIF, MI, CC, EN, AG and Q CV . SD is an evaluation index that reflects the distribution and contrast of the fused image. Humans tend to be attracted to areas with higher contrast, so the higher the SD, the better the fusion effect. SF is a measure of the degree of preservation of detail and texture information. MI is based on information theory to measure the amount of information preserved between the source image and the fused image. The correlation coefficient (CC) measures the degree of linear correlation between the fused image and the source image. Entropy (EN) is based on information theory to measure the amount of information retained from the source image contained in the fused image. The average gradient (AG) shows the gradient information in the fused image in a quantitative way, and retains the detail and texture information to a certain extent. VIF is a human perception indicator based on natural scene statistics and the human visual system. In addition to Q CV In addition to the index, the higher the score of other indicators, the better the fusion effect.
[0151] (3) DWI and FLAIR fusion results and discussion
[0152] like Figure 3 , 4As shown, this embodiment performs typical DWI-FLAIR fusion results under two groups of samples, and uses enlarged box diagrams to describe the ischemic core and the surrounding area. In addition, in order to compare the pixel distribution of the fused image and the pixel distribution of the ischemic core area, the figure plots the histograms of the fused image and the ischemic core area generated by different fusion methods, so as to more intuitively display the image pixel distribution. The two display methods of the enlarged box diagram and the histogram are combined to compare the differences between the fusion results of the present invention and other representative methods.
[0153] exist Figure 3 In the figure, according to the enlarged box diagram, the fused images generated by Crossfuse, GeSeNet and Our methods not only highlight the ischemic core area well, but also retain the texture details of the FLAIR image. On the contrary, the fused images of MDHU, MATR, CNP, UMFCMGR, DAT Fuse and CDDFuse methods have problems such as unclear details and low brightness. Figure 3 From the pixel histograms of the source image / fused image and the local area magnified image, it can be seen that there are more low-brightness areas in the DWI and FLAIR source images, among which DWI is mainly distributed in the range of 0-100, FLAIR is mainly distributed in the range of 0-20 and 100-160, and fewer pixels are distributed in the range of 100-180 and 200-250. The histograms obtained by several fusion methods such as MATR, DATFuse, and UMFCMGR are generally biased to the left, and the pixels are mainly concentrated in the range of 0-50 and 100-150, and are sparsely distributed in the range of 200-250. Only GeSeNet, Crossfuse, Swinfusion, CDDFuse and the method of this embodiment are more evenly distributed in the range of 150-250 pixels. In particular, the method of this embodiment retains more brightness information, highlights the ischemic core and surrounding areas very well, and the image details are clearer and more suitable for human visual perception.
[0154] Similarly, from Figure 4 From the comprehensive enlarged box diagram and histogram, it can be seen that the EKGFusion fusion method proposed in this embodiment is effective and retains the multi-sequence image information of ischemic stroke.
[0155] (4) Quantitative results presentation
[0156] For 128 pairs of test images, this example objectively compares the fusion results of EKGFusion with those of the other 19 representative methods. CV (Q CV =Q CV / 100) were appropriately linearly transformed to facilitate comparison in the same figure. The details are as follows Figure 5 And as shown in Table 1.
[0157] Figure 5 The order of the bars in is: Figure 5 The legend of the bars in the lower middle part is in the order from left to right and from top to bottom, that is, the order of the bars is: S-ADE, SwinFusion, Li, LEGEF, MATR, MSMG-PCNN-EA, CDDFuse, CNN, CNP, ACOF, Crossfuse...Our.
[0158] Figure 5 The bar chart shows the quantitative fusion results of 128 pairs of DWI-FLAIR images under different fusion methods, except for Q CV Except for the index, the other indexes are at the best level, which shows to some extent that the fusion image quality under the fusion method proposed by us is better. In order to facilitate the quantitative analysis of each index, this embodiment shows the specific numerical indexes in Table 1.
[0159] Table 1 Tabular display of 128 pairs of DWI-FLAIR quantitative results
[0160]
[0161] (5) Experts’ subjective evaluation
[0162] For ischemic stroke medical image fusion, the ultimate goal is to provide radiologists with fusion images with auxiliary observation, so as to facilitate the qualitative, quantitative and localization analysis of the ischemic core area of ischemic stroke by doctors. Therefore, the subjective evaluation of radiologists can prove the clinical significance and advantages of the EKGFusion fusion model proposed in this embodiment from a clinical perspective. This embodiment covers the fusion results of 9 representative methods under traditional fusion methods and deep learning fusion methods, and distributes them to 9 doctors in the Department of Radiology of the First Affiliated Hospital of Xinjiang Medical University. These doctors have clinical experience of more than 10 years (1 doctor), 6 to 9 years (2 doctors), 4 years (2 doctors) and less than 3 years (4). In this questionnaire, 15 scoring items were designed for 15 groups of fusion samples, and the score was set to [1-10]. The higher the score, the more the radiologists regard the fusion image generated under this method as more helpful to them in clinical observation and clinical diagnosis, which represents the better fusion effect under this method. For each group of samples, there are two original source images, DWI and FLAIR, and 9 fused images as scoring options. These images are generated by the following 9 representative methods (ETEM, Crossfuse, DATFuse, HID, IFCNN, Li, MATR, MDHU, Ours), which have been anonymously distributed during the scoring stage.
[0163] The statistical results are as follows Figure 6 As shown in the figure, the 15 fusion results generated by 9 different methods are marked as Sample1-15, and the bold line represents the average score of the 15 fusion samples given by 9 experts.
[0164] Figure 6 The order of the bars in is: Figure 6 The legends of the bars in the lower middle part follow the order from left to right and from top to bottom, that is, the order of the bars is: Sample1, Sample2, Sample3, Sample4, Sample5, Sample6, Sample7, ...Sample12...
[0165] Sample14, Sample15, Average value.
[0166] according to Figure 6 As shown, the fused images generated by the EKGFusion model and the Crossfuse model ranked first in the average subjective evaluation score. From the perspective of clinical experts, the fused images generated by the method proposed in this embodiment can better help them screen for ischemic stroke and reduce working time.
[0167] In summary, the model of the present invention not only achieves SOTA in objective indicators, but also has a more interpretable subjective fusion effect, which greatly improves the work efficiency of clinicians in diagnosing ischemic stroke.
[0168] (6) Generalization research
[0169] In order to prove the necessity and effectiveness of each component designed in EKGFusion, this embodiment conducts ablation experiments on three aspects: expert knowledge injection, feature extraction module and loss function composition. The quantitative indicators are shown in Table 2 and Table 3, and the visualization results are shown in Figure 7 .like Figure 7 As shown, the lack of a model structural component caused EKGFusion to perform slightly worse than the M11 condition in terms of signal presentation of the ischemic core and surrounding areas as well as the overall structural details of the fused images.
[0170] For the image fusion model, the selection and design of the loss function is very important, which will affect the quality and visual effect of the fused image to a certain extent. Therefore, this embodiment conducts an ablation study on the composition of the loss function and the loss function coefficients, attempting to improve the performance of the image fusion model and the quality of the generated results by reasonably selecting and designing the loss function.
[0171] As shown in Table 3, when L is removed total L psnrWhen assembling, the M9 structure is in Q CV The quantitative results under the indicator are weaker than those of M11, showing that L psnr The loss function has a good effect in improving the subjective visual effect. total L SM When L is removed, M10 obtains quantitative indicators of poor fusion performance in all indicators except SF and AG. It is worth noting that CC indicator performs the worst, reaching -0.7112. total Any component in will affect the fusion result of EKGFusion, thus affecting people's visual effects.
[0172] The selection of loss function coefficients also has a certain degree of influence on the performance of the image fusion model. Therefore, this embodiment uses four sets of loss functions to perform ablation studies on α, β, γ and δ respectively. Figure 8 shown.
[0173] Figure 8 The order of the bars in is: Figure 8 The legends of the bars in the lower part of each small graph are in order from left to right. For example, in the α graph in the upper left corner, the order of the bars is: 0.001, 0.005, 0.01, 0.05.
[0174] through Figure 8 As shown in Figure 2, when α, β, γ and δ are 0.05, 0.005, 1.5 and 1.5 respectively, the image fusion model achieves the best fusion effect.
[0175] Table 2 Composition of quantitative ablation experiments in each module
[0176]
[0177]
[0178] Table 3. Comparison of quantitative ablation experimental results of each module.
[0179] EN SD MI SF VIF AG CC <![CDATA[Q CV ]]> M1 4.0135 68.5304 3.1026 12.6963 1.0267 3.1673 0.9865 176.8507 M2 4.4955 67.7729 3.4106 13.1137 1.0839 3.3706 0.9864 186.2338 M3 4.7382 71.2358 3.3821 13.4320 1.1138 3.5245 0.9859 209.1697 M4 4.2291 67.1724 3.3953 12.4206 1.0398 3.0776 0.9868 164.2571 M5 4.3961 67.1302 3.2884 12.9839 1.0623 3.3399 0.9864 182.0720 M6 4.5315 71.4985 3.5566 12.5634 1.0733 3.0670 0.9865 187.6425 M7 4.6309 71.0418 3.5945 12.4520 1.1040 3.1679 0.9867 191.3760 M8 4.2061 70.1422 3.2316 12.8584 1.0380 3.1862 0.9865 190.7337 M9 4.6794 68.7372 3.5840 13.0107 1.1105 3.3501 0.9865 186.3057 M10 4.6458 38.0645 1.8366 24.1361 0.0120 5.5570 -0.7112 3007.6199 M11 4.6469 68.4431 3.5487 12.8745 1.1031 3.2879 0.9865 181.9410
[0180] Combined with the above, it can be seen that under the Ischemic Stroke 2024 dataset, compared with 19 mainstream methods, the EKGFusion model proposed in this invention has reached SOTA under 8 objective evaluation indicators. In addition, combined with the subjective evaluation scores of 9 radiology experts, the EKGFusion model also shows the best visual effect and has broad application value.
[0181] The above is only a preferred embodiment of the embodiment of the present invention, and does not impose any form of limitation on the embodiment of the present invention. Any simple modification, equivalent changes and modifications made to the above embodiment based on the technical essence of the embodiment of the present invention are still within the scope of the technical solution of the embodiment of the present invention.
Claims
1. A fusion model of multi-sequence magnetic resonance images of ischemic stroke, It is characterized in that The fusion model includes: a lesion detection module, two progressive feature guidance modules, an edge detection module, and a loss function module; The lesion detection module is used to focus on the high signal lesion area in the DWI image; The progressive feature guiding module is used to guide the modal image branch to perform feature extraction; The edge detection module is used to extract the edge features of the lesion.
2. The fusion model according to claim 1, It is characterized in that The lesion detection module combines a median filter and binarization, and uses local pixel contrast to achieve adaptive separation of ischemic lesions and background areas, thereby quickly focusing on high-signal lesion areas in DWI images.
3. The fusion model according to claim 1, It is characterized in that The progressive feature guidance module includes: a Restormer module and a CNN module; The CNN module is used to obtain cross-modal shallow features; The Restormer module extracts global features from high-resolution input images by applying a self-attention mechanism on the feature dimension.
4. The fusion model according to claim 1, It is characterized in that The edge detection module combines a multi-layer convolution structure with a Scharr edge detection algorithm to generate information-rich edge features of the ischemic core area.
5. The fusion model according to claim 4, It is characterized in that The edge detection module extracts image features by constructing multiple groups of 3*3 convolutional layers Conv, and after using concatenate and addition operations, uses 1×1 convolution to reduce the feature quantity to a single channel map, and obtains the output features under multiple groups of convolutional layers, using the following formula: At this time, Conv1 is a 5*5 convolution layer and Relu activation function, Conv2 is a 3*3 convolution layer and Relu activation function, Cat is a concatenate operation, and Conv is a 1*1 convolution operation. represents the output feature of the i-th layer, ε 1 (.),ε 2 (.),ε 3 (.),ε 4 (.) and ε(.) are set to the same operation; The Scharr gradient filter is used to learn the gradient information in the horizontal and vertical directions, and the gradient information is fed into the 1×1 convolution layer using the following formula: in, Represents ε i The output features of (.), Scharr_xy represents the Scharr gradient filter operation.
6. The fusion model according to claim 1, It is characterized in that The loss function module adopts the following formula: L total (U,D,F)=αL psnr (U,F)+βL psnr (U,D)+γL SM (U,F)+δL SM (U,D); Among them, L total (U, D, F) represents the total loss function, L psnr (x,y) represents the peak signal-to-noise ratio loss function, L SM (x,y) represents the structural loss and mean square error loss.
7. The fusion model according to claim 6, It is characterized in that The peak signal-to-noise ratio loss function adopts the following formula: Among them, MSE represents mean square error; The structural loss and mean square error loss are expressed as follows: L SM (U,D)=L SSIM (U,D)+L MSE (U,D); L SM (U,F)=L SSIM (U,F)+L MSE (U,F); L SSIM (x,y)=1-SSIM(x,y); L MSE (x,y)=MSE(x,y); Among them, SSIM(x,y) is a structural similarity measure commonly used in the statistical fusion process to calculate the information loss and image distortion, and calculates the correlation loss, brightness and contrast distortion. MSE(x,y) represents the mean square error.
8. A method for fusion of multi-sequence magnetic resonance images of ischemic stroke. It is characterized in that The method comprises: preprocessing the multi-sequence magnetic resonance images of ischemic stroke and inputting them into the fusion model described in any one of claims 1 to 7 for fusion.
9. The fusion method according to claim 8, It is characterized in that The ischemic stroke multi-sequence magnetic resonance images include DWI, FLAIR, and MASK images; The DWI and FLAIR images are subjected to global and local feature extraction respectively, the DWI image is input into the lesion detection module to generate a MASK image, and then the MASK image is input into the edge detection module for feature extraction; The features output by the edge detection module are then used to guide the progressive feature guidance modules of the DWI and FLAIR branches to perform feature extraction.
Citation Information
Cited By
Image fusion method based on raindrop layer interference reduction and background detail information enhancement
CN119313574A
Image fusion method based on raindrop layer interference reduction and background detail information enhancement
CN119313574B
