Magnetic Susceptibility Image Reconstruction Method and System Based on Multi-Source Information Fusion Network

Abstract

The present invention discloses a method and system for reconstructing a susceptibility image based on a multi-source information fusion network. A multi-source information fusion network is constructed, which includes a feature encoding module, a feature decoding module, a feature cross-fusion module, and a convolutional modulation module. A field map is obtained, and the field map is input into the feature encoding module to extract a first feature map and a second feature map. The first feature map is input into the feature decoding module, and a field map latent code is output. An amplitude map is obtained, and the second feature map and the amplitude map are input into the feature cross-fusion module for cross-fusion to obtain a cross-fusion feature map. A pre-estimation result of susceptibility is obtained, and the field map latent code, the cross-fusion feature map, and the pre-estimation result of susceptibility are input into the convolutional modulation module, and a susceptibility image is output. By using the fusion of multi-source information, intelligent and accurate reconstruction of the quantitative susceptibility map of tissues and organs can be achieved, providing a reliable quantitative information basis for clinically evaluating the susceptibility of tissues and organs quantitatively.

Description

Technical Field

[0001] The present invention relates to the technical field of magnetic resonance image processing based on convolutional neural networks, and particularly relates to a susceptibility image reconstruction method and system based on a multi-source information fusion network. Background Art

[0002] Magnetic resonance imaging is a technology that can image the internal structure of an object. Due to its advantages such as no radiation damage, high soft tissue resolution, and support for multi-parameter imaging, magnetic resonance imaging technology has become an indispensable clinical diagnostic tool. Nowadays, magnetic resonance technology has been widely applied in important scientific research fields such as brain science, neuroscience, and mental diseases. Different substances will produce different magnetization responses under the action of an external magnetic field. This phenomenon is called the magnetic susceptibility of the substance, and the degree of magnetization is measured by the susceptibility value. Quantitative Susceptibility Mapping (QSM) is an emerging magnetic resonance imaging technology for measuring the magnetic susceptibility characteristics of tissues. Using this technology, the magnetic susceptibility of substances in tissues can be measured, so as to perform accurate quantitative analysis on indicators such as iron content, calcification degree, and blood oxygen saturation. These applications are of great significance for the research and diagnosis of neurodegenerative diseases, brain injuries, and other organ lesions.

[0003] Most magnetic resonance imaging technologies usually obtain information from the amplitude map, and the phase map containing susceptibility information is directly discarded. The QSM technology can make good use of the information in the phase map and can obtain a quantitative susceptibility image after multiple steps of processing. The QSM technology includes the following key processes. First, there is a phase ambiguity, that is, phase wrapping, between the phase recorded by the imaging system and the true phase. Therefore, phase unwrapping needs to be performed before calculating the susceptibility distribution to obtain the original phase that has not been wrapped. Then, since the magnetic susceptibility of air is much greater than that of human tissues, at the junction of tissues and air, the background magnetic field will have large fluctuations, resulting in a small magnetic susceptibility contrast of local tissues and more significant phase wrapping. Therefore, after the phase unwrapping step, it is necessary to remove the interference of the background field and retain the high-quality local field distribution, that is, the background field removal step. Finally, based on a specific reconstruction algorithm, a quantitative susceptibility image can be obtained from the local field map, that is, the dipole inversion step.

[0004] In susceptibility imaging, the dipole inversion step is the most crucial one. Due to the existence of singular angles in the dipole kernel, obtaining the susceptibility map from the locally field map preprocessed from the acquired data is an ill-posed inverse problem. Collecting data in multiple directions for susceptibility reconstruction (COSMOS) is a relatively effective method. However, multiple sampling data requires a large amount of time and is difficult to implement in clinical practice. Therefore, many single-direction reconstruction methods have been proposed to solve this problem, such as methods based on threshold truncation, regularization methods combining amplitude map structure priors, etc. However, these methods have problems such as large artifacts, underestimation of susceptibility, or the need to manually design feature extraction rules, design specific regularization terms, and adjust parameters during algorithm implementation, and the processing process is relatively cumbersome.

[0005] In recent years, thanks to the development of hardware devices and related theories, especially the emergence of high-performance GPUs and the implementation of effective training methods, deep neural networks have developed rapidly and been applied. Among them, convolutional neural networks have been widely used in the fields of image processing and computer vision and achieved good results. Deep learning methods directly learn the feature distribution transformation between input data and label data by training deep neural networks, and no longer require manual design of feature extraction rules. In complex inverse problems such as image denoising, super-resolution, and compressive sensing reconstruction, deep learning methods have achieved good results and shown great development potential. In the field of QSM, Jongho Lee et al. first proposed a quantitative susceptibility reconstruction network based on convolutional neural networks. Steffen Bollmann et al. obtained a network that can be used to reconstruct real human brain susceptibility images by training on simulated data. Xu Li et al. combined the model expansion of the proximal optimization algorithm and used a convolutional network to implement the optimization process of the proximal operator, thereby realizing the reconstruction of susceptibility.

[0006] Due to the diversity of actual image acquisition parameters such as voxel size, field strength, and echo time, the data distribution is different. The existing deep learning-based susceptibility reconstruction methods do not consider making an adaptive design for the data differences in the network structure design. Therefore, when facing a large change in data distribution, the performance of the network drops sharply, that is, the generalization ability is insufficient. In addition, the existing methods do not make good use of the effective structure information in the amplitude map and lack the structure prior in the amplitude information, and there is a lack of reconstruction ability on clinical data. At the same time, the results obtained by the existing deep learning-based reconstruction methods have the problem of over-smoothing, and the tissue structure information in the data is not obvious, which is not conducive to clinical imaging diagnosis. Summary of the Invention

[0007] In view of the above-mentioned technical problems, an object of the embodiments of the present application is to propose a method and system for reconstructing susceptibility images based on a multi-source information fusion network to solve the technical problems mentioned in the above background art section.

[0008] In a first aspect, the present invention provides a method for reconstructing susceptibility images based on a multi-source information fusion network, including the following steps:

[0009] S1. Construct a multi-source information fusion network, which includes a feature encoding module, a feature decoding module, a feature cross-fusion module, and a convolutional modulation module;

[0010] S2. Obtain a field map, input the field map into the feature encoding module to extract a first feature map and a second feature map; input the first feature map into the feature decoding module and output a field map latent code;

[0011] S3. Obtain an amplitude map, input the second feature map and the amplitude map into the feature cross-fusion module for cross-fusion to obtain a cross-fusion feature map;

[0012] S4. Obtain a pre-estimation result of susceptibility, input the field map latent code, the cross-fusion feature map, and the pre-estimation result of susceptibility into the convolutional modulation module, and output a susceptibility image.

[0013] Preferably, the feature encoding module is composed of n + 2 Encoder modules, where n Encoder modules do not change the size and number of channels of the feature map, and the additional 2 Encoder modules are used to increase the number of channels of the feature map. At the same time, trilinear interpolation downsampling is used. The Encoder module includes a first convolutional layer and a second convolutional layer. The output result of the first convolutional layer is input into the second convolutional layer and element-wise added to the output result of the second convolutional layer to form a residual block; the first feature map is the feature output of the Encoder module with downsampling, and the second feature map is the feature output of the first n Encoder modules.

[0014] Preferably, the feature decoding module is composed of corresponding n Decoder modules, and each Decoder module includes a third convolutional layer, a fourth convolutional layer, a global average pooling layer, and a linear layer connected in sequence.

[0015] Preferably, the feature cross-fusion module uses convolutional layers to perform multi-level feature fusion on the multi-level second feature map, and at the same time extracts amplitude features from the amplitude map, adds the result of amplitude feature extraction to the result of multi-level feature fusion element-wise to obtain the cross-fusion result of multi-level feature map information and amplitude map information, and uses a multi-branch convolutional layer to extract different information from the cross-fusion result as output to obtain a cross-fusion feature map:

[0016]

[0017]

[0018] Among them, F BM is the cross-fusion result of multi-level feature map information and amplitude map information, and f fuse , f mag , f i respectively represent the 1×1×1 convolutional layer used for multi-level feature fusion, the convolutional layer used for extracting amplitude map information, and the convolutional layer applied to the feature map of the i-th branch. represents the output feature map of the i-th Encoder module in the feature encoding module, C represents concatenation, and I mag represents the input amplitude map, and M is the mask extracted from the amplitude map. represents the cross-fusion feature map output by the j-th branch.

[0019] Preferably, the convolutional modulation module has three input sources: the initial input, the cross-fusion feature map, and the field map latent code; among them, the field map latent code multiplies the convolutional kernel parameters in the channel dimension to achieve modulation of the convolutional kernel parameters:

[0020]

[0021] Among them, s i is the obtained corresponding field map latent code, i is the corresponding convolutional kernel channel, w i is the parameter of the original convolutional kernel, and w i ′ is the modulated convolutional kernel parameter;

[0022] The modulated convolutional kernel parameters are normalized:

[0023]

[0024] Among them, ε is a positive number.

[0025] Preferably, the pre-estimation result of magnetic susceptibility is used as the initial input of the convolutional modulation module, and the result of dividing the field map by the thresholded dipole kernel is used. The data corresponding to the ill-conditioned region in the k-space of this result is set to 0; the initial input passes through the convolutional layer modulated by the field map latent code and then is element-wise added to the cross-fusion feature map of the corresponding branch. The added result is input to the next-level modulated convolutional layer and then is also element-wise added to the cross-fusion feature map of the corresponding branch. After successive stacking and reconstruction, the result output by the last-level modulated convolutional layer passes through a 1×1×1 convolution to obtain the magnetic susceptibility image.

[0026] Preferably, the loss function of the multi-source information fusion network is:

[0027]

[0028] Among them, χ is the label image, and χ is the susceptibility image output by the multi-source information fusion network. f LoG is the filtering operation using the radially symmetric Laplacian of Gaussian operator LoG. σ is a constant, and k1 and k2 are the weight values of the two parts of the loss respectively.

[0029] In a second aspect, the present invention provides a susceptibility image reconstruction system based on a multi-source information fusion network, including:

[0030] A network construction unit configured to construct a multi-source information fusion network, which includes a feature encoding module, a feature decoding module, a feature cross-fusion module, and a convolutional modulation module;

[0031] An encoding and decoding unit configured to obtain a field map, input the field map into the feature encoding module to extract a first feature map and a second feature map; input the first feature map into the feature decoding module and output a field map latent code;

[0032] A cross-fusion unit configured to obtain an amplitude map, input the second feature map and the amplitude map into the feature cross-fusion module for cross-fusion to obtain a cross-fusion feature map;

[0033] A convolutional modulation unit configured to obtain a pre-estimation result of susceptibility, input the field map latent code, the cross-fusion feature map, and the pre-estimation result of susceptibility into the convolutional modulation module, and output a susceptibility image.

[0034] In a third aspect, the present invention provides an electronic device, including one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the first aspect.

[0035] In a fourth aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method described in any implementation manner of the first aspect is implemented.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] For the ill-posed inverse problem of reconstructing a quantitative susceptibility image from a field map, the present invention adopts a deep convolutional neural network, constructs a multi-source information fusion network according to the task characteristics, designs a specific loss function, realizes the reconstruction of the quantitative susceptibility image, solves the problems that the traditional processing method requires manual design of feature extraction rules and has a slow processing speed, and at the same time alleviates the problems of insufficient reconstruction of tissue structure details and insufficient reconstruction ability of lesion areas in clinical data in susceptibility reconstruction based on deep learning, and has better generalization performance.

[0038] In the method proposed by the present invention, the multi-source information fusion network uses data sources including the field map, amplitude map, and pre-estimated susceptibility results for susceptibility reconstruction, achieving full fusion of existing information. Compared with only using field map data, more accurate reconstruction results are obtained. The method of convolutional modulation is used to modulate the parameters of the convolutional layer using the latent codes of the data, enabling the network to adapt to the distribution of the input data. The feature cross-fusion module cross-fuses multi-level feature maps, improving the utilization rate of the feature maps. The introduced amplitude map provides structural information as a prior to enhance the network's reconstruction ability for regions such as lesions. In addition, the method proposed by the present invention can include the pre-estimated susceptibility results as the input of the convolutional modulation module, thereby accelerating the convergence of the network and improving the stability of network training. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0040] Figure 1 It is a schematic flowchart of a susceptibility image reconstruction method based on a multi-source information fusion network according to an embodiment of the present application;

[0041] Figure 2 It is a schematic diagram of the model of the multi-source information fusion network of the susceptibility image reconstruction method based on a multi-source information fusion network according to an embodiment of the present application, where (a) is the main framework of the model, and (b) is a detailed display of the model sub-module;

[0042] Figure 3 It shows the influence on the model by replacing the initial pre-estimated susceptibility result with the field map (MCnet_deltaB), removing the Fusion module (MCnet_NoFusion), and removing the modulation convolutional module (MCnet_NoMod-Conv) for the method of the present invention (labeled as MCnet) through the Loss-epoch curve in the susceptibility image reconstruction method based on a multi-source information fusion network according to an embodiment of the present application, where (a) is the loss curve on the training set and (b) is the loss curve on the validation set;

[0043] Figure 4 It is the susceptibility reconstruction results of different methods on healthy human brain data in the susceptibility image reconstruction method based on a multi-source information fusion network according to an embodiment of the present application. The upper two rows are the axial and coronal views of the reconstruction results respectively, and the lower two rows are the difference maps corresponding to the upper two rows and the label results;

[0044] Figure 5 Magnetic susceptibility reconstruction results of different methods of the magnetic susceptibility image reconstruction method based on a multi-source information fusion network in the embodiments of the present application on clinical human brain data, including field maps, amplitude maps, and magnetic susceptibility result maps reconstructed by each method;

[0045] Figure 6 Reconstruction results of different methods of the magnetic susceptibility image reconstruction method based on a multi-source information fusion network in the embodiments of the present application on calcification data in the 2019 QSM Challenge, including two views and corresponding enlarged views;

[0046] Figure 7 Schematic diagram of the magnetic susceptibility image reconstruction system based on a multi-source information fusion network in the embodiments of the present application. Detailed implementation manners

[0047] In order to make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Apparently, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0048] Figure 1 A magnetic susceptibility image reconstruction method based on a multi-source information fusion network provided by an embodiment of the present application is shown, including the following steps:

[0049] S1. Construct a multi-source information fusion network, which includes a feature encoding module, a feature decoding module, a feature cross-fusion module, and a convolution modulation module.

[0050] Specifically, referring to Figure 2 , the overall architecture of the multi-source information fusion network (MCnet) includes a feature encoding module, a feature decoding module, a feature cross-fusion module, and a convolution modulation module. The feature encoding module uses residual blocks as basic units to extract multi-level feature maps from the input field map; the feature decoding module extracts the field map latent code from the feature maps; the feature cross-fusion module combines the constraints of the amplitude map to perform cross-fusion on the multi-level feature maps; the convolution modulation module modulates the convolution parameters with the field map latent code, performs adaptive feature extraction on the previous-level feature map, and combines the feature map obtained by the feature cross-fusion module to finally reconstruct an accurate magnetic susceptibility image.

[0051] In the embodiments of the present application, three-dimensional convolution kernels are used. The inputs of the multi-source information fusion network are the field map (δB), the normalized amplitude map (Mag), and the magnetic susceptibility pre-estimation result (χ int ), and the output is the magnetic susceptibility image (χ). Figure 3In (a) is the overall architecture of the network, and (b) is the internal structure of the main sub-module. Modulate represents the convolutional modulation module, Cross-Fusion represents the feature cross-fusion module. The size of the input and output image blocks is 64×64×32, and numbers such as {32, 64, 128} represent the number of channels. The number of channels of the input and output feature maps is both 1. In other positions of the figure, the number of channels is only marked when the number of channels of the output feature map changes. The multi-source information fusion network uses the residual block as the basic unit for feature extraction, that is Figure 2 the Encoder module in

[0052] S2, obtain the field map, input the field map into the feature encoding module to extract the first feature map and the second feature map; input the first feature map into the feature decoding module, and output the field map latent code.

[0053] In a specific embodiment, the feature encoding module is composed of n + 2 Encoder modules. Among them, n Encoder modules do not change the size and number of channels of the feature map. The additional 2 Encoder modules are used to increase the number of channels of the feature map. At the same time, trilinear interpolation downsampling is used. The Encoder module includes a first convolutional layer and a second convolutional layer. The output result of the first convolutional layer is input into the second convolutional layer and element-wise added to the output result of the second convolutional layer to form a residual block; the first feature map is the feature output of the Encoder module with downsampling, and the second feature map is the feature output of the first n Encoder modules.

[0054] Specifically, the feature encoding module uses the residual block as the basic encoding unit, and performs an element-wise addition of a cross-connection from the input end to the output end every two layers of convolution to form a residual block. The output data stream of the feature encoding module includes two parts: the output of the residual block with downsampling, that is, the first feature map, which is used to obtain the field map latent code, and the feature output of the first n residual blocks, that is, the second feature map, which is used as the input of the feature cross-fusion module.

[0055] In a specific embodiment, the feature decoding module consists of n corresponding Decoder modules. Each Decoder module includes a third convolutional layer, a fourth convolutional layer, a global average pooling layer, and a linear layer connected in sequence. After passing through the Decoder module, a latent code regarding the field map data can be obtained, and the output dimension of the linear layer is the same as the number of convolutional kernel channels. The feature decoding module uses continuous convolutional layers, a global average pooling layer, and a linear layer to process the feature map output by the feature encoding module to obtain the field map latent code. Here, the latent code is a characterization of the hidden features of the data. Compared with the superficial and easily understandable surface features of the data, it does not represent the specific meaning of the data features, but is an expression of the deep relationship hidden under the surface features. In the embodiments of the present application, it is obtained through training based on deep learning methods.

[0056] S3. Obtain the amplitude map, input the second feature map and the amplitude map into the feature cross-fusion module for cross-fusion to obtain a cross-fused feature map.

[0057] In a specific embodiment, the feature cross-fusion module uses convolutional layers to perform multi-level feature fusion on the multi-level second feature map, and at the same time extracts amplitude features from the amplitude map. Add the results of the amplitude feature extraction and the results of the multi-level feature fusion element-wise to obtain the cross-fusion result of the multi-level feature map information and the amplitude map information. Use a multi-branch convolutional layer to extract different information from the cross-fusion result as the output to obtain a cross-fused feature map:

[0058]

[0059]

[0060] where, F BM is the cross-fusion result of the multi-level feature map information and the amplitude map information, f fuse , f mag , f i respectively represent the 1×1×1 convolutional layer used for multi-level feature fusion, the convolutional layer used to extract amplitude map information, and the convolutional layer applied to the feature map of the i-th branch, represents the output feature map of the i-th Encoder module in the feature encoding module, C represents concatenation, I mag represents the input amplitude map, M is the mask extracted from the amplitude map, represents the cross-fused feature map output by the j-th branch.

[0061] S4. Obtain the pre-estimated magnetic susceptibility result, input the field map latent code, the cross-fused feature map, and the pre-estimated magnetic susceptibility result into the convolutional modulation module, and output to obtain the magnetic susceptibility image.

[0062] In a specific embodiment, the convolutional modulation module has three input sources: an initial input, a cross-fusion feature map, and a field map latent code. Among them, the field map latent code multiplies the convolutional kernel parameters in the channel dimension to achieve the modulation of the convolutional kernel parameters:

[0063]

[0064] where s i is the obtained corresponding field map latent code, i is the corresponding convolutional kernel channel, and w i is the parameter of the original convolutional kernel, and w i ′ is the modulated convolutional kernel parameter;

[0065] To avoid the interference caused by the offset of the intensity value, the modulated convolutional kernel parameters are normalized:

[0066]

[0067] where ε is a positive number.

[0068] To avoid the division-by-zero operation, a small positive number ε is used, and ε is set to 10 -8 .

[0069] In a specific embodiment, the magnetic susceptibility pre-estimation result is used as the initial input of the convolutional modulation module. The result of dividing the field map by the thresholded dipole kernel is used, and the data corresponding to the ill-conditioned region in the k-space of this result is set to 0, thereby reducing the adverse effects brought by unreliable data points. The initial input passes through a convolutional layer modulated by the field map latent code and then is element-wise added to the cross-fusion feature map of the corresponding branch. The result of the addition is input to the next-level modulated convolutional layer and is also element-wise added to the cross-fusion feature map of the corresponding branch. After successive stacking and reconstruction, the result output by the last-level modulated convolutional layer passes through a 1×1×1 convolution to obtain the magnetic susceptibility image. Introducing the magnetic susceptibility pre-estimation result is used to accelerate network convergence, enhance the stability of network training, and thus improve the final reconstruction accuracy.

[0070] Specifically, in the modulated convolutional module, Mod is the operation of modulating the convolutional parameters, and Demod is the operation of normalizing the convolutional parameters, that is, demodulation. The number of modulated convolutions used is n, which corresponds to the n input and output channels in the feature cross-fusion module. Considering both the calculation speed and the reconstruction accuracy, n in the embodiments of the present application is set to 3.

[0071] In a specific embodiment, in order to obtain better magnetic susceptibility reconstruction metrics while taking into account retaining the organizational structure details of the reconstruction result, the loss function of the multi-source information fusion network is:

[0072]

[0073] where χ is the label image, χ is the susceptibility image output by the multi-source information fusion network, and f LoG is the filtering operation using the radially symmetric Laplacian of Gaussian operator LoG, σ is a constant, and the selected size balances the extraction of details and the robustness of the results. k1 and k2 are the weight values of the two parts of the loss respectively, used to balance the proportion of the two parts of the loss.

[0074] The training data used in the embodiments of this application comes from 9 healthy volunteers. Among them, 4 volunteers scanned 5 echoes in 5 directions respectively, with TR / TE1 / ΔTE = 28 / 5 / 5 ms, the imaging field of view was 220×220×110, and the matrix size was 224×224×126. The other 5 volunteers scanned 9 echoes in 4 directions respectively, with TR / TE1 / ΔTE = 45 / 2 / 2 ms, the imaging field of view was 220×220×110, and the matrix size was 224×224×110. All data were made into COSMOS as the labels for training. Limited by the GPU memory, during training, the field map, amplitude map, and COSMOS were overlapped and cut into three-dimensional blocks of 64×64×32.

[0075] The deep learning framework used to build the multi-source information fusion network is Pytorch, the batch size is set to 8, and the value of γ is 10 -8 , and ADAMW is used as the optimizer, with the initial learning rate set to 10 -4 , and the learning rate is adjusted to 10 after traversing the training data 20 times -5 , and the training time of the proposed network on a single NVIDIA TITAN X (Pascal) graphics card is about 12 hours.

[0076] Reference Figure 3 , and the influence of the method proposed in the embodiments of this application on the susceptibility reconstruction results is shown through the Training Loss-Epoch curve and the Validation-Epoch curve after removing the feature cross-fusion module (MCnet_NoFusion), removing the convolutional modulation module (MCnet_NoMod-Conv), and replacing the initial susceptibility pre-estimation result (MCnet_deltaB) respectively. The results show that the network without using the susceptibility pre-estimation result (MCnet_deltaB) has a larger initial Loss, which proves the effectiveness of the susceptibility pre-estimation result in accelerating the network convergence. After removing the feature cross-fusion module and removing the convolutional modulation module, the final convergent Loss of MCnet is larger than the convergent Loss of the complete network, indicating the effectiveness of the design of these two modules. In summary, the design of each module of MCnet can improve the reconstruction effect to varying degrees.

[0077] This application will also compare the reconstruction results of several reconstruction methods, namely TKD, SFCR, QSMnet, and LPCNN, on healthy human brain data, clinical human brain data, and the data of the 2019 QSM Challenge. The threshold of TKD is set to 0.2. In SFCR, λ1 and λ2 are set to 50 and 1, and γ1 and γ2 are set to 2000 and 20. The model code of QSMnet is rewritten according to the original paper, and the LPCNN model uses the source code provided by the author. The method of the present invention is labeled as MCnet.

[0078] Figure 4 The magnetic susceptibility reconstruction results of different methods on healthy human brain data are shown. The upper two rows in the figure show the reconstruction results and local magnifications in the transverse and coronal planes; the lower two rows in the figure are the difference images between the results of different methods and the label (COSMOS); the arrows in the figure indicate the regions where there are obvious differences in the results of each method. It can be seen from the positions pointed by the arrows in the figure that the present invention more accurately reconstructs the deep gray matter nucleus region. From the difference maps in the transverse and coronal planes, it can be seen that the results of the present invention are closer to the gold standard COSMOS. In the region around the globus pallidus, the detailed structures are more completely retained in the results of the present invention.

[0079] Table 1 lists the evaluation indexes of the test results of each method on healthy human brain data, including Peak Signal to Noise Ratio (PSNR), Root Mean Squared Error (RMSE), Structural Similarity (SSIM), and High Frequency Error Norm (HFEN). The higher the values of the peak signal-to-noise ratio and structural similarity indexes, the better; the lower the values of the root mean square error and high frequency error norm indexes, the better. It can be seen from Table 1 that MCnet has obtained the best reconstruction result.

[0080] Table 1 Comparison of objective evaluation indexes of different methods

[0081]

[0082] Table 2 lists the basic performance parameters of each model. Among them, the data dimension size used in the running time test is 224×224×104, including two cases of running on CPU and GPU. The CPU running environment of the deep learning model is Intel Xeon E5-2620, and the GPU running environment is a 12GB NVIDIA Pascal Titan X GPU and 64GB RAM. The CPU running environment of the traditional method is AMD Ryzen7 4800H and 32GB RAM. From the number of parameters and the running time on different platforms, it can be seen that the method proposed in the present invention has high execution performance.

[0083] Comparison of performance parameters of each model in Table 2

[0084]

[0085] The susceptibility reconstruction results of different methods on clinical human brain data are as Figure 5 shown. All deep learning-based methods did not perform fine-tuning of model parameters. Compared with other methods, the MCnet reconstruction result of the hematoma area is more accurate, with less artifacts generated, and at the same time, the contrast of other structures in the data is maintained.

[0086] Figure 6 This is the result of directly testing different deep learning-based methods on the 2019 QSM Challenge data. From the difference map, it can be seen that compared with other deep learning-based methods, the difference of the reconstruction result of the method proposed in the present invention is smaller. The reconstruction result indicators in Table 3 also reflect the advantages of the proposed method. In the calcification area, the contour of the method proposed in the present invention is clearer, achieving a more accurate result. In the blood vessel area (indicated by the arrow), the reconstruction result is significantly better than other methods.

[0087] Table 3 Comparison of result indicators on the 2019 QSM Challenge data

[0088]

[0089] Further referring to Figure 7 and as an implementation of the methods shown in the above figures, the present application provides an embodiment of a susceptibility image reconstruction system based on a multi-source information fusion network. This system embodiment corresponds to Figure 1 the method embodiment shown and can be specifically applied to various electronic devices.

[0090] The embodiment of the present application provides a susceptibility image reconstruction system based on a multi-source information fusion network, including:

[0091] The network construction unit 1 is configured to construct a multi-source information fusion network, and the multi-source information fusion network includes a feature encoding module, a feature decoding module, a feature cross-fusion module, and a convolutional modulation module;

[0092] The encoding and decoding unit 2 is configured to obtain a field map, input the field map into the feature encoding module to extract a first feature map and a second feature map; input the first feature map into the feature decoding module, and output a field map latent code;

[0093] The cross-fusion unit 3 is configured to obtain an amplitude map, input the second feature map and the amplitude map into the feature cross-fusion module for cross-fusion, and obtain a cross-fusion feature map;

[0094] The convolutional modulation unit 4 is configured to obtain a pre-estimation result of magnetic susceptibility, input the field map latent code, the cross-fusion feature map, and the pre-estimation result of magnetic susceptibility into the convolutional modulation module, and output a magnetic susceptibility image.

[0095] It should be noted that the computer-readable medium described in this application may be a computer-readable signal medium or a computer-readable medium or any combination of the above two. The computer-readable medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device, apparatus, or component, or any combination of the above. More specific examples of the computer-readable medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, the computer-readable medium may be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution device, apparatus, or component. And in this application, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which the computer-readable program code is carried. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than the computer-readable medium, and this computer-readable medium can send, propagate, or transmit a program for use by or in combination with an instruction execution device, apparatus, or component. The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination of the above.

[0096] Computer program code for performing the operations of this application can be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or it can also be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0097] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram can represent a module, a segment of a program, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks can occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based device that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0098] The units described in the embodiments of this application can be implemented in software or in hardware. The described units can also be provided in a processor.

[0099] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or may exist alone without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the electronic device, the electronic device is caused to: construct a multi-source information fusion network, the multi-source information fusion network including a feature encoding module, a feature decoding module, a feature cross-fusion module, and a convolutional modulation module; obtain a field map, input the field map into the feature encoding module to extract a first feature map and a second feature map; input the first feature map into the feature decoding module to output a field map latent code; obtain an amplitude map, input the second feature map and the amplitude map into the feature cross-fusion module for cross-fusion to obtain a cross-fusion feature map; obtain a pre-estimation result of magnetic susceptibility, input the field map latent code, the cross-fusion feature map, and the pre-estimation result of magnetic susceptibility into the convolutional modulation module, and output a magnetic susceptibility image.

[0100] The above description is only a preferred embodiment of the present application and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by the specific combination of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the technical solutions formed by mutually replacing the above features with the technical features (but not limited to) having similar functions disclosed in the present application.

Claims

1. A method for reconstructing a magnetic susceptibility image based on a multi-source information fusion network, characterized in that, The following steps are involved: S1, constructing a multi-source information fusion network, wherein the multi-source information fusion network includes a feature encoding module, a feature decoding module, a feature cross fusion module and a convolution modulation module; S2, obtaining a field image, inputting the field image into the feature encoding module to extract a first feature image and a second feature image; inputting the first feature image into the feature decoding module, and outputting a field image latent code; S3, obtaining an amplitude map, inputting the second feature map and the amplitude map into a feature cross-fusion module for cross-fusion, and obtaining a cross-fusion feature map; the feature cross-fusion module uses a convolution layer to perform multi-level feature fusion on the multi-level second feature map, and at the same time extracts amplitude features from the amplitude map, and adds the result of the amplitude feature extraction to the result of the multi-level feature fusion to obtain a cross-fusion result of the multi-level feature map information and the amplitude map information, and uses a multi-branch convolution layer to extract different information from the cross-fusion result as output, and obtains the cross-fusion feature map: Among them, F BM is the cross - fusion result of multi - level feature map information and amplitude map information, f fuse , f mag , f i respectively represent the 1×1×1 convolutional layer used for multi - level feature fusion, the convolutional layer used for extracting amplitude map information, and the convolutional layer applied to the feature map of the i - th branch, represents the output feature map of the i - th Encoder module in the feature encoding module, C represents concatenation, I mag represents the input amplitude map, M is the mask extracted from the amplitude map, represents the cross - fusion feature map output by the j - th branch; S4, obtaining a magnetic susceptibility pre-estimation result, inputting the field image latent code, the cross-fusion feature map and the magnetic susceptibility pre-estimation result into a convolution modulation module, and outputting a magnetic susceptibility image; the convolution modulation module has three input sources: an initial input, a cross-fusion feature map, and a field image latent code, wherein the field image latent code is multiplied by a convolution kernel parameter in a channel dimension to achieve modulation of the convolution kernel parameter: w i ' = s i ·w i ; Among them, s i is the obtained corresponding field map latent code, i is the corresponding convolution kernel channel, w i is the parameter of the original convolution kernel, w i ' is the parameter of the modulated convolution kernel; The modulated convolution kernel parameters are normalized: Among them, ε is a positive number.

2. The magnetic susceptibility image reconstruction method based on a multi-source information fusion network according to claim 1, wherein The feature encoding module is composed of n+2 Encoder modules, wherein the n Encoder modules do not change the size and the number of channels of the feature map, and the additional 2 Encoder modules are used to increase the number of channels of the feature map, and trilinear interpolation downsampling is used at the same time. The Encoder module includes a first convolutional layer and a second convolutional layer, and the output result of the first convolutional layer is input into the second convolutional layer and element-wise added to the output result of the second convolutional layer to form a residual block; the first feature map is the feature output of the Encoder module with downsampling, and the second feature map is the feature output of the first n Encoder modules.

3. The magnetic susceptibility image reconstruction method based on a multi-source information fusion network according to claim 1, wherein The feature decoding module is composed of corresponding n Decoder modules, and each Decoder module includes a third convolution layer, a fourth convolution layer, a global average pooling layer and a linear layer connected in sequence.

4. The magnetic susceptibility image reconstruction method based on a multi-source information fusion network according to claim 1, wherein The magnetic susceptibility pre-estimation result is used as the initial input of the convolution modulation module, and the result after dividing the field map by the thresholded dipole kernel is adopted, and the data of the pathological area corresponding to the result in the k-space is set to 0; the initial input is subjected to the convolution layer of the field map latent code modulation, and then element-by-element addition is performed with the cross-fusion feature map of the corresponding branch. The result of the addition is input to the modulation convolution layer of the next level, and then element-by-element addition operation is performed with the cross-fusion feature map of the corresponding branch. After step-by-step superposition and reconstruction, the result output by the last level of modulation convolution layer is subjected to a layer of 1×1×1 convolution to obtain the magnetic susceptibility image.

5. The method for reconstructing a magnetic susceptibility image based on a multi-source information fusion network according to claim 1, wherein The loss function of the multi-source information fusion network is: where χ is the label image, is the magnetic susceptibility image output by the multi-source information fusion network, and f LoG is the filtering operation using the radially symmetric Laplacian of Gaussian operator LoG, σ is a constant, and k1 and k2 are the weight values of the two parts of the loss respectively.

6. A magnetic susceptibility image reconstruction system based on a multi-source information fusion network, characterized in that include: A network construction unit, configured to construct a multi-source information fusion network, where the multi-source information fusion network includes a feature encoding module, a feature decoding module, a feature cross-fusion module, and a convolution modulation module; An encoding and decoding unit, configured to obtain a field map, input the field map into the feature encoding module to extract a first feature map and a second feature map; input the first feature map into the feature decoding module, and output a field map latent code; A cross-fusion unit, configured to obtain an amplitude map, input the second feature map and the amplitude map into the feature cross-fusion module for cross-fusion to obtain a cross-fusion feature map; the feature cross-fusion module uses a convolutional layer to perform multi-level feature fusion on the multi-level second feature map, and at the same time performs amplitude feature extraction on the amplitude map, adds the result of the amplitude feature extraction to the result of the multi-level feature fusion element by element to obtain the cross-fusion result of the multi-level feature map information and the amplitude map information, and uses a multi-branch convolutional layer to extract different information from the cross-fusion result as output to obtain the cross-fusion feature map: Among them, F BM is the cross - fusion result of multi - level feature map information and amplitude map information, f fuse , f mag , f i respectively represent the 1×1×1 convolutional layer used for multi - level feature fusion, the convolutional layer used for extracting amplitude map information, and the convolutional layer applied to the feature map of the i - th branch, represents the output feature map of the i - th Encoder module in the feature encoding module, C represents concatenation, I mag represents the input amplitude map, M is the mask extracted from the amplitude map, represents the cross - fusion feature map output by the j - th branch; A convolution modulation unit, configured to obtain a magnetic susceptibility pre-estimation result, input the field map latent code, the cross-fusion feature map, and the magnetic susceptibility pre-estimation result into the convolution modulation module, and output a magnetic susceptibility image; the convolution modulation module has three input sources: an initial input, a cross-fusion feature map, and a field map latent code, where the field map latent code multiplies the convolution kernel parameter in the channel dimension to achieve the modulation of the convolution kernel parameter: w i ' = s i ·w i ; Among them, s i is the obtained corresponding field map latent code, i is the corresponding convolution kernel channel, w i is the parameter of the original convolution kernel, and w i ' is the parameter of the modulated convolution kernel; The modulated convolution kernel parameter is normalized: where ε is a positive number.

7. An electronic device, comprising: One or more processors; A storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method according to any one of claims 1-5.

Images