Unsupervised deep learning-based image correction method and device for dMRI image

Abstract

The invention provides an unsupervised deep learning-based image correction method and device for a dMRI image, and a computer readable medium. The image correction method comprises the following steps: acquiring dMRI data, wherein the dMRI data comprises two pieces of data with opposite phase encoding directions; calculating FOD images according to the dMRI data, wherein the FOD images comprise a first FOD image and a second FOD image; establishing an affine transformation network, and inputting the first FOD image and the second FOD image into the affine transformation network to obtain a first coarse correction image; establishing a distortion transformation network, and inputting the first coarse correction image and the second FOD image into the distortion transformation network to obtain a distortion field between the first FOD image and the second FOD image; and performing distortion correction on the first FOD image and the second FOD image according to the distortion field to respectively obtain a first corrected image and a second corrected image.

Description

Technical Field

[0001] The present application mainly relates to the field of image processing, and in particular to an image correction method, an image correction device and a computer-readable medium based on unsupervised deep learning for dMRI images. Background Art

[0002] The high-resolution, multi-shell diffusion MRI (dMRI) data provided by the Human Connectome Project (HCP) provide unprecedented convenience for studying human brain pathways. However, after processing dMRI data with traditional methods and learning-based methods, significant residual distortion still exists in brain regions such as the brainstem. The typical method is to obtain B0 images from phase-encoding (PE) and then use the topup tool in FSL (FSL-FMRIB Software Library) to estimate the distortion field. With the rapid development of deep learning technology, some learning-based methods are used to process B0 images, which can achieve correction accuracy similar to topup with less computing time. However, due to the poor image quality of B0 images in some areas, these deep learning methods have limitations in achieving high correction accuracy, especially in the brainstem region. Therefore, a method is needed to effectively improve the distortion correction of areas with poor image quality such as the brainstem region. Summary of the invention

[0003] In response to the above technical problems, the present application provides an end-to-end image correction method, an image correction device and a computer-readable medium based on unsupervised deep learning for dMRI images, which can achieve excellent distortion correction performance in a shorter prediction time.

[0004] To solve the above technical problems, the present application provides an image correction method based on unsupervised deep learning for dMRI images, comprising: acquiring dMRI data, the dMRI data including data in two opposite phase encoding directions; calculating an FOD image according to the dMRI data, the FOD image including a first FOD image and a second FOD image, wherein the first FOD image corresponds to a first phase encoding direction, the second FOD image corresponds to a second phase encoding direction, and the first phase encoding direction is opposite to the second phase encoding direction; establishing an affine transformation network, inputting the first FOD image and the second FOD image into the affine transformation network to obtain a first coarse corrected image; establishing a distortion transformation network, inputting the first coarse corrected image and the second FOD image into the distortion transformation network to obtain a distortion field between the first FOD image and the second FOD image; performing distortion correction on the first FOD image and the second FOD image according to the distortion field to obtain a first corrected image and a second corrected image, respectively.

[0005] In one embodiment of the present application, before calculating the FOD image based on the dMRI data, it also includes: using an eddy current correction method to process the dMRI data to obtain processed dMRI data; the step of calculating the FOD image based on the dMRI data includes: calculating the FOD image based on the processed dMRI data.

[0006] In one embodiment of the present application, the affine transformation network includes several convolution modules, a maximum pooling layer and a fully connected layer, and the output of the fully connected layer includes a parameter set μ,

[0007] μ=(θ x ,θ y ,θ z ,t x ,t y ,t z ),

[0008] Among them, x, y, z are three coordinate axes, θ x ,θ y ,θ z Respectively represent the rotation angle of the corresponding coordinate axis, t x ,t y ,t z The steps of inputting the first FOD image and the second FOD image into the affine transformation network to obtain a first coarse correction image respectively represent the translation distance of the corresponding coordinate axis, and comprising:

[0009] The affine matrix A is calculated using the following formula:

[0010]

[0011]

[0012] The affine matrix A is applied to the first FOD image through a spatial transformation network to obtain the first coarse corrected image.

[0013] In one embodiment of the present application, the loss function of the affine transformation network is:

[0014]

[0015] Where A is the affine matrix, L sim is the similarity loss function, I + represents the first FOD image, I - Represents the second FOD image.

[0016] In one embodiment of the present application, it also includes: adopting a multi-resolution strategy to train the affine transformation network, including: obtaining a plurality of first sub-FOD images and a plurality of second sub-FOD images at different resolutions from the first FOD image and the second FOD image respectively; obtaining a plurality of groups of training images, each group of training images including a pair of first sub-FOD images and second sub-FOD images with opposite phase encoding directions; starting from a group of training images with the lowest resolution, inputting the current group of training images into the affine transformation network to obtain a current affine matrix, applying the current affine matrix and the next group of training images to the affine transformation network through the spatial transformation network to obtain a next affine matrix corresponding to the next group of training images, until the last group of training images is used as the input of the affine transformation network, and the target affine matrix is ​​obtained.

[0017] In one embodiment of the present application, the distortion transformation network includes U-Net, and the output of the U-Net is the distortion field.

[0018] In one embodiment of the present application, the loss function of the distortion transformation network is:

[0019]

[0020] Where, φ represents the distortion field, L sim is the similarity loss function, I + represents the first FOD image, I - represents the second FOD image, represents a first corrected image corresponding to said first FOD image, represents the second corrected image corresponding to the second FOD image, L reg represents the regularized loss function used to optimize the distortion field, p represents the image voxel position, and Ω represents the entire image domain.

[0021] In one embodiment of the present application, a multi-resolution strategy is adopted to train the distortion transformation network, including: obtaining a plurality of first sub-FOD images and a plurality of second sub-FOD images at different resolutions from the first FOD image and the second FOD image, respectively; obtaining a plurality of groups of training images, each group of training images including a pair of first sub-FOD images and second sub-FOD images with opposite phase encoding directions; starting from a group of training images with the lowest resolution, inputting the current group of training images into the distortion transformation network to obtain a current distortion field, applying the current distortion field and the next group of training images to the distortion transformation network through a spatial transformation network to obtain a next distortion field corresponding to the next group of training images, until the last group of training images is used as input to the distortion transformation network to obtain a target distortion field.

[0022] In one embodiment of the present application, the steps of obtaining multiple first sub-FOD images and multiple second sub-FOD images at different resolutions from the first FOD image and the second FOD image respectively include: using a Gaussian blur method to process the first FOD image and the second FOD image to obtain multiple first sub-FOD images and multiple second sub-FOD images at different resolutions.

[0023] In one embodiment of the present application, it also includes: freezing the parameters of the affine transformation network when training the distortion transformation network.

[0024] In one embodiment of the present application, the method further includes: synthesizing the first corrected image and the second corrected image to obtain a corrected dMRI image.

[0025] In order to solve the above-mentioned technical problems, the present application also proposes an image correction device based on unsupervised deep learning for dMRI images, comprising: a data acquisition module for acquiring dMRI data; a FOD image calculation module for calculating the FOD image based on the dMRI data; a correction network module, comprising an affine transformation network and a distortion transformation network; a memory for storing instructions executable by a processor; and a processor for executing the instructions to implement the method as described above.

[0026] In order to solve the above technical problem, the present application further proposes a computer-readable medium storing a computer program code, wherein the computer program code implements the above method when executed by a processor.

[0027] The image correction method and device of the present application proposes a distortion correction network DiscoNet, including an affine transformation network and a distortion transformation network. The trained DiscoNet can perform distortion correction on the first FOD image and the second FOD image according to the distortion field to obtain a first corrected image and a second corrected image, respectively. The present application also designs loss functions for the affine transformation network and the distortion transformation network, respectively, so that the network can converge quickly and obtain the correction result quickly. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings are included to provide a further understanding of the present application. They are included and constitute a part of the present application. The accompanying drawings illustrate embodiments of the present application and together with the present specification serve to explain the principles of the present application. In the accompanying drawings:

[0029] Figure 1 is an exemplary flow chart of an image correction method according to an embodiment of the present application;

[0030] Figure 2 is a network structure block diagram corresponding to the image correction method according to an embodiment of the present application;

[0031] Figure 3 is an exemplary block diagram of an affine transformation network established in an embodiment of the present application;

[0032] Figure 4 is an exemplary block diagram of a distortion transformation network established in an embodiment of the present application;

[0033] Figure 5 Used to illustrate the multi-resolution training strategy in the image correction method of one embodiment of the present application;

[0034] Figure 6 These are the distortion correction effect diagrams of the six methods;

[0035] Figure 7 It is the MAD and MSD quantification results of the six methods in three ROIs;

[0036] Figure 8 An unsupervised deep learning-based image correction device for dMRI images according to an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0038] As shown in this application, unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not refer to the singular, but also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0039] Unless otherwise specifically stated, the relative arrangement, numerical expressions and numerical values ​​of the parts and steps set forth in these embodiments do not limit the scope of the application. Meanwhile, it should be understood that, for ease of description, the sizes of the various parts shown in the accompanying drawings are not drawn according to actual proportional relationships. The technology, methods and equipment known to those of ordinary skill in the relevant art may not be discussed in detail, but in appropriate cases, the technology, methods and equipment should be considered as a part of the specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters represent similar items in the following drawings, and therefore, once a certain item is defined in an accompanying drawing, it does not need to be further discussed in subsequent drawings.

[0040] In addition, it should be noted that the use of words such as "first" and "second" to define components is only for the convenience of distinguishing the corresponding components. If not otherwise stated, the above words have no special meaning and cannot be understood as limiting the scope of protection of this application. In addition, although the terms used in this application are selected from well-known and commonly used terms, some terms mentioned in the specification of this application may be selected by the applicant at his or her discretion, and their detailed meanings are explained in the relevant parts of the description of this article. In addition, it is required to understand this application not only by the actual terms used, but also by the meaning implied by each term.

[0041] Flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, various steps may be processed in reverse order or simultaneously. At the same time, other operations may be added to these processes, or one or more operations may be removed from these processes.

[0042] The image correction method provided in this application is an end-to-end method based on an unsupervised learning framework, which can correct the magnetic susceptibility distortion in dMRI images. More specifically, this application proposes a distortion correction network DiscoNet, which can achieve excellent distortion correction performance in a shorter prediction time.

[0043] Figure 1 is an exemplary flow chart of an image correction method according to an embodiment of the present application. Figure 1 As shown, the image correction method of this embodiment includes:

[0044] Step S10: Acquire dMRI data, where the dMRI data includes data in two opposite phase encoding directions;

[0045] Step S20: Calculating a FOD image according to the dMRI data, the FOD image comprising a first FOD image and a second FOD image, wherein the first FOD image corresponds to a first phase encoding direction, the second FOD image corresponds to a second phase encoding direction, and the first phase encoding direction is opposite to the second phase encoding direction;

[0046] Step S30: establishing an affine transformation network, inputting the first FOD image and the second FOD image into the affine transformation network to obtain a first coarse correction image;

[0047] Step S40: establishing a distortion transformation network, inputting the first coarse correction image and the second FOD image into the distortion transformation network, and obtaining a distortion field between the first FOD image and the second FOD image;

[0048] Step S50: performing distortion correction on the first FOD image and the second FOD image according to the distortion field to obtain a first corrected image and a second corrected image respectively.

[0049] The above steps S10 to S50 are described below with reference to the accompanying drawings.

[0050] In step S10, the present application does not limit where the dMRI data is obtained. In one embodiment, the data can be obtained from the HCP. The dMRI data should include data with two opposite phase encoding (PE) directions. For example, the left-right (R / L) or the front-back (P / A) phase encoding direction can be used as the first phase encoding direction and the second phase encoding direction, respectively.

[0051] In step S20, the present application does not limit how to calculate the FOD image from the dMRI data, and the relevant technology in the field can be used to calculate the FOD image. The FOD (Fiber Orientation Distribution) image can be calculated from the multi-shell dMRI data using a multi-compartment model. At each voxel, the FOD is expressed as spherical harmonic coefficients (SPHARMs). In one embodiment of the present application, the base order of SPHARMs is selected as 8 to estimate the FOD image, where the first 6 coefficients are used to estimate the distortion field.

[0052] Figure 2 : is a network structure diagram corresponding to the image correction method of an embodiment of the present application. Figure 2 As shown, the first PE data 201 corresponds to the positive PE data in the dMRI data, and the second PE data 202 corresponds to the negative PE data in the dMRI data. Positive and negative indicate that the phase encoding directions of the two are opposite.

[0053] In step S20, the first PE data 201 is calculated to obtain a first FOD image 210, ie, a positive PE FOD image, and the second PE data 202 is calculated to obtain a second FOD image 220, ie, a negative PE FOD image.

[0054] In some embodiments, before step S20, the following steps may also be included:

[0055] Step S11: Processing the dMRI data using an eddy current correction method to obtain processed dMRI data, wherein the eddy current correction method is a conventional technique in the art and will not be elaborated here.

[0056] It should be noted that the use of the eddy current correction method to perform eddy current correction on the dMRI data is beneficial to improving the accuracy of subsequent image correction. If step S11 is not executed, but step S20 is directly executed, and the DiscoNet designed in the present application is used to process the original dMRI image, a certain correction effect can also be obtained, but the effect is not as good as executing step S11. At the same time, the existing dMRI distortion correction methods often use eddy current correction to pre-process the original dMRI data. Therefore, the use of the eddy current correction method in the present application is also conducive to a more reasonable comparison of the method of the present application with other methods.

[0057] exist Figure 2In the embodiment shown, after the first PE data 201 is processed by the eddy current correction method (indicated by arrows in the figure), a first FOD image 210 is obtained. + After the second PE data 202 is processed by the eddy current correction method (indicated by arrows in the figure), a second FOD image 220 is obtained. - express.

[0058] In step S30, an affine transformation network (AffineNet) is established.

[0059] refer to Figure 2 , where Indicates concatenation, that is, I + and I - After connection, it is input into DiscoNet230, which outputs the distortion field φ. DiscoNet230 includes an affine transformation network.

[0060] Figure 3 is an exemplary block diagram of an affine transformation network established in an embodiment of the present application. Figure 3 As shown, the first FOD image 210 and the second FOD image 220 are both inputs of the affine transformation network 300. In some embodiments, the first FOD image 210 and the second FOD image 220 are connected as a 12-channel image and input into the affine transformation network 300. The affine transformation network 300 may specifically include several convolution modules, a maximum pooling layer (Maxpool) and a fully connected layer (FullyConnected). Figure 3 Four convolutional blocks are shown in Figure 1. Each convolutional block includes a 3D convolutional layer with a convolution kernel size of 3*3*3 and a LeakyReLU activation layer with an activation parameter of 0.2. The fully connected layer includes two linear layers and a ReLU activation layer, which are used to estimate the parameter set μ including 6 parameters.

[0061] μ=(θ x ,θ y ,θ z ,t x ,t y ,t z ),

[0062] Among them, x, y, z are three coordinate axes, θ x ,θ y ,θ z Respectively represent the rotation angle of the corresponding coordinate axis, t x ,t y ,t zRespectively represent the translation distance of the corresponding coordinate axis.

[0063] Since the dMRI data include two opposite data from the same subject, there is only a rotation or translation type misalignment between the corresponding two dMRI images, and there is no scaling difference. Therefore, the rotation angle and translation distance can be estimated by the affine transformation network 300, and the linear space misalignment that may exist between the two FOD images can be estimated.

[0064] The set transformation matrix can be calculated from the parameter set μ: x , R y , R z , T, where:

[0065]

[0066] In step S30, the first FOD image 210 and the second FOD image 220 are input into the affine transformation network 300 to obtain a first rough correction image I a The steps include:

[0067] Step S31: Calculate the affine matrix A using the following formula (5):

[0068]

[0069] Step S32: Apply the affine matrix A to the first FOD image 210 through a spatial transformation network to obtain a first coarse correction image I a .

[0070] exist Figure 3 In, use Represents the spatial transformer network STN (Spatial Transformer Network). The first coarse correction image I a is a coarse alignment image of the second FOD image 220. It should be noted that Figure 3 The positions of the first FOD image 210 and the second FOD image 220 in the image can be interchanged, and are not limited to Figure 3 shown.

[0071] In some embodiments, during the training process, the loss function of the affine transformation network 300 is:

[0072]

[0073] Among them, A is an affine matrix, L sim is the similarity loss function, I + Represents the first FOD image, I _ Represents the second FOD image.

[0074] L sim Used to measure the second FOD image and the first rough correction image I a During the training process, when the similarity reaches the convergence condition, the training ends.

[0075] In step S40, a distortion transformation network is established. Figure 2 DiscoNet230 in

[15] also includes a distortion transformation network.

[0076] Since the small differences between the two dMRI images cannot be captured by using only the affine transformation network 300, the image correction method of the present application also uses a distortion transformation network. In one embodiment, the distortion transformation network is a non-rigid transformation network. The distortion transformation network of the present application includes a U-Net for generating a distortion field φ.

[0077] Figure 4 is an exemplary block diagram of a distortion transformation network established in an embodiment of the present application. Figure 4 As shown, the structure of U-Net is based on multiple convolutional layers with skip connections 410 ( Figure 4 The U-Net is formed in the form of an encoder-decoder (indicated by dotted lines in the figure). In one embodiment, each convolution block in the encoder and decoder includes a three-dimensional convolution layer with a kernel size of 3×3×3 and a LeakyReLU activation layer with a parameter of 0.2. The encoder includes 4 downsampling blocks 420 with 32 convolution filters to reduce the input size by a factor of 2. The decoder includes 4 upsampling blocks 430, followed by 3 refined convolution layers 440 (filter size is 16). In addition, the features learned at each scale can be transferred by jumping between the encoder and decoder. The output of U-Net is a deformation field distortion field φ i We transform the learned distortion field φ through the spatial transformer network STN i Applied to two FOD images in opposite directions, an average distortion-free image can be finally obtained by synthesis.

[0078] Combination Figure 3 and Figure 4 As shown, after the coarse correction by the affine transformation network 300, the first coarse correction image I a The first FOD image and the second FOD image are input into the distortion transformation network 400, so that the distortion field between the first FOD image and the second FOD image can be obtained. Figure 4 , and then according to the distortion field φ i First FOD image and the second FOD image Distortion correction is performed to obtain a first corrected image Positive PE deformed FOD and a second corrected image Negative PE deformed FOD. i represents the i-th group of training images of multiple resolutions obtained when the multi-resolution strategy is used to train the network, which will be described in detail later. When the multi-resolution strategy is not used, i = 1, then Figure 4 φ i-1 Does not exist.

[0079] In some embodiments, during the training process, the loss function of the distortion transform network 400 is:

[0080]

[0081] Where φ represents the distortion field, L sim is the similarity loss function, I + Represents the first FOD image, I _ represents the second FOD image, represents a first corrected image corresponding to the first FOD image, represents the second corrected image corresponding to the second FOD image, L reg represents the regularized loss function used to optimize the distortion field, p represents the image voxel position, and Ω represents the entire image domain.

[0082] Combination Figure 4 , the loss function L reg Used to optimize the distortion field φ. Similarity loss function L sin Used to measure the similarity between the first corrected image Positive PE deformed FOD and the second corrected image Negative PE deformed FOD.

[0083] In some embodiments, the similarity loss function L in the affine transformation network 300 and the distortion transformation network 400 is sim The reason for using MSD is that images from opposite PE directions have the same modality and acquisition parameters, so it is suitable to use MSD to calculate the similarity. In other embodiments, the similarity loss function L sim Functions commonly used in this field, such as MI and NCC, may be used.

[0084] In some embodiments, the affine transformation network 300 is trained first, and after the training, the parameters or weights of the affine transformation network 300 are frozen, and then the distortion transformation network 400 is trained.

[0085] The image correction method of the present application uses the affine transformation network 300 and the distortion transformation network 400 at the same time, and corrects the dMRI image distortion in an end-to-end, unsupervised learning method, which can achieve a better correction effect.

[0086] In some embodiments, the image correction method of the present application further includes: adopting a multi-resolution strategy to train the distortion transformation network 400 and the affine transformation network 300. The following first describes the use of a multi-resolution strategy to train the distortion transformation network 400. In this process, it is assumed that the affine transformation network 300 has been trained, and its parameters and weights are frozen and will not change with the training process.

[0087] Figure 5 The multi-resolution training strategy in the image correction method of an embodiment of the present application is used to illustrate the multi-resolution training strategy. In order to achieve multi-resolution, firstly, a plurality of first sub-FOD images with different resolutions are obtained from the first FOD image and the second FOD image. and multiple second sub-FOD images in, and The first FOD image and the second FOD image originally obtained have the highest resolution. Conventional means in the art can be used to obtain the first sub-FOD image and the second sub-FOD image with reduced resolution.

[0088] In one embodiment, the first FOD image and the second FOD image are processed by a Gaussian blur method to obtain a plurality of first sub-FOD images and a plurality of second sub-FOD images at different resolutions.

[0089] Furthermore, the plurality of first sub-FOD images and the plurality of second sub-FOD images may be sorted from low to high according to resolution to obtain a plurality of groups of training images, each group of training images including a pair of first sub-FOD images and second sub-FOD images with opposite phase encoding directions. The resolution of the first sub-FOD image and the second sub-FOD image in each group is the same. Figure 5 As shown, the training image corresponding to resolution 1 is The training image corresponding to resolution 2 is And so on.

[0090] Figure 5 Each DenseNet in i The module corresponds to a distortion transformation network trained by the first sub-FOD image and the second sub-FOD image of the current resolution i.

[0091] When using multi-resolution training strategy, the first set of training images Take φ1 as an example and input it into the distortion transform network DenseNet1 module as the current group training image. The DenseNet1 module outputs the current distortion field φ1. Combine the current distortion field φ1 and the next set of training images Applied to the distortion transform network DenseNet2 module, the DenseNet2 module outputs the next distortion field φ2. It should be noted that the DenseNet2 module and the DenseNet1 module are representations of the same distortion transform network at different training stages.

[0092] Specifically, in Figure 5 The zoomed in image shows the DenseNet corresponding to the multi-resolution medium resolution i i The specific structure of the module. It shows that at the current resolution i, the previous resolution i-1 is composed of DenseNet i-1 The output distortion field φ i-1 Participated in DenseNet at the current resolution i i In this embodiment, DenseNet i-1 The parameters or weights in are frozen and will not change with the current training process. The distortion field φ i-1 Spatial Transformation Network Applied to the first FOD image I + Then as the first FOD image at resolution i Figure 5 Also shown is an affine transformation network AffineNet, which is used to transform the first FOD image I + and the second FOD image I - Get the trained affine matrix A. The affine matrix A passes through the spatial transformation network Applied to the second FOD image I - , and the distortion field φ i-1 Spatial Transformation Network Then as the second FOD image at resolution i

[0093] like Figure 5 As shown, DenseNet at resolution i i The output and distortion field φ i-1 Through the spatial transformation network Then the output distortion field φ i After training at multiple resolution levels from low to high, DenseNet n Output target distortion field φ n .

[0094] Going back to step S50, Figure 5 and Figure 2 As shown, according to the target distortion field φ obtained through multi-resolution strategy training n By performing distortion correction on the first FOD image and the second FOD image, better correction results can be obtained.

[0095] Similar to the multi-resolution strategy for training the distortion transformation network 400, in the image correction method of the present application, the multi-resolution strategy can also be used to train the affine transformation network 300. Specifically, refer to Figure 5 The structure from Resolution 1 to the original resolution in the upper middle part, where DenseNet is replaced by AffineNet, can be used to assist in explaining the process of training the affine transformation network 300 using a multi-resolution strategy. The process specifically includes:

[0096] Obtaining a plurality of first sub-FOD images and a plurality of second sub-FOD images at different resolutions from the first FOD image and the second FOD image respectively;

[0097] Acquire multiple groups of training images, each group of training images includes a pair of first sub-FOD images and second sub-FOD images with opposite phase encoding directions;

[0098] Starting from the set of training images with the lowest resolution, the current set of training images is input into the affine transformation network to obtain the current affine matrix, and the current affine matrix and the next set of training images are applied to the affine transformation network through the spatial transformation network to obtain the next affine matrix corresponding to the next set of training images, until the last set of training images is used as the input of the affine transformation network and the target affine matrix is ​​obtained.

[0099] Furthermore, the affine transformation network 300 may be first trained at multiple resolutions, and then its parameters or weights may be frozen and used for multi-resolution training of the distortion transformation network 400 .

[0100] The inventors of the present application selected dMRI data of 100 subjects from HCP. Each dMRI scan included image volumes from 97 gradient directions distributed on three shells, with b values ​​of 1000, 2000, and 3000 s / mm2, respectively. 2 The image size of each scan is 144×168×110, and the isotropic spatial resolution is 1.25 mm. The results obtained by using DiscoNet of the present application are compared with other methods to study the technical effect of the present application.

[0101] In order to quantify the distortion correction performance of the image correction method of the present application, the inventors evaluated the difference between the two PEs processed by DiscoNet. First, the mean square deviation (MSD) of the fractional anisotropy (FA) of the dMRI data generated by the tensor model was measured. After distortion correction, the minimum angular difference (MAD) of the main fiber direction of the FOD data between the two relative PEs was measured. The angular difference is obtained by calculating the minimum angle between the main fiber (largest amplitude) of the positive PE FOD and the top three fibers (by amplitude) of the negative PE FOD.

[0102] The brainstem region is an important area for studying pathological changes of brain diseases such as Alzheimer's disease, and dMRI images will be severely distorted. The correction effect of this area is more clinically significant, so the inventor used the pons region in the brainstem as the region of interest (ROI) in the experiment. In addition, the corpus callosum (cc) is the largest white matter pathway and nerve bundle in the brain, which will be significantly distorted in the dMRI image. The correction effect of the corpus callosum is very important for fiber tracking in white matter related analysis. The entire white matter area (wm) is also under consideration. The inventor of the present application used Freesurfer 6.0 to segment the brainstem region and corpus callosum region in the T1 image, and then corrected the pons in the brainstem and the corpus callosum in the corpus callosum.

[0103] Figure 6 The distortion correction effects of the DiscoNet network proposed in this application are compared with those of some other methods. These methods include topup, S-Net, flow-net, FODReg, and DrC-Net. Figure 6 It can be seen that the fiber bundle trajectories generated by the FOD of the two PEs are still distorted in the cerebellar peduncles after correction by topup, S-Net and flow-net, and their directions are opposite, which will lead to erroneous fiber trajectories in the merged data.

[0104] Figure 7 The quantitative results of the six methods in three ROIs were compared, including MAD and MSD. Compared with other methods, the DiscoNet of this application has smaller MAD and MSD in three ROIs. In the severely distorted pons and cc regions, DiscoNet has better correction performance than DrC-Net, with MADs 1.584 and 0.798 lower, respectively. In the entire white matter region, DiscoNet's MAD and MSD are also better than other methods. In addition, the use of additional Gaussian blur layers reduces overfitting and significantly improves the distortion correction results of various indicators.

[0105] The inventors also compared the time. All six methods were run on the training set and the test set, and the average time required to estimate the distortion field was calculated. The results showed that the methods topup and FODReg took more than an hour to estimate the distortion field of a case. The method of the present application only took about 2 seconds to estimate a distortion field and achieved excellent correction results.

[0106] Figure 8 The image correction device 800 is an unsupervised deep learning-based image correction device for dMRI images according to an embodiment of the present application. The image correction device 800 can be used to perform the image correction method of the present application, but the image correction method of the present application is not limited to being performed using only the image correction device 800.

[0107] refer to Figure 8 The image correction device 800 includes a data acquisition module 810, a FOD image calculation module 820, a correction network module 830, a memory 840 and a processor 850. Among them, the data acquisition module 810 is used to acquire dMRI data, which can be used to execute the step S10 described above. The FOD image calculation module 820 is used to calculate the FOD image based on the dMRI data, which can be used to execute the step S20 described above. The correction network module 830 includes an affine transformation network and a distortion transformation network, which can be used to execute the steps S30 to S50 described above. The memory 840 is used to store instructions that can be executed by the processor. The processor 850 is used to execute the instructions to implement the image correction method of the present application. The image correction device 800 can be used to quickly and effectively perform distortion correction on dMRI images, especially to improve the image correction quality of the brainstem region.

[0108] It should be understood that the embodiments described above are only illustrative. The embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or any combination thereof. For hardware implementation, the processor may be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, and / or other electronic units designed to perform the functions described herein, or a combination thereof.

[0109] Some aspects of the present application may be performed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The above hardware or software may be referred to as "data blocks", "modules", "engines", "units", "components" or "systems". The processor may be one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DAPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, or combinations thereof. In addition, various aspects of the present application may be expressed as computer products located in one or more computer-readable media, which include computer-readable program codes. For example, computer-readable media may include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, tapes ...), optical disks (e.g., compact disks CDs, digital versatile disks DVDs ...), smart cards, and flash memory devices (e.g., cards, sticks, key drives ...).

[0110] A computer-readable medium may include a propagated data signal containing computer program code, such as in baseband or as part of a carrier wave. The propagated signal may have a variety of manifestations, including electromagnetic, optical, etc., or a suitable combination. A computer-readable medium may be any computer-readable medium other than a computer-readable storage medium, which may be connected to an instruction execution system, device or apparatus to communicate, propagate or transmit a program for use. The program code on the computer-readable medium may be propagated via any suitable medium, including radio, cable, fiber optic cable, radio frequency signal, or similar medium, or any combination of the above mediums.

[0111] The basic concepts have been described above. Obviously, for those skilled in the art, the above invention disclosure is only used as an example and does not constitute a limitation of the present application. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements and amendments to the present application. Such modifications, improvements and amendments are suggested in the present application, so such modifications, improvements and amendments still belong to the spirit and scope of the exemplary embodiments of the present application.

[0112] At the same time, the present application uses specific words to describe the embodiments of the present application. For example, "one embodiment", "an embodiment", and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of the present application. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more in different positions in this specification does not necessarily refer to the same embodiment. In addition, some features, structures or characteristics in one or more embodiments of the present application can be appropriately combined.

[0113] In some embodiments, numbers describing the number of components and attributes are used. It should be understood that such numbers used in the description of the embodiments are modified by the modifiers "about", "approximately" or "substantially" in some examples. Unless otherwise specified, "about", "approximately" or "substantially" indicate that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification are approximate values, which may change according to the required features of individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and adopt the general method of retaining the digits. Although the numerical domains and parameters used to confirm the breadth of the range in some embodiments of the present application are approximate values, in specific embodiments, the setting of such numerical values ​​is as accurate as possible within the feasible range.

Claims

1. An image correction method based on unsupervised deep learning for dMRI images, characterized in that: include: Acquiring dMRI data, wherein the dMRI data includes data in two opposite phase encoding directions; Calculate a FOD image according to the dMRI data, the FOD image comprising a first FOD image and a second FOD image, wherein the first FOD image corresponds to a first phase encoding direction, the second FOD image corresponds to a second phase encoding direction, and the first phase encoding direction is opposite to the second phase encoding direction; Establishing an affine transformation network, and inputting the first FOD image and the second FOD image into the affine transformation network to obtain a first coarse correction image; Establishing a distortion transformation network, inputting the first coarse correction image and the second FOD image into the distortion transformation network, and obtaining a distortion field between the first FOD image and the second FOD image; Distortion correction is performed on the first FOD image and the second FOD image according to the distortion field to obtain a first corrected image and a second corrected image respectively.

2. The image correction method according to claim 1, characterized in that: Before calculating the FOD image according to the dMRI data, the method further includes: Processing the dMRI data using an eddy current correction method to obtain processed dMRI data; The step of calculating the FOD image according to the dMRI data includes: calculating the FOD image according to the processed dMRI data.

3. The image correction method according to claim 1, characterized in that: The affine transformation network includes several convolution modules, a maximum pooling layer and a fully connected layer, and the output of the fully connected layer includes a parameter set μ, μ=(θ x ,i y ,i z ,t x ,t y ,t z ), Among them, x, y, z are three coordinate axes, θ x ,θ y ,θ z Respectively represent the rotation angle of the corresponding coordinate axis, t x ,t y ,t z The steps of inputting the first FOD image and the second FOD image into the affine transformation network to obtain a first coarse correction image respectively represent the translation distance of the corresponding coordinate axis, and comprising: The affine matrix A is calculated using the following formula: A=R x °R y °R z °T The affine matrix A is applied to the first FOD image through a spatial transformation network to obtain the first coarse corrected image.

4. The image correction method according to claim 3, characterized in that: The loss function of the affine transformation network is: L affine (A,I + ,I - )=L sim (A°I - ,I + ), Where A is the affine matrix, L sim is the similarity loss function, I + represents the first FOD image, I -m Represents the second FOD image.

5. The image correction method according to claim 3, characterized in that: Also includes: The affine transformation network is trained using a multi-resolution strategy, including: Obtaining a plurality of first sub-FOD images and a plurality of second sub-FOD images at different resolutions from the first FOD image and the second FOD image respectively; Acquire multiple groups of training images, each group of training images includes a pair of first sub-FOD images and second sub-FOD images with opposite phase encoding directions; Starting from a group of training images with the lowest resolution, the current group of training images is input into the affine transformation network to obtain a current affine matrix, and the current affine matrix and the next group of training images are applied to the affine transformation network through the spatial transformation network to obtain a next affine matrix corresponding to the next group of training images, until the last group of training images is used as input to the affine transformation network, and a target affine matrix is ​​obtained.

6. The image correction method according to claim 1, characterized in that: The distortion transformation network includes a U-Net, and the output of the U-Net is the distortion field.

7. The image correction method according to claim 6, characterized in that: The loss function of the distortion transformation network is: THE joint (φ,I + ,THE - )=L sim (THE + °φ,I - °φ -1 )+λL reg (φ), Where, φ represents the distortion field, L sim is the similarity loss function, I + represents the first FOD image, I - represents the second FOD image, I + °φ represents the first corrected image corresponding to the first FOD image, I - °φ -1 represents the second corrected image corresponding to the second FOD image, L reg represents the regularized loss function used to optimize the distortion field, p represents the image voxel position, and Ω represents the entire image domain.

8. The image correction method according to claim 6, characterized in that: The distortion transform network is trained using a multi-resolution strategy, including: Obtaining a plurality of first sub-FOD images and a plurality of second sub-FOD images at different resolutions from the first FOD image and the second FOD image respectively; Acquire multiple groups of training images, each group of training images includes a pair of first sub-FOD images and second sub-FOD images with opposite phase encoding directions; Starting from a group of training images with the lowest resolution, the current group of training images is input into the distortion transformation network to obtain a current distortion field, and the current distortion field and the next group of training images are applied to the distortion transformation network through a spatial transformation network to obtain a next distortion field corresponding to the next group of training images, until the last group of training images is used as input to the distortion transformation network to obtain a target distortion field.

9. The image correction method according to claim 5 or 8, characterized in that: The steps of respectively obtaining multiple first sub-FOD images and multiple second sub-FOD images at different resolutions from the first FOD image and the second FOD image include: using a Gaussian blur method to process the first FOD image and the second FOD image to obtain multiple first sub-FOD images and multiple second sub-FOD images at different resolutions.

10. The image correction method according to claim 6, characterized in that: Also includes: While training the distortion transformation network, parameters of the affine transformation network are frozen.

11. The image correction method according to claim 1, characterized in that: Also includes: The first corrected image and the second corrected image are synthesized to obtain a corrected dMRI image.

12. An image correction device based on unsupervised deep learning for dMRI images, characterized in that: include: A data acquisition module, used for acquiring dMRI data; A FOD image calculation module, used for calculating the FOD image according to the dMRI data; Correction network module, including affine transformation network and distortion transformation network; a memory for storing instructions executable by a processor; A processor, configured to execute the instructions to implement the method according to any one of claims 1 to 11.

13. A computer readable medium storing computer program code, characterized in that: The computer program code implements the method according to any one of claims 1 to 11 when executed by a processor.

Images