Systems and methods for deblurring medical images using deep neural networks

Abstract

The present invention is entitled "System and method for deblurring medical images using deep neural networks". The present invention provides methods and systems for deblurring medical images using deep neural networks. In one embodiment, a method for deblurring a medical image includes: receiving a blurred medical image and one or more acquisition parameters corresponding to the acquisition of the blurred medical image; incorporating the one or more acquisition parameters into a trained deep neural network; and mapping the blurred medical image to a deblurred medical image by the trained deep neural network. The deep neural network can thereby receive at least partial information about the type, degree and / or spatial distribution of blur in the blurred medical image, so that the trained deep neural network can selectively deblur the received blurred medical image.

Description

Technical Field

[0001] Embodiments of the subject matter disclosed herein relate to processing medical images, such as magnetic resonance images (MRI), computed tomography (CT) images, X-ray images, ultrasound images, etc., and more particularly, to reducing blur in medical images using deep neural networks. Background Art

[0002] Medical imaging systems (such as magnetic resonance imaging (MRI) systems, computed tomography (CT) systems, positron emission tomography (PET) systems, X-ray systems, ultrasound systems, etc.) are widely used to obtain internal physiological information of a subject (e.g., a patient). Medical images obtained by these imaging modalities may include one or more types of blur that may be caused by transient signal acquisition, tissue / patient specific effects, imaging equipment properties, etc. Blurring may reduce the resolution and diagnostic quality of medical images, and may further reduce the efficacy of downstream image processing methods that may have been trained on clear, non-blurred medical images. Deep learning methods have been proposed for deblurring images, however current deep learning methods perform inconsistently in deblurring and generally fail to produce a sufficient degree of deblurring. Therefore, it is generally desirable to explore deep learning techniques to identify new methods for consistently deblurring medical images. Summary of the invention

[0003] The present disclosure at least partially solves the above-mentioned problems. In one embodiment, the present disclosure provides a method for deblurring a medical image using a deep neural network. The method includes: receiving a blurred medical image and one or more acquisition parameters corresponding to the acquisition of the blurred medical image; incorporating the one or more acquisition parameters into a trained deep neural network; and mapping the blurred medical image to a deblurred medical image by the trained deep neural network. By receiving both the blurred medical image and the acquisition parameters, wherein the acquisition parameters correspond to one or more parameters of the imaging system used during the acquisition of the blurred medical image, the deep neural network can be at least partially informed of the type, degree and / or distribution of blur present in the blurred medical image, thereby enabling the deep neural network to map the blurred medical image to the corresponding deblurred medical image with a greater degree of consistency, even when there are multiple types of blur in the medical image, or when the degree / type of blur varies with pixels / voxels.

[0004] The above advantages and other advantages and features of the present specification will be apparent from the following detailed description when taken alone or in conjunction with the accompanying drawings. It should be understood that the above summary is provided to introduce a selection of concepts further described in the detailed description in a simplified form. This is not meant to identify key or essential features of the claimed subject matter, the scope of which is uniquely defined by the claims that follow the detailed description. In addition, the claimed subject matter is not limited to specific implementations that address any disadvantages mentioned above or in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Various aspects of the present disclosure may be better understood by reading the following detailed description and referring to the accompanying drawings, in which:

[0006] Figure 1 is a schematic diagram illustrating an image processing system for deblurring a medical image using a deep neural network and acquisition parameter transformation according to an exemplary embodiment;

[0007] Figure 2 is a diagram showing an exemplary embodiment of the Figure 1 A schematic diagram of the layout of a first embodiment of a deep neural network and acquisition parameter transformation used in an image processing system;

[0008] Figure 3 is a diagram showing an exemplary embodiment of the Figure 1 A schematic diagram of the layout of a second embodiment of a deep neural network and acquisition parameter transformation used in an image processing system;

[0009] Figure 4 is a diagram showing an exemplary embodiment of the Figure 1 A schematic diagram of the architecture of the deep neural network used in the system;

[0010] Figure 5 is a diagram showing an exemplary embodiment of the Figure 1 A schematic diagram of the acquisition parameter transformation architecture used in the system;

[0011] Figure 6 is a diagram showing an exemplary embodiment of the Figure 1 A schematic diagram of the acquisition parameter transformation architecture used in the system;

[0012] Figure 7 is a flow chart illustrating a method for deblurring a medical image using a deep neural network and acquisition parameter transformation according to an exemplary embodiment;

[0013] Figure 8 The signal attenuation due to T2 and the corresponding point spread function are shown.

[0014] Fig. 9 A comparison between a blurred medical image and a deblurred medical image is shown according to an exemplary embodiment.

[0015] The accompanying drawings illustrate specific aspects of the described systems and methods for deblurring medical images using a deep neural network and one or more acquisition parameter transformations. Together with the following description, the accompanying drawings illustrate and explain the structures, methods, and principles described herein. In the accompanying drawings, the sizes of components may be exaggerated or otherwise modified for clarity. Well-known structures, materials, or operations are not shown or described in detail to avoid obscuring aspects of the described components, systems, and methods. DETAILED DESCRIPTION

[0016] Medical images are prone to blurring, which limits image resolution and reduces diagnostic quality. In one example, blurring in magnetic resonance (MR) images may be caused by signal modulation during transient acquisition and / or variable flip angles and / or relaxation effects during readout columns. An example is T2 blurring in single-shot fast spin echo. In another example, in CT imaging, blurring may occur as a function of detector size, number of detectors, source-to-detector distance, collimator geometry, slice thickness, dose, etc. In another example, in PET imaging, blurring may occur as a function of detector type, detector geometry, ring radius, positron range (isotope type), depth / line of response, and voxel activity. In yet another example, in ultrasound imaging, blurring may occur as a function of transducer (crystal and array) geometry, center frequency, ringing, spatial pulse length, focal depth, beam apodization, sidelobe amplitude, slice thickness, etc. Thus, blur occurs in a wide range of medical imaging modalities and can occur as a function of one or more acquisition parameters, where, as used herein, acquisition parameters will be understood to include one or more parameters of the imaging system, the imaged tissue, and / or the environmental conditions (temperature, humidity, ambient magnetic field strength, etc.) when the medical image is acquired.

[0017] The process of deblurring medical images can be complicated when the type / source of blur is unknown and / or when the degree of blur varies throughout the image. Take MR as an example. In single-shot fast spin echo, blur may be caused by a combination of low (and potentially variable) refocusing angles and other factors, including tissue-specific relaxation times, view ordering, partial volume effects, B1 field inhomogeneities, etc. In some cases, each voxel / pixel may have a unique blur function. Deblurring is an ill-posed inverse problem that is difficult to solve using conventional methods. In addition, conventional methods have the disadvantage of noise amplification, resulting in clearer images but more noise.

[0018] The following description relates to various embodiments for deblurring medical images using a deep neural network, which may at least partially address the above-mentioned problems. Specifically, a medical image is acquired by an imaging system, wherein one or more acquisition parameters are applied during the acquisition of the medical image. The acquisition parameters may include various imaging system settings for acquiring the medical image. For example, the acquisition parameters of an MR image may include one or more of the following: echo train length, repetition time (TR), echo time (TE), flip angle, inversion time, etc. The acquisition parameters may also include one or more of the following: size of the medical image, voxel spatial size (the spatial volume represented by each voxel in the medical image), sampling pattern, acquisition order, acceleration factor, fat saturation setting (on or off), B0 shim mode selection, RF drive mode selection, physiological signals, physiological state, image reconstruction parameters, etc. The acquired medical image is then processed using a trained deep neural network and one or more acquisition parameter transformations, and the acquired medical image may include blur. The one or more acquisition parameters applied during the acquisition of the medical image are input into the acquisition parameter transformation, and the output from the acquisition parameter transformation can be combined into the deep neural network in various ways. By incorporating the output of the acquisition parameter transformation into a deep neural network, information about the type, degree, and distribution of blur within the medical image can be determined by the deep neural network and used to deblur the medical image in a more consistent manner without increasing noise within the medical image.

[0019] In some embodiments, the output of the acquisition parameter transform may include a point spread function based on one or more acquisition parameters, wherein a deconvolution kernel and / or a convolution kernel used in the deep neural network may be based on the point spread function determined by the acquisition parameter transform. In this manner, a deblurred / sharp medical image may be generated from a corresponding blurred medical image. Furthermore, the deep neural network and the acquisition parameter transform (in embodiments where the acquisition parameter transform includes a neural network) may be trained and then put into use.

[0020] As used herein, deblurring is the process of removing blur artifacts from an image, such as blur caused by defocus aberrations or motion blur or other types of blur artifacts. In general, blur can be modeled as a convolution of a (space-varying and / or time-varying) point spread function on a hypothetical sharp input image, where both the sharp input image and the point spread function may be unknown. Therefore, deblurring includes a process of at least partially inverting or mitigating the convolution of a sharp image by one or more point spread functions to obtain or approximate the sharp input image and is therefore referred to as an inverse problem. Therefore, a sharp image (or a sharp medical image) can be produced by deblurring a blurred image (or a blurred medical image), and conversely, a blurred image can be produced by introducing one or more blur artifacts into the sharp image. It should be understood that in some cases, the sharp medical image can be directly acquired using an imaging system.

[0021] refer to Figure 1 , a medical image processing system 100 according to an exemplary embodiment is shown. In some embodiments, the medical image processing system 100 is incorporated into a medical imaging system (e.g., an MRI system, a CT system, an X-ray system, a PET system, an ultrasound system, etc.). In some embodiments, the medical image processing system 100 is disposed at a device (e.g., an edge device, a server, etc.) that is communicatively coupled to the medical imaging system via a wired and / or wireless connection. In some embodiments, the medical image processing system 100 is disposed at a separate device (e.g., a workstation) that can receive images from the medical imaging system or from a storage device that stores images generated by the medical imaging system. The medical image processing system 100 may include an image processing system 31, a user input device 32, and a display device 33.

[0022] The image processing system 31 includes a processor 204 configured to execute machine-readable instructions stored in a non-transitory memory 206. The processor 204 can be single-core or multi-core, and the program executed thereon can be configured to perform parallel or distributed processing. In some embodiments, the processor 204 can optionally include separate components distributed in two or more devices, which can be remotely located and / or configured for coordinated processing. In some embodiments, one or more aspects of the processor 204 can be virtualized and executed by a remotely accessible networked computing device configured with a cloud computing configuration.

[0023] The non-transitory memory 206 may store a deep neural network module 208, an acquisition parameter transformation module 210, a training module 212, and medical image data 214. The deep neural network module 208 may include one or more deep neural networks, the one or more deep neural networks including a plurality of parameters (including weights, biases, activation functions), and instructions for implementing the one or more deep neural networks to receive a blurred medical image and map the blurred medical image to an output, wherein a deblurred medical image corresponding to the blurred medical image may be generated from the output. For example, the deep neural network module 208 may store instructions for implementing a neural network (such as Figure 4 4. The deep neural network module 208 may include instructions for a convolutional neural network (CNN) of the CNN architecture 400 shown. The deep neural network module 208 may include trained and / or untrained neural networks, and may also include various data or metadata related to one or more neural networks stored therein.

[0024] The non-transitory memory 206 also stores an acquisition parameter transformation module 210, in which one or more trained and / or untrained acquisition parameter transformations and associated data and / or metadata may be stored. The acquisition parameter transformation may be configured to map acquisition parameters to outputs, wherein the outputs may be incorporated into one or more deep neural networks of the deep neural network module 208. In some embodiments, the acquisition parameter transformation may be configured to predict one or more point spread functions based on one or more input acquisition parameters. In some embodiments, the acquisition parameter transformation may include an analytical function or model, wherein the function / model receives one or more acquisition parameters as model independent variables and generates an output using a function mapping from the acquisition parameters. In one example, the acquisition parameter transformation may include a Fourier transform for mapping an MRI echo sequence to an intensity point spread function.

[0025] The non-transitory memory 206 may also store a training module 212 that includes instructions for training one or more of the deep neural network stored in the deep neural network module 208 and / or the acquisition parameter transformation stored in the acquisition parameter transformation module 210. The training module 212 may include instructions that, when executed by the processor 204, cause the image processing system 31 to perform one or more of the steps of the method 700 discussed in detail below. In some embodiments, the training module 212 includes instructions for implementing one or more gradient descent algorithms, applying one or more loss functions and / or training routines for adjusting parameters of one or more deep neural networks of the deep neural network module 208 and / or the acquisition parameter transformation of the acquisition parameter transformation module 210. In some embodiments, the training module 212 includes instructions for intelligently selecting training data pairs from the medical image data 214. In some embodiments, the training data pairs include corresponding blurred and deblurred / clear medical image pairs of the same anatomical region. In some embodiments, the training module 212 includes instructions for generating training data pairs by applying / adding one or more blur artifacts to a clear medical image to produce a blurred medical image. In some embodiments, the training module 212 is not provided at the image processing system 31. The deep neural network module 208 includes a trained and validated network.

[0026] The non-transitory memory 206 also stores medical image data 214. The medical image data 214 includes, for example, MR images captured from an MRI system, ultrasound images acquired by an ultrasound system, and the like. For example, the medical image data 214 may store blurred and / or sharp medical images. In some embodiments, the medical image data 214 may include a plurality of training data pairs including blurred and sharp medical image pairs. In some embodiments, the medical image data may include one or more blur point spread functions for generating blurred medical images from sharp medical images.

[0027] In some embodiments, non-transitory memory 206 may include components disposed on two or more devices that may be remotely located and / or configured for coordinated processing. In some embodiments, one or more aspects of non-transitory memory 206 may include a remotely accessible networked storage device configured in a cloud computing configuration.

[0028] The medical image processing system 100 may further include a user input device 32. The user input device 32 may include one or more of a touch screen, a keyboard, a mouse, a trackpad, a motion sensing camera, or other devices configured to enable a user to interact with and manipulate data within the image processing system 31. For example, the user input device 32 may enable a user to select a medical image to perform deblurring on.

[0029] The display device 33 may include one or more display devices utilizing virtually any type of technology. In some embodiments, the display device 33 may include a computer monitor and may display unprocessed and processed MR images and / or parametric maps. The display device 33 may be combined with the processor 204, the non-transitory memory 206, and / or the user input device 32 in a shared housing, or may be a peripheral display device and may include a monitor, a touch screen, a projector, or other display device known in the art that enables a user to view medical images and / or interact with various data stored in the non-transitory memory 206.

[0030] It should be understood that Figure 1 The medical image processing system 100 shown in FIG. 1 is for illustration and not limitation. Another suitable image processing system may include more, fewer, or different components.

[0031] Go to Figure 2, a schematic diagram of a first embodiment of a deblurring process 200 for deblurring a medical image is shown. The deblurring process 200 may be implemented by the medical image processing system 100 to at least partially remove one or more blurring artifacts from a blurred medical image. The deblurring process 200 is used to illustrate an embodiment of a deblurring system, wherein acquisition parameters 302 may be mapped to outputs via an acquisition parameter transformation 304, which are then coupled to a deep neural network 324 via an input layer 322 and used to remove one or more blurring artifacts from a blurred image 310. By mapping the acquisition parameters to the outputs and coupling the outputs to the deep neural network 324, the deep neural network 324 may receive at least partial information about the extent, type, and distribution of blurring artifacts in the blurred image 310, as blurring artifacts may occur as a function of one or more of the acquisition parameters.

[0032] The deblurring process 200 includes an acquisition parameter transform 304 that receives the acquisition parameters 302 as input and maps the acquisition parameters 302 to one or more outputs, where the one or more outputs can be incorporated into an input layer 322 of a deep neural network 324. In some embodiments, the acquisition parameter transform 304 includes a trained neural network that receives the one or more acquisition parameters at an input layer and maps the one or more acquisition parameters to a first output including a plurality of values, where the plurality of values ​​can be equal in number to a plurality of pixels or voxels of the blurred image 310. In some embodiments, the acquisition parameter transform 304 includes an analytical mapping, such as an equation or model, that receives the one or more acquisition parameters and maps to a first output. Figure 2 In the illustrated embodiment, the acquisition parameter transform 304 may produce an equal number of output values ​​as there are pixel values ​​or voxel values ​​in the blurred image 310, thereby enabling each of the multiple pixels or voxels of the blurred image 310 to have at least a single output value from the acquisition parameter transform 304 cascaded therewith.

[0033] The input layer 322 includes the acquisition layer 306 generated by the output of the acquisition parameters 304, the blurred image 310, and optionally the parameter maps 308, which may include one or more parameter maps of the imaging space, such as B1 + Field, B oField, T1 map, T2 map, gradient nonlinearity map, coding error map, motion map (where motion / velocity can be estimated or measured), etc. The input layer 322 is received by the deep neural network 324 and mapped to the deblurred image 320. The deep neural network 324 includes learned convolution filters 314 (learned during the training process) and learned deconvolution filters 318 (learned during the training process). By propagating data from the input layer 322 through the convolution layers and deconvolution layers of the deep neural network 324, the deblurred image 320 is produced.

[0034] Acquisition parameters 302 include one or more settings used by the imaging system during acquisition of the blurred image 310 and / or one or more physiological properties and / or environmental conditions of the imaged tissue / patient. For example, if the blurred image 310 is an MR image, the acquisition parameters 302 may include one or more of echo train length, repetition time, echo time, echo spacing, target flip angle, sampling pattern, acquisition order, physiological signals, or other parameters / settings used by the MRI system during acquisition of the blurred image 310 or associated with the imaged patient. The blurred image 310 is a medical image of an anatomical region, including a plurality of values, each of which may be referred to as a pixel (for a 2D image) or a voxel (for a 3D image). The blurred image 310 includes one or more blurring artifacts, which may generally be described as a spread / smearing of intensity values ​​over a larger area of ​​the image than the original / underlying anatomical region. Blurring may be mathematically described as a convolution of the original non-blurred image with a point spread function, where the point spread function is used to spread point sources of intensity from the non-blurred image to produce intensity regions that occupy a relatively larger area of ​​the image. For example, the blurred image 310 includes an MR image of a human abdomen that includes spatially heterogeneous blur, where the degree and / or type of blur varies based on the pixels / voxels within the blurred image 310. That is, the blurred image 310 may include a first region having a first degree of blur and a second region including a second degree of blur, where the first degree of blur and the second degree of blur and the type of blur are different.

[0035] The acquisition parameters 302 may include vectors or scalars and may be input into the acquisition parameter transform 304. The acquisition parameter transform 304 is configured to map the acquisition parameters 302 to one or more outputs, which may be incorporated into the deep neural network 324 and / or concatenated with the parameter map 308 and the blurred image 310 and then input into the deep neural network 324. In some embodiments, multiple outputs may be generated by multiple acquisition parameter transforms, wherein the acquisition parameter transforms may include functions, neural networks, and / or numerical models. Each output generated by one or more acquisition parameter transforms may be incorporated into the deep neural network 324. In some embodiments, each output of each acquisition parameter transform in the multiple acquisition parameter transforms may be input into the deep neural network 324 via the input layer 322.

[0036] The acquisition layer 306 includes one or more feature maps / layers, each layer having the same size as the blurred image 310, thereby enabling direct concatenation of the acquisition layer 306 and the blurred image 310. In some embodiments, the acquisition parameter transform 304 may receive an acquisition parameter 302 including a single scalar value, and may map the single scalar value to a plurality of values, wherein the plurality of values ​​is equal in number to a plurality of pixels in the blurred image 310. In some embodiments, the acquisition parameter transform 304 may map the plurality of acquisition parameters 302 to a plurality of acquisition parameter layers. In one example, if the blurred image 310 includes a 512×512×120 voxel volume, the acquisition layer 306 may include one or more layers, each layer having a size of 512×512×120, thereby enabling direct concatenation with the blurred image 310. By mapping the acquisition parameters 302, which may include scalars (single acquisition parameter) or vectors (multiple acquisition parameters), to a higher dimensional space, information about the spatial heterogeneity of blur within the imaging space can be appended to the blurred image 310 and then input into the deep neural network 324, thereby providing the deep neural network 324 with additional information about the spatial distribution of blur in the blurred image 310, which may occur as a function of one or more acquisition parameters 302.

[0037] The parametric map 308 may include one or more parameters of the imaging space, where the imaging space corresponds to the blurred image 310. In some embodiments, the parametric map 308 may include one or more physical properties / parameters of the region imaged by the blurred image 310. In one example, the parametric map 308 may include magnetic field strength, proton density field map, density field map, B1 + Field inhomogeneity mapping, etc.

[0038] The input layer 322 (which includes the acquisition layer 306, the parameter map 308, and the blurred image 310) may be propagated through multiple layers within the deep neural network 324 to map the intensity values ​​of the blurred image 310 to the intensity values ​​of the deblurred image 320. The deep neural network 324 includes the learned convolution filters 314 and the learned deconvolution filters 318. The deep neural network 324 may also include one or more densely connected layers (not shown) as well as one or more pooling layers (not shown), one or more upsampling layers (not shown), and one or more ReLU layers (not shown), or any layers conventional in the field of machine learning.

[0039] The output of deep neural network 324 may be used to produce a deblurred image 320 that includes an image of the same anatomical region as blurred image 310 , but in which at least one or more blurring artifacts are at least partially removed.

[0040] Go to Figure 3 , a schematic diagram of a second embodiment of a deblurring process 300 for deblurring a medical image is shown. Elements of the deblurring process 300 previously introduced and discussed in previous figures may retain their numbering. The deblurring process 300 may be implemented by the medical image processing system 100 to at least partially remove one or more blur artifacts from a blurred medical image. The deblurring process 300 is used to illustrate an embodiment of a deblurring system, wherein acquisition parameters 302 may be mapped to one or more values ​​via an acquisition parameter transform 304, which may then be used to set one or more convolution filter weights and / or deconvolution filter weights of a deep neural network 324. The convolution filter weights and / or deconvolution filter weights set based on the output of the acquisition parameter transform 304 may enable the deep neural network to be informed of the type / degree of blur present in the blurred image 310. By applying convolution filters and deconvolution filters specifically selected for a given set of acquisition parameters, one or more blur artifacts may be more consistently and more thoroughly removed from the blurred image 310. By mapping the acquisition parameters to outputs and incorporating the outputs into the deep neural network 324, the deep neural network 324 may receive at least partial information about the extent, type, and distribution of blur artifacts in the blurred image 310, since blur artifacts may occur as a function of one or more of the acquisition parameters.

[0041] The deblurring process 300 includes an acquisition parameter transform 304 that receives the acquisition parameters 302 as input and maps the acquisition parameters 302 to one or more outputs, wherein the one or more outputs can be directly incorporated into the deep neural network 324 by setting one or more parameters of the deep neural network 324 based on the outputs (see Figure 6For a more detailed discussion). In some embodiments, the acquisition parameter transform 304 includes a Fourier transform, and a first output of the acquisition parameter transform 304 includes a point spread function corresponding to the blur present in the blurred image 310 acquired using the one or more acquisition parameters 302.

[0042] The input layer 322 includes the blurred image 310 and optionally the parametric map 308, which may include one or more parametric maps of the imaging space, such as B1 + Field, B o field, T1 map, T2 map, gradient nonlinearity map, encoding error map, motion map (where motion / velocity can be estimated or measured), etc. The input layer 322 is received by the deep neural network 324 and mapped to the deblurred image 320.

[0043] The deep neural network 324 includes a convolution filter 312 (set based on the output from the acquisition parameter transform 304), a learned convolution filter 314 (learned during the training process), a deconvolution filter 316 (set based on the output from the acquisition parameter transform 304), and a learned deconvolution filter 318 (learned during the training process). By propagating data from the input layer 322 through the convolution layers and deconvolution layers of the deep neural network 324, a deblurred image 320 is produced.

[0044] The acquisition parameters 302 may include vectors or scalars and may be input into an acquisition parameter transform 304. The acquisition parameter transform 304 is configured to map the acquisition parameters 302 to one or more outputs, which may be incorporated into a deep neural network 324. The outputs from the acquisition parameter transform 304 may be directly incorporated into the deep neural network 324 by setting a convolution filter 312 and / or a deconvolution filter 316 based on the outputs of the acquisition parameter transform 304. In some embodiments, each output incorporated into the deep neural network 324 may be generated by a separate acquisition parameter transform, such that there is a one-to-one correspondence between the output and the acquisition parameter transform. In other embodiments, a single acquisition parameter transform may map the acquisition parameters 302 to multiple outputs, wherein the multiple outputs may each be transmitted via multiple different mechanisms / channels (such as reference Figure 3 The above) is combined into the deep neural network 324.

[0045] The input layer 322 (which includes the parameter map 308 and the blurred image 310) can be propagated through multiple layers within the deep neural network 324 to map the intensity values ​​of the blurred image 310 to the intensity values ​​of the deblurred image 320. The deep neural network 324 includes the convolution filter 312, the learned convolution filter 314, the deconvolution filter 316, and the learned deconvolution filter 318. Therefore, the deep neural network 324 includes parameters / layers learned during training using training data, as well as parameters selected based on the output from the acquisition parameter transform 304.

[0046] The deep neural network 324 includes a convolution filter 312 and a deconvolution filter 316, which may be determined or selected based on the output of the acquisition parameter transform 304. In some embodiments, the acquisition parameter transform 304 maps one or more acquisition parameters to a plurality of values, wherein the plurality of values ​​include a plurality of weights, and wherein one or more filters within the convolution filter 312 and / or the deconvolution filter 316 are set based on the plurality of weights. In some embodiments, the one or more convolution filters 312 and / or the one or more deconvolution filters 316 are set equal to the output of the acquisition parameter transform 304. In some embodiments, the output of the acquisition parameter transform 304 may include a plurality of weights representing a point spread function or an inverse mapping of a point spread function, the plurality of weights being determined based on the one or more acquisition parameters input into the acquisition parameter transform 304.

[0047] The deep neural network 324 also includes learned convolution filters 314 and learned deconvolution filters 318, which include filters learned during training of the deep neural network 324. By including both parameters learned during the training process and parameters selected / determined based on outputs from one or more acquisition parameter transforms 304, the deep neural network 324 can be enabled to at least partially anticipate the type, degree, and / or distribution of blur artifacts present in the blurred image 310, which can enable a more accurate and consistent mapping of blurred image intensity values ​​to deblurred image intensity values, thereby improving the diagnostic quality of the one or more blurred images and reducing the need to re-image anatomical regions of the patient.

[0048] In some embodiments, the outputs from the plurality of acquisition parameter transformations may be input into the deep neural network 324 via the input layer 322 and by setting one or more convolution filters 312 or deconvolution filters 316 based on the outputs. That is, Figure 2 and Figure 3The embodiments shown may be combined. The set of parameters transformed as part of the input layer 322 may be the same or different than the set of parameters transformed into the filters 312 / 316. There may or may not be overlap. In some embodiments, the acquisition parameter transform 304 may map the acquisition parameters 302 to multiple outputs, each of which is transmitted via multiple different mechanisms / channels (such as Figure 5 and Figure 6 ) are combined into the deep neural network 324.

[0049] Go to Figure 4 , shows a CNN architecture 400 for mapping a blurred medical image to a deblurred medical image based on the output of an acquisition parameter transform. The CNN architecture 400 provides a more detailed description of a deep neural network (such as the deep neural network 324) that can perform deblurring of a blurred medical image. In some embodiments, a subset of the parameters of the CNN architecture 400 can be selected / determined based on the output of one or more acquisition parameter transforms, while other parameters of the CNN architecture 400 can be learned using a training algorithm. For example, Figure 3 As indicated in the illustrated deblurring process 300 , the CNN architecture 400 may be combined with one or more outputs of the acquisition parameter transformation.

[0050] The CNN architecture 400 represents a U-net architecture, which can be divided into an autoencoder portion (descending portion, elements 402b-430) and an autodecoder portion (ascending portion, elements 432-456a). The CNN architecture 400 is configured to receive medical images including one or more blur artifacts, which may include magnetic resonance (MR) images, computed tomography (CT) images, positron emission tomography (PET) images, X-ray images, or ultrasound images. In one embodiment, the CNN architecture 400 is configured to receive data from a blurred medical image (such as the blurred medical image 402a) of an anatomical region including multiple pixels / voxels, and map the input blurred medical image data to a deblurred medical image (such as the deblurred medical image 456b) of the same anatomical region based on the output of the acquisition parameter transformation. The CNN architecture 400 includes a series of mappings from an input image tile 402b that can be received by an input layer through multiple feature mapping maps and finally to an output deblurred medical image 456b that can be generated based on the output of the output layer 456a. In some embodiments, the CNN architecture 400 is configured to receive an output in the form of an acquisition parameter layer from a first acquisition parameter transform and concatenate the acquisition parameter layer data from the blurred medical image 402a, wherein the concatenated input image data and the acquisition parameter layer may be fed into the input tile 402b and propagated through the layers of the CNN architecture 400. In some embodiments, the CNN architecture 400 is configured to set one or more convolution filters and / or one or more deconvolution filters based on the output of the acquisition parameter transform. The CNN architecture 400 may be configured to receive multiple outputs from corresponding multiple acquisition parameter transforms, which may be incorporated into the CNN architecture 400 according to one or more of the above embodiments.

[0051] Various elements comprising the CNN architecture 400 are labeled in the legend 458. As indicated in the legend 458, the CNN architecture 400 includes multiple feature maps (and / or replicated feature maps) connected by one or more operations (indicated by arrows). The arrows / operations receive inputs from an external file or a previous feature map, and transform / map the received inputs to outputs to produce the next feature map. Each feature map may include multiple neurons, wherein in some embodiments, each neuron may receive inputs from a subset of neurons in a previous layer / feature map, and may compute a single output based on the received inputs, wherein the output may be propagated / mapped to a subset or all of the neurons in the next layer / feature map.

[0052] The feature map may be described using the terms length, width, and depth, where each term refers to the number of neurons that comprise the feature map (e.g., a given feature map is how many neurons long, how many neurons wide, and how many neurons deep). The length and width used by the reference feature map correspond to the spatial dimensions of the image being processed, and in some cases may correspond to the number of pixels / voxels of the image. The depth used by the reference feature map may correspond to the number of features in each feature channel.

[0053] The transformations / mappings performed between each feature map are indicated by arrows, where each different type of arrow corresponds to a different type of transformation, as indicated in legend 458. The solid black arrow pointing to the right indicates a 3×3 convolution with a stride of 1, where the output of the 3×3 grid of features from the immediately preceding feature map (where the 3×3 grid extends through all layers of the immediately preceding feature map) is mapped to a single feature at a single depth of the current feature map by performing a dot product between the output / activation of the 3×3 grid of feature channels and a 3×3 filter (where each layer / each unit depth of the immediately preceding feature map contains 9 weights). In some embodiments, the convolution filter weights may be selected based on the output from the acquisition parameter transform. In some embodiments, the convolution filter weights may be learned during the training process. The filter used to perform the 3×3 convolution is referred to herein as a convolution filter (convolution filter / convolutional filter) or a convolution kernel (convolution kernel / convolutional kernel).

[0054] The downward pointing arrow indicates a 2×2 max pooling operation, where the maximum value of a 2×2 grid of feature channels at a single depth from the immediately previous feature map is propagated to a single feature at a single depth in the current feature map, resulting in an output feature map with a spatial resolution reduced by a factor of 4 compared to the immediately previous feature map. In one example, max pooling of a 2×2 activation grid from the immediately previous feature map, where the 2×2 activation grid includes (2, 1.4, 10, 4.4), produces an output of (10), since 10 is the maximum activation value within the 2×2 grid.

[0055] The upward pointing arrows indicate 2×2 up-convolutions with a stride of 2, which include performing a transposed convolution (also referred to herein as a deconvolution) using a deconvolution filter including a plurality of weights (the filter used to perform the transposed convolution is also referred to herein as a deconvolution filter (deconvolutional filter / deconvolution filter)), mapping the output of a single feature channel at each feature depth from the immediately previous feature map to a 2×2 feature grid at a single feature depth in the current feature map, thereby increasing the spatial resolution of the immediately previous feature map by a factor of 4.

[0056] The dashed arrow pointing to the right indicates copying and cropping of a feature map to be concatenated with another feature map that comes later. The cropping enables the dimensions of the copied feature map to match the dimensions of the feature map to be concatenated with the copied feature map. It should be understood that when the dimensions of the first feature map being copied and the dimensions of the second feature map to be concatenated with the first feature map are equal, no cropping may be performed.

[0057] The right-pointing arrow with a hollow head indicates a 1×1 convolution with a stride of 1, where each feature channel in the immediately previous feature map is mapped to a single feature channel of the current feature map, or in other words, where a 1-to-1 mapping of feature channels occurs between the immediately previous feature map and the current feature map. The processing at each feature map may include the convolution and deconvolution described above, as well as activation, where the activation function is a nonlinear function that limits the processed output values ​​to within a bounded range.

[0058] In addition to the operations indicated by the arrows in legend 458, CNN architecture 400 also includes a solid filled rectangle corresponding to a feature map, where the feature map includes a height (e.g., Figure 4 The length from top to bottom shown corresponds to the y spatial dimension in the xy plane), the width ( Figure 4 , assuming that the magnitude is equal to the height, corresponding to the x spatial dimension in the xy plane) and the depth (as Figure 4 , corresponding to the number of features in each feature channel). Similarly, CNN architecture 400 includes a hollow (unfilled) rectangle corresponding to the copied and cropped feature map, where the copied feature map includes a height (e.g. Figure 4 The length from top to bottom shown corresponds to the y spatial dimension in the xy plane), the width ( Figure 4 , assuming that the magnitude is equal to the height, corresponding to the x spatial dimension in the xy plane) and the depth (as Figure 4 The length from left to right shown corresponds to the number of features in each feature channel).

[0059] Starting at the input image tile 402b (also referred to herein as the input layer), data corresponding to the blurred medical image 402a is input and mapped to a first set of features. In some embodiments, the blurred medical image 402a (which may include one or more layers corresponding to one or more features of the image (such as each intensity value of a multi-color image)) may also include one or more cascaded acquisition parameter layers generated by one or more acquisition parameter transformations. In some embodiments, the acquisition parameter layers cascaded with the blurred medical image 402a may indicate the desired / expected type or intensity of blur artifacts at each pixel location of the blurred medical image 402a. The blurred medical image 402a may include a two-dimensional (2D) or three-dimensional (3D) image / map of an anatomical region of a patient. In some embodiments, the input data from the blurred medical image 402a is pre-processed (e.g., normalized) before being processed by the neural network.

[0060] The output layer 456a may include an output layer of neurons, wherein each neuron may correspond to a pixel (or residual) of the predicted deblurred medical image 456b, wherein the output of each neuron may correspond to the predicted pixel intensity at a specified location within the output deblurred medical image 456b.

[0061] In this manner, the CNN architecture 400 may implement a mapping of multiple intensity values ​​from a blurred medical image 402a to multiple intensity values ​​of a deblurred medical image 456b, wherein the blurriness of one or more blur artifacts present in the blurred medical image 402a is reduced or eliminated in the deblurred medical image 456b. In some embodiments, the CNN architecture 400 may implement a mapping of one or more features of a pixel / voxel of a blurred medical image to one or more attributes of a deblurred medical image. The CNN architecture 400 illustrates the transformation of a feature map that occurs when an input image tile is propagated through a neuron layer of a convolutional neural network to produce a deblurred medical image. In one example, the CNN architecture 400 may implement a mapping of multiple pixel / voxel intensity values ​​of a blurred medical image to a residual map, wherein a deblurred medical image may be produced by combining the residual map with the input blurred medical image 402a (such as by pixel-by-pixel addition of values).

[0062] The weights (and biases) of the convolutional layers in the CNN architecture 400 may be learned during training and / or may be combined / set based on the outputs from one or more acquisition parameter transformations, as will be described below with reference to Figure 5 , Figure 6 and Figure 7Discussed in more detail. The CNN architecture 400 can be trained by calculating the difference between a predicted deblurred medical image and a ground truth deblurred medical image, wherein the ground truth deblurred medical image can include a medical image without blur artifacts. The difference between the predicted deblurred medical image and the ground truth deblurred medical image can be used to determine the loss, and the loss can be back-propagated through the neural network using gradient descent or any other parameter optimization method known in the field of machine learning to update the weights (and biases) of each feature map. During the training process of the CNN architecture 400, a plurality of training data pairs can be used, including blurry medical images and corresponding ground truth deblurred medical images.

[0063] although Figure 4 Not shown, but it should be understood that the present disclosure encompasses neural network architectures that include one or more regularization layers, including batch normalization layers, dropout layers, Gaussian noise layers, and other regularization layers known in the field of machine learning, which can be used during training to mitigate overfitting and improve training efficiency while reducing training time.

[0064] It should be understood that Figure 4 The CNN architecture 400 shown in is for illustration and not limitation. Any suitable neural network such as ResNet, recursive neural network, generalized regression neural network (GRNN), etc. may be used herein to predict a deblurred medical image from a blurry medical image using the output from one or more acquisition parameter transformations. One or more specific embodiments of the present disclosure are described above to provide a thorough understanding. These described embodiments are merely examples of systems and methods for predicting a deblurred medical image from a blurry medical image using a deep neural network and one or more acquisition parameters and one or more acquisition parameter transformations. Those skilled in the art will appreciate that the specific details described in the embodiments may be modified in implementation without departing from the essence of the present disclosure.

[0065] Go to Figure 5 , showing an acquisition parameter transformation 500. The acquisition parameter transformation 500 may be stored in the acquisition parameter transformation module 210 within the image processing system 31 and may be used to map one or more acquisition parameters to one or more acquisition parameter layers, which may be concatenated with data from a blurred medical image and received by a deep neural network for mapping to a deblurred medical image, such as Figure 3As shown in the deblurring process 300 of . The acquisition parameter transform 500 is configured to map the vector (1D) of acquisition parameters to an acquisition parameter layer 518 whose size and dimensions match the size and dimensions of the blurred medical image. The acquisition parameter layer 518 can then be cascaded with the data from the blurred medical image, and the combined blurred medical image data and acquisition parameter layer data can be input into a deep neural network, thereby providing the deep neural network with additional information about the acquisition parameters of the blurred medical image, which can thereby inform the deep neural network of the type and / or degree of blur within the blurred medical image.

[0066] The acquisition parameter transform 500 is configured to receive a vector of n acquisition parameters (1D data object), including the first acquisition parameter AP1 502 to the nth acquisition parameter acquisition parameter transform 504, where n is a positive integer greater than 1. In some embodiments, the acquisition parameter transform 500 may be configured to receive a single acquisition parameter without departing from the scope of the present disclosure. Each of the n acquisition parameters includes a corresponding input node / neuron in the first / input layer of the acquisition parameter transform 500, so that there is a one-to-one correspondence between the acquisition parameters and the input nodes / neurons. As shown in the acquisition parameter transform 500, the n acquisition parameters match the n nodes in the input layer. Each node from node (1,1) 506 to node (1,n) 508 can receive a single acquisition parameter as input, and can map the single received acquisition parameter to an output by passing the acquisition parameter through an activation function, wherein the activation function can include a bias term.

[0067] The output from each input node of input node (1,1) 506 to input node (1,n) 508 is received by each first hidden layer node of first hidden layer node (2,1) 510 to first hidden layer node (2,J) 512, where J is a positive integer greater than 1, which may be equal to or not equal to n. In other words, the output from each node of the first layer of acquisition parameter transformation 500 is received by each node of the first hidden layer of acquisition parameter transformation 500, and thus the first hidden layer of acquisition parameter transformation 500 may be referred to as a fully connected layer. Each node of the first hidden layer of acquisition parameter transformation 500 calculates a dot product using the output from each node of the previous layer and each corresponding weight according to the following equation.

[0068]

[0069] where is the i-th neuron in the previous layer, Y j is the jth neuron in the subsequent layer, W ji is the weight, and B jIn some embodiments, the activation function f is a rectified linear unit (ReLU) function, such as a normal ReLU function, a leaky ReLU function, a parametric ReLU function, etc.

[0070] The acquisition parameter transformation 500 may include a positive integer number of fully connected hidden layers, similar to the first hidden layer described above, and indicated by the horizontal ellipsis between the first hidden layer and the output layer. The acquisition parameter transformation 500 may also include one or more dropout layers, or other regularization layers that may facilitate training and mitigate overfitting of the training data.

[0071] The acquisition parameter transformation 500 also includes output nodes (C, 1) 514 to output nodes (C, R) 516, where C specifies that the output layer is the Cth layer of the acquisition parameter transformation 500, and the Cth layer includes R nodes, where C and R are integers greater than 1. Each of the output nodes (C, 1) 514 to output nodes (C, R) 516 receives the weighted output from each node of the immediately previous hidden layer and calculates activation therefrom, similar to the way each hidden layer calculates activation based on the weighted input from each previous node / neuron in the immediately previous layer. Each activation value is then multiplied by the corresponding weight, and the result is used to generate the acquisition parameter layer 518.

[0072] exist Figure 5 In the illustrated embodiment, each output node includes X times Y weights (where the value of each of the X times Y weights may be different for each output node) that are used to map activation values ​​from each output node to a separate depth of the acquisition parameter layer 518. The sizes of X and Y are selected based on the size of the input image (the blurred medical image) to be cascaded with the acquisition parameter layer 518. For example, if the blurred medical image includes a 3D image of 512 pixels by 512 pixels by 100 pixels, then X may be set to 512, Y may be set to 512, and R may be set to 100.

[0073] In one embodiment, output node (C, 1) 514 (which represents the first output node of the output layer of acquisition parameter transform 500) produces a single output / activation value, which is then mapped to a first depth 520 of acquisition parameter layer 518 using X times Y weights. Similarly, output node (C, R) 516 (the last node in the output layer) produces an activation value, which is propagated to the last Rth depth of acquisition parameter layer 518 using X times Y weights. The blurred medical image that can be cascaded with acquisition parameter layer 518 has a width of X, a height of Y, and a depth of R, and thus represents a 3D image, such as a 3D image of a patient's anatomy. As a specific example, if R is equal to 3, X is equal to 10, and Y is equal to 10, the output layer will include 3 output nodes, and each of the 3 output nodes will generate 100 output values ​​(by multiplying the single activation value of each output node by 100 associated weights), where the first depth of the acquisition parameter layer 518 includes the 100 output values ​​generated by the output node (C, 1), the second depth of the acquisition parameter layer 518 (not shown) includes the 100 output values ​​generated by the output node (C, 2) (not shown), and the third depth of the acquisition parameter layer 518 includes the 100 output values ​​generated by the output node (C, R). In this way, one or more acquisition parameters can be mapped to a data object of the same size and dimension as the blurred medical image, thereby achieving a direct cascade of the acquisition parameter layer 518 and the blurred medical image, and then the blurred medical image data is input into the deep neural network.

[0074] It should be understood that the present disclosure covers Figure 5 , for mapping one or more acquisition parameters to a data object having the same dimensions and size as the blurred medical image. For example, each output node may produce a value used in each depth of the acquisition parameter layer 518. In another example, each output node of the output layer may output a single value to be combined with the acquisition parameter layer 518, so that, for example, if the acquisition parameter layer 518 includes a total of 100 values, the output layer includes 100 output nodes. In some embodiments, the acquisition parameter layer 518 may have the same dimensions as the blurred medical image, but have a smaller size than the blurred medical image.

[0075] Go to Figure 6, showing an acquisition parameter transform 600. The acquisition parameter transform 600 may be stored in the acquisition parameter transform module 210 within the image processing system 31 and may be used to map one or more acquisition parameters to one or more convolution filters or deconvolution filters, which may be incorporated into one or more convolution layers and / or deconvolution layers of a deep neural network, respectively. The acquisition parameter transform 600 is configured to map a vector (1D) of acquisition parameters to a plurality of filter weights, wherein the filter weights may be used for one or more convolution filters and / or deconvolution filters of a deep neural network, such as Figure 3 The deep neural network 324 shown or Figure 4 CNN architecture 400 is shown.

[0076] The acquisition parameter transform 600 is configured to receive a vector (1D data object) of n acquisition parameters, including the first acquisition parameter AP1 602 to the nth acquisition parameter acquisition parameter transform 604, where n is a positive integer greater than 1. In some embodiments, the acquisition parameter transform 600 may be configured to receive a single acquisition parameter without departing from the scope of the present disclosure. Each of the n acquisition parameters includes a corresponding input node / neuron in the first / input layer of the acquisition parameter transform 600, so that there is a one-to-one correspondence between the acquisition parameter and the input node / neuron. As shown in the acquisition parameter transform 600, the n acquisition parameters match the n nodes in the input layer. Each node from node (1,1) 606 to node (1,n) 608 can receive a single acquisition parameter as input, and can map the single received acquisition parameter to an output by passing the acquisition parameter through an activation function, wherein the activation function can include a bias term.

[0077] The output from each input node of input node (1,1) 606 to input node (1,n) 608 is received by each first hidden layer node of first hidden layer node (2,1) 610 to first hidden layer node (2,J) 612, where J is a positive integer greater than 1, which may be equal to or not equal to n. In other words, the output from each node of the first layer of acquisition parameter transformation 600 is received by each node of the first hidden layer of acquisition parameter transformation 600, and thus the first hidden layer of acquisition parameter transformation 600 may be referred to as a fully connected layer. Each node of the first hidden layer of acquisition parameter transformation 600 calculates a dot product using the output from each node of the previous layer and each corresponding weight according to the following equation.

[0078]

[0079] Where X i is the i-th neuron in the previous layer, Y j is the jth neuron in the subsequent layer, W jiis the weight, and B j In some embodiments, the activation function f is a rectified linear unit (ReLU) function, such as a normal ReLU function, a leaky ReLU function, a parametric ReLU function, etc.

[0080] The acquisition parameter transformation 600 may include a positive integer number of fully connected hidden layers, similar to the first hidden layer described above, and indicated by the horizontal ellipsis between the first hidden layer and the output layer. The acquisition parameter transformation 600 may also include one or more dropout layers, or other regularization layers that may facilitate training and mitigate overfitting of the training data.

[0081] The acquisition parameter transformation 600 also includes a first output node (C, 1) 614 to a last output node (C, R) 616, where C specifies that the output layer is the Cth layer of the acquisition parameter transformation 600, and the Cth layer includes R nodes, where C and R are integers greater than 1, which may be equal to each other or unequal to each other. Each of the output nodes from the first output node (C, 1) 614 to the last output node (C, R) 616 receives the weighted output from each node of the immediately previous hidden layer and calculates an activation value therefrom, similar to the manner in which each hidden layer calculates activation based on the weighted input from each node / neuron in the immediately previous layer. Each activation value is then multiplied by the corresponding weight, and the result is used to generate a filter 618, where the filter 618 includes a convolution filter and / or a deconvolution filter having i times R weights (that is, the filter 618 includes i columns and R rows).

[0082] exist Figure 6 In the illustrated embodiment, each of the R output nodes includes i associated weights, where the i associated weights are used to map activation values ​​from each output node to i different filter weights, such that each output node produces filter weights in a single row of filter 618. It should be understood that although shown as including a single depth, filter 618 may include any positive integer number of depths, such that filter 618 may include i×R×D weights, where D is the depth of the filter. In one embodiment, filter 618 includes a 3×3 convolutional filter with a depth of 1, such that the output layer of acquisition parameter transform 600 includes 3 output nodes, each with 3 different weights, thereby mapping 3 activation values ​​to a total of 9 filter weights.

[0083] Filter 618 may be incorporated into a deep neural network such as Figure 3 A deep neural network 324 or Figure 4The invention may be implemented in one or more convolutional layers and / or deconvolutional layers of a CNN architecture 400) and may enable a deep neural network to more consistently deconvolve / deblur blurry medical images.

[0084] It should be understood that the present disclosure covers Figure 6 618 for mapping one or more acquisition parameters to convolution filters and / or deconvolution filters. For example, each output node can produce a value used in each depth of filter 618. In another example, each output node of the output layer can output a single value for incorporation into filter 618, such that if filter 618 includes, for example, a total of 27 values, then the output layer includes 27 output nodes.

[0085] A deep neural network can be trained by using multiple pairs of blurred medical images and corresponding clear (or original) images. In some embodiments, in the clear-blurred medical image pair, the blurred medical image is reconstructed by a medical device from the acquired raw data, and the clear image is obtained by processing the blurred image by a known denoising / deblurring method or any combination thereof. In some embodiments, in the clear-blurred medical image pair, a clear image and a blurred image are acquired for the same anatomical region but using different acquisition parameters. The blurred image is used as an input to the deep neural network, and the clear image is used as a ground truth for reference.

[0086] The difference between the deblurred medical image and the corresponding sharp medical image output by the deep neural network is determined and back-propagated through the layers / feature maps of the deep neural network.

[0087] In some embodiments, parameters of the deep neural network and one or more acquisition parameter transforms are adjusted in a single stage, wherein the difference between the predicted deblurred medical image and the ground truth deblurred medical image is used to adjust the parameters in both the entire deep neural network and the acquisition parameter transform in a single stage. Alternatively, the deep neural network and one or more acquisition parameter transforms can be trained in alternating stages, wherein during a first stage, the parameters of the deep neural network remain fixed, while the parameters of the one or more acquisition parameter transforms are adjusted based on the training data. During a second stage, the parameters of the one or more acquisition parameter transforms may remain fixed, while the parameters of the deep neural network are adjusted based on the training data. The alternation of training stages can continue until a threshold accuracy of the prediction is met, or until the parameters of the one or more acquisition parameter transforms and the deep neural network have converged (that is, when the rate of change of the parameters of the one or more acquisition parameter transforms and the deep neural network per round is less than a predetermined threshold rate of change).

[0088] refer to Figure 7, a flow chart of a method 700 for deblurring a blurred medical image using a deep neural network and one or more acquisition parameter transformations according to an exemplary embodiment is shown. The method 700 can be implemented by the medical image processing system 100, an edge device connected to an imaging device, a cloud communicating with an imaging device, or any suitable combination thereof.

[0089] Method 700 begins at operation 702, in which a blurred medical image is received. In some embodiments, the image processing system receives the blurred medical image from the imaging system via a communication coupling (such as through a network). In some embodiments, the image processing system receives the blurred medical image from a non-transitory memory. Although described with reference to a single blurred medical image for simplicity, it should be understood that the present disclosure provides mapping multiple blurred medical images to multiple (or single) deblurred medical images. For example, the number of input layers corresponding to the multiple blurred medical images can be increased to accommodate the number of blurred medical images to be deblurred without departing from the disclosure provided herein.

[0090] At operation 704, one or more acquisition parameters associated with the blurred medical image are received. The acquisition parameters associated with or corresponding to the blurred medical image may include one or more settings, parameters, or conditions used or present during the acquisition of the blurred medical image. In some embodiments, the acquisition parameters include settings of an imaging device used during the scan / image acquisition. In some embodiments, the acquisition parameters include one or more properties of an anatomical region of a patient imaged during the scan / image acquisition. The acquisition parameters may be stored with or indexed by the medical image to which they correspond, such that fast and computationally efficient retrieval of one or more acquisition parameters associated with the blurred medical image may be achieved.

[0091] At operation 706, the received acquisition parameters are input into one or more trained acquisition parameter transforms that map the acquisition parameters to a first output. In some embodiments, the first output may include one or more acquisition parameter layers that may be concatenated with the blurred medical image and input into a deep neural network. In some embodiments, the first output may include one or more values ​​that may be used to set one or more weights in a convolution filter and / or a deconvolution filter of the deep neural network. In some embodiments, the output may be used for both the input layer and the weights of the filter. In some embodiments, the acquisition parameter transform includes an input layer configured to receive the acquisition parameters, an output layer configured to produce the first output, and at least one fully connected layer between the input layer and the output layer. In some embodiments, the blurred medical image includes a plurality of pixels or voxels, and the first output from the acquisition parameter transform includes a plurality of output values, wherein the plurality of output values ​​may be evenly divided by the plurality of pixels or voxels, such that an integer number of output values ​​from the first output may be concatenated with data from each pixel or voxel of the blurred input image. See Figure 5 and Figure 6 , for a more detailed description of the acquisition parameter transformation and the types of outputs that can be generated therefrom. Although the acquisition parameter transformation is described herein as including learnable parameters, it should be understood that the present disclosure also provides acquisition parameter transformations including fixed models / simulations, where the parameters are fixed, such as according to a physical model or equation, which can be used to map one or more acquisition parameters to a first output having the properties described herein.

[0092] At operation 708, the first output from the one or more acquisition parameter transformations is incorporated into a deep neural network, such as the deep neural network 324 or the CNN architecture 400. Incorporating the first output from the one or more acquisition parameter transformations into the deep neural network may be as described with reference to Figure 3 , Figure 4 , Figure 5 and Figure 6The first output occurs as described above, and therefore will be briefly described herein. In some embodiments, combining the first output with the deep neural network includes: concatenating the first output with a parametric map and a blurred medical image, wherein the parametric map may include a map of one or more physical properties or physiological properties of the imaging space / imaged anatomical region; and inputting the first output, the parametric map, and the blurred medical image into an input layer of the trained deep neural network. In some embodiments, combining the first output with the deep neural network includes: concatenating the first output with the blurred medical image; and inputting both the first output and the blurred medical image into an input layer of the trained deep neural network. In some embodiments, the first output includes a plurality of values, and combining the first output into the deep neural network includes setting a plurality of weights in the trained deep neural network based on the plurality of values, wherein in some embodiments, the deep neural network includes a CNN, and the plurality of weights include a deconvolution filter of a deconvolution layer of a convolutional neural network or a convolution filter of a convolution layer. In some embodiments, the output can be used for both the weights of the input layer and the filter.

[0093] At operation 710, the blurred medical image is mapped to a second output using a deep neural network. Mapping the blurred medical image to the second output includes: inputting data from the blurred medical image (including any additional cascaded data) into an input layer / input tile of the deep neural network; and propagating the input data through each layer of the deep neural network until a second output is generated by an output layer of the deep neural network. In some embodiments, the deep neural network includes a convolutional neural network, wherein one or more filters (convolution filters or deconvolution filters) are set based on a first output from one or more acquisition parameter transformations, and the one or more filters are applied to the data from the blurred medical image as the data is propagated through the deep neural network.

[0094] At operation 712, a deblurred medical image is generated using a second output from the deep neural network. In some embodiments, the second output includes a residual map, and using the second output to produce a deblurred medical image from a blurred medical image includes combining the residual map with the blurred medical image to produce the deblurred medical image. In other words, the residual map may include a plurality of values, i.e., one or each pixel or voxel of the input blurred medical image, describing the intensity difference between the intensity of each pixel or voxel of the blurred image and the intensity of each pixel or voxel of the corresponding deblurred medical image. Combining the residual map with the blurred medical image to produce the deblurred medical image may include a pixel-by-pixel addition of the values ​​between the residual map and the blurred medical image. In some embodiments, the second output from the deep neural network includes a map of pixel / voxel intensity values ​​of the deblurred medical image.

[0095] At operation 714, the image processing system displays the deblurred medical image via a display device. In some embodiments, the user can select the deblurred image for further image processing, such as image segmentation, background noise suppression, pathology identification, super-resolution, etc., via a user input device using a model trained on clear / non-blurred medical images.

[0096] In this manner, the method 700 integrates information about one or more acquisition parameters into the deep neural network used using one or more acquisition parameter transformations to achieve deblurring of blurry medical images in a time-efficient and more consistent manner, thereby generating a deblurred medical image. The deblurred medical image can be more efficiently processed by additional downstream image processing models, which may have been trained using clear medical images.

[0097] Go to Figure 8 , shows an example of a point spread function 804 and a corresponding transverse magnetization signal decay 802. The nominal transverse magnetization signal decay 802 is model-fitted to a nominal T2 value (or some fractional combination of T2 values, which may result in a multi-exponential decay) based on one or more acquisition parameters (such as echo spacing, echo train length, sample scheduling / order, and filtering of the raw k-space data). Transforming the transverse magnetization signal decay 802 from the frequency domain to the spatial domain may be accomplished via a Fourier transform, resulting in the point spread function 804. In some embodiments, inverting the transverse magnetization signal decay 802 prior to transforming to the spatial domain may be used to generate weights for one or more convolution kernels or deconvolution kernels, which may be used, for example, at the convolution filter 312 and / or the deconvolution filter 316.

[0098] In some embodiments, a point spread function similar to point spread function 804 can be predicted by an acquisition parameter transformation based on one or more acquisition parameters, and inverted to produce a deconvolution filter, which can be incorporated into a deep neural network and used to deconvolve a blurred medical image. Since blur can be modeled as a convolution of a point spread function on a sharp image, deconvolving a blurred image using an inverted point spread function can achieve consistent deblurring, especially when the inverse point spread function is predicted based on the acquisition parameters used during acquisition of the blurred medical image to be deblurred rather than a general point spread function.

[0099] In some embodiments, the point spread function 804 may be generated by an analytical model as a function of one or more acquisition parameters.

[0100] Go to Fig. 9, shows an example of a blurred medical image and a corresponding deblurred medical image generated according to an embodiment of the present disclosure. The blurred medical image 902 includes an MR cross-sectional image of a human abdomen and includes one or more blurring artifacts that reduce the clarity of various boundaries of the imaged anatomical regions. The clear / deblurred medical image 904 includes the same anatomical region as depicted in the blurred medical image 902, however the clear medical image 904 has been deblurred according to the deblurring method disclosed herein (such as method 700). It can be seen that the boundaries of the anatomical regions imaged in the clear medical image 904 are clearer / less trailing than the corresponding anatomical regions imaged in the blurred medical image 902, which can enable more accurate analysis and / or diagnosis based on the imaged anatomical regions. In addition, the degree of blur within the blurred medical image 902 varies throughout the image, however, the deblurred medical image 904 shows a consistent degree of deblurring.

[0101] The blurred medical image 906 includes an MR cross-sectional image of a human hip region and includes one or more blurring artifacts that reduce the clarity of various boundaries of the imaged anatomical region. The clear / deblurred medical image 908 includes the same anatomical region as depicted in the blurred medical image 906, however the clear medical image 908 has been deblurred according to a deblurring method disclosed herein (such as method 700). It can be seen that the fine structures in the tissue and bone in the anatomical region captured in the deblurred medical image 908 are more clearly defined than the corresponding fine structures in the blurred medical image 906. Therefore, the deblurred medical image 908 can enable more accurate analysis and / or diagnosis.

[0102] although Fig. 9 Two specific examples of anatomical regions imaged via MRI are provided that can be deblurred using the systems and methods disclosed herein, but it should be understood that the present disclosure provides for deblurring of substantially any medical image of any anatomical region. In one embodiment, a single deep neural network can be trained using training data pairs of substantially similar anatomical regions (these substantially similar anatomical regions are captured / acquired using a single medical imaging modality), and the deep neural network can be used to deblur blurred medical images of anatomical regions that are substantially similar to the training data pairs. In other embodiments, a single deep neural network can be trained using training data pairs of multiple different medical imaging modalities including different anatomical regions, thereby producing a more generalized deep neural network that can enable the use of a single deep neural network to deblur a wide range of medical images of various anatomical regions.

[0103] The technical effect of incorporating one or more acquisition parameters into a trained deep neural network is that the deep neural network can receive at least partial information about the type, degree and / or spatial distribution of blur in the blurry medical image, so that the trained deep neural network can selectively deblur the received blurry medical image with a higher degree of consistency.

[0104] One or more specific embodiments of the present disclosure are described above to provide a thorough understanding. These described embodiments are merely examples of systems and methods for selectively denoising medical images using deep neural networks. Those skilled in the art will appreciate that the specific details described in the embodiments may be modified in practice without departing from the essence of the present disclosure.

[0105] When introducing the elements of various embodiments of the present disclosure, the words "one", "a kind of" and "the" are intended to mean one or more elements present in these elements. The terms "first", "second", etc. do not represent any order, amount or importance, but are used to distinguish one element from another element. The terms "comprise", "include" and "have" are intended to be inclusive, and mean that additional elements may also exist in addition to the listed elements. As used herein, the terms "connected to", "coupled to", etc., an object (e.g., material, element, structure, member, etc.) can be connected to or coupled to another object, regardless of whether the one object is directly connected or coupled to another object, or whether there are one or more intervening objects between the one object and another object. In addition, it should be understood that the reference to "an embodiment" or "embodiment" of the present disclosure is not intended to be interpreted as excluding the existence of additional embodiments that also include the cited features.

[0106] In addition to any previously indicated modifications, numerous other variations and alternative arrangements may be devised by those skilled in the art without departing from the spirit and scope of the present description, and the appended claims are intended to cover such modifications and arrangements. Therefore, although the information has been described above in detail in conjunction with what are currently considered to be the most practical and preferred aspects, it will be apparent to those of ordinary skill in the art that many modifications may be made, including but not limited to form, function, mode of operation, and use, without departing from the principles and concepts set forth herein. Likewise, as used herein, the examples and embodiments are intended to be illustrative in all respects only and should not be construed as limiting in any way.

Claims

1. A method for deblurring a medical image, the method comprising: receiving a blurred medical image and one or more acquisition parameters corresponding to acquisition of the blurred medical image, the one or more acquisition parameters comprising parameters of imaging system settings used to acquire the medical image, the one or more acquisition parameters comprising one or more of echo train length, repetition time, echo time, flip angle, inversion time, acquisition sequence, acceleration factor, fat saturation setting, B0 shim mode, RF drive mode, and image reconstruction parameters; incorporating the one or more acquired parameters into a trained deep neural network; as well as The blurred medical image is mapped to a deblurred medical image by the trained deep neural network.

2. The method of claim 1 , wherein incorporating the one or more acquisition parameters into the trained deep neural network comprises: mapping the one or more acquisition parameters to one or more weights; as well as The one or more weights are incorporated into the one or more trained deep neural networks.

3. The method of claim 2, wherein the trained deep neural network comprises a convolutional neural network, and wherein incorporating the one or more weights into the trained deep neural network comprises setting one or more convolution filter weights and / or one or more deconvolution filter weights based on the one or more weights.

4. The method of claim 1 , wherein incorporating the one or more acquisition parameters into the trained deep neural network comprises: mapping the one or more acquisition parameters to a plurality of values; concatenating the plurality of values ​​with a plurality of pixel or voxel data of the blurred medical image; as well as The plurality of values ​​and the plurality of pixel or voxel data are input into an input layer of the trained deep neural network.

5. The method according to claim 1 further includes training the deep neural network by using a plurality of pairs of sharp medical images and corresponding blurred medical images.

6. The method of claim 1, wherein the medical image is one of: a magnetic resonance (MR) image, a computed tomography (CT) image, a positron emission tomography (PET) image, an X-ray image, or an ultrasound image.

7. The method of claim 1 , wherein the one or more acquisition parameters comprise one or more of: echo train length, echo spacing, flip angle, voxel spatial dimension, sampling pattern, acquisition order, acceleration factor, physiological state, image reconstruction parameters, detector size, number of detectors, source-to-detector distance, collimator geometry, slice thickness, dose, detector type, detector geometry, ring radius, positron range, response depth / line, transducer geometry, center frequency, ringing, spatial pulse length, focal depth, beam toeing, or sidelobe amplitude.

8. A method for reducing blur in a medical image, the method comprising: receiving a blurred medical image acquired by an imaging system; receiving one or more acquisition parameters of the imaging system applied during acquisition of the blurred medical image, the one or more acquisition parameters comprising one or more of an echo train length, a repetition time, an echo time, a flip angle, an inversion time, an acquisition sequence, an acceleration factor, a fat saturation setting, a B0 shim mode, an RF drive mode, and an image reconstruction parameter; mapping the one or more acquisition parameters to a first output using an acquisition parameter transform; Incorporating the first output into a convolutional neural network (CNN); mapping the blurred medical image to a second output using the CNN; and generating a deblurred medical image using the second output; Wherein the second output comprises a residual map, and wherein producing the deblurred medical image using the second output comprises combining the residual map with the blurred medical image to produce the deblurred medical image, the residual map describing the intensity difference between the intensity of each pixel or voxel of the blurred medical image and the intensity of each pixel or voxel of the corresponding deblurred medical image.

9. The method of claim 8, wherein the first output is one or more weights, and wherein incorporating the first output into the CNN comprises setting one or more convolution filter weights and / or one or more deconvolution filter weights based on the one or more weights.

10. The method of claim 9, wherein the acquisition parameter transformation comprises mapping the one or more acquisition parameters to an analytical model of the one or more weights. The method of claim 10 , wherein the analytical model is based on a point spread function.

12. The method of claim 8, wherein the first output is a plurality of values, and wherein incorporating the first output into the CNN comprises concatenating the plurality of values ​​with a plurality of pixel or voxel data of the blurred medical image as input to the CNN.

13. The method of claim 12, wherein the acquisition parameter transformation comprises another trained deep neural network mapping the one or more acquisition parameters to the plurality of values.

14. A system for deblurring a medical image, the system comprising: A memory storing the trained deep neural network and acquisition parameter transformations; and a processor communicatively coupled to the memory and configured to: Receiving blurry medical images; receiving one or more acquisition parameters, wherein the one or more acquisition parameters correspond to acquisition of the blurred medical image, the one or more acquisition parameters comprising one or more of an echo train length, a repetition time, an echo time, a flip angle, an inversion time, an acquisition sequence, an acceleration factor, a fat saturation setting, a B0 shim mode, an RF drive mode, and an image reconstruction parameter; mapping the acquisition parameters to a first output using the acquisition parameter transform; Incorporating the first output into the trained deep neural network; mapping the blurred medical image to a second output using the trained deep neural network; and generating a deblurred medical image using the second output; Wherein the second output comprises a residual map, and wherein producing the deblurred medical image using the second output comprises combining the residual map with the blurred medical image to produce the deblurred medical image, the residual map describing the intensity difference between the intensity of each pixel or voxel of the blurred medical image and the intensity of each pixel or voxel of the corresponding deblurred medical image.

15. The system of claim 14, wherein the trained deep neural network is a trained convolutional neural network, wherein the first output is one or more weights, and wherein the processor is configured to incorporate the first output into the trained deep neural network by setting one or more convolution filter weights and / or one or more deconvolution filter weights of the convolutional neural network based on the one or more weights.

16. The system of claim 15, wherein the acquisition parameter transformation comprises an analytical model that maps the one or more acquisition parameters to the one or more weights.

17. The system of claim 16, wherein the analytical model is based on a point spread function.

18. The system of claim 14, wherein the first output is a plurality of values, and wherein the processor is configured to incorporate the first output into the trained deep neural network by concatenating the plurality of values ​​with a plurality of pixel or voxel data of the blurred medical image as input to the trained deep neural network.

19. The system of claim 18, wherein the acquisition parameter transformation comprises another trained deep neural network that maps the one or more acquisition parameters to the plurality of values.

Images