System and method for generating thin image slices from thick image slices

Abstract

Disclosed herein are systems and methods for generating thin slice images from thick slice images. In some instances, a deep learning system may calculate a residual from a thick slice image and add the residual to the thick slice image to generate a thin slice image. In some instances, the deep learning system includes a neural network. In some instances, the neural network may include one or more levels, wherein one or more of the levels include one or more blocks. In some instances, each level includes a convolution block and a nonlinear activation function block. In some instances, the levels of the neural network may be arranged in a cascade.

Description

[0001] STATEMENT AS TO RIGHTS TO INVENTS MADE UNDER FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

[0002] This invention was made with government support under Contract Nos. NIH R01 AR063643 and R01 EB002524 awarded by the National Institutes of Health (NIH). The government has certain rights in this invention.

[0003] CROSS-REFERENCE TO RELATED APPLICATIONS

[0004] This application claims priority to earlier-filed U.S. Provisional Application No. 62 / 641,836, filed on March 12, 2018, which is incorporated herein by reference in its entirety for any purpose. Technical Field

[0005] The examples described herein generally relate to processing image slices from a three-dimensional image. More specifically, the examples described herein relate to processing thick image slices from a three-dimensional image to produce thin slices of a three-dimensional image. In some examples, the thin slices may be analyzed, observed, or otherwise used to diagnose and / or treat a disease. Background Art

[0006] In three-dimensional (3D) imaging modalities such as magnetic resonance imaging (MRI), computed tomography (CT), positron emission tomography (PET), and ultrasound (US), a volume is imaged as a series of imaging planes (e.g., slices) with finite thickness (e.g., 0.5 mm, 1 mm, 0.5 cm, 1 cm). The slices are combined to produce a 3D image of the volume, wherein each voxel of the 3D image corresponds to a portion of the imaged volume. The resolution of the 3D image may be based at least in part on the magnitude of the volume represented by each voxel. The larger the volume represented by the voxel, the lower the resolution of the 3D image. One technique for improving the resolution of a 3D image is to acquire more slices within the volume. That is, the thickness of the slices decreases and the volume represented by each voxel decreases. For example, imaging the volume as a series of 0.5 mm slices may produce a higher resolution 3D image than when the volume is imaged as a series of 1.0 mm slices.

[0007] Higher resolution images may provide more information to an observer than lower resolution images. For example, clinicians may be able to make more accurate diagnoses from medical images with higher resolution than from those with lower resolution. However, in some applications, acquiring thinner slices of a volume may not be feasible. For example, some imaging systems may be limited to a minimum slice thickness that cannot be reduced any further. In some applications, acquiring thinner slices may require additional imaging time. This may reduce patient comfort if, for example, the patient must remain in a cramped space for a longer period of time and / or must hold his breath during slice acquisition. For some imaging modalities, acquiring thinner slices may require additional exposure to ionizing radiation, which may increase health risks for the patient.

[0008] Therefore, there is a need to improve the resolution of 3D images without reducing the thickness of slices acquired during imaging. Summary of the invention

[0009] Examples of the techniques described herein may provide a super-resolution technique that uses a deep learning system to generate thin-slice 3D images (e.g., MRI, fMRI, PET, US, and / or CT) from thicker input slices, and compares this approach to alternative through-plane interpolation methods.

[0010] According to some examples of the present disclosure, a deep learning system comprising a 3D convolutional neural network is described, which is configured to learn a residual-based transformation between a high-resolution thin-slice image and a lower-resolution thick-slice image. In some examples, the neural network is trained to learn a residual-based transformation between a high-resolution thin-slice image and a lower-resolution thick-slice image at the same central position. Although 3D musculoskeletal MR images are provided as examples, this method is generally applicable to any 3D image and imaging of any region of the human or animal body (including the brain, liver, cardiovascular system, etc.).

[0011] Neural networks (e.g., 3D convolutional neural networks) according to the principles of the present disclosure may be able to solve high-resolution thin-slice MRI from lower-resolution thicker slices. In some applications, the neural networks can achieve superior quantitative and qualitative diagnostic performance while reducing the number of slices that need to be acquired compared to conventionally utilized and state-of-the-art methods for MRI, CT, and / or other 3D imaging modalities.

[0012] A method for generating a thin slice image from a thick slice image according to an example of the present disclosure may include receiving a first image having a first resolution at a neural network; performing a convolution on the first image by the neural network; performing a nonlinear activation function on the first image by the neural network; repeating the convolution and the nonlinear activation function; generating a residual based on another convolution by the neural network; and summing the residual with the first image by the neural network to generate a second image having a second resolution, wherein the second resolution is higher than the first resolution.

[0013] A system for generating a thin slice image from a thick slice image according to an example of the present disclosure may include a non-transitory computer-readable medium including instructions for implementing a neural network, wherein the neural network includes a hierarchy including a convolution block and a rectified linear unit non-linear activation block, wherein the hierarchy is configured to generate a residual from a first image having a first resolution received by the neural network, wherein the neural network is configured to sum the first image with the residual to generate a second image having a second resolution, wherein the second resolution is higher than the first resolution; and a processor configured to execute the instructions to implement the neural network.

[0014] A system for generating a high-resolution image from a low-resolution image according to an example of the present disclosure may include an image acquisition unit configured to acquire a first image of a feature of interest at a first resolution; a computing system configured to implement a deep learning system, wherein the deep learning system is configured to receive the first image of the feature of interest and generate a second image of the feature of interest at a second resolution based at least in part on the first image, wherein the second resolution is higher than the first resolution; and a display configured to display the second image of the feature of interest. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a block diagram of a system arranged according to the examples described herein.

[0016] Figure 2 is an illustration of a neural network arranged according to the examples described herein.

[0017] Figure 3 yes Figure 2 An illustration of a portion of a neural network is shown in FIG.

[0018] Figure 4 yes Figure 2 and 3 An illustration of a portion of a neural network is shown in FIG.

[0019] Figure 5An example workflow for implementing a neural network according to the examples described herein is shown.

[0020] Figure 6 are exemplary coronal images with different downsampling factors and corresponding 2D structural similarity maps according to examples described herein.

[0021] Figure 7 A comparison of example embodiments of the present disclosure with other resolution enhancement techniques is described.

[0022] Figure 8 is a set of simulated ground-truth images and the resulting generated images according to the examples described in this article.

[0023] Fig. 9 Example images showing a horizontal tear in the lateral meniscal body according to examples described herein.

[0024] Fig.10 Example images showing a mild case of grade 2A chondromalacia of the lateral patellar cartilage according to examples described herein.

[0025] Fig.11 is an illustration of a system arranged according to the examples described herein. DETAILED DESCRIPTION

[0026] Certain details are set forth below to provide a thorough understanding of the described embodiments. However, it will be apparent to one skilled in the art that the embodiments may be practiced without these specific details. In some cases, well-known brain imaging techniques and systems, circuits, control signals, timing protocols, and / or software operations are not shown in detail to avoid unnecessarily obscuring the described embodiments.

[0027] Rapidly acquiring high-resolution 3D medical images with sufficient signal-to-noise ratio is challenging due to hardware limitations in modalities such as magnetic resonance imaging (MRI), functional magnetic resonance imaging (fMRI), positron emission tomography (PET), and computed tomography (CT).

[0028] In MRI, clinical musculoskeletal imaging protocols typically include 2D fast spin echo (FSE) sequences scanned in various scan planes, usually with equal contrast. Although such sequences provide excellent in-plane resolution, there is always an associated risk of missing subtle lesions due to partial volume effects in slices with high section thickness (typically 2.5-4 mm) and with slice gaps. Image acquisition of thick slices also prevents image reconstruction into arbitrary scan planes, which makes it challenging to query tissues (e.g., anterior cruciate ligament and articular cartilage of the femoral trochlea) in oblique orientations.

[0029] Several methods have been proposed recently for thin-section musculoskeletal MRI, of which 3D fast spin echo (using vendor products SPACE, CUBE or VISTA) is a popular choice. However, 3D FSE sequences usually compromise in-plane resolution for through-plane resolution and suffer from significant echo chain blurring, even with variable flip angle methods. Therefore, even if thin-section images are acquired, the overall diagnostic quality of 3D FSE is still limited. 3D radial sequences are used for isotropic musculoskeletal imaging, but are not yet widely adopted due to different artifact appearances. In addition to sequence-based methods, compressed sensing (CS) and parallel imaging are also promising methods for accelerating thin-section MRI, but they are not ideal independent methods. CS MRI is not common in 2D FSE clinical protocols because only one dimension can be used to produce incoherent sampling and long reconstruction times are required. Parallel imaging acceleration is ultimately limited by SNR and g-factor. In addition, the geometry of common knee coils (e.g., rigid 8-channel) is not suitable for acceleration in all directions.

[0030] Single image super-resolution (an active field in image processing) can potentially improve MRI spatial resolution to produce thin-section MRI without compromising SNR or requiring additional MRI hardware or scan time. For example, interpolation is an original implementation of super-resolution. Due to the challenges of actually acquiring thin-section imaging, MRI vendors provide retrospective slice interpolation using zero-filled Fourier interpolation (FI) through options such as 'ZIP2' and 'Interpolation' for GE Healthcare and Siemens Healthineers MRI scanners, respectively. Similarly, medical image viewing platforms such as OsiriX use trilinear interpolation during image manipulation and multi-planar reconstruction of data. Low-order linear interpolation methods such as FI and trilinear interpolation are widely used in clinical and research protocols that attempt to achieve thin-section imaging, however, they do not produce images with high diagnostic quality.

[0031] In addition to interpolation, another major approach to MRI super-resolution imaging requires exploiting image sparsity and only requires a single image as input. The current state-of-the-art MRI single-image super-resolution algorithm is based on sparse coding super-resolution (ScSR), which was originally developed for natural images but later adapted to a few MRI applications. Although promising, this super-resolution method has not yet been widely used in medical imaging due to limited resolution improvements and slow execution speed for 3D data.

[0032] According to the principles of the present disclosure, a deep learning system may be configured to apply super-resolution techniques used with natural 2D images to 3D medical images. In some instances, a deep learning system may be implemented using an artificial intelligence system, a machine learning system, a neural network (e.g., a convolutional neural network), and / or other computing techniques. In some instances, a software-based post-processing neural network may at least partially overcome the limitations of 3D medical imaging just described. In some instances, the neural network may be a convolutional neural network. In some instances, the neural network may employ deep learning techniques. Because images are used to train and deploy the neural network, the neural network is independent of the image acquisition hardware platform.

[0033] An example describing musculoskeletal MRI will be used to explain the architecture of the deep learning system and its training and implementation methods. However, the deep learning system can be generally applied to different 3D medical images including MRI, fMRI, PET and CT. The application area is not limited to musculoskeletal imaging and includes images covering any area such as the brain, liver and cardiovascular system.

[0034] The described example includes a system that uses deep learning for MRI super-resolution to generate thin-slice images and maintains high in-plane resolution to reduce the total scan time. As used herein, "thick" and "thin" are used as relative terms. That is, the size of a slice referred to as "thick" is larger than the size of a "thin" image in the corresponding dimension in at least one dimension (e.g., vertical, azimuth, lateral). The resolution of the image generated by the slice can be based on the magnitude of the volume represented by the voxel in the image. The larger the volume represented by the voxel, the lower the resolution of the image. The image generated from the thick slice can have a larger volume associated with each voxel than the image generated from the thin slice. The proposed deep learning system and method can be referred to as 'DeepResolve' because it helps to solve high-resolution features from low-resolution input. DeepResolve may include a convolutional neural network for deep learning (e.g., machine learning). Specifically, the example described herein uses a publicly available data set to train a neural network to generate high-resolution thin-slice knee MR images from slices at the same orientation but with slice thickness 2-8 times higher. The deep learning system may not necessarily produce images that are identical to the ground truth; rather, in some applications, the deep learning system may enhance the low-resolution images to make them more similar to the ground truth (e.g., high-resolution images) than the current state-of-the-art methods typically utilized. This may enhance the diagnostic value of images acquired from thick slices in some applications.

[0035] Figure 11 is a schematic illustration of a system arranged according to the examples described herein. System 100 includes image slices (e.g., thick slices) 102, a training data set 104, a computing system 106, a processor 108, executable instructions for generating thin slices from input image slices 110 (also referred to herein as executable instructions), a memory 112, a display 114, and a network interface 116. Additional, fewer, and / or other components may be used in other examples. In some examples, some or all of the components of system 100 may be included with an imaging system. For example, computing system 106 may be included in an imaging system. For example, the imaging system may be an MRI, CT, and / or ultrasound imaging system.

[0036] The image slices 102 and / or the training data set 104 may be provided by an imaging system. In some examples, the image slices 102 and / or the training data set 104 may be provided by multiple imaging systems. The imaging systems may have the same modality or different modalities. The image slices 102 and / or the training data set 104 may be stored in a memory that is accessible to the computing system 106 and / or transmitted to the computing system 106 (e.g., using wired or wireless communication). The computing system 106 may be configured to generate a deep learning system based on the training data set 104. The deep learning system may include one or more functions for generating thin slice images from the image slices 102. The functions may be generated and / or optimized based on the training data set 104.

[0037] The training data set 104 may train a deep learning system to find the difference (e.g., residual) between a thick slice image and a corresponding thin slice image. After training, the deep learning system may generate a thin slice image from an input image slice 102 by applying the function to the image slice 102. The function may supplement the image slice 102 to generate a thin slice (e.g., high resolution) image. In some examples, the deep learning system may calculate the difference between the input image slice 102 and a desired high resolution image (e.g., thin slice image) and add the difference to the input image slice 102 to generate a high resolution image. In some examples, the deep learning system may include one or more neural networks. In some examples, one or more of the neural networks may be convolutional neural networks. One or more of the neural networks may include multiple layers, wherein each layer includes one or more functions or function portions for generating a thin slice image from the image slice 102.

[0038] The computing system 106 may be configured to generate thin slice (e.g., higher resolution) images from the image slices 102 by the deep learning system generated from the training data set 104. In some examples, the memory 112 may be encoded with executable instructions 110 for the deep learning system, which is configured to generate thin slice images based on the thick slice 102 input. In some examples, the deep learning system may be partially or completely implemented in hardware (e.g., ASIC, FPGA). In some examples, the executable instructions 110 may include instructions in a hardware description language (e.g., VHDL), which can be used to design hardware to implement some or all of the deep learning system.

[0039] The examples described herein may utilize a computing system that may generally include hardware and / or software for implementing a deep learning system for generating thin slice images. For example, a computing system 106 may include one or more processors 108. For example, the processor 108 may be implemented using one or more central processing units (CPUs), graphics processing units (GPUs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other processor circuits. In some examples, the processor 108 may execute some or all of the executable instructions 110. The processor 108 may communicate with a memory 112. The memory 112 may generally be implemented by any computer-readable medium (e.g., a read-only memory (ROM), a random access memory (RAM), flash, a solid-state drive, etc.). Although a single memory 112 is shown, any number of memories 112 may be used, and they may be integrated with the processor 108 in a single computing system 106 and / or located in another computing system and communicate with the processor 108.

[0040] In some examples, the system 100 may include a display 114, which may be in communication with the computing system 106 (e.g., using a wired and / or wireless connection), or the display 114 may be integrated with the computing system 106. The display 114 may display one or more image slices 102 and / or one or more thin slices generated by the computing system 106. Any number or variety of displays may be present, including one or more LED, LCD, plasma, or other display devices.

[0041] In some examples, the system 100 may include a network interface 116. The network interface 116 may provide a communication interface with any network (e.g., LAN, WAN, Internet). The network interface 116 may be implemented using a wired and / or wireless interface (e.g., Wi-Fi, Bluetooth, HDMI, USB, etc.). The network interface 116 may transmit data, which may include image slices 102 and / or thin slice images generated by a deep learning system implemented by the computing system 106.

[0042] refer to Figure 2-4, describing an example deep learning system. During training, a set of i ground truth high-resolution thin-section images with the same field of view and matrix size and the corresponding low-resolution thick slice image interpolated to the thin slice orientation A set of may be provided as a training set. The deep learning system may include one or more functions. In an example implementation, the deep learning system includes a function f=f(X i ), which computes the residual image r between the ground truth image and the interpolated low-resolution image i This function can be trained by optimizing (using L2-loss):

[0043]

[0044] In the implementation examples described herein, the L2 loss function is used to optimize the deep learning system. In other examples, additional loss functions may be used, such as L1-norm, pSNR, SSIM, and / or any other metric for measuring image differences. Other optimization functions may be used in other examples.

[0045] In some examples, the deep learning system may be an unsupervised learning algorithm. In some examples, the function generated and / or optimized by training (e.g., the function in equation (1)) may be implemented in one or more neural networks that may have multiple layers. In some examples, the function may be implemented on multiple layers of the neural network. Each layer of the neural network may include one or more blocks. Each block may implement at least a portion of the function. In some examples, each layer of the neural network includes the same blocks (e.g., blocks that are repeated in each layer), but in other examples, the layers of the neural network may include different blocks.

[0046] In an example implementation, a deep learning system can be modeled as a cascade of convolutional filters. Figure 2 As illustrated in , each level (e.g., layer) 202 of the deep learning system 200 may include a convolution block 204 and a rectified linear unit nonlinear activation function block (ReLu) 206, which will be described in more detail later. The deep learning system 200 may calculate a residual (e.g., a residual image) 208 from an input low-resolution (LR) image 210 (e.g., a thick slice image) to generate a corresponding high-resolution (HR) image 212 (e.g., a thin slice image). Although 2D convolutions may be used, the deep learning system may utilize 3D convolutions that may provide additional spatial information to improve super-resolution performance.

[0047] After the deep learning system has been trained, the estimated residual can be directly calculated And a high-resolution super-resolution image can be generated by the following formula (e.g., to generate thin-section images):

[0048]

[0049] An example implementation of a deep learning system includes a neural network having 19 layers of paired convolution 204 and rectified linear unit (ReLU) nonlinear activation function blocks 206 (e.g., activation functions), and additional layers including convolution blocks 207. However, other numbers of layers may be used in other examples. An input LR image 210 may pass through a first layer and be processed by convolution blocks 204 and ReLu blocks 206. The processed input image may then be passed as input to the convolution blocks 204 and ReLu blocks 206 of the next layer. This process is repeated until the input LR image 210 has been processed by all layers of the neural network and a residual 208 is output.

[0050] like Figure 3 As illustrated in , the filter sizes of the first and last layers of the neural network of the implementation example of the deep learning system have dimensions of 32×32×32x1, while all other layers have dimensions of 32×32×32×64, where the first 3 numbers represent the x, y, z spatial dimensions and the last number represents the number of filters (also referred to as the number of filter channels). The last layer may not include an activation function to ensure that the generated residual contains both positive and negative values. However, in other examples, an activation function may be included in the last layer. Continuing this example, as Figure 4 As illustrated in , during the training of the neural network, each input LR image 210 of the 3D dataset is divided into isotropic 32x32x32 pixel tiles 402, and all tiles from all training dataset images are used during the training process. During the application phase (e.g., when the trained neural network is used to generate thin slice images), the input image may not have to be divided into tiles.

[0051] In some examples, one or more layers of the deep learning system may be trained to recognize one or more features. In some examples, different layers may recognize different features. Feature recognition may be used as part of optimizing the function generated for each layer. In some examples, the features may be in the form of feature maps. As previously mentioned, an example deep learning system is implemented using cascaded convolution filters paired with nonlinear activation functions. In convolution block 204, the input image (32×32×32 pixel patch) may be convolved by multiple (e.g., 64) 3-dimensional filters with a size of 3×3×3 and a span of 1×1×1. Thus, in the first neural network layer, the input patch is transformed into a feature map with a size of 32×32×32×64. Subsequent layers except the last layer maintain the same size, in which feature reduction is performed to produce a residual image of size 32×32×32. Convolution is performed on the zero-padded input and the output is cropped to the size of the original input in order to maintain the same input and output size. Other filter arrangements may be used in other examples. For example, input patches of different sizes may also be used as feature maps of different sizes.Other feature reduction and / or padding methods may be performed based at least in part on the size of the input image and the desired size of the output image.

[0052] Continuing with the example implementation, in the ReLU block 206, a nonlinear activation function of the form R(x)=max(0,x) is used to ensure that the feature map output is a nonlinear representation of the input. The input low-resolution patch is transformed by the cascaded convolution and ReLU pairs 202, except for the last layer 207 that produces a residual image of size 32×32×32. In this example, the last (20th) convolution layer does not include a ReLU activation to ensure that the residual image can include positive and negative values. However, in other examples, the last layer may include a ReLU activation block 206. The residual image 208 is added to the input image to produce a near high-resolution image. In some examples, the mean square error L2-loss can be compared to the original high-resolution image to evaluate the training. If the mean square error L2-loss is unacceptably high, the deep learning system may require additional training and / or additional or fewer layers may be added to the deep learning system in addition to further training.

[0053] In an example implementation, using a small convolution filter size of 3×3×3 in a deep learning system can avoid averaging high-frequency images and convolution details together. Filters with larger supports can be effectively decomposed into several smaller filters, introducing unnecessary redundancy during the network training process. Therefore, small convolution kernels can effectively improve and / or maximize high-resolution feature extraction. In some examples, a pooling layer can be used, but it can average high-resolution details together. However, this can be desirable in some applications for memory optimization. In some examples, one or more zero padding in the feature map can be performed before convolution and cut to the original input size after convolution in order to maintain equivalent patch input and convolution output size. This ensures that the details at the edge of the patch maintain fidelity. In addition, in some examples, patches can overlap. For example, a patch overlap of 50% of the patch size can be used during the deep learning system training phase. However, in some examples, the test data can be divided into large patches and not overlapped to reduce the network effects caused at the edge of the patch.

[0054] An example implementation of a deep learning system has a depth of 20. The depth may be selected based at least in part on the convolutional filters used and the image patch size. For example, after the first 3×3×3 convolution, the receptive field of each weight in the second layer is 3×3×3. After the second 3×3×3 convolution, the total receptive field of the next weight is 5×5×5, which can be generalized as (2D+1) 3 , where D is the current network depth. Thus, for a patch size of 32×32×32, the receptive field of the weights in layer 16 corresponds to the features encountered in the entire image patch. Additional deeper layers can provide even higher levels of abstraction. Since super-resolution is an inherently ill-posed problem, providing spatial cues at different length scales by adding additional layers can be a prudent optimization, which can be useful given a high-dimensional deep learning system and hyperparameter space. Increasing the described of a deep learning system can improve super-resolution quality. In addition, deep learning models can follow a power-law loss relationship, which can help determine the ideal training data size.

[0055] Return to reference Figure 1 , the reference may be implemented by the processor 108 of the computing system 106 based on the training data set 104 Figure 2-4 The processor 108 may generate executable instructions 110 to implement the training described in reference Figure 2-4The deep learning system described herein and the executable instructions 110 are stored on the memory 112 of the computing system 106. The processor 108 can then execute the stored executable instructions 110 to implement the deep learning system. Once the executable instructions 110 are stored, the executable instructions 110 can be transferred to another computing system. That is, once the deep learning system is trained, it can be provided to other computing systems without retraining. In some examples, the executable instructions 110 can be used to generate hardware to implement some or all of the deep learning system (e.g., an ASIC), which can be provided to one or more computing systems for implementing the deep learning system.

[0056] The training, validation and test data for the example implementation of the deep learning system described herein were obtained from the Osteoarthritis Initiative (OAI). A total of 176 3D sagittal steady-state dual echo (DESS) data sets from OAI were used for training. All images had double knee weight-bearing radiographs to analyze the Kellgren-Lawrence (KL) osteoarthritis grade.

[0057] For training, thick slice images are simulated from a training set of thin slice images. The ratio of the base true slice thickness to the reduced-sampled low-resolution slice thickness is called the reduced-sample factor (DSF). The high-resolution thin slice image represents that the thick slice representation is generated by performing anti-aliasing filtering on the thin slice image in the left and right directions and then reducing the sampling. A 1D finite impulse response low-pass 48-order Hamming windowing filter with a normalized passband of 1 / DSF is generated for retrospective reduced sampling. This simulated acquisition of thicker slices has magnetization from surrounding slices. The training data set of the example implementation scheme includes the base true high-resolution image and the simulated low-resolution image. In some examples, the low-resolution image can be scaled up using 1D triple cubic interpolation (TCI) in the slice direction (left and right) at the base true slice orientation. However, other methods for obtaining thick slice images can be used. For example, a data set including thin slice images and thick slice images obtained from the same volume can be used to train a deep learning system.

[0058] Other datasets of different resolutions may be used in other examples. A training dataset for processing a particular feature may include previous patient images having that feature. For example, in gastrointestinal imaging, a volumetric image of the liver and / or a liver containing lesion morphology of interest may be used to train a deep learning system to generate improved resolution images of the liver.

[0059] In some examples, the training data can be preprocessed prior to training the deep learning system. In some applications, preprocessing the training data can improve the training process and produce a better trained deep learning system. Preprocessing can include formatting the data to ensure uniform size, cropping the image to contain only relevant anatomical structures, uniformly scaling the intensity values, and / or subdividing into tiles.

[0060] Thus, the deep learning systems described herein may be trained using high resolution images (e.g., "ground truth images") and low resolution images simulated based on those high resolution images. The simulated low resolution images may be downsampled versions of the high resolution images. In some examples, after downsampling, the images may be interpolated so that the simulated low resolution images are the same or similar size as the high resolution images (e.g., "ground truth images"). In this way, a set of simulated low resolution images is provided that has a known association with the set of high resolution images. The deep learning systems described herein may be trained on the set of simulated low resolution images (e.g., coefficients and other parameters for operating the deep learning systems may be selected). The sets of coefficients and other parameters may be selected to provide substantially accurate output of the "ground truth images" from the simulated low resolution images.

[0061] Figure 5 An example workflow 500 according to an example of the present disclosure is described. In an example embodiment, a high-resolution base real slice (a) is used to simulate the acquisition of slices (b) with higher cross-sectional thickness for different downsampling factors (DSF). These slices are then triple-interpolated to the base real slice position (c), and the residual function (d) identified by the training deep learning system can be added to the low-resolution thick slice to produce a high-resolution thin slice (e). During inference, a residual image for a test input is generated using the learned residual model. Therefore, the residual can be added to the low-resolution input in order to output a super-resolution image. In other words, the deep learning system can receive a thick slice image as input and generate a simulated thin slice image as output. In some applications, the deep learning system can allow fewer image slices to be acquired while still maintaining sufficient resolution for diagnostic purposes. This can allow for reduced patient imaging time in some applications, and can also improve patient comfort and / or reduce patient exposure to ionizing radiation.

[0062] Figure 6Exemplary coronal deep learning images with different downsampling factors (DSF) (panes af) and corresponding 2D structural similarity (SSIM) maps (pane gl) according to examples of the present disclosure are illustrated. The deep learning images and SSIM maps can be compared to the underlying true slices (e.g., thin slice images actually acquired). Although images with higher DSFs may be smoothed, the directionality of the blur is also more pronounced. In the femoral bone marrow, pixels with incomplete fat saturation appear more blurred in the left-right direction compared to the superior-inferior direction (dashed arrows). The medial collateral ligament may be a tissue to illustrate the image fidelity in the coronal reconstruction, which is that it is very thin in the left-right direction (solid arrows). The SSIM map illustrates that the MCL is not reproduced with high diagnostic quality for DSFs 6 and 8. Coronal images with different DSFs and their SSIM maps indicate that as the DSF increases, the fine details of the medial collateral ligament (MCL) and anterior cruciate ligament (ACL) become blurred (arrows on the SSIM map). Additionally, the remaining femoral bone marrow signal is attributed to incomplete fat suppression, which appears as oversmoothing, reducing the local SSIM. In this example, sagittal images are least affected by blurring as DSF increases, while axial and coronal reconstructions show higher blurring as DSF increases.

[0063] exist Figure 7A comparison of an example implementation of a deep learning system with other super-resolution methods including TCI, FI, and ScSR methods for DSF 3x and a ground truth image is shown in FIG. An example coronal ground truth image (pane a) and a resolution enhanced image with a downsampling factor (DSF) 3x at the same slice orientation for (pane b) DeepResolve, (pane c) Fourier interpolation (FI), (pane d) triple cubic interpolation (TCI), and (pane e) sparse coded super-resolution (ScSR) are shown. Between the compared methods, the residual map (pane f) and the difference image (e.g., difference map) produced by the deep learning system are scaled 5 times to emphasize subtle differences, and the ground truth image (pane gj) is shown. Also shown (pane kn) are the corresponding 2D structural similarity (SSIM) maps for the resolution enhanced image compared to the ground truth. The deep learning system of the present disclosure maintains fine features in the coronal reconstruction and is most similar to the original ground truth image in terms of quality row visual quality and quantitative structural similarity metrics. High resolution features such as the MCL (solid arrow), small bone spurs on the lateral tibial plateau (dashed arrow), and inflammation with sharp features (dashed arrow) are easily visualized on the deep learning system images, however, visualization is much more challenging than other methods (ae). This is further indicated by the difference map (gj) and pixel-wise structural similarity map (kn), where unlike the deep learning system of the present disclosure, TCI, FI, and ScSR all have lower similarity around the medial collateral ligament.

[0064] The deep learning system described in this article can provide the ability to diagnose subtle lesions from thick slices. For example, three pairs of artificial meniscus and cartilage lesions with different signal intensities were simulated, and lesions that were subtle in space and signal intensity were manually generated.

[0065] Figure 8 Simulated ground truth images and resulting generated images according to examples of the present disclosure are shown. The ground truth image (pane a) is modified to include artificial meniscus lesions (solid arrows) and cartilage lesions (dashed arrows) of different signal intensities (panes bd). The simulated meniscus tear is only two pixels thick in the horizontal and vertical directions. By taking the ground truth simulation as input, the DSF of 3x shows that the deep learning system (pane eh) reproduces the lesion reasonably well and has lower blur compared to the TCI image (pane il). Compared to TCI, the deep learning system of the present disclosure always has better similarity metrics (higher SSIM and pSNR, lower MSE) in terms of ground truth.

[0066] Fig. 9The example image shows an example of a horizontal tear in the lateral meniscus body that can be identified by a high-intensity DESS signal. Tears (arrows) can be found relatively similarly in the sagittal ground truth, DeepResolve, and TCI images (panes ac). However, there is some blur in the posterior femoral cartilage where the cartilage thickness is overestimated. The coronal reconstruction also indicates the same meniscus tear (pane df). The scaled section (yellow bracket) of the coronal image shows that the deep learning system image appears smoother and has less noise than the ground truth image (pane gi). Comparing the TCI image with the ground truth and deep learning system images shows that it is significantly more blurred in the TCI image. The boundary of the inflammation in the TCI image (green arrow) does not have the same contour as the ground truth, and the central cartilage has a stepped jagged appearance (dashed arrow) instead of a smooth edge.

[0067] Fig.10 The example image shows an example of a subtle case of grade 2A chondromalacia of the lateral patellar cartilage identified by low-intensity DESS signal. The cartilage (indicated by arrows) shows similarity on all three sagittal images. However, the axial reconstruction shows slightly higher fidelity of the deep learning system image compared to the TCI image (pane ac). The joint effusion (dashed arrow) on the axial image around the lateral femoral condyle shows a significant jagged appearance on the TCI image compared to the ground truth and deep learning system (pane df). The scaled cross-section of the axial reconstruction shows that the contours and uneven signals of the articular cartilage are better maintained in the deep learning system image than in the TCI image, but ultimately not as good as in the ground truth image (pane gi).

[0068] The example of a lateral meniscal tear in all three sets of images shows that the sagittal TCI images appear blurrier compared to the deep learning system images, however, the coronal TCI reconstruction is significantly blurrier than the deep learning system of the present disclosure. The outlines of fine structures such as cartilage, meniscus, and joint inflammation in the deep learning system images appear more similar to the base real images. In contrast to the significant meniscal tear, very subtle grade 2A lateral patellar chondromalacia (according to the modified Noyes scale) was found to be similarly presented on all three sagittal image sets, but the axial deep learning system reconstruction had better image quality compared to TCI. The subtle outlines of the cartilage and cartilage signal inhomogeneities are adequately depicted on the base real images but appear blurrier on the deep learning system and TCI images. However, the deep learning system of the present disclosure does maintain higher image quality compared to TCI. As shown in FIG. Figure 8-10As described in , the deep learning system of the present disclosure has the potential for diagnostic use. For example, a healthy MCL typically has a sub-millimeter thickness in the coronal plane and the deep learning system is able to maintain fine MCL details in acquisitions that simulate a slice thickness of 2.1 mm. In another example, meniscal tears and fine soft bone lesions are visible in images generated by the deep learning system of the present disclosure. In some applications, the deep learning system can be used to acquire slices with a slice thickness of 2-3 mm and then transform them into thinner slices (e.g., higher resolution) for clinical use in multi-planar reconstruction. Such methods may also be particularly suitable for newer embodiments of DESS that implement simultaneous T 2 Relaxation times, morphometry, and semiquantitative radiological assessments. For example, in some applications, thicker slices can be used to generate data for quantitative T 2 High SNR of measurements, while thin sections can be used for accurate morphometry and semi-quantitative whole joint assessment. Bilateral knee imaging approaches that acquire hundreds of slices can also benefit from the deep learning system of the present disclosure.

[0069] Fig.11 1 is an illustration of a system 1100 arranged according to examples described herein. System 1100 may include an image acquisition unit 1102, a computing system 1108, and a workstation 1118. In some embodiments, image acquisition unit 1102, computing system 1108, and workstation 1118 may be integrated into a single unit.

[0070] In some instances, the image acquisition unit 1102 may be a magnetic resonance imaging (MRI) imaging system. In some instances, the image acquisition unit 1102 may be a computed tomography (CT) imaging system. In some instances, the image acquisition unit 1102 may be an ultrasound (US) imaging system. In some instances, the image acquisition unit 1102 may be another imaging modality capable of acquiring image slices of limited thickness. The image acquisition unit 1102 may acquire one or more image slices 1106 from an individual 1104. The individual 1104 may be a human or animal individual. In some instances, the image slice 1106 may include one or more features of interest. In some instances, the features of interest may be anatomical features. Fig.11 In the example shown in , image slice 1106 includes a knee joint. In some examples, image slice 1106 may be provided to computing system 1108.

[0071] In some examples, computing system 1108 may be provided by Figure 1The computing system 1108 may be implemented by the computing system 106 shown in FIG. The computing system 1108 may include one or more memories (not shown) capable of storing executable instructions and one or more processors (not shown) capable of executing the executable instructions. The one or more memories may be configured to store one or more image slices 1106 provided by the image acquisition unit 1102. The computing system 1108 may be configured to implement a deep learning system 1110. The deep learning system 1110 may be configured to generate a high-resolution (e.g., thin-slice) image 1114 from the image slices 1106. In some examples, the deep learning system 1110 may be implemented by Figure 2-4 The high-resolution image 1114 may be provided to a workstation 1118. In some examples, the computing system 1108 may be included in the workstation 1118. In some examples, the computing system 1108 may be remote from the workstation 1118 and may be connected to the workstation 1118 via a network interface (e.g., Figure 1 The network interface 116 shown in FIG. 1 communicates with the workstation 1118 .

[0072] Optionally, in some examples, computing system 1108 may be configured to implement second deep learning system 1112. Second deep learning system 1112 may be configured to receive high-resolution image 1114 and output diagnostic information 1116. Deep learning system 1112 may be trained to recognize anatomical features in high-resolution image 1114 associated with one or more pathologies (e.g., a torn meniscus) and output diagnostic information 1116 associated with the identified pathologies. However, in other examples, other deep learning systems may be used to implement deep learning system 1112. Diagnostic information 1116 may be provided to workstation 1118.

[0073] The workstation 1118 may include a display 1120 and a user interface 1122. Examples of data that may be provided on the display 1120 are shown in box 1124. In some examples, the display 1120 may provide individual information 1126. The individual information may include name, date of birth, ID number, and / or other information. In some examples, the display 1120 may provide image slices 1106 and / or high-resolution images 1114. In some examples, the display 1120 may provide multiple image slices 1106 and / or high-resolution images 1114 simultaneously. In some examples, the display 1120 may provide a variety of other information 1130, such as an option menu (e.g., edit image, save image, obtain measurement values ​​from image), information about the displayed image (e.g., image acquisition unit 1102 settings, anatomical views of slices 1106, 1114) and / or other information (e.g., diagnostic information 1116, physician annotations).

[0074] The workstation 1118 may include a user interface 1122. The user interface 1122 may include any type of hard user controls and / or soft user controls. Examples of hard user controls include, but are not limited to, keyboards, trackballs, mice, and switches. Examples of soft user controls include, but are not limited to, touch screens with software-implemented graphical buttons, sliders, and / or toggles. The user interface 1122 may be configured to receive user input from a user 1132. For example, the user input may determine when and / or under what conditions (e.g., thickness) the image slices 1106 are acquired. In another example, the user input may determine what image is displayed on the display 1120.

[0075] In some applications, the deep learning systems described herein (e.g., deep learning system 1110) may permit an image acquisition unit (e.g., image acquisition unit 1102) to acquire fewer slices (e.g., slice 1106) of a volume of a subject 1104 while still providing images of diagnostic quality, such as high-resolution images 1114. This may improve the performance of the image acquisition unit by reducing image acquisition time. In some applications, the deep learning systems described herein may improve the resolution of slices acquired by the image acquisition unit. For example, image acquisition unit 1102 may acquire several thin image slices 1106 of a volume. The deep learning system 1110 may enhance the resolution of the thin image slices 1106 by simulating even thinner slices (e.g., higher resolution) images 1114.

[0076] A user of the deep learning system described herein (e.g., user 1132) may use the output of the deep learning system (e.g., high-resolution image 1114) to make a diagnosis for a subject (e.g., individual 1104). For example, a physician may be able to diagnose chondromalacia and / or staged osteoarthritis of the lateral patellar cartilage based on features of interest (e.g., knee) visible in high-resolution image 1114. User 1132 may further make a treatment determination based on the diagnosis made from high-resolution image 1114. Examples of treatment options include, but are not limited to, surgery, physical therapy, and supportive bracing. In some instances, it may be impossible to make a diagnosis and / or treatment determination based on the image slices 1106 initially acquired.

[0077] In summary, a deep learning system as described herein may be able to resolve high-resolution thin-slice features from slices that were initially significantly thicker. In other words, the deep learning system produces an image in which each voxel represents a smaller volume than the voxel in the image originally acquired. It should be understood that although the example implementation uses musculoskeletal MRI, other examples of deep learning systems and images of other modalities (e.g., brain MRI or CT) may also be used. In addition, the deep learning system can be used to enhance resolution in all three dimensions. The deep learning system can be implemented on one or more computer systems, such as a system having one or more processing units (e.g., processors, such as central processing units (CPUs) and / or graphics processing units (GPUs)) and a computer-readable medium (e.g., memory, storage device) encoded with executable instructions for performing the deep learning techniques and / or training described herein.

[0078] From the foregoing it will be appreciated that, although specific embodiments have been described herein for purposes of illustration, various modifications may be made while remaining within the scope of the claimed technology.

Claims

1. A method for generating a thin-section image from a thick-section image, the method comprising: Receiving, at a neural network, a first image having a first resolution; performing a convolution on the first image through the neural network; performing a nonlinear activation function on the first image through the neural network; Repeat the convolution and nonlinear activation function; generating a residual based on the convolution; and The residual is summed with the first image through the neural network to generate a second image having a second resolution, wherein the second resolution is higher than the first resolution.

2. The method of claim 1 , wherein the performing the convolution and the performing the nonlinear activation function are performed and repeated in a plurality of layers of the neural network, wherein the first image is an input to a first layer of the plurality of layers, an output of the first layer of the plurality of layers is an input to a second layer of the plurality of layers, and an output of a last layer of the plurality of layers is the residual.

3. The method of claim 1, further comprising training the neural network on a training data set, wherein the training data set comprises a plurality of first images and a plurality of second images.

4. The method of claim 1, further comprising testing the neural network on a test dataset, wherein the test dataset comprises a plurality of first images. The method of claim 1 , wherein performing the convolution comprises performing a three-dimensional convolution.

6. The method of claim 1, wherein training the neural network comprises dividing the first image into a plurality of pixel patches.

7. The method of claim 1, wherein the non-linear activation function comprises the form of R(x)=max(0,x).

8. The method of claim 1, further comprising acquiring the first image from an imaging system.

9. The method of claim 8, wherein the imaging system is a magnetic resonance imaging system.

10. A system for generating a thin-section image from a thick-section image, the system comprising: A non-transitory computer-readable medium comprising instructions for implementing a neural network, wherein the neural network comprises a layer comprising a convolution block and a rectified linear unit non-linear activation block, wherein the layer is configured to generate a residual from a first image having a first resolution received by the neural network, wherein the neural network is configured to sum the first image with the residual to generate a second image having a second resolution, wherein the second resolution is higher than the first resolution; and A processor is configured to execute the instructions to implement the neural network.

11. The system of claim 10, further comprising a display configured to display the second image.

12. The system of claim 10, wherein the convolution block applies a three-dimensional convolution and thresholding using a rectified linear unit function to the first image.

13. The system of claim 10, wherein the neural network comprises a plurality of layers, wherein an output of a first layer of the plurality of layers is provided as an input to a second layer of the plurality of layers.

14. The system of claim 13, wherein a last level of the plurality of levels does not include the rectified linear unit non-linear activation block.

15. The system of claim 13, wherein a last level of the plurality of levels has a size smaller than other levels of the plurality of levels.

16. The system of claim 10, wherein the neural network divides the first image into a plurality of pixel patches. The system of claim 16 , wherein the plurality of pixel tiles overlap.

18. The system of claim 17, wherein the plurality of pixel tiles overlap by 50%.

19. The system of claim 10, wherein the convolution block applies zero-padded convolution and the output of the zero-padded convolution is cropped to the original size of the input image.

20. The system of claim 10, wherein the convolutional block outputs a plurality of feature maps.

21. A system for generating a high-resolution image from a low-resolution image, the system comprising: an image acquisition unit configured to acquire a first image of a feature of interest at a first resolution; a computing system configured to implement a deep learning system, wherein the deep learning system is configured to receive the first image of the feature of interest and perform a convolution on the first image, perform a non-linear activation function on the first image, repeat the convolution and the non-linear activation function, generate a residual based on the convolution, and sum the residual with the first image to generate a second image of the feature of interest at a second resolution, wherein the second resolution is higher than the first resolution; and A display is configured to display the second image of the feature of interest.

22. The system of claim 21, wherein the image acquisition unit is a magnetic resonance imaging system.

23. The system of claim 21, wherein the deep learning system generates the second image by augmenting the first image.

24. The system of claim 21, wherein the computing system is configured to implement a second deep learning system and the second image of the feature of interest is provided as an input to the second deep learning system.

Images