Coarse-slice resolution upsampling for 2D magnetic resonance imaging

Abstract

For coarse slice resolution upsampling, a machine learning model or network infers data or samples along or between slices from 2D MRI data (e.g., from slice stacks) (e.g., creating additional slices). Fast scans can be used to create slice stacks with poor slice-to-slice resolution, and 3D volumes with better resolution across slices can be provided by inference from the slice stacks.

Description

Technical Field

[0001] This invention relates to magnetic resonance (MR) imaging (MRI). Background Technology

[0002] For clinical MRI, high-resolution images are desired. Running 3D sequences is often impractical due to the length of time patients are required to hold their breath or remain still during the procedure and / or the length of time scanning is required when the menstrual cycle is gated. For this reason, high-resolution images are acquired in two dimensions with fine in-plane resolution. A two-dimensional (2D) volume of slices stacked with fine in-plane resolution but coarse slice-to-slice spacing can be acquired more quickly. Poor resolution is provided along one dimension of the patient volume, where high resolution is generally expected along all dimensions. Interpolating new slices to increase resolution across slices may not yield satisfactory results. Summary of the Invention

[0003] By way of description, the following preferred embodiments include methods, systems, instructions, and non-transitory computer-readable media for coarse slice resolution upsampling. A machine learning model or network infers data or samples along or between slices (e.g., creating additional slices) from 2D MRI data (e.g., from a slice stack). A slice stack with poor slice-to-slice resolution can be created using rapid scans, and a 3D volume with better resolution across slices can be provided through inference from the slice stack.

[0004] In a first aspect, a method for coarse-slice resolution upsampling for magnetic resonance imaging (MRI) is provided. An MRI system scans a patient. The scan uses a two-dimensional (2D) imaging protocol to provide samples in a slice stack, where the resolution within each slice is greater than the resolution between slices. A machine learning network performs upsampling between slices. The machine learning network outputs additional slices for upsampling. MRI images are generated from the slice stack and the additional slices.

[0005] In a second aspect, a method for coarse-slice resolution upsampling for magnetic resonance imaging is provided. A two-dimensional (2D) magnetic resonance volume is input into a deep convolutional network. A three-dimensional (3D) magnetic resonance volume is formed from the 2D magnetic resonance volume and the output of the deep convolutional network. A magnetic resonance image is generated from the 3D magnetic resonance volume.

[0006] In a third aspect, a system for increasing the resolution of magnetic resonance imaging is provided. A magnetic resonance scanner is configured to scan a patient. An image processor is configured to reconstruct a volumetric stack of two-dimensional (2D) magnetic resonance slices from the data from the scan, and to infer a three-dimensional (3D) magnetic resonance volume from the volumetric stack via a machine learning model. The 3D magnetic resonance volume has a higher resolution along the slices than the volumetric stack. A display is configured to show a patient image generated from the 3D magnetic resonance volume.

[0007] Any one or more of the aspects or concepts summarized in the illustrative examples above or below may be used alone or in combination. An aspect or concept described for one illustrative example or aspect may be used in other embodiments or aspects. An aspect or concept described for a method or system may be used in other aspects or concepts of a system, method, or non-transitory computer-readable storage medium.

[0008] This invention is defined by the following claims, and nothing in this section should be construed as limiting those claims. The illustrative examples below summarize further aspects. Further aspects and advantages of the invention are discussed below in conjunction with preferred embodiments and may be claimed thereafter, independently or in combination. Attached Figure Description

[0009] Figure 1 This is a flowchart of an example of a method for coarse slice resolution upsampling in MRI; Figure 2 An example 2D MR volume is shown; Figure 3 An example machine learning model for increasing resolution across slices is shown; Figure 4 Example MR images are shown; Figure 5 This is a block diagram of an example MR system used for medical imaging using machine-trained models; Figure 6 An example neural network for upsampling is shown; and Figure 7 An example convolutional network for upsampling is shown. Detailed Implementation

[0010] Coarse slice resolution upsampling is provided for 2D MR imaging. Machine learning models, such as deep convolutional networks, are used to generate high-resolution 3D MR images from corresponding coarse stacks of 2D slices. Fine in-plane resolution is used in inference to optimize upsampling across planes or slices. Machine learning models leverage in-plane high-resolution information (in 2D) already present within one or more 3D volumes to infer finer slice intervals.

[0011] Figure 1 This is a flowchart of one embodiment of a method for coarse slice resolution upsampling for MR imaging. A machine learning model is applied to upsample slices across a 2D MRI volume. This provides speed for 2D MR imaging, but MR images can be generated from higher resolution information across slices, similar to 3D MR imaging.

[0012] This method is by Figure 5 The MR system or another system performs the operation. The MR system scans the patient. The image processor reconstructs and increases the resolution of coarse slices in the reconstructed volume. The display shows the MR image generated from the MR volume. Other components, such as a remote server or workstation, can be used to perform reconstruction, upsampling, and / or display.

[0013] The method is performed in the order shown (top-down or numerical order) or other orders. Additional, different, or fewer actions may be provided. For example, preset, default, or user-input settings are used to configure the scan prior to action 100. As another example, instead of or in addition to the display of action 130, the image is stored in memory (e.g., a computerized patient record) or transmitted via a computer network. In another example, actions 100 and / or 110, such as retrieving the reconstructed 2D MR volume from memory, are not provided.

[0014] In action 100, the MR system scans the patient. The scan results in measurements. A pulse sequence is created based on the MR system configuration (e.g., the selected imaging protocol). The pulse sequence is transmitted from a coil into the patient's body. The resulting response is measured by receiving radio frequency signals at the same or different coils. The scan results in k-space measurements as scan data.

[0015] The scan is protocol-guided. The scanner can fully sample the volume. A 3D sequence with full and / or uniform sampling across the entire volume is performed. 3D MR imaging can be time-consuming, such as 3-5 minutes per bed position. Pharyngeal gating and / or re-scanning due to the patient's inability to hold their breath can cause 3D MR scans to take even longer.

[0016] Sparse sampling can be performed. For 2D MR volumetric imaging, multiple slices are scanned. For each slice, full resolution or high resolution is provided, such as 1 mm or less per sample location. Slices are not spaced at this resolution. Slices can be spaced 2-10 mm or more. For example, slices are parallel but spaced 6-8 mm apart. 2D imaging protocols provide slice stacking, where the resolution within each slice is greater than the resolution between slices.

[0017] Figure 2An example with three slices 200 is shown. This representation is after reconstruction in action 110. Although three slices 200 are shown, four or more slices (e.g., dozens or hundreds) can be stacked. In the X and Y dimensions (slice dimensions) of this example, a relatively high resolution (e.g., 0.9 mm) is provided. In the Z dimension (across slices or between slices), a lower resolution (e.g., 5 mm or more) is provided for the 2D MR volume. For example, a plane 210 extending along one dimension (e.g., the Y dimension) and the Z dimension of a slice has a relatively high resolution along the Y dimension and a relatively poor resolution (e.g., 6 mm) across slices (along the Z dimension).

[0018] Any of a variety of 2D MRI protocols can be used. For example, spin echo, T1, T2, or other protocols using slice sampling can be performed. In one approach, a half-Fourier single-shot excitation turbine spin echo (HASTE) protocol is performed. Using HASTE, the patient volume can be scanned for each bed position in approximately 20 seconds (approximately 10% bias). Other protocols can be used. Scanning for each bed position for less than 20 seconds, 30 seconds, 1 minute, or 2 minutes provides clinically more useful scans, but the disadvantage is lower resolution across slices. 2D MR scanning protocols can be used when 3D scanning protocols are impractical or impossible.

[0019] In action 110, the image processor reconstructs the patient's representation based on k-space scan data. The image processor reconstructs the MR volume, such as... Figure 2 The slices are stacked to form a 2D MR volume of 200. For MR reconstruction, k-space data is converted into an image or object representation, such as scalar values ​​representing different spatial locations. Pixel values ​​or voxel values ​​are reconstructed into an MR volume. This forms a spatial distribution of measurements in the object or image space. This spatial distribution is representative of the patient's image.

[0020] Reconstruction can be achieved using optimization algorithms such as SENSE, GRAPPA, or iterative reconstruction. Alternatively, reconstruction can utilize a machine learning model. The k-space data is fed into the machine learning model, which outputs a volumetric representation (e.g., slice stacking).

[0021] Prior to reconstruction, further processing can be performed on the input k-space measurements. Further processing, such as spatial filtering, color mapping, and / or display formatting, can be performed on the output representation or reconstruction.

[0022] In Action 120, the image processor upsamples between or along slices. Additional slices are created between stacked existing slices. A 3D volume is formed from the stack of 2D slices. Any increase in resolution across slices can be provided, such as increasing the slice size so that the slice dimension Z has a resolution similar to (e.g., + / - 10%) that of the in-plane or intra-slice resolution (e.g., approximately 1.0 or 0.9 mm). In another approach, the increase is provided by resampling, such that a uniformly sampled 3D volume (e.g., 0.9 mm or 1.0 mm per voxel) is provided in all three dimensions. Any increase in resolution is provided, such as doubling the resolution along the Z dimension or slice dimensions. The resolution within a slice (e.g., the X and Y dimensions) can remain the same, increase, or decrease.

[0023] Machine learning models (such as machine learning networks) upsample or increase the resolution. The machine learning model outputs additional slices or voxels of slices to increase the resolution across slices. Figure 3 An example is shown. The machine learning network 310 receives a stack of 300 slices (see example 310). Figure 2 ), and outputs a 3D volume of 320 with higher resolution along the Z dimension (across slices). Figure 3 In the examples, machine learning models double the resolution across slices, such as by forming additional slices for each slice of the input stack of 300, or by forming a 3D volume with twice the resolution along the Z dimension of the input stack of 300. Other increases in resolution, such as 5 or 6 times, can be provided.

[0024] Any machine learning model can be used. Any architecture or layer structure used for machine learning, such as a convolutional neural network, can be used. The architecture defines the structure, learnable parameters, and the relationships between the parameters. In one embodiment, a convolutional, transformer-based, or other neural network is used. Any number of layers and nodes within layers can be used. DenseNet, U-Net, encoder-decoder, deep iterative bottom-up convolutional neural network (CNN), image-to-image, and / or other networks can be used. A portion of the network can include dense blocks (i.e., multiple layers sequentially outputting to the next layer and the last layer in a dense block). Downsampling layers and / or upsampling layers can be included. Skip connections can be used. Any known or later-developed neural network or other deep learning network can be used. Any number of hidden layers and / or nodes can be provided between the input and output layers.

[0025] Figure 3An example of a neural network 310 is shown. The neural network 310 is a cascade of different convolutional, pooling, and upsampling layers 312. The input layer receives slices or 2D MR volumes stacked 300. This network structure forms an encoder with increasing abstraction from the initial layer 312 and a decoder with decreasing abstraction to the final layer 312. Layers 312 may include convolutional layers. Each convolutional layer has a corresponding convolutional kernel, where each kernel is formed by learnable parameters. The output layer outputs additional slices and / or 3D MR volumes 320. Additional, different, or fewer layers may be used. Other types of layers may also be used or can be used alternatively.

[0026] Machine training (such as deep learning) is used to train a defined architecture. The model is trained to receive input (stack 300) and generate output (stack 320) as a response. The values ​​of learnable parameters of the architecture (e.g., kernels) are learned.

[0027] Training data is used to train the model. Training data is obtained from memory, scanning, or transmission. For machine training, training data is created, collected, or accessed.

[0028] Training data includes multiple datasets, such as 2D MR volumes (stack 300) and corresponding outputs (attached slices and / or 3D MR volumes). For example, full 3D volumetric scans are performed on many patients. The resulting reconstructed 3D MR volumes are collected as ground truth. The 3D MR volumes can be downsampled or sampled to simulate 2D MR scans, thus providing one or more stacks 300 for each ground truth 3D MR volume 320 to be output. Dozens, hundreds, or thousands of training samples can be obtained, such as from patient scans, phantom scans, scan simulations, and / or by image processing to create additional samples. In one approach, patient data from one or more hospitals, one or more imaging facilities, or patient health records are used.

[0029] For training, training data stack 300 is input into network 310. The output produced by network 310 is compared with the ground truth. The values ​​of learnable parameters that result in the minimum loss (most similar output) from the ground truth are learned for various inputs to the training data. A computer (e.g., an image processor, workstation, or server) or another machine trains the model (e.g., network 310) to infer additional slices and / or 3D MR volumes 320 from the input stack 300 of 2D MR. Neural network 310 is machine-trained. In one embodiment, deep learning is used to train the model. This training learns both the features of the input data and the transformation of those features to the desired output. Backpropagation, RMSprop, ADAM, or another optimization is used to learn the values ​​of learnable parameters of the network (e.g., a convolutional neural network (CNN) or a fully connected network (FCN)). The difference from the output (e.g., the inferred 3D MR volume 320) to the ground truth is minimized. Any measure of difference, such as the sum of mean squared error (MSE) or mean absolute error (MAE), can be used. L1 loss, L2 loss, or other losses may be used.

[0030] Once trained, the architecture with the learned values ​​is applied. The image processor takes a stack 300 of reconstructed patient scans as input to the machine learning model, which outputs a 3D MR volume 320, such as formatted as additional slices or the full 3D MR volume. In other methods, the machine learning model can be trained to receive k-space data and output a reconstructed 3D MR volume 320.

[0031] When applying a machine learning model to one or more different patients and corresponding different scan data, the same learned weights or learned values ​​are used. At least for a given time period (e.g., weeks, months, or years) or a given number of uses (e.g., tens or hundreds of times), the model and values ​​of the learnable parameters do not vary from patient to patient. These fixed values ​​and the corresponding fixed model are applied sequentially and / or by different processors to different patients. The model can be updated, such as by retraining or replacement, but it will not learn new values ​​as part of the application for a given patient.

[0032] Actions 122 and 124 represent example implementations of increasing coarse slice resolution using machine learning networks or other models. Additional, different, or fewer actions can be provided for upsampling across slices.

[0033] In action 122, the image processor inputs a 2D MR volume into the deep convolutional network 310. A stack of 300 slices 200 is fed into the input channel of the machine learning network. The input slices (images or samples in a plane or region) have a first resolution within the slice that is finer than the resolution across the slice.

[0034] Machine learning networks respond to inputs and outputs. Input data is propagated into network 310, where the values ​​of learned parameters and architecture-controlled features are computed. Outputs are generated by inference. In response to inputs, higher resolution or finer slice intervals along the slice dimension are inferred.

[0035] Additional inputs can be provided. For example, clinical information about the patient can also be input. The machine learning network is trained to use the additional inputs along with 2D MR volumes. As another example, the patient is scanned from different orientations. Multiple 2D MR volumes with slice dimensions relative to the patient's different orientations are input to generate the output. Alternatively, each 2D MR volume is input separately and combined to obtain the resulting 3D MR volume.

[0036] In action 124, the image processor forms a 3D MR volume based on the 2D MR volume and the output of the depth convolutional network 310. The network 310 infers a finer slice interval than the input slice interval. Through inference, the resolution along at least one dimension becomes higher. The fine in-plane resolution of the slices is used for inference to optimize upsampling across planes or slices. The machine learning model utilizes the in-plane high-resolution information (in 2D) already present within the 3D volume(s) to infer the finer slice interval.

[0037] In one approach, network 310 infers or outputs additional slices in response to input. The image processor then adds the additional slices to a stack 300 of the 2D MR volume to create a 3D MR volume. The inferred intermediary slices are combined with the original slices. In another approach, additional slices are inferred as part of the 3D MR volume. Network 310 outputs the 3D MR volume. The complete 3D MR volume is output. Slices in the 3D MR volume may or may not include the input slices 200 (of the 2D MR volume) of the input stack 300. The 3D MR volume is inferred directly by the network.

[0038] The same deep machine learning model (e.g., Network 310) can be used for different patients. The same or different copies of the same machine learning model are applied to different patients, resulting in patient-specific representations or reconstructions using the same values ​​or weights of the model's learned parameters. Different patients and / or the same patients can be scanned at different times while using the same or fixed trained machine learning model. Other copies of the same deep machine learning model can be used for other patients.

[0039] In action 130, an image processor and / or another processor (e.g., a graphics processing unit) generates an MR image from the output (e.g., from slice stack 320 and additional slices). The MR image is generated from a 3D MR volume. A display (e.g., a screen or display device) displays the MR image. After any post-processing or as part of post-processing, the MR image is formatted for display on the display. The display generates an image for viewing by a user, radiologist, internist, clinician, and / or patient. This image aids in diagnosis and / or other clinical purposes.

[0040] The displayed image can represent a planar part or region within the patient. This region can have any orientation relative to the 3D MR volume and the patient. Where the orientation extends at least partially along the stack (i.e., across slices), the increase in resolution across slices or between slices results in an increase in resolution in the MR image. One dimension of a region in the MR image can span slices of a 2D MR volume. Higher resolution is provided to the MR image due to the increase in resolution or upsampling between slices. Pixels along one dimension in the MR image are derived from slices inferred by a machine learning model and additional slices.

[0041] Figure 4 An example is shown. HASTE MR imaging was used on the test subject. The slices are transverse (cross-section), so the slices are perpendicular to the patient's longitudinal axis. The left column shows the same image along the coronal plane (the coronal plane is perpendicular to the cross-section). The right column shows the same image along the sagittal plane (the sagittal plane is perpendicular to the cross-section). For both the coronal and sagittal planes, the Y-axis passes through the slice. The top row shows MR images created from a 2D MR volume without upsampling via a machine learning model. The middle row shows a finer-resolution MR image generated using simple linear interpolation between slices. The bottom row shows MR images at a finer resolution, but based on the output of a 3D MR volume from a machine learning model. The MR images in the middle and bottom rows have the same higher resolution as the top row, but the MR images generated from the 3D MR volume created by the machine learning model look more realistic (sharper or less blurry) than the MR images created by simple interpolation.

[0042] In alternative or additional methods, the displayed image is a volumetric or surface rendering from voxels (a three-dimensional distribution) to a two-dimensional display. 3D MR volumes are used for rendering, such as 3D rendering from a single viewpoint. Higher resolution 3D MR volumes can result in better 3D rendering.

[0043] Figure 5An embodiment of a system for increasing the resolution of MR imaging is illustrated. A high-quality 3D surrogate is generated from a volumetric stack of 2D MRI slices by a machine learning model 552. The system scans or acquires a reconstruction of a given patient and applies the machine learning model 552 to the upsampling.

[0044] This system is implemented by an MR scanner 502 or system, computer, server, or other processor. The MR scanner 502 is merely exemplary, and various MR scanning systems can be used to collect MR data. Figure 5 In some embodiments, the system is or includes an MR scanner 502 or an MR system. The MR scanner 502 is configured to scan a patient. The scan acquires a 2D MR volume (a volume representation of the patient along coarsely spaced slices or planes). The scan provides scan data in the scan domain. The system scans the patient to provide k-space measurements (measurements in the frequency domain).

[0045] In the MR medical scanner 502, the main magnetic coil 500 generates a static basic magnetic field (B0) within the body of the patient 530. The gradient coil 510, in response to a gradient signal provided to it by the gradient and shimming coil control module, generates a position-dependent shimming magnetic field gradient in three orthogonal directions and generates a sequence of magnetic field pulses.

[0046] RF coil 520 (whole-body and / or local coils) generates magnetic field pulses in response to RF pulse signals, which rotate the proton spins within the imaging body of patient 530. Gradient and shimming coil control modules, in conjunction with the RF module, control slice selection, phase encoding, readout gradient magnetic field, RF transmission, and magnetic resonance signal detection under the guidance of a central controller to acquire magnetic resonance signals representing planar slices of patient 530.

[0047] In response to an applied RF pulse signal, RF coil 520 receives an MR signal. The MR signal is detected and processed to provide an MR dataset to image processor 540 for processing into an image (i.e., for reconstruction in the object domain from k-space data in the scan domain). In some embodiments, image processor 540 is located in a central controller, control processor, or control system.

[0048] The MR scanner 502 is configured to scan the patient 530 at each bed position in less than 30 seconds. Instead of scanning the entire 3D MR volume at the same or similar resolution along all three dimensions, the scan is made faster through sparser scanning. Multiple slices or planes are scanned at a coarser resolution between slices and a finer resolution within slices. Longer scan durations can also be used.

[0049] Image processor 540 is an image processor for reconstructing a patient representation from k-space data, upsampling (increasing resolution across slices), and / or rendering MR images. Image processor 540 is a general-purpose processor, digital signal processor, 3D data processor, graphics processing unit, application-specific integrated circuit, field-programmable gate array, artificial intelligence processor, tensor processor, digital circuit, analog circuit, and combinations thereof, and / or another device now known or developed hereafter for upsampling using machine learning model 552. Image processor 540 can be a single device, multiple devices, or a network. For more than one device, parallel or sequential processing partitioning can be used. Different devices comprising image processor 540 can perform different functions, such as reconstruction by one device, upsampling by another device, and rendering by yet another device. In one embodiment, image processor 540 is a control processor of MR scanner 502 or another processor. Other image processors, either within or outside MR scanner 502, can be used.

[0050] Image processor 540 is configured by software, firmware, and / or hardware to apply machine learning model 552 to increase resolution between slices. Image processor 540 operates according to instructions stored on a non-transitory medium (e.g., memory 550) to perform the various actions described herein.

[0051] Image processor 540 is configured to reconstruct a representation in the object domain. The reconstruction is a 2D MR volume or slice stack, with resolution within the slices greater than resolution between slices. Image processor 540 is configured to reconstruct a volumetric stack of 2D MR slices based on data (k-space data) from the scan.

[0052] Image processor 540 is configured to implement machine learning model 552, such as Figure 3 The convolutional neural network 310 or another model is used. The image processor 540 is configured to infer a 3D MR volume from a stack of 2D MR volumes by applying a machine learning model 552. Based on the inference by the machine learning model 552, the 3D MR volume has higher resolution along slices, across slices, or between slices compared to the reconstructed volume stack input to the machine learning model 552. The machine learning model 552 infers additional slices, either as slices to be added to the stack or as slices to form the output 3D MR volume. The additional slices are added to the volume stack to form the 3D MR volume.

[0053] As mentioned above, regarding Figure 1 The image processor 540 applies a machine learning model 552 trained as described above. The machine learning model 552 can be... Figure 3The network or another architecture. For example, machine learning model 552 is formed as an encoder-decoder, U-Net, or another image-to-image network (input space samples to output space samples) deep convolutional network.

[0054] Image processor 540 is used to generate an image of the patient. This image is generated from a 3D MR volume. Due to the increased resolution (higher or better) between slices, the image can better represent the patient without requiring a longer scan.

[0055] Display 560 is a CRT, LCD, plasma, projector, printer, or other display device. Display 560 is configured by loading an image onto a display plane or buffer. Display 560 is configured to display images of the patient generated from a 3D MR volume. The images have the same or similar (+ / -10%) resolution dimensions along the slice. Displaying the patient's images aids in diagnosis.

[0056] Memory 550 stores scan data, machine learning model 552, 2D MR volume, 3D MR volume, MR images, and / or other data. Memory 550 is additionally or alternatively a non-transitory computer-readable storage medium with processing instructions. Memory 550 stores data representing instructions executable by the programmed processor 540, such as instructions for upsampling.

[0057] Instructions for implementing the processes, methods, and / or techniques discussed herein are provided on a non-transitory computer-readable storage medium or memory, such as a cache, buffer, RAM, removable media, hard disk drive, or other computer-readable storage medium. Computer-readable storage media include various types of volatile and non-volatile storage media. Functions, actions, or tasks shown in the accompanying drawings or described herein are performed in response to one or more sets of instructions stored in or on a computer-readable storage medium. Functions, actions, or tasks are independent of a particular type of instruction set, storage medium, processor, or processing strategy, and can be performed individually or in combination by software, hardware, integrated circuits, firmware, microcode, and the like. Similarly, processing strategies can include multiprocessing, multitasking, parallel processing, and the like. In one embodiment, instructions are stored on a removable media device for reading by a local or remote system. In other embodiments, instructions are stored at a remote location for transmission over a computer network or via a telephone line. In yet another embodiment, instructions are stored within a given computer, CPU, GPU, or system.

[0058] Figure 6Embodiments of an artificial neural network 600 (e.g., network 310 or machine learning model 552) according to one or more embodiments are shown. Alternative terms for “artificial neural network” are “neural network,” “artificial neural network,” or “neural network.” Machine learning networks described herein (such as those in…) Figure 1 One or more machine learning-based networks used in step 120 Figure 3 Network 310 Figure 5 The machine learning model 552, or any other machine learning network described herein, can be implemented using an artificial neural network 600.

[0059] The artificial neural network 600 includes nodes 602-622 and edges 632, 634...636, where each edge 632, 634...636 is a directed connection from a first node 602-622 to a second node 602-622. Generally, the first node 602-622 and the second node 602-622 are different nodes, but it is also possible that the first node 602-622 and the second node 602-622 are the same. For example, in... Figure 6 In the diagram, edge 632 is a directed connection from node 602 to node 606, and edge 634 is a directed connection from node 604 to node 606. Edges 632, 634...636 from the first node 602-622 to the second node 602-622 are also represented as "incoming edges" for the second node 602-622 and "outgoing edges" for the first node 602-622.

[0060] In this embodiment, nodes 602-622 of the artificial neural network 600 can be arranged in layers 624-630, wherein these layers may include an inherent order introduced by edges 632, 634...636 between nodes 602-622. Specifically, edges 632, 634...636 may exist only between adjacent layers of nodes. Figure 6 In the illustrated embodiment, there is an input layer 624 comprising only nodes 602 and 604 without incoming edges, an output layer 630 comprising only node 622 without outgoing edges, and hidden layers 626 and 628 between the input layer 624 and the output layer 630. Generally, the number of hidden layers 626 and 628 can be arbitrarily chosen. The number of nodes 602 and 604 in the input layer 624 is typically related to the number of input values ​​of the neural network 600, and the number of nodes 622 in the output layer 630 is typically related to the number of output values ​​of the neural network 600.

[0061] Specifically, (real) numbers can be assigned as values ​​to each node 602-622 of the neural network 600. Here, x (n)i This represents the value of the i-th node 602-622 in the n-th layer 624-630. The values ​​of nodes 602-622 in the input layer 624 are equivalent to the input values ​​of the neural network 600, and the value of node 622 in the output layer 630 is equivalent to the output value of the neural network 600. Furthermore, each edge 632, 634...636 may include a weight as a real number, specifically a real number in the interval [-1, 1] or the interval [0, 1]. Here, w... (m,n) i,j This represents the weight of the edge between the i-th node (602-622) of layer m (624-630) and the j-th node (602-622) of layer n (624-630). Additionally, the abbreviation w... (n) i,j Defined for weight w (n,n+1) i,j .

[0062] Specifically, to calculate the output value of neural network 600, the input value is propagated through the neural network. Specifically, the values ​​of nodes 602-622 in layer (n+1) of layer 624-630 can be calculated based on the values ​​of nodes 602-622 in layer n (624-630) using the following formula: .

[0063] In this paper, the function f is the transfer function (another term is "activation function"). Known transfer functions are step functions, sigmoid functions (e.g., logistic functions, generalized logistic functions, hyperbolic tangent functions, arctangent functions, error functions, smoothstep functions), or rectifier functions. Transfer functions are primarily used for normalization purposes.

[0064] Specifically, these values ​​are propagated layer by layer through the neural network, where the value of the input layer 624 is given by the input of the neural network 600, the value of the first hidden layer 626 can be calculated based on the value of the input layer 624 of the neural network, the value of the second hidden layer 628 can be calculated based on the value of the first hidden layer 626, and so on.

[0065] To set the value w of the edge (m,n) i,j Training data is required to train the neural network 600. Specifically, the training data includes training input data and training output data (denoted as t). i For training, the neural network 600 is applied to the training input data to generate the computed output data. Specifically, the training data and the computed output data include a number of values ​​equal to the number of nodes in the output layer.

[0066] Specifically, the weights within the neural network (600) are recursively adapted using a comparison between the calculated output data and the training data (backpropagation algorithm). Specifically, the weights are changed according to the following formula: Where γ is the learning rate, and if the (n+1)th layer is not the output layer, it can be based on δ. (n+1) j Recursively apply the number δ (n) j The calculation is as follows: , And if the (n+1)th layer is the output layer 630, then it is calculated as follows: Where f' is the first derivative of the activation function, and y (n+1) j It is the comparison training value of the j-th node of the output layer 630.

[0067] Figure 7 A convolutional neural network 700 according to one or more embodiments is illustrated. Machine learning networks described herein (such as, for example, in...) Figure 1 The machine learning-based network used in step 120 Figure 3 Network 310 Figure 5 The machine learning model 552, or any other machine learning network described herein, can be implemented using a convolutional neural network 700.

[0068] exist Figure 7 In the illustrated embodiment, the convolutional neural network 700 includes an input layer 702, a convolutional layer 704, a pooling layer 706, a fully connected layer 708, and an output layer 710. Alternatively, the convolutional neural network 700 may include several convolutional layers 704, several pooling layers 706, several fully connected layers 708, and other types of layers. The order of these layers can be arbitrarily chosen; typically, the fully connected layer 708 is used as the last layer before the output layer 710.

[0069] Specifically, within the convolutional neural network 700, the nodes 712-720 of a layer 702-710 can be considered as arranged as a d-dimensional matrix or a d-dimensional image. In particular, in the two-dimensional case, the value of node 712-720 indexed by i and j in the nth layer 702-710 can be represented as x. (n) [i, j] However, the arrangement of nodes 712-720 in a layer 702-710 has no effect on the computation itself performed within the convolutional neural network 700, because they are given only by the structure and weights of the edges.

[0070] Specifically, the convolutional layer 704 is characterized by forming the structure and weights of the input edges for the convolution operation based on a certain number of kernels. In particular, the structure and weights of the input edges are chosen such that the value x of node 714 of the convolutional layer 704... (n) k Based on the value x of node 712 of the previous layer 702. (n-1) And calculated as convolution x (n) k =K k * x (n-1) In the two-dimensional case, convolution* is defined as: .

[0071] Here, the k-th core K k This is a d-dimensional matrix (a two-dimensional matrix in this embodiment), which is typically small compared to the number of nodes 712-718 (e.g., a 3×3 or 5×5 matrix). Specifically, this means that the weights of the incoming edges are not independent but are chosen such that they produce the convolution formula. Specifically, for the kernel, which is a 3×3 matrix, there are only 9 independent weights (each entry in the kernel matrix corresponds to one independent weight), regardless of the number of nodes 712-720 in the corresponding layers 702-710. Specifically, for convolutional layer 704, the number of nodes 714 in this convolutional layer is equal to the number of nodes 712 in the previous layer 702 multiplied by the number of kernels.

[0072] If the nodes 712 of the previous layer 702 are arranged as a d-dimensional matrix, then using multiple kernels can be interpreted as adding an additional dimension (represented as the "depth" dimension) so that the nodes 714 of the convolutional layer 714 are arranged as a (d+1)-dimensional matrix. If the nodes 712 of the previous layer 702 have already been arranged as a (d+1)-dimensional matrix including the depth dimension, then using multiple kernels can be interpreted as extending along the depth dimension so that the nodes 714 of the convolutional layer 704 are also arranged as a (d+1)-dimensional matrix, where the size of this (d+1)-dimensional matrix with respect to the depth dimension is a multiple of the kernel size in the previous layer 702.

[0073] The advantage of using convolutional layer 704 is that it can take advantage of the spatial local correlation of the input data by enforcing a local connectivity pattern between nodes in neighboring layers, in particular by connecting each node only to a small region of nodes in the previous layer.

[0074] exist Figure 7In the illustrated embodiment, the input layer 702 includes 36 nodes 712 arranged as a two-dimensional 6×6 matrix. The convolutional layer 704 includes 72 nodes 714 arranged as two two-dimensional 6×6 matrices, each of which is the result of convolving the values ​​of the input layer with a kernel. Equivalently, the nodes 714 of the convolutional layer 704 can be interpreted as a three-dimensional 6×6×2 matrix, where the last dimension is the depth dimension.

[0075] The pooling layer 706 is characterized by the activation function of its node 716, which forms the pooling operation based on a nonlinear pooling function f, as well as the structure and weights of the incoming edges. For example, in the two-dimensional case, the value x of node 716 in the pooling layer 706... (n) It can be based on the value x of node 714 of the previous layer 704. (n-1) And calculated as: .

[0076] In other words, by using pooling layer 706, the number of nodes 714 and 716 can be reduced by replacing a number of adjacent nodes 714 (d1·d2) in the previous layer 704 with a single node 716, which is calculated based on the values ​​of the number of adjacent nodes. Specifically, the pooling function f can be a maximum value function, an average value function, or an L2 norm function. Also, for pooling layer 706, the weights of the incoming edges are fixed and are not modified through training.

[0077] The advantage of using pooling layer 706 is that it reduces the number of nodes 714 and 716 and the number of parameters. This leads to a reduction in the computational cost of the network and helps control overfitting.

[0078] exist Figure 7 In the illustrated embodiment, pooling layer 706 is max pooling, replacing four adjacent nodes with only one node whose value is the maximum of the four adjacent node values. Max pooling is applied to each d-dimensional matrix of the previous layer; in this embodiment, max pooling is applied to each of two two-dimensional matrices, thereby reducing the number of nodes from 72 to 18.

[0079] The fully connected layer 708 is characterized by the fact that most, in particular all, edges between node 716 of the previous layer 706 and node 718 of the fully connected layer 708 exist, and the weight of each edge can be adjusted individually.

[0080] In this embodiment, the nodes 716 of the preceding layer 706 of the fully connected layer 708 are displayed both as a two-dimensional matrix and additionally as unrelated nodes (indicated as a row of nodes, where the number of nodes is reduced for better presentability). In this embodiment, the number of nodes 718 in the fully connected layer 708 is equal to the number of nodes 716 in the preceding layer 706. Alternatively, the numbers of nodes 716 and 718 can be different.

[0081] Furthermore, in this embodiment, the value of node 720 in output layer 710 is determined by applying the Softmax function to the value of node 718 in the previous layer 708. By applying the Softmax function, the sum of the values ​​of all nodes 720 in output layer 710 is 1, and all values ​​of all nodes 720 in output layer 710 are real numbers between 0 and 1.

[0082] The convolutional neural network 700 may also include ReLU (Modified Linear Unit) layers or activation layers with non-linear transfer functions. Specifically, the number and structure of nodes included in the ReLU layer are equivalent to the number and structure of nodes included in the previous layer. In particular, the value of each node in the ReLU layer is calculated by applying a correction function to the value of the corresponding node in the previous layer.

[0083] The inputs and outputs of different convolutional neural network blocks can be wired using summation (residual / dense neural networks), element-wise multiplication (attention), or other differentiable operators. Therefore, if the entire pipeline is differentiable, the convolutional neural network architecture can be nested rather than sequential.

[0084] Specifically, a convolutional neural network 700 can be trained based on the backpropagation algorithm. To prevent overfitting, regularization methods can be used, such as dropout of nodes 712-720, random pooling, the use of artificial data, weight decay based on L1 or L2 norm, or maximum norm constraints. Different loss functions can be combined to train the same neural network to reflect the joint training objective. A subset of neural network parameters can be excluded from the optimization to retain weights pre-trained on a different dataset.

[0085] The following are various illustrative examples. Different combinations of aspects are summarized based on these examples. Different combinations of methods or aspects can be used. Example method actions can be provided in the system, and vice versa. Examples used in the application can be used for training.

[0086] Illustrative Example 1. A method for coarse slice resolution upsampling for magnetic resonance imaging, the method comprising: scanning a patient by a magnetic resonance system using a two-dimensional (2D) imaging protocol to provide samples in a slice stack, wherein the resolution within each slice is greater than the resolution between slices; upsampling between slices by a machine learning network, the machine learning network outputting additional slices for upsampling; and generating a magnetic resonance image from the slice stack and the additional slices.

[0087] Illustrative Example 2. According to the method of Illustrative Example 1, the scanning includes scanning using a 2D imaging protocol, which includes the Half Fourier Single Excited Turbine Spin Echo (HASTE) protocol.

[0088] Illustrative Example 3. The method according to any one of Illustrative Examples 1-2, wherein the scan includes scanning each bed position for less than 30 seconds.

[0089] Illustrative Example 4. The method according to any one of Illustrative Examples 1-3, wherein the scanning includes scanning at an intra-slice resolution of 1 mm per pixel or less than 1 mm per pixel after reconstruction and at a slice-by-slice resolution of 5 mm or greater between slices.

[0090] Illustrative Example 5. According to the method of Illustrative Example 4, wherein upsampling includes upsampling such that additional slices between slices in the stack provide a slice-by-slice resolution of 1 mm or less between the slices and the additional slices.

[0091] Illustrative Example 6. According to the method of Illustrative Example 5, generating a magnetic resonance image includes generating a magnetic resonance image having dimensions along the slice stack and additional slices.

[0092] Illustrative Example 7. The method of any one of Illustrative Examples 1-6, wherein the upsampling includes upsampling performed by a machine learning network including a convolutional neural network.

[0093] Illustrative Example 8. According to any one of the illustrative examples 1-7, the upsampling includes inputting slices in a stack into a machine learning network that outputs additional slices in response to the input.

[0094] Illustrative Example 9. The method of any one of Illustrative Examples 1-8, wherein upsampling includes forming a three-dimensional magnetic resonance volume from a stack of slices.

[0095] Illustrative Example 10. A method according to any one of Illustrative Examples 1-9, wherein generating a magnetic resonance image includes generating a 2D image representing a patient region, wherein pixels of the 2D image along one dimension are derived from slices and additional slices.

[0096] Illustrative Example 11. A method for coarse slice resolution upsampling for magnetic resonance imaging, the method comprising: inputting a two-dimensional (2D) magnetic resonance volume into a deep convolutional network; forming a three-dimensional (3D) magnetic resonance volume from the 2D magnetic resonance volume and the output of the deep convolutional network; and generating a magnetic resonance image from the 3D magnetic resonance volume.

[0097] Illustrative Example 12. According to the method of Illustrative Example 11, the 2D magnetic resonance volume includes slices with a first resolution in the slices, the first resolution being finer than a second resolution across the slices, and the input includes slice intervals finer than the second resolution inferred by a deep convolutional network based on information in the slices with the first resolution.

[0098] Illustrative Example 13. A method according to any one of Illustrative Examples 11-12, wherein forming includes outputting a 3D magnetic resonance volume from a deep convolutional network in response to input.

[0099] Illustrative Example 14. According to any one of Illustrative Examples 11-13, the formation includes combining the information as an additional slice output by a deep convolutional network in response to the input with a 2D magnetic resonance volume.

[0100] Illustrative Example 15. A method according to any one of Illustrative Examples 11-14, wherein generation includes generating a magnetic resonance image, the magnetic resonance image comprising a 2D magnetic resonance image representing a region, one dimension of the region being a slice across the 2D magnetic resonance volume.

[0101] Illustrative Example 16. A system for increasing the resolution of magnetic resonance imaging, the system comprising: a magnetic resonance scanner configured to scan a patient; an image processor configured to reconstruct a volumetric stack of two-dimensional (2D) magnetic resonance slices from data from the scan to infer a three-dimensional (3D) magnetic resonance volume from the volumetric stack via a machine learning model, the 3D magnetic resonance volume having a greater resolution along the slices than the volumetric stack; and a display configured to display a patient image generated from the 3D magnetic resonance volume.

[0102] Illustrative Example 17. A system according to Illustrative Example 16, wherein the image processor is configured to infer a 3D magnetic resonance volume by inferring additional slices, and the image processor is configured to insert additional slices into a volume stack to form a 3D magnetic resonance volume.

[0103] Illustrative Example 18. A system based on any one of Illustrative Examples 16-17, wherein the machine learning model includes a deep convolutional network.

[0104] Illustrative Example 19. A system according to any one of Illustrative Examples 16-18, wherein the magnetic resonance scanner is configured to scan each bed position for less than 30 seconds.

[0105] Illustrative Example 20. A system according to any one of Illustrative Examples 16-19, wherein the image has a dimension along the slice, which has the same resolution as the dimension within one of the slices.

[0106] While the subject matter has been described with reference to exemplary embodiments, it is not limited thereto. Rather, the appended claims should be interpreted broadly to include other variations and embodiments that may be made by those skilled in the art.

Claims

1. A method for coarse slice resolution upsampling for magnetic resonance imaging, the method comprising: scanning a patient by a magnetic resonance system, the scan using a two-dimensional (2D) imaging protocol to provide a sample in a stack of slices, where the resolution within each slice is greater than the resolution between the slices; upsampling between the slices by a machine learning network, the machine learning network outputting additional slices for the upsampling; and generating a magnetic resonance image from the stack of slices and the additional slices.

2. The method of claim 1, wherein scanning comprises scanning with a 2D imaging protocol, the 2D imaging protocol comprising a half-Fourier single-shot turbo spin echo (HASTE) protocol.

3. The method of claim 1, wherein scanning comprises scanning for less than 30 seconds per bed position.

4. The method of claim 1, wherein scanning comprises scanning with an in-slice resolution of 1 mm per pixel or less than 1 mm per pixel after reconstruction and a slice-to-slice resolution of 5 mm or more between the slices.

5. The method of claim 4, wherein upsampling comprises upsampling such that the additional slices between the slices in the stack provide a slice-to-slice resolution of 1 mm or less between the slices and the additional slices.

6. The method of claim 5, wherein generating a magnetic resonance image comprises generating a magnetic resonance image having a dimension along the stack of slices and the additional slices.

7. The method of claim 1, wherein upsampling comprises upsampling by a machine learning network comprising a convolutional neural network.

8. The method of claim 1, wherein upsampling comprises inputting the slices in the stack into a machine learning network, the machine learning network outputting the additional slices in response to the input.

9. The method of claim 1, wherein upsampling comprises forming a three-dimensional magnetic resonance volume from the stack of slices.

10. The method of claim 1, wherein generating a magnetic resonance image comprises generating a 2D image representing a region of the patient, where the pixels of the 2D image along one dimension are from the slices and the additional slices.

11. A method for coarse slice resolution upsampling for magnetic resonance imaging, the method comprising: inputting a two-dimensional (2D) magnetic resonance volume to a deep convolutional network; forming a three-dimensional (3D) magnetic resonance volume from the 2D magnetic resonance volume and an output of the deep convolutional network; and generating a magnetic resonance image from the 3D magnetic resonance volume.

12. The method of claim 11, wherein the 2D magnetic resonance volume comprises slices having a first resolution in the slices, the first resolution being finer than a second resolution across the slices, and wherein inputting comprises inferring by the deep convolutional network a slice spacing that is finer than the second resolution based on information in the slices having the first resolution.

13. The method of claim 11, wherein forming comprises outputting by the deep convolutional network the 3D magnetic resonance volume in response to the input.

14. The method of claim 11, wherein forming comprises combining the information as additional slices output by the deep convolutional network in response to the input with the 2D magnetic resonance volume.

15. The method of claim 11, wherein generating includes generating a magnetic resonance image, the magnetic resonance image comprising a 2D magnetic resonance image representing a region, one dimension of the region being a slice across the 2D magnetic resonance volume.

16. A system for increasing the resolution of magnetic resonance imaging, the system comprising: A magnetic resonance scanner, which is configured to scan a patient; An image processor is configured to reconstruct a volumetric stack of two-dimensional (2D) magnetic resonance slices from data from a scan, in order to infer a three-dimensional (3D) magnetic resonance volume from the volumetric stack via a machine learning model, the 3D magnetic resonance volume having a greater resolution along the slice than the volumetric stack; and The display is configured to show patient images generated from a 3D magnetic resonance volume.

17. The system of claim 16, wherein the image processor is configured to infer the 3D magnetic resonance volume by inferring additional slices, and the image processor is configured to insert the additional slices into the volume stack to form the 3D magnetic resonance volume.

18. The system of claim 16, wherein the machine learning model comprises a deep convolutional network.

19. The system of claim 16, wherein the magnetic resonance scanner is configured to scan each bed position for less than 30 seconds.

20. The system of claim 16, wherein the image has a dimension along the slice, the dimension along the slice having the same resolution as the dimension within one of the slices.

Images