Methods, systems, and computer-readable media for generating synthetic imaging data

Abstract

A computer-implemented method, system, and computer-readable medium for generating synthetic imaging data of an anatomical portion of a human subject are disclosed. The method includes: receiving raw imaging data of an anatomical portion of a human subject, the raw imaging data including a plurality of images providing a first imaging modality depiction of the anatomical portion; receiving a convolutional neural network model trained to predict synthetic imaging data based on the raw imaging data; and converting the raw imaging data to synthetic imaging data including a plurality of images similar to the second imaging modality depiction of the anatomical portion by using the convolutional neural network model. Converting the raw imaging data to synthetic imaging data includes: using the convolutional neural network model, generating one or more synthetic images from the plurality of images of the raw imaging data; aggregating a plurality of groups of the one or more generated synthetic images; and determining the synthetic imaging data based on the aggregated plurality of groups of the one or more generated synthetic images.

Description

[0001] This application is a divisional application of the application with national application number 201780065652.5, international application date July 14, 2017, national entry date April 23, 2019, and invention name “Neural Network for Generating Synthetic Medical Images”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to U.S. Provisional Patent Application No. 62 / 384,171, filed on September 6, 2016, and U.S. Provisional Patent Application No. 62 / 408,676, filed on October 14, 2016, the entire disclosures of which are incorporated herein by reference. Technical Field

[0004] The present disclosure generally relates to using machine learning algorithms to generate synthetic images for radiation therapy. More specifically, the present disclosure relates to systems and methods for using neural networks to generate computed tomography (CT) images from magnetic resonance imaging (MRI) images. Background Art

[0005] Traditionally, CT imaging has been used as the main source of image data in the planning process of external radiation therapy. CT images provide an accurate representation of the patient's geometry, and CT values ​​can be directly converted to electron density for radiation dose calculation. However, since conventional CT images are generated by using an imaging radiation source, the use of CT may expose the patient to additional radiation doses. In recent years, there has been an interest in replacing CT images with MRI images during radiotherapy treatment planning. This is because MRI has no ionizing radiation and provides superior soft tissue contrast to achieve more accurate target and structural delineation. MRI also captures functional information of the human body, such as tissue metabolism and function. However, MRI intensity values ​​are not directly related to electron density, and conventional MRI sequences cannot obtain MRI signals from bones. Therefore, it is beneficial if "pseudo-CT images" or "synthetic images", such as pseudo-or synthetic CT images (referred to herein as "synthetic CT images"), can be derived from the acquired MRI images.

[0006] Synthetic CT images derived from MRI images can be used to assist in patient dose calculations in radiotherapy treatment planning or to generate digitally reconstructed radiographs for image guidance. Synthetic CT images can also be used for patient placement during radiotherapy. Therefore, it is desirable to accurately generate synthetic CT images using MRI image data to protect patients from additional radiation exposure caused by CT imaging. Preferably, the synthetic CT images will accurately resemble (resemble) the "real" CT images acquired by a CT scanner. In other words, the synthetic CT should be as close to the real CT image as possible, for example in terms of relevant voxel intensity values ​​and contrast characteristics. Generating accurate synthetic CT images is not a simple task because there is no direct mathematical relationship between CT image intensity values ​​(CT numbers) and MRI intensity values.

[0007] Existing methods for automatic synthetic CT image generation can be roughly divided into three categories: tissue classification-based methods, atlas-based approaches, and model-based methods. Tissue classification or tissue segmentation methods first classify MRI images into regions of different tissue types (e.g., air, fat, soft tissue, or bone), and then use volume density assignment to assign different CT numbers to each tissue type. Automatic tissue segmentation is a difficult problem, especially when using traditional MRI images. For example, bone and air are inseparable or difficult to distinguish in traditional MRI images. Therefore, more complex MRI sequences (e.g., MRI images that can acquire images with higher resolution) are often required for more accurate tissue segmentation. Even if tissue segmentation can be performed, dividing the anatomical structure into several tissue types only provides a very rough approximation of the actual anatomical structure. For example, both the prostate and the bladder are soft tissues, but their CT intensities are typically different.

[0008] Atlas-based methods are also referred to as registration-based approaches. In the context of synthetic CT generation, the term "atlas" generally refers to a data set that includes paired atlas MRI images and atlas CT images. In an example, the pairing of MRI images with CT images can be performed by image registration. In order to generate a synthetic CT image from a given original MRI image, the atlas-based method first aligns each atlas MRI image from the "atlas" with the original MRI image by calculating an image registration transformation (usually requiring a deformable or nonlinear image registration, rather than a simple linear registration). The calculated registration transformation can then be applied to map the corresponding atlas CT image to the original MRI image, and a deformed atlas CT is generated. The deformed atlas CT image provides an estimate of the true CT image (e.g., a synthetic CT image).

[0009] Typically, in order to obtain better accuracy, multiple deformed atlas CTs are calculated and can be combined together, for example by averaging, to generate a final synthetic CT image corresponding to the original MRI image. However, most existing image registration methods can only process single-channel MRI images. In addition, if a large number of atlases are used, the calculation time may increase because the original MRI image must be registered with each atlas MRI image. This process is slow for the following reasons: Since the image registration operation involves all (atlas selection - less than all images may be used) atlas MRI images and atlas CT images, these techniques require continuous access to the complete atlas database. In the example, atlas selection can be used to select a subset of all atlas MRI images and atlas CT images.

[0010] Model-based or learning-based methods aim to build a prediction model based on available training data. Once a prediction model has been trained, it can be applied to any new MRI image of the same type (e.g., any new MRI image acquired using the same MRI sequence as the MRI sequence used to acquire the training data) to predict the corresponding CT image. The training process usually uses a supervised learning algorithm. Specifically, a supervised learning algorithm uses a known set of input data and a known response or output to the data, and then trains the model to generate a reasonable prediction of the response to the new data. For synthetic CT prediction, since the response variable (e.g., CT value) is a real number, a regression (rather than a classification) model is required. Existing model-based methods and traditional learning algorithms also require the extraction of specific types of "features" to be used as input to the model. Existing methods differ in the type of features that can be used by each method as input to its regression model. These existing methods also require manually designed features. In addition, existing methods predict the CT number of a single voxel. Therefore, these methods must be applied in a sliding window manner (e.g., by applying the model to a series of sub-regions of the image) or voxel by voxel to generate a complete synthetic CT image. Such sliding window methods usually require a lot of computing time and resources to generate a complete synthetic CT image.

[0011] Therefore, new systems and methods are needed for generating synthetic CT images from other images, such as MRI images, using artificial intelligence techniques. Summary of the invention

[0012] In one aspect, the present disclosure features a computer-implemented method for generating a composite image of an anatomical portion based on an original image of the anatomical portion acquired by an imaging device using a first imaging modality. The method may include receiving an original image of the anatomical portion acquired by an imaging device using a first imaging modality, and receiving a convolutional neural network model trained to predict a composite image based on the original image. The method may also include converting the original image into a composite image by the convolutional neural network model by at least one processor. The composite image may be similar to a second imaging modality depiction of the anatomical portion, wherein the second imaging modality may be different from the first imaging modality. The first imaging modality may be magnetic resonance imaging, and the second imaging modality may be computed tomography. The first imaging modality and the second imaging modality are selected from the following group: magnetic resonance imaging, computed tomography, ultrasound imaging, positron emission tomography, and single photon emission computed tomography. The method may also include: receiving a plurality of training original images acquired using a first imaging modality; receiving a plurality of training target images acquired using a second imaging modality, wherein each training target image may correspond to a training original image; determining a convolutional neural network architecture; and training a convolutional neural network model using the training original images and the corresponding training target images. The original image may be a two-dimensional image, and the composite image may be a two-dimensional image. The original image may include a stack of two-dimensional images, and the composite image may include a stack of corresponding two-dimensional images. The original image may be a three-dimensional volume, and the composite image may be a three-dimensional volume. The original image may be a three-dimensional image, and generating the composite image may include: selecting a plurality of stacks of adjacent two-dimensional images from the three-dimensional image; converting each stack of adjacent two-dimensional images into a stack of composite two-dimensional images using the convolutional neural network model; and determining the composite image by aggregating the stack of composite two-dimensional images by at least one processor. The original image may be a three-dimensional image, and generating the composite image may include: creating a first stack of two-dimensional images according to a first plane of the three-dimensional image and creating a second stack of two-dimensional images according to a second plane of the three-dimensional image; converting the first stack and the second stack of two-dimensional images into a first stack and a second stack of composite two-dimensional images using the convolutional neural network model; and determining the composite image by aggregating the first stack and the second stack of composite two-dimensional images. Adjacent two-dimensional images may be in the same plane and carry dependent structural information on an axis orthogonal to the plane. Multiple stacks of adjacent two-dimensional images may partially overlap, and aggregating a stack of synthetic two-dimensional images may include averaging the overlapping synthetic two-dimensional images. Converting a first stack of two-dimensional images into a first stack of synthetic two-dimensional images may use a first convolutional neural network model, and converting a second stack of two-dimensional images into a second stack of synthetic two-dimensional images may use a second convolutional neural network model. The first convolutional neural network model and the second convolutional neural network model may correspond to two different anatomical planes.The original image may include a multi-channel image of an anatomical part acquired using different acquisition channels or different acquisition settings of an imaging device. The multi-channel image may include a T1-weighted magnetic resonance image and a T2-weighted magnetic resonance image. The convolutional neural network model may include: a first component configured to determine a feature map of the original image; and a second component configured to determine a synthetic image based on the feature map. The first component may include a plurality of encoding layers and the second component may include a plurality of decoding layers. The first component may be configured to reduce the size of the feature map by downsampling, and the second component may be configured to increase the size of the feature map by upsampling. The convolutional neural network may include a plurality of convolutional layers, wherein the model parameter set may include learnable filter weights for use by the plurality of convolutional layers. The convolutional neural network may include 10 to 500 layers. The convolutional neural network model may include: converting a training original image into a synthetic image using the convolutional neural network model; determining a difference between the synthetic image and a corresponding training target image; and updating a model parameter set of the convolutional neural network based on the difference. Training may be completed when the difference between the synthetic image and the target image is less than a predetermined threshold. The difference may be measured by a loss function calculated based on the synthetic image and the target image.

[0013] In one aspect, the present disclosure may feature a system for generating a composite image of an anatomical part based on an original image of the anatomical part acquired by an imaging device using a first imaging modality. The system may include an input interface configured to receive an original image of the anatomical part acquired by an imaging device using a first imaging modality, and to receive a convolutional neural network model trained to predict a composite image based on the original image. The system may also include at least one storage device configured to store the original image and the convolutional neural network model. The system may also include an image processor configured to convert the original image into a composite image through the convolutional neural network model. The composite image (e.g., a pseudo CT image) may be stored in at least one storage device. The composite image may be similar to a second imaging modality depiction of the anatomical part, wherein the second imaging modality (e.g., CT) may be different from the first imaging modality. The original image may include a multi-channel image of the anatomical part acquired using different acquisition channels or different acquisition settings of the imaging device. The first imaging modality may be magnetic resonance imaging, and the second imaging modality may be computed tomography. The first imaging modality and the second imaging modality may be selected from the following group: magnetic resonance imaging, computed tomography, ultrasound imaging, positron emission tomography, and single photon emission computed tomography. The input interface may also be configured to: receive a plurality of training original images acquired using the first imaging modality, receive a plurality of training target images acquired using the second imaging modality, wherein each training target image may correspond to the training original image. The image processor may also be configured to train the convolutional neural network model using the training original images and the corresponding training target images. The original image may be a two-dimensional image, and the synthesized image may be a two-dimensional image. The original image may include a stack of two-dimensional images, and the synthesized image may include a stack of corresponding two-dimensional images. The original image may be a three-dimensional volume, and the synthesized image may be a three-dimensional volume. The original image may be a 3D image and the image processor may also be configured to: select a plurality of stacks of adjacent two-dimensional images from the 3D image; convert each stack of adjacent two-dimensional images into a stack of synthesized two-dimensional images using the convolutional neural network model; and determine the synthesized image by aggregating the stack of synthesized two-dimensional images. The adjacent two-dimensional images may be in the same plane and may carry dependent structural information on an axis orthogonal to the plane. The plurality of stacks of adjacent two-dimensional images may partially overlap, and aggregating the stacks of composite two-dimensional images may include averaging the overlapping composite two-dimensional images.The original image may be a three-dimensional image, and the image processor may be further configured to: create a first stack of two-dimensional images according to a first plane of the three-dimensional image and create a second stack of two-dimensional images according to a second plane of the three-dimensional image; convert the first stack and the second stack of two-dimensional images into a first stack and a second stack of synthetic two-dimensional images using a convolutional neural network model; and determine a synthetic image by aggregating the first stack and the second stack of synthetic two-dimensional images. The image processor may be configured to: convert the first stack of two-dimensional images into a first stack of synthetic two-dimensional images using a first convolutional neural network model, and convert the second stack of two-dimensional images into a second stack of synthetic two-dimensional images using a second convolutional neural network model, wherein the first convolutional neural network model and the second convolutional neural network model may correspond to two different anatomical planes. The original image may include a multi-channel image of an anatomical part acquired using different acquisition channels of an imaging device. The multi-channel image may include a T1-weighted magnetic resonance image and a T2-weighted magnetic resonance image. The convolutional neural network model may include: a first component configured to determine a feature map of the original image; and a second component configured to determine a synthetic image according to the feature map. The first component may include a plurality of encoding layers and the second component may include a plurality of decoding layers. The first component may be configured to reduce the size of the feature map by downsampling, and the second component may be configured to increase the size of the feature map by upsampling. The convolutional neural network may include a plurality of convolutional layers, wherein the model parameter set may include learnable filter weights for use by the plurality of convolutional layers. The convolutional neural network may include 10 to 500 layers. The image processor may also be configured to: convert the training original image into a synthetic image using the convolutional neural network model; determine the difference between the synthetic image and the corresponding training target image; and update the model parameter set based on the difference. The difference may be measured by a loss function calculated based on the synthetic image and the corresponding training target image.

[0014] In one aspect, the present disclosure may feature a non-transitory computer-readable medium containing instructions that, when executed by at least one processor, cause the at least one processor to perform a method for generating a composite image of an anatomical portion based on an original image of the anatomical portion acquired by an imaging device using a first imaging modality. The method may include receiving an original image of the anatomical portion acquired by an imaging device using a first imaging modality, and receiving a convolutional neural network model trained to predict a composite image based on the original image. The method may also include converting the original image into a composite image by the convolutional neural network model. The composite image may be similar to a second imaging modality depiction of the anatomical portion, wherein the second imaging modality (e.g., CT) may be different from the first imaging modality. The method may also include: receiving a plurality of training original images acquired using a first imaging modality; receiving a plurality of training target images acquired using a second imaging modality, each training target image corresponding to a training original image; determining a convolutional neural network architecture; and training the convolutional neural network model using the training original images and the corresponding training target images. The original image may be a three-dimensional image, and generating the composite image may further include: selecting a plurality of stacks of adjacent two-dimensional images from the three-dimensional image; converting each stack of adjacent two-dimensional images into a stack of composite two-dimensional images using a convolutional neural network model; and determining the composite image by aggregating the stack of composite two-dimensional images by at least one processor. The adjacent two-dimensional images may be in the same plane and carry dependent structural information on an axis orthogonal to the plane. The plurality of stacks of adjacent two-dimensional images may partially overlap, and aggregating the stack of composite two-dimensional images may include averaging the overlapping composite two-dimensional images. The original image may be a three-dimensional image, and generating the composite image may include: creating a first stack of two-dimensional images according to a first plane of the three-dimensional image and creating a second stack of two-dimensional images according to a second plane of the three-dimensional image; converting the first stack and the second stack of two-dimensional images into a first stack and a second stack of composite two-dimensional images using a convolutional neural network model; and determining the composite image by aggregating the first stack and the second stack of composite two-dimensional images. Converting the first stack of two-dimensional images into a first stack of composite two-dimensional images may include using a first convolutional neural network model, and converting the second stack of two-dimensional images into a second stack of composite two-dimensional images may include using a second convolutional neural network model. The first convolutional neural network model and the second convolutional neural network model may correspond to two different anatomical planes. The original image may include a multi-channel image of an anatomical part acquired using different acquisition channels of an imaging device. The multi-channel image may include a T1-weighted magnetic resonance image and a T2-weighted magnetic resonance image. The convolutional neural network model may include: a first component configured to determine a feature map of the original image; and a second component configured to determine a synthesized image based on the feature map. The first component may include multiple encoding layers and the second component may include multiple decoding layers.The first component may be configured to reduce the size of the feature map by downsampling, and the second component may be configured to increase the size of the feature map by upsampling. The convolutional neural network may include a plurality of convolutional layers, wherein the model parameter set may include learnable filter weights for use by the plurality of convolutional layers. The convolutional neural network may include 10 to 500 layers. The model parameter set for training the convolutional neural network model may include: converting a training original image into a synthetic image using the convolutional neural network model; determining a difference between the synthetic image and a corresponding training target image; and updating the model parameter set based on the difference. The difference may be measured by a loss function calculated based on the synthetic image and the corresponding training target image.

[0015] In one aspect, the present disclosure may feature a computer-implemented method for generating a convolutional neural network model for predicting a synthetic image of an anatomical portion based on an original image of the anatomical portion acquired by an imaging device using a first imaging modality. The method may include receiving a plurality of training original images acquired using the first imaging modality. The method may also include receiving a plurality of training target images acquired using a second imaging modality, each training target image corresponding to the training original image. The method may also include training a set of model parameters of a convolutional neural network model using the training original images and the corresponding training target images by at least one processor. The first imaging modality may be magnetic resonance imaging, and the second imaging modality may be computed tomography. The first imaging modality and the second imaging modality may be selected from the following group: magnetic resonance imaging, computed tomography, ultrasound imaging, positron emission tomography, and single photon emission computed tomography. The training original images may include multi-channel images of the anatomical portion acquired using different acquisition channels of the imaging device. The multi-channel images may include T1-weighted magnetic resonance images and T2-weighted magnetic resonance images. The convolutional neural network model may include: a first component configured to determine a feature map of an original image; and a second component configured to determine a synthesized image based on the feature map. The first component may include multiple encoding layers and the second component includes multiple decoding layers. The first component may be configured to reduce the size of the feature map by downsampling, and the second component may be configured to increase the size of the feature map by upsampling. The convolutional neural network may include multiple convolutional layers, and the model parameter set may include learnable filter weights for use by the multiple convolutional layers. The convolutional neural network may include 10 to 500 layers. The model parameter set for training the convolutional neural network model may include: converting the training original image into a synthesized image using the convolutional neural network model; determining the difference between the synthesized image and the corresponding training target image; and updating the model parameter set based on the difference. The difference can be measured by a loss function calculated based on the synthesized image and the corresponding training target image.

[0016] Other purposes and advantages of the embodiments will be explained in part in the following description, and in part will become apparent from the description, or can be learned by practicing the embodiments. It should be understood that both the above general description and the following detailed description are exemplary and illustrative only, and are not intended to limit the claims.

[0017] As used herein, the terms "comprises," "comprising," or any other variations thereof, are intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate the disclosed embodiments and, together with the description, serve to explain the disclosed principles. In the drawings:

[0019] Figure 1A An exemplary radiation therapy system according to some embodiments of the present disclosure is shown.

[0020] Figure 1B An exemplary radiation therapy system is shown that may include a radiation therapy output configured to provide a treatment beam.

[0021] Figure 1C An exemplary system is shown that includes a combined radiation therapy system and an imaging system, such as a computed tomography (CT) imaging system.

[0022] Figure 1D A partial cross-sectional view of an exemplary system including a combined radiation therapy system and an imaging system, such as a magnetic resonance (MR) imaging system, is shown.

[0023] Figure 2 Orthogonal views of exemplary multi-leaf collimator configurations for shaping, directing, or intensity modulating a radiation therapy beam are shown.

[0024] Figure 3 An exemplary Gamma Knife radiotherapy system is shown.

[0025] Figure 4A and Figure 4B The difference between an exemplary MRI image and a corresponding CT image is depicted.

[0026] Figure 4C Depicted are exemplary synthetic CT images generated from acquired MRI images.

[0027] Figure 5 An exemplary image conversion system according to some implementations of the present disclosure is depicted.

[0028] Figure 6 An exemplary image pre-processing process according to some embodiments of the present disclosure is shown.

[0029] Fig. 7A Some embodiments of the present disclosure are shown Figure 5 Operation of an exemplary convolutional layer of a prediction model is shown.

[0030] Figure 7B Some embodiments of the present disclosure are shown Figure 5 Operation of an exemplary convolutional layer stack for a prediction model is shown.

[0031] Figure 7C Some embodiments of the present disclosure are shown Figure 5 Operation of exemplary max pooling and unpooling layers of the prediction model shown.

[0032] Fig.7D and Fig. 7E Depicted are some embodiments of the present disclosure. Figure 5 An exemplary convolutional neural network used by the image translation system shown.

[0033] Figure 7F Depicted are some embodiments of the present disclosure. Fig.7D and Fig. 7E Example feature maps generated by selected layers of the neural network shown.

[0034] Figure 8 is a flowchart illustrating an exemplary training process for training a CNN model according to some embodiments of the present disclosure.

[0035] Fig. 9A is a diagram showing the use of some embodiments of the present disclosure Figure 8 Flowchart of an exemplary image conversion process for a trained CNN model obtained by the process in .

[0036] Fig. 9B is a diagram showing the use of some embodiments of the present disclosure Figure 8 Flowchart of an exemplary 3D image conversion process of at least one trained CNN model obtained by the process in. DETAILED DESCRIPTION

[0037] Exemplary embodiments will now be described in detail with reference to the accompanying drawings. Where possible, the same reference numerals will be used throughout the accompanying drawings to refer to the same or similar parts. In these figures, the leftmost digit of the reference numeral identifies the figure in which the reference numeral first appears. Wherever convenient, the same reference numerals are used throughout the accompanying drawings to refer to the same or similar parts. Although examples and features of the disclosed principles are described herein, modifications, adjustments, and other implementations are possible without departing from the scope of the disclosed embodiments. In addition, the words "comprising", "having", "containing", and "including" and other similar forms are intended to be equivalent in meaning and to be interpreted as open-ended, wherein one or more items following any of these words are not intended to be an exhaustive list of such one or more items, nor are they intended to be limited to the listed one or more items. Moreover, unless the context clearly dictates otherwise, the singular forms "a", "an", and "the" are intended to include plural references. The term "exemplary" is used in the sense of "example" rather than "ideal".

[0038] The disclosed embodiments utilize the principle of transfer learning to use limited data to initialize a convolutional neural network with the parameters of an existing neural network trained for image classification. The disclosed embodiments advantageously may not require linear or deformable inter-subject image registration, and instead may directly learn the mapping between an MRI image and a corresponding CT image. The disclosed embodiments may advantageously provide better accuracy than atlas-based methods using patch refinement and fusion. This is because atlas-based methods typically rely on patch comparisons to find similar atlas candidates, which is also common in other atlas-based or patch-based methods proposed in the literature. Small local patches have limited image information. Therefore, using the original image intensity of the patch as a feature may suffer from a large amount of redundancy in the data and reduce discrimination. In contrast, the disclosed embodiments may advantageously automatically learn a hierarchy of image features of different scales and complexity levels from a complete image.

[0039] One or more methods consistent with the disclosed embodiments have been tested on 18 patients, and the results compare favorably with other reported results in the literature, such as those obtained using the following existing methods: fuzzy c-means clustering methods, methods using atlases for registration, methods using Gaussian mixture regression models, and atlas registration followed by patch fusion. In addition to improving accuracy, the disclosed embodiments also enable shorter computational times when deployed. For example, even though training a model (e.g., a deep convolutional neural network (DCNN)) may take several days, the training only needs to be performed once and the training process can be accelerated by using multiple GPUs, cloud computers, or supercomputers. Applying the trained model on a single GPU to create a synthetic image for a new patient may take only a few seconds. In contrast, existing model-based or atlas-based methods may take longer (e.g., several minutes or hours).

[0040] The disclosed embodiments can also accommodate large amounts of training data (hundreds of pairs of original and target images). Due to the high model capacity, the neural network can benefit greatly from large amounts of data. Therefore, as additional training data becomes available, the accuracy of the disclosed embodiments can be increased. Increasing the amount of training data may increase the training time, but the size of the final neural network and the synthetic image generation time can remain unchanged. In contrast, it may not be practical to increase the training data when using existing atlas-based methods or other model-based methods (such as existing Gaussian process models) because these existing methods require that all training data be retained. In addition, the computational time of these methods is generally directly proportional to the number of atlases used. In addition, the accuracy of atlas-based methods may saturate quickly, and therefore, an atlas selection process is often required to avoid accuracy degradation when using a large number of atlases. However, the use of increased training data can improve the accuracy of the final synthetic image generated by the trained neural network model.

[0041] Figure 1A An exemplary radiation therapy system 10 for providing radiation therapy to a patient is shown. The radiation therapy system 10 includes an image processing device 12. The image processing device 12 may be connected to a network 20. The network 20 may be connected to the Internet 22. The network 20 may connect the image processing device 12 with one or more of the following: a database 24, a hospital database 26, an oncology information system (OIS) 28, a radiation therapy device 30, an image acquisition device 32, a display device 34, and a user interface 36. The image processing device 12 is configured to generate a radiation therapy treatment plan 16 to be used by the radiation therapy device 30.

[0042] The image processing device 12 may include a memory device 16, a processor 14, and a communication interface 18. The memory device 16 may store computer executable instructions, such as an operating system 43, a radiation therapy treatment plan 42 (e.g., an original treatment plan, an adjusted treatment plan, etc.), a software program 44 (e.g., artificial intelligence, deep learning, a neural network, radiation therapy planning software), and any other computer executable instructions to be executed by the processor 14. In an embodiment, the software program 44 may convert a medical image in one format (e.g., MRI) into another format (e.g., CT) by generating a synthetic image such as a pseudo CT image. For example, the software program 44 may include an image processing program that trains a prediction model for converting a medical image 46 of one modality (e.g., an MRI image) into a synthetic image of a different modality (e.g., a pseudo CT image); alternatively, the trained prediction model may convert a CT image into an MRI image. The memory device 16 may store data, including medical images 46, patient data 45, and other data required to create and implement the radiation therapy treatment plan 42.

[0043] In addition to the memory 16 storing the software program 44, it is also expected that the software program 44 can be stored on a removable computer medium, such as a hard drive, a computer disk, a CD-ROM, a DVD, an HD, a Blu-ray DVD, a USB flash drive, an SD card, a memory stick, or any other suitable medium; and when the software program 44 is downloaded to the image processing device 12, the software program 44 can be executed by the image processor 14.

[0044] The processor 14 may be communicatively coupled to the memory device 16, and the processor 14 may be configured to execute computer executable instructions stored thereon. The processor 14 may send or receive a medical image 46 to or from the memory 16. For example, the processor 14 may receive a medical image 46 from the image acquisition device 32 via the communication interface 18 and the network 20 to store it in the memory 16. The processor 14 may also send the medical image 46 stored in the memory 16 via the communication interface 18 to the network 20 to store it in the database 24 or the hospital database 26.

[0045] In addition, the processor 14 can create a radiation therapy treatment plan 42 using a software program 44 (e.g., treatment planning software) along with medical images 46 and patient data 45. The medical images 46 can include information such as imaging data associated with a patient's anatomical region, organ, or segmented data volume of interest. The patient data 45 can include information such as: (1) functional organ modeling data (e.g., serial organs versus parallel organs, appropriate dose response models, etc.); (2) radiation dose data (e.g., dose-volume histogram (DVH) information); or (3) other clinical information about the patient and treatment process (e.g., other surgeries, chemotherapy, previous radiation therapy, etc.).

[0046] In addition, the processor 14 can utilize the software program to generate intermediate data, such as updated parameters to be used by the neural network model; or generate intermediate 2D or 3D images, which can then be stored in the memory 16. The processor 14 can then transmit the executable radiation therapy treatment plan 42 to the radiation treatment device 30 via the communication interface 18 to the network 20, where the radiation therapy plan will be used to treat the patient with radiation. In addition, the processor 14 can execute the software program 44 to implement functions such as image conversion, image segmentation, deep learning, neural networks, and artificial intelligence. For example, the processor 14 can execute a software program 44 for training medical images or delineating medical image contours; such software 44, when executed, can train a boundary detector using a shape dictionary.

[0047] The processor 14 may be a processing device, including: one or more general-purpose processing devices, such as a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), etc. More specifically, the processor 14 may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processor that implements other instruction sets, or a processor that implements a combination of instruction sets. The processor 14 may also be implemented by one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a system on a chip (SoC), etc. As will be appreciated by those skilled in the art, in some embodiments, the processor 14 may be a special-purpose processor rather than a general-purpose processor. The processor 14 may include one or more known processing devices, such as those from Intel. TM Pentium manufactured TM 、Core TM , Xeon TM or series of microprocessors from AMD TMMade by Turion TM , Athlon TM 、Sempron TM 、Opteron TM FX TM 、Phenom TM series of microprocessors, or any of a variety of processors manufactured by Sun Microsystems. Processor 14 may also include a graphics processing unit, such as a graphics processing unit from Nvidia TM Manufactured series, by Intel TM GMA, Iris TM series or by AMD TM Made by Radeon TM Processor 14 may also include an accelerated processing unit, such as an AMD TM Desktop A-4(6,8) series manufactured by Intel TM Xeon Phi manufactured TM The disclosed embodiments are not limited to any type of processor that is otherwise configured to meet the computational requirements of identifying, analyzing, maintaining, generating and / or providing large amounts of data or manipulating such data to perform the methods disclosed herein. In addition, the term "processor" may include more than one processor, for example, a multi-core design or multiple processors each having a multi-core design. Processor 14 may execute sequences of computer program instructions stored in memory 16 to perform various operations, processes, methods described in more detail below.

[0048] The memory device 16 may store medical images 46. In some embodiments, the medical images 46 may include one or more MRI images (e.g., 2D MRI, 3D MRI, 2D flow MRI, 4D MRI, 4D volume MRI, 4D imaging MRI, etc.), functional MRI images (e.g., fMRI, DCE-MRI, diffusion MRI), computed tomography (CT) images (e.g., 2D CT, cone beam CT, 3D CT, 4D CT), ultrasound images (e.g., 2D ultrasound, 3D ultrasound, 4D ultrasound), positron emission tomography (PET) images, X-ray images, fluoroscopic images, radiotherapy portal images, single photon emission computed tomography (SPECT) images, computer-generated synthetic images (e.g., pseudo CT images), etc. In addition, the medical images 46 may also include medical image data, such as training images, ground truth images, and contour images. In an embodiment, the medical images 46 may be received from the image acquisition device 32. Thus, the image acquisition device 32 may include an MRI imaging device, a CT imaging device, a PET imaging device, an ultrasound imaging device, a fluoroscopic device, a SPECT imaging device, an integrated linear accelerator and MRI imaging device, or other medical imaging devices for obtaining medical images of a patient. The medical image 46 may be received and stored in any data type or any format type that the image processing device 12 may use to perform operations consistent with the disclosed embodiments. The memory device 12 may be a non-transitory computer-readable medium such as a read-only memory (ROM), a phase-change random access memory (PRAM), a static random access memory (SRAM), a flash memory, a random access memory (RAM), a dynamic random access memory (DRAM) such as a synchronous DRAM (SDRAM), an electrically erasable programmable read-only memory (EEPROM), a static memory (e.g., flash memory, a flash disk, a static random access memory), and other types of random access memory, a cache memory, a register, a compact disk read-only memory (CD-ROM), a digital versatile disk (DVD) or other optical storage device, a cassette, other magnetic storage device, or any other non-transitory medium that can be used to store information including images, data, or computer executable instructions (e.g., stored in any format) that can be accessed by the processor 14 or any other type of computer device. The computer program instructions can be accessed by the processor 14, read from the ROM or any other suitable memory location, and loaded into the RAM for execution by the processor 14. For example, the memory 16 may store one or more software applications. Software applications stored in memory 16 may include, for example, an operating system 43 for common computer systems and for devices controlled by the software.Furthermore, the memory 16 may store an entire software application or only a portion of a software application that is executable by the processor 14. For example, the memory device 16 may store one or more radiation therapy treatment plans 42.

[0049] The image processing device 12 can communicate with the network 20 via the communication interface 18, which is communicatively coupled to the processor 14 and the memory 16. The communication interface 18 can provide a communication connection between the image processing device 12 and components of the radiation therapy system 10 (e.g., allowing data to be exchanged with external devices). For example, in some embodiments, the communication interface 18 can have appropriate interface circuitry to connect to the user interface 36, which can be a hardware keyboard, keypad, or touch screen through which a user can enter information into the radiation therapy system 10.

[0050] The communication interface 18 may include, for example, a network adapter, a cable connector, a serial connector, a USB connector, a parallel connector, a high-speed data transmission adapter (e.g., optical fiber, USB 3.0, thunderbolt, etc.), a wireless network adapter (e.g., a WiFi adapter), a telecommunication adapter (e.g., 3G, 4G / LTE, etc.), etc. The communication interface 18 may include one or more digital and / or analog communication devices that allow the image processing device 12 to communicate with other machines and devices, such as remotely located components, via the network 20.

[0051] The network 20 can provide the functionality of a local area network (LAN), a wireless network, a cloud computing environment (e.g., software as a service, platform as a service, infrastructure as a service, etc.), a client-server, a wide area network (WAN), etc. For example, the network 20 can be a LAN or a WAN, which can include other systems S1 (38), S2 (40), and S3 (41). The systems S1, S2, and S3 can be the same as the image processing device 12, or can be different systems. In some embodiments, one or more systems in the network 20 can form a distributed computing / simulation environment that collaborates to perform the embodiments described herein. In some embodiments, one or more systems S1, S2, and S3 can include a CT scanner that obtains CT images (e.g., medical images 46). In addition, the network 20 can be connected to the Internet 22 to communicate with servers and clients remotely residing on the Internet.

[0052] Thus, the network 20 may allow for data transfer between the image processing device 12 and a number of different other systems and devices, such as the OIS 28, the radiotherapy device 30, and the image acquisition device 32. In addition, data generated by the OIS 28 and / or the image acquisition device 32 may be stored in the memory 16, the database 24, and / or the hospital database 26. Data may be transmitted / received via the network 20 through the communication interface 18 for access by the processor 14, as desired.

[0053] The image processing device 12 can communicate with the database 24 through the network 20 to send / receive a variety of different types of data stored on the database 24. For example, the database 24 may include machine data as information associated with the radiation therapy device 30, the image acquisition device 32, or other machines related to radiation therapy. The machine data information may include radiation beam size, arc arrangement, beam on and off duration, control points, segmentation, multi-leaf collimator (MLC) configuration, gantry speed, MRI pulse sequence, etc. The database 24 may be a storage device. Those skilled in the art will appreciate that the database 24 may include multiple devices located in a central or distributed manner.

[0054] In some embodiments, the database 24 may include a processor-readable storage medium (not shown). Although the processor-readable storage medium in the embodiments may be a single medium, the term "processor-readable storage medium" should be deemed to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated cache memory and server) storing one or more sets of computer-executable instructions or data. The term "processor-readable storage medium" should also be deemed to include any medium capable of storing or encoding an instruction set for execution by a processor and causing the processor to execute any one or more methods of the present disclosure. Therefore, the term "processor-readable storage medium" should be deemed to include, but is not limited to, solid-state memory, optical media, and magnetic media. For example, a processor-readable storage medium may be one or more volatile, non-transient, or non-volatile tangible computer-readable media.

[0055] The image processor 14 may communicate with the database 24 to read images into the memory 16 or store images from the memory 16 to the database 24. For example, the database 24 may be configured to store a plurality of images (e.g., 3D MRI, 4D MRI, 2D MRI slice images, CT images, 2D fluoroscopic images, X-ray images, raw data from MR scans or CT scans, Digital Imaging and Communications in Medicine (DIMCOM) data, etc.) received by the database 24 from the image acquisition device 32. The database 24 may store data to be used by the image processor 14 when executing the software program 44 or when creating a radiation therapy treatment plan 42. The image processing device 12 may receive imaging data 46 (e.g., 2D MRI slice images, CT images, 2D fluoroscopic images, X-ray images, 3D MRI images, 4D MRI images, etc.) from the database 24, the radiation therapy device 30 (e.g., MRI-Linac), and / or the image acquisition device 32 to generate the treatment plan 42.

[0056] In an embodiment, the radiotherapy system 100 may include an image acquisition device 32 that is capable of acquiring medical images of a patient (e.g., magnetic resonance imaging (MRI) images, 3D MRI, 2D flow MRI, 4D volume MRI, computed tomography (CT) images, cone beam CT, positron emission tomography (PET) images, functional MRI images (e.g., fMRI, DCE-MRI, and diffusion MRI), X-ray images, fluoroscopic images, ultrasound images, radiotherapy portal images, single photon emission computed tomography (SPECT) images, etc.). The image acquisition device 32 may be, for example, an MRI imaging device, a CT imaging device, a PET imaging device, an ultrasound device, a fluoroscopic device, a SPECT imaging device, or any other suitable medical imaging device for obtaining one or more medical images of a patient. The images acquired by the image acquisition device 32 may be stored in the database 24 as imaging data and / or test data. As an example, the images acquired by the image acquisition device 32 may also be stored in the memory 16 by the image processing device 12 as medical image data 46.

[0057] In an embodiment, for example, the image acquisition device 32 can be integrated with the radiation therapy device 30 into a single device (e.g., an MRI device combined with a linear accelerator, also referred to as "MRI-Linac"). Such MRI-Linac can be used, for example, to determine the location of a patient's target organ or target tumor so as to accurately guide the radiation therapy to a predetermined target according to the radiation therapy treatment plan 42.

[0058] The image acquisition device 32 can be configured to acquire one or more images of the patient's anatomical structure for a region of interest (e.g., a target organ, a target tumor, or both). Each image (usually a 2D image or slice) can include one or more parameters (e.g., 2D slice thickness, orientation, and position, etc.). In an example, the image acquisition device 32 can acquire 2D slices with any orientation. For example, the orientation of a 2D slice may include a sagittal orientation, a coronal orientation, or an axial orientation. The processor 14 can adjust one or more parameters such as the thickness and / or orientation of a 2D slice to include a target organ and / or a target tumor. In an example, a 2D slice can be determined based on information such as a 3D MRI volume. For example, in the case of using a radiotherapy device 30, such a 2D slice can be acquired by the image acquisition device 32 in "real time" while the patient is receiving radiotherapy treatment. "Real time" means acquiring data at least in milliseconds or less.

[0059] The image processing device 12 can generate and store a radiation therapy treatment plan 42 for one or more patients. The radiation therapy treatment plan 42 can provide information about the specific radiation dose to be applied to each patient. The radiation therapy treatment plan 42 can also include other radiation treatment information, such as beam angles, dose-histogram-volume information, the number of radiation beams to be used during treatment, the dose per beam, etc.

[0060] Image processor 14 may be implemented using, for example, a processor such as that manufactured by Elekta AB of Stockholm, Sweden. The radiation therapy treatment plan 42 is generated by a software program 44 of the treatment planning software of the image processor 14. To generate the radiation therapy treatment plan 42, the image processor 14 can communicate with the image acquisition device 32 (e.g., a CT device, an MRI device, a PET device, an X-ray device, an ultrasound device, etc.) to access the patient image and depict the target, such as a tumor. In some embodiments, it may be necessary to depict one or more organs at risk (OARs), such as healthy tissue surrounding a tumor or adjacent to a tumor. Therefore, the OARs can be segmented if the OARs are close to the target tumor. In addition, if the target tumor is close to the OARs (e.g., the prostate adjacent to the bladder and rectum), by segmenting the OARs from the tumor, the treatment planning device 110 can not only study the dose distribution in the target, but also study the dose distribution in the OARs.

[0061] In order to depict the target organ or target tumor from the OAR, the medical image of the patient who is receiving radiation therapy can be obtained non-invasively by the image acquisition device 32, for example, MRI image, CT image, PET image, fMRI image, X-ray image, ultrasound image, radiation therapy portal image, SPECT image, etc., to reveal the internal structure of the body part. Based on the information from the medical image, the 3D structure of the relevant anatomical part can be obtained. In addition, during the treatment planning process, multiple parameters can be considered to achieve a balance between effective treatment of the target tumor (for example, the target tumor receives enough radiation dose to achieve effective treatment) and low irradiation of the OAR (for example, the OAR receives the lowest possible radiation dose). Other parameters that can be considered include the position of the target organ and the target tumor, the position of the OAR, and the movement of the target relative to the OAR. For example, the 3D structure can be obtained by depicting the outline of the target or depicting the outline of the OAR in each 2D layer or slice of the MRI or CT image and combining the outline of each 2D layer or slice. The segmentation can be performed manually (e.g., by a physician, dosimetrist, or paramedic) or automatically (e.g., using a program such as the atlas-based automatic segmentation software ABAS manufactured by Elekta AB of Stockholm, Sweden). TM ) Generate a contour. In some embodiments, the 3D structure of the target tumor or OAR can be automatically generated by the treatment planning software.

[0062] After the target tumor and OARs have been located and delineated, the dosimetrist, physician, or medical staff can determine the radiation dose to be applied to the target tumor, as well as any maximum dose that can be received by the OARs adjacent to the tumor (e.g., left and right parotid glands, optic nerves, eyes, lenses, inner ears, spinal cord, brainstem, etc.). After the radiation dose has been determined for each anatomical structure (e.g., target tumor, OAR), a process known as inverse planning can be performed to determine one or more treatment planning parameters that will achieve the desired radiation dose distribution. Examples of treatment planning parameters include volume delineation parameters (e.g., which define target volumes, contour sensitive structures, etc.), margins around target tumors and OARs, beam angle selection, collimator settings, and beam on time. During the inverse planning process, the physician can define dose constraint parameters that set boundaries on how much radiation an OAR can receive (e.g., limiting the full dose to the tumor target and zero dose to any OAR; limiting the dose to 95% of the target tumor; limiting the spinal cord, brainstem, and optic nerve structures to receive ≤45Gy, ≤55Gy, and <54Gy, respectively). The results of the inverse planning can constitute a radiation therapy treatment plan 42 that can be stored in the memory 16 or database 24. Some of these treatment parameters can be related. For example, adjusting one parameter (e.g., the weighting for different targets, such as increasing the dose to the target tumor) in an attempt to change the treatment plan may affect at least one other parameter, which in turn may result in the development of a different treatment plan. Therefore, the image processing device 12 can generate a customized radiation therapy treatment plan 42 with these parameters so that the radiation therapy device 30 provides radiation therapy treatment to the patient.

[0063] In addition, the radiation therapy system 10 may include a display device 34 and a user interface 36. The display device 34 may include one or more display screens that display medical images, interface information, treatment plan parameters (e.g., contours, doses, beam angles, etc.), treatment plans, targets, localization targets, and / or tracking targets, or any related information to the user. The user interface 36 may be a keyboard, keypad, touch screen, or any type of device that a user can use to input information to the radiation therapy system 10. Alternatively, the display device 34 and the user interface 36 may be integrated into a device such as a tablet computer, for example, an Apple Lenovo Samsung wait.

[0064] In addition, any and all components of the radiation therapy system 10 can be implemented as virtual machines (e.g., VMWare, Hyper-V, etc.). For example, a virtual machine can be software used as hardware. Therefore, a virtual machine can include at least one or more virtual processors, one or more virtual memories, and one or more virtual communication interfaces that are used together as hardware. For example, the image processing device 12, the OIS 28, and the image acquisition device 32 can be implemented as virtual machines. Assuming that processing power, memory, and computing power are available, the entire radiation therapy system 10 can be implemented as a virtual machine.

[0065] Figure 1B An exemplary radiation therapy device 102 is shown, which may include a radiation source such as an x-ray source or a linear accelerator, a multi-leaf collimator (not shown), a couch 116, an imaging detector 114, and a radiation therapy output 104. The radiation therapy device 102 may be configured to emit a radiation beam 108 to provide treatment to a patient. The radiation therapy output 104 may include one or more attenuators or collimators, such as described below. Figure 2 An illustrative example of a multi-leaf collimator (MLC) is described.

[0066] Return to reference Figure 1B , a table or bed 116 can be used to position a patient in the area 112 to receive a radiation therapy dose according to the radiation therapy treatment plan. The radiation therapy output 104 can be mounted or attached to a gantry 106 or other mechanical support. When the bed 116 is inserted into the treatment area, one or more chassis motors (chassis notor) (not shown) can rotate the gantry 106 and the radiation therapy output 104 around the bed 116. In an embodiment, the gantry 106 can rotate continuously around the bed 116 when the bed 116 is inserted into the treatment area. In other embodiments, the gantry 106 can rotate to a predetermined position when the bed 116 is inserted into the treatment area. For example, the gantry 106 can be configured to rotate the treatment output 104 around an axis ("A"). Both the couch 116 and the radiation therapy output 104 can be independently moved to other positions around the patient, for example, can be moved in a transverse direction ("T"), can be moved in a lateral direction ("L"), or can be rotated about one or more other axes, such as about a transverse axis (shown as "R"). A controller communicatively connected to one or more actuators (not shown) can control the couch 116 to move or rotate so as to appropriately position the patient in or out of the radiation beam 108 location according to the radiation therapy treatment plan. Because both the couch 116 and the gantry 104 can move independently of each other in multiple degrees of freedom, this allows the patient to be positioned so that the radiation beam 108 can accurately target the tumor.

[0067] Figure 1BThe coordinate system shown (including axes A, T, and L) can have an origin located at an isocenter 110. The isocenter can be defined as the location where the radiotherapy beam 108 intersects the origin of the coordinate axes so as to deliver a prescribed radiation dose to a location on or in the patient. For example, the isocenter 110 can be defined as the location where the radiotherapy beam 108 intersects the patient for various rotational positions of the radiotherapy output 104 positioned by the gantry 106 about the axis A.

[0068] The gantry 106 may also have an attached imaging detector 114. The imaging detector 114 is preferably located opposite the radiation source 104 and may be located within the field of the treatment beam 108 in an example.

[0069] Preferably, the imaging detector 114 can be mounted on the gantry 106 opposite the radiation therapy output 104 so as to maintain alignment with the treatment beam 108. As the gantry 106 rotates, the imaging detector 114 rotates about the rotation axis. In an embodiment, the imaging detector 114 can be a flat panel detector (e.g., a direct detector or a scintillator detector). In this manner, the imaging detector 114 can be used to monitor the treatment beam 108, or the imaging detector 114 can be used to image the patient's anatomy, such as for portal imaging. The control circuit system of the radiation therapy device 102 can be integrated within the system 100 or remote from the system 100, and is functionally composed of, for example, Figure 1A The image processing device 12 is shown.

[0070] In the illustrative example, one or more of the couch 116, the treatment output 104, or the gantry 106 can be automatically positioned, and the treatment output 104 can establish the treatment beam 108 according to the specified dose for a particular treatment delivery instance. A treatment delivery sequence can be specified according to a radiation therapy treatment plan, such as using one or more different orientations or positions of the gantry 106, the couch 116, or the treatment output 104. The treatment deliveries can occur sequentially, but can be staggered in desired treatment sites on or in the patient, such as at the isocenter 110. A prescribed cumulative dose of radiation therapy can thereby be delivered to the treatment site while reducing or avoiding damage to tissue near the treatment site.

[0071] Figure 1CAn exemplary radiation therapy apparatus 102 is shown, which may include a combined linear accelerator and imaging system, such as a computed tomography (CT) imaging system. The CT imaging system may include an imaging X-ray source 118 that provides X-ray energy, for example, in the kiloelectronvolt (keV) energy range. The imaging X-ray source 118 provides a fan-shaped and / or cone-shaped beam 120 that is directed to an imaging detector 122, such as a flat panel detector. The radiation therapy system 102 may be similar to that described with respect to Figure 1B The depicted system 102, for example, includes a radiation therapy output 104, a gantry 106, a platform 116, and an additional flat panel detector 114. An X-ray source 118 may provide a relatively low energy diagnostic X-ray beam for imaging.

[0072] exist Figure 1C In the illustrative example of , the radiotherapy output 104 and the x-ray source 118 can be mounted on the same rotating gantry 106, rotated 90 degrees apart from each other. In other examples, two or more x-ray sources can be mounted along the periphery of the gantry 106, for example, each x-ray source has its own detector arrangement to provide diagnostic imaging at multiple angles simultaneously. Similarly, multiple radiotherapy outputs 104 can be provided.

[0073] Figure 1D An exemplary radiation therapy system 300 is depicted, which may include a combined radiation therapy device and imaging system, such as a magnetic resonance (MR) imaging system (e.g., referred to in the art as MR-Linac) consistent with disclosed embodiments. As shown, the system 300 may include a couch 210, an image acquisition device 140, and a radiation delivery device 130. The system 300 administers radiation therapy to a patient according to a radiation therapy treatment plan. In some embodiments, the image acquisition device 140 may correspond to Figure 1A The image acquisition device 32 in the embodiment can acquire an original image having a first modality (for example, Figure 4A ) or a target image having a second modality (e.g., Figure 4B CT image shown).

[0074] The bed 210 can support a patient (not shown) during the treatment process. In some implementations, the bed 210 can move along a horizontal translation axis (labeled "I") so that the bed 210 can move a patient lying on the bed 210 into and / or out of the system 300. The bed 210 can also rotate around a central vertical rotation axis that is transverse to the translation axis. To allow such movement or rotation, the bed 210 can have motors (not shown) that enable the bed to move in various directions and rotate along various axes. A controller (not shown) can control these movements or rotations according to the treatment plan to appropriately position the patient.

[0075] In some embodiments, the image acquisition device 140 may include an MRI machine for acquiring 2D or 3D MRI images of the patient before, during, and / or after the treatment process. The image acquisition device 140 may include a magnet 146 for generating a main magnetic field for magnetic resonance imaging. The magnetic field lines generated by the operation of the magnet 146 may extend substantially parallel to the central translation axis I. The magnet 146 may include one or more coils whose axes extend parallel to the translation axis I. In some embodiments, one or more coils in the magnet 146 may be spaced apart so that the central window 147 of the magnet 146 is free of coils. In other embodiments, the coils in the magnet 146 may be thin enough or have a reduced density so that they are substantially transparent to the radiation of the wavelength generated by the radiotherapy device 130. The image acquisition device 140 may also include one or more shielding coils that can generate a magnetic field of approximately equal magnitude and opposite polarity outside the magnet 146 to cancel or reduce any magnetic field outside the magnet 146. As described below, the radiation source 134 of the radiation therapy device 130 can be positioned in a region where the magnetic field is at least cancelled to a first order or reduced.

[0076] The image acquisition device 140 may also include two gradient coils 148 and 149, which can generate a gradient magnetic field superimposed on the main magnetic field. The gradient coils 148 and 149 can generate a gradient in the generated magnetic field, which allows the protons to be spatially encoded so that the position of the protons can be determined. The gradient coils 148 and 149 can be arranged around a common central axis with the magnet 146 and can be shifted along the central axis. The shifting can create a gap or window between the coils 148 and 149. In an embodiment where the magnet 146 also includes a central window 147 between the coils, the two windows can be aligned with each other.

[0077] In some embodiments, the image acquisition device 140 may be an imaging device other than MRI, such as X-ray, CT, CBCT, spiral CT, PET, SPECT, optical tomography, fluoroscopic imaging, ultrasound imaging, or a radiation therapy portal imaging device, etc. As will be appreciated by those of ordinary skill in the art, the above description of the image acquisition device 140 relates to a specific embodiment and is not intended to be limiting.

[0078] The radiation therapy device 130 may include a radiation source 134 such as an X-ray source or a linear accelerator and a multi-leaf collimator (MLC) 132 (hereinafter referred to as Figure 2140). The radiotherapy device 130 may be mounted on a chassis 138. When the couch 210 is inserted into the treatment area, one or more chassis motors (not shown) may cause the chassis 138 to rotate about the couch 210. In an embodiment, the chassis 138 may continuously rotate about the couch 210 when the couch 210 is inserted into the treatment area. The chassis 138 may also have an attached radiation detector (not shown), which is preferably positioned opposite the radiation source 134 and wherein the rotation axis of the chassis 138 is located between the radiation source 134 and the detector. In addition, the radiotherapy system 300 may include a control circuit system (not shown) for controlling, for example, one or more of the couch 210, the image acquisition device 140, and the radiotherapy device 130. The control circuit system of the radiotherapy device 130 may be integrated within the radiotherapy system 300 or remote from the radiotherapy system 300.

[0079] During a radiation therapy treatment procedure, a patient may be positioned on the couch 210. The system 300 may then move the couch 210 into a treatment area defined by the magnetic coils 146, 148, 149 and the chassis 138. The control circuitry may then control the radiation source 134, the MLC 132, and the chassis motors to deliver radiation to the patient through the window between the coils 148 and 149 according to the radiation therapy treatment plan.

[0080] Figure 1B , Figure 1C and Figure 1D An example of a radiotherapy device configured to provide radiotherapy treatment to a patient is generally shown, including a configuration that can rotate the radiotherapy output about a central axis (e.g., axis "A"). Other radiotherapy output configurations can be used. For example, the radiotherapy output can be mounted to a robotic arm or manipulator with multiple degrees of freedom. In another example, the treatment output can be fixed, such as located in an area laterally separated from the patient, and a platform supporting the patient can be used to align the radiotherapy isocenter with a designated target site in the patient's body.

[0081] As mentioned above, Figure 1B , Figure 1C and Figure 1D The described radiotherapy apparatus includes a multi-leaf collimator for shaping, directing or intensity modulating a radiotherapy beam to a designated target site within a patient. Figure 2An exemplary multi-leaf collimator (MLC) 132 is shown, which includes leaves 132A-132J that can be automatically positioned to define an aperture that approximates a cross-section or projection of a tumor 140. The leaves 132A-132J allow for modulation of the radiation therapy beam. Depending on the radiation therapy plan, the leaves 132A-132J can be made of a material that is designated to attenuate or block the radiation beam in areas other than the aperture. For example, the leaves 132A-132J can include a metal plate, such as tungsten, with the long axis of the plate oriented parallel to the beam direction and the ends oriented orthogonal to the beam direction (e.g., Figure 1B 140 or other target site. This is in contrast to using a static collimator configuration, or using an MLC 132 configuration that is determined exclusively using "offline" treatment planning techniques. Radiation therapy techniques that use an MLC 132 to produce a specified radiation dose distribution for a tumor or a specific region within a tumor may be referred to as intensity modulated radiation therapy (IMRT).

[0082] Figure 3 An example of another type of radiation therapy device 130 (e.g., a Leksell Gamma Knife) is shown in accordance with some embodiments of the present disclosure. Figure 3 As shown, during the radiotherapy treatment, the patient 202 can wear a coordinate frame 220 to keep the patient's body part (e.g., head) undergoing surgery or radiotherapy stable. The coordinate frame 220 and the patient placement system 222 can establish a spatial coordinate system that can be used when imaging the patient or during radiosurgery. The radiotherapy device 130 may include a protective shell 214 for surrounding multiple radiation sources 212. The radiation source 212 can generate multiple radiation beams (e.g., small beams) through a beam channel 216. Multiple radiation beams can be configured to focus on an isocenter 218 from different directions. Although each individual radiation beam may have a relatively low intensity, when multiple doses from different radiation beams are accumulated at the isocenter 218, the isocenter 218 can receive a relatively high level of radiation. In some embodiments, the isocenter 218 may correspond to a target undergoing surgery or treatment, such as a tumor.

[0083] Figure 4A and Figure 4B Depicted are differences between exemplary images acquired using different imaging modalities, based on which image acquisition device 32 is used. For example, different imaging modalities characterize patient tissue in different ways. Thus, images of the same underlying object acquired using different imaging modalities may not generally resemble each other in terms of image characteristics. For example, Figure 4AA two-dimensional (2D) image of a patient's head acquired using an MRI apparatus for image acquisition is shown; and Figure 4B 2D images of the same object acquired using a CT device are shown. Figure 4A and Figure 4B As shown in , the two images represent the same object, but they exhibit significantly different image characteristics. Figure 4A In the MRI image, the skull and vitreous body have approximately the same intensity values. Figure 4B Very different intensity values ​​for the skull and vitreous are depicted, as indicated by the brightness of the skull shown in the CT image (eg, white outline). Thus, the MRI image intensity values ​​and the CT image intensity values ​​may differ substantially.

[0084] The disclosed systems and methods involve obtaining an original image (e.g., Figure 4A exemplarily shown MRI image) to generate a target image similar to that acquired using a second modality (e.g., Figure 4B A synthetic image (e.g., a pseudo CT image) of a CT image (exemplarily shown).

[0085] Reference Figure 4C , which compares MR and CT images of a patient's head from three different planes and provides an error map. For example, Figure 4C The top row depicts three original images acquired using MRI from three different anatomical planes. The second row depicts synthetic CT images generated based on the MRI images shown in the top row using the disclosed image conversion system and method. The third row depicts images of three planes of the same object acquired using CT. The fourth row shows an error map representing the difference between the synthetic CT images (second row) and the acquired CT images (third row). As shown in the fourth row, for most parts of the head region, the synthetic CT image provides pixel values ​​that are very similar to the CT values ​​in the acquired CT image. Figure 4C In the example of , the error map shows that the errors (e.g., the intensity differences between corresponding voxels in the synthetic CT image and the corresponding real CT image) are minimal (e.g., within the noise level of a typical CT image) and are confined to the interfaces between different tissue types, particularly around the bone-air boundary.

[0086] Figure 5An exemplary image conversion system 400 consistent with the disclosed embodiments is depicted. In some embodiments, the image conversion system 400 may include an image acquisition device 140, an image database 420, an image conversion unit 440, a training database 410 storing training images 412, a training unit 430, and a network 460. The image conversion system 400 may perform two functions: (a) providing a training unit 430 for training a prediction model using the training images 412; (b) providing an image conversion unit 440 configured to generate a synthetic image using the trained prediction model. In an example, the image acquisition device 140 may correspond to Figure 1A The image acquisition device 32 is shown.

[0087] The disclosed systems and methods use predictive models to generate synthetic images. Consistent with the above disclosure, the term "synthetic image" as used herein generally refers to any computer-generated image data that represents or is similar to an image acquired with a different modality (e.g., a CT image) based on actual image data (e.g., an MRI image) acquired from an imaging device using one modality. As explained below, such a synthetic image is converted from an original image acquired using a first modality (e.g., MRI) to an image generated using a second modality (e.g., CT). Predictive models can advantageously enable automatic conversion of images without the need for tissue classification, atlas registration, or model-based methods that require manually customized features.

[0088] In some embodiments, the disclosed systems, methods, devices, and processes relate to generating a synthetic CT image based on an image acquired using another modality, such as an MRI image. In some aspects, the prediction model can be a neural network, such as various types of convolutional neural networks (CNNs) or deep neural networks and other types of artificial intelligence known in the art. Therefore, synthetic image generation can include two stages. In the first stage, a prediction model can be generated (also referred to as "training" or "learning") using training data, the training data including training MRI images and their corresponding real CT images. The training process can automatically use the training data to learn the parameters or weights of the prediction model. These parameters or weights can be iteratively learned according to an optimality criterion. In the second stage, a synthetic CT image (e.g., also referred to as a pseudo CT image) can be generated based on the MRI image using the prediction model.

[0089] An "original image" may refer to an image acquired using a first imaging modality, such as magnetic resonance imaging (MRI). A "target image" may refer to an image acquired using a second modality different from the first modality, such as computed tomography (CT).

[0090] As described above, a "synthetic image" may refer to an image that is generated based on one or more "original images" but is similar to a "target image". In some embodiments, a synthetic image may be referred to as a pseudo CT image. That is, the synthetic image may have the characteristics of a target image. For example, a synthetic CT image ("synthetic image") generated based on one or more MRI images ("original images") may have the characteristics of a real CT image ("target image") acquired by a CT scanner. Throughout this disclosure, examples will be discussed in which the original image is an MRI image and the target image is a CT image. But this is not a limitation. Alternatively, for example, the original image may be a CT image and the target image is an MRI image. In examples in which a synthetic CT image may be generated based on one or more MRI images, the synthetic CT image may be used in a radiation therapy planning process without exposing the patient to ionizing radiation as in a typical CT imaging process.

[0091] As used herein, an image is "similar to" another image if the difference between the two images is less than a predetermined threshold criterion. In an example, if the voxel-to-voxel intensity difference between the two images is less than 20 Hounsfield units, and in some examples, less than 100 Hounsfield units on average, the synthetic CT image may be "similar to" the corresponding CT image. For example, the difference may be determined as an error map indicating the difference in pixel values ​​between the two images. The difference may be measured by a loss function, such as a mean absolute error or a mean square error calculated based on the two images. However, other types of image threshold criteria may be used to show a synthetic image (e.g., a pseudo CT image) consistent with the present disclosure as being similar to a target image. In addition, systems and methods consistent with the present disclosure do not necessarily need to apply any such predetermined threshold criteria to make the target image similar to the original image. In other words and as described below, systems and methods consistent with the present disclosure may use any such predetermined threshold criteria to potentially verify the degree of similarity between the original image and the target image. Systems and methods consistent with the present disclosure may generate a synthetic image (e.g., including a pseudo CT image) that meets a predetermined threshold criterion.

[0092] In order for the training unit 430 to train the prediction model, training images may be utilized. Consistent with the disclosed embodiments, the training database 410 may be configured to store one or more sets of training images 412. Each set of training images may include one or more original images ("original images") acquired using a first imaging modality and corresponding target images ("target images") acquired using a second imaging modality. One or more original images and corresponding target images are acquired for the same object and may include corresponding views of the object (e.g., images of the same anatomical part taken at substantially the same orientation but using different types of image modalities). As described below, the original images and the target images may be 2D images or 3D images. In some embodiments, the database 410 may be part of an oncology information system that manages an oncology treatment plan for a patient. In some aspects, the database 410 may receive these image groups from an image database having images previously acquired by the image acquisition device 140 during one or more radiotherapy treatment processes.

[0093] The image database 420 can store images acquired using the first imaging modality. In some embodiments, the image acquisition device 140 can acquire images stored in the image database 420. The images stored in the image database 420 can also correspond to images acquired during one or more radiation therapy treatment processes. As described below, the images can be 2D images and / or 3D images.

[0094] In some aspects, the image conversion system 400 can be configured to perform a training phase. In some embodiments, the training unit 430 can be configured to train the prediction model using training data from the training database 410. As will be appreciated by those skilled in the art, the training unit 430 can be implemented in hardware and / or software. During the training phase, the training unit 430 can be configured to generate an estimated synthetic image based on the training original image received from the training database 410. The training unit 430 can also generate an error map by comparing the estimated synthetic image with the training target image (e.g., a CT image), and can then adjust the parameters of the prediction model based on the generated error map that identifies the difference between the synthetic image and the real CT image. The training unit 430 can be configured to continue training the prediction model until a specific stopping criterion is met (checking the error between the model prediction and the real CT image, or training based on the number of training iterations, or when the model error is less than a threshold). In an example, the stopping criterion may be met when the number of training iterations exceeds a threshold (e.g., when there have been more than 100 training epochs—in neural network training, an epoch may be equal to the number of iterations to traverse all the training data once). In an example, the stopping criterion may be met if the voxel-to-voxel intensity difference between the two images is less than a threshold (e.g., 20 Hounsfield units). The training unit 430 may then save the trained prediction model, which may be used later by the image conversion unit 440.

[0095] In another aspect, the image conversion system 400 can also be configured to perform a conversion phase. During the conversion phase, the image conversion unit 440 can be configured to receive the trained prediction model from the training unit 430. In some embodiments, the image conversion unit 440 can be configured to generate a synthetic image based on the original image 422 received from the image database 420. As will be appreciated by those skilled in the art, the image conversion unit 440 can be implemented in hardware and / or software. The image conversion unit 440 can be configured to generate a synthetic image using the trained prediction model and the received original image 422.

[0096] In some embodiments, the image conversion system 400 can be configured to display the composite image to be accessed by a user interface (e.g., a graphical user interface provided with a computer, which can also include a tablet computer, iPad, mobile device, etc.), store the composite image in an image database 420 for further use in treatment planning, and provide the composite image to the treatment planning software 115. As another example, the image conversion system 400 can be configured to store the original image and the composite image in the training database 410. These stored images can become part of the training images 412. In various aspects, the storage of the original image and the composite image can be automatic, semi-automatic, or manual. As will be appreciated by those skilled in the art, the image conversion unit 440 can be implemented in hardware and / or software.

[0097] 1 , the image acquisition device 140 may be configured to acquire images using one or more imaging modalities (as described above), including MRI, functional MRI (e.g., fMRI, DCE-MRI, and diffusion MRI), CT, CBCT, spiral CT, PET, SPECT, X-ray, optical tomography, fluoroscopic imaging, ultrasound imaging, and radiotherapy portal imaging, etc. In some embodiments, the image acquisition device 140 may provide the acquired images to the training database 410, the image database 420, the training unit 430, and the image conversion unit 440.

[0098] The network 460 may be configured to provide communication between the components in FIG. 4 . For example, the network 460 may be any type of network (including infrastructure) that provides communication, exchanges information, and / or facilitates electronic information exchange between one or more devices. In this regard, the network 460 may include a wired connection (e.g., a LAN or other hardwired connection), a wireless connection (e.g., WiFi, Bluetooth, ZigBee, NFC, WiMAX, LETS, etc.), a computer bus, a serial connection, a parallel connection, an Ethernet connection, a local or wide area network, an Internet connection, a satellite connection, or any other suitable connection including a connection to a cloud computing service, or any combination thereof, that enables the components of the image conversion system 400 to send and receive information to and from each other in any format and under any communication protocol.

[0099] It is intended that FIG. 4 illustrates only an exemplary arrangement of the image conversion system 400. In some embodiments, additional components may be added, and / or the depicted components may be combined, divided, modified, or removed. For example, the training phase may be performed separately in advance by another system. Thus, the image conversion system 400 may not include the training database 410 or the training unit 430. Furthermore, in some aspects, at least one component of the image conversion system 400 may be located geographically remote from the remaining components and may communicate with the remaining components via the network 460. For example, the training phase including the training unit 430 and the training database 410 may be located in a research and development department; while the conversion phase including the image conversion unit and the original image may be located in a radiation therapy clinic.

[0100] In still other embodiments, two or more components of the image conversion system 400 may be implemented at a single location (e.g., a radiation therapy treatment room). In addition, although FIG. 4 shows various components of the image conversion system 400 as discrete, the image conversion system 400 may implement some of these components in a single device. For example, the database 410 and the training unit 430 may be implemented in a single device, such as a tablet computer, a laptop computer, a desktop computer, a workstation, a server, or a dedicated image conversion device. Similarly, the training unit 430 and the image conversion unit 440 may be implemented in a single device.

[0101] Figure 6 An exemplary image pre-processing process 600 is shown. Process 600 can be performed by training unit 430 as part of a training phase or prior to processing performed in a training phase. Certain aspects of process 600 (e.g., receiving a raw image 601) can also be performed by image conversion unit 440 as part of a conversion phase or prior to a conversion phase. At 601, image conversion system 400 can receive a raw image 422 of a patient acquired using a first imaging modality. Consistent with the disclosed embodiments, the raw image 422 can be received from image acquisition device 140, or alternatively, image conversion system 400 can receive training images 412 from training database 410. The image conversion system then provides the raw image to be normalized.

[0102] At 603, the training unit 430 may normalize the original image 422. The normalization process may improve the accuracy of the image conversion system 400 by normalizing the input image data provided to the prediction model. Here, at 603, the training unit 430 may perform various processing functions on the original image 422 to generate a normalized original image with standardized alignment, resolution, and / or intensity value distribution. For example, the training unit 430 may align the acquired original image 422 with a standard CT image to adjust the orientation of the object in the image. The training unit 430 may also resample the original image 422 so that the normalized original image has a reference resolution and / or a reference field of view. For example, the original image may be upsampled using an interpolation method, or downsampled using an average or maximum pooling method. In an example, the original image may be upsampled or downsampled to provide an original image having the same spatial resolution as the corresponding target image.

[0103] In addition, the training unit 430 can correct the intensity value distribution of the original image 422 to better approximate the standardized intensity value distribution. For example, in the case where the original image 422 is an MRI image (e.g., exemplary image 611), the training unit 430 can use the N3 bias field correction algorithm known in the art to correct the intensity inhomogeneity in the original image. As known in the art, other correction algorithms can be applied, such as joint histogram registration, landmarks or histogram matching, or dynamic histogram warping. At the end of 603, a normalized original image is created and provided to 605.

[0104] At 605, the training unit 230 may generate a binary mask (e.g., exemplary image 615) based on the normalized original image. The training unit 430 may be configured to remove unwanted portions of the normalized original image 422 (e.g., corresponding to a stereotactic head frame) using a binary mask. The training unit 430 may generate a binary mask by applying thresholding and spatial filtering to the normalized original image. The training unit 430 may automatically determine a threshold based on a histogram of image intensity values ​​of the original image 422 according to methods known to those skilled in the art. For example, the training unit 430 may determine a threshold to apply to each intensity value of the normalized original image or to compare with each intensity value of the normalized original image. The training unit 430 may compare each intensity value of the normalized original image with a threshold. The predetermined threshold may be a default intensity value. Through this comparison, the training unit 430 may generate a binary mask image having an intensity value of a logical "1" or "0". The intensity value in the binary mask image depends on whether the corresponding intensity value of the original original image meets or exceeds the threshold. In some embodiments, the training unit 430 may further perform spatial filtering on the generated binary mask image using known morphological operations to generate a final binary mask.

[0105] Then, the training unit 430 can use the generated binary mask to remove the stereotactic head frame in the original image. For example, the portion of the original image within the binary mask can maintain its original intensity value. Then, the other portions of the target image outside the binary mask can be set to a predetermined intensity value.

[0106] At 607, the training unit 430 may receive a target image 412 (e.g., a CT image 617) of the same object acquired using another imaging modality. The target image 412 may be received from the training database 410. The target image 412 may correspond to the object shown in the original image 611. For example, the original image 611 may be an MRI image of a particular portion of the patient's anatomical structure, and the target image 412 may be a corresponding CT image of the same anatomical portion. The training unit 430 may receive the target image 412 from the image acquisition device 140 or the training database 410. For example, in the case where the target image 412 is a CT image, the target image 412 may include a portion of a stereotactic head frame (e.g., image element 618a and image element 618b corresponding to the head frame). Typically, the stereotactic head frame is not shown in the corresponding MRI image (e.g., the original image 611).

[0107] At 609, the training unit 430 may normalize the target image 412. The normalization of the original image 611 is described above, and here, at 609, the normalization of the target image may improve the accuracy of the image conversion system 400. Fig.7D The input data of the prediction model of the neural network 700 is shown to normalize the target image. The training unit 430 can perform various processing functions on the target image to generate a normalized target image with standardized alignment, resolution and / or intensity value distribution.

[0108] In some aspects, an image registration or image alignment process is performed to align the target image with the original image for each pair of training images. This is necessary because the target image and the original image may be acquired by different imaging devices or at different scanning times, so they may not be aligned spatially.

[0109] In some aspects, a binary mask can also be used to segment the target image. For example, the binary mask can correspond to the portion of the image associated with the patient. Then, other portions of the target image outside the binary mask can be set to a predetermined intensity value. The predetermined value can be a default intensity value. For example, in the case where the target image is a CT image, the portion can be set to a Henkel scale value of -1000. In this way, by using a binary mask, a portion of the CT image showing, for example, a stereotactic head frame (e.g., image 619) can be eliminated or reduced in whole or in part because the portion of the head frame is outside the binary mask.

[0110] However, it should be noted that the image conversion system 400 may include the process 600 as an optional step. In some embodiments, the image conversion system 400 does not utilize every aspect of the process 600. For example, the image conversion system 400 may directly train the prediction model using the acquired image. For example, if the image does not contain image features such as a head frame, the process 600 may not be utilized. The disclosed method may be able to train the prediction model without the need for preprocessing to adapt it to the training images. For example, the image conversion system 400 may still generate high-quality synthetic images using a prediction model trained based on training original images and training target images that have not been preprocessed using the functions included in the process 600.

[0111] With or without preprocessing, the training unit 430 can use the training images (including the training original images and the training target images) to train the prediction model. Specifically, the prediction model can be a convolutional neural network (CNN), which is a known type of supervised machine learning. The convolutional neural network may include a stack of different layers that transform the input into the output. The input size, output size, and relationship between the input and output of these layers may be different. Each layer can be connected to one or more upstream and downstream layers in the stack of layers. Therefore, the performance of the convolutional neural network can depend on the number of layers, and the complexity of the convolutional neural network can increase with the increase of the number of layers. If the convolutional neural network has more than one stage of nonlinear feature transformation, which generally means that the number of layers in the network is above a certain number, the convolutional neural network can be regarded as "deep". For example, some convolutional neural networks can include about 10 to 30 layers, or in some cases more than a few hundred layers. Examples of convolutional neural network models include AlexNet, VGGNet, GoogLeNet, ResNet, etc. These convolutional neural network models are designed for image recognition tasks, and in the disclosed embodiments, they can be used at the encoding part of the complete convolutional neural network model.

[0112] 7A to 7E The structure and various components of an exemplary convolutional neural network model are shown. Among other things, a convolutional neural network can generally include multiple convolutional layers, deconvolution layers, pooling (e.g., downsampling) layers, and up-pooling (e.g., upsampling) layers. Fig. 7A The operation of an exemplary convolutional layer of a convolutional neural network is shown. Such a convolutional layer can be configured to perform a two-dimensional convolution or a three-dimensional convolution on the input of the convolutional layer using a set of learnable filter functions. The learnable filter function can be defined by a weight matrix W and an offset value b, where each weight is applied to an image pixel during the convolution operation. The weight matrix W and the offset b are in the model parameters that need to be learned during the training phase. The convolutional layer can be configured to further apply an activation function to the results of these convolutions. Non-limiting examples of activation functions include a rectified linear unit function, a sigmoid function (S-type function), a hyperbolic tangent function, or a similar nonlinear function. The combination of the filter function and the activation function can be expressed as:

[0113] h=max(0,W*X+b) Formula 1

[0114] Formula 1 consists of two parts: a learnable filter function W*X+b and an activation function max(0,·). X can be the input of the filter function W*X+b. In some embodiments, X can be two-dimensional (e.g., an image) or three-dimensional (e.g., a stack of images). The activation function max(0,·) can be applied to the output of the learnable filter function to generate an output h. The output or feature map can indicate the importance of the feature in the input. The activation function max(0,·) can be a rectified linear unit function, which can be used as an activation filter by zeroing the negative values ​​of the convolution operation. In some embodiments, zero padding can be used to ensure that the spatial size of h is equivalent to the spatial size of X. As a non-limiting example, when X is 1024x1024, zero padding can be used to ensure that the spatial size of h is also 1024x1024. When using an nxn filter function, n-1 rows (or columns) of zeros can be added to the edges of the matrix X before X is convolved with the filter. As a non-limiting example, the learnable filter function may be a 3x3 filter, in which case 2 rows of zeros and 2 columns of zeros may be added to X. For example, a row of zeros and a column of zeros may be added to each edge of X.

[0115] For illustration, Fig. 7A The application of a learnable filter function and an activation function to a two-dimensional input feature map in a convolutional layer is shown. Here, an exemplary 3x3 filter function (including a weight matrix 711a and offset values ​​711b) and an activation function are applied to an exemplary 4x4 input feature map 713a to generate a 4x4 output feature map 713c. First, the input feature map 713a can be padded with 2 rows of zeros and 2 columns of zeros, as shown in zero padding 715. Here, a column (or row) of zeros can be added to each edge of the input feature map 713a. However, as will be appreciated by those skilled in the art, the amount of zero padding can depend on the size of the weight matrix 711a, and the specific amount and arrangement of zero padding shown are not intended to be limiting. After zero padding, the input feature map 713a becomes 6x6 and can have a value x in its nth row and mth column. n,m (n,m=1,2,...,6). The weight matrix 711a may include a total of 9 weights. For example, the weight in the jth row and the kth column of the filter matrix is ​​w j,k . In the example shown, the offset value 711b may be a single offset value. The prediction model may be configured to convolve the weight matrix 711a with the zero-padded input feature map 713a, the result of which is shown as the middle image 713b. To perform the convolution, at each location, the weight matrix 711a will overlap a 3x3 portion of the zero-padded input feature map 713a. For example, Fig. 7A As shown, the weight matrix 711a can be 3,5Each pixel value of the input feature map 713a can be multiplied by the corresponding weight overlapped with it, and the multiplication results are added to derive the value h in the intermediate map 713b 2,4 (Convolution output 719b). The convolution layer can generate an output feature map 713c by adding each element of the intermediate map 713b to the offset value 711b and applying the activation function max(0,·) to each resulting sum. For example, Fig. 7A As shown, when the offset value 711b is equal to the value h 2,4 When the sum is less than zero, the activation function output 719c can adjust it to zero. As the learnable filter slides to overlap different parts of the input feature map 713a, other values ​​of the output feature map 713c can be calculated.

[0116] In a similar manner, the convolution layer can apply a learnable filter function and an activation function to a three-dimensional input feature map in the convolution layer. Here, the weight matrix can be three-dimensional. As in the two-dimensional case, the prediction model can be configured to convolve the weight matrix with the zero-padded input feature map to generate an intermediate map. However, in the three-dimensional case, the weight matrix can overlap with the volume of the zero-padded input feature map at each position. For example, when the input feature map includes a stack of three images, the weight matrix can be 3x3x3 and can overlap with the 3x3x3 volume of the zero-padded input feature map during the convolution operation. As in the two-dimensional case, each pixel value of the input feature map can be multiplied with the corresponding weights overlapped therewith, and the multiplication results can be added to derive the corresponding values ​​in the three-dimensional intermediate map. The corresponding value can be added to the offset value, and the activation function can be applied to the result to generate a three-dimensional output feature map.

[0117] The convolutional neural network may include a convolutional layer stack having one or more convolutional layers. In some embodiments, these stacks may include two to five convolutional layers. In addition, different convolutional layer stacks may include different numbers of convolutional layers. Figure 7B The operation of convolutional layer 720, i.e., an exemplary convolutional layer stack, is shown. Convolutional layer 720 may be configured to receive feature map 721a and output feature map 721c. Convolutional layer 720 may include convolutional layer 723a and convolutional layer 723b, convolutional layer 723a may be configured to use feature map 721a to generate feature map 721b, and convolutional layer 723b may be configured to use feature map 721b to generate feature map 721c. The number of feature maps in each of feature map 721a, feature map 721b, and feature map 721c may be predetermined.

[0118] In some embodiments, the feature map 721a may include one or more original images of the object. For example, the feature map 721a may include a T1-weighted MRI image, a T2-weighted MRI image, and an MRI image generated using a contrast agent. As another example, the feature map 721a may include different slices in a 2D image stack of the object. These slices may be adjacent. In various embodiments, the feature map 721a may include a stack of one or more feature maps generated by another component of the prediction model. For example, the feature map 721a may include a stack of sixty-four feature maps generated by the previous convolutional layer of the prediction model.

[0119] The feature map 721b may include feature maps generated by the convolution layer 723a. In some embodiments, the convolution layer 723a may generate these feature maps according to Formula 1. For example, the convolution layer 723a may apply a learnable filter function and an activation function to generate each feature map in the feature map 721b. Each learnable filter function may be described by a weight matrix and an offset value. The values ​​of the weight matrix and the offset are the parameters of the learnable filter that can be learned from the data in the training phase of the convolutional neural network model. The number of parameters in the weight matrix may depend on the spatial size of the learnable filter function and the number of feature maps in the feature map 721a. For example, when the input feature map 721a includes l feature maps and the spatial size of the filter function is m×n, the weight matrix may include m×n×l parameters. When the output feature map 721b includes k feature maps, k learnable filter functions may be used, and therefore, the total number of parameters in the convolution layer 723a will be k×(m×n×l+1). As Figure 7B As shown, the feature maps in feature map 721b and feature map 721a may have the same spatial size. In various embodiments, the number of feature maps in feature map 721a and feature map 721b may be different. For example, feature map 721a may include three feature maps, while feature map 721b may include more or fewer feature maps.

[0120] Feature map 721b can be provided as an input to the next convolutional layer 723b that generates feature map 721c. Similar to convolutional layer 723a, convolutional layer 723b can generate these feature maps according to Equation 1, including applying one or more learnable filter functions and activation functions. In addition, the number of feature maps in feature map 721b and feature map 721c can be the same, or they can be different. For example, feature map 721b can include 64 feature maps, while feature map 721c can include 128 feature maps.

[0121] Figure 7CThe operation of an exemplary maximum pooling layer and an upper pooling layer of a prediction model is shown. The pooling layer can receive an input feature map from a convolutional layer and downsample the received input feature map to generate an output feature map with a reduced size. The output feature map can have a lower spatial resolution than the input feature map, and then, the subsequent convolutional layer can learn image features with a larger spatial range or greater spatial invariance than the image features with higher spatial resolution learned by the previous convolutional layer. Therefore, the prediction model can use a pooling layer to help learn features at different spatial resolution levels, thereby improving conversion accuracy. For example, a pooling layer can use a 2x2 window (i.e., a non-overlapping window) with a stride of two input feature values ​​to downsample the feature map by twice in each dimension. It is expected that the stride may be different and the window used may be any other suitable size, such as 3x3, 4x4, 8x8, 16x16, 32x32, etc.

[0122] In some embodiments, Figure 7C As shown, the pooling layer can be a maximum pooling layer, which selects a single value equal to the maximum value within the pooling window. Figure 7C In the example of , the pooling window is a 2x2 window. Other suitable sizes of the pooling window are contemplated. Figure 7C A portion of a pooling layer 731a is depicted, showing the values ​​within the pooling window 733a. The prediction model can be configured to output the maximum value (i.e., 0.8) within the pooling window 733a to a corresponding position in the downsampled feature map 735a, as indicated by the arrow connecting the pooling layer 731a and the downsampled feature map 735a. The prediction model can also be configured to store the positions of the maximum values ​​within the pooling layer 731a. For example, the prediction model can be configured to store these positions in a binary mask 737, as indicated by the arrow connecting the pooling layer 731a and the binary mask 737. The size of the binary mask 737 can be the same as the size of the pooling layer 731a, and can indicate which input features have maximum values ​​in each window. A pixel value of "1" (or "true") in the binary mask 737 can indicate that the corresponding pixel of the pooling layer 731a contains the maximum value within the window, and a pixel value of "0" (or "false") can similarly indicate that the corresponding pixel of the feature map does not contain the maximum value within the window. As Figure 7CAs shown, the pixels of the binary mask 737 corresponding to the pixel values ​​0.3, 0.6, and 0.1 in the pooling window 733a have a pixel value of "0" because those pixel values ​​are not the maximum pixel values ​​in the pooling window 733a, and the pixels of the binary mask 737 corresponding to the pixel value 0.8 in the pooling window 733a have a pixel value of "1" because 0.8 is the maximum pixel value in the pooling window 733a. The prediction model can be configured to provide this information about the maximum value position to the upper pooling layer, as shown by the arrow connecting the binary mask 737 and the upper pooling layer 731b.

[0123] The upper pooling layer can increase the size of the input feature map by upsampling. The upsampling can increase the spatial resolution of the feature map, thereby increasing the ability of the image conversion system 400 to accurately depict high-resolution details in the synthesized image. In some embodiments, each upper pooling layer can use a 2x2 upper pooling window size that matches the 2x2 pooling window used in the corresponding pooling layer. Therefore, the upper pooling layer can double the size of the feature map in each dimension. Other suitable sizes of the upper pooling window are contemplated.

[0124] In some embodiments, Figure 7C As shown, the prediction model can be configured to correlate the pixel values ​​in the feature map 735b before upsampling with the pixel values ​​in the upper pooling layer 731b. Figure 7C As shown, the pixels in the feature map 735b before upsampling can correspond to the upper pooling window 733b in the upper pooling layer 731b. The binary mask can also define the correspondence between the pixels in the feature map 735b before upsampling and the pixels in the upper pooling layer 731b. The binary mask can be generated by the corresponding pooling layer. For example, the binary mask 737 generated by the corresponding pooling layer 731a can be used by the upper pooling layer 731b. As described above, the binary mask 737 can indicate which pixel values ​​of the pooling layer 731a are stored in the feature map 735a after downsampling. Figure 7C As shown, a pixel with a pixel value of 0.7 in the feature map 735b before upsampling may correspond to an upper right pixel in the upper pooling window 733b, because the upper right value of the corresponding portion of the binary mask 737 has a value of "1". The prediction model may be configured to set the corresponding pixel in the upper pooling layer 731b to the pixel value of the pixel in the feature map 735b before upsampling, as shown by the arrow connecting the feature map 735b before upsampling and the upper pooling layer 731b. The remaining elements of the upper pooling window 733b may be set to predetermined values, such as zero.

[0125] Fig.7D and Fig. 7EA neural network 700 consistent with the disclosed embodiments is depicted, which is an exemplary implementation of a prediction model for use with the image conversion system 400. The neural network 700 can be configured to convert a digital image represented by a matrix of values, where each value corresponds to a pixel in the image. The size of the matrix can correspond to the spatial size of the image. For example, a matrix of 1024x1024 can correspond to an image of spatial size 1024x1024. The matrix representing the original image and the generated matrix representing the composite image can have the same size (or spatial resolution). Unlike conventional methods that predict a composite image on a pixel-by-pixel basis (one value of the matrix at a time), the disclosed method predicts an entire composite image of equal spatial size to the original image (all values ​​of the matrix at a time).

[0126] As described herein, neural network 700 may include input 741 and output 751. In some embodiments, the spatial size (or spatial resolution) of input 741 and output 751 may be the same. For example, input 741 may receive at least one 1024x1024 pixel original image, and output 751 may output at least one 1024x1024 pixel calculated composite image. However, neural network 700 may also process image data having other spatial sizes.

[0127] The input 741 of the neural network 700 can be configured to accept one or more original images. When training the neural network 700 to convert 3D images, the accuracy of the neural network 700 can be increased when the input 741 receives a stack of adjacent images, because the adjacent 2D images may contain dependent structural information. The dependent structural information may include spatial dependencies between anatomical structures shown in the stack of adjacent 2D images. These spatial dependencies may be along an axis orthogonal to the anatomical plane of the 2D image. As a non-limiting example, the shape and type of the anatomical structure represented by the first pixel group of the first image in the stack can also be represented by the second pixel group of the second image adjacent to the first image. This is because the first image and the second image are spatially adjacent to each other along an axis orthogonal to the anatomical plane. Therefore, the two images will have some dependency or continuity on the anatomical structures shown in these images. Therefore, the shape, size and / or type of the anatomical structure in one image can provide additional information about the shape, size and / or type of the anatomical structure in another adjacent image along the same plane. The effect of relying on structural information may depend on various factors, such as the number of adjacent images in the stack, the anatomical structures depicted in the images, and / or the imaging modality used to obtain the images.

[0128] In some embodiments, the input 741 may also be configured to receive a multi-channel MRI image. For example, one or more first channels of the input 741 may be configured to receive a T1-weighted MRI image, and one or more second channels of the input 741 may be configured to receive a T2-weighted MRI image. As known in the art, T1 is a longitudinal relaxation constant, and T2 is a transverse relaxation constant, which controls the rate at which excited protons in a tissue recover their equilibrium state. These rate constants are different from each other and vary from tissue to tissue. Depending on the MRI imaging parameters, the contrast and brightness of the tissue in the MRI image may be determined primarily by the T1 rate constant (T1-weighted MRI image) or the T2 rate constant (T2-weighted MRI image). Therefore, the T1-weighted MRI image and the T2-weighted MRI image convey different information about the imaging object. The neural network 700 may be configured to use additional information provided by using a first channel of a T1-weighted MRI image and another channel of a T2-weighted MRI image to improve the accuracy of the conversion. Additionally or alternatively, channels may be dedicated to images associated with other tissue parameters known in the art (eg, spin density), or to MRI images acquired using contrast agents.

[0129] like Fig.7D and Fig. 7E As shown, image data can flow from input 741 to output 751 through encoding stage 740 and decoding stage 750. Encoding stage 740 can be configured to extract feature maps from input images. In some implementations, encoding stage 740 may not include fully connected layers. Fully connected layers are typically layers in which each pixel of the input feature map contributes to each pixel of the output feature map. As known in the art, fully connected layers require a very large number of parameters and generate features suitable for image classification tasks. These features are of little benefit in this application, and excluding fully connected layers can reduce the number of parameters required for training convolutional neural networks. Decoding stage 750 can be configured to convert the feature map output from encoding stage 740 into a synthetic image having the same spatial resolution as the original image.

[0130] like Fig.7D and Fig. 7E As shown, the neural network 700 may include a stack of convolutional layers (e.g., convolutional layer 743a and convolutional layer 753a), a pooling layer (e.g., pooling layer 745a), and an upper pooling layer (e.g., upper pooling layer 755a). Fig. 7A and Figure 7B As described above, the stack of convolutional layers can be configured to generate an output feature map based on the input feature map (or original image). Figure 7C As described above, the pooling layer can be configured to downsample the feature map, and the upper pooling layer can be configured to upsample the feature map.

[0131] like Fig.7D and Fig. 7E As shown, neural network 700 may be arranged in layers having different levels of spatial resolution. Fig.7D and Fig. 7E Five such levels are depicted, but the neural network 700 may include additional levels or fewer levels. Each level may have an encoding unit and a decoding unit (the encoding unit and decoding unit combined above all levels include an encoding stage 740 and a decoding stage 750). Each spatial resolution level in the hierarchy can be associated with a feature map of a specific spatial size, depending on the spatial size of the input image. Except for the lowest level, the output of the encoding unit of each level can be provided to the input of the encoding unit of the next level. For example, the output of the convolution layer 743a on the highest level can be provided to the input of the pooling layer 745a on the next lower level. Except for the highest level, the output of the decoding unit can be provided to the input of the decoding unit of the next highest level. For example, the output of the convolution layer 753d on the second to last level can be provided to the input of the upper pooling layer 755c on the next higher level.

[0132] In addition, one or more layers in the encoding stage 740 can be directly connected to corresponding layers in the decoding stage 750. For example, the last layer in the convolutional layer 743a can output one or more feature maps to the decoding stage 750. These feature maps can be combined (e.g., stacked) with the output feature maps of the upper pooling layer 755b to generate input to the convolutional layer 753b. Fig.7D and Fig. 7E Such direct connections between the convolutional layers 743a to 743d and the upper pooling layers 755a to 755d, respectively, are shown. Fig.7D and Fig. 7E , the feature map provided is shown as the dashed portion of the upper pooling layer 755a to the upper pooling layer 755d. The direct connection between the encoding stage 740 and the decoding stage 750 can enable the high-resolution features learned in the encoding stage 740 to be used in the decoding stage 750. This can allow or improve the ability of the decoding stage 750 to generate more accurate synthetic images. Such a direct connection can also increase the flexibility of the neural network 700. For example, in the case where the higher resolution features provided by the direct connection are sufficient to generate an accurate synthetic image, the neural network 700 can be trained to assign very small weights to the lower spatial resolution features.

[0133] The highest level of the neural network 700 may include an input 741, a convolution layer 743a, an upper pooling layer 755a, a convolution layer 753a, and an output layer 759. In some embodiments, the convolution layer 743a, the convolution layer 753a, and the output layer 759 may be a stack of one or more convolution layers. For example, the convolution layer 743a and the convolution layer 753a may each include two to five convolution layers. Alternatively, the output layer 759 may include a single convolution layer. Fig. 7A and Figure 7B As discussed, these convolutional layers can generate output feature maps by applying one or more learnable filter functions to their corresponding input feature maps. For example, one or more convolutional layers including convolutional layer 743a and convolutional layer 753a can each apply 50 to 100 (e.g., 64) learnable filter functions to generate a corresponding number of output feature maps.

[0134] The next lower level of the neural network 700 may include a pooling layer 745a, a convolution layer 743b, an upper pooling layer 755b, and a convolution layer 753b. In some embodiments, the convolution layer 743b and the convolution layer 753b may be a stack of one or more convolution layers. For example, the convolution layer 743b and the convolution layer 753b may each include two to five convolution layers. Fig. 7A and Figure 7B As discussed, these convolutional layers can generate output feature maps by applying one or more learnable filter functions to their corresponding input feature maps. For example, one or more convolutional layers including convolutional layer 743b and convolutional layer 753b can each apply 100 to 200 (e.g., 128) learnable filter functions to generate a corresponding number of output feature maps. Pooling layer 745a can correspond to the upper pooling layer 755a in the next highest level. For example, pooling layer 745a can be configured to generate a binary mask indicating the position of the maximum value in its input feature layer, and provide these instructions to the upper pooling layer 755a. The upper pooling layer 755a can be configured to use the binary mask when upsampling its input feature map.

[0135] The next lower level of the neural network 700 may include a pooling layer 745b, a convolution layer 743c, an upper pooling layer 755c, and a convolution layer 753c. In some embodiments, the convolution layer 743c and the convolution layer 753c may be a stack of one or more convolution layers. For example, the convolution layer 743c and the convolution layer 753c may each include two to six convolution layers. Fig. 7A and Figure 7BAs discussed, these convolutional layers can generate output feature maps by applying one or more learnable filter functions to their corresponding input feature maps. For example, one or more convolutional layers including convolutional layer 743c and convolutional layer 753c can each apply 150 to 300 (e.g., 256) learnable filter functions to generate a corresponding number of output feature maps. Pooling layer 745b can correspond to the upper pooling layer 755b in the next highest level. For example, pooling layer 745b can be configured to generate an indication of the position of the maximum value in its input feature map, and provide these indications to upper pooling layer 755b. Upper pooling layer 755b can be configured to use these indications when upsampling feature maps.

[0136] The next lower level of the neural network 700 may include a pooling layer 745c, a convolution layer 743d, an upper pooling layer 755d, and a convolution layer 753d. In some embodiments, the convolution layer 743d and the convolution layer 753d may be a stack of one or more convolution layers. For example, the convolution layer 743d and the convolution layer 753d may each include two to six convolution layers. Fig. 7A and Figure 7B As discussed, these convolutional layers can generate output feature maps by applying one or more learnable filter functions to their corresponding input feature maps. For example, one or more convolutional layers including convolutional layer 743d and convolutional layer 753d can each apply 300 to 600 (e.g., 512) learnable filter functions to generate a corresponding number of output feature maps. Pooling layer 745c can correspond to the upper pooling layer 755c in the next highest level. For example, pooling layer 745c can be configured to generate an indication of the position of the maximum value in its input feature layer, and provide these indications to the upper pooling layer 755c. The upper pooling layer 755c can be configured to use these indications when upsampling the feature map.

[0137] The lowest level of neural network 700 may include pooling layer 745d, convolution layer 743e, and convolution layer 753e. In some embodiments, convolution layer 743e and convolution layer 753e may be stacks of convolution layers. For example, convolution layer 743e and convolution layer 753e may each include two to five convolution layers. Fig. 7A and Figure 7BAs discussed, these convolutional layers can generate output feature maps by applying one or more learnable filter functions to their corresponding input feature maps. For example, one or more convolutional layers including convolutional layer 743e and convolutional layer 753e can each apply 800 to 1200 (e.g., 1024) learnable filter functions to generate a corresponding number of output feature maps. Pooling layer 745d can correspond to the upper pooling layer 755d in the next highest level. For example, pooling layer 745d can be configured to generate an indication of the position of the maximum value in its input feature layer, and provide these indications to upper pooling layer 755d. Upper pooling layer 755d can be configured to use these indications when upsampling feature maps.

[0138] If targeted Fig.7D and Fig. 7E As described, one or more original images (e.g., original image 761) may be provided as input 741 to neural network 700. Neural network 700 may be configured to pass input 741 through encoding stage 740 and decoding stage 750 to generate one or more target images (e.g., target image 769) as output 751. Figure 7F Depicted are exemplary feature maps generated by selected layers of neural network 700 during steps of the encoding and decoding process. Specifically, Figure 7F Depicted are feature maps generated during each step of the encoding phase 740, starting with the original image 761 and ending with feature map 763e. As shown, the relative spatial size of these exemplary feature maps decreases during the encoding phase (enlarged versions of feature map 763d and feature map 763e are provided for ease of understanding). These feature maps also depict the increased spatial extent of the features in each map and the decreased spatial resolution of each map. Fig. 7E Also depicted are feature maps generated during each step of the decoding stage 750, starting with feature map 773e and ending with target image 779. As shown, the relative spatial size of these exemplary feature maps increases during the decoding stage (enlarged versions of feature map 773e and feature map 773d are provided for ease of understanding). These feature maps also depict the reduced spatial extent of the features in each map and the increased spatial resolution of each map.

[0139] As a first encoding step, the convolutional layer 743a can be configured to receive one or more original images and generate a feature map. Figure 7CAs described, the convolution layer 743a may include a stack of one or more convolution layers. In some embodiments, the convolution layer 743a may be configured to receive one or more original images, such as the original image 761, and output a feature map, such as the feature map 763a. As described above, the disclosed systems and methods are not limited to images with a specific spatial size. For example, in some embodiments, the spatial size of one or more original images may be greater than 128x128 pixels, for example, 1024x1024 pixels. In some embodiments, one or more original images may include at least one T1-weighted MRI image, a T2-weighted MRI image, and an MRI image generated using a contrast agent. As another example, one or more original images may include different slices in a 2D image stack. In some embodiments, the convolution layer 743a may be configured to output the feature map to the next lower level of the encoding stage 740. The next lower level of the encoding stage 740 may include a pooling layer 745a and a convolution layer 743b. The convolution layer 743a may also be configured to output the feature map directly to the decoding stage 750.

[0140] As a second encoding step, the pooling layer 745a can be configured to downsample the feature map received from the convolutional layer 743a, thereby generating a downsampled feature map with a smaller spatial size. The convolutional layer 743b can be configured to receive these downsampled feature maps and generate an output feature map, for example, a feature map 763b. As shown, the feature map 763b can have a smaller spatial size than the feature map 763a. In some aspects, the spatial size of the feature map 763b can be half the spatial size of the feature map 763a in at least one dimension. In some embodiments, the convolutional layer 743b can be configured to output the feature map to the next lower level of the encoding stage 740. The next lower level of the encoding stage 740 can include the pooling layer 745b and the convolutional layer 743c. The convolutional layer 743b can also be configured to provide the output feature map directly to the decoding stage 750.

[0141] As the third encoding step, the pooling layer 745b can be configured to downsample the feature map received from the convolutional layer 743b, thereby generating a downsampled feature map with a smaller spatial size. The convolutional layer 743c can be configured to receive these downsampled feature maps and generate an output feature map, for example, a feature map 763c. As shown, the feature map 763c can have a smaller spatial size than the feature map 763b. In some aspects, the spatial size of the feature map 763c can be half the spatial size of the feature map 763b in at least one dimension. In some embodiments, the convolutional layer 743c can be configured to output the feature map to the next lower level of the encoding stage 740. The next lower level of the encoding stage 740 can include the pooling layer 745c and the convolutional layer 743d. The convolutional layer 743c can also be configured to provide the output feature map directly to the decoding stage 750.

[0142] As a fourth encoding step, the pooling layer 745c can be configured to downsample the feature map received from the convolutional layer 743c, thereby generating a downsampled feature map with a smaller spatial size. The convolutional layer 743d can be configured to receive these downsampled feature maps and generate an output feature map, for example, a feature map 763d. As shown, the feature map 763d can have a smaller spatial size than the feature map 763c. In some aspects, the spatial size of the feature map 763d can be half the spatial size of the feature map 763c in at least one dimension. In some embodiments, the convolutional layer 743d can be configured to output the feature map to the lowest level of the encoding stage 740. The lowest level of the encoding stage 740 can include the pooling layer 745d and the convolutional layer 743e. The convolutional layer 743d can also be configured to provide the output feature map directly to the decoding stage 750.

[0143] As a fifth encoding step, the pooling layer 745d can be configured to downsample the feature map received from the convolutional layer 743d, thereby generating a downsampled feature map with a smaller spatial size. The convolutional layer 743e can be configured to receive these downsampled feature maps and generate output feature maps, for example, feature map 763e. As shown, feature map 763e can have a smaller spatial size than feature map 763d. In some aspects, the spatial size of feature map 763e can be half the spatial size of feature map 763d in at least one dimension. In some embodiments, the convolutional layer 743e can be configured to provide these output feature maps to the lowest level of the encoding stage 750.

[0144] In the first decoding step, the convolution layer 753e can be configured to use the feature maps received from the convolution layer 743e to generate output feature maps, such as feature map 773e. In some embodiments, the convolution layer 753e can be configured to provide these output feature maps to a higher level of the decoding stage 750. The higher level of the decoding stage 750 can include an upper pooling layer 755d and a convolution layer 753d.

[0145] In the second decoding step, the upper pooling layer 755d can be configured to upsample the feature map received from the convolutional layer 753e, thereby generating an upsampled feature map with an increased spatial size. Figure 7C As described, the upper pooling layer 755d can use the binary mask generated by the pooling layer 745d to assign the value in the feature map received from the convolution layer 753e to the corresponding upsampled feature map. The convolution layer 753d can be configured to use the upsampled feature map received from the upper pooling layer 755d to generate an output feature map, for example, feature map 773d. In some aspects, the convolution layer 753d can also use the feature map received from the convolution layer 743d to generate an output feature map. For example, the convolution layer 753d can be configured to apply one or more learnable filter functions to the feature map stack including the feature map received from the convolution layer 743d and the upsampled feature map received from the upper pooling layer 755d. In some embodiments, the output feature map generated by the convolution layer 753d can be provided to a higher level of the decoding stage 750 by the convolution layer 753d. The higher level of the decoding stage 750 may include the upper pooling layer 755c and the convolution layer 753c.

[0146] In the third decoding step, the upper pooling layer 755c can be configured to upsample the feature map received from the convolutional layer 753d, thereby generating an upsampled feature map with an increased spatial size. Figure 7CAs described, the upper pooling layer 755c can use the binary mask generated by the pooling layer 745c to assign the value in the feature map received from the convolution layer 753d to the corresponding upsampled feature map. The convolution layer 753c can be configured to generate an output feature map, such as a feature map 773c, using the upsampled feature map received from the upper pooling layer 755c. In some aspects, the convolution layer 753c can also use the feature map received from the convolution layer 743c to generate an output feature map. For example, the convolution layer 753c can be configured to apply one or more learnable filter functions to the feature map stack including the feature map received from the convolution layer 743c and the upsampled feature map received from the upper pooling layer 755c. In some embodiments, the output feature map generated by the convolution layer 753c can be provided to a higher level of the decoding stage 750 by the convolution layer 753c. The higher level of the decoding stage 750 can include the upper pooling layer 755b and the convolution layer 753b.

[0147] In the fourth decoding step, the upper pooling layer 755b can be configured to upsample the feature map received from the convolutional layer 753c, thereby generating an upsampled feature map with an increased spatial size. Figure 7C As described, the upper pooling layer 755b can use the binary mask generated by the pooling layer 745b to assign the value in the feature map received from the convolution layer 753c to the corresponding upsampled feature map. The convolution layer 753b can be configured to generate an output feature map, for example, a feature map 773b, using the upsampled feature map received from the upper pooling layer 755b. In some aspects, the convolution layer 753b can also use the feature map received from the convolution layer 743b to generate an output feature map. For example, the convolution layer 753b can be configured to apply one or more learnable filter functions to the feature map stack including the feature map received from the convolution layer 743b and the upsampled feature map received from the upper pooling layer 755b. In some embodiments, the output feature map generated by the convolution layer 753b can be provided to the highest level of the decoding stage 750 by the convolution layer 753b. The highest level of the decoding stage 750 can include the upper pooling layer 755a, the convolution layer 753a and the output layer 759.

[0148] In the fifth decoding step, the upper pooling layer 755a may be configured to upsample the feature map received from the convolutional layer 753b, thereby generating an upsampled feature map with an increased spatial size. Figure 7CAs described, the upper pooling layer 755a can use the binary mask generated by the pooling layer 745a to assign the value in the feature map received from the convolution layer 753b to the corresponding upsampled feature map. The convolution layer 753a can be configured to generate an output feature map, such as a feature map 773a, using the upsampled feature map received from the upper pooling layer 755a. In some aspects, the convolution layer 753a can also use the feature map received from the convolution layer 743a to generate an output feature map. For example, the convolution layer 753a can be configured to apply one or more learnable filter functions to a feature map stack including a feature map received from the convolution layer 743a and an upsampled feature map received from the upper pooling layer 755a. In some embodiments, the output layer 759 can be configured to generate at least one target image, such as a target image 779, using the output feature map received from the convolution layer 753a. In some embodiments, the at least one target image can be provided to the output 751.

[0149] exist Fig.7D and Fig. 7E In the exemplary embodiment shown, the neural network 700 may include 27 convolutional layers with more than 30 million parameters for converting three-channel images. This exemplary arrangement is not intended to be restrictive. For example, the number of layers between the encoding stage 740 and the decoding stage 750 in the neural network 700 may be in the range of about 10 to several hundred (e.g., 500). As the number of layers increases, the number of model parameters also increases, thereby providing better prediction accuracy. However, a large number of layers may also lead to increased computational costs, especially during the training phase. As the computing power of the processor increases, the number of suitable layers may change. Similarly, the neural network 700 may include more or less spatial resolution levels. In some embodiments, different numbers of input channels may be used, or different image sizes may be used.

[0150] Figure 88 is a flow chart illustrating an exemplary training process 800 for training the neural network 700 consistent with the disclosed embodiments. In some embodiments, the training process 800 can be performed by the above-mentioned training unit 430 to learn the model parameters θ={W1, b1, W2, b2, ...} of the neural network 700. The training process 800 can start when the training unit 430 receives a training image group (step 810). As described above, the training image group may include one or more original images acquired using a first imaging modality and corresponding target images acquired using a second imaging modality. For example, the training image may include a single-channel MRI image or a multi-channel MRI image of an anatomical region of a patient, such as a head, torso, abdomen, and / or limbs. In some embodiments, a corresponding target image of the same anatomical region or a similar anatomical region of the patient can be acquired by CT scanning.

[0151] like Figure 8 As shown, in step 812, the training unit 430 can initialize the prediction model. For example, the training unit 430 can initialize the iteration index used to train the CNN model to, for example, zero. In some embodiments, the training unit 430 can also initialize the parameters of the neural network 700. For example, the convolutional layer weights can be initialized to random values, and / or the convolutional layer biases can be initialized to zero. In some embodiments, the training unit 430 can use the parameters from the pre-trained model to initialize the parameters of the encoding stage 730. For example, the training unit 430 can borrow weights from a model trained for other tasks such as image classification. This can enable the image conversion system 400 to utilize the transfer learning characteristics of the neural network to accelerate the convergence of acceptable solutions.

[0152] Steps 814 to 822 may be performed iteratively until one or more stopping criteria are met (e.g., the iterative process converges according to a predetermined criterion). At each iteration, a new batch of training data may be randomly selected and used to train the CNN model until a maximum number of iterations is reached. In some embodiments, each batch may include 30 to 60 sets of training images. For example, each batch may include approximately 50 pairs of original images and target images.

[0153] In step 814, the training unit 430 may be configured to select a batch of training data. In some embodiments, the training data may be randomly selected from the training data stored in the database 410. The training unit 430 may optionally be configured to pre-process the training data. For example, consistent with the above process 600, the training unit 430 may be configured to receive one or more training images 412 from the database 410 and process them to remove certain image features. In step 816, the training unit 430 may be configured to calculate a synthetic image based on the current parameters of the neural network 700 for each set of training images. In step 818, the training unit 430 may be configured to generate an error map (e.g., as described above for the synthetic image) based on the calculated synthetic image and the training data. Figure 2 For example, when the training data includes an original image and a corresponding target image, the training unit 430 may be configured to generate an error map as a difference between the synthesized image and the target image.

[0154] In step 820, the training unit 430 may be configured to determine or update the parameters of the neural network 700 based on the error map generated by step 818. For example, the parameters of the neural network 700 may be determined or updated using a back propagation algorithm known to those skilled in the art. In some embodiments, the back propagation algorithm may be configured to minimize a loss function for the parameters of the neural network 700. For example, the back propagation algorithm may update the model parameters θ=(θ1, θ2, ..., θ L ) to reduce the value of the loss function. Those skilled in the art will be familiar with various loss functions that can be implemented by the training unit 430, such as the mean absolute error (MAE) of the model prediction, which can be determined as follows:

[0155]

[0156] Among them, y i (θ) indicates the CT intensity value calculated at each voxel of the composite image, z i represents the CT intensity value (e.g., true value) at each voxel of the target image, n is the total number of image voxels, and E(θ) represents MAE. Using MAE as a loss function can improve learning so that the learning process is robust to outliers such as noise and artifacts in the training data. Alternatively, the training unit 430 can be configured to use the mean square error (MSE) of the model prediction, which can be determined as follows:

[0157]

[0158] Here, E(θ) represents MSE.

[0159] The back-propagation algorithm can be used to train the weights W of the convolutional filters, for examplek The gradient of the loss function is calculated with respect to the model parameters (θ) of the offset value b. Then, the model parameters can be iteratively updated by using a stochastic gradient descent algorithm. In some embodiments, batch normalization can be performed after each convolutional layer to reduce internal covariant shift.

[0160] Simple data augmentation can also be performed during model training to artificially increase the amount of training data. In such data augmentation, new pairs of original and target images can be created by randomly shifting pixels or flipping images in each spatial dimension for each pair of MR and CT images. By using such data augmentation, exemplary embodiments consistent with the present disclosure can increase the amount of training data and thus generate a more highly trained prediction model within the training unit 430, thereby generating more accurate synthetic images through the model.

[0161] In step 822, the training unit 430 may be configured to determine whether a stopping criterion has been met. Various stopping criteria may be used, such as a predetermined maximum number of iterations or a predetermined image quality metric (e.g., a loss function value indicating that the difference between the estimated synthesized image and the target image is sufficiently small). For example, the training unit 430 may be configured to determine whether the index is equal to or greater than a predetermined maximum number of iterations. Additionally or alternatively, the training unit 430 may be configured to determine whether the accuracy of the synthesized image meets or exceeds the expected accuracy, for example, by determining whether the loss function is less than a certain threshold criterion. If the training unit 430 determines that the stopping criterion has not been met, the training process 800 may return to step 814. If the training unit 430 determines that the stopping criterion has been met, the training process 800 may proceed to step 824. In step 824, the training unit 430 may be configured to store the updated version of the neural network 700 for later use by the image conversion unit 440.

[0162] Fig. 9A is a diagram showing the use of some embodiments of the present disclosure Figure 8Flowchart of an exemplary image conversion process 900A of a trained convolutional neural network (e.g., neural network 700) obtained by process 800 in . Image conversion process 900A may be performed by image conversion unit 440. In an embodiment, image conversion unit 440 may be configured to generate a synthetic image based on an original image acquired using the same imaging modality as the training image. For example, in the case where neural network 700 is trained based on MRI images, image conversion unit 440 may be configured to generate a synthetic image based on the MRI images. Similarly, in the case where neural network 700 is trained based on single-channel MRI images, image conversion unit 440 may be configured to use the same type of single-channel MRI images (e.g., T1-weighted images). In the case where neural network 700 is trained based on multi-channel MRI images, image conversion unit 440 may be configured to use the same type of multi-channel MRI images.

[0163] like Fig. 9A As shown, process 900A may begin when image conversion unit 440 receives one or more raw images in step 912. In some aspects, image conversion unit 440 may be configured to receive one or more raw images from image database 420, another component of image conversion system 400, or another system. In step 914, image conversion unit 440 may optionally be configured to pre-process the one or more raw images in accordance with process 600 described above to remove artifacts associated with the raw images in order to improve the accuracy of the conversion process.

[0164] In step 918, the image conversion unit 440 may be configured to receive the neural network 700. In some aspects, the neural network 700 may be received from the training unit 430, another component of the image conversion system 400, or another system. In step 920, the image conversion unit 440 may be configured to calculate one or more synthetic images using the neural network 700. Then, in step 922, the image conversion unit 440 may output the one or more synthetic images. The output may include displaying a representation of the one or more synthetic images for further analysis or observation, storing the one or more synthetic images in a non-transitory medium, or providing the one or more synthetic images to a computer process, program, and / or application. The non-transitory medium and the computer process, program, and / or application may be on a remote system.

[0165] Process 900A may be performed to convert both 2D original images and 3D original images. In the case of converting 3D images, the neural network 700 may be modified to generate a 3D composite image. As described above, the input 731 of the neural network 700 may be configured to accept a stack of adjacent 2D images selected from the 3D original image. Similarly, the output layer 749 of the neural network 700 may be configured to generate a corresponding number of adjacent composite images, which may be combined into a 3D composite image. In some embodiments, the 3D original image may be divided into a stack of 2D original images. In some embodiments, the stack of original images may overlap. For example, the first stack may include first to third image slices, the second stack may include second to fourth image slices, and the third stack may include third to fifth image slices. Then, the image conversion unit 440 may be configured to convert the stack of these original images into a stack of corresponding composite images. For example, when the image stack includes three images, the image conversion unit 440 may be configured to also generate three composite images corresponding to each of the three original images in the stack. Consistent with the stack of original images, the stack of composite images may also overlap. Thus, for a given slice in a 3D composite image, multiple 2D composite images may be generated. These multiple composite images may be fused to derive a composite image of the slice. Various rules may be used for fusion. For example, multiple images may be averaged or a median value may be used, etc.

[0166] Fig. 9B An exemplary flow chart illustrating a 3D image conversion process 900B consistent with disclosed embodiments is depicted. In some embodiments, process 900B may be performed by image conversion unit 440. In step 928, process 900B begins by configuring image conversion unit 440 to receive one or more 3D raw images. The 3D images may be received from image database 420 or from another component of image conversion system 400. The 3D images may include images of part or all of a patient's head, torso, abdomen, limbs, or any other anatomical region.

[0167] In step 930, the image conversion unit 440 may be configured to select an anatomical plane of the received 3D original image. In step 932, the image conversion unit 440 may be configured to generate one or more stacks of 2D original images based on the received 3D image and the selected plane. For example, the image conversion unit 440 may be configured to select from three anatomical planes including an axial plane, a sagittal plane, and a coronal plane. The image conversion unit 440 may be configured to generate a 2D original image from the received 3D original image along the selected anatomical plane. In some aspects, the image conversion unit 440 may be configured to generate a series of separate original images or a series of stacks of original images.

[0168] In step 934, the image conversion unit 440 may be configured to receive a neural network (e.g., neural network 700). The received neural network may correspond to the selected anatomical plane. For example, the received neural network may have been trained using images (or stacks of images) along the selected anatomical plane. Such images (or stacks of images) along the selected anatomical plane may collectively include dependent structural information for an axis orthogonal to the selected anatomical plane. Since the received neural network has been trained using images along the selected anatomical plane, the neural network may be able to use the dependent structural information to improve conversion efficiency. For example, the parameters or weights of the received neural network may reflect the dependent structural information. In response to a change in the selected anatomical plane (e.g., from sagittal to coronal), the image conversion unit 440 may be configured to receive an additional neural network corresponding to the newly selected anatomical plane. For example, the image conversion unit 440 may receive an axial neural network, a coronal neural network, or a sagittal neural network depending on the selected plane. Just as a previously received neural network was able to use dependent structural information for an axis orthogonal to a previously selected anatomical plane to improve conversion efficiency, a newly received neural network may be able to use dependent structural information for an axis orthogonal to a newly selected anatomical plane to improve conversion efficiency. In some embodiments, image conversion unit 440 may be configured to receive a neural network trained using images (or stacks of images) along multiple planes (e.g., multiple anatomical planes, such that the stack may include a coronal image, a sagittal image; or the stack may include an axial image and a sagittal image, etc.). Thus, image conversion unit 440 may be configured to use the neural network to generate a composite image from the original image along any selected anatomical plane or multiple anatomical planes.

[0169] In step 936, the image conversion unit 440 can be configured to use the received neural network to generate a corresponding composite image or a stack of composite images. As described above, neural networks trained using images (or stacks of images) along a specific anatomical plane can be used to improve conversion efficiency because these networks can use dependent structural information specific to the specific anatomical plane. The image conversion unit 440 can repeat step 936 until all selected original images or stacks of selected original images have been converted to composite images.

[0170] In some embodiments, the image conversion unit 440 may perform step 930 after step 934. For example, the trained neural network received by the image conversion unit 440 may be trained using 2D raw images along multiple planes (e.g., all three planes). In this case, the image conversion unit 440 may be configured to iteratively select each of the different planes and generate a 2D raw image along the selected plane. These 2D raw images along the three planes may be independently input to the neural network 700 to generate composite images along each plane. For example, an axial 2D raw image may be input to the neural network 700 that outputs an axial composite image. A sagittal 2D raw image may be input to the neural network 700 that obtains a sagittal composite image. Finally, a coronal 2D raw image may be input to the neural network 700 that obtains a coronal composite image. Alternatively, the 2D raw images of all three planes may be combined into a single sequence (e.g., axial 2D raw images, followed by sagittal 2D raw images, and then followed by coronal 2D raw images) and that sequence input to the same neural network 700 trained using images stacked from these three planes.

[0171] In step 938, the image conversion unit 440 may be configured to aggregate the resulting 2D composite images into a 3D composite image. Such aggregation may include stacking the composite images along an axis orthogonal to the selected plane to obtain the 3D composite image.

[0172] In step 940, the image conversion unit 440 may determine whether all planes of the 3D original image have been processed. For example, the image conversion unit 440 may continue to process each plane until a predefined 3D image volume has been processed. Alternatively, the image conversion unit 440 may continue processing until a predetermined number of 2D planar images have been processed. If all planes have been processed based on the criteria used in step 940, the process 900B continues to step 942. If there are planes that have not been processed, the image conversion unit 440 may be configured to return to step 930 and select additional image planes for processing.

[0173] Because the 3D composite image of each plane provides a pixel value for any given pixel in the 3D image, each pixel may have three different pixel values. Therefore, the 3D composite images can be combined to determine the final 3D composite image. In step 942, the image conversion unit 440 can be configured to determine the final 3D composite image. The image conversion unit 440 can be configured to determine the final 3D composite image by combining (e.g., "fusion" as known in the art) the 3D composite images corresponding to each selected plane (e.g., the axial plane, the sagittal plane, and the coronal plane). In some embodiments, the value of the voxel of the fused 3D composite image can be an average value. For example, if the CT values ​​of the three fused 3D composite images are 70, 80, and 30, respectively, the voxel can have an average CT value of 60. As another example, an intermediate voxel value can be used, such as 70 in the above example. Other methods of combining voxel values ​​can be used, and the above example is not intended to be limiting. In this way, dependent structural information specific to each anatomical plane can be incorporated into the final 3D composite image.

[0174] In addition, the present disclosure may also be configured as follows:

[0175] Solution 1. A computer-implemented method for generating a composite image of an anatomical portion based on an original image of the anatomical portion acquired by an imaging device using a first imaging modality, the method comprising:

[0176] receiving the original image of the anatomical portion acquired by the imaging device using the first imaging modality;

[0177] receiving a convolutional neural network model trained to predict the synthesized image based on the original image; and

[0178] The original image is converted into the composite image by at least one processor through the convolutional neural network model, and the composite image is similar to a second imaging modality depiction of the anatomical part, wherein the second imaging modality is different from the first imaging modality.

[0179] Option 2. A method according to Option 1, wherein the first imaging modality is magnetic resonance imaging and the second imaging modality is computed tomography.

[0180] Option 3. A method according to Option 1, wherein the first imaging modality and the second imaging modality are selected from the following group: magnetic resonance imaging, computed tomography, ultrasound imaging, positron emission tomography, and single photon emission computed tomography.

[0181] Solution 4. The method according to Solution 1 further includes:

[0182] receiving a plurality of original training images acquired using the first imaging modality;

[0183] receiving a plurality of training target images acquired using the second imaging modality, each training target image corresponding to a training original image;

[0184] Determine the convolutional neural network architecture; and

[0185] The convolutional neural network model is trained using the training original images and the corresponding training target images.

[0186] Option 5. The method according to Option 1, wherein the original image is a two-dimensional image and the synthesized image is a two-dimensional image.

[0187] Option 6. A method according to Option 1, wherein the original image comprises a stack of two-dimensional images, and the composite image comprises a stack of corresponding two-dimensional images.

[0188] Option 7. A method according to Option 1, wherein the original image is a three-dimensional volume and the synthesized image is a three-dimensional volume.

[0189] Solution 8. The method according to Solution 1, wherein the original image is a three-dimensional image, and wherein generating the composite image further comprises:

[0190] selecting a plurality of stacks of adjacent two-dimensional images from the three-dimensional image;

[0191] converting each stack of adjacent two-dimensional images into a stack of synthetic two-dimensional images using the convolutional neural network model; and

[0192] The composite image is determined by the at least one processor by aggregating a stack of the composite two-dimensional images.

[0193] Solution 9. The method according to Solution 1, wherein the original image is a three-dimensional image, and wherein generating the composite image further comprises:

[0194] creating a first stack of two-dimensional images from a first plane of the three-dimensional images and creating a second stack of two-dimensional images from a second plane of the three-dimensional images;

[0195] converting the first stack of two-dimensional images and the second stack of two-dimensional images into a first stack of synthetic two-dimensional images and a second stack of synthetic two-dimensional images using the convolutional neural network model; and

[0196] The composite image is determined by aggregating the first stack of composite two-dimensional images and the second stack of composite two-dimensional images.

[0197] Option 10. A method according to Option 1, wherein the original image includes a multi-channel image of the anatomical part acquired using different acquisition channels or different acquisition settings of the imaging device.

[0198] Solution 11. The method according to Solution 4, wherein training the convolutional neural network model further comprises:

[0199] Using the convolutional neural network model to convert the training original image into a synthetic image;

[0200] determining a difference between the composite image and the corresponding training target image; and

[0201] A set of model parameters of the convolutional neural network model is updated based on the difference.

[0202] Option 12. The method according to Option 11, wherein the training is completed when the difference between the synthetic image and the target image is less than a predetermined threshold.

[0203] Option 13. A method according to Option 11, wherein the difference is measured by a loss function calculated based on the synthetic image and the target image.

[0204] Embodiment 14. A system for generating a composite image of an anatomical portion based on an original image of the anatomical portion acquired by an imaging device using a first imaging modality, the system comprising:

[0205] An input interface, the input interface being configured to:

[0206] receiving the raw image of the anatomical portion acquired by the imaging device using the first imaging modality; and

[0207] Receiving a convolutional neural network model trained to predict the synthesized image based on the original image;

[0208] at least one storage device configured to store the original image and the convolutional neural network model; and

[0209] An image processor configured to convert the original image into the composite image through the convolutional neural network model, wherein the composite image resembles a second imaging modality depiction of the anatomical portion, wherein the second imaging modality is different from the first imaging modality.

[0210] Option 15. A system according to Option 14, wherein the original image includes a multi-channel image of the anatomical part acquired using different acquisition channels or different acquisition settings of the imaging device.

[0211] Option 16. A system according to Option 14, wherein the first imaging modality is magnetic resonance imaging and the second imaging modality is computed tomography.

[0212] Option 17. A system according to Option 14, wherein the first imaging modality and the second imaging modality are selected from the following group: magnetic resonance imaging, computed tomography, ultrasound imaging, positron emission tomography, and single photon emission computed tomography.

[0213] Embodiment 18. The system according to embodiment 14, wherein the input interface is further configured to:

[0214] receiving a plurality of original training images acquired using the first imaging modality;

[0215] receiving a plurality of training target images acquired using the second imaging modality, each training target image corresponding to a training original image; and

[0216] Wherein, the image processor is further configured to:

[0217] The convolutional neural network model is trained using the training original images and the corresponding training target images.

[0218] Option 19. A system according to Option 14, wherein the original image is a two-dimensional image and the synthesized image is a two-dimensional image.

[0219] Option 20. A system according to Option 14, wherein the original image comprises a stack of two-dimensional images, and the composite image comprises a stack of corresponding two-dimensional images.

[0220] Option 21. A system according to Option 14, wherein the original image is a three-dimensional volume and the synthesized image is a three-dimensional volume.

[0221] Solution 22. The system according to Solution 14, wherein the original image is a 3D image, and wherein the image processor is further configured to:

[0222] selecting a plurality of stacks of adjacent two-dimensional images from the 3D image;

[0223] converting each stack of adjacent two-dimensional images into a stack of synthetic two-dimensional images using the convolutional neural network model; and

[0224] The composite image is determined by aggregating a stack of the composite two-dimensional images.

[0225] Embodiment 23. A non-transitory computer-readable medium comprising instructions, which when executed by at least one processor causes the at least one processor to perform a method for generating a composite image of an anatomical portion based on an original image of the anatomical portion acquired by an imaging device using a first imaging modality, the method comprising:

[0226] receiving the original image of the anatomical portion acquired by the imaging device using the first imaging modality;

[0227] receiving a convolutional neural network model trained to predict the synthesized image based on the original image; and

[0228] The original image is converted into the composite image by the convolutional neural network model, and the composite image is similar to a second imaging modality depiction of the anatomical part, wherein the second imaging modality is different from the first imaging modality.

[0229] Embodiment 24. The non-transitory computer readable medium according to embodiment 23, further comprising:

[0230] receiving a plurality of original training images acquired using the first imaging modality;

[0231] receiving a plurality of training target images acquired using the second imaging modality, each training target image corresponding to a training original image;

[0232] Determine the convolutional neural network architecture; and

[0233] The convolutional neural network model is trained using the training original images and the corresponding training target images.

[0234] Embodiment 25. The non-transitory computer readable medium of embodiment 23, wherein the original image is a three-dimensional image, and wherein generating the composite image further comprises:

[0235] selecting a plurality of stacks of adjacent two-dimensional images from the three-dimensional image;

[0236] converting each stack of adjacent two-dimensional images into a stack of synthetic two-dimensional images using the convolutional neural network model; and

[0237] The composite image is determined by the at least one processor by aggregating a stack of the composite two-dimensional images.

[0238] Various operations or functions are described herein that may be implemented or defined as software code or instructions. Such content may be directly executable ("object" or "executable" form), source code, or differential code ("incremental" or "patch" code). Software implementations of the embodiments described herein may be provided via an article having code or instructions stored thereon, or via a method of operating a communication interface to send data through the communication interface.

[0239] A machine-readable storage medium or a computer-readable storage medium may include one or more non-transitory media (e.g., a centralized database or a distributed database, and / or associated cache memory and servers). Such a machine-readable storage medium or a computer-readable storage medium may store computer-executable instructions or data that can enable a machine to perform the described functions or operations. Such a machine-readable storage medium or a computer-readable storage medium may include any mechanism for storing information in a form accessible to a machine (e.g., a computing device, an electronic system, etc.), such as a recordable / non-recordable medium (e.g., a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium, an optical storage medium, a flash memory device, etc.). For example, the term "machine-readable storage medium" or "computer-readable storage medium" should therefore be deemed to include, but is not limited to, solid-state memory, optical media, and magnetic media.

[0240] The communication interface includes any mechanism that interfaces with any medium, whether hardwired, wireless, optical, etc., to communicate with another device, such as a memory bus interface, a processor bus interface, an Internet connection, a disk controller, etc. The communication interface can be configured by providing configuration parameters and / or sending signals to prepare the communication interface to provide data signals describing the software content. The communication interface can be accessed via one or more commands or signals sent to the communication interface.

Claims

1. A computer-implemented method for generating synthetic imaging data of an anatomical portion of a human subject, the method comprising: receiving raw imaging data of an anatomical portion of the human subject, the raw imaging data being acquired by an imaging device using a first type of imaging modality, and the raw imaging data comprising a plurality of images providing a first imaging modality depiction of the anatomical portion; receiving a convolutional neural network model trained to predict the synthesized imaging data based on the raw imaging data; as well as The method of claim 1 , wherein the raw imaging data is converted into the synthetic imaging data by using the convolutional neural network model, the synthetic imaging data comprising a plurality of images similar to a second imaging modality depiction of the anatomical portion from a second type of imaging modality, wherein the second type of imaging modality is different from the first type of imaging modality, and wherein converting the raw imaging data into the synthetic imaging data comprises: generating one or more composite images from the plurality of images of the raw imaging data using the convolutional neural network model; aggregating multiple groups of one or more generated composite images; and The composite imaging data is determined from aggregated sets of one or more generated composite images.

2. The method according to claim 1, wherein: The first type of imaging modality and the second type of imaging modality are provided by different modalities provided from two of the following: magnetic resonance imaging, computed tomography, ultrasound imaging, positron emission tomography or single photon emission computed tomography.

3. The method according to claim 1, wherein: The composite imaging data includes the plurality of images provided as a stack of two-dimensional images.

4. The method according to claim 1, wherein: The raw imaging data is provided in a three-dimensional volume, and the synthesized imaging data is provided in a three-dimensional volume.

5. The method according to claim 1, in, Receiving the raw imaging data includes selecting a plurality of stacks of adjacent two-dimensional images from the raw imaging data; and Wherein, generating the one or more synthetic images includes using the convolutional neural network model to generate a stack of synthetic two-dimensional images from each stack of adjacent two-dimensional images.

6. The method according to claim 1, in, Receiving the raw imaging data includes creating a first stack of two-dimensional images from a first plane of the three-dimensional imaging data and creating a second stack of two-dimensional images from a second plane of the three-dimensional imaging data; and Wherein, generating the one or more synthetic images includes using the convolutional neural network model to generate first and second stacks of synthetic two-dimensional images from first and second stacks of two-dimensional images.

7. The method according to claim 1, further comprising: receiving a plurality of training original images acquired using the first type of imaging modality; receiving a plurality of training target images acquired using the second type of imaging modality, each training target image corresponding to a training original image; Determine the convolutional neural network architecture; as well as Based on the determined convolutional neural network architecture, the convolutional neural network model is trained using the training original images and the corresponding training target images.

8. The method according to claim 7, wherein: Training the convolutional neural network model includes: Using the convolutional neural network model to convert the training original image into a synthetic image; determining a difference between the composite image and the corresponding training target image; and Model parameters of the convolutional neural network model are updated based on the difference.

9. A system for generating synthetic imaging data of an anatomical portion of a human subject, the system comprising: An input interface, the input interface being configured to: receiving raw imaging data of an anatomical portion of the human subject, the raw imaging data being acquired by an imaging device using a first type of imaging modality and comprising a plurality of images providing a first imaging modality depiction of the anatomical portion, and receiving a convolutional neural network model trained to predict the synthesized imaging data based on the raw imaging data; at least one storage device configured to store the raw imaging data and the convolutional neural network model; as well as an image processor configured to convert the raw imaging data to the synthetic imaging data by using the convolutional neural network model, the synthetic imaging data comprising a plurality of images similar to a second imaging modality depiction of the anatomical portion from a second type of imaging modality, wherein the second type of imaging modality is different from the first imaging modality, and wherein the image processor is configured to convert the raw imaging data to the synthetic imaging data by: generating one or more composite images from the plurality of images of the raw imaging data using the convolutional neural network model; aggregating multiple groups of one or more generated composite images; and The composite imaging data is determined from aggregated sets of one or more generated composite images.

10. The system according to claim 9, wherein: The first type of imaging modality and the second type of imaging modality are provided by different modalities provided from two of the following: magnetic resonance imaging, computed tomography, ultrasound imaging, positron emission tomography or single photon emission computed tomography.

11. The system according to claim 9, wherein: The composite imaging data includes the plurality of images provided as a stack of two-dimensional images.

12. The system according to claim 9, wherein: The raw imaging data is provided in a three-dimensional volume, and the synthesized imaging data is provided in a three-dimensional volume.

13. The system according to claim 9, in, The operation of receiving the raw imaging data includes the operation of selecting a plurality of stacks of adjacent two-dimensional images from the raw imaging data; and Among them, the operation of generating the one or more synthetic images includes the operation of using the convolutional neural network model to generate a stack of synthetic two-dimensional images from each stack of adjacent two-dimensional images.

14. The system according to claim 9, in, The operation of receiving the raw imaging data includes the operation of creating a first stack of two-dimensional images from a first plane of the three-dimensional imaging data and creating a second stack of two-dimensional images from a second plane of the three-dimensional imaging data; and Among them, the operation of generating the one or more synthetic images includes the operation of using the convolutional neural network model to generate a first stack and a second stack of synthetic two-dimensional images from a first stack and a second stack of two-dimensional images.

15. The system according to claim 10, wherein: The image processor is further configured to: receiving a plurality of training original images acquired using the first type of imaging modality; receiving a plurality of training target images acquired using the second type of imaging modality, each training target image corresponding to a training original image; Determine the convolutional neural network architecture; as well as Based on the determined convolutional neural network architecture, the convolutional neural network model is trained using the training original images and the corresponding training target images.

16. The system of claim 15, wherein: The operation of training the convolutional neural network model includes the following operations: Using the convolutional neural network model to convert the training original image into a synthetic image; determining a difference between the composite image and the corresponding training target image; and Model parameters of the convolutional neural network model are updated based on the difference.

17. A computer readable medium comprising instructions which, when executed by at least one processor, cause the at least one processor to perform a method for generating synthetic imaging data of an anatomical portion of a human subject, the method comprising: obtaining raw imaging data of an anatomical portion of the human subject, the raw imaging data being acquired by an imaging device using a first type of imaging modality and comprising a plurality of images providing a first imaging modality depiction of the anatomical portion; identifying a convolutional neural network model trained to predict the synthetic imaging data based on the raw imaging data; as well as The method of claim 1 , wherein the raw imaging data is converted into the synthetic imaging data by using the identified convolutional neural network model, the synthetic imaging data comprising a plurality of images similar to a second imaging modality depiction of the anatomical portion from a second type of imaging modality, wherein the second type of imaging modality is different from the first type of imaging modality, and wherein converting the raw imaging data into the synthetic imaging data comprises: generating one or more composite images from the plurality of images of the raw imaging data using the identified convolutional neural network model; aggregating multiple groups of one or more generated composite images; and The composite imaging data is determined from aggregated sets of one or more generated composite images.

18. The computer-readable medium of claim 17, wherein: The first type of imaging modality and the second type of imaging modality are provided by different modalities provided from two of the following: magnetic resonance imaging, computed tomography, ultrasound imaging, positron emission tomography or single photon emission computed tomography.

19. The computer-readable medium of claim 17, wherein: The composite imaging data includes the plurality of images provided as a stack of two-dimensional images.

20. The computer-readable medium of claim 17, wherein: The raw imaging data is provided in a three-dimensional volume, and the synthesized imaging data is provided in a three-dimensional volume.

21. The computer readable medium of claim 17, in, Obtaining the raw imaging data includes selecting a plurality of stacks of adjacent two-dimensional images from the raw imaging data; and Wherein, generating the one or more synthetic images includes using the identified convolutional neural network model to generate a stack of synthetic two-dimensional images from each stack of adjacent two-dimensional images.

22. The computer readable medium of claim 17, in, Obtaining the raw imaging data includes creating a first stack of two-dimensional images from a first plane of three-dimensional imaging data and creating a second stack of two-dimensional images from a second plane of the three-dimensional imaging data; and Wherein, generating the one or more synthetic images includes using the identified convolutional neural network model to generate first and second stacks of synthetic two-dimensional images from first and second stacks of two-dimensional images.

23. The computer-readable medium of claim 17, wherein: The method executed by the instructions also includes: Obtaining a plurality of original training images acquired using the first type of imaging modality; Obtaining a plurality of training target images acquired using the second type of imaging modality, each training target image corresponding to a training original image; Determine the convolutional neural network architecture; and Based on the determined convolutional neural network architecture, the convolutional neural network model is trained using the training original images and the corresponding training target images.

24. The computer readable medium of claim 23, wherein: Training the convolutional neural network model also includes: Using the convolutional neural network model to convert the training original image into a synthetic image; determining a difference between the composite image and the corresponding training target image; and Model parameters of the convolutional neural network model are updated based on the difference.

Images