Image Synthesis Using Adversarial Networks

Abstract

A statistical learning technique that does not rely on paired imaging information is described herein. The technique can be computer-implemented and can be used to train a statistical learning model to perform image synthesis, for example, to support radiotherapy treatment planning. In an example, the trained statistical learning model can include a convolutional neural network established as a generator convolutional network, and the generator can be trained at least in part using a separate convolutional neural network established as a discriminator convolutional network. The generator convolutional network and the discriminator convolutional network can form an adversarial network architecture for use during training. After training, the generator convolutional network can be provided for image synthesis, for example, for receiving imaging data corresponding to a first imaging modality type and synthesizing imaging data corresponding to a different second imaging modality type.

Description

[0001] Priority claim

[0002] This application claims the benefit of priority to U.S. Application Serial No. 15 / 952,686, filed on April 13, 2018, entitled “IMAGE SYNTHESIS USING ADVERSARIAL NETWORKS SUCH AS FOR RADIATION THERAPY,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present document relates generally, but not by way of limitation, to the synthesis of medical images and, more particularly, to devices and techniques for automatically converting imaging data from a first imaging modality type (e.g., magnetic resonance imaging (MR) imaging) to a different second imaging modality type (e.g., computed tomography (CT) imaging) using, at least in part, a statistical learning model to, for example, support radiation therapy delivery or radiation therapy treatment planning. Background Art

[0004] In radiotherapy or radiosurgery, medical imaging of the patient is often used to make treatment plans. Analysis of such imaging often involves depicting target volumes and critical organs in medical images. For example, it is generally believed that segmentation or contouring of tumors and organs at risk (OARs) based on patient images is a prerequisite for radiotherapy planning. Segmentation or other image processing can be performed on imaging data corresponding to a specific imaging modality type (e.g., computed tomography (CT) imaging). Semi-automatic or automatic segmentation techniques can be used depending on whether the input imaging data is CT imaging data. In some use cases, it may be desirable to use magnetic resonance (MR) imaging primarily or exclusively as input imaging data for processing. For example, such MR imaging data may not be suitable for use with existing radiotherapy planning tools that rely on CT imaging, or may not be suitable for use with tools that rely on specific MR weighting methods or MR imaging sequences that are different from available input imaging data. Image synthesis techniques can be used to convert images from a first modality type to a different second modality type, for example to facilitate compatibility with radiotherapy treatment plans or therapy delivery tools.

[0005] In one approach, image transformation can be performed using machine learning techniques trained with paired images. The phrase "paired images" can generally refer to the use of different acquired images of the same patient to generate training data, where the different acquired images are obtained using different imaging modalities (e.g., CT and MR, cone beam imaging and conventional CT, T1-weighted MR and T2-weighted MR, as illustrative examples). This approach that relies on paired imaging data may present various challenges. For example, in order for the training technique to be effective, it may be necessary to perfectly or nearly perfectly align the paired images to train a model that can faithfully synthesize output images. Such alignment may be extremely difficult or even impossible to achieve because images are typically acquired from different imaging devices or at different time points when the imaging object has moved (e.g., from one scanner to another). Such motion may produce distortions or anatomical changes between imaging acquisitions. Such challenges can be partially addressed using registration techniques, but such registration techniques also present their own challenges. Due to imperfect training data, registration errors will lead to degraded model performance. Summary of the invention

[0006] As described herein, image synthesis generally involves receiving input imaging data and synthesizing output imaging data using a trained statistical learning model, wherein the output imaging data can then be used in downstream activities such as radiation therapy treatment planning or radiation therapy delivery. The inventors have recognized that, among other things, there are problems in applying statistical learning techniques (e.g., machine learning involving deep convolutional neural network (DCNN) techniques) to perform image synthesis (e.g., converting images from a first imaging modality type to a different second imaging modality type) when trained using unpaired images. Among other things, the inventors have developed statistical learning techniques that do not rely on paired imaging information to train a statistical learning model for performing image synthesis. In an example, the trained statistical learning model can include a convolutional neural network established as a generator convolutional network, and the generator can be trained at least in part using a separate convolutional neural network established as a discriminator convolutional network. The generator convolutional network and the discriminator convolutional network can form an adversarial network architecture, wherein the discriminator convolutional network is adjusted during training so that the value of the target is increased or maximized over the input distribution to the discriminator convolutional network, and the generator convolutional network is adjusted so that the value of the target is decreased or minimized over the input distribution to the generator convolutional network.

[0007] Compared to other methods, the inventors have also recognized that, among other things, the methods described herein do not need to include simultaneously training two separate generator convolutional networks and two separate discriminator convolutional networks in order to be used with unpaired imaging data. In this way, the methods described herein can provide substantial savings in processing time and memory consumption (e.g., providing approximately two times savings compared to other methods involving two generator networks and two discriminator networks).

[0008] In an example, a computer-implemented method may be used to synthesize a medical image using a trained statistical learning model, the method comprising: receiving medical imaging data obtained using a first imaging modality type; applying the trained statistical learning model to the received medical imaging data to synthesize a medical image corresponding to a different second imaging modality type; and providing the synthesized medical image for presentation or for further processing. The trained statistical learning model may be established at least in part using a similarity determination between training imaging data provided at a model input and synthesized imaging data at a model output, wherein the training imaging data corresponds to the first imaging modality type and the synthesized imaging data corresponds to the second imaging modality type, and the trained statistical learning model may be established at least in part using a separate statistical learning model established to discriminate between actual imaging data corresponding to the second imaging modality and synthesized imaging data.

[0009] In an example, a computer-implemented method may include establishing a trained statistical learning model for synthesizing medical images without requiring paired imaging for training, wherein the method includes: receiving training medical imaging data corresponding to a first imaging modality type; applying a statistical learning model to the received medical imaging data to synthesize imaging data corresponding to a different second imaging modality type; adjusting the statistical learning model at least in part using a similarity determination between the training imaging data and the synthesized imaging data at the model output; and adjusting the statistical learning model at least in part using a separate statistical learning model, the separate statistical learning model being established to discriminate between the synthesized imaging data and actual imaging data corresponding to the second imaging modality type.

[0010] In an example, a system for performing various computer-implemented techniques may include: a processing circuit including at least one processor; and a storage medium including instructions that, when executed by the at least one processor, cause the processor to: receive medical imaging data obtained using a first imaging modality type; apply a trained statistical learning model to the received medical imaging data to synthesize a medical image corresponding to a different second imaging modality type; and provide the synthesized medical image for presentation or further processing. The trained statistical learning model may be established at least in part using a similarity determination between training imaging data provided at a model input and synthesized imaging data at a model output, the training imaging data corresponding to the first imaging modality type, and the synthesized imaging data corresponding to the second imaging modality type, and the trained statistical learning model may be established at least in part using a separate statistical learning model established to discriminate between the synthesized imaging data and actual imaging data corresponding to the second imaging modality type.

[0011] This summary of the invention is intended to provide an overview of the subject matter of this patent application. It is not intended to provide an exclusive or exhaustive description of the invention. The specific description is included to provide additional information about this patent application. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In the accompanying drawings, which are not necessarily drawn to scale, the same reference numerals may describe similar components in different views. The same reference numerals with different letter suffixes may represent different instances of similar components. The accompanying drawings generally illustrate various embodiments discussed in this document by way of example and not by way of limitation.

[0013] Figure 1 An example is shown including a radiation therapy system for providing radiation therapy to a patient, for example.

[0014] Figure 2 An example is shown that includes a radiation therapy apparatus that may include a radiation source (eg, an x-ray source or a linear accelerator), an examination table, an imaging detector, and a radiation therapy output.

[0015] Figure 3 An example is shown including an architecture for performing training of at least one statistical learning model (e.g., a convolutional neural network (CNN)) and for providing a synthesized image using an image synthesis technique based on the CNN model.

[0016] Figure 4An example is shown that includes an adversarial training architecture for performing training on at least one statistical learning model (e.g., a convolutional neural network (CNN)), for example, for providing a synthetic image using an image synthesis technique based on the CNN model.

[0017] Figure 5 An example is shown that includes a statistical learning model (e.g., a convolutional neural network (CNN)), which is used, for example, to perform classification of medical imaging data (e.g., for discriminating between synthesized imaging data and actual imaging data), which can, for example, be used to train another statistical learning model to perform image synthesis.

[0018] Figure 6 An example is shown that includes a statistical learning model (e.g., a convolutional neural network (CNN)) that is used, for example, to receive medical imaging data corresponding to a first imaging modality type and to perform synthesis of medical imaging data corresponding to a different second imaging modality type using the received medical imaging data.

[0019] Figure 7 Techniques (eg, methods) that can be computer-implemented, for example, for performing medical imaging data synthesis using a statistical learning model, are generally described.

[0020] Figure 8 Techniques (e.g., methods) that can be computer-implemented, for example, for implementing an adversarial training architecture to build a statistical learning model for performing medical imaging data synthesis are generally described.

[0021] Fig.9A , Fig. 9B and Fig. 9C An illustrative example of a view including imaging data corresponding to a first imaging modality type is shown.

[0022] Fig. 10A , Fig. 10B and Fig. 10C illustrative examples of views including synthetic imaging data corresponding to a second imaging modality type are shown, the synthetic imaging data being generated using images respectively applied to Fig.9A , Fig. 9B and Fig. 9C A statistical learning model of the imaging data is provided.

[0023] Fig.11A , Fig. 11B and Fig. 11C illustrative examples of views including actual imaging data acquired using a second imaging modality type, which are presented herein for use with Fig. 10A , Fig. 10B and Fig. 10C The synthetic imaging data were compared. DETAILED DESCRIPTION

[0024] As mentioned above, image conversion from one imaging modality type to another imaging modality type can be performed in an automated manner (e.g., a computer-implemented manner), such as using a trained statistical learning model to synthesize an output image. Such cross-modality image synthesis has many applications in radiation therapy. For example, it may be desirable to generate a CT-like image (e.g., referred to as a "pseudo-CT" image or a "synthetic" CT image) based on an MR image to facilitate the use of the acquired MR imaging data to formulate a treatment plan. In another example, a higher quality CT-like image (e.g., having less noise, fewer artifacts, and having a higher resolution) can be synthesized from a relatively low quality CT or cone beam CT (CBCT) image as input. In this latter example, the synthesis provides a method for improving the image quality of a relatively low quality image (the lower quality input imaging data may be due to imaging constraints, such as reducing the imaging dose or using a shorter imaging duration in order to increase imaging speed or reduce latency).

[0025] As another example, MR imaging can involve the use of many different sequences (e.g., T1-weighted sequences, T2-weighted sequences, or FLAIR sequences, as illustrative examples), which can result in highly variable image appearances. When such images are provided as input to an off-the-shelf imaging registration or imaging segmentation routine, the use of different MR imaging sequences can present challenges. The inventors have recognized that output images can be synthesized to provide imaging data that is compatible with the registration or segmentation routine. For example, T2-weighted imaging data representing a first imaging modality type can be provided, and from the T2-weighted imaging data, T1-weighted imaging data can be synthesized using a trained statistical learning model. One or more of registration or segmentation can then be performed on the synthesized imaging data.

[0026] Deep learning techniques including deep convolutional neural networks (DCNNs) can be used to challenge medical imaging analysis problems, such as tumor detection, disease classification, or structural segmentation, as illustrative examples only. A property that contributes to the success of DCNNs is that such DCNNs are able to discern or "learn" complex models directly from raw input data without relying on hand-crafted features. In general, the techniques described herein can be used to provide image synthesis facilitated by DCNNs, such as receiving input imaging data obtained using a first image modality type, and providing a synthesized output including output imaging data corresponding to a different second imaging modality.

[0027] Figure 1An example is shown that includes a radiation therapy system 100 for providing radiation therapy to a patient, for example. The radiation therapy system 100 includes an image processing device 112. The image processing device 112 may be connected to a network 120. The network 120 may be connected to the Internet 122. The network 120 may connect the image processing device 112 with one or more of: a database 124, a hospital database 126, an oncology information system (OIS) 128, a radiation therapy device 130, an image acquisition device 132, a display device 134, and a user interface 136. The image processing device 112 may be configured to generate a radiation therapy treatment plan 142 to be used by the radiation therapy device 130.

[0028] The image processing device 112 may include a memory device 116, a processor 114, and a communication interface 118. The memory device 116 may store computer executable instructions such as an operating system 143, a radiation therapy treatment plan 142 (e.g., an original treatment plan, an adjusted treatment plan, etc.), a software program 144 (e.g., artificial intelligence, deep learning, a neural network, radiation therapy treatment planning software), and any other computer executable instructions to be executed by the processor 114 (e.g., any computer executable instructions corresponding to or using one or more techniques as shown and described elsewhere herein). In one embodiment, the software program 144 may convert a medical image in one format (e.g., MRI) to another format (e.g., CT) by generating a synthetic image (e.g., a pseudo-CT image). For example, the software program 144 may include an image processing program for training a statistical learning model to convert a medical image 146 corresponding to one modality (e.g., an MRI image) to a synthetic image corresponding to a different modality (e.g., a CT image); alternatively, the statistical learning model may convert a CT image into an MRI image.

[0029] Illustrative examples for performing such image synthesis using statistical learning models are described elsewhere herein (e.g., below). In another embodiment, the software program 144 may align a patient image (e.g., a CT image or an MR image) with a dose distribution (also represented as an image) for the patient so that corresponding image voxels are appropriately associated with dose voxels through the network. In yet another embodiment, the software program 144 may replace a function of the patient image that emphasizes some aspect of image information, such as a signed distance function or a processed version of the image. Such a function may emphasize edges or differences in voxel texture or any other structural aspect useful for neural network learning. In another embodiment, the software program 144 may replace a function of the dose distribution that emphasizes some aspect of dose information. Such a function may emphasize tilt gradients around a target or any other structural aspect useful for neural network learning. The memory device 116 may store data, including medical images 146, patient data 145, and other data required to create and implement the radiation therapy treatment plan 142.

[0030] In addition to the memory 116 storing the software program 144, it is also contemplated that the software program 144 can be stored on a removable computer medium such as a hard drive, a computer disk, a CD-ROM, a DVD, an HD, a Blu-ray DVD, a USB flash drive, an SD card, a memory stick, or any other suitable medium; and that the software program 144 can be executed by the image processor 114 when downloaded to the image processing device 112.

[0031] The processor 114 may be communicatively coupled to the memory device 116, and the processor 114 may be configured to execute computer executable instructions stored thereon. The processor 114 may send the medical image 146 to the memory 116 or receive the medical image 146 from the memory 116. For example, the processor 114 may receive the medical image 146 from the image acquisition device 132 via the communication interface 118 and the network 120 to store it in the memory 116. The processor 114 may also send the medical image 146 stored in the memory 116 to the network 120 via the communication interface 118 to store the medical image 146 in the database 124 or the hospital database 126.

[0032] In addition, the processor 114 can create a radiation therapy treatment plan 142 using a software program 144 (e.g., treatment planning software) along with medical images 146 and patient data 145. The medical images 146 can include information such as imaging data associated with a patient's anatomical region, organ, or volume of interest segmentation data. The patient data 145 can include information such as: (1) functional organ modeling data (e.g., serial organs versus parallel organs, appropriate dose response models, etc.); (2) radiation dose data (e.g., dose-volume histogram (DVH) information); or (3) other clinical information about the patient and treatment process (e.g., other surgeries, chemotherapy, previous radiation therapy, etc.).

[0033] In addition, the processor 114 can utilize the software program to: generate intermediate data, such as, for example, updated parameters to be used by the neural network model; or generate intermediate 2D or 3D images, which can then be subsequently stored in the memory 116. The processor 114 can send an executable radiation therapy treatment plan 142 to the radiation therapy device 130 via the communication interface 118 to the network 120, where the radiation therapy plan will be used to treat the patient with radiation. In addition, the processor 114 can execute the software program 144 to implement functions such as image conversion, image segmentation, deep learning, neural networks, and artificial intelligence. For example, the processor 114 can execute a software program 144 that trains or delineates medical images; such software 144, when executed, can train a boundary detector or utilize a shape dictionary.

[0034] The processor 114 may be a processing device including one or more general-purpose processing devices, such as a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), etc. More specifically, the processor 214 may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processor that implements other instruction sets, or a processor that implements a combination of instruction sets. The processor 114 may also be implemented by one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a system on a chip (SoC), etc. As will be appreciated by those skilled in the art, in some embodiments, the processor 114 may be a special-purpose processor rather than a general-purpose processor. The processor 114 may include one or more known processing devices, such as those from Intel. TM Pentium manufactured TM 、Core TM , Xeon TM or series of microprocessors; from AMD TM Made by Turion TM , Athlon TM 、Sempron TM 、Opteron TM FX TM 、Phenom TM series of microprocessors; or any of a variety of processors manufactured by Sun Microsystems. Processor 114 may also include a graphics processing unit, such as a graphics processing unit from Nvidia TM Manufactured series of GPUs; from Intel TM GMA, Iris TM series of GPUs; or from AMD TM Made by Radeon TM Processor 114 may also include an accelerated processing unit, such as an Intel TM Xeon Phi TM The disclosed embodiments are not limited to any type of processor that is otherwise configured to meet the computational requirements of identifying, analyzing, maintaining, generating and / or providing large amounts of data or manipulating such data to perform the methods disclosed herein. In addition, the term "processor" may include more than one processor, such as a multi-core design or multiple processors each having a multi-core design. The processor 114 may execute sequences of computer program instructions stored in the memory 116 to perform various operations, processes, and methods described in more detail below.

[0035] The memory device 116 may store medical images 146. In some embodiments, the medical images 146 may include one or more MRI images (e.g., 2D MRI, 3D MRI, 2D flow MRI, 4D MRI, 4D volume MRI, 4D imaging MRI, etc.), functional MRI images (e.g., fMRI, DCE-MRI, diffusion MRI), computed tomography (CT) images (e.g., 2D CT, cone beam CT, 3D CT, 4D CT), ultrasound images (e.g., 2D ultrasound, 3D ultrasound, 4D ultrasound), positron emission tomography (PET) images, X-ray images, fluoroscopic images, radiotherapy portal images, single photon emission computed tomography (SPECT) images, computer-generated synthetic images (e.g., pseudo CT images), etc. In addition, the medical images 146 may also include medical image data, such as training images and ground truth images, contour images, and dose images. In an embodiment, the medical images 146 may be received from the image acquisition device 132. Thus, the image acquisition device 132 may include an MRI imaging device, a CT imaging device, a PET imaging device, an ultrasound imaging device, a fluoroscope device, a SPECT imaging device, an integrated linear accelerator and MRI imaging device, or other medical imaging device for obtaining medical images of a patient. The medical image 146 may be received and stored in any type of data or in any type of format that the image processing device 112 may use to perform operations consistent with the disclosed embodiments. The memory device 116 may be a non-transitory computer-readable medium, such as a read-only memory (ROM), a phase-change random access memory (PRAM), a static random access memory (SRAM), a flash memory, a random access memory (RAM), a dynamic random access memory (DRAM) such as a synchronous DRAM (SDRAM), an electrically erasable programmable read-only memory (EEPROM), a static memory (e.g., a flash memory, a flash disk, a static random access memory), and other types of random access memory, a cache memory, a register, a compact disk read-only memory (CD-ROM), a digital versatile disk (DVD) or other optical storage device, a cassette, other magnetic storage device, or any other non-transitory medium that can be used to store information (e.g., stored in any format) that can be accessed by the processor 114 or any other type of computer device, including images, data, or computer-executable instructions. The computer program instructions can be accessed by the processor 114, read from the ROM or any other suitable memory location, and loaded into the RAM for execution by the processor 114. For example, the memory 116 may store one or more software applications.The software applications stored in the memory 116 may include, for example, an operating system 143 for common computer systems and for software-controlled devices. In addition, the memory 116 may store an entire software application or only a portion of a software application that is executable by the processor 114. For example, the memory device 116 may store one or more radiation therapy treatment plans 142.

[0036] The image processing device 112 can communicate with the network 120 via a communication interface 118, which can be communicatively coupled to the processor 114 and the memory 116. The communication interface 118 can provide a communication connection between the image processing device 112 and components of the radiation therapy system 100 (e.g., allowing data to be exchanged with external devices). For example, in some embodiments, the communication interface 118 can have appropriate interface circuitry to connect to a user interface 136, which can be a hardware keyboard, keypad, or touch screen, through which a user can input information into the radiation therapy system 100.

[0037] The communication interface 118 may include, for example, a network adapter, a cable connector, a serial connector, a USB connector, a parallel connector, a high-speed data transmission adapter (e.g., optical fiber, USB 3.0, thunderbolt, etc.), a wireless network adapter (e.g., a WiFi adapter), a telecommunication adapter (e.g., 3G, 4G / LTE, etc.), etc. The communication interface 118 may include one or more digital or analog communication devices that allow the image processing device 112 to communicate with other machines and devices (e.g., remotely located components) via the network 120.

[0038] The network 120 can provide the functionality of a local area network (LAN), a wireless network, a cloud computing environment (e.g., software as a service, platform as a service, infrastructure as a service, etc.), a client-server, a wide area network (WAN), etc. For example, the network 120 can be a LAN or a WAN, which can include other systems S1 (138), S2 (140), and S3 (141). The systems S1, S2, and S3 can be the same as the image processing device 112, or can be different systems. In some embodiments, one or more of the systems in the network 120 can form a distributed computing / simulation environment that collaboratively performs the embodiments described herein. In some embodiments, one or more of the systems S1, S2, and S3 can include a CT scanner that obtains a CT image (e.g., a medical image 146). In addition, the network 120 can be connected to the Internet 122 to communicate with servers and clients remotely residing on the Internet.

[0039] Thus, the network 120 may allow data transfer between the image processing device 112 and a variety of other systems and devices, such as the OIS 128, the radiation therapy device 130, and the image acquisition device 132. In addition, data generated by the OIS 128 and / or the image acquisition device 132 may be stored in the memory 116, the database 124, and / or the hospital database 126. Data may be transmitted / received via the network 120 through the communication interface 118 for access by the processor 114, as desired.

[0040] The image processing device 112 can communicate with the database 124 through the network 120 to send / receive various types of data stored on the database 124. For example, the database 124 may include machine data, which is information associated with the radiation therapy device 130, the image acquisition device 132, or other machines related to radiation therapy. The machine data information may include radiation beam size, arc placement, beam on and off duration, machine parameters, segments, multi-leaf collimator (MLC) configuration, gantry speed, MRI pulse sequence, etc. The database 124 may be a storage device and may be equipped with an appropriate database management software program. Those skilled in the art will understand that the database 124 may include multiple devices located in a central or distributed manner.

[0041] In some embodiments, the database 124 may include a processor-readable storage medium (not shown). Although the processor-readable storage medium in the embodiments may be a single medium, the term "processor-readable storage medium" should be considered to include a single medium or multiple media (e.g., a centralized or distributed database or associated cache memory and server) storing one or more computer-executable instruction sets or data. The term "processor-readable storage medium" should also be considered to include any medium capable of storing or encoding an instruction set executed by a processor and causing the processor to perform any one or more methods in the present disclosure. The term "processor-readable storage medium" should be understood accordingly to include, but is not limited to, solid-state memory, optical and magnetic media. For example, a processor-readable storage medium may be or may include one or more volatile, non-transient or non-volatile tangible computer-readable media.

[0042] The image processor 114 may communicate with the database 124 to read images into the memory 116 or to store images from the memory 116 to the database 124. For example, the database 124 may be configured to store a plurality of images (e.g., 3D MRI, 4D MRI, 2D MRI slice images, CT images, 2D fluoroscopic images, X-ray images, raw data from MR scans or CT scans, Digital Imaging and Communications in Medicine (DIMCOM) data, etc.) received by the database 124 from the image acquisition device 132. The database 124 may store data to be used by the image processor 114 when executing the software program 144 or when creating the radiation therapy treatment plan 142. The database 124 may store data generated by a trained neural network, including network parameters constituting a model learned by the network and resulting prediction data. The image processing device 112 can receive imaging data 146 (e.g., 2D MRI slice images, CT images, 2D fluoroscopic images, X-ray images, 3D MRI images, 4D MRI images, etc.) from a database 124, a radiotherapy device 130 (e.g., an MRI linear accelerator), and / or an image acquisition device 132 to generate a treatment plan 142.

[0043] In an embodiment, the radiation therapy system 100 may include an image acquisition device 132, which may acquire medical images of the patient (e.g., magnetic resonance imaging (MRI) images, 3D MRI, 2D flow MRI, 4D volume MRI, computed tomography (CT) images, cone beam CT, positron emission tomography (PET) images, functional MRI images (e.g., fMRI, DCE-MRI, and diffusion MRI), X-ray images, fluoroscopic images, ultrasound images, radiation therapy portal images, single photon emission computed tomography (SPECT) images, etc.). The image acquisition device 132 may be, for example, an MR imaging device, a CT imaging device, a PET imaging device, an ultrasound device, a fluoroscopic device, a SPECT imaging device, or any other suitable medical imaging device for obtaining one or more medical images of the patient. The images acquired by the image acquisition device 132 may be stored in the database 124 as imaging data or test data. As an example, the images acquired by the imaging acquisition device 132 may also be stored in the memory 116 by the image processing device 112 as medical image data 146.

[0044] In an embodiment, for example, the image acquisition device 132 can be integrated with the radiation therapy device 130 into a single device (e.g., an MRI device combined with a linear accelerator, which is also referred to as an "MRI linear accelerator"). Such an MRI linear accelerator can be used, for example, to determine the location of a target organ or target tumor in a patient's body to accurately guide the radiation therapy to a predetermined target according to the radiation therapy treatment plan 142.

[0045] The image acquisition device 132 can be configured to acquire one or more images of the patient's anatomical structure for a region of interest (e.g., a target organ, a target tumor, or both). Each image (which is typically a 2D image or slice) may include one or more parameters (e.g., 2D slice thickness, orientation, and position, etc.). In an embodiment, the image acquisition device 132 can acquire 2D slices with any orientation. For example, the orientation of the 2D slice may include a sagittal orientation, a coronal orientation, or an axial orientation. The processor 114 may adjust one or more parameters (e.g., the thickness or orientation of the 2D slice) to include the target organ or target tumor. In an embodiment, the 2D slice may be determined based on information such as a 3D MRI volume. Such 2D slices may be acquired by the image acquisition device 132 in "near real time" while the patient is receiving radiation therapy treatment, such as when using the radiation therapy device 130. "Near real time" means acquiring data in a duration of milliseconds or less.

[0046] The image processing device 112 can generate and store a radiation therapy treatment plan 142 for one or more patients. The radiation therapy treatment plan 142 can provide information about the specific radiation dose to be applied to each patient. The radiation therapy treatment plan 142 can also include other radiation therapy information, such as beam angles, dose-volume histogram information, the number of radiation beams to be used during treatment, the dose of each beam, etc.

[0047] The image processor 114 may be used by using a software program 144 such as treatment planning software (e.g., manufactured by Elekta AB of Stockholm, Sweden). ) to generate a radiation therapy treatment plan 142. To generate the radiation therapy treatment plan 242, the image processor 114 can communicate with the image acquisition device 132 (e.g., a CT device, an MRI device, a PET device, an X-ray device, an ultrasound device, etc.) to access the patient image and depict the target (e.g., a tumor). In some embodiments, it may be necessary to depict one or more organs at risk (OARs), such as healthy tissue surrounding a tumor or immediately adjacent to a tumor. Therefore, in the case where the OAR is close to the target tumor, segmentation of the OAR can be performed. In addition, if the target tumor is close to the OAR (e.g., the prostate is close to the bladder and rectum), by segmenting the OAR from the tumor, the radiation therapy system 100 can not only study the dose distribution in the target, but also study the dose distribution in the OAR.

[0048] In order to depict the target organ or target tumor relative to the OAR, a medical image (e.g., MRI image, CT image, PET image, fMRI image, X-ray image, ultrasound image, radiotherapy portal image, SPECT image, etc.) of a patient undergoing radiotherapy can be obtained by the image acquisition device 132 in a non-invasive manner to reveal the internal structure of the body part. Based on the information in the medical image, the 3D structure of the relevant anatomical part can be obtained. In addition, during the treatment planning process, many parameters can be considered to balance the efficient treatment of the target tumor (e.g., so that the target tumor receives enough radiation dose to achieve effective treatment) with the low irradiation of the OAR (e.g., the OAR receives the lowest possible radiation dose). Other parameters that can be considered include: the location of the target organ and target tumor, the location of the OAR, and the movement of the target relative to the OAR. For example, the 3D structure can be obtained by outlining the target in each 2D layer or slice of the MRI image or CT image or outlining the OAR and combining the outlines of each 2D layer or slice. The contours may be generated manually (e.g., by a physician, dosimetrist, or medical staff using a program such as MONACO manufactured by Elekta AB of Stockholm, Sweden). TM ) or automatically generated (e.g., using a program such as the Atlas-based automatic segmentation software ABAS manufactured by Elekta AB of Stockholm, Sweden TM ). In certain embodiments, the 3D structure of the target tumor or OAR can be automatically generated by treatment planning software, for example, using one or more of the techniques shown and described elsewhere herein.

[0049] After the target tumor and OAR have been located and delineated, the dosimetrist, physician, or medical staff can determine the dose of radiation to be applied to the target tumor and any maximum dose values ​​that may be received by the OARs adjacent to the tumor (e.g., left and right parotid glands, optic nerves, eyes, lenses, inner ears, spinal cord, brainstem, etc.). After the radiation dose is determined for each anatomical structure (e.g., target tumor, OAR), a process known as inverse planning can be performed to determine one or more treatment planning parameters that will achieve the desired radiation dose distribution. Examples of treatment planning parameters include: volume delineation parameters (e.g., which define the target volume, outline sensitive structures, etc.), margins around the target tumor and OAR, beam angle selection, collimator settings, and beam opening times. During the inverse planning process, the physician can define dose constraint parameters that set limits on how much radiation an OAR can receive (e.g., limiting the application of a full dose to a tumor target and a zero dose to any OAR; limiting the application of 95% of the dose to a target tumor; limiting the spinal cord, brainstem, and optic nerve structures to receive ≤45Gy, ≤55Gy, and <54Gy, respectively). The results of the inverse planning can constitute a radiation therapy treatment plan 142, which can be stored in the memory 116 or the database 124. Some of these treatment parameters may be interrelated. For example, adjusting one parameter (e.g., the weight of different targets, such as increasing the dose to the target tumor) in an attempt to change the treatment plan may affect at least one other parameter, which in turn may result in a different treatment plan being formulated. Therefore, the image processing device 112 can generate a customized radiation therapy treatment plan 142 with these parameters to enable the radiation therapy device 130 to provide radiation therapy treatment to the patient.

[0050] In addition, the radiation therapy system 100 may include a display device 134 and a user interface 136. The display device 134 may include one or more display screens that display medical images, interface information, treatment plan parameters (e.g., contours, doses, beam angles, etc.), treatment plans, targets, target positioning and / or target tracking, or any related information to the user. The user interface 136 may be a keyboard, keypad, touch screen, or any type of device through which a user can input information to the radiation therapy system 100. Alternatively, the display device 134 and the user interface 136 may be integrated into a computer such as a tablet computer (e.g., an Apple Lenovo Samsung etc.) device.

[0051] Additionally, a subset or all of the components of the radiation therapy system 100 may be implemented as a virtual machine (e.g., VMWare TM , Hyper-V TMFor example, a virtual machine may be software used as hardware. Thus, a virtual machine may include at least one or more virtual processors, one or more virtual memories, and one or more virtual communication interfaces that are used together as hardware. For example, the image processing device 112, the OIS 128, and the image acquisition device 132 may be implemented as a virtual machine. If processing power, memory, and computing power are available, the entire radiation therapy system 100 may be implemented as a virtual machine.

[0052] Figure 2 An example is shown including a radiotherapy device 202, which may include a radiation source (e.g., an x-ray source or a linear accelerator), a treatment table 216, an imaging detector 214, and a radiotherapy output 204. The radiotherapy device 202 may be configured to emit a radiation beam 208 to provide treatment to a patient. The radiotherapy output 204 may include one or more attenuators or collimators, such as a multi-leaf collimator (MLC). The patient may be positioned in an area 212 supported by the treatment table 216 to receive a radiation therapy dose according to a radiotherapy treatment plan.

[0053] The radiation therapy output 204 may be mounted or attached to a gantry 206 or other mechanical support. When the table 216 is inserted into the treatment area, one or more chassis motors (not shown) may rotate the gantry 206 and the radiation therapy output 204 about the table 216. In an embodiment, the gantry 206 may be capable of continuously rotating about the table 216 when the table 216 is inserted into the treatment area. In another embodiment, the gantry 206 may be rotated to a predetermined position when the table 216 is inserted into the treatment area. For example, the gantry 206 may be configured to rotate the treatment output 204 about an axis ("A"). Both the table 216 and the radiation therapy output 204 may be capable of independently moving to other positions around the patient, for example, being capable of moving in a transverse direction ("T"), being capable of moving in a lateral direction ("L"), or being capable of rotating about one or more other axes, such as about a transverse axis (indicated as "R"). A controller communicatively connected to one or more actuators (not shown) can control the movement or rotation of the table 216 to appropriately position the patient in or out of the radiation beam 208 according to the radiation therapy treatment plan. For example, both the table 216 and the gantry 206 can be independently movable with multiple degrees of freedom relative to each other, allowing the patient to be positioned so that the radiation beam 208 can be precisely aimed at the tumor.

[0054] Figure 2The origin of the coordinate system shown in FIG. 2 (including axes A, T, and L) can be located at an isocenter 210. The isocenter can be defined as the location where the central axis of the radiotherapy beam 208 intersects the origin of the coordinate axes, such as to deliver a prescribed radiation dose to a patient's body surface or body. Alternatively, the isocenter 210 can be defined as the location where the central axis of the radiotherapy beam 208 intersects the patient for various rotational positions of the radiotherapy output 204 positioned by the gantry 206 about the axis A.

[0055] The gantry 206 may also have an attached imaging detector 214. The imaging detector 214 is preferably located opposite the radiation source 204, and in embodiments, the imaging detector 214 may be located within the field of the treatment beam 208. The imaging detector 214 may be mounted on the gantry 206 preferably opposite the radiation therapy output 204 to maintain alignment with the treatment beam 208. As the gantry 206 rotates, the imaging detector 214 rotates about the axis of rotation. In embodiments, the imaging detector 214 may be a flat panel detector (e.g., a direct detector or a scintillator detector). In this manner, the imaging detector 214 may be used to monitor the treatment beam 208, or the imaging detector 214 may be used to image the patient's anatomical structure, such as portal imaging. The control circuitry of the radiation therapy device 202 may be integrated within the system 100 or remote from the system 100.

[0056] In an illustrative embodiment, one or more of the table 216, the treatment output 204, or the gantry 206 can be automatically positioned, and the treatment output 204 can establish the treatment beam 208 according to the specified dose for a particular treatment delivery instance. A treatment delivery sequence can be specified according to a radiation therapy treatment plan, such as using one or more different orientations or positions of the gantry 206, the table 216, or the treatment output 204. The treatment deliveries can occur sequentially, but can intersect at a desired treatment site on or in the patient's body, such as at the isocenter 210. Thus, a prescribed cumulative dose of radiation therapy can be delivered to the treatment site while reducing or avoiding damage to tissue near the treatment site.

[0057] Figure 2The examples generally illustrate embodiments of a radiotherapy device configured to provide radiotherapy treatment to a patient, including a configuration in which the radiotherapy output can rotate about a central axis (e.g., axis "A"). Other radiotherapy output configurations can be used. For example, the radiotherapy output can be mounted to a robotic arm or manipulator having multiple degrees of freedom. In yet another embodiment, the treatment output can be fixed, such as located in an area laterally separated from the patient, and a platform supporting the patient can be used to align the radiotherapy isocenter with a designated target site in the patient's body. In another embodiment, the radiotherapy device can be a combination of a linear accelerator and an image acquisition device. As will be appreciated by one of ordinary skill in the art, in some embodiments, the image acquisition device can be an MRI, X-ray, CT, CBCT, spiral CT, PET, SPECT, optical tomography, fluorescence imaging, ultrasound imaging, or a radiotherapy portal imaging device, among others.

[0058] Figure 3 An example of a system 300 architecture is shown for performing training of at least one statistical learning model (e.g., a convolutional neural network (CNN)) and for providing a synthesized image using an image synthesis technique based on the CNN model. The system 300 may be included as Figure 1 A portion of the radiation therapy system 100 shown, or Figure 3 The system 300 can be communicatively coupled to such a radiation therapy system. Figure 3 As shown, the system 300 may include components for performing two phases, namely a training phase and an image synthesis phase. To perform the training phase, the system 300 may include a training image database 301 and a CNN model training unit 302. To perform synthesis, the system 300 may include an image synthesis unit 303 based on a CNN model and a medical image database 304. In some embodiments, the system 300 may include Figure 3 More or less of the components shown. For example, in the case where a CNN model for image synthesis is pre-trained and provided, the system 300 may only include an image synthesis unit 303 based on the CNN model and a medical image database 304. The system 300 may optionally include a network 305. In some embodiments, the network 305 may be replaced by a wired data communication system or device.

[0059] In some embodiments, the various components of system 300 may be remotely located from one another or in different spaces and may be connected via network 305, such as Figure 3In some embodiments, some components of the system 300 may be located at the same location or in one device. For example, the training image database 301 may be located at a location with the CNN model training unit 302, or may be a part of the CNN model training unit 302. As another example, the CNN model training unit 302 and the image synthesis unit 303 may be in the same computer or processing device.

[0060] like Figure 3 As shown, the CNN model training unit 302 can communicate with the training image database 301 to receive one or more training image sets. The training image set stored in the training image database 301 can be obtained from a medical image database (e.g., a medical image database containing medical images during a previously acquired radiotherapy treatment process). The training image set may include 3D imaging data or 2D imaging data. The 3D imaging data can be selectively divided into one or more ordered stacks of adjacent 2D images. The count of adjacent 2D images in each stack can be determined based on various factors, such as the size of the 3D image, the specific framework of the CNN model, the relationship between anatomical structures in adjacent 2D images along an axis orthogonal to the 2D image, or considering other factors.

[0061] Consistent with the disclosed embodiments, training images may be acquired using a variety of imaging modalities, including MRI, functional MRI (e.g., fMRI, DCE-MRI, and diffusion MRI), CT, CBCT, spiral CT, PET, SPECT, X-ray, optical tomography, fluorescence imaging, ultrasound imaging, and radiation therapy portal imaging. In some embodiments, training data may be collected from an oncology information system or other centralized repository. For example, the training data may be collected by Figure 1 The image acquisition device 232 acquires the training image.

[0062] Return to reference Figure 3 , the CNN model training unit 302 may use the training images received from the training image database 301 to train the CNN model to receive imaging data corresponding to a first imaging modality type and synthesize imaging data corresponding to a different second imaging modality type. For example, the training architecture may include Figure 4) (corresponding to the adversarial training architecture). The CNN model training unit 302 may include a processor and a non-transitory computer-readable medium. The processor may perform training by executing instructions of the training process stored in the computer-readable medium. The CNN model training unit 302 may additionally include input and output interfaces to communicate with the training image database 301, the network 305, or a user interface (not shown). The user interface may be used to select a training image set, adjust one or more parameters of the training process, or to select or modify the framework of the CNN model, which are only illustrative examples. The CNN model training unit 302 can be implemented using hardware specially programmed by software that executes the training process.

[0063] The image synthesis unit 303 may receive at least one trained CNN model from the CNN model training unit 302. The image synthesis unit 303 may include a processor and a non-transitory computer readable medium (such as Figure 1 The processor may synthesize the imaging information according to the instructions stored in the medium. The image synthesis unit 303 may additionally include input and output interfaces to communicate with the medical image database 304, the network 305, or a user interface. The user interface may be used to select imaging information to be converted from a first imaging modality type to a second imaging modality type, including initiating an image synthesis process, displaying an input image or an output image, or performing further analysis based on the input imaging data or the synthesized output imaging data. In combination with examples elsewhere in this document (e.g., in combination with Figure 4 , Figure 5 , Figure 6 , Figure 7 or Figure 8 Various techniques for performing CNN-based image synthesis are described in the examples shown and described.

[0064] The image synthesis unit 303 can communicate with the medical image database 304 to receive input imaging data for use in synthesizing different output imaging data. The images stored in the medical image database 304 can be obtained from, for example, a medical image database containing 2D or 3D images of a radiotherapy treatment session. As described herein, the images can be obtained based on images obtained by a medical imaging device (e.g., Figure 2 The image acquisition device 232 shown in FIG. 2 reconstructs the image data by acquiring a 2D projection image. Return to reference Figure 3 , the image synthesis unit 303 can provide one or more synthesized images 306 for further processing, such as supporting the formulation of radiation therapy treatment planning or radiation therapy delivery. In an example, the images 306 output by the image synthesis unit 303 can be stored in the training image database 401 or used in other ways, such as for further training of the statistical learning model.

[0065] The imaging data to be converted using the image synthesis unit 303 can be acquired using various imaging modalities, including MRI, functional MRI (e.g., fMRI, DCE-MRI, and diffusion MRI), CT, CBCT, spiral CT, PET, SPECT, X-ray, optical tomography, fluorescence imaging, ultrasound imaging, and radiotherapy portal imaging, etc. In some embodiments, the medical image database 304 can be an integrated part of the image synthesis unit 303, or located at the same location as the image synthesis unit 303, such as in a radiotherapy treatment room. In the system 300, the network 305 can provide connections between any of the above components. For example, the network 305 can be a local area network (LAN), a wireless network, a cloud computing environment (e.g., software as a service, platform as a service, infrastructure as a service), a client-server, a wide area network (WAN), etc.

[0066] Figure 4 An example of an adversarial training architecture 400 is shown for performing adversarial training on at least one statistical learning model (e.g., a generator convolutional neural network (CNN) “G 1 ” 420), for example, to provide a synthetic image 428 using an image synthesis technique based on a CNN model. Figure 4 The training architecture 400 schematically illustrated in FIG. 4 does not require the use of "paired" imaging information. The input imaging data 414 used for training may include images (e.g., images 416) corresponding to a first imaging modality type (e.g., MR imaging data). Unpaired imaging data 422, such as images 426, corresponding to a different second imaging modality type, may also be used, such as images that do not need to be registered or aligned with corresponding images in the input imaging data 414.

[0067] As mentioned elsewhere herein, statistical learning models can be trained in a manner that is independent of "paired" imaging data. "Paired" imaging data in a training context can refer to a pairing of first imaging data corresponding to a first imaging modality type and associated "paired" second imaging data corresponding to a different second imaging modality type. In a "paired" scheme, the second imaging data includes images that are aligned (e.g., registered) or otherwise obtained to correspond as closely as possible to corresponding images from the first imaging data. In contrast, the use of "unpaired" imaging data does not require alignment or close correspondence between training images for the first imaging data and training images for the different second imaging data. For example, the first imaging data may be associated with or otherwise correspond to a first group of patients, while the second imaging data may be associated with or otherwise correspond to a partially different or completely different group of patients.

[0068] Typically, unpaired imaging data is more readily available. As an illustrative example, if it is not required that each CT image has a well-aligned corresponding MR image or that each MR image has a well-aligned corresponding CT image, many CT images and many MR images (e.g., hundreds or more) may be used for training.

[0069] exist Figure 4 In the example of FIG. 4 , the architecture 400 may include a single generator convolutional neural network (CNN) 420 “G 1 " and a single discriminator convolutional neural network 430 "D 1 In the adversarial scheme shown for architecture 400, a generator 420 is responsible for mapping or converting its input (e.g., image 416) from one domain (e.g., one modality type) to another modality type (e.g., to produce synthetic imaging data 428, such as synthetic image 418). As an illustrative example, the input image 416 is an MR image, and the output image 418 is a "pseudo" CT image or a synthetic CT (SCT) image. A discriminator 430 facilitates training of the generator 420 in the following manner: The discriminator 430 aims to discriminate between synthetic imaging data 428 from the generator 420 relative to unpaired training imaging data 422 representing actual or "real" CT imaging data (e.g., including CT images 426 obtained using CT imaging rather than synthesized).

[0070] The architecture 400 may be referred to as a type of generative adversarial network (GAN) in which a generator 420 and a discriminator 430 compete, wherein the generator is tuned or optimized to provide synthetic imaging data 428 that corresponds as closely as possible to actual imaging data 422 to "fool" the discriminator 430. The output 436 of the discriminator 430 may be a categorical value, such as a value between zero and one, where one represents a determination that the input image to the discriminator represents an actual image, and zero represents a determination that the input image is "fake." In the extreme case, as training proceeds, the generator 420 may produce imaging data 428 that is so similar to the actual imaging data 422 that the discriminator 430 cannot distinguish between the synthetic imaging data 428 and the actual imaging data 422 used for training. Once training is complete (e.g., the generator 420 is tuned to provide imaging data that is close to a specified correlation with the actual imaging data according to the discriminator 430), the generator 420 model may be saved and used alone for image synthesis.

[0071] Without sufficient constraints, the generator 420 model may create images that look real to the discriminator 430 but have very poor correlation with the input to the generator 420 model. As an illustrative example, in the worst case, the generator 420 may always produce a fixed output image that is independent of the input image. In another illustrative example, the generator 420 may randomly output a real CT image from the training imaging data 422 corpus.

[0072] because Figure 4 The method shown does not rely on paired training data, so the output of the generator 420 is not directly compared to some corresponding ground truth. Instead, the cross-modal similarity measure 412 can be used, for example, to implement a similarity determination, directly comparing the synthetic imaging data 428 from the output of the generator 420 with the imaging data 414 at the input of the generator 420. This avoids the mode collapse problem and helps ensure that the generator 420 produces synthetic imaging data 428 that is well correlated with the generator 420 input imaging data 414. Figure 4 The illustrated training architecture 400 may be partially or fully computer-implemented, including, for example, using a training unit 402 .

[0073] The representation of the optimization method implemented by the training unit 402 can be expressed as follows:

[0074]

[0075] In Formula 1, G* represents the generator 420 with adjusted (e.g., optimized) parameters, such as provided by the output 434 of the training unit 402, and "L(G, D)" represents the "loss" function or objective function "L". The method shown in Formula 1 may refer to adjusting the discriminator 430 convolutional network so that the value of the objective is increased or maximized over the input distribution for the discriminator 430, and adjusting the generator 420 so that the value of the objective is reduced or minimized over the input distribution for the generator 420. Various methods may be used to implement such training, such as including performing a stochastic gradient descent (SGD) iteration to adjust the generator 420 using the output 434, then performing a stochastic gradient descent iteration to adjust the discriminator 430 using the output 432, and so on. Such adjustment may include: modifying parameters within the corresponding deep convolutional neural networks defining the generator 420 and the discriminator 430, respectively. Other training techniques may be used, such as SGD with momentum, Nesterov accelerated gradient technique, AdaGrad, Adadelta, RMSprop, or Adam. Other techniques may be used, for example involving evolutionary methods, genetic techniques, simulated annealing or particle swarm optimization, just as illustrative examples.

[0076] As an illustrative example, the discriminator 430 may be implemented using a convolutional neural network acting as a classifier, such as in Figure 5 The example of illustratively shown in FIG. 4 shows that the generator 420 can use a Figure 6 The architecture shown can be implemented as a convolutional neural network, for example including short-range or long-range skip connections between convolutional layers. Figure 4 , the objective function may include a corresponding adversarial objective 410 (e.g., represented by “L GAN ”) and similarity measure 412 (e.g., represented by “L SIM ” indicates the contribution of:

[0077] L=L GAN (G, D) + λ·L sim (G) Formula 2

[0078] The value "λ" may represent an adjustable weighting factor. In the expression of Equation 2, and as shown below in Equation 3, the term L GAN It may include two items. The first item may represent the expected value (e.g., Among them, "E Y ” is from the probability distribution “P B (y)", the expected value of the probability distribution "P B The term “D(y)” represents the distribution of actual imaging data used as input to the discriminator network for training purposes, and “D(y)” represents the operation of the discriminator network on the actual imaging data used for training), and the second term may represent the expected value that the input to the discriminator including the synthetic image from the first generator is not recognized by the first discriminator as actual imaging data (e.g., Among them, "E x ” is from the probability distribution “P A (x)”, the expected value of the probability distribution “P A (x)” represents the input distribution of imaging data “x”, “G(x)” represents the operation of the generator network on the input imaging data in order to produce a synthetic image, and “D(G(x))” represents the operation of the discriminator network on the synthetic image provided by the generator network), as shown below in Equation 3:

[0079]

[0080] The similarity measure 412 may be expressed as shown below in Equation 4:

[0081]

[0082] The term “Sim(x, G(x))” may represent a similarity function that indicates the similarity between two images of different modalities (e.g., an input image “x” and a synthesized image “G(x)” provided by the generator 420). Inter-modality image similarity metrics are generally available and may include the use of techniques including one or more of mutual information, normalized mutual information, cross-correlation, local cross-correction, or other techniques. The use of a cross-modal similarity function enables Figure 4 The training architecture 400 can be used with unpaired imaging data. Figure 4 The methods shown in are not the only methods that can be used. Figure 4 Compared to the method shown in FIG. 1 , the training of the image synthesis or image modality conversion model can be performed using the “CycleGAN” method using unpaired imaging data. However, such an approach has disadvantages because typically in the CycleGAN method, two image synthesis models are trained simultaneously, including a first synthesis model that converts from a first imaging modality type to a second imaging modality type (e.g., similar to G 1 ) and a second synthetic model that converts from the second imaging modality type back to the first imaging modality type. In addition, such CycleGAN methods usually require training two "discriminators" at the same time. Training two models is unnecessary and consumes significantly more resources in terms of processing and memory consumption. Figure 4 The architecture 400 shown in and generally described in this document provides a simpler adversarial network architecture compared to CycleGAN, and the techniques described herein can still be trained using unpaired imaging data.

[0083] Used with Figure 4 The illustrated architecture 400 incorporates imaging data for training that may be site-specific or may be obtained from a centralized repository (e.g., representing multiple clinical sites). Such training information may include, for example, one or more variations in resolution, contrast, imaged region, imaging device type or manufacturer, or other variations, such as to help provide generator 420 parameters that are robust to such variations across one or more of patients, sites, or specific imaging device configurations.

[0084] Figure 5 An example is shown that includes a statistical learning model (e.g., a convolutional neural network (CNN) 520) that is used, for example, to perform classification of medical imaging data (e.g., to distinguish between synthesized imaging data and actual imaging data), which can be used to train another statistical learning model to perform image synthesis. Figure 5As shown, a CNN model 520 for image classification may receive an image 522 as input from one or more images 526 provided for training a separate generator convolutional neural network. Various methods (e.g., Figure 4 or methods discussed elsewhere in this paper) to train the CNN 520.

[0085] A series of one or more convolutional layers 528 may be used to build the classification output 529 from the CNN model 520. Each convolutional layer 528 may have a number of parameters, such as a width ("W") and height ("H") determined by the upper input layer (e.g., the size of the input to the convolutional layer 528) and a count ("N") of filters or kernels in the layer and their sizes. The number of filters may be referred to as the channels of a particular convolutional layer. Thus, each convolutional layer 528 may be configured as follows: Figure 5 , but such representation is illustrative only. The input to each convolutional layer 528 is typically convolved with a filter over its width and height, and a 2D activation map or feature map corresponding to the filter is produced. The above convolution is performed for all filters of each convolutional layer, and the resulting activation maps or feature maps are stacked along the channel dimension to generate a 3D output. The output of the previous convolutional layer can be used as the input to the next convolutional layer.

[0086] In some embodiments, the CNN model 520 includes one or more pooling layers. A pooling layer can be added between two consecutive convolutional layers 528 in the CNN model 510. The pooling layer can operate independently on each channel of its input (e.g., activation maps or feature maps from a previous convolutional layer), and the pooling layer reduces its spatial dimension by performing a form of non-linear downsampling. In some examples, information from non-adjacent layers can "skip" intermediate layers and can be aggregated with the output of the following convolutional layer before passing through the pooling layer, as described in detail in the example of FIG. Figure 6 As discussed. Figure 5 As shown, the function of the pooling layer may include gradually reducing the spatial dimension of the extracted activation map or feature map to reduce the number of parameters and calculations in the network and control overfitting. The number and placement of the pooling layer may be determined based on various factors, for example, based on the configuration of the convolutional network architecture, the size of the input, the size of the convolutional layer 528, or the application of the CNN model 520.

[0087] Various nonlinear functions can be used to implement the pooling layer. For example, max pooling can be used. Max pooling can divide the input image slices into a set of overlapping or non-overlapping sub-regions with a predetermined span. For each sub-region, the maximum value in the corresponding sub-region in the max pooling output partition. This can effectively downsample each slice of the input along both its width and its height, while the channel dimension remains unchanged. Other suitable functions can be used to implement the pooling layer, such as average pooling or even L2 norm pooling.

[0088] The CNN model 520 may optionally include one or more additional layers. As a non-limiting example, a rectified linear unit (ReLu) layer or a parameterized ReLU (PReLU) layer may be selectively added after the convolutional layer to generate an intermediate activation map or feature map. For example, the ReLu layer may desirably increase the nonlinear properties of the predictor function and the entire CNN model 520 without affecting the corresponding dimensions of the convolutional layer 528. In addition, the ReLu layer may reduce or avoid saturation during the back-propagation training process.

[0089] As mentioned above, the classification output 529 from the CNN is a classification value, such as a value between zero and one, where one represents a determination that the input image 522 to the CNN model 520 represents an actual image, and zero represents a determination that the input image 522 is a "fake" synthesized by a separate generator statistical learning model.

[0090] Figure 6 An example is shown that includes a statistical learning model (e.g., a convolutional neural network (CNN) 600A), which is used, for example, to receive medical imaging data corresponding to a first imaging modality type (e.g., a first image 626), and to perform synthesis of medical imaging data corresponding to a different second imaging modality type using the received medical imaging data (e.g., to provide a second image at an output layer 632A). Figure 6 Examples can be used to provide Figure 4 The example mentioned in the "generator" model. Return to reference Figure 6 , CNN 600A can be classified as a "fully convolutional neural network" (FCN). Different FCN architectures can be used, such as an FCN architecture with an encoding portion 624A. The encoding portion is similar to a typical CNN that extracts a hierarchy of image features from low complexity to high complexity. The decoding portion 634A then transforms the features and gradually reconstructs a composite image from a coarser resolution to a finer resolution.

[0091] The architecture of CNN 600A shows long-range connections 640A, 640B, and 640C. Short-range connections may also be used, such as within encoding portion 624A or within decoding portion 634A. Figure 6 The use of long-range connections (between portions 624A and 634A) allows the use of higher-resolution features from the encoding portion (624A) as input to the (de)convolutional layers in the decoding portion (634A). Such a configuration enhances the ability of the decoding portion (634A) to generate high-resolution predictions. This shortcut also makes model 600A more flexible. For example, if high-resolution features (at the "top" of the network) are sufficient to produce accurate synthesis, model 600A can automatically learn through training to skip coarse-level features (at the "bottom" of the network, e.g. Figure 6 At layer 629A in FIG. 1 ). a Figure 6 The long-range connections 640A, 640B, 640C, and 640D shown in FIG. 1 provide corresponding copies 642A, 642B, 642C, and 642E, respectively, by taking the output of the corresponding convolutional layer.

[0092] Reference Figure 6 CNN 600A may provide input imaging data to a first convolutional layer having, for example, a 320×320 spatial size, and the convolutional layer may include 64 filters or “channels.” The input stack may represent a training image or a series of images to be converted. Figure 6 The CNN 600A may include five different resolution layers, for example, a first downsampling layer 628A having an output size of 160×160×128 channels, a second downsampling layer 628B having an output size of 80×80×256 channels, a third downsampling layer 628C having an output size of 40×40×512 layers, and a bottom layer 629A having an output size of 20×20×512 channels. The output of the bottom layer 629A may include feature domain information (e.g., a feature map or a set of feature vectors). Other numbers of layers or channels may be used, or different resolutions may be used.

[0093] As mentioned above, Figure 5 The mentioned pooling is used to achieve downsampling. Figure 6The decoding portion 634A of the CNN 600A may include upsampling or "deconvolution" layers, such as a first upsampling layer 636C, a second upsampling layer 636B, and a third upsampling layer 636A. The final or output layer 632A may provide a synthesized image having a resolution similar to or matching the resolution of the input 2D imaging data. Consistent with various embodiments of the present disclosure, the image segmentation method, system, device, or process based on the above-mentioned CNN model includes two stages: a training stage, using a training data set to train the CNN model; and an image synthesis stage (e.g., a runtime stage), using the trained CNN model to synthesize an image having a modality type different from the input image.

[0094] Figure 7 Generally, a technique 700 (e.g., a method) that can be implemented by a computer, for example, for performing medical imaging data synthesis using a statistical learning model, is shown. At 702, medical imaging data can be obtained using a first imaging modality type. As mentioned with respect to other examples described in this document, such imaging data can be obtained directly from a connected imaging device (e.g., an MR scanner), or such imaging data can be obtained from a medical imaging database. At 704, a trained statistical learning model can be applied to the received medical imaging data to synthesize a medical image corresponding to a different second imaging modality type. Such a learning model can include a generator convolutional neural network having ... Figure 4 and Figure 8 At 706, the synthesized medical image can be presented for display to a user or for further processing. For example, optionally, at 708, the synthesized image can be displayed on a workstation for radiation therapy treatment planning or management. In an example, optionally, at 710, an update or receipt of an update to a radiation therapy treatment protocol can be performed, for example, based at least in part on the synthesized medical image.

[0095] Figure 8Generally, a technique 800 (e.g., a method) that can be implemented by a computer, for example, for implementing an adversarial training architecture to establish a statistical learning model for performing medical imaging data synthesis is shown. At 802, training medical imaging data, such as data corresponding to a first imaging modality type, can be received. As mentioned elsewhere, such training data can be specific to a location, or can represent, for example, a wider range of imaging device configurations, settings, patients, or locations. At 804, a statistical learning model (e.g., a generator model for image synthesis) can be applied to the received medical image data to synthesize medical imaging data corresponding to a different second imaging modality type. At 806, the statistical learning model for synthesizing medical imaging data at 804 can be adjusted, for example, at least in part using a similarity determination between the training imaging data and the synthesized imaging data at the model output. At 808, a separate statistical learning model (e.g., a discriminator model for imaging data classification) can also be used to adjust the statistical learning model for synthesizing medical imaging data at 804. For example, the separate statistical learning model can be established to discriminate between the synthesized imaging data and the actual imaging data corresponding to the second imaging modality type. Likewise, the actual imaging data used to build the discriminator model may be site-specific, or may represent, for example, a larger range of imaging device configurations, settings, patients, or sites.

[0096] Fig.9A , Fig. 9B and Fig. 9C An illustrative example of views (axial, sagittal, and coronal views, respectively) including imaging data corresponding to a first imaging modality type (eg, MR imaging data) is shown. Fig. 10A , Fig. 10B and Fig. 10C An illustrative example of a view including synthetic imaging data corresponding to a second imaging modality type (e.g., CT imaging) is shown, the synthetic imaging data being obtained using the Fig.9A , Fig. 9B and Fig. 9C A statistical learning model (e.g., a deep convolutional neural network including a generator model) of the imaging data is provided. Fig.11A , Fig. 11B and Fig. 11C Illustrative examples of views including actual imaging data acquired using a second imaging modality type (e.g., CT imaging) are shown, and are presented herein for use with, respectively, Fig. 10A , Fig. 10B and Fig. 10C The synthetic imaging data is compared and shown to correspond well to Fig.11A , Fig. 11B and Fig. 11C The “ground truth” image of .

[0097] Various annotations

[0098] Each of the non-limiting aspects described above in this document may stand alone, or may be combined in various permutations or combinations or with one or more of the other aspects or other subject matter described in this document.

[0099] The above detailed description includes reference to the accompanying drawings, which form a part of the detailed description. The accompanying drawings illustrate the detailed description by way of illustration, in which the present invention can be put into practice. These embodiments are also generally referred to as "examples". Such examples may include elements other than the elements shown or described. However, the inventors also contemplate examples that only provide those elements shown or described. In addition, the inventors also contemplate examples (or one or more aspects thereof) of any combination or arrangement of those elements shown or described with respect to the specific examples (or one or more aspects thereof) shown or described herein or with respect to other examples (or one or more aspects thereof).

[0100] In the event of an inconsistent usage between this document and any document incorporated by reference, the usage in this document controls.

[0101] As is common in patent documents, in this document, the terms "a" or "an" are used to include one or more than one, independent of any other instance or usage of "at least one" or "one or more". In this document, unless otherwise indicated, the term "or" is used to represent a non-exclusive or, such that "A or B" includes "A but not B", "B but not A", and "A and B". In this document, the terms "including" and "in..." are used as the plain English equivalents of the corresponding terms "comprising" and "wherein". In addition, in the appended claims, the terms "including" and "comprising" are open-ended, that is, systems, devices, articles, combinations, formulations, or processes that include elements other than those listed after such terms in the claim are still considered to fall within the scope of the claim. In addition, in the appended claims, the terms "first", "second", and "third", etc. are used only as labels and are not intended to impose numerical requirements on their objects.

[0102] The method examples described herein may be at least partially machine or computer-implemented. Some examples may include a computer-readable medium or machine-readable medium encoded with instructions, which can be operated to configure an electronic device to perform the method described in the above examples. The implementation of these methods may include code such as microcode, assembly language code, higher-level language code, etc. Such code may include computer-readable instructions for performing various methods. The code may form part of a computer program product. In addition, in an example, for example, during execution or at other times, the code may be tangibly stored on one or more volatile, non-transient or non-volatile tangible computer-readable media. Examples of these tangible computer-readable media may include, but are not limited to, hard disks, removable disks, removable optical disks (e.g., compact disks and digital video disks), cassettes, memory cards or memory sticks, random access memory (RAM), read-only memory (ROM), etc.

[0103] The above description is intended to be illustrative rather than restrictive. For example, the above examples (or one or more aspects thereof) can be used in combination with each other. After reviewing the above description, other embodiments may be used by those skilled in the art, for example. An abstract is provided to comply with 37C.FR§1.72(b) to allow readers to quickly determine the nature of the technical disclosure. The abstract is submitted with the understanding that the abstract will not be used to interpret or limit the scope or meaning of the claims. In addition, in the above specific embodiments, various features may be combined together to organize the present disclosure. This should not be interpreted to mean that for any claim, the disclosed features that are not claimed for protection are necessary. Instead, the subject matter of the invention may be less than all the features of a specific disclosed embodiment. Therefore, the attached claims are incorporated into the specific embodiments as examples or embodiments, wherein each claim itself exists as a separate embodiment, and it is conceivable that such embodiments can be combined with each other in various combinations or arrangements. The scope of the present invention should be determined with reference to the attached claims and the full range of equivalents given by such claims.

Claims

1. A computer-implemented method for synthesizing a medical image using a trained statistical learning model, the method comprising: receiving medical imaging data acquired using a first imaging modality type; applying the trained statistical learning model to the received medical imaging data to synthesize a medical image corresponding to a different second imaging modality type; as well as providing the synthesized medical image for presentation or for further processing; wherein the trained statistical learning model is established using a similarity determination between training imaging data provided at a model input and synthetic imaging data at a model output without reconstructing an image of the first imaging modality type, the training imaging data corresponding to the first imaging modality type, and the synthetic imaging data corresponding to the second imaging modality type; and wherein the trained statistical learning model is established using a single generator convolutional network and a separate statistical learning model, wherein the separate statistical learning model is established to discriminate between actual imaging data corresponding to the second imaging modality type and the synthetic imaging data; The single generator convolutional network and the single discriminator convolutional network form an adversarial network architecture, and the adversarial network architecture is constructed as follows: Inputting the synthetic imaging data to the single discriminator convolutional network and adjusting the single discriminator convolutional network to increase or maximize the value of a discriminator objective over the distribution of the synthetic imaging data; and The single generator convolutional network is adjusted so that the value of the discriminator objective is reduced or minimized and the similarity is increased.

2. The computer-implemented method of claim 1 , wherein: The trained statistical learning model includes a first generator convolutional network, and the separate statistical learning model includes a first discriminator convolutional network.

3. The computer-implemented method of claim 2, wherein: The trained statistical learning model is established without the need to use a second generator convolutional network and a second discriminator convolutional network, wherein the second generator convolutional network is trained to synthesize a medical image corresponding to the first imaging modality type based on an input image corresponding to the second imaging modality type, and the second discriminator convolutional network is established to discriminate between synthesized imaging data corresponding to the first imaging modality type and actual imaging data corresponding to the first imaging modality type.

4. The computer-implemented method of claim 1 , wherein: The target is represented as the sum of contributions from the adversarial target and the similarity determination.

5. The computer-implemented method of claim 2, wherein: The target includes the following items: 1) an item indicating that an input to the first discriminator convolutional network including actual imaging data is recognized by the first discriminator convolutional network as an expected value of actual imaging data; and 2) an item indicating that an input to the first discriminator convolutional network including a synthetic image from the first generator convolutional network is not recognized by the first discriminator convolutional network as an expected value of actual imaging data.

6. The computer-implemented method of claim 1 , wherein: The similarity determination includes at least one of: mutual information determination; or mutual correlation determination.

7. The computer-implemented method of claim 1 , wherein: The first imaging modality type comprises magnetic resonance (MR) imaging; and Wherein, the second imaging modality type includes computed tomography (CT) imaging.

8. A computer-implemented method according to any one of claims 1 to 7, wherein: Presenting the synthesized medical image includes displaying the image on a MR-LINAC workstation for use in radiation therapy treatment planning or management.

9. A computer-implemented method according to any one of claims 1 to 7, comprising: A radiation therapy treatment protocol is updated or an update to the radiation therapy treatment protocol is received based at least in part on the synthesized medical image.

10. A computer-implemented method for building a trained statistical learning model for synthesizing medical images without requiring paired imaging for training, the method comprising: receiving training medical imaging data corresponding to a first imaging modality type; applying a statistical learning model to the received training medical imaging data to synthesize imaging data corresponding to a different second imaging modality type; using a similarity determination between the training medical imaging data and the synthesized imaging data at the model output to adjust the statistical learning model without reconstructing an image of the first imaging modality type; as well as tuning the statistical learning model using a single generator convolutional network and a separate statistical learning model built to discriminate between the synthetic imaging data and actual imaging data corresponding to the second imaging modality type; The single generator convolutional network and the single discriminator convolutional network form an adversarial network architecture, and the adversarial network architecture is constructed as follows: Inputting the synthetic imaging data to the single discriminator convolutional network and adjusting the single discriminator convolutional network to increase or maximize the value of a discriminator objective over the distribution of the synthetic imaging data; and The single generator convolutional network is adjusted so that the value of the discriminator objective is reduced or minimized and the similarity is increased.

11. The computer-implemented method of claim 10, wherein: The statistical learning model includes a first generator convolutional network, and the separate statistical learning model includes a first discriminator convolutional network.

12. The computer-implemented method of claim 11, wherein: Adjusting the statistical learning model includes using a stochastic gradient descent technique.

13. The computer-implemented method of claim 10, wherein: The target is represented as the sum of contributions from the adversarial target and the similarity determination.

14. The computer-implemented method of claim 11, wherein: The target includes the following items: 1) an item indicating that an input to the first discriminator convolutional network including actual imaging data is recognized by the first discriminator convolutional network as an expected value of actual imaging data; and 2) an item indicating that an input to the first discriminator convolutional network including a synthetic image from the first generator convolutional network is not recognized by the first discriminator convolutional network as an expected value of actual imaging data.

15. The computer-implemented method of claim 10, wherein: The similarity determination includes at least one of: mutual information determination; or mutual correlation determination.

16. A computer-implemented method according to any one of claims 10 to 15, wherein: The first imaging modality type comprises magnetic resonance (MR) imaging; and Wherein, the second imaging modality type includes computed tomography (CT) imaging.

17. A system comprising: processing circuitry comprising at least one processor; as well as A storage medium comprising instructions, which when executed by the at least one processor cause the processor to perform the following operations: receiving medical imaging data acquired using a first imaging modality type; applying the trained statistical learning model to the received medical imaging data to synthesize a medical image corresponding to a different second imaging modality type; as well as providing the synthesized medical image for presentation or for further processing; wherein the trained statistical learning model is established using a similarity determination between training imaging data provided at a model input and synthetic imaging data at a model output without reconstructing an image of the first imaging modality type, the training imaging data corresponding to the first imaging modality type, and the synthetic imaging data corresponding to the second imaging modality type; and wherein the trained statistical learning model is established using a single generator convolutional network and a separate statistical learning model, wherein the separate statistical learning model is established to discriminate between the synthetic imaging data and actual imaging data corresponding to the second imaging modality type; The single generator convolutional network and the single discriminator convolutional network form an adversarial network architecture, and the adversarial network architecture is constructed as follows: Inputting the synthetic imaging data to the single discriminator convolutional network and adjusting the single discriminator convolutional network to increase or maximize the value of a discriminator objective over the distribution of the synthetic imaging data; and The single generator convolutional network is adjusted so that the value of the discriminator objective is reduced or minimized and the similarity is increased.

18. The system of claim 17, wherein: The instructions cause the processor to update a radiation therapy treatment protocol or receive an update to a radiation therapy treatment protocol based at least in part on the synthesized medical image.

Images