Atlas-based segmentation using deep learning

Abstract

The present application discloses atlas-based segmentation using deep learning. A computer-implemented method for operating a trained machine learning model classifier in an atlas-based segmentation process includes: applying the machine learning model classifier to a subject image to generate a classification of respective regions of the subject image, the machine learning model classifier having been trained by data generated by a deep learning model separate from the machine learning model classifier; estimating structural labels for the respective regions of the subject image based on the classification of the respective regions of the subject image; and defining structural labels for the respective regions of the subject image by combining the estimated structural labels with structural labels generated by performing atlas-based segmentation on the subject image; wherein the machine learning model classifier is trained using data generated by the deep learning model to classify a plurality of anatomical structures in the atlas image; and wherein the deep learning model includes a convolutional neural network that generates the data.

Description

[0001] This application is a divisional application of Chinese patent application No. 201980013586.6, filed on August 14, 2020, entitled “Atlas-Based Segmentation Using Deep Learning.” The international filing date of the parent application is February 12, 2019, with international application number PCT / US2019 / 017626.

[0002] Priority claim

[0003] This application claims the benefit of priority to U.S. application serial number 15 / 896,895, filed on February 14, 2018, which is hereby incorporated by reference in its entirety. Technical Field

[0004] Embodiments of the present disclosure generally relate to medical image and artificial intelligence processing techniques. In particular, the present disclosure relates to using deep learning models in image segmentation and structure labeling workflows. Background Art

[0005] In radiotherapy or radiosurgery, treatment planning is typically performed based on medical images of the patient, and requires delineation of the target volume and normal critical organs in the medical images. Therefore, structural segmentation or contouring of various patient anatomical structures in medical images is a prerequisite and crucial step in radiotherapy treatment planning; if performed manually, contouring and segmentation are among the most tedious and time-consuming steps.

[0006] Accurate and automatic computer-based segmentation or delineation of anatomical structures can greatly assist in the design and / or modification of optimal treatment plans. However, accurate and automatic segmentation of medical images remains a challenging task due to the deformation and variability of the shape, size, position, etc. of target areas and critical organs in different patients. Atlas-based automatic segmentation (e.g., such as those produced by Elekta AB, Stockholm, Sweden) software) is one approach that has been used to address this task, as atlas-based segmentation involves applying pre-segmentation to an atlas dataset with structures of interest already identified and labeled.

[0007] Atlas-based automatic segmentation, also known as registration-based automatic segmentation, performs image segmentation by performing atlas registration on a subject's image, followed by label fusion or refinement. The accuracy of the segmentation produced by atlas-based automatic segmentation generally depends on the specific atlas registration method applied, but the accuracy of atlas-based automatic segmentation is also improved by label fusion methods that combine segmentations produced by multiple atlases. In addition, some previous methods have attempted to improve the accuracy of atlas-based automatic segmentation by combining it with machine learning-based segmentation methods. For example, the applicant's prior patent application by Xiao Han, entitled "Method and apparatus for learning-enhanced atlas-based auto-segmentation", published under patent number 9,122,950, relates to a technique for improving the accuracy of atlas-based segmentation using an automatic structure classifier, which is trained using a machine learning algorithm.

[0008] Newer research suggests using deep learning methods to perform segmentation and identify various states from medical images. Deep learning, based on deep convolutional neural networks (CNNs), offers another powerful approach to the problem of medical image segmentation. Compared to existing atlas-based automatic segmentation techniques, deep learning can use much larger training datasets to train and operate structural segmentation models. However, deep learning has some significant drawbacks that hinder its widespread use. Training deep learning models is typically very slow—even taking many days—and is typically performed offline. However, once a deep learning model is trained, applying it to new images can be very fast, typically taking minutes or even seconds. In addition, deep learning models generally work better if they are trained using large amounts of training data, such as hundreds or thousands of images with true segmentations. Although the availability of such training data may be limited, the ability of deep learning models to easily accommodate and respond to large amounts of training data is a key advantage of deep learning methods. Consequently, various methods for performing image segmentation using deep learning CNNs are now being discussed.

[0009] There are other practical limitations that prevent deep learning from being deployed as a primary approach to performing image segmentation. First, the large training datasets typically required to build accurate and useful deep learning models for specific segmentation features are not easy to accumulate or manage. Second, different medical devices can use different imaging protocols and / or different delineation protocols for segmentation; therefore, a model trained using data from one device and manual delineation may not be applicable to data from a different device and may lead to biased segmentation results. Third, training deep learning models typically requires deep technical expertise, so it may be difficult for individual medical users to retrain models on private datasets or adapt deep learning models to their specific needs. For example, a user may need to segment more structures in an image than are available in the pretrained model. Therefore, although deep learning offers a variety of techniques that appear promising for identifying anatomical features in medical imaging, it has not yet been successfully adopted in many real-world settings. Summary of the Invention

[0010] The present disclosure includes processes that incorporate deep learning models and methods into workflows of atlas-based segmentation operations to achieve improved automatic segmentation accuracy and identification of anatomical structures and features. The present disclosure includes multiple illustrative examples related to the use of segmentation and deep learning operations as they relate to a radiation therapy treatment workflow that includes atlas-based automatic segmentation; however, it is apparent that the use of deep learning models and segmentation improvements can be incorporated into other medical imaging workflows for various diagnostic, assessment, and interpretive settings.

[0011] In an example, an implementation of a method for performing atlas-based segmentation using deep learning includes the following operations, the operations comprising: applying a deep learning model to a subject image, the deep learning model being trained to generate deep learning segmentation data that identifies anatomical features in the subject image; registering the atlas image with the subject image, the atlas image being associated with annotation data that identifies anatomical features in the atlas image, such that the registration uses the deep learning segmentation data to improve the registration result between the atlas image and the subject image; generating a mapping atlas by registering the atlas image with the subject image; and using the mapping atlas to identify anatomical features in the subject image.

[0012] Other examples of using deep learning to perform atlas-based segmentation may include: performing registration to improve the registration results between the atlas image and the subject image by applying deep learning segmentation data to determine initial registration estimates or constraints based on anatomical features identified in the subject image; registering multiple atlas images with the subject image to identify the respective locations and boundaries of anatomical features in the subject image by combining results from multiple mapping atlases; performing structural labeling of the multiple anatomical features in the subject image based on the multiple mapping atlases, and generating a structural labeling map for the subject image based on the structural labeling of the multiple anatomical features; and applying the deep learning model to the atlas image to generate additional deep learning segmentation data that identifies anatomical features in the atlas image and improves the registration results of the anatomical features between the atlas image and the subject image. Additionally, in other examples, a deep learning model can be trained based on multiple medical images that classify individual voxels of anatomical features in a segmented label map, wherein the multiple medical images are used to train a deep learning model that includes images from various medical devices, and the various medical devices utilize variations in imaging and contouring protocols to identify anatomical features in the multiple medical images.

[0013] In addition, in an example, an implementation of a method for defining and operating a machine learning classifier for labeling used in an atlas-based segmentation process using deep learning includes the following operations, the operations comprising: applying a deep learning model to an atlas image, the deep learning model being suitable for generating data by analyzing multiple anatomical structures in the atlas image; training a machine learning model classifier using the data generated by applying the deep learning model, so that the machine learning model classifier is trained to classify anatomical structures in the atlas image; applying the trained machine learning model classifier to a subject image to produce a classification of respective regions of the subject image; estimating structural labels for respective regions of the subject image based on the classification of respective regions of the subject image; and defining structural labels for respective regions of the subject image by combining the estimated structural labels with structural labels generated by performing atlas-based segmentation on the subject image.

[0014] Other examples of machine learning classifier training and operation may include: the data generated by applying the deep learning model includes a feature map generated by analyzing the input image in the intermediate convolutional layer of the convolutional neural network; aligning multiple atlas images with the subject image, generating multiple mapping atlases on the subject image based on aligning the multiple atlas images with the subject image, and generating structural labels of the subject image from the multiple mapping atlases; performing label refinement and label fusion on multiple labels indicated from the multiple mapping atlases; training the machine learning model classifier by using the segmentation results generated by applying the deep learning model to the multiple atlas images; training the machine learning model classifier by using the segmentation feature data generated by applying the deep learning model to the multiple atlas images; and generating a label map of the subject image from the structural labels of each region of the subject image, so that the label map identifies each segmentation of the subject image.

[0015] The examples described herein can be implemented in a variety of embodiments. For example, one embodiment includes a computing device comprising processing hardware (e.g., a processor or other processing circuitry) and memory hardware (e.g., a storage device or volatile memory), the memory hardware including instructions implemented thereon such that the instructions, when executed by the processing hardware, cause the computing device to implement, perform, or coordinate electronic operations for these techniques and system configurations. Another embodiment discussed herein includes a computer program product, such as can be implemented by a machine-readable medium or other storage device, that provides instructions for implementing, performing, or coordinating electronic operations for these techniques and system configurations. Another embodiment discussed herein includes a method that can operate on the processing hardware of a computing device to implement, perform, or coordinate electronic operations for these techniques and system configurations.

[0016] In other embodiments, the logic, commands, or instructions for implementing various aspects of the above-described electronic operations may be provided in a distributed or centralized computing system (including any number of form factors regarding computing systems such as desktop or notebook personal computers, mobile devices such as tablet computers, netbooks, and smartphones, client terminals, and server-hosted machine instances. Another embodiment discussed herein includes incorporating the techniques discussed herein into other forms, including other forms of programming logic, hardware configurations, or dedicated components or modules, including devices having various means for performing the functions of such techniques. Various algorithms for implementing the functions of such techniques may include sequences of some or all of the above-described electronic operations or other aspects depicted in the accompanying drawings and the following detailed description.

[0017] The above summary is intended to provide an overview of the subject matter of this patent application. It is not intended to provide an exclusive or exhaustive description of the invention. The detailed description is included to provide more information about this patent application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In the accompanying drawings, which are not necessarily drawn to scale, like reference numerals describe substantially similar components throughout the several views. Like reference numerals with different letter suffixes represent different instances of substantially similar components. The accompanying drawings generally illustrate various embodiments discussed in this document by way of example and not by way of limitation.

[0019] Figure 1 An exemplary radiation therapy system suitable for performing image segmentation processing is shown.

[0020] Figure 2 An exemplary image-guided radiation therapy apparatus is shown.

[0021] Figure 3 An exemplary flow chart illustrating the operation of a deep learning model is shown.

[0022] Figure 4 An exemplary convolutional neural network model for image segmentation is shown.

[0023] Figure 5 An exemplary data flow suitable for use with deep learning segmentation data in the atlas registration process is shown.

[0024] Figure 6 A flowchart illustrating example operations for performing deep learning-assisted atlas-based segmentation is shown.

[0025] Figure 7 An exemplary data flow suitable for use with deep learning segmentation feature data during machine learning model training is shown.

[0026] Figure 8 An exemplary data flow suitable for use with deep learning segmentation feature data during machine learning model classification is shown.

[0027] Figure 9 A flowchart illustrating example operations for performing deep learning-assisted atlas-based segmentation using a machine learning classifier is shown. DETAILED DESCRIPTION

[0028] In the following detailed description, reference is made to the accompanying drawings that form a part thereof and are shown by way of illustration specific to the embodiments shown, and the present invention may be practiced in the embodiments specific to the embodiments shown. These embodiments, also referred to herein as “examples,” are described in sufficient detail to enable those skilled in the art to practice the invention, with the understanding that these embodiments may be combined or other embodiments may be utilized, and that structural, logical, and electrical changes may be made without departing from the scope of the invention. Therefore, the following detailed description is not limiting, and the scope of the invention is defined by the appended claims and their equivalents.

[0029] The present disclosure includes various techniques for improving the operation of image segmentation processes, including performing image segmentation in a manner that provides technical advantages over manual (e.g., human-assisted or human-guided) and conventional atlas-based or artificial intelligence-based methods. These technical advantages include reduced processing time to generate output, improved efficiency of image analysis and visualization operations, and improvements in processing, memory, and network resources associated with performing image segmentation workflow activities. These improved image segmentation workflow activities can be applicable to various medical imaging processing activities used for imaging-based medical treatment and diagnostic actions, as well as accompanying information systems and artificial intelligence environments that manage data to support such treatment and diagnostic actions.

[0030] As discussed further herein, the following use and deployment of deep learning models enables improvements in the accuracy and usefulness of registration results produced by registering atlas images with subject images in an atlas-based segmentation workflow. The deep learning segmentation data provides additional information, in addition to the original atlas and subject image data, to improve the mapping of one or more anatomical features in the atlas registration. Image registration based solely on image (intensity) data is a difficult problem and has many locally suboptimal solutions due to ambiguity and noise in the image data. The segmentation results produced by the deep learning models provide additional information and constraints to help guide the segmentation workflow to an improved solution in both registration computation and feature recognition.

[0031] Figure 1An exemplary radiation therapy system suitable for performing image segmentation processing is shown. The image segmentation processing is performed to enable the radiation therapy system to provide radiation therapy to a patient based on specific aspects of the captured medical imaging data. The radiation therapy system includes an image processing computing system 110 that hosts segmentation processing logic 120. The image processing computing system 110 can be connected to a network (not shown), and such a network can be connected to the Internet. For example, the network can connect the image processing computing system 110 with one or more medical information sources (e.g., a radiology information system (RIS), a medical record system (e.g., an electronic medical record (EMR) / electronic health record (EHR) system), an oncology information system (OIS)), one or more image data sources 150, an image acquisition device 170, and a treatment device 180 (e.g., a radiation therapy device). As an example, the image processing computing system 110 can be configured to perform image segmentation operations as part of an operation to generate and customize a radiation therapy treatment plan to be used by the treatment device 180 by executing instructions or data from the segmentation processing logic 120.

[0032] The image processing computing system 110 may include a processing circuit system 112, a memory 114, a storage device 116, and other hardware and software operable features such as a user interface 140, a communication interface, etc. The storage device 116 may store computer-executable instructions such as an operating system, radiation therapy treatment plans (e.g., original treatment plans, modified treatment plans, etc.), software programs (e.g., radiation therapy treatment planning software; artificial intelligence implementations such as deep learning models, machine learning models, and neural networks), and any other computer-executable instructions to be executed by the processing circuit system 112.

[0033] In an example, the processing circuit system 112 may include a processing device, such as one or more general-purpose processing devices such as a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), etc. More particularly, the processing circuit system 112 may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processor that implements other instruction sets, or a processor that implements a combination of instruction sets. The processing circuit system 112 may also be implemented by one or more special-purpose processing devices such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), a system on a chip (SoC), etc. As will be understood by those skilled in the art, in some examples, the processing circuit system 112 may be a special-purpose processor rather than a general-purpose processor. The processing circuit system 112 may include one or more known processing devices, such as those from Intel TM Pentium manufacturedTM 、Core TM 、Xeon TM or series of microprocessors and those from AMD TM Turion TM , Athlon TM 、Sempron TM , Opteron TM FX TM 、Phenom TM series of microprocessors or any of the various processors manufactured by Sun Microsystems. Processing circuit system 112 may also include processors such as those from Nvidia TM Manufactured series and by Intel TM GMA, Iris TM series or by AMD TM Radeon TM The processing circuit system 112 may also include a graphics processing unit such as a graphics processing unit of the Intel series GPU. TM Xeon Phi manufactured TM A series of accelerated processing units. The disclosed embodiments are not limited to any type of processor that is otherwise configured to meet the computational demands of identifying, analyzing, maintaining, generating, and / or providing large amounts of data or manipulating such data to perform the methods disclosed herein. Additionally, the term "processor" may include more than one processor, such as a multi-core design or multiple processors each having a multi-core design. Processing circuitry 112 may execute sequences of computer program instructions stored in memory 114 and accessed from storage device 116 to perform various operations, processes, and methods that will be described in more detail below.

[0034] Memory 114 may include read-only memory (ROM), phase-change random access memory (PRAM), static random access memory (SRAM), flash memory, random access memory (RAM), dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM), electrically erasable programmable read-only memory (EEPROM), static memory (e.g., flash memory, flash disk, static random access memory), and other types of random access memory, cache memory, registers, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage device, magnetic cassette, other magnetic storage device, or any other non-transitory medium that can be used to store information including images, data, or computer-executable instructions (e.g., stored in any format) that can be accessed by processing circuitry 112 or any other type of computer device. For example, computer program instructions may be accessible to processing circuitry 112, may be read from ROM or any other suitable memory location, and may be loaded into RAM for execution by processing circuitry 112.

[0035] The storage device 116 may constitute a drive unit including a computer-readable medium having stored thereon one or more sets of instructions and data structures (e.g., software) implemented or utilized by any one or more of the methods or functions described herein (including, in various examples, the segmentation processing logic 120 and the user interface 140). During execution of the instructions by the image processing computing system 110, the instructions may also reside, in whole or in part, within the memory 114 and / or within the processing circuit system 112, wherein the memory 114 and the processing circuit system 112 also constitute machine-readable media.

[0036] The memory device 114 and the storage device 116 may constitute a non-transitory computer-readable medium. For example, the memory device 114 and the storage device 116 may store or load instructions for one or more software applications on a computer-readable medium. The software applications stored or loaded using the memory device 114 and the storage device 116 may include, for example, operating systems for general-purpose computer systems and devices controlled by software. The image processing computing system 110 may also operate various software programs including software code for implementing the segmentation processing logic 120 and the user interface 140. In addition, the memory device 114 and the storage device 116 may store or load an entire software application, a portion of a software application, or code or data associated with the software application that can be executed by the processing circuit system 112. In another example, the memory device 114 and the storage device 116 may store, load, and manipulate one or more radiation therapy treatment plans, imaging data, segmentation data, artificial intelligence model data, labeling and mapping data, and the like. It is contemplated that the software programs may be stored not only on storage device 116 and memory 114, but also on removable computer media such as a hard drive, computer disk, CD-ROM, DVD, HD, Blu-ray DVD, USB flash drive, SD card, memory stick, or any other suitable media; such software programs may also be transmitted or received over a network.

[0037] Although not shown, the image processing computing system 110 may include communication interfaces, network interface cards, and communication circuitry. Example communication interfaces may include, for example, a network adapter, a cable connector, a serial connector, a USB connector, a parallel connector, a high-speed data transfer adapter (e.g., fiber optic, USB 3.0, Thunderbolt, etc.), a wireless network adapter (e.g., an IEEE 802.11 / Wi-Fi adapter), a telecommunications adapter (e.g., for communicating with 3G, 4G / LTE, and 5G networks, etc.), and the like. Such communication interfaces may include one or more digital and / or analog communication devices that allow the machine to communicate with other machines and devices, such as remotely located components, via a network. The network may provide the functionality of a local area network (LAN), a wireless network, a cloud computing environment (e.g., software as a service, platform as a service, infrastructure as a service, etc.), a client-server, a wide area network (WAN), and the like. For example, the network may be a LAN or WAN that may include other systems, including additional image processing computing systems or image-based components associated with medical imaging or radiation therapy operations.

[0038] In an example, the image processing computing system 110 can obtain image data 160 from an image data source 150 for hosting on the storage device 116 and the memory 114. In an example, a software program running on the image processing computing system 110 can convert a medical image in one format (e.g., MRI) into another format (e.g., CT), for example, by generating a synthetic image such as a pseudo-CT image. In another example, the software program can register or correlate a patient medical image (e.g., a CT image or an MR image) with a dose distribution (e.g., also represented as an image) of a radiation therapy treatment for that patient, such that corresponding image voxels and dose voxels are properly correlated. In yet another example, the software program can replace a function of the patient image, such as a signed distance function or a processed version of the image that emphasizes certain aspects of the image information. Such a function might emphasize edges or differences in voxel texture or other structural aspects. In another example, the software program can visualize, hide, emphasize, or de-emphasize anatomical features, segmented features, or certain aspects of dose or treatment information within the medical image. The storage device 116 and memory 114 may store and host data used to accomplish these purposes, including image data 160 , patient data, and other data required to create and implement radiation therapy treatment plans and associated fractionation operations.

[0039] The processing circuitry 112 can be communicatively coupled to the memory 114 and the storage device 116, and can be configured to execute computer-executable instructions stored thereon from the memory 114 or the storage device 116. The processing circuitry 112 can execute the instructions to cause a medical image from the image data 160 to be received or acquired in the memory 114 and processed using the segmentation processing logic 120. For example, the image processing computing system 110 can receive the image data 160 from the image acquisition device 170 or the image data source 150 via the communication interface and the network for storage or buffering in the storage device 116. The processing circuitry 112 can also send or update the medical image stored in the memory 114 or the storage device 116 to another database or data store (e.g., a medical device database) via the communication interface. In some examples, one or more systems can form a distributed computing / simulation environment that uses a network to collaboratively perform the embodiments described herein. Additionally, such a network can be connected to the Internet to communicate with servers and clients remotely residing on the Internet.

[0040] In another example, the processing circuit system 112 can utilize a software program (e.g., treatment planning software) and the image data 160 and other patient data to create a radiation therapy treatment plan. In an example, the image data 160 can include atlas information or other information, such as data associated with patient anatomical regions, organs, or volume of interest segmentation data. The patient data can include information such as: (1) functional organ modeling data (e.g., serial versus parallel organs, appropriate dose response models, etc.); (2) radiation dose data (e.g., dose-volume histogram (DVH) information); or (3) other clinical information about the patient and treatment process (e.g., other surgeries, chemotherapy, previous radiation therapy, etc.). In another example, the atlas data provides segmentation or labeling of anatomical features that is specific to a patient, a set of patients, a process or type of treatment, a set of processes or treatments, an image acquisition device, a medical facility, etc.

[0041] In addition, the processing circuit system 112 can utilize the software program to generate intermediate data, such as updated parameters to be used by a neural network model, a machine learning model, an atlas segmentation workflow, or other aspects related to the segmentation of the image data 160. Moreover, using the techniques discussed further herein, such a software program can utilize the segmentation processing logic 120 to implement the segmentation workflow 130. The processing circuit system 112 can then send the executable radiation therapy treatment plan to the treatment device 180 via the communication interface and network, where the radiation therapy plan will be used to treat the patient with radiation via the treatment device, consistent with the results of the segmentation workflow. Other outputs and uses of the software program and the segmentation workflow 130 can occur using the image processing computing system 110.

[0042] As discussed herein (e.g., with reference to Figure 3 and Figure 4 Deep learning processing discussed and referenced Figures 5 to 9 In addition to the segmentation processing discussed above, processing circuitry 112 may execute software programs that invoke segmentation processing logic 120 to implement functions including image segmentation, machine learning, deep learning, neural networks, and other aspects of automated processing and artificial intelligence. For example, processing circuitry 112 may execute software programs that train, delineate, label, or analyze features of medical images; such software, when executed, may train a boundary detector or utilize a shape dictionary.

[0043] In an example, the image data 160 may include one or more MRI images (e.g., 2D MRI, 3D MRI, 2D flow MRI, 4D MRI, 4D volume MRI, 4D imaging MRI, etc.), functional MRI images (e.g., fMRI, DCE-MRI, diffusion MRI), computed tomography (CT) images (e.g., 2D CT, cone-beam CT, 3D CT, 4D CT), ultrasound images (e.g., 2D ultrasound, 3D ultrasound, 4D ultrasound), positron emission tomography (PET) images, X-ray images, fluoroscopic images, radiotherapy portal images, single photon emission computed tomography (SPECT) images, computer-generated synthetic images (e.g., pseudo-CT images), etc. In addition, the image data 160 may also include or be associated with medical image processing data such as training images, ground truth images, contour images, and dose images. In an example, image data 160 may be received from an image acquisition device 170 and stored in one or more of an image data source 150 (e.g., a picture archiving and communication system (PACS), a vendor neutral archive (VNA), a medical record or information system, a data warehouse, etc.). Thus, the image acquisition device 170 may include an MRI imaging device, a CT imaging device, a PET imaging device, an ultrasound imaging device, a fluoroscopic device, a SPECT imaging device, an integrated linear accelerator and MRI imaging device, or other medical imaging device for acquiring medical images of a patient. The image data 160 may be received and stored in any type of data or in any type of format (e.g., in Digital Imaging and Communications in Medicine (DICOM) format) that the image acquisition device 170 and the image processing computing system 110 may use to perform operations consistent with the disclosed embodiments.

[0044] In an example, the image acquisition device 170 can be integrated with the treatment device 180 into a single device (e.g., an MRI device combined with a linear accelerator, also referred to as an "MRI-Linac"). Such an MRI-Linac can be used, for example, to determine the location of a target organ or target tumor within a patient's body, thereby accurately directing radiation therapy to a predetermined target according to a radiation therapy treatment plan. For example, a radiation therapy treatment plan can provide information about a specific radiation dose to be applied to each patient. The radiation therapy treatment plan can also include other radiation therapy information, such as beam angles, dose-volume histogram information, the number of radiation beams to be used during treatment, the dose of each beam, and the like.

[0045] The image processing computing system 110 can communicate with an external database over a network to send / receive a variety of data related to image processing and radiation therapy operations. For example, the external database can include machine data, which is information associated with the treatment device 180, the image acquisition device 170, or other machines related to radiation therapy or medical procedures. The machine data information can include radiation beam size, arc placement, beam on and off durations, machine parameters, segments, multi-leaf collimator (MLC) configuration, gantry speed, MRI pulse sequence, etc. The external database can be a storage device and can be equipped with an appropriate database management software program. In addition, such a database or data source can include multiple devices or systems located in a central or distributed manner.

[0046] The image processing computing system 110 can collect and acquire data via a network and communicate with other systems using one or more communication interfaces that can be communicatively coupled to the processing circuitry 112 and the memory 114. For example, the communication interface can provide a communication connection between the image processing computing system 110 and components of the radiation therapy system (e.g., allowing data to be exchanged with external devices). For example, in some examples, the communication interface can have appropriate interface circuitry with an output device 142 or an input device 144 to connect to a user interface 140, which can be a hardware keyboard, keypad, or touch screen through which a user can enter information into the radiation therapy system.

[0047] As an example, the output device 142 may include a display device that outputs a representation of the user interface 140 and one or more aspects, visualizations, or representations of a medical image. The output device 142 may include one or more display screens that display a medical image, interface information, treatment plan parameters (e.g., contours, doses, beam angles, markers, mappings, etc.), a treatment plan, a target, target positioning, and / or target tracking, or any information relevant to the user. The input device 144 connected to the user interface 140 may be a keyboard, keypad, touch screen, or any type of device by which a user can input information to the radiation therapy system. Alternatively, the output device 142, input device 144, and features of the user interface 140 may be integrated into a device such as a smartphone or tablet computer (e.g., Apple Lenovo Samsung etc.) in a single device.

[0048] Furthermore, any and all components of the radiation therapy system can be implemented as virtual machines (e.g., via a virtualization platform such as VMWare, Hyper-V, etc.). For example, a virtual machine can be software that acts as hardware. Thus, a virtual machine can include at least one or more virtual processors, one or more virtual memories, and one or more virtual communication interfaces that together act as hardware. For example, the image processing computing system 110, the image data source 150, or similar components can be implemented as virtual machines or within a cloud-based virtualization environment.

[0049] The segmentation processing logic 120 or other software program can cause the computing system to communicate with the image data source 150 to read images into the memory 114 and storage device 116, or to store images or associated data from the memory 114 or storage device 116 to and from the image data source 150. For example, the image data source 150 can be configured to store and provide a plurality of images (e.g., 3D MRI, 4D MRI, 2D MRI slice images, CT images, 2D fluoroscopic images, X-ray images, raw data from MR scans or CT scans, Digital Imaging and Communications in Medicine (DICOM) metadata, etc.) from an image set in image data 160 hosted by the image data source 150 and obtained from one or more patients via the image acquisition device 170. The image data source 150 or other database can also store data to be used by the segmentation processing logic 120 when executing the software program that performs the segmentation operation or when creating a radiation therapy treatment plan. In addition, various databases can store data generated by trained deep learning neural networks, image atlases, or machine learning models, including network parameters that constitute the model learned by the network and the resulting predicted data. In connection with performing image segmentation as part of a therapeutic or diagnostic operation, the image processing computing system 110 can therefore acquire and / or receive image data 160 (e.g., 2D MRI slice images, CT images, 2D fluoroscopic images, X-ray images, 3D MRI images, 4D MRI images, etc.) from an image data source 150, an image acquisition device 170, a treatment device 180 (e.g., MRI-Linac), or other information system.

[0050] The image acquisition device 170 can be configured to acquire one or more images of the patient's anatomical structure for an area of ​​interest (e.g., a target organ, a target tumor, or both). Each image, typically a 2D image or slice, can include one or more parameters (e.g., 2D slice thickness, orientation, and position, etc.). In an example, the image acquisition device 170 can acquire 2D slices of any orientation. For example, the orientation of a 2D slice can include a sagittal orientation, a coronal orientation, or an axial orientation. The processing circuit system 112 can adjust one or more parameters, such as the thickness and / or orientation of the 2D slice, to include the target organ and / or target tumor. In an example, the 2D slice can be determined based on information such as a 3D MRI volume. While the patient is receiving radiation therapy treatment, for example, when using the treatment device 180, such 2D slices can be acquired by the image acquisition device 170 in "near real time" (wherein "near real time" means acquiring data in at least a few milliseconds or less).

[0051] The segmentation processing logic 120 in the image processing computing system 110 is depicted as implementing a segmentation workflow 130 having various aspects of segmentation and image processing operations. In the example, the segmentation workflow 130 operated by the segmentation processing logic 120 is coupled to a deep learning segmentation function 132 (e.g., Figure 3 and Figure 4 As shown, segmentation processing is performed by using a deep learning model), atlas registration 134 (e.g., as Figure 5 and 6 As shown, performing atlas-based automatic segmentation enhanced by deep learning operations), machine learning labeling 136 (e.g., as Figures 7 to 9 As shown, machine learning operations for segmentation enhanced by deep learning operations are performed) and label fusion and refinement 138 (e.g., as shown Figures 5 to 9 As shown, the use of the labeling output enhanced by deep learning operations is performed in combination. Other segmentation and image processing functions not specifically depicted can be incorporated into the segmentation workflow 130.

[0052] In treatment planning software such as those manufactured by Elekta AB of Stockholm, Sweden, ), the segmentation processing logic 120 and the segmentation workflow 130 can be used when generating a radiation therapy treatment plan. To generate a radiation therapy treatment plan, the image processing computing system 110 can communicate with an image acquisition device 170 (e.g., a CT device, an MRI device, a PET device, an X-ray device, an ultrasound device, etc.) to capture and access images of the patient and depict the target (e.g., a tumor). In some examples, it may be necessary to depict one or more organs at risk (OARs), such as healthy tissue surrounding or immediately adjacent to a tumor. Therefore, when the OAR is close to the target tumor, segmentation of the OAR can be performed. In addition, if the target tumor is close to the OAR (e.g., the prostate is immediately adjacent to the bladder and rectum), by segmenting the OAR from the tumor, the radiation therapy system can study not only the dose distribution in the target, but also the dose distribution in the OAR.

[0053] To delineate the target organ or tumor from the OAR, medical images (e.g., MRI images, CT images, PET images, fMRI images, X-ray images, ultrasound images, radiation therapy portal images, SPECT images, etc.) of a patient undergoing radiation therapy can be non-invasively acquired by image acquisition device 170 to reveal the internal structure of the body part. Based on the information from the medical images, the 3D structure of the relevant anatomical part can be obtained. Furthermore, during the treatment planning process, a number of parameters can be considered to strike a balance between effective treatment of the target tumor (e.g., ensuring that the target tumor receives a sufficient radiation dose for effective treatment) and low irradiation of the OAR (e.g., ensuring that the OAR receives the lowest possible radiation dose). Other parameters that can be considered include the location of the target organ and tumor, the location of the OAR, and the movement of the target relative to the OAR. For example, the 3D structure can be obtained by outlining the target within each 2D layer or slice of the MRI or CT image, or by outlining the OAR and combining the outlines of each 2D layer or slice. The segmentation can be generated manually (e.g., by a physician, dosimetrist, or healthcare professional using a program such as MONACO™ manufactured by Elekta AB of Stockholm, Sweden) or automatically (e.g., using atlas-based automatic segmentation software such as MONACOT™ manufactured by Elekta AB of Stockholm, Sweden). In some examples, the 2D or 3D structure of the target tumor or OAR can be automatically generated by the treatment planning software using the segmentation processing logic 120.

[0054] After the target tumor and OARs have been located and delineated, a dosimetrist, physician, or medical staff member can determine the dose of radiation to be applied to the target tumor and any maximum dose that can be received by OARs adjacent to the tumor (e.g., left and right parotid glands, optic nerve, eye, lens, inner ear, spinal cord, brainstem, etc.). After the radiation dose has been determined for each anatomical structure (e.g., target tumor, OAR), a process known as inverse planning can be performed to determine one or more treatment planning parameters that will achieve the desired radiation dose distribution. Examples of treatment planning parameters include: volume delineation parameters (e.g., which define the target volume, contour-sensitive structures, etc.), margins around the target tumor and OARs, beam angle selection, collimator settings, and beam opening times. During the inverse planning process, the physician can define dose constraint parameters that set limits on how much radiation the OARs can receive (e.g., full dose for tumor targets, zero dose for any OARs; 95% dose for target tumors; spinal cord, brainstem, and optic nerve structures receiving ≤45 Gy, ≤55 Gy, and <54 Gy, respectively). The results of the inverse planning can constitute a radiotherapy treatment plan that can be stored. Some of these treatment parameters may be interrelated. For example, adjusting one parameter (e.g., the weighting for different objectives, such as increasing the dose to the target tumor) in an attempt to change the treatment plan may affect at least one other parameter, which in turn can lead to the development of a different treatment plan. Therefore, the image processing computing system 110 can generate a customized radiotherapy treatment plan with these parameters so that the treatment device 180 can provide the appropriate radiotherapy treatment to the patient.

[0055] Figure 2 An exemplary image-guided radiation therapy apparatus 202 is shown, comprising a radiation source, such as an X-ray source or a linear accelerator, a couch 216, an imaging detector 214, and a radiation therapy output 204. The radiation therapy apparatus 202 can be configured to emit a radiation beam 208 to provide therapy to a patient. The radiation therapy output 204 can include one or more attenuators or collimators, such as a multi-leaf collimator (MLC).

[0056] As an example, a patient may be positioned in area 212 supported by a treatment couch 216 to be treated according to a radiation therapy treatment plan (e.g., Figure 1The radiation therapy output 204 may be mounted or attached to a gantry 206 or other mechanical support. When the couch 216 is inserted into the treatment area, one or more chassis motors (not shown) may rotate the gantry 206 and the radiation therapy output 204 about the couch 216. In one example, the gantry 206 may rotate continuously about the couch 216 as the couch 216 is inserted into the treatment area. In another example, the gantry 206 may rotate to a predetermined position when the couch 216 is inserted into the treatment area. For example, the gantry 206 may be configured to rotate the therapy output 204 about an axis ("A"). Both the couch 216 and the radiation therapy output 204 may be independently movable to other positions around the patient, for example, in a transverse direction ("T"), in a lateral direction ("L"), or rotated about one or more other axes, such as a transverse axis (denoted as "R"). A controller communicatively connected to one or more actuators (not shown) can control the movement or rotation of the couch 216 to appropriately position the patient in or out of the radiation beam 208 according to the radiation therapy treatment plan. Because both the couch 216 and the gantry 206 can move independently of each other in multiple degrees of freedom, this allows the patient to be positioned so that the radiation beam 208 can be precisely targeted at the tumor.

[0057] Figure 2 The coordinate system shown in FIG, including axes A, T, and L, can have an origin located at an isocenter 210. The isocenter can be defined as the location where the central axis of the radiation therapy beam 208 intersects the origin of the coordinate axes, e.g., to deliver a prescribed radiation dose to a location on or within a patient. Alternatively, the isocenter 210 can be defined as the location where the central axis of the radiation therapy beam 208 intersects the patient for various rotational positions of the radiation therapy output 204 about axis A, as positioned by the gantry 206.

[0058] The gantry 206 may also have an attached imaging detector 214. The imaging detector 214 is preferably located opposite the radiation source (output 204) and, in an example, may be located within the field of the therapy beam 208.

[0059] The imaging detector 214 can be preferably mounted on the gantry 206 relative to the radiation therapy output 204, for example to maintain alignment with the therapy beam 208. As the gantry 206 rotates, the imaging detector 214 rotates about the rotation axis. In an example, the imaging detector 214 can be a flat panel detector (e.g., a direct detector or a scintillator detector). In this manner, the imaging detector 214 can be used to monitor the therapy beam 208, or the imaging detector 214 can be used to image the patient's anatomy, such as for portal imaging. The control circuitry of the radiation therapy device 202 can be integrated within the radiation therapy system or remote from the radiation therapy system.

[0060] In the illustrative example, one or more of the couch 216, therapy output 204, or gantry 206 can be automatically positioned, and the therapy output 204 can establish a therapy beam 208 according to a specified dose for a particular therapy delivery instance. A sequence of therapy deliveries can be specified according to a radiation therapy treatment plan, for example, using one or more different orientations or positions of the gantry 206, couch 216, or therapy output 204. The therapy deliveries can occur sequentially but can intersect at desired therapy sites on or within the patient, such as at an isocenter 210. Thus, a prescribed cumulative dose of radiation therapy can be delivered to the therapy site while reducing or avoiding damage to tissue near the therapy site.

[0061] therefore, Figure 2 Specifically shown is an example of a radiotherapy device 202 that is operable to provide radiotherapy treatment to a patient, the radiotherapy device 202 having a configuration in which a radiotherapy output can rotate about a central axis (e.g., axis "A"). Other radiotherapy output configurations can be used. For example, the radiotherapy output can be mounted to a robotic arm or manipulator having multiple degrees of freedom. In yet another example, the therapy output can be fixed, for example, positioned in an area laterally separated from the patient, and a platform supporting the patient can be used to align the radiotherapy isocenter with a designated target site in the patient's body. In another example, the radiotherapy device can be a combination of a linear accelerator and an image acquisition device. As will be appreciated by one of ordinary skill in the art, in some examples, the image acquisition device can be an MRI, X-ray, CT, CBCT, spiral CT, PET, SPECT, optical tomography, fluorescence imaging, ultrasound imaging, or a radiotherapy portal imaging device, among others.

[0062] Figure 3 An exemplary flow chart for deep learning operations is shown, in which a deep learning model such as a deep convolutional neural network (CNN) can be trained and used to perform segmentation operations. For example, Figure 3 The deep learning model can include Figure 1The deep learning segmentation functionality 132 is provided as part of the segmentation processing logic 120 in the radiation therapy system.

[0063] In an example, the input 304 for deep learning operations may include a defined deep learning model that receives or acquires an initial set of values ​​and training data. The training data may include, for example, hundreds or thousands of images with true segmentation marks, contours, and other identifiers of segmentation features. The deep learning model may be constructed to include an artificial neural network, such as the one described below with reference to Figure 4 The deep CNN model discussed. Deep learning networks can be trained as part of online or offline training methods, and deep learning networks can also be integrated into specific segmentation and radiation therapy use cases (and adjusted or retrained with operating parameters or additional training data depending on the use case). For example, when trained on a series of images, a deep learning network can be used to generate an indication in the form of a classification, probability, or other predicted outcome for new subject images.

[0064] During the training 308 of the deep learning model, a batch of training data can be selected or provided from an existing image dataset. The selected training data can include a set of patient images and corresponding true segmentation labels that identify anatomical structures, features, or characteristics in the patient images. Various algorithms in the deep learning model can be applied to the selected training data and then compared to the expected results (e.g., true segmentation values ​​corresponding to the segmentation labels) to calculate differences that can provide an indication of training errors. These errors can be used during a process called backpropagation to correct errors in the parameters of the deep learning network (e.g., layer node weights and biases), for example to reduce or minimize errors in segmentation value estimates during subsequent trials. The errors can be compared to a predetermined criterion, such as a minimum value that persists for a specified number of training iterations. When the expected results are analyzed for another iteration of deep learning model training, if the errors do not meet the predetermined criterion, backpropagation can be used to update the model parameters of the deep learning model and another batch of training data can be selected from the training dataset. If the error meets a predetermined criterion, training can be terminated, and the trained model can then be deployed during a deep learning prediction phase 312 (including an additional testing or inference phase) to predict segmentation results for images of subjects different from the training data. Thus, the trained model can be utilized to receive and interpret new image data, and the trained model can provide prediction results (e.g., segmentation classification, labeling, mapping, probability, etc.) on the new image data.

[0065] Thus, during training 308 and prediction 312 (deployment) of a deep learning model, multiple parameters in the convolutional layers of the deep learning model may be varied and used to optimize the model output to a desired state. In the context of medical image feature recognition, a fairly large training dataset is required to successfully train the parameters of the model to handle a wide variety of real-world image use cases and produce predictions that are as close to the ground truth as possible. However, due to the large variation in medical images from different patients and imaging sources, many different segmentation approaches and preferences, and inconsistencies and difficulties in training deep learning models based on training data, deep learning may not provide a standalone solution for medical image segmentation. Therefore, as discussed in the following paragraphs, incorporating deep learning data and predictions into various atlas-based automated segmentation processes may provide an effective hybrid approach with significant advantages over existing segmentation methods.

[0066] As previously noted, various atlas-based automatic segmentation methods have been developed to perform outlining and labeling of anatomical structures in radiotherapy treatment planning. Atlas-based automatic segmentation methods map contours in new (subject) images based on previously defined anatomical structures in reference images (particularly atlases). Although some atlas registration methods have been very effective, the shape and size of some organs may vary from patient to patient, and some organs may undergo large-scale deformation at different stages for the same patient. This may reduce the registration accuracy and affect the automatic segmentation performed by atlas-based automatic segmentation methods, or even completely hinder the use of atlas-based methods.

[0067] Incorporating information from deep learning models into various aspects of the segmentation workflow involving atlas-based automatic segmentation methods can provide significant improvements over conventional methods that rely solely on the use of atlases. The use of deep learning models and artificial neural network operations in various aspects of the segmentation workflow also provides advantages over machine learning methods for image segmentation and methods that rely on the use of deep learning alone. As discussed in the examples below, segmentation data generated or predicted from deep learning models can be incorporated into various stages of atlas-based automatic segmentation methods, thereby providing a hybrid approach to segmentation (and applicable machine learning classification) that emphasizes the advantages of deep learning and atlas-based image registration.

[0068] An exemplary deep learning model that can be generated for image segmentation operations includes a convolutional neural network (CNN). CNN is a machine learning algorithm that can be trained through supervised learning. Supervised learning is a branch of machine learning that infers a predictive model based on a training data set. Each individual sample of training data is a pair of samples that includes a data set (e.g., an image) and a desired output value or data set. The supervised learning algorithm analyzes the training data and generates a predictor function. Once the predictor function is obtained through training, the predictor function is able to reasonably predict or estimate the correct output value or data set for a valid input. The predictor function can be formulated based on various machine learning models, algorithms, and / or processes.

[0069] The architecture of a CNN model includes a stack of different layers for transforming input into output. Examples of different layers may include one or more convolutional layers, nonlinear operator layers (e.g., rectified linear unit (ReLu) functions, sigmoid functions, or hyperbolic tangent functions), pooling or subsampling layers, fully connected layers, and / or final loss layers. Each layer may connect one upstream layer and one downstream layer. The input may be considered an input layer, while the output may be considered a final output layer.

[0070] In order to improve the performance and learning ability of the CNN model, the number of different layers can be selectively increased. The number of different intermediate layers from the input layer to the output layer can become very large, thereby increasing the complexity of the architecture of the CNN model. CNN models with a large number of intermediate layers are called deep CNN models. For example, some deep CNN models can include more than 20 to 30 layers, while other deep CNN models can even include more than hundreds of layers. Examples of deep CNN models include AlexNet, VGGNet, GoogLeNet, ResNet, etc.

[0071] The present disclosure utilizes the powerful learning capabilities of CNN models, especially deep CNN models, to segment the anatomical structures of medical images in combination with segmentation and feature labeling workflows. Consistent with the disclosed examples, a trained CNN model can be used to segment medical images to label or classify each voxel of an input 3D image or each pixel of an input 2D image using anatomical structures. Advantageously, the use of a CNN model for image segmentation in embodiments of the present disclosure allows for automatic segmentation of anatomical structures from a large set of training examples without the need for manual feature extraction (which is typically required with traditional machine learning methods). In addition, as described with reference to Figures 5 to 9 As discussed, using data from CNN models can provide significant benefits to atlas-based segmentation and labeling operations, both for image registration and labeling aspects of atlas-based segmentation.

[0072] As used herein, the deep learning model used in the disclosed segmentation methods and workflows may refer to any neural network model developed, adapted, or modified based on the framework of a convolutional neural network. For example, in embodiments of the present disclosure, the deep learning model used for segmentation may selectively include: intermediate layers between the input layer and the output layer, such as one or more deconvolution layers, upsampling layers or uppooling layers, pixel-by-pixel prediction layers, and / or copy and crop operator layers.

[0073] Figure 4 A simplified example of a deep learning model for image segmentation implemented in a CNN model is shown. Figure 4 As shown, the CNN model 410 for image segmentation can receive a stack of adjacent 2D images as input and output a predicted 2D label map of one of the images (e.g., the image in the middle of the stack). Based on feature extraction and labeling, the 2D label map generated from the CNN model can provide a structural label for one, two, or more images in the stack.

[0074] like Figure 4 As shown, CNN model 410 may generally include two components: a first feature extraction component 420 and a second pixel-by-pixel labeling component 430. For example, feature extraction component 420 may extract one or more features from a stack of adjacent 2D images 422. In the following example, anatomical features from a stack of 2D images constituting a 3D dataset are segmented. However, the following segmentation examples and design of CNN model 410 may also be applied to segmenting or classifying individual 2D images or other forms of medical imaging data.

[0075] In this example, feature extraction section 420 uses convolutional neural network 424 to receive a stack of adjacent 2D images 422 as input and output at least one feature vector or matrix representing features of the stack. Pixel-by-pixel labeling section 430 uses the output of feature extraction section 420 to predict a 2D label map 432 for an intermediate image 426 in the stack of adjacent 2D images 422. Pixel-by-pixel labeling section 430 can be performed using any suitable method, such as a slice-based method or a full-map method. For example, using a stack of adjacent 2D images containing relevant structural information for training and as input to CNN model 410 improves the accuracy of the 2D label map 432 predicted by CNN model 410. This further improves the accuracy of the predicted 3D label map for a 3D image constructed from the 2D label maps predicted for each image slice of the 3D image.

[0076] In an example, features may be identified in a CNN based on spatial correlation relationships between anatomical structures shown in the stack of adjacent 2D images along an axis orthogonal to the anatomical plane of the 2D images. As a non-limiting example, the shape and type of an anatomical structure represented by a first set of pixels in a first image in the stack may also be represented by a second set of pixels in a second image adjacent to the first image. The reason is that the spatial proximity of the first image and the second image along an axis orthogonal to the anatomical plane allows for a certain correlation or continuity of the anatomical structures shown in these images. Therefore, the shape, size and / or type of an anatomical structure in one image may provide information about the shape, size and / or type of an anatomical structure in another adjacent image along the same plane.

[0077] As another non-limiting example, when the stack of adjacent 2D images includes three consecutive images, such as a first image slice, a second image slice, and a third image slice stacked in sequence, an anatomical structure may be shown in a first pixel set in the first image slice of the stack and a third pixel set in the third image slice of the stack, but the anatomical structure may not be shown in a corresponding second pixel set (e.g., pixels having a spatial position similar to the spatial position of the first pixel set and / or the third pixel set) of a second image slice located between and adjacent to the first and third image slices. In this case, the corresponding pixels in the second image slice may be incorrectly labeled. This discontinuity of anatomical structure in the stack of three adjacent 2D image slices can be used as relevant structural information for training the CNN model 410.

[0078] As another non-limiting example, in a stack of three adjacent 2D images, such as a first image slice, a second image slice, and a third image slice stacked sequentially, a first set of pixels in the first image slice and a third set of pixels in the third image slice of the stack may both indicate background, but a corresponding second set of pixels in a second image slice located between and adjacent to the first and third image slices may indicate anatomical structure. The corresponding pixels in the second image slice may be affected by noise that produces false positive signals. This discontinuity in the background in the stack of three adjacent 2D image slices can also be used as relevant structural information for training the CNN model 410.

[0079] Different types of correlated structural information can be selectively used based on various factors, such as the number of adjacent images in the stack; the type, shape, size, location, and / or number of anatomical structures to be segmented; and / or the imaging modality used to acquire the images. Using such correlated structural information from several stacks of adjacent 2D images obtained from a 3D image improves the accuracy of segmenting the 3D image or generating a 3D label map.

[0080] In some examples, the convolutional neural network 424 of the CNN model 410 includes an input layer, for example, a stack of adjacent 2D images 422. Because a stack of adjacent 2D images is used as input, the input layer has a volume whose spatial dimensions are determined by the width and height of the 2D images, and whose depth is determined by the number of images in the stack. As described herein, the depth of the input layer of the CNN model 410 can be adjusted as desired to match the number of images in the input stack of adjacent 2D images 422.

[0081] In some embodiments, the convolutional neural network 424 of the CNN model 410 includes one or more convolutional layers 428. Each convolutional layer 428 can have multiple parameters, such as a width ("W") and a height ("H") determined by the upper input layer (e.g., the size of the input to the convolutional layer 428), as well as the number of filters or kernels in the layer ("N") and their sizes. The number of filters can be referred to as the depth of the convolutional layer. Therefore, each convolutional layer 428 can be described in terms of a 3D volume. The input to each convolutional layer 428 is convolved with a filter across its width and height, and a 2D activation map or feature map corresponding to the filter is produced. Convolution is performed for all filters of each convolutional layer, and the resulting activation maps or feature maps are stacked along the depth dimension to generate a 3D output. The output of the previous convolutional layer can be used as the input to the next convolutional layer.

[0082] In some embodiments, the convolutional neural network 424 of the CNN model 410 includes one or more pooling layers (not shown). A pooling layer can be added between two consecutive convolutional layers 428 in the CNN model 410. The pooling layer operates independently on each depth slice of the input (e.g., the activation map or feature map from the previous convolutional layer), and the pooling layer reduces its spatial dimension by performing a form of nonlinear downsampling. In addition, information from non-adjacent layers can "skip" intermediate layers and can be aggregated with other inputs in the pooling layer. In an example, the function of the pooling layer can include gradually reducing the spatial dimension of the extracted activation map or feature map to reduce the number of parameters and calculations in the network and control overfitting. The number and placement of pooling layers can be determined based on various factors, for example, the design of the convolutional network architecture, the size of the input, the size of the convolutional layer 428, or the application of the CNN model 410.

[0083] Various nonlinear functions can be used to implement the pooling layer. For example, max pooling can be used. Max pooling can divide the input image slice into a set of overlapping or non-overlapping sub-regions with a predetermined span. For each sub-region, max pooling outputs the maximum value in the corresponding sub-region within the partition. This effectively downsamples each slice of the input along both its width and its height, while the depth dimension remains unchanged. Other suitable functions such as average pooling or even L2-norm pooling can be used to implement the pooling layer.

[0084] In various embodiments, the CNN model 410 can optionally include one or more additional layers in its convolutional neural network 424. As a non-limiting example, a rectified linear unit (ReLu) layer (not shown) or a parameterized ReLU (PReLU) (not shown) can be optionally added after the convolution layer to generate an intermediate activation map or feature map. For example, the ReLu layer can increase the nonlinear characteristics of the predictor function and the entire CNN model 410 without affecting the corresponding dimensions of the convolution layer 428. In addition, the ReLu layer can reduce or avoid saturation during the backpropagation training process.

[0085] As another non-limiting example, one or more fully connected layers 429 may be added after the convolutional layers and / or pooling layers. A fully connected layer is fully connected to all activation maps or feature maps of the previous layer. For example, a fully connected layer may take the output of the last convolutional layer or the last pooling layer as input in the form of a vector and perform high-level determinations and output a feature vector arranged along the depth dimension. The output vector may be referred to as an output layer. This vector may contain information about the anatomical structure in a stack of images 422 that are input to the CNN model 410. In addition, information from the output layer extracted from the 2D imaging slice according to a 2D or "2.5D" CNN model may be used to identify sub-regions of the 3D imaging data. This output data from the CNN model 410 may also be used in conjunction with a 3D CNN applied to the sub-regions.

[0086] In the second part of CNN model 410, pixel-by-pixel labeling can be performed using one or more features extracted by convolutional neural network 424 as input to generate a predicted 2D label map 432. The 2D label map can provide structural labels for intermediate images in a stack of adjacent 2D images. In an example, the 2D label map can be used to automatically determine a sub-region of 3D imaging to which a second 3D CNN model can be applied (e.g., in a cascaded or chained manner). A patch-based approach can be used to predict a 2D label map 432 for an intermediate image 426 in the input stack of adjacent 2D images 422. Each image in the stack of adjacent 2D images can similarly be divided into overlapping or non-overlapping rectangular patches, each having a center pixel. This generates a stack of adjacent 2D image patches. The stack of 2D image patches can be used as both training data and input to CNN model 410. The patches can be designed so that the center pixels of the patches, taken together, substantially constitute the entire 2D image. CNN model 410 can classify the center pixel of each intermediate patch in the stack, for example, predicting the anatomical structure represented by the center pixel. For example, the CNN model 410 can predict a feature vector for the center pixel of the middle slice in the stack, thereby allowing the anatomical structure of the center pixel to be classified. This classification is repeated until all center pixels of the middle slices in all stacks of adjacent 2D image slices are classified or labeled, thereby achieving the segmentation of the middle image in the stack of adjacent 2D images. For example, in a slice-based approach, when all center pixels constituting the entire middle image 426 are classified, pixel-by-pixel labeling of the middle image 426 in the input stack of adjacent 2D images 422 can be performed.

[0087] In the patch-based approach described above, pixel-by-pixel labeling of the intermediate image 426 in the input stack of adjacent 2D images 422 is performed when all center pixels constituting the entire intermediate image 426 are classified.

[0088] In another example, a full mapping method can be used to predict a 2D label map 432 for an intermediate image 426 in a stack of input adjacent 2D images 422. In this case, the 2D label map 432 for the intermediate image 426 is generated as the output of the CNN model 410 based on the stack of input adjacent 2D images 422. The convolutional neural network 424 in the CNN model 410 is used to extract an activation map or feature map as output, which is received by a pixel-by-pixel labeling structure including one or more operation layers to predict the 2D label map. In this case, the last layer of the convolutional neural network 424 can be a convolution layer that outputs the activation map or feature map.

[0089] As a non-limiting example, a pixel-by-pixel prediction layer (not shown) can be added to the CNN model 410 to perform pixel-by-pixel labeling. The pixel-by-pixel prediction layer converts the coarse output feature map (e.g., feature vector) of the convolutional neural network 424 into a dense (e.g., providing more information per pixel) predicted pixel-by-pixel 2D label map 432 of the intermediate image 426 in the input stack of adjacent 2D images 422. Various functions can be used to implement the pixel-by-pixel prediction layer, such as backward upsampling or unpooling (e.g., bilinear or nonlinear interpolation) and backward convolution (deconvolution).

[0090] As another non-limiting example, a deconvolution network 434 can be used as part of the CNN model 410 to perform pixel-by-pixel labeling. The deconvolution network 434 can be a mirrored version of the convolutional neural network 424 of the CNN model 410. In contrast to the convolutional neural network 424, which gradually reduces the spatial dimensions of the extracted activation map or feature map, the deconvolution network 434 increases the intermediate activation map or feature map by using selected deconvolution layers 436 and / or depooling layers (not shown). Depooling layers (e.g., upsampling layers) can be used to put pixels in the feature map back to their previous or original pool positions, thereby generating an increased but sparse activation map or feature map. Deconvolution layers can be used to associate a single pixel of an input activation map or feature map with multiple output pixels, thereby increasing and increasing the density of the activation map or feature map. The deconvolution network 434 can be trained and used together with the convolutional neural network 424 to predict a 2D label map.

[0091] As another non-limiting example, a loss layer (not shown) can be included in the CNN model 410. The loss layer can be the last layer in the convolutional neural network 434 or the CNN model 410. During training of the CNN model 410, the loss layer can determine how the network training penalizes the deviation between the predicted 2D label map and the 2D true label map. The loss layer can be implemented by various suitable loss functions. For example, a cross entropy loss function can be used as the final loss layer of the CNN model 410.

[0092] Consistent with the embodiments of the present disclosure, the image segmentation method, system, apparatus and / or process based on the above-mentioned CNN model includes two stages: a training stage, which uses a training dataset including images labeled with different anatomical structures for each voxel to "train" or "learn" the CNN model; and a segmentation stage, which uses the trained CNN model to predict the anatomical structure of each voxel of the input 3D image (or pixel of the input 2D image) or label each voxel of the input 3D medical image (or pixel of the input 2D image) for the anatomical structure. Figure 4 The general structure of the convolutional neural network shown is also applicable to 3D models, where a set of 3D images is provided instead of a stack of 2D images. The following describes in detail image segmentation methods, systems, devices, and / or processes for incorporating these CNN models and other types of deep learning models into atlas-based segmentation workflows. Other variations of the types of deep learning models and other neural network processing methods can also be implemented using this technology.

[0093] Figure 5 An exemplary data flow 500 suitable for use with deep learning segmentation data in an atlas registration process is shown. As shown in the data flow, the atlas registration process is used together with multiple atlases 521, 522, 523 (atlases 1, 2, ..., N) and associated atlas metadata to implement various aspects of an automatic segmentation process for a subject image 515. Specifically, the data flow 500 generates structural labels for various pixels or voxels of the subject image 515 based on 1 to N mapping atlases (atlases 561, 562, 563) (operation 570); such structural labels can be generated as a structural labeling map of the subject image (result 580). As discussed below, the data flow 500 can be considered as an atlas-based automatic segmentation process modified to incorporate the data results of a deep learning model.

[0094] In an example, an atlas-based automatic segmentation process is used to segment a subject image using one or more already segmented images (e.g., from a previously treated patient). These already segmented images and their annotations (e.g., metadata indicating a structural landmark map, structural surface, or other illustration) are referred to as an atlas. In a process known as image registration, after aligning the new subject image with the atlas image via image matching, the structural landmarks defined in the atlas are mapped to the new subject image using a calculated image transformation, which then produces structural landmarks and segmentation results for the subject image. The accuracy of the atlas-based automatic segmentation process can be improved by using multiple atlases. For example, when performing radiation therapy treatment at different stages or periods, atlas segmentation is often updated in parts; combining multiple atlases generated from these different stages or periods can make the entire set of atlases very accurate for a particular patient. Furthermore, applying and combining multiple atlases to a new subject image can be performed very quickly, but if there are significant changes in the subject image, applying and combining multiple atlases to a new subject image often comes at the expense of accuracy and precision.

[0095] The accuracy of the atlas-based automatic segmentation process depends largely on the accuracy of the atlas registration. However, because the image information may be so ambiguous that different structures with similar or overlapping intensity distributions cannot be displayed, registering the atlas with the subject image can be a difficult problem when considering image data. The combination of deep learning results in the following examples provides other ways to improve the accuracy and precision of applying atlases in atlas registration. For example, a deep learning model trained on a large training data set can provide very accurate segmentation results for some of the same structures processed during atlas registration. Therefore, in data flow 500, segmentation data 544 or other results from applying the deep learning model (operation 534) can be used to guide the registration of the image with the corresponding atlas image (in image registration operations 551, 552, 553) by structurally constrained deformable registration or similar registration techniques.

[0096] Figure 5 The data flow in FIG. 5 specifically depicts the receipt of a subject image and an atlas (operation 510), which corresponds to a plurality of atlas images and metadata annotations 521, 522, 523 and a subject image 515. In one branch of the data flow 500, a deep learning model that has been trained to recognize anatomical structures or features can be applied (operation 534) to generate segmentation data (result 544). In the example, as shown in FIG. Figure 3 and Figure 4 As discussed with respect to deep learning methods, a deep learning model may constitute a model trained to perform segmentation of 2D or 3D medical imaging data.

[0097] As shown, the segmentation results (predictions or outputs) of the deep learning model are incorporated into various aspects of atlas image registration (in operations 551, 552, 553). As in the conventional atlas workflow, atlas image registration is configured to perform atlas image registration based on atlas image data and associated atlas metadata (e.g., from atlas 521, 522, 523). However, in data flow 500, the results of the deep learning segmentation model used on the subject image (results 544) (and in some examples, the results of the deep learning model used on the atlas image) are used to assist atlas image registration (in image registration operations 551, 552, 553).

[0098] Application of the deep learning segmentation model can produce segmentation data 544 that identifies a subset of anatomical structures or features in the subject image 515. Such identified structures or features can be used to "arrange" or "guide" the registration performed on the atlas image and the subject image. In an example, image registration is performed on the subject image 515, and the features identified by deep learning affect the application of image registration operations with each of 1 to N atlases (operations 551, 552, 553). For example, the features identified by deep learning can produce initial registration positions, constraints, or a mapping of a particular atlas image or atlas annotation to the subject image.

[0099] In another (optional) example, deep learning segmentation data can be generated from each atlas image (data 521, 522, 523) and analyzed in another branch of the workflow before image registration. For example, image data from each atlas image (data 521, 522, 523) can be applied to a corresponding use of a deep learning model (operations 531, 532, 533) to produce segmentation data (results 541, 542, 543). This segmentation data can be used to further assist in image registration and classification or mapping of structures to subject images.

[0100] The use of multiple atlas images and image registration can assist the atlas-based automatic segmentation process by allowing mapping variations among 1 to N atlases. Thus, in an example, multiple mapping atlases are generated from the image registration (operations 551, 552, 553) (results 561, 562, 563). The data from the mapping atlases can be combined to generate structural labels on corresponding regions and portions of the image (operation 570). The structural labels generated from the multiple mapping atlases can then be used to generate a structural labeling map for the subject image (result 580). For example, the structural labeling map can provide indications of various 2D pixel or 3D voxel regions that are classified as structures of interest or structures of no interest. In other examples, the data stream 500 can be modified for use with a single atlas or for evaluating multiple subject images using a deep learning model.

[0101] As an example scenario, the deep learning segmentation results can be used directly to compute an initial registration solution (e.g., a linear registration) to provide an improved starting point for computing the final atlas-subject image registration for various anatomical organs or structures. For example, consider a scenario in which a deep learning model is a deep learning segmentation model that is trained on a large dataset to recognize the contours of a prostate. The deep learning segmentation model can be used to automatically segment the prostate from both an atlas image and a new subject image. For example, the prostate segmentation results from the deep learning model can be used to compute a linear transformation that aligns the atlas's prostate with the subject's prostate. In this way, the deep learning model provides a starting point for improving the accuracy of the final image registration results of an atlas that utilizes nonlinear registration or deformable registration.

[0102] Thus, in data flow 500, the deep learning model (applied in operation 534 or optionally in operations 531, 532, 533) affects the final atlas-based automatic segmentation result only through its impact on the atlas registration step. This addresses two significant limitations of using only deep learning or other machine learning models to produce the final segmentation result. First, especially because deep learning models are typically pre-trained at a previous time point using a large training data set, the deep learning model may not have all the structures required for the new image. The training data can be used to pre-train the model on a subset of structures such as the prostate / bladder / rectum, while a new user or medical device may need to segment other structures such as neurovascular bundles. Using this technology, the deep learning model can be used to provide very accurate segmentation of a specific organ or subset of organs, while the atlas is used to segment or estimate the remaining structures. Second, the deep learning model can be trained using data from different clinics whose contouring protocols are different from the new image data to be segmented. Using this technique, deep learning models can be applied to segment structures in the model on both the atlas and new subject images to ensure that the segmentation results for these structures are consistent between the atlas and the subject.

[0103] Therefore, deep learning segmentation results can be used to improve atlas registration in the atlas-based automatic segmentation data stream 500 or other variations of atlas-based segmentation. However, the final segmentation result of the structure in the subject image can still be obtained by the conventional atlas-structure deformation and label fusion / refinement process defined for atlas-based segmentation. To reiterate, the deep learning-assisted image registration results can be viewed as "deforming" or adapting the atlas structure to the new subject image. Since such atlases in the atlas-based automatic segmentation data stream are typically specific to the user or medical device, the desired contouring protocol and other specific characteristics of the atlas may still be used and consistent with the new segmentation task.

[0104] Figure 6 A process flow 600 of exemplary operations for performing deep learning-assisted atlas-based segmentation is shown. The process flow 600 is shown from the perspective of an image processing system that receives and processes data using the results of deep learning incorporated into an atlas-based automatic segmentation process. However, the corresponding operations may be performed by other devices or systems.

[0105] Process flow 600 depicts optional prerequisites for a deep learning assisted atlas-based segmentation workflow that may be performed at an earlier time by another entity, user, or medical device or in another setting. These prerequisites are depicted as including selecting and training a deep learning structure segmentation model using a training dataset (operation 610) and defining and selecting one or more atlas models for structure segmentation (operation 620). For example, one may select a structure segmentation model based on the atlas model described above. Figure 3 and Figure 4 A deep learning structure segmentation model can be trained or constructed using aspects of deep learning and CNNs as mentioned in

[15] . Atlas models can be designed or constructed based on manually or automatically segmented images and annotations (e.g., structure labels or structure surfaces) on such segmented images; in some examples, atlas models can be created in conjunction with a radiation therapy treatment workflow.

[0106] Process flow 600 continues with segmentation operations performed on a medical image of a subject by applying a deep learning model. These operations include: obtaining an image of the subject for segmentation processing (operation 630); and performing a branch of the segmentation workflow using the deep learning segmentation model to identify one or more structures in the image of the subject (operation 640). For example, the deep learning segmentation model can generate a classification of pixels or voxels, regions of pixels or voxels, or similar image portions, or identification of structures, features, or other identifiers.

[0107] Process flow 600 continues with segmentation operations performed on a medical image of a subject using deep learning enhanced atlas registration techniques. These operations include: obtaining an atlas image and atlas data (e.g., annotations or other metadata) from an atlas model (operation 650); and applying the segmentation data of the subject image from the deep learning model to perform or assist in atlas image-subject image registration (operation 670). Such atlas registration may involve fitting or deforming the atlas image to the subject to generate a segmentation result (e.g., a structural landmark or structural contours / surfaces) on the subject image. In another (optional) example, additional segmentation data used in image registration is generated by identifying one or more structures or features in the atlas image using a deep learning segmentation model (operation 660). Although only one atlas is depicted in certain elements within process flow 600, it should be understood that the segmentation workflow can be performed on multiple atlases (applying operations 650, 660, 670 to one or more atlases) in parallel or sequentially.

[0108] Process flow 600 continues with operations to perform labeling (including label fusion and refinement) on the subject image (operation 680). The labeling may include: combining registered images from multiple atlases, combining segmentation and labeling of features from another artificial intelligence (e.g., machine learning) algorithm, etc. Label fusion may be combined with the deformed atlas segmentation (structural label map or structural contour / surface) to produce a structural segmentation of the subject image. Figures 7 to 9 Other examples of label fusion and refinement using data generated by a trained machine learning model are discussed below. Finally, process flow 600 concludes by providing the results of the segmentation of the subject's image. This provision can take the form of generating metadata or image data annotations, outputting the results in a graphical user interface, defining or associating a classification of one or more anatomical structures of interest, generating a label map or structural label estimates, generating contours or surfaces for each structure, storing or transmitting an indication of the segmentation, modifying the treatment plan, defining treatment areas or areas to be excluded from treatment (including delineation of OARs), etc.

[0109] In another example, an atlas-based automatic segmentation and labeling method may include the use of a machine learning method enhanced by deep learning. For example, a random forest (RF), support vector machine (SVM), boosted tree (BT) or similar classifier model can be used to train a pixel- or voxel-based structure classifier that can improve the segmentation accuracy of the atlas-based automatic segmentation. However, using machine learning methods with previous atlas-based automatic segmentation and labeling methods typically involves the use of manually specified image features that are used as input to the machine learning model to predict a structural label for each image pixel or voxel. In the following non-limiting example, a deep learning model result may be provided as an additional input to train or refine applicable image features of the machine learning classifier. Such a deep learning model result may include a deep learning segmentation result (e.g., the final result when the deep learning model is applied), or alternatively, such a deep learning model result may include an intermediate result generated from the operation of the deep learning model.

[0110] In the example, a deep learning segmentation model using CNN has the ability to automatically learn a hierarchy of image features from training images. Therefore, when a CNN model is applied to a new subject image to calculate a segmentation result, the model involves first extracting a sequence of image features and then converting these features into a segmentation map or segmentation probability. The output of each convolutional layer of the CNN model is often referred to as a "feature map". In the example, these feature maps or similar outputs from layers within a CNN or other deep learning model can be extracted as features for training a machine learning classifier for labeling images. For example, Figure 4 As shown, the deep learning model can generate many immediate outputs (feature maps) from the intermediate layers of the CNN model when analyzing the subject's image. Each feature map is a multi-channel image that can be used as additional input to train the machine learning model.

[0111] Thus, in an example, the deep learning model segmentation results can be used as additional input to an online or offline machine learning model (e.g., BT, RF, or SVM or other machine learning classifiers) to assist or supplement the atlas-based automatic segmentation workflow (e.g., data flow 500 or process flow 600). For example, the deep learning segmentation label map can be used as an additional channel to the original image data and used to train a multi-channel image pixel or voxel classification model (e.g., BT, RF, or SVM or other machine learning classifiers).

[0112] In another example, a transform of the deep learning segmentation labeling map can be calculated and used as another input channel to extract features for a machine learning classifier model. For example, such features can be used to calculate a distance transform of the segmentation map; in many settings, the distance to specific structures can provide valuable information (e.g., the distance to the prostate region can help determine whether a voxel belongs to the bladder).

[0113] Figure 7 An exemplary data flow 700 suitable for use with deep learning segmentation feature data during machine learning model training is shown. Data flow 700 provides a simplified representation of some of the operations performed in data flow 500, including receiving a subject image 715 and 1 to N atlases 721, 722, 723 (operation 710) and performing atlas registration (operations 731, 732, 733) on the 1 to N atlases 721, 722, 723 to produce mapped atlases 761, 762, 763. However, the atlas registration operation is optional; the machine learning model can be trained solely based on the raw atlas data 721, 722, 723 and the output of applying the deep learning model to each atlas image as discussed below. Additionally, although not depicted, the subject image 715 can be trained by applying a deep learning segmentation model (e.g., as Figure 5 ) to assist atlas registration (operations 731, 732, 733).

[0114] However, data flow 700 more specifically illustrates the results of training a machine learning structure classifier (operation 780) using atlas image data to produce a trained machine learning model 790 that can be used for segmentation labeling and feature representation. Figure 8 Incorporating this trained machine learning model into the segmentation process is described in more detail in the data flow of FIG. ). In data flow 700, machine learning model 790 is shown as being trained based on atlas image segmentation results 750 produced by a deep learning model analyzing atlas images; in optional examples, machine learning model 790 is shown as being additionally trained based on mapped atlas feature data 765 or atlas image segmentation feature data 770.

[0115] Specifically, in addition to the sequence of segmentation features of the atlas images from the various layers of the deep learning model (data 770), the deep learning model 740 also analyzes each atlas image 721, 722, 723 to extract segmentation results (data 750). The segmentation data 750, 770 are provided as training input to the machine learning model classifier and are used in the training phase (operation 780). For each original atlas image, the ground truth can be determined from the atlas data itself; therefore, the feature information of the original atlas image or the mapped atlas image can be used to train the machine learning classifier that produces the classification. Therefore, compared to conventional techniques for training machine learning classifiers from manual feature definitions based on gradients, lines, or textures, the output data from the applied deep learning model automatically provides a feature set related to the anatomical structure to be classified.

[0116] Other aspects of the training data used in the machine learning training (e.g., in a previous machine learning-assisted atlas-based segmentation process) can optionally be incorporated into or used as training inputs to the machine learning model classifier. For example, feature data (data 765) provided from the various mapping atlases 761, 762, 763 generated by registering the image with the subject image 715 can be combined with the features used to train the structure classifier (operation 780) during the training phase, and can supplement or replace the features used to train the structure classifier (operation 780) during the training phase. In a similar manner, metadata or other feature data provided from the original atlases 721, 722, 723 can be combined with the features used to train the structure classifier (operation 780) during the training phase, and can supplement or replace the features used to train the structure classifier (operation 780) during the training phase.

[0117] Figure 8 An exemplary data flow 800 suitable for use with deep learning segmentation feature data in a machine learning model classification process is shown. The data flow 800 specifically indicates the use of a machine learning model 850 trained from an atlas or deep learning segmentation feature data. For example, Figure 7 The machine learning model 850 is trained in the same manner as the machine learning model 790 shown.

[0118] Data flow 800 also provides a simplified representation of some of the operations performed in data flow 500, including receiving a subject image 815 and a plurality of atlases 820 and performing atlas registration (operation 830) on the plurality of atlases 820 and the subject image 815 to produce atlas-based structural labels 840. Although not depicted, the atlas-based structural labels can be generated by applying a deep learning segmentation model (e.g., Figure 5 ) to assist in atlas registration (operation 830).

[0119] The data flow 800 also illustrates combining (e.g., fusing) the structure labels (e.g., labels 840) generated from the atlas-based segmentation operation on the subject image 815 with the structure labels generated by applying the machine learning model 850 to the subject image 815. Specifically, a branch of the data flow 800 illustrates applying the trained structure classifier to the subject image 815 (operation 860) to generate a classifier-based structure label estimate 865 for the subject image. In an optional example, the application of the trained structure classifier (operation 860) may also utilize the label data 845 generated from the atlas-based segmentation operation or the atlas-based structure labeling.

[0120] The classifier structure label estimate 865 generated from the machine learning structure classifier can be combined with the atlas-based structure label 840 (operation 870) and used to generate a structure label map 880 for the subject's image. Any number of label fusion or combination techniques can be used to combine the segmentation labeling results. Thus, the machine learning enhanced structure label map 880 can be used instead of the structure label map generated from the subject's image. Figure 5 The data stream 500 generates a structural markup graph 580 .

[0121] Figure 9 A process flow 900 of exemplary operations for performing deep learning-assisted atlas-based segmentation using a machine learning classifier is shown. The process flow 900 is also shown from the perspective of an image processing system that trains and utilizes a machine learning structure classifier using the results of deep learning incorporated into an atlas-based automatic segmentation process. However, the corresponding operations can be performed by other devices or systems (including in an offline training setting outside of an atlas-based segmentation workflow).

[0122] Process flow 900 depicts the generation of segmentation feature data for an atlas image using a deep learning segmentation model (operation 910). Such segmentation feature data may include a sequence of image features extracted by the CNN model and / or a final CNN segmentation map or segmentation probability generated from the CNN model. The segmentation data is used to train a machine learning structure classifier—e.g., training a machine learning structure classifier based on the deep learning segmentation feature data (e.g., feature map) (operation 920) and training a machine learning structure classifier based on the deep learning segmentation result data (operation 930)—to produce the machine learning classifier models discussed above (e.g., RF, SVM, BT).

[0123] The trained structure classifier is applied to the subject image (operation 940), and the trained structure classifier is used to generate an estimate of the structure label using the structure classifier (operation 950). Outputs from the machine learning model may include: classification labels; probabilities; or other forms of classification data indicating the structure, characteristics, or other indications of individual pixels, voxels, or regions of the subject image (or set of subject images). The structure label information generated from the machine learning model can be combined with the information from the atlas-based automatic segmentation process (including reference Figure 5 and Figure 6

[00106] The process flow then combines (e.g., fuses, combines, or unites) the structural labeling information from the image of the subject using the deep learning-assisted atlas-based segmentation process discussed above (operation 960). (In other examples, the structural labeling information can be combined with information from a conventional atlas-based segmentation process that does not involve deep learning.) Finally, the process flow concludes by generating a label map for the subject image (operation 970). As discussed above, the output of the label map can be provided with reference to the output of the segmentation results (e.g., using operation 690).

[0124] As previously discussed, various electronic computing systems or devices can implement one or more of the methods or functional operations discussed herein. In one or more embodiments, the image processing computing system 110 can be configured, adapted, or used to: control or operate the image-guided radiation therapy device 202; perform or implement deep learning training or prediction operations 308, 312; operate the CNN model 410; perform or implement data flows 500, 700, 800; perform or implement the operations of the flowcharts 600, 900; or perform any one or more of the other methods discussed herein (e.g., as part of the segmentation processing logic 120 and the segmentation workflow 130). In various embodiments, such an electronic computing system or device operates as a standalone device or can be connected (e.g., networked) to other machines. For example, such a computing system or device can operate in the capacity of a server or client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Features of computing system or device 110 may be implemented by a personal computer (PC), tablet PC, personal digital assistant (PDA), cellular phone, network appliance, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be performed by the machine.

[0125] As also described above, the functionality discussed above can be implemented by instructions, logic, or other information storage on a machine-readable medium. Although a machine-readable medium may have been described in various examples with reference to a single medium, the term "machine-readable medium" may include a single medium or multiple media (e.g., a centralized or distributed database and / or associated cache memories and servers) that store one or more instructions or data structures. The term "machine-readable medium" should also be considered to include any tangible medium that is capable of storing, encoding, or carrying instructions executed by a machine and causing the machine to perform any one or more of the methods of the present invention or capable of storing, encoding, or carrying data structures utilized by or associated with such instructions.

[0126] Supplementary Notes

[0127] The above detailed description includes reference to the accompanying drawings, which form a part of the detailed description. The accompanying drawings show the detailed description in an illustrative manner, rather than by a restrictive manner, in which the present invention can be put into practice. These embodiments are also referred to as "examples" in this article. Such examples may include elements other than those shown or described. However, the inventors also contemplate examples that only provide examples of those elements shown or described. In addition, the inventors also contemplate examples of any combination or arrangement of those elements (or one or more aspects of those elements) shown or described with respect to a particular example (or one or more aspects of a particular example) or with respect to other examples (or one or more aspects of other examples) shown or described in this article.

[0128] All publications, patents, and patent documents referenced in this document are incorporated herein by reference in their entirety, as if individually incorporated by reference. If there is an inconsistent usage between this document and those documents incorporated by reference, the usage in the incorporated reference(s) should be considered supplementary to that of this document; for conflicting inconsistencies, the usage in this document controls.

[0129] In this document, when introducing elements of aspects of the present invention or elements of embodiments thereof, the terms "a," "an," "the," and "said" are used, as is common in patent documents, to include one or more than one or more of the elements, independent of any other instance or use of "at least one" or "one or more." In this document, unless otherwise indicated, the term "or" is used to refer to non-exclusivity or such that "A or B" includes "A but not B," "B but not A," and "A and B."

[0130] In the following claims, the terms "including" and "in which" are used as the plain-English equivalents of the respective terms "comprising" and "wherein." Furthermore, in the following claims, the terms "comprising," "including," and "having" are intended to be open-ended, to mean that there may be additional elements in addition to the listed elements, such that anything following such a term in a claim (e.g., comprising, including, having) is considered to fall within the scope of the claim. Moreover, in the following claims, the terms "first," "second," and "third," etc. are used merely as labels and are not intended to impose numerical requirements on their objects.

[0131] The present invention further relates to a computing system adapted, configured or operated to perform the operations herein. The system may be specially constructed for the desired purpose, or the system may include a general-purpose computer selectively started or reconfigured by a computer program (e.g., instruction, code, etc.) stored in the computer. Unless otherwise stated, the order of execution or execution of the operations in the embodiments of the present invention shown and described herein is not necessary. That is, unless otherwise stated, operations may be performed in any order, and embodiments of the present invention may include more or less operations than those disclosed herein. For example, it is contemplated that it is within the scope of the various aspects of the present invention to execute or perform a particular operation before, simultaneously with, or after another operation.

[0132] In view of the above, it will be seen that the several objects of the present invention are achieved and other advantageous results are obtained. Having described various aspects of the present invention in detail, it will be apparent that modifications and variations are possible without departing from the scope of the various aspects of the present invention as defined in the appended claims. Since various changes can be made in the above-described constructions, products, and methods without departing from the scope of the various aspects of the present invention, it is intended that all matter contained in the above description and shown in the accompanying drawings shall be interpreted as illustrative and not restrictive.

[0133] The above description is intended to be illustrative, rather than restrictive. For example, the examples described above (or one or more aspects of the examples) can be used in combination with each other. In addition, without departing from the scope of the present invention, many modifications can be made to adapt specific situations or materials to the teachings of the present invention. Although the size, type and example parameters, functions and implementations of the materials described herein are intended to limit the parameters of the present invention, they are by no means restrictive, but exemplary embodiments. After reviewing the above description, many other embodiments will be apparent to those skilled in the art. Therefore, the scope of the present invention should be determined with reference to the full range of equivalents given by the appended claims and such claims.

[0134] Furthermore, in the above detailed description, various features may be combined to simplify the disclosure. This should not be interpreted as intending that unclaimed disclosed features are essential to any claim. Rather, inventive subject matter may lie in fewer than all features of a particular disclosed embodiment. Accordingly, the appended claims are hereby incorporated into the detailed description, with each claim independently serving as a separate embodiment. The scope of the present invention should be determined by reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

[0135] Regarding the implementation methods including the above embodiments, the following technical solutions are also disclosed:

[0136] Solution 1. A computer-implemented method for performing atlas-based segmentation using deep learning, the method comprising:

[0137] applying a deep learning model to an image of a subject, the deep learning model being trained to generate deep learning segmentation data that identifies anatomical features in the image of the subject;

[0138] registering an atlas image with the subject image, the atlas image being associated with annotation data identifying anatomical features in the atlas image, wherein the registering uses the deep learning segmentation data to improve the registration result between the atlas image and the subject image;

[0139] generating a mapping atlas by registering the atlas image with the subject image; and

[0140] Anatomical features in the subject image are identified using the mapping atlas.

[0141] Option 2. A method according to Option 1, wherein the registration improves the registration result between the atlas image and the subject image by applying the deep learning segmentation data to determine an initial registration estimate or constraint based on anatomical features identified in the subject image.

[0142] Option 3. The method of option 1, wherein the atlas image is one of a plurality of atlas images, and wherein the mapping atlas is one of a plurality of mapping atlases, the method further comprising:

[0143] registering the plurality of atlas images with the subject image, the plurality of atlas images being associated with respective annotation data identifying anatomical features in the respective atlas images, wherein the registering uses the deep learning segmentation data to improve the registration result between the plurality of atlas images and the subject image; and

[0144] generating the plurality of mapping atlases by registering the plurality of atlas images with the subject image, the plurality of mapping atlases identifying respective locations and boundaries of anatomical features in the subject image;

[0145] Wherein, identifying anatomical features in the subject image includes combining results from the plurality of mapping atlases.

[0146] Option 4. The method of option 3, wherein the anatomical feature is one of a plurality of anatomical features, and wherein identifying the anatomical feature in the subject image from the plurality of mapping atlases further comprises:

[0147] performing structural labeling of the plurality of anatomical features in the image of the subject based on the plurality of mapping atlases; and

[0148] A structural labeling map of the subject image is generated based on the structural labeling of the plurality of anatomical features.

[0149] Solution 5. The method according to solution 1, further comprising:

[0150] applying the deep learning model to the atlas image, the deep learning model being trained to: generate additional deep learning segmentation data that identifies anatomical features in the atlas image;

[0151] Wherein, registering the atlas image with the subject image further comprises using the other deep learning segmentation data, wherein the other deep learning segmentation data is further used to improve the registration result of the identified anatomical features between the atlas image and the subject image.

[0152] Solution 6. The method according to solution 1, further comprising:

[0153] applying a machine learning model to the subject image, training the machine learning model based on feature data from a layer of the deep learning model, wherein the machine learning model provides a structure classifier indicative of a predicted anatomical structure classification; and

[0154] generating a classifier structure label for the subject image from the predicted anatomical structure classification;

[0155] Wherein, identifying the anatomical features comprises combining structural labels from the atlas-based segmentation with classifier structural labels from the machine learning model to identify a structural label map of the subject image.

[0156] Option 7. The method according to Option 6, wherein the machine learning model is a boosted tree (BT), random forest (RF) or support vector machine (SVM) classifier.

[0157] Option 8. A method according to Option 6, wherein the machine learning model is also trained based on at least one of the following: atlas feature data from the atlas image, mapped atlas feature data generated from a mapping of the atlas image, or a transformation of the segmentation labeling map from the deep learning model.

[0158] Option 9. A method according to Option 1, wherein the deep learning model is a convolutional neural network, wherein the anatomical features are segmented from a set of 3D images; and wherein the deep learning model is trained based on multiple medical images, and the multiple medical images classify individual voxels of the anatomical features in the segmented label map.

[0159] Option 10. A method according to Option 9, wherein the multiple medical images used to train the deep learning model include images from various medical devices, wherein the various medical devices utilize variations in imaging and contouring protocols to identify anatomical features in the multiple medical images.

[0160] 11. A computer-implemented method for operating a trained machine learning classifier in an atlas-based segmentation process using deep learning, the method comprising:

[0161] applying a deep learning model to an atlas image, the deep learning model being adapted to generate data by analyzing a plurality of anatomical structures in the atlas image;

[0162] training a machine learning model classifier using data generated by applying the deep learning model, the machine learning model classifier being trained to classify anatomical structures in the atlas images;

[0163] applying the trained machine learning model classifier to a subject image to produce a classification of respective regions of the subject image;

[0164] estimating a structural label for each region of the subject image based on the classification of each region of the subject image; and

[0165] The estimated structural labels are combined with structural labels generated by performing atlas-based segmentation on the subject image to define structural labels for each region of the subject image.

[0166] Option 12. A method according to Option 11, wherein the deep learning model includes a convolutional neural network trained to perform segmentation of an input image, and wherein the data generated by applying the deep learning model includes a feature map generated by analyzing the input image in an intermediate convolutional layer of the convolutional neural network.

[0167] Solution 13. The method of solution 11, wherein the atlas image is one of a plurality of atlas images, and the method further comprises performing the atlas-based segmentation on the subject image by:

[0168] registering a plurality of atlas images with the subject image using segmentation data generated by applying the deep learning model to the subject image;

[0169] generating a plurality of mapping atlases on the subject image based on registering the plurality of atlas images with the subject image; and

[0170] A structural signature of the subject image is generated from the plurality of mapping atlases.

[0171] Option 14. The method according to Option 13, wherein generating structural labels of the subject image from the multiple mapping atlas sets includes: performing label refinement and label fusion on multiple labels indicated from the multiple mapping atlas sets.

[0172] Option 15. A method according to Option 11, wherein the atlas image is one of a plurality of atlas images, and wherein the training of the machine learning model classifier is also performed using a segmentation result generated by applying the deep learning model to the plurality of atlas images.

[0173] Option 16. A method according to Option 11, wherein the atlas image is one of a plurality of atlas images, and wherein the training of the machine learning model classifier is also performed using segmentation feature data generated by applying the deep learning model to the plurality of atlas images.

[0174] Option 17. The method according to Option 11, further comprising:

[0175] A label map for the subject image is generated from the structural labels for respective regions of the subject image, the label map identifying respective segmentations of the subject image, wherein each region of the subject image includes a respective structural label corresponding to a plurality of voxels.

[0176] Solution 18. A machine-readable storage medium comprising instructions, wherein the instructions, when executed by a circuit system of a processing device, cause the circuit system to perform the method according to any one of claims 11 to 17.

[0177] Solution 19. A system for performing atlas-based segmentation using deep learning, the system comprising:

[0178] processing circuitry comprising at least one processor; and

[0179] A storage medium comprising instructions that, when executed by the at least one processor, cause the processor to:

[0180] obtaining images of the subject;

[0181] applying a deep learning model to the subject image, the deep learning model being trained to: generate deep learning segmentation data that identifies anatomical features in the subject image;

[0182] performing registration of an atlas image with the subject image, the atlas image being associated with annotation data identifying anatomical features in the atlas image, wherein the registration uses the deep learning segmentation data to improve a registration result between the atlas image and the subject image;

[0183] generating a mapping atlas by registering the atlas image with the subject image; and

[0184] Identification of anatomical features in the subject image is performed using the mapping atlas.

[0185] Option 20. A system according to Option 19, wherein the registration improves the registration result between the atlas image and the subject image by applying the deep learning segmentation data to determine an initial registration estimate or constraint based on anatomical features identified in the subject image.

[0186] 21. The system of claim 19, wherein the atlas image is one of a plurality of atlas images, and wherein the mapping atlas is one of a plurality of mapping atlases, wherein the instructions further cause the processor to:

[0187] registering the plurality of atlas images with the subject image using the deep learning segmentation data, the plurality of atlas images being associated with respective annotation data identifying anatomical features in the respective atlas images, wherein the registering uses the deep learning segmentation data to improve a registration result between the plurality of atlas images and the subject image; and

[0188] generating the plurality of mapping atlases by registering the plurality of atlas images with the subject image, the plurality of mapping atlases identifying respective locations and boundaries of anatomical features in the subject image;

[0189] Wherein, identifying anatomical features in the subject image includes combining results from the plurality of mapping atlases.

[0190] 22. The system of claim 21, wherein the anatomical feature is one of a plurality of anatomical features, and wherein identifying the anatomical feature in the image of the subject from the plurality of mapping atlases causes the processor to:

[0191] performing structural labeling of the plurality of anatomical features in the image of the subject based on the plurality of mapping atlases; and

[0192] A structural labeling map of the subject image is generated based on the structural labeling of the plurality of anatomical features.

[0193] Option 23. The system of Option 19, wherein the instructions further cause the processor to:

[0194] applying the deep learning model to the atlas image, the deep learning model being trained to: generate additional deep learning segmentation data that identifies anatomical features in the atlas image;

[0195] Wherein, registering the atlas image with the subject image further comprises using the other deep learning segmentation data, wherein the other deep learning segmentation data is further used to improve the registration result of the anatomical features between the atlas image and the subject image.

[0196] Option 24. The system of Option 19, wherein the instructions further cause the processor to:

[0197] applying a machine learning model to the subject image, the machine learning model trained based on feature data from a layer of the deep learning model, wherein the machine learning model provides a structure classifier indicative of a predicted anatomical structure classification; and

[0198] generating a classifier structure label for the subject image from the predicted anatomical structure classification;

[0199] wherein identifying the anatomical features comprises: combining structural labels from the atlas-based segmentation and classifier structural labels from the machine learning model to identify a structural label map of the subject image; and

[0200] Wherein, the machine learning model is a boosted tree (BT), random forest (RF) or support vector machine (SVM) classifier.

[0201] Option 25. A system according to Option 19, wherein the deep learning model is a convolutional neural network, wherein the deep learning model is trained based on multiple medical images, wherein the multiple medical images classify individual voxels of anatomical features in a segmentation label map, wherein the multiple medical images used to train the deep learning model include images from various medical devices, and wherein the various medical devices utilize variations in imaging and contouring protocols to identify anatomical features in the multiple medical images.

Claims

1. A computer-implemented method for using a trained machine learning model classifier in an atlas-based segmentation process, the method comprising: applying the trained machine learning model classifier to a subject image to classify one or more segmented anatomical features in respective regions of the subject image, the trained machine learning model classifier having been previously trained on data generated by a deep learning model separate from the trained machine learning model classifier, wherein the trained machine learning model classifier is trained using data generated by the deep learning model to classify a plurality of anatomical structures in an atlas image, and wherein the deep learning model comprises a convolutional neural network adapted to generate data by analyzing the plurality of anatomical structures in the atlas image; estimating structural labels for respective regions of the subject image based on the one or more segmented anatomical features classified in the subject image; performing atlas-based segmentation on the subject image to produce an atlas structural labeling, wherein the atlas-based segmentation comprises registration of a plurality of atlases, the registration of the plurality of atlases aided by segmentation data generated from analysis of the subject image or the plurality of atlases using the deep learning model; and defining a structural label for each region of the subject image by combining the estimated structural label with the atlas structural label; wherein the defined structural markers are generated by fusing (i) a first mapping of the one or more segmented anatomical features of the subject image generated by applying the trained machine learning model classifier to the subject image and (ii) a second mapping of the one or more segmented anatomical features of the subject image generated by performing the atlas-based segmentation on the subject image.

2. The method according to claim 1, wherein The convolutional neural network of the deep learning model is trained to perform segmentation of an input image, and wherein the data generated by applying the deep learning model includes a feature map produced by analyzing the input image in an intermediate convolutional layer of the convolutional neural network.

3. The method of claim 1 , further comprising performing the atlas-based segmentation on the subject image by: registering a plurality of atlas images with the subject image using segmentation data generated by applying the deep learning model to the subject image; generating a plurality of mapping atlases on the subject image based on registering the plurality of atlas images with the subject image; as well as The atlas structure signature of the subject image is generated from the plurality of mapping atlases.

4. The method according to claim 3, wherein: Generating the atlas structured labeling of the subject image from the plurality of mapping atlases includes performing label refinement and label fusion on a plurality of labels indicated from the plurality of mapping atlases.

5. The method according to claim 1, wherein Training of the machine learning model classifier is also performed using segmentation results produced by applying the deep learning model to multiple atlas images.

6. The method according to claim 1, wherein Training of the machine learning model classifier includes using feature data obtained from layers of the convolutional neural network of the deep learning model.

7. The method according to claim 1, further comprising: A label map for the subject image is generated from the structural labels for respective regions of the subject image, the label map identifying respective segmentations of the subject image, wherein each region of the subject image includes a respective structural label corresponding to a plurality of voxels.

8. A non-transitory computer-readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform a method for using a trained machine learning model classifier in an atlas-based segmentation process, the method comprising: applying the trained machine learning model classifier to a subject image to classify one or more segmented anatomical features in respective regions of the subject image, the trained machine learning model classifier having been previously trained on data generated by a deep learning model separate from the trained machine learning model classifier, wherein the trained machine learning model classifier is trained using data generated by the deep learning model to classify a plurality of anatomical structures in an atlas image, and wherein the deep learning model comprises a convolutional neural network adapted to generate data by analyzing the plurality of anatomical structures in the atlas image; estimating structural labels for respective regions of the subject image based on the one or more segmented anatomical features classified in the subject image; performing atlas-based segmentation on the subject image to produce an atlas structural labeling, wherein the atlas-based segmentation comprises registration of a plurality of atlases, the registration of the plurality of atlases aided by segmentation data generated from analysis of the subject image or the plurality of atlases using the deep learning model; and defining a structural label for each region of the subject image by combining the estimated structural label with the atlas structural label; wherein the defined structural markers are generated by fusing (i) a first mapping of the one or more segmented anatomical features of the subject image generated by applying the trained machine learning model classifier to the subject image and (ii) a second mapping of the one or more segmented anatomical features of the subject image generated by performing the atlas-based segmentation on the subject image.

9. The non-transitory computer readable medium of claim 8, wherein: The convolutional neural network of the deep learning model is trained to perform segmentation of an input image, and wherein the data generated by applying the deep learning model includes a feature map produced by analyzing the input image in an intermediate convolutional layer of the convolutional neural network.

10. The non-transitory computer-readable medium of claim 8, the method further comprising performing the atlas-based segmentation on the subject image by: registering a plurality of atlas images with the subject image using segmentation data generated by applying the deep learning model to the subject image; generating a plurality of mapping atlases on the subject image based on registering the plurality of atlas images with the subject image; as well as The atlas structure signature of the subject image is generated from the plurality of mapping atlases.

11. The non-transitory computer readable medium of claim 10, wherein: Generating the atlas structured labeling of the subject image from the plurality of mapping atlases includes performing label refinement and label fusion on a plurality of labels indicated from the plurality of mapping atlases.

12. The non-transitory computer readable medium of claim 8, wherein: Training of the machine learning model classifier is also performed using segmentation results produced by applying the deep learning model to multiple atlas images.

13. The non-transitory computer readable medium of claim 8, wherein: Training of the machine learning model classifier includes using feature data obtained from layers of the convolutional neural network of the deep learning model.

14. The non-transitory computer readable medium of claim 8, the method for operating a trained machine learning model classifier further comprising: A label map for the subject image is generated from the structural labels for respective regions of the subject image, the label map identifying respective segmentations of the subject image, wherein each region of the subject image includes a respective structural label corresponding to a plurality of voxels.

15. A system for using a trained machine learning model classifier in an atlas-based segmentation process, the system comprising: An input interface, wherein the input interface is configured to: receiving an atlas image, the atlas image corresponding to a region of an anatomical structure; as well as Receiving a deep learning model adapted to generate data by analyzing a plurality of anatomical structures in the atlas image, wherein the deep learning model comprises a convolutional neural network; at least one storage device configured to store the atlas image and the deep learning model; and An image processor configured to: Applying the deep learning model to the atlas image; training a machine learning model classifier using data generated by applying the deep learning model to provide the trained machine learning model classifier to classify anatomical structures in the atlas images; applying the trained machine learning model classifier to a subject image to classify one or more segmented anatomical features in respective regions of the subject image; estimating structural labels for respective regions of the subject image based on the one or more segmented anatomical features classified in the subject image; performing atlas-based segmentation on the subject image to produce an atlas structural labeling, wherein the atlas-based segmentation comprises registration of a plurality of atlases, the registration of the plurality of atlases aided by segmentation data generated from analysis of the subject image or the plurality of atlases using the deep learning model; and defining a structural label for each region of the subject image by combining the estimated structural label with the atlas structural label; wherein the defined structural markers are generated by fusing (i) a first mapping of the one or more segmented anatomical features of the subject image generated by applying the trained machine learning model classifier to the subject image and (ii) a second mapping of the one or more segmented anatomical features of the subject image generated by performing the atlas-based segmentation on the subject image.

16. The system according to claim 15, wherein: The convolutional neural network of the deep learning model is trained to perform segmentation of an input image, and wherein the data generated by applying the deep learning model includes a feature map produced by analyzing the input image in an intermediate convolutional layer of the convolutional neural network.

17. The system according to claim 15, wherein: The image processor is further configured to perform the atlas-based segmentation on the subject image by: registering a plurality of atlas images with the subject image using segmentation data generated by applying the deep learning model to the subject image; generating a plurality of mapping atlases on the subject image based on registering the plurality of atlas images with the subject image; as well as The atlas structure signature of the subject image is generated from the plurality of mapping atlases.

18. The system according to claim 17, wherein: Generating the atlas structured labeling of the subject image from the plurality of mapping atlases includes performing label refinement and label fusion on a plurality of labels indicated from the plurality of mapping atlases.

19. The system of claim 15, wherein: Training of the machine learning model classifier is also performed using segmentation results produced by applying the deep learning model to multiple atlas images.

20. The system of claim 15, wherein: The training of the machine learning model classifier uses feature data, which is obtained from the layers of the convolutional neural network of the deep learning model.

21. The system of claim 15, wherein the image processor is further configured to: A label map for the subject image is generated from the structural labels of the respective regions of the subject image, the label map identifying respective segmentations of the subject image, wherein Each region of the subject image includes each structural marker corresponding to a plurality of voxels.

Images