Systems and methods for scalable segmentation model training

Abstract

A system for training a segmentation model, comprising: an interface configured to allow a user to: upload and store training data in a storage device of a cloud-based network; provide access to the training data stored in the storage device; initiate a request to train a segmentation model; monitor training of the segmentation model; and download the trained segmentation model; and a computing infrastructure configured to: pre-process the training data using a first set of computing resources of the cloud-based network to obtain processed training data, and store the processed training data in the storage device; deploy a training application on a second set of computing resources of the cloud-based network to train the segmentation model based on the processed training data; provide access to monitor the training; and provide access to the trained segmentation model.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to medical imaging, and more particularly, to systems and methods for cloud-based scalable segmentation model training. BACKGROUND

[0002] Radiation therapy is an important tool for treating cancerous tumors in patients. Unfortunately, ionizing radiation applied to treat a patient does not inherently distinguish between a tumor and neighboring healthy structures (e.g., organs at risk). Thus, the administration of ionizing radiation must be carefully tailored in order to limit the radiation applied to the target (i.e., the tumor), while avoiding unnecessary radiation of surrounding anatomical structures, with the goal of delivering a lethal radiation dose to the tumor, while maintaining an acceptable dose to neighboring structures.

[0003] As part of the radiation treatment planning process, medical images of the tumor and surrounding anatomy are acquired. The medical images can be used as a basis for radiation treatment simulation, and can be used to plan various aspects of the therapy, including but not limited to beam geometry and position, radiation energy and dose. The medical images are typically processed to delineate a target region (e.g., pixels or voxels where a tumor or other region is imaged that is desired to be irradiated) and separate surrounding structures (e.g., pixels or voxels where organs at risk (OARs) or other anatomical structures are imaged that should be avoided from being irradiated). This delineation, referred to as contouring or segmentation, involves defining respective boundaries that define the contours of different anatomical structures in the image. However, if the anatomical structures are not contoured correctly in the image, this can result in insufficient irradiation of the target and / or undesired irradiation of the surrounding structures.

[0004] Manual contouring of structures in medical images can be a time-consuming phase in the radiation treatment planning process. To address this problem, automated segmentation models, such as deep learning segmentation models, have been proposed. Deep learning-based segmentation methods are automated segmentation methods that utilize many layers or stages of non-linear data processing for feature learning as well as pattern analysis and / or classification.

[0005] Generally, deep learning segmentation methods involve a training phase and an inference phase, as shown in Figure 1 During the training phase, a deep neural network (DNN) model uses a training dataset (10) of medical images to generate a particular output (20). For example, the training dataset (10) can include 2-D or 3-D images having ground truth contours for anatomical structures imaged by different pixels or voxels. During the inference phase, the trained DNN model operates (40) on medical image(s) (30) of a patient to automatically process (50) features of the medical image(s).

[0006] Because deep learning based segmentation methods rely heavily on the dataset they are trained on, they do not guarantee generalization to other datasets. For example, when a segmentation DNN model is trained on a dataset produced at one clinic, but it is applied to patient data produced at other clinics, or to patient data produced about patients from different geographic regions, this can result in suboptimal contours. Furthermore, because different clinics have different contouring practices, a segmentation DNN model that performs well at one clinic can not be acceptable at another.

[0007] Embodiments of the disclosed subject matter can address one or more of the problems and shortcomings described above. SUMMARY

[0008] In a first aspect, the invention provides a system for training a segmentation model. The system includes an interface and a computing infrastructure. The interface is configured to allow a user to upload and store training data in a storage device of a cloud-based network, provide access to the training data stored in the storage device, initiate a request for training a segmentation model, monitor training of the segmentation model, and download the trained segmentation model. The computing infrastructure is configured to preprocess the training data using a first set of computing resources of the cloud-based network to obtain processed training data, and store the processed training data in the storage device, deploy a training application on a second set of computing resources of the cloud-based network to train the segmentation model based on the processed training data, provide access to monitor the training, and provide access to the trained segmentation model.

[0009] In a second aspect, the invention provides a method for cloud-based segmentation model training. The method includes uploading and storing training data in a storage device of a cloud-based network, initiating a request for training a segmentation model, initiating automatic deployment of a training application on computing resources of the cloud-based network, the segmentation model being trained by the training application using the stored training data, monitoring the training, and accessing the trained segmentation model.

[0010] In a third aspect, the invention provides a non-transitory computer readable storage medium and a computer processing system. The non-transitory computer readable storage medium has embodied thereon a sequence of programming instructions, and the computer processing system executes the sequence of programming instructions embodied on the computer readable storage medium to cause the computer processing system to access training data uploaded by a client to a cloud storage device, deploy a training application on cloud-based computing resources to train a segmentation DNN model based on the training data, allow the training data to be monitored by the client, allow the client to determine whether the segmentation DNN model is trained, and provide the trained segmentation DNN model to the client, wherein the deployment of the training application is scalable.

[0011] Optional features are defined in dependent claims.

[0012] Embodiments of the disclosed subject matter provide a model training solution to allow remote clients / users / customers to be able to train segmentation models on their own training datasets.

[0013] Embodiments of the disclosed subject matter also provide a model training solution to allow remote clients / users / customers to be able to apply trained segmentation models on their own or on other datasets.

[0014] In embodiments, the model training solution is a cloud-based training solution.

[0015] In embodiments, the cloud-based model training solution is a scalable training solution.

[0016] In embodiments, the training can be scaled based on need, and / or number of segmentation requests, and / or size of data, and / or size of model.

[0017] Embodiments of the disclosed subject matter also provide a computing infrastructure to support applications of the cloud-based model training scalable solution.

[0018] In embodiments, the computing infrastructure includes systems and methods to support the cloud-based scalable training solution.

[0019] In embodiments, the system can include a computing interface through which the client / user / can upload and store training data in a storage device of a cloud-based network; provide access to the training data stored in the storage device; initiate a request for training a segmentation model; monitor training of the segmentation model; download the trained segmentation model.

[0020] In embodiments, the system can also include a computing system operably coupled with the client device through the computing interface and configured to pre-process the training data using a first set of computing resources of the cloud-based network; store the processed training data in a storage device of the cloud-based network; based on a training request from the client device, deploy a training application on a second set of computing resources of the cloud-based network to train a segmentation model based on the processed training data; provide access to the client device to monitor the training; and provide access to the trained segmentation model.

[0021] In embodiments, the computer interface is a web-based interface.

[0022] In embodiments, the first set of computing resources includes a computing device, a computing server, or a virtual machine, and the second set of computing resources includes one or more virtual machines.

[0023] In an embodiment, the second set of computing resources comprises a scalable cluster of computing engines, the scalable cluster comprising a management computing engine and a plurality of worker computing engines, the cluster configured to scale up or scale down based on a number of segmentation model training requests.

[0024] In an embodiment, the training application comprises a machine learning framework configured to support deployment of data and computations across different platforms and different tasks.

[0025] In an embodiment, the segmentation model to be trained is provided by the client / user / customer device.

[0026] In an embodiment, the segmentation model to be trained is selected by the user from a database of segmentation models provided by the computing system via a computer interface.

[0027] In an embodiment, the segmentation model is an automatic segmentation model.

[0028] In an embodiment, the automatic segmentation model is a segmentation neural network model.

[0029] In an embodiment, the training can comprise training the segmentation neural network model to approximate contours of different anatomical structures in medical images.

[0030] In an embodiment, the computing system can provide output checkpoints at different intervals during training, or upon request by the user, so that the user can monitor the training process.

[0031] In an embodiment, the computing system can provide access to information about training progress, training performance, and training completion.

[0032] In an embodiment, the system is configured to apply the trained segmentation model to a dataset obtained from the client user; apply another segmentation model on the same dataset; compare the segmentation results to obtain a comparison result; and compare the result to a predetermined threshold evaluation.

[0033] In an embodiment, the system is configured to apply the trained segmentation model to a dataset obtained from the client device; and evaluate the segmentation result against a predetermined threshold.

[0034] In an embodiment, the system can provide access to: a training log for monitoring the training process and a model file for downloading the trained segmentation model.

[0035] Embodiments of the disclosed subject matter also provide a system and method for training a segmentation model, the system including a web-based computer interface that allows a client user to select training data from a remote database, and initiate a request to train a segmentation model using the selected training data; and a computing infrastructure configured to pre-process the selected training data, deploy a training application to train a segmentation model based on the pre-processed training data, and provide access to the trained segmentation model for further use by the user.

[0036] In embodiments, the user can also apply the trained segmentation model to generate contours on medical images of a patient.

[0037] Embodiments of the disclosed subject matter also provide a computing platform configured to: access training data stored in a storage device of a cloud-based network; pre-process the training data using a first set of computing resources of the cloud-based network, and store the pre-processed training data in the storage device; deploy a training application on a second set of computing resources of the cloud-based network to train a segmentation model based on the pre-processed training data; provide access to monitor the training of the segmentation model; and provide access to the trained segmentation model.

[0038] Embodiments of the disclosed subject matter also provide a method for scalable cloud-based training of a segmentation model.

[0039] In embodiments, the method can include uploading and storing training data in a storage device of a cloud-based network; initiating a request for training a segmentation model, the initiating automatically deploying a training application on computing resources of the cloud-based network through which the segmentation model is trained using the stored training data; monitoring the training; and accessing the trained segmentation model.

[0040] Embodiments of the disclosed subject matter also provide a non-transitory computer readable storage medium having sequences of programming instructions embodied thereon, and a computer processing system that executes the sequences of programming instructions embodied on the computer readable storage medium to cause the computer processing system to perform the model training methods described herein.

[0041] In embodiments, the computer-readable storage media can cause the computer processing system to prompt a client user to upload training data in a storage device of a cloud-based network, provide access to training data stored in the storage device, and initiate a request for training a segmentation model. The computer-readable storage media can also cause the computer processing system to pre-process the training data using a first set of computing resources of the cloud-based network, store the processed training data in a storage device of the cloud-based network, deploy a training application on a second set of computing resources of the cloud-based network to train the segmentation model based on the processed training data, and provide access to monitor the training. The computer-readable storage media can also cause the computer processing system to allow the user to monitor the training process and access the trained segmentation model.

[0042] Objects and advantages of embodiments of the disclosed subject matter will be apparent after consideration of the following description. BRIEF DESCRIPTION OF DRAWINGS

[0043] Embodiments will be described below with reference to the accompanying drawings, which are not necessarily drawn to scale. These drawings are intended to illustrate certain embodiments of the disclosure and are not intended to limit the scope of the disclosure in any way. Some features can not be shown to help illustrate underlying features. Wherever possible, like reference numbers have been used to refer to like elements throughout the drawing(s). As used herein, various embodiments can mean one, some, or all embodiments.

[0044] Figure 1 , Figure 2A , Figure 2B is a simplified schematic diagram of the operation of a neural network model during the training and inference phases in accordance with various embodiments of the disclosed subject matter.

[0045] Figure 2C is a simplified node diagram of a deep neural network in accordance with various embodiments of the disclosed subject matter.

[0046] Figure 2D is a simplified schematic diagram of the operation of a neural network model during the training phase in accordance with various embodiments of the disclosed subject matter.

[0047] Figure 3 is a simplified schematic diagram of a cloud-based system for training a segmentation model in accordance with various embodiments of the disclosed subject matter.

[0048] Figure 4 is a simplified schematic diagram of an image processing system of a client device in accordance with various embodiments of the disclosed subject matter.

[0049] Figure 5 is a simplified schematic diagram showing aspects of a medical image processing and radiotherapy system in accordance with various embodiments of the disclosed subject matter.

[0050] Figure 6 It is a schematic diagram of structural and dose data acquired in medical image processing according to various embodiments of the disclosed subject matter.

[0051] Figure 7 These are simplified schematic diagrams of computing systems / networks based on various embodiments of the disclosed subject matter.

[0052] Figures 8A-8C This is a simplified schematic diagram of cloud computing resources according to various embodiments of the disclosed subject matter.

[0053] Figure 9-11 This is a simplified schematic diagram of a cloud-based scalable training network based on various embodiments of the disclosed subject matter.

[0054] Figure 12A-12F This is a simplified schematic diagram of training a distributed segmentation DNN network using a computer cluster, according to various embodiments of the disclosed subject matter.

[0055] Figures 13A-13C This is a process flowchart of a cloud-based scalable segmented DNN model training solution based on various embodiments of the disclosed subject matter.

[0056] Figure 14 This is a processing flowchart of the inference phase of a trained segmentation DNN model using various embodiments of the disclosed subject matter. Detailed Implementation

[0057] The operation of segmentation models based on deep neural networks (DNN models) typically involves a training phase (TP) and an inference phase (IP), for example... Figure 1 , 2A As shown in -2C. As used herein, the term "deep learning model" or "deep neural network model" refers to a class of computer-based machine learning algorithms that utilize many layers or stages of data processing (particularly at least two "hidden" layers between the input and output layers) to perform feature learning, pattern analysis, and / or classification. Typically, these DNN models are formed by hierarchical networks of processing elements (called neurons or nodes) interconnected by connections (called synapses or weights). The node layers are trained end-to-end (i.e., from the input layer to the output layer) to extract features from the input and classify the features to produce an output (e.g., a classification label or category).

[0058] Figure 2C A simplified node diagram 250 of an exemplary DNN model is shown. The DNN model includes transforming the input (provided to node 258 of input layer 252) into the output (at node 262 of output layer 256). Figure 2CThe intermediate layers (Layer 1 through Layer n) between the input layer 252 and the output layer 256 are referred to as hidden layers 254. At least two hidden layers are provided so that the neural network is considered “deep.” Each hidden layer has respective nodes 260 that perform specific computations and are interconnected to nodes in adjacent layers. For example, each node 260 can include a weighting function that provides weights to respective inputs and an activation function that processes the weighted inputs to generate a respective output. Different hidden layers 254 can include, but are not limited to, a final loss layer, a nonlinear operator layer, a pooling layer, a subsampling layer, an upsampling layer, a fully connected layer, and a convolutional layer. Although Figure 2C The hidden layers 254 are illustrated as having more nodes 260 per layer than the number of nodes 258 / 262 in the input layer 252 and the output layer 256, but other numbers and configurations are possible. Figure 2C The simplified diagram shown in FIG. 1 is exemplary only, and other diagrams based on a selected DNN (e.g., a convolutional neural network) are possible according to one or more contemplated embodiments.

[0059] Generally, in a training phase (TP), the segmentation DNN model 20 uses a training dataset 10 of medical images 11 to generate a particular output 21. For example, the training dataset 10 can include two-dimensional (2-D) or three-dimensional (3-D) images 11 having ground truth contours of anatomical structures imaged by different pixels or voxels. To train the DNN model 20, the training dataset 10 can include additional ground truth information, such as cut-plane locations and / or user-defined ROIs (e.g., bounding boxes). As used herein, “training” refers to, for example, adjusting one or more parameters of the nodes in the hidden layers of the DNN model 20 to cause the output 21 of the DNN model 20 to more closely match the corresponding ground truth. Figure 2D The one or more parameters of the nodes in the hidden layers of the DNN model 20 are determined by the iterative training process S100 shown in FIG. 1, which adjusts the parameters so that the output 21 of the DNN model 20 more closely matches the corresponding ground truth. For example, as shown in FIG. 1, the nodes 260 in the hidden layers 254 can include filters or kernels, the parameters of which (e.g., kernel weights, size, shape, or structure) can be adjusted during the training process. Figure 2C The one or more parameters of the nodes in the hidden layers of the DNN model 20 are determined by the iterative training process S100 shown in FIG. 1, which adjusts the parameters so that the output 21 of the DNN model 20 more closely matches the corresponding ground truth. For example, as shown in FIG. 1, the nodes 260 in the hidden layers 254 can include filters or kernels, the parameters of which (e.g., kernel weights, size, shape, or structure) can be adjusted during the training process.

[0060] Figure 2D The training process S100 of the iterative model is shown. In step S102, the training dataset 10 provided in S101 is propagated through the nodes of the hidden layers of the DNN model. In S103, the resulting data from the hidden layers is provided to the nodes of the output layer of the DNN model. In step S104, the data from the output layer is compared to the ground truth via a loss function 22. For example, the loss function 22 can be a mean squared error, a dice loss, a cross-entropy based loss, or any other loss function known in the art.

[0061] During training S102, the DNN model is given feedback by the loss function 22 about how well its output matches the correct output. Once the iteration criteria are met at S105 (e.g., the loss function meets a predetermined threshold, a threshold number of iterations have been reached, or no further improvement is seen between iterations), at S107, the DNN model is fixed. Otherwise, the training process S100 proceeds to S106, where the DNN model is modified, e.g., by adjusting the parameters of the hidden layer nodes, in order to improve the match between the output and the desired output. The training process S100 can repeat iterations until the desired iteration criteria are met at S105. The DNN model is then considered trained, and the trained DNN model of S107 can be stored in an image segmentation model database. Multiple DNN models can be trained according to this process.

[0062] During the inference phase (IP), the trained DNN model 20 can operate on a medical image set 30 containing a medical image 31 of a new patient to automatically process the features of the medical image 31, such as determining the contours 40 of unknown anatomical structures in the image(s). The contour image can then be used to generate a treatment plan for the patient.

[0063] Each respective DNN model can run on a corresponding DNN engine, which refers to any suitable hardware and / or software component(s) of a computer system capable of executing an algorithm according to any suitable deep learning model. In embodiments, the deep learning model(s) can be based on any existing or later developed neural network or combination thereof. Exemplary neural networks include, but are not limited to, convolutional neural networks (ConvNet or CNN) (e.g., U-Net, deep CNN, LeNet, V-Net, AlexNet, VGGNet, Xception, DenseNet, GoogleNet / Inception, etc.), residual neural networks (ResNet), recurrent neural networks (RNN) (e.g., Hopfield, Echo State, Long Short-Term Memory (LSTM) neural networks, recurrent neural networks, generative adversarial neural networks (GAN), normalized flow, and graph networks, and deep belief networks (DBN).

[0064] To generate the medical images (whether 2-D or 3-D) of the training set 10 and / or the patient set 30, any suitable medical imaging modality can be used, such as, but not limited to, X-ray, computed tomography (CT), cone-beam computed tomography (CBCT), helical CT, positron emission tomography (PET), magnetic resonance imaging (MRI), functional MRI, single photon emission computed tomography (SPECT), optical tomography, ultrasound imaging, fluorescence imaging, radiotherapy portal imaging, or any combination thereof. For example, the image data can include a series of 2-D images or slices, each representing a cross-sectional view of the patient anatomy. Alternatively or additionally, the image data can include a volume or 3-D image of the patient, or a time series of 2-D or 3-D images of the patient.

[0065] In general, the image data set 10 used to train the segmentation DNN model and the patient data set 30 for inference are generated at different clinics. Moreover, the training data used to train the segmentation model is data of patients different from the new patient. Because imaging and contouring protocols can be different at different clinics, and because patient anatomy can also be different across different geographic locations, using a benchmark truth that can not be applicable to the current patient using different locations, and / or using patient data that is not compatible with the new patient, and / or using a segmentation DNN model trained using a contouring protocol that is not compatible with the contouring protocol of the clinic applying the inference process, can generate inaccurate contours for the new patient. Inaccurate contours result in inaccurate treatment plans for the patient. For example, when executed by a radiotherapy system such as the Accuray CyberKnife® shown, inaccurate treatment plans result in inaccurate treatment of the patient. Figure 4

[0066] Figure 3 Aspects of a computer networking infrastructure are shown that can provide a solution to one or more of these problems. For example, Figure 3 A cloud-based training solution is shown whereby a segmentation DNN model can be trained (i.e., training phase) using a training data set generated and / or accepted by a clinic that applies the segmentation DNN model (i.e., inference phase).

[0067] Figure 3 The computer networking infrastructure of the Accuray CyberKnife® can also provide a training solution whereby a segmentation DNN model can be trained (i.e., training phase) and applied (i.e., inference phase) using a patient's own data set.

[0068] Figure 3 The computer networking infrastructure of the Accuray CyberKnife® can also provide a training solution whereby a segmentation DNN model can be trained (i.e., training phase) and applied (i.e., inference phase) using a data set generated at the same clinic that generates the inference data set. ​

[0069] Figure 3 The computer networking infrastructure can also provide training solutions, thereby enabling the training (i.e., the training phase) of segmentation DNN models using patient-specific segmentation models and / or previously developed models.

[0070] Figure 3 The computer networking infrastructure can also provide training solutions, thereby using previously generated datasets to train (i.e., the training phase) segmentation DNN models.

[0071] Figure 3 The computer networking infrastructure can also provide scalable training solutions, which can be used to train (i.e., the training phase) segmentation DNN models based on the computing resources that need to be scaled up or down.

[0072] The cloud-based training solution disclosed in this paper also provides applications of trained segmentation DNN models to generate contours of a patient's anatomical structures in medical images, and / or use the contoured images to generate treatment plans for the patient, and / or use the generated treatment plans on the patient during radiotherapy.

[0073] like Figure 3 As shown, the computer networking infrastructure may include client / user / client computing devices 100, computing systems / networks 400, and cloud networking systems including cloud storage devices 300 and cloud computing resources 500, all connected via one or more wired and / or wireless communication links (such as via network 600). Typically, client computing devices 100 may engage in communication with cloud storage devices 300 to securely upload training and / or other patient data to one or more storage devices of cloud storage devices 300. Client computing devices 100 may also engage in communication with computing systems / networks 400 to request the initiation of a training process, thereby training a segmentation model, such as a segmentation DNN model, for example, based on training data stored in cloud storage devices 300. Upon receiving a request from client computing devices 100, computing systems / networks 400 may engage in secure communication with computing devices of cloud computing resources 500 to deploy training applications across the computing devices to train the segmentation DNN model. Computing systems / networks 400 are also configured to allow, for example, secure downloading of the trained segmentation DNN model by client computing devices 100 for inference.

[0074] In an exemplary embodiment, the client computing device 100 is a treatment planning device, for example... Figure 5The treatment planning device 700 shown is configured to perform any suitable number of treatment planning tasks or steps, such as segmentation, dose prediction, projection data prediction, treatment plan generation, etc. When used in this way, the client computing device 100 is configured to automatically generate contours using a trained segmentation DNN model and to generate plans to be used with a radiotherapy system (e.g., Figure 5 The radiotherapy system 1000 shown executes a treatment plan 740 for the patient.

[0075] Alternatively, the client computing device 100 may include: an image processing module (such as...) Figure 4 and 5 The image processing module 710 shown generates, for example, the outlines of a patient's anatomical structures on a medical image using a trained segmentation DNN model, and as... Figure 4 and 5 The treatment planning module 720 shown generates a treatment plan 740, for example, using a contoured medical image.

[0076] Alternatively, the client computing device 100 may provide the image processing functions of the image processing module 710 to generate the outline of the patient's anatomical structure, wherein the treatment planning module 720 is separate from the client computing device 100.

[0077] exist Figure 4 In the exemplary embodiment shown, the client computing device 100 includes an image processing module 710, a treatment planning module 720, a network interface 200 (i.e., a web interface, such as a web browser), and an input / output device 202 that is operatively coupled to an input / output interface 2100 of the radiotherapy system 1000 via, for example, a network 601.

[0078] Image processing module 710 is configured to perform automatic segmentation using a trained segmentation DNN model to generate structural data (such as) identifying various anatomical structures (such as, but not limited to, malignant tumors (i.e., targets) and any organ of danger (OAR)). Figure 6 The structural data 730A shown may identify, and / or segment / locate anatomical landmarks, and / or segment / locate anatomical or physician-expected areas, such as, but not limited to, areas that can be used to assist in treatment planning (e.g., nodule areas). The structural data 730A may also identify other anatomical structures, such as other organs, tissues, bones, blood vessels, etc. The structural data 730A may also include any appropriate data relating to the patient's anatomical structures, malignancies (i.e., targets), any organ at risk (OAR), and the outline, shape, size, and location of any other anatomical structure.

[0079] The image processing module 710 can include a computer system 102 configured to implement the automated segmentation processes disclosed herein, the computer system 102 including a bus 122 or other mechanism for passing information between components. The computer system 102 can also include a processor 104 coupled to the bus 122, such as but not limited to a general-purpose or special-purpose processor (e.g., a graphics processing unit (GPU)). The processor 104 can be a processor of a cloud-based system, and / or a processor of one or more network or Internet host servers. The image processing module 710 can also include an input / output device 202, such as a communication device (such as a network interface card) providing access to the network 600 for communicating with the computing system / network 400 or the network 601 for communicating with the radiation therapy system 1000, and / or input / output ports allowing a user to interact with the computer system 102, e.g., via user input devices (e.g., an interactive graphical user interface (GUI)) including a mouse, keyboard, display, etc.

[0080] The GUI can include, but is not limited to, user-selectable and / or user-adjustable graphical controls such as, but not limited to, sliders, option buttons, text boxes, drop-down boxes, windows, animations, and / or any other GUI components for: selecting data from internal and / or external and / or remote datasets; selecting and / or adjusting digital renderings of 3D and / or 2D images and / or image slices, and / or anatomical structures and / or outlines of anatomical structures displayed by the processor 104; selecting to upload and / or download data to and / or from cloud storage devices and / or remote databases; displaying and / or accessing a list of available segmentation models, and / or a list of supported anatomical structures displayed by the processor 104 or the computer system 102; etc. The GUI is configured to allow a user to input data, manipulate input and output data, and make any edits to the data, generated outlines, and displayed outputs. The user can interact with the computer system 102 directly or remotely through the networks 600, 601 or via any other method.

[0081] The computer system 102 can also include a memory 106 storing information and instructions to be executed by the processor 104. The memory 106 can include a combination of random access memory (RAM), read only memory (ROM), static storage such as a disk or diskette, or any other type of computer readable medium. For example, the computer readable medium can be any available media that can be accessed by the processor 104 and can include both volatile and nonvolatile media, removable and non-removable media, and communication media. Communication media can include computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and includes any information delivery media.

[0082] Memory 106 can store software modules that provide functionality when executed by processor 104. These modules can include, for example, an operating system 108 that can provide operating system functionality for computer system 102, one or more inference engines 112 configured to access and / or link to a plurality of segmentation DNN models, such as those saved in image segmentation models 120 database of image processing module 710, or image segmentation DNN models database 410 of computing system / network 400, or any other remote segmentation model database available to client computing device 100 or computing system / network 400 or radiation therapy system 1000. Memory 106 can also include additional functionality 101 modules.

[0083] One or more inference engines 112 are modules that include hardware and / or software components capable of executing algorithms according to segmentation models (including, but not limited to, segmentation DNN models) stored in image segmentation models 120 database or models database 410 or remote databases.

[0084] One or more of inference engines 112 can receive medical image data (whether training data or medical image(s) for inference) from image database 123 of image processing module 710, or from image database 420 of computing system / network 400 via network 600, or from any other remote image database available to client computing device 100 of radiation therapy system 1000 via other networks, or from radiation therapy system 1000 via input / output device 202 and network 601, and generate contours for one or more anatomical structures in the received medical images.

[0085] Client computing device 100 can also include one or more proxy servers 118, such as but not limited to servers that facilitate data communication and data movement between client computing device 100, remote computing system / network 400, and / or cloud-based networks including cloud storage 300 and cloud computing resources 500. Proxy servers 118 can also be capable of establishing and maintaining secure communication sessions with computing system / network 400.

[0086] Client computing device 100 can also include a firewall 116 configured to deny all incoming communication sessions via the Internet unless the session is initiated from behind firewall 116 or firewall 116 has been explicitly configured to support the session. By placing proxy servers 118 behind firewall 116, proxy servers 118 can initiate these communication sessions through firewall 116 and thus avoid any potential security risks to client computing device 100.

[0087] In an embodiment, firewall 116 may include one or more dedicated routers 114 or other server devices that protect client device 100 from unauthorized attempts to access devices, applications, and services therein, while allowing authorized communication originating at client computing device 100. Firewall 116 may also provide intrusion detection, web filtering, virus scanning, application-layer gateway, and any other applications and services.

[0088] In one or more embodiments, the client computing device 100 may further include one or more virtual private network (VPN) gateways for communicating with the computing system / network 400 and / or cloud storage device 300 and cloud computing resources 500 services.

[0089] The client computing device 100 may also include a network interface 200 configured to operate a web browser to retrieve, present, and / or navigate information on the World Wide Web. The browser may include web display tools that provide or otherwise support the display of information related to segmentation model training, including but not limited to scalable segmentation model training that can be performed using the computing system / network 400, and segmentation model inference performed by the client computing device 100 and / or the computing system / network 400.

[0090] The client computing device 100 may also include a treatment planning module 720 to generate dosage data 730B, such as... Figure 6 As shown, this indicates the radiation dose that will be delivered to the target and any other anatomical structures to be irradiated, and specifies the maximum permissible radiation dose allowed to be delivered to other anatomical structures (such as, for example, OARs). Therefore, the treatment plan 740 includes information 730 related to the structural data 730A and dose data 730B. The treatment plan 740 may also include any other additional data, such as prescriptions, disease stages, biological or radiological data, genetic data, laboratory data, past treatments or medical history, or any combination thereof. The treatment plan 740 may also take into account constraints imposed on the treatment process by the radiotherapy system 1000 used to deliver radiation to the patient 1001.

[0091] Figure 5 An exemplary radiotherapy system 1000 is also illustrated, which can be used to deliver radiation according to a treatment plan 740 generated by a client computing device 100. The radiotherapy system 1000 can deliver radiation 1022 to a target location 1022 within a patient 1001 located on a treatment bed 1012, and can allow for various radiation dose verification protocols. Radiotherapy may include photon-based radiotherapy, particle therapy, electron beam therapy, or any other type of treatment.

[0092] In one embodiment, the radiation therapy system 1000 can include a radiation therapy device 1010, such as but not limited to a LINAC, operable to generate one or more beams of megavoltage (MV) X-ray radiation 1020 for treatment. The LINAC can also be operable to generate one or more beams of kilovoltage (kV) X-ray radiation, for example, for patient imaging. The radiation therapy system 1000 has a gantry 1002 supporting a radiation treatment head 1014 having one or more radiation sources 1006 and various beam-modulating elements, such as but not limited to a flattening filter 1004 and a collimating assembly 1008. The collimating assembly 1008 can include, for example, a multi-leaf collimator (MLC), an up-down jaw, and / or other collimating elements. The collimating assembly 1008 and / or the flattening filter 1004 can be positioned within the radiation beam path by respective actuators (not shown), which can be controlled by the controller 2000.

[0093] The gantry 1002 can be a ring gantry (i.e., it extends through a full 360° arc to create a complete ring or circle), although other types of mounting arrangements can also be employed. For example, a static column, or a C-arm, partial ring gantry, or robotic arm can be used. Any frame capable of positioning the treatment head 1014 at various rotational and / or axial positions relative to the patient 1001 can also be used.

[0094] In one embodiment, the radiation therapy device is a MV energy intensity modulated radiation therapy (IMRT) device. The intensity profile in such a system is customized according to the treatment requirements of the individual patient. The IMRT fields are delivered by the MLC 1008, which can be a computer-controlled mechanical beam-shaping device attached to the head 1014 and includes a set of metal fingers or leaves. For each beam direction, the optimized intensity profile is achieved by the sequential delivery of various sub-fields with optimized shapes and weights. From one sub-field to the next, the leaves can move either with the radiation beam on (i.e., dynamic multi-leaf collimation (DMLC)) or with the radiation beam off (i.e., segmented multi-leaf collimation (SMLC)).

[0095] Alternatively or additionally, the radiation therapy device 1001 can be a tomotherapy device, a helical tomotherapy device, or a simplified intensity modulated arc therapy (SIMAT) device, a volumetric modulated arc therapy (VMAT) device, or a volumetric high definition (or super-arc) therapy (HDRT). In fact, any type of IMRT device can be used as the radiation therapy device 1010 of the radiation therapy system 1000, and can also include on-board volumetric imaging 1011, which can be used to generate in-treatment image data generated during a treatment session.

[0096] Each type of radiotherapy device can be accompanied by a corresponding radiation plan and radiation delivery procedure.

[0097] The controller 2000, which can be but is not limited to a graphics processing unit (GPU), can include a computer with suitable hardware such as a processor, and an operating system for running various software programs and / or communication applications. The controller 2000 can include a software program operative to communicate with the radiotherapy device 1010, which software program is operable to receive data from external software programs and hardware. The computer can also include any suitable input / output (I / O) interface 2100, which can be adapted to allow communication between the controller 2000 and a user (e.g., medical personnel) of the radiotherapy system 1000. For example, the controller 2000 can be equipped with an I / O interface, a console, a storage device, a memory, a keyboard, a mouse, a monitor, a printer, a scanner, and a department information system (DIS) such as a communication and management interface (DICOM) for storing and transferring medical imaging information and related data, and capable of integrating medical imaging equipment such as scanners, servers, workstations, printers, network hardware, etc.

[0098] Alternatively or additionally, the I / O interface 2100 can provide access to one or more networks such as the networks 600 and 601 shown in FIGS. 6 and 7, for example, for transferring data between the controller 2000 and remote systems. For example, the controller 2000 can be networked with other computers and radiotherapy systems via the I / O interface 2100. The radiotherapy system 1000, the radiotherapy device 1010, and the controller 2000 can communicate with the networks 600 and 601, and databases and servers (e.g., dose calculation servers (e.g., distributed dose calculation framework)) and client computing devices 100. The controller 2000 can also be configured to transfer data related to medical images between different parts of the medical device. Figure 3 and Figure 4 For example, the controller 2000 can be networked with other computers and radiotherapy systems via the I / O interface 2100. The radiotherapy system 1000, the radiotherapy device 1010, and the controller 2000 can communicate with the networks 600 and 601, and databases and servers (e.g., dose calculation servers (e.g., distributed dose calculation framework)) and client computing devices 100. The controller 2000 can also be configured to transfer data related to medical images between different parts of the medical device.

[0099] The radiation therapy system 1000 can also include a number of modules (e.g., as part of the controller 2000, or as separate modules within the radiation therapy system 1000, or integrated into other components of the radiation therapy system 1000) that include programmed instructions that cause the radiation therapy system 1000 to perform different functions related to adaptive radiation therapy or other radiation therapy. For example, the radiation therapy system 1000 can include a treatment planning module operable to generate a treatment plan for the patient 1001 based on a number of data input to the system by medical personnel, a patient positioning module operable to position and align the patient 1001 relative to a desired location, such as an isocenter of a gantry, for a particular radiation therapy treatment, an image acquisition module operable to instruct the radiation therapy system and / or imaging device to acquire images of the patient 1001 prior to a radiation therapy treatment (i.e., pre-treatment / reference images for treatment planning and patient positioning) and / or during a radiation therapy treatment (i.e., in-session images during a treatment session), and to instruct the radiation therapy system 1000 and / or imaging device or other imaging device or system to acquire images of the patient 1001.

[0100] The radiation therapy system 1000 can also include a radiation dose prediction module operable to predict a dose to be delivered to the patient 1001 prior to the start of a radiation treatment, a dose calculation module operable to calculate an actual dose implemented to the patient 1001 during a radiation treatment, a treatment delivery module operable to instruct the radiation therapy device to deliver a treatment plan to the patient 1001, a correlation module operable to correlate planned images with in-treatment images acquired during a radiation therapy, a reconstruction module operable to reconstruct a three-dimensional target volume from in-treatment images, an analysis module operable to calculate a measure of displacement, and a feedback module operable to instruct the controller to stop the radiation therapy in real-time based on a comparison of the calculated displacement to a predetermined threshold (range).

[0101] The radiation therapy system 1000 can also include one or more contouring modules operable to generate contours of target volumes and other structures in pre-treatment (planning, reference) and in-treatment (treatment session) images, an image registration module operable to register pre-treatment images with subsequent in-treatment images, a dose calculation module operable to calculate cumulative dose, a contour propagation module operable to propagate contours from one image to another, a contour verification module operable to verify generated contours, a registration deformation vector field generation module operable to determine a deformation vector field (DVF) as a result of image deformation processing. The radiation therapy system 1000 can also include modules for electron density map generation, isodose distribution generation, dose volume histogram (DVH) generation, image synchronization, image display, treatment plan generation, treatment plan optimization, automatic optimization parameter generation, update and selection, and adaptive instructions and treatment information transfer. The modules can be written, for example, in C or C++ programming language. Computer program code for carrying out operations described herein can be written in any programming language, such as C or C++ programming languages.

[0102] The computing system / network 400 can be a standalone computing platform configured to securely communicate with the client device 100, the cloud storage 300, the cloud computing resources 500, and the radiation therapy system 1000 to provide some or all of the client computing device's network-available web portals, services, and / or applications for supporting requests from the client computing device 100, such as but not limited to cloud-based training and / or inference of segmentation DNN models. The computing system / network 400 can use the cloud storage 300 and the cloud computing resources 500 to deploy applications and services to the client computing device 100. The computing system / network 400 can include a plurality of interfaces and modules, as shown, for example, that can provide to a user of the client computing device 100: requesting, through the network interface 200, initialization of a training process to train one or more segmentation DNN models, and in response to the request, receiving, through the network interface 200, the trained segmentation DNN model(s) for use in the client computing device 100. Figure 7 The computing system / network 400 can also include a plurality of interfaces and modules, as shown, for example, that can provide to a user of the client computing device 100: requesting, through the network interface 200, initialization of a training process to train one or more segmentation DNN models, and in response to the request, receiving, through the network interface 200, the trained segmentation DNN model(s) for use in the client computing device 100. Figures 8A-8C The computing system / network 400 can also include a plurality of interfaces and modules, as shown, for example, that can provide to a user of the client computing device 100: requesting, through the network interface 200, initialization of a training process to train one or more segmentation DNN models, and in response to the request, receiving, through the network interface 200, the trained segmentation DNN model(s) for use in the client computing device 100.

[0103] Additionally or alternatively, the computing system / network 400 can provide some or all of the client computing device's network-available web portals, services, and / or applications for supporting cloud-based training and cloud-based application of trained segmentation DNN models.

[0104] Additionally or alternatively, the computing system / network 400 can be included in the radiation therapy system 1000.

[0105] Additionally or alternatively, the computing system / network 400 can be included in the controller 2000 of the radiation therapy system 1000.

[0106] Additionally or alternatively, the controller 2000 of the radiation therapy system 1000 can implement and execute all or part of the functionality of the computing system / network 400.

[0107] Figure 7 An example computing system / network 400 is shown. The computing / network system 400 can include a computer system 402 that includes a bus 422 or other mechanism for communicating information and a processor 404, such as but not limited to a general or special purpose processor (e.g., a graphics processing unit (GPU)), coupled to the bus 422. The processor 404 can be a processor of a cloud-based system, and / or a processor of one or more network or Internet host servers. The computer system 402 can also include an input / output module 440, such as a communication device (such as a network interface card) that provides access to the network 600 for communicating with the client computing device 100, or to the network 601 for communicating with the radiation therapy system 1000, and / or input / output ports that allow a user to interact with the computer system 402, e.g., via user input devices such as a keyboard, a mouse, a display, etc., such as, for example, an interactive graphical user interface (GUI).

[0108] The computer system 402 can also include a memory 406 that stores information and instructions that will be executed by the processor 404. The memory 406 can include any combination of random access memory (RAM), read only memory (ROM), static memory such as a disk or compact disk, or any other type of computer readable medium. For example, the computer readable medium can be any available medium that is accessible by the processor 404, and can include volatile and non-volatile media, removable and non-removable media, and communication media. Communication media can include computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism, and includes any information delivery media.

[0109] Memory 406 can store software modules that provide functionality when executed by processor 404. The modules can include, for example, an operating system 408 that can provide operating system functionality for computer system 402, and additional functionality 401 modules, such as, but not limited to, web-based modules, such as the model training monitoring interface 416 configured to provide users of client computing device 100 the ability to generate reports, view analytics, and perform other tasks. Computer system 402 can also include a training data processing interface 412 configured to provide processing instructions to one or more pre-processing virtual machines 510 of cloud computing resources 500 shown in FIG. 5B to pre-process training data stored in cloud storage 300. Figures 8A-8C Computer system 402 can also include a training application programming interface 414 configured to deploy instructions to scalable training infrastructure 520 of cloud computing resources 500 shown in FIG. 5B to train segmentation models, for example, based on the processed training data. Figures 8A-8C

[0110] Computer system 402 can also include a model database 410 configured to store a library of segmentation DNN models, including segmentation DNN models trained using the cloud-based training solutions described throughout this application, and / or segmentation DNN models received from a remote segmentation model database accessible by computer system / network 400.

[0111] Computer system 402 can also include an image database 420 configured to store a library of image data acquired and / or generated by computer system / network 400. Computer system / network 400 can also include a network interface 430 configured to securely communicate with cloud storage 300 and cloud computing resources 500 via network 600.

[0112] Cloud storage 300 can include a plurality of storage devices configured to securely store data received from client device 100, computer system / network 400, and / or radiation therapy system 1000. Cloud storage 300 can be an encrypted cloud storage that provides client computing device 100 the ability to encrypt any files and data it sends to cloud storage 300 for storage, and limit access to any of the data it stores in cloud storage 300. Cloud storage 300 can also be a cloud storage that complies with global privacy laws. Cloud storage 300 can also be a cloud storage that provides two-factor authentication and user permissions to ensure that access to client data is protected.

[0113] Cloud computing resources 500, 500A, 500B​ Figure 8A C) can include multiple computing devices, including but not limited to multiple virtual machines, servers, computing clusters, etc., which allow for secure data retrieval, computation, and transmission between the client device 100, the cloud storage 300, the computing system / network 400, and the radiotherapy system 1000.

[0114] In one or more embodiments, the cloud computing resources 500, 500A, 500B can include a first set of computing devices dedicated to pre-processing training data received from the client computing device 100, and a second set of computing devices dedicated to training the segmentation model.

[0115] In one exemplary embodiment, the first set of computing devices can include one or more virtual machines, while the second set of computing devices can include one or more virtual machines or clusters of virtual machines that emulate a computing system and mimic the functionality of a physical computer.

[0116] Alternatively, the first set of computing devices 510A can include one or more CPUs 511 and / or one or more GPUs 512, as shown in FIG. 5A, and the second set of computing devices 520A can include multiple CPUs 521 and GPUs 522. Figure 8B

[0117] Alternatively, the first set of computing devices 510B can include virtual machines, and the second set of computing devices 520B can include a cluster of servers supporting multiple virtual machines 522B, as shown in FIG. 5B. Figure 8C

[0118] In one exemplary embodiment, the second set of computing devices 520B can include Docker Swarm to establish and manage a cluster of Docker nodes as a single virtual system, and / or Kubernetes to enable parallel deployment of training applications as independent containers on Kubernetes, Swarm.

[0119] In some other exemplary embodiments, the second set of computing devices 520B can include a distributed cluster of CPU- or GPU-enabled virtual machines, including a management CPU / GPU 521B and a cluster of virtual workers 522B, which allow for scalable distribution of the training application(s) across the cluster 522B. Such distribution is configured to scale automatically based on the number of training requests received.

[0120] Figure 9 ​​An exemplary cloud-based training solution is shown in which a segmentation model 120 is trained on a training dataset 110 received from a client computing device 100 using cloud-based resources. The segmentation model 120 can be a segmentation DNN model stored in a model database 120 of the client computing device 100, or a segmentation DNN model stored in a model database 410 of a computing system / network 400 and provided to the client computing device 100, or a segmentation DNN model stored in a remote segmentation model database accessible to the client computing device 100, or a segmentation DNN model stored in a remote segmentation model database and accessed by the computing system / network 400 and provided to the client computing device 100.

[0121] The training data 110 can be a dataset generated at the client computing device 100 and stored in the image database 123, or a dataset generated at a different location from the client computing device 100 and determined to be acceptable by a user of the client computing device 100 and stored in the image database 123 or the image database 420 or a remote database, or a patient’s own dataset configured to be used by the client computing device 100 in the inference phase and stored in the image database 123. Alternatively or additionally, the training data 110 can be a dataset generated at a particular clinic / hospital / medical facility, and / or at different clinics / hospitals / medical facilities, and / or at different locations within a clinic / hospital / medical facility, and the dataset has been automatically added to the training database and stored in a central database.

[0122] In an exemplary embodiment, the training data 110 can include a medical image set containing ground truth contours of anatomical structures.

[0123] Alternatively, the training data 110 can include a medical image set and an anatomical structure set.

[0124] In an embodiment, the anatomical structure set is compiled by a user of the client computing device 100.

[0125] In an embodiment, the anatomical structure set is a previously compiled anatomical structure stored in the database 120 or accessible from a remote database.

[0126] The training data 110 and the segmentation DNN model 120 can be securely uploaded from the client computing device 100 to respective storage devices of the cloud storage 300 via the network 600. A user of the client computing device 100 can provide access to the training data 110 and the segmentation DNN model stored in the cloud storage 300 to authorized users.

[0127] Through the network interface 200 and / or the client application, a user of the client computing device 100 can initiate a request 210 for training of the stored partitioned DNN model 120, for example, by selecting a training request option displayed on a display device of the client computing device 100. The client computing device 100 can also provide information to the computing system / network 400 by which the computing system / network can access data stored in the cloud storage 300.

[0128] Upon receiving the training request 210, the computing system / network 400 is configured to automatically initiate, via the training data processing interface 412, a pre-processing operation on the training data 110 stored in the cloud storage 300 using the first set of computing devices 510. The first set of computing devices 510 can be one or more virtual machines as shown in FIG. 4. Figures 8A-8C

[0129] Alternatively, the computing system / network 400 does not automatically initiate the pre-processing operation on the training data 110 upon receiving the training request 210, but waits for a pre-processing request / input / command from the client computing device 100 before it initiates the pre-processing operation.

[0130] In an example embodiment, the pre-processing operation includes one or more operations that change the raw feature vectors of the training data into a representation that is more suitable for downstream training applications. By pre-processing the raw training data 110, the training data 110 can be converted into a format that can be more easily processed by the computing devices of the scalable training infrastructure 520 for training. For example, the pre-processing operation can include one or more of: denoising, which removes noise from the data; standardization or mean removal and variance scaling, which normalizes the training data by removing outliers; nonlinear transformation, which manipulates the raw training data to produce a single input; normalization, which organizes the training data for more efficient access; encoding of categorical features, which encodes certain data features into integers; discretization (quantization, binning), which divides continuous features into discrete values; feature binarization, which can threshold numerical features to obtain Boolean values; imputation of missing values, which can add values that are missing in the training data; generation of polynomial features, which can generate higher order terms and interaction terms from the training data; and application of custom transformers, which converts existing functions into transformers to assist with data cleaning or processing. Additionally or alternatively, the pre-processing operation can include a scaling operation to scale image resolutions to a common image resolution or an image resolution that is more suitable for training.

[0131] ​The computing system / network 400 can determine which pre-processing operations should be applied to the training data 110 based on the format of the data set that is beneficial for training the segmentation DNN model 120. After the computing system / network 400 determines which pre-processing operations would be beneficial for the stored segmentation DNN model 120, the computing system / network 400 can initiate the appropriate pre-processing operations using one or more virtual machines 510 and / or computer processing units CPUs 511, 512 of the cloud computing resources 500 (500A, 500B).

[0132] Alternatively, the computing system / network 400 can automatically initiate the previously determined pre-processing operations.

[0133] Alternatively, the computing system / network 400 can provide the client computing device 100 with the option to select the pre-processing operations to be applied to the training data via the network interface 200.

[0134] The pre-processing operations can be selected to be applied to one or both of the medical images and anatomical structures included in the training data 110.

[0135] Once processed, the processed training data 310 can be stored in the same storage device of the cloud storage 300 so as to be accessible by the computing system / network 400 for training operations.

[0136] The computing system / network 400 can then deploy a training application via the training application programming interface 414 on the computing resources of the cloud computing resources 500 (e.g., the scalable training infrastructure 520) to perform training of the segmentation DNN model 120 on the processed training data 310. The deployment of the training application includes the deployment of one or more machine learning frameworks across one or more of the computing resources, including but not limited to the Tensorflow, CUDA, and Keras frameworks, for example, which subsequently provide the support for deploying the processed training data 310, the segmentation DNN model 120, and the training computations required across different computing platforms and different tasks. The deployment of the training application also includes providing the computing resources with the processed training data 310 and all or a portion of the segmentation DNN model 120.

[0137] As each of the segmented DNN models can generate contours of multiple anatomical structures, in order to train the segmented DNN model 120 to predict contours of a desired anatomical structure (i.e., a target anatomical structure), the desired anatomical structure can be the anatomical structure that is to be predicted for the patient during inference, rather than providing data related to the entire set of anatomical structures included in the processed training data 310 as input to the computing resources, the computing system / network 400 is configured to provide a target anatomical structure dataset as input data to the computing resources along with the medical images included in the processed training data 310. Thus, the processed training data 310 provided as input data to the computing resources can include a set of medical images and a set of target anatomical structures.

[0138] In an embodiment, the user of the client computing device 100 can select one or more anatomical structures from the set of anatomical structures initially compiled / displayed to the user as target anatomical structures. This selection can be communicated to the computing system / network 400 via the network interface 200. Upon receiving this information, the computing system / network 400 is configured to identify data associated with the selected target anatomical structures in the processed training data 310. This can be done automatically using the structure ID, or manually by allowing the user to map the anatomical structures. Once the identification is made, the computing system / network 400 is configured to deploy the processed training data 310 associated with the selected anatomical structures (i.e., target anatomical structures) along with the segmented DNN model 120 on the computing resources.

[0139] In an embodiment, the selection of the target anatomical structures is initiated by the computing system / network 400 via the network interface 200.

[0140] In an embodiment, the selection of the target anatomical structures by the user is automatically sent to the computing system / network 400 as part of the training request 210.

[0141] In an embodiment, the list of anatomical structures can be provided to the user by displaying, for example, a list of anatomical structures on the display screen of the client computing device 100, and providing the client computing device 100 with an option to select a target anatomical structure from the list of anatomical structures. The list of anatomical structures can be automatically provided to the user of the client computing device 100 upon the user initiating the training request 210.

[0142] Alternatively, the user can be provided with the anatomical structures by prompting the user to select a list of anatomical structures displayed in the image database 123, or the database 120, or any other remote database available to the client computing device 100, and manually selecting a target anatomical structure from the list.

[0143] The computing system / network 400 is configured to deploy the training application across a required suitable number of computing resources of the cloud computing resources 500, for example, based on the number of received training requests, and / or based on the amount of tasks each computing device will need to perform, and / or based on the complexity of each request and / or task. The computing system / network 400 is configured to determine the required computing resources for each request and to scale the computing resources appropriately. Figure 12A-12F An exemplary deployment and execution of the training process for obtaining the trained segmentation DNN model is shown.

[0144] Once the execution of the model training is completed and / or terminated, the trained segmentation DNN model 430 can be provided to the client computing device 100 for further use in inference to predict the contours of the patient’s anatomical structure.

[0145] In embodiments, providing the trained segmentation DNN model 430 includes providing a link to the client computing device 100 so that the trained segmentation DNN model 430 can be downloaded by the client computing device 100 and stored in the database 120.

[0146] In embodiments, providing the trained segmentation DNN model 430 includes providing a link to the client computing device 100 so that the trained segmentation DNN model 430 can be accessed by the client computing device 100 through the computing system / network 400.

[0147] In embodiments, providing the trained segmentation DNN model 430 includes providing access to a segmentation service bundle including the trained segmentation DNN model 430 through a payment of a service fee.

[0148] In embodiments, providing the trained segmentation DNN model 430 includes providing access to the trained segmentation DNN model 430 via a software package that integrates different segmentation processes.

[0149] In embodiments, the trained segmentation DNN model 430 can be stored in the database 120 and / or the image database 420, and / or an accessible remote database, and / or a cloud storage 300 to be accessed by the client computing device 100.

[0150] The computing system / network 400 is further configured to provide a monitoring service to the client computing device 100 via the model training monitoring interface 415. Through this service, a user of the client computing device 100 can monitor the training progress, training performance, and training completion via the network interface 200.

[0151] The computing system / network 400 is also configured to deploy an inference process on one or more computing resources via an inference engine 409. For the inference process, the computing system / network 400 may apply a trained segmentation DNN model 430 to the patient dataset 30 to generate a first set of contours. Simultaneously or sequentially, the computing system / network 400 may apply another segmentation DNN model (such as a previously trained segmentation model stored in a model database 410) to the patient dataset 30 to generate a second set of contours. The computing system / network 400 is also configured to compare the first set of contours and the second set of contours, and provide the results, for example, in the form of a training log, to a client computing device 100, allowing the user to compare how well the newly trained segmentation DNN model 430 performs on the patient data with how well an already available segmentation DNN model performs on the same patient data.

[0152] Alternatively or additionally, the computing system / network 400 may also provide access to live monitoring of the training process, and the option for the user of the client computing device to terminate the training process when the user of the client computing device determines that the segmentation DNN model has been trained satisfactorily.

[0153] In another exemplary embodiment, such as Figure 10 As shown, instead of uploading the segmentation DNN model from the client computing device 100 to the cloud storage device 300 for training by the computing system / network 400 using processed training data 310, a list containing multiple trained segmentation models stored in the model database 410 of the computing system / network 400 can be made available to the client for selection via a web browser 200A. After selecting a segmentation DNN model 411 from the list, the user can proceed as described in the reference... Figure 9 The embodiments described herein are used to initiate preprocessing and training requests 220, and can also be applied to Figure 9 The preprocessing, training, and monitoring steps.

[0154] Additionally, since the trained segmentation DNN model 440 is a segmentation model already available and trained on patient data, the user can compare the trained DNN segmentation model 430 with the trained segmentation model 440 and determine which trained segmentation model is more suitable for inference.

[0155] Optionally, different computing resources can be used to simultaneously train the client-provided segmentation DNN model 120 and the segmentation DNN model 411, and the training results can be monitored.

[0156] Optionally, the processed training data 310 can include additional subsets of data. For example, the data set 310 can include a validation set that is used to track the quality of the segmentation DNN model during training but is not otherwise used as input to the DNN model during training. Alternatively or additionally, the processed training data 310 can include a test subset that is used only after training to quantify the quality (e.g., accuracy, dice score) of the trained model and / or to verify that the model has not over-learned or under-learned the data.

[0157] In another example embodiment, as shown in FIG. 5B, instead of uploading the training data and the segmentation model from the client computing device 100 to the cloud storage 300, the user is provided with an option to select a training data set 111 from a set of patient data stored in the image database 420 and a segmentation DNN model 412 from the segmentation model database 410 of the computing system / network 400 via the interface 200B. Once the selections are made, the computing system / network 400 can automatically perform the preprocessing of the training data set 111 to obtain the processed training data 320 and train the segmentation DNN model 412 using the processed training data 320, similar to the preprocessing and training processes described for the embodiments of FIGS. 4A and 4B. Figure 11 Figure 9 and 10

[0158] The trained segmentation DNN model 450 can be provided to the client computing device 100 in a similar manner as previously described.

[0159] The computing system / network 400 can also provide information on the training progress, training performance, and training completion.

[0160] Figure 12A-12F Training applications are shown to scale across different deployment configurations of the scalable training infrastructure 520. The scalable training infrastructure 520 can be a CPU or GPU managed (521B) cluster of worker virtual machines 522B (W1-Wn). There are multiple ways to train the segmentation DNN model. Figure 12A ​​A data-parallel distributed training process is shown in which a training application 530 containing a processed training dataset 310 / 320 including a medical image set 330 and a target anatomical structure set 340 is deployed across a cluster of virtual machines 522B by a management processor, e.g., a management CPU / GPU 521B, such that the image set 330 is divided into n image subsets (image subset 1 - image subset n), each image subset including a portion of the image set 330, and the segmentation DNN model 120 / 411 / 412 is replicated on n virtual machines 522B. In operation, each of the virtual machines 522B performs forward and backward passes on each image subset. When the virtual machines 522B complete the process, it shares updates with the other virtual machines 522B, and the acquired values are used to compute updated weights for the entire image set 330, and the weights are synchronized across the replicated model 120 / 411 / 412. The results 540, which can include a trained training log 541 for monitoring the ongoing trained, and a trained model file 542 for accessing the trained segmentation model, can be made accessible to a user of the client computing device 100 via the computing system / network 400. The number n of virtual machines to be used can be determined based on, for example, the size of the image set 330 and / or other applicable considerations, such as, but not limited to, the number of segmentation requests received by the management processor and / or the number of different segmentation requests received by the management processor. The number of virtual machines 522B to be used can also be determined based on the data size of one or more segmentation requests, and / or the size of the segmentation model to be trained, and / or the number of segmentation requests.

[0161] In Figure 12BIn the illustrated model parallel distributed training process configuration, the segmented DNN model 120 / 411 / 412 is divided across n virtual machines 522B, while the image set 330 is replicated across n virtual machines 522B. The segmented DNN model 120 / 411 / 412 can be divided into n model portions (model portion 1 - model portion n). In example embodiments, each virtual machine 522B can hold a layer of the DNN network. During forward and backward passes, information can be passed between the virtual machines 522B. Results 540, including a training log 541 and a trained model file 542, can be made accessible to a user of the client computing device 100 via the computing system / network 400. The number of virtual machines n to be used can be determined based on the size of the image set 330 and / or other applicable considerations, such as but not limited to the number of DNN network layers, and / or the number and / or type of segmentation requests received by the manager 521B. The number of virtual machines 522B to be used can also be determined based on the data size of one or more segmentation requests, and / or the size of the segmented model to be trained, and / or the number of segmentation requests.

[0162] In Figure 12C In the illustrated configuration, in which a first request is made to train a segmented DNN model using image set 1 1230, and a second request is made to train the same segmented DNN model using image set 2 1250 for the same target set of anatomical structures 340, a segmented application deployment can be made such that the segmented DNN model is replicated across a first virtual machine 522B and a second virtual machine 522B, where the first virtual machine 522B performs forward and backward passes for image set 1 1230, and the second virtual machine 522B performs forward and backward passes for image set 2 1250. When the virtual machines 522B complete the process, it shares updates with the other virtual machine 522B, and the acquired values are used to compute updated weights for both image data sets, and the weights are synchronized across the replicated model. Results 540, including a training log 541 and a trained model file 542, can be made accessible to a user of the client computing device 100 via the computing system / network 400. The number of virtual machines 522B to be used can also be determined based on the data size of one or more segmentation requests, and / or the size of the segmented model to be trained, and / or the number of segmentation requests.

[0163] In an alternative configuration, as Figure 12DAs shown, each of the image set 1 1230 and the image set 2 1250 can be replicated on a respective number of virtual machines 522B, and the split DNN model can be divided into a respective number of model portions (model portion 1 - model portion n) for the image set 1 1230 and model portion 1 - model portion n for the image set 2 1250. Each virtual machine 522B can hold a layer of the DNN network. During forward and backward passes, information can be passed between the virtual machines 522B. The results 540, including the training log 541 and the trained model file 542, can be made accessible to the user of the client computing device 100 via the computing system / network 400. The number n of virtual machines to be used can be determined based on the respective sizes of the two image sets, and / or based on other applicable considerations, including but not limited to the number of split requests received by the manager 521B and / or the number of DNN network layers of the model.

[0164] In another embodiment in which the image set 1 1230 is used to train a first split DNN model and a second split DNN model (model 1, model 2), as shown in FIG. 12B, the image set 1 1230 can be replicated across a first virtual machine 522B and a second virtual machine 522B, and the first virtual machine 522B can hold a first model (model 1) and the second virtual machine 522B can hold a second model (model 2). Each of the virtual machines performs forward and backward passes without passing information between the virtual machines. When the virtual machines 522B complete the process, the obtained values are used to compute updated weights for their respective image sets. The results 540, which can include the training log 541 and the respective trained model files 542 (i.e., model file 1, model file 2), can be made accessible to the user of the client computing device 100 via the computing system / network 400. Figure 12E

[0165] In an alternative configuration, as shown in FIG. 12C, the image set 1 1230 can be replicated across a first virtual machine 522B and a second virtual machine 522B, and the first virtual machine 522B can hold a first model (model 1) and the second virtual machine 522B can hold a second model (model 2). Each of the virtual machines performs forward and backward passes without passing information between the virtual machines. When the virtual machines 522B complete the process, the obtained values are used to compute updated weights for their respective image sets. The results 540, which can include the training log 541 and the respective trained model files 542 (i.e., model file 1, model file 2), can be made accessible to the user of the client computing device 100 via the computing system / network 400. Figure 12F ​As shown, the image set 1 1230 can be divided into n image subsets (image subset 1 - image subset n) across n virtual machines 522B, and the first and second models (model 1, model 2) are replicated across respective sets of n virtual machines 522B. In operation, the first set of virtual machines 522B performs forward and backward passes for each image subset 523. When the virtual machines 522B complete the process, it shares the updates with the other virtual machines 522B, and the obtained values are used to compute updated weights for the entire image set 1 1230, and the weights are synchronized across the replicated models 120 / 411 / 412. The result 540, including the training log 541 and the trained model file 542, can be made accessible to the user of the client computing device 100 via the computing system / network 400. The number n of virtual machines to be used can be determined based on the size of the image set 1 1230 and / or other applicable considerations. The number of virtual machines 522B to be used can also be determined based on the data size of one or more segmentation requests, and / or the size of the segmentation models to be trained, and / or the number of segmentation requests.

[0166] In Figure 12A-12F application deployment configurations, the updates at different virtual machines 522B can be done in a synchronous or asynchronous manner.

[0167] Although a particular deployment configuration is shown in Figure 12A-12F , it should be understood that any other combination of data parallel and / or model parallel deployment configurations across n virtual machines is possible, and any and all possible configurations included herein are possible. It should additionally be understood that scaling up or down of computing resources, such as virtual machines in the cloud resources 500, can be done as needed, including the data size of one or more segmentation requests, and / or the size of the segmentation models to be trained, and / or the number of segmentation requests.

[0168] It should further be understood that the computing system / network 400 can be a management processor, and the management processor and / or the computing system / network 400 is configured to manage the cluster of virtual machines 522B, collect the result 540, scale resources, manage monitoring of the training process, and handle any failures in the segmentation training process.

[0169] In one or more embodiments, the computing system / network 400 is further configured to enable secure access to the training log 541 by the user of the client computing device 100 to enable the user to monitor the segmentation model training process progress. The training log 541 can be accessed directly through the network interface 200, or can be downloaded or streamed through the computing system / network 400.

[0170] In one or more embodiments, the computing system / network 400 is also configured to allow a user of the client computing device 100 to stop training of the segmentation DNN model through the network interface 200.

[0171] Referring to Figures 13A-13C An exemplary operation for segmentation model training will be described. Figure 13A And 13C is a generalized schematic of a cloud-based segmentation model training operation. In a first step, a customer can securely upload raw data, such as but not limited to training data desired for training a segmentation model, to a secure cloud storage location. The uploaded raw data can then be pre-processed using cloud-based computing resources, such as but not limited to virtual machines. The pre-processed data can be securely stored back into the cloud-based storage. The customer can then select a target anatomical structure and provide access to the pre-processed training data. A cloud-based computing network, including scalable training infrastructure, can then access the pre-processed training data so that a plurality of computing machines, such as but not limited to a cluster of managed virtual machines, can be used to train a segmentation model to generate a trained segmentation model based on the pre-processed data and the selected anatomical structure. The trained model is then made accessible to the customer for further use.

[0172] Figure 13C is Figures 13A-13B is an exemplary process flow S200 diagram of the various steps of the training infrastructure for the segmentation DNN model of

[0173] The training data set can include a set of two-dimensional (2-D) or three-dimensional (3-D) medical images or image slices having a ground truth profile of an anatomical structure imaged by different pixels or voxels. The training data set can have images that have been segmented (i.e., outlined), where a ground truth label map provides a known anatomical structure label for each pixel of representative image slices of the training images. In other words, the pixels of the ground truth label map can be associated with a known anatomical structure.

[0174] Along with the set of medical images, a list of anatomical structures is also uploaded to the cloud storage device along with the segmentation model desired to be trained. In an embodiment, the training data set includes the set of medical images and the list of anatomical structures. The training data set can also include additional ground truth information, such as, for example, cut plane locations and / or user-defined ROIs (e.g., bounding boxes).

[0175] In step S202, preprocessing of the training data can be initiated. The preprocessing can be initiated by the user via the web interface by selecting a preprocessing option displayed to the user on the display screen of the computing device 100. The preprocessing option can be displayed to the user after termination of the uploading of the training dataset to the cloud storage device, or as a first step of the initiation of the training process of S203. The user can be given the option to select one or more of the available preprocessing operations. Alternatively, the preprocessing operations comprise a previously determined set of preprocessing operations.

[0176] Then in S203, the user is given the option to initiate the segmentation DNN model training process, e.g. via the web interface. Upon selection of the training option, the user can be prompted to select the target anatomical structure from a list of stored anatomical structures. Upon selection of the target anatomical structure, in S204, the training application is initiated, through which a framework to support data communication and data flow between different devices, platforms and networks and the operations and tasks required for the training process are deployed on a scalable cluster of cloud computing devices. Through the deployment of the training application, the preprocessed training data from the cloud storage device associated with the target anatomical structure and the segmentation model are used as input data across multiple cloud computing devices. The deployment can be done across a suitable number of cloud computing devices as required. For example, the computing devices can be scaled up and down based on the number of training requests received, and / or based on the number of tasks each computing device will need to perform, and / or based on the complexity of each request and / or task. The computing system / network 400 is configured to determine the computing resources required for each request and scale the computing resources appropriately.

[0177] The training process comprises determining one or more parameters of the nodes in the hidden layers of the segmentation DNN model, e.g. through an iterative process of changing the parameters such that the output of the segmentation DNN model more closely matches the corresponding ground truth. For example, the nodes in the hidden layers can comprise filters or kernels, the parameters of which (e.g. kernel weights, size, shape or structure) can be adjusted during the training process.

[0178] In an exemplary embodiment, each hidden layer of the segmentation DNN model can be represented by a virtual machine of a cloud computing cluster emulating the neural DNN engine. During the iterative model training process, training data is propagated through the nodes of the hidden layers of the DNN model (i.e., through the virtual machines), and in S205, the resulting data from the hidden layers is provided to the nodes of the output layer of the DNN model (i.e., the results from each virtual machine). The resulting data from the output layer S206 (i.e., the resulting profile) can be monitored by the user in S207 via the training log (541), for example, in order to determine in S208 whether the resulting data is acceptable. The purpose of the training S204 is to train the segmentation DNN model to perform automatic segmentation of anatomical structures in the image(s) by mapping the input data (i.e., the medical image) to the exemplary output data (i.e., the ground truth profile). In some embodiments, the training S204 can involve finding weights that minimize the training error (e.g., as determined by a loss function) between the ground truth profile and the estimated profile generated by the deep learning engine.

[0179] In one or more embodiments, the user can determine that the resulting data S206 is acceptable by comparing the resulting data S206 to the ground truth via a loss function.

[0180] Alternatively, the user can determine in S208 that the resulting data S206 is acceptable by comparing the resulting data S206 to resulting data obtained, for example, by applying an existing model (e.g., a previously determined model stored in a database) to the processed training data, or to resulting data obtained by applying an existing model to an existing training data set.

[0181] Alternatively, for example, the user can determine whether the resulting data S206 is acceptable by comparing the resulting data S206 to a predetermined threshold.

[0182] Alternatively, in S206, the user can also set a predetermined number of iterations to obtain the resulting data, and when the number of iterations is reached, the resulting data is accepted.

[0183] Alternatively, the user can determine whether the resulting data S206 is acceptable by verifying the degree of matching of the output to the correct / desired / threshold output during the training feedback received via the training log 541.

[0184] Once the user accepts the result data in S208, the segmentation DNN model can be fixed in S210. In S211, the trained segmentation DNN model can be provided to the user for download or streaming, or in S211 the trained segmentation DNN model is accessed via an additional service. Otherwise, the training S200 proceeds to S209 in which the segmentation DNN model is modified, for example by adjusting the parameters of the hidden layer nodes, in order to improve the match between the output and the desired output. The training process S204 can repeat iterations until the desired iteration criteria are met in S208. The segmentation DNN model is then considered to be trained.

[0185] The trained segmentation DNN model of S210 can also be stored in an image segmentation model database of the computing system / network 400, or in an image segmentation model database of the client computing device 100, or in an image segmentation model database of the secure cloud storage 300.

[0186] Once the execution of the model training is completed and / or terminated by the user, the trained segmentation DNN model S210 can be provided to the client computing device 100 for further use in inference to predict the contours of the patient’s anatomical structures.

[0187] Figure 14 An exemplary inference process S300 is shown in FIG. 3. The process S300 uses the trained segmentation DNN model of S210, for example, to process a medical image(s) of a patient to automatically segment (i.e., contour) unknown anatomical structures shown therein. In the inference phase, the process S300 can start in S301 in which a patient dataset comprising one or more medical images is provided to the trained segmentation DNN model S210. The image(s) of the patient can be acquired using any medical imaging modality, and need not be of the same modality as the images used to train the segmentation DNN model.

[0188] The process S300 can then proceed to S302 in which the medical image(s) are processed by the trained segmentation DNN model S210 and output contours data based on its training. In some embodiments, the contour data can be combined with the original medical image(s) in S303 such that the contours are superimposed on the respective anatomical structures in the images, for example for visualization by a user or for radiotherapy planning. Alternatively, the segmentation DNN model S210 can directly produce contours on the medical images as output without a separate combining step S303.

[0189] In embodiments, the inference process is deployed on one or more computing resources by the computing system / network 400 via the inference engine 409.

[0190] Alternatively or additionally, the inference process is deployed by the client computing device 100 using the inference engine 112.

[0191] Alternatively or additionally, in some embodiments, non-imaging information can be used with the medical image(s) of the patient at the inference stage. For example, non-imaging information, such as field parameters of a planned radiotherapy, can be used to define appropriate cutting planes. In radiotherapy planning, depending on whether the organs are serial or parallel, the field off parts of the anatomical structure can not be of interest. If the field geometry is known, the field off cutoff planes can be detected. In particular, at S301, the one or more medical images can be provided with non-imaging information. The non-imaging information can be provided to the trained segmentation DNN model S210, e.g., for generating the contour data in S302.

[0192] It will be appreciated that a cloud-based system and method for training a segmentation model is provided, including an interface and a computing infrastructure, the interface configured to: allow a user to upload and store training data in a storage device of a cloud-based network; provide access to the training data stored in the storage device; initiate a request for training a segmentation model; monitor training of the segmentation model; and download the trained segmentation model; the computing infrastructure configured to: pre-process the training data using a first set of computing resources of the cloud-based network to obtain processed training data, and store the processed training data in the storage device; deploy a training application on a second set of computing resources of the cloud-based network to train the segmentation model based on the processed training data; provide access to monitor the training; and provide access to the trained segmentation model.

[0193] It will also be appreciated that a cloud-based scalable system and method for training a segmentation model is provided, the system configured to: access training data stored in a storage device of a cloud-based network; pre-process the training data using a first set of computing resources of the cloud-based network, and store the pre-processed training data in the storage device; deploy a training application on a second set of computing resources of the cloud-based network to train the segmentation model based on the pre-processed training data; provide access to monitor the training of the segmentation model; and provide access to the trained segmentation model.

[0194] It will also be appreciated that a system for training a segmentation model is provided, the system comprising: an interface configured to allow a user to select training data from a database and to initiate a request to train a segmentation model using the selected training data; and a computing infrastructure configured to pre-process the selected training data, deploy a training application to train the segmentation model based on the pre-processed training data, and provide access to the trained segmentation model.

[0195] The training application can comprise one or more machine learning frameworks configured to support deployment of data and computation across different platforms and different tasks.

[0196] The segmentation model to be trained can be provided by the user through the interface.

[0197] The selected training data can comprise a plurality of medical images and a plurality of anatomical structures.

[0198] The automatic segmentation model can be a segmentation neural network model.

[0199] The training can comprise training the segmentation neural network model to approximate contours of the selected anatomical structures in the medical images.

[0200] It will also be appreciated that a system for training a segmentation model is provided, the system configured to: access training data stored in a storage device of a cloud-based network; use a first set of computing resources of the cloud-based network to pre-process the training data and store the pre-processed training data in the storage device; deploy a training application on a second set of computing resources of the cloud-based network to train the segmentation model based on the pre-processed training data; provide access to monitor the training of the segmentation model; and provide access to the trained segmentation model.

[0201] The first set of computing resources can comprise a computing device, a computing server, or a virtual machine.

[0202] The second set of computing resources can comprise one or more virtual machines.

[0203] The second set of computing resources can comprise a scalable cluster of computing engines comprising a management computing engine and a plurality of worker computing engines.

[0204] The cluster can be configured to scale based on a number of segmentation model training requests.

[0205] The training application can comprise one or more machine learning frameworks configured to support deployment of computation across different platforms and different tasks.

[0206] The deployment of the training application can be initiated at a remote device through a web-based training interface.

[0207] The training data can be received from a remote device and access to the training data can be provided by the remote device.

[0208] The segmentation model to be trained can be received from a remote device.

[0209] The segmentation model to be trained can be provided by the system.

[0210] The system can also be configured to provide a plurality of segmentation models and to allow a remote device to select one or more segmentation models from the set for training via a web-based training interface.

[0211] The training data can comprise a set of medical images comprising a plurality of medical images and a set of anatomical structures comprising a plurality of anatomical structures.

[0212] The segmentation model can be an automatic segmentation model.

[0213] The automatic segmentation model can be a segmentation neural network model.

[0214] The training can comprise training the segmentation neural network model to approximate contours of different anatomical structures in the medical images.

[0215] The different anatomical structures to be contoured can be selected from the set of anatomical structures stored in the storage device.

[0216] The selection can be made at the remote device via a network interface.

[0217] The anatomical structures can comprise one or more of: a target, an organ, a tissue, a blood vessel, and a bone.

[0218] Providing access for monitoring the training can comprise providing output checkpoints at different intervals during the training or upon request from the remote device.

[0219] The system can also be configured to provide information on training progress, training performance, and training completion.

[0220] The system can also be configured to apply the trained segmentation model on a data set and to apply a different segmentation model on the data set, compare the results, and provide the compared results as information on training performance.

[0221] It is also to be understood that a method for cloud-based training of a segmentation model is provided, comprising: uploading and storing training data in a storage device of a cloud-based network; initiating a request for training of a segmentation model; the initiating automatically deploying a training application on a computing resource of the cloud-based network, by which the segmentation model is trained using the stored training data; monitoring the training; and accessing the trained segmentation model.

[0222] It is also to be understood that a non-transitory computer readable storage medium having stored thereon a sequence of programming instructions embodied as computer readable program code is also disclosed, and that a computer processing system uses the sequence of programming instructions stored on the computer readable storage medium to perform any one of the method steps described herein, or a combination thereof.

[0223] It should be understood that aspects of the disclosed subject matter can be implemented in whole or in part in hardware, hardware programmed by software, software stored on a computer-readable medium (e.g., a non-transitory computer-readable medium), or any combination of the above. For example, components of the disclosed subject matter, including such as controllers, modules, models, neural networks, or any other features, can include, but are not limited to, a personal computer, or workstation, or other such computing system including a processor (e.g., a graphics processing unit), a microprocessor, a microcontroller device, or consisting of control logic including integrated circuits such as, for example, an application-specific integrated circuit (ASIC). Features discussed herein can be executed on a single or distributed processor(s) (single core and / or multi-core), by components distributed across multiple computers or systems, or by components co-located in a single processor or system. For example, aspects of the disclosed subject matter can be implemented via a programmed general purpose computer, an integrated circuit device (e.g., an ASIC), a digital signal processor (DSP), an electronic device programmed with microcode (e.g., a microprocessor or microcontroller), a hard-wired electronic or logic circuit, a programmable logic circuit (e.g., a programmable logic device (PLD), a programmable logic array (PLA), a field programmable gate array (FPGA), a programmable array logic (PAL)), software stored on a computer-readable medium or signal, an optical computing device, a networking system of electronic and / or optical devices, a special purpose computing device, a semiconductor chip, a software module or object stored on a computer-readable medium or signal.

[0224] When implemented in software, the functions can be stored or transmitted over as one or more instructions or code on a computer-readable medium. The steps of a process, method, or algorithm disclosed herein can be embodied in a processor-executable software module which can reside on a computer-readable medium. The instructions can be compiled from a source code instruction set or can be provided as high-level language instructions. The programming sequences and associated data can be stored in a computer-readable medium (e.g., a non-transitory computer-readable medium) such as a computer memory or storage device, which can be any suitable memory device, such as, but not limited to, read-only memory (ROM), programmable read-only memory (PROM), electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), flash memory, a disk drive, etc.

[0225] As used herein, computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media can be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.

[0226] Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a transmission medium such as coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the transmission medium is included in the definition of computer-readable medium. Further, it should be appreciated that any processes, methods, or algorithms described herein can be expressed as a set of codes and / or instructions that can be directly executable by the machine and / or computer-readable medium, which can be incorporated in the computer program product.

[0227] Those skilled in the art will readily recognize that the above description is not exhaustive and that various aspects of the disclosed subject matter can be practiced in different ways than specifically disclosed above. Indeed, the disclosed subject matter is in no way limited to the embodiments described above, but can be practiced with any known or later developed systems, structures, devices, and / or software by those having ordinary skill in the art without departing from the scope of the disclosed subject matter as provided by the functional description herein.

[0228] In this application, the use of the singular includes the plural, and the use of "or" means "and / or", unless specifically stated otherwise. Furthermore, the use of the term "including", as well as other forms such as "include", "includes", and "included", is intended to be broad and encompass the terms "comprising" and / or "comprise". Also, the term "comprising" is used in the sense of "including" and means that other elements can also be present. More particularly, the use of the term "comprising" means that other elements can also be present, in addition to the listed elements.

[0229] Any range described herein is to be understood to include the endpoints and all values between the endpoints. Whenever "substantially", "about", "approximately", "near" or like terms are used in conjunction with a particular value, unless specifically stated otherwise, the intended meaning is a variation of up to and including 10% from the stated value.

[0230] The present disclosure allows for multiple alternatives, modifications, and variations. Although specific examples are illustrated and described herein, it will be appreciated that any arrangement can be substituted for the specific ones shown and described. For example, the disclosed features can be combined, rearranged, omitted, etc. to produce additional embodiments, and certain disclosed features can be beneficially used without corresponding use of other features. Thus, the applicant intends any patent to cover all such alternatives, modifications, equivalents, and variations that fall within the spirit and scope of the present invention.

Claims

1. A system for training a segmentation model, comprising: The interface is configured to allow users to: Upload and store the training data in a cloud-based network storage device; Provide access to the training data stored in the storage device; Initiate a request to train the segmentation model; Monitor the training of the segmentation model; as well as Download the trained segmentation model; The computing infrastructure is configured as follows: The first set of computing resources of the cloud-based network is used to preprocess the training data to obtain the processed training data, and the processed training data is stored in the storage device. A training application is deployed on a second set of computing resources in the cloud-based network to train the segmentation model based on the processed training data; Provide access to monitor the training; as well as Provides access to the trained segmentation model. The preprocessing of the training data includes applying one or more operations from a plurality of operations to transform the training data into a representation that can be processed by the second set of computational resources, and The plurality of operations are configured to be displayed on a display device for selection by the user via the interface, the selection being made based on the second set of computing resources.

2. The system according to claim 1, wherein the interface is a web-based interface.

3. The system according to claim 1, wherein the first set of computing resources includes computing devices, computing servers, or virtual machines.

4. The system of claim 1, wherein the second set of computing resources includes one or more virtual machines.

5. The system of claim 1, wherein the second set of computing resources comprises a scalable cluster of computing engines, the cluster including a management computing engine and a plurality of working computing engines, the cluster being configured to be scaled up or down to deploy the training application across a required number of working computing engines as determined by the computing infrastructure, wherein the computing infrastructure determines the required number of working computing engines based on one or more of the following: the number of segmentation model training requests, the complexity of each request, and the amount of task that needs to be performed by each of the plurality of working computing engines.

6. The system of claim 1, wherein the training application includes one or more machine learning frameworks configured to support the deployment of data and computation across different platforms and different tasks.

7. The system of claim 1, wherein the segmentation model to be trained is provided by the user through the interface.

8. The system of claim 7, wherein the segmentation model to be trained is selected from a plurality of segmentation models provided to the user from the computing infrastructure via the interface.

9. The system according to claim 1, wherein the training data includes: A medical image set including multiple medical images; and an anatomical structure set including multiple anatomical structures.

10. The system according to claim 9, wherein the segmentation model is an automatic segmentation model.

11. The system of claim 10, wherein the automatic segmentation model is a segmentation neural network model.

12. The system of claim 11, wherein the training comprises: The segmentation neural network model is trained to approximate the contours of the multiple anatomical structures in the multiple medical images.

13. The system of claim 12, wherein the anatomical structure comprises one or more of the following: a target, an organ, a tissue, a blood vessel, and a bone.

14. The system of claim 1, wherein providing access to monitor the training comprises: Output checkpoints are provided at different intervals during training, or at the user's request.

15. The system of claim 14, wherein the computing infrastructure is further configured to provide access to information regarding training progress, training performance, and training completion.

16. The system of claim 15, further configured to: The trained segmentation model is applied to the dataset obtained from the user; Apply another segmentation model to the dataset obtained from the user; The results from the segmentation model trained on the dataset obtained from the user and the other segmentation model are compared to obtain a comparison result; as well as The comparison results are evaluated against a predetermined threshold.

17. The system of claim 15, further configured to: Apply the trained segmentation model to the dataset obtained from the user; and The segmentation results obtained from the segmentation model trained on the dataset obtained from the user are evaluated against a predetermined threshold.

18. The system of claim 1, wherein providing access to the trained segmentation model comprises: The user is provided with training logs and model files for downloading the trained segmentation model.

Images