Machine learning approach to segmentation of an organ image

Abstract

A method comprising: receiving a set of images, receiving non-binary segmentation values associated with each image assigning a probability that a pixel corresponds to a specified anatomical structure

Description

BACKGROUND

[0001] Radiation therapy can involve a planning stage and a treatment stage. In the planning stage, an imager can be used to obtain internal medical images of a lesion. These images can be used such as to measure the location, size, contour, and amount of critical structures to be treated. This information can be used to establish a dose distribution, and various other irradiation parameters in an attempt to irradiate the lesion while minimizing damage to surrounding healthy tissue. These images can be factored into a subsequent treatment planning stage, e.g., involving the delineation of target volumes and normal critical organs in the images. Accurately tracking the various objects, such as a tumor, healthy tissue, or other aspects of patient anatomy can be challenging, e.g., when the patient is moving (e.g., breathing), when an image can include noise, or when an image represents organs proximate to one another. SUMMARY

[0002] A medical image segmentation technique can involve some form of expert human labeling to provide accurate and consistent identification of anatomic structures of interest. For example, a physician can manually label a medical image to identify whether certain image pixels or image voxels represent a specified anatomical structure. A challenge of approaches relying on entirely manual labeling is that a resulting treatment plan can be highly subjective based on, e.g., the contouring techniques of the particular expert and the relative perception of “risk” held by the particular expert. Manual labeling can also be time-consuming and in certain approaches can yield misleading or difficult-to-interpret visual characterizations, such as a chart indicating binary segmentations and without visual indicia of the particular expert user’s confidence in their manual labeling.

[0003] This document describes a method for supporting radiation treatment planning. The method can include, e.g., obtaining a set of images of at least one patient and receiving a set of non-binary segmentation values. Each value in the set can be within a range of possible segmentation values. The set of non-binary segmentation values can assign a probability that an individual pixel or voxel of one of the set of images corresponds to a specified anatomical structure of the at least one patient.

[0004] In an example, a machine learning model can be trained, e.g., using the set of nonbinary segmentation values paired with respective images of the set of images. Here, the machine learning model can be trained such as to output a probability map indicating nonbinary segmentation values for pixels or voxels of a provided image. In an example, the probability map, when output by the trained machine learning model can indicate a probability that an individual pixel of an image of the set of images represents a contour of the specified anatomical structure of the at least one patient. In an example, the set of non-binary segmentation values can be received via radiation oncologist labels of images in the set of images.

[0005] In an example, a probability map can be generated of the first tissue image based on the plurality of non-binary segmentation values, the probability map including a representation of user-confidence of the association of the individual pixel or voxel to the first anatomical structure. In an example, the probability map can be used as an input to a trained machine learning (ML) model. For example, the ML model can be trained such as to establish or adjust a threshold segmentation value for associating an individual pixel or voxel of a second tissue image to the anatomical structure of the patient. For example, the method can include confirming or correcting an individual user input non-binary segmentation value based on the threshold segmentation value. In an example, the method can include using the confirmed or corrected non-binary segmentation values to determine a deformable image registration (DIR) including the first tissue image and the second tissue image.

[0006] In an example, generating the probability map can be also based on one of measured organ motion over time, measured organ deformation over time, or image distortion. In an example, the method can include using the non-binary segmentation values to generate a visual representation included such as for planning dosage of radiation treatment. For example, the visual representation can include a dose-volume histogram (DVH). In an example, volume data of the DVH corresponding to an individual dosage can include a range of values based on the non-binary segmentation values. In an example, the DVH can include a planar representation, including an area, of the respective ranges of values for a plurality of dosages.

[0007] In an example, the planar representation can include a plurality of sections each corresponding to an individual accuracy probability of the non-binary segmentation values. In an example, individual sections can be represented by different colors. In an example, the method can include automatically detecting an outside body contour of the first anatomical structure. In an example, the method can include superimposing at least one visual identifier of the first anatomical structure on the first tissue image, superimposing being based on the nonbinary segmentation values.

[0008] In an example, the method can include performing a radiotherapy dosage calculation based in part on the plurality of confirmed or corrected non-binary segmentation values. In an example, the method can include performing a plurality of radiotherapy dosage calculations each corresponding to an individual pixel or voxel of the first tissue image based on the plurality of confirmed or corrected respective non-binary segmentation values.

[0009] Each of the non-limiting examples described herein can stand on its own, or can be combined in various permutations or combinations with one or more of the other examples.

[0010] This Summary is intended to provide an overview of the subject matter of the present patent application. It is not intended to provide an exclusive or exhaustive explanation of the invention. The detailed description is included to provide further information. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In the drawings, which are not necessarily drawn to scale, like numerals describe substantially similar components throughout the several views. Like numerals having different letter suffixes represent different instances of substantially similar components. The drawings illustrate generally, by way of example but not by way of limitation, various examples discussed in the present document.

[0012] FIG. 1 depicts an example of portions of a user interface for supporting radiation treatment planning.

[0013] FIG. 2A depicts an example of a segmented tissue image.

[0014] FIG. 2B depicts an example of a segmented tissue image.

[0015] FIG. 2C depicts an example of a segmented tissue image.

[0016] FIG. 2D depicts a dose-volume histogram (DVH) visually representing non-binary segmentation.

[0017] FIG. 3 illustrates a flowchart of exemplary operations for estimating a probability map using image segmentation.

[0018] FIG. 4 illustrates an exemplary radiotherapy system adapted for performing image patient state estimation processing.

[0019] FIG. 5 illustrates an exemplary image-guided radiotherapy device.

[0020] FIG. 6 illustrates a partially cut-away view of an exemplary system including a combined radiation therapy system and an imaging system, such as a nuclear magnetic resonance (MR) imaging system.

[0021] FIG. 7 illustrates an exemplary model machine learning engine for use in estimating a probability map using image segmentation.

[0022] FIG. 8 is a flowchart illustrating an example of operations of the segmentation processing logic in performing process. DETAILED DESCRIPTION

[0023] Radiotherapy is used to treat cancers and other ailments in mammalian (e.g., human and animal) tissue. The direction, shape, and intensity of the radiation beam should be accurately controlled to ensure the tumor receives the prescribed radiation, and the placement of the beam should be such as to minimize damage to the surrounding healthy tissue (called the organ(s) at risk (OARs)). Treatment planning can be used to control radiation beam parameters, and a radiotherapy device effectuates a treatment by delivering a spatially varying dose distribution to the patient. The treatment planning procedure can include an image-guided radiation therapy (IGRT) technique, such as involving two or three-dimensional imaging, to identify a target region (e.g., the tumor) and to identify critical organs near the tumor. Segmentation can be performed to help identify the organs-at-risk (OARs) and the area to be treated (for example, a planning target volume (PTV)). Segmentation can be relied upon in determining a dose plan for the patient indicating the desirable amount of radiation to be received by the, one or more, PTV (e.g., target) or the OARs. A PTV can have an irregular volume and can be unique as to its size, shape, and position. A treatment plan can be calculated after optimizing several plan parameters to ensure that enough dose is provided to the PTV(s) while as low a dose as possible is provided to surrounding healthy tissue. A radiation therapy treatment plan is generally determined by balancing efficient control of the dose to treat the tumor against sparing any OAR.

[0024] FIG. 1 depicts an example of a user interface for supporting radiation treatment planning. In an approach to image segmentation, such as for IGRT, a contouring technique can be performed by a physician, e.g., via a user interface 100. For example, a physician can manipulate a cursor 104 such as to manually outline a subset of pixels of an individual tissue image, the subset of pixels depicting a perceived edge of an organ. For example, the subset of pixels can be distinguished or displayed by an overlay element 105 drawn by the cursor over a tissue image 102 including the target region. For example, the overlay element 105 can be superimposed on the tissue image. In an example, the physician can outline the subset of pixels by drawing a contour or arc around or along the perceived edge. This technique can involve the physician manually drawing a large number of contours to segment an organ such as the liver to a desired precision.

[0025] The user interface 100 can include a plurality of contouring modes, such as selectable via a toolbar 106, to aid in manual segmentation. For example, to help reduce the number of contours the physician must manually draw, manual segmentation can be aided by automated contour interpolation. For example, interpolation circuitry can predict how a contour situated between two manually drawn contours looks like. An automated contour interpolation technique can include connecting points on a contour A with corresponding points on a contour B, such as connecting the points via a straight line. The intersection of those lines with the plane in which the interpolated contour should be created gives the vertices of the interpolated contour. Determining the best correspondence of points on the two contours can be challenging, especially when the two contours differ much in shape. Results of automated contour interpolation can differ significantly from a physician’s expectations or can produce results that include anomalies, such as a surface being erroneously self-intersecting. Also, since the two contours are connected using straight lines, the result can be inadequate at representing the smooth shapes of most anatomical structures. Thus, despite an automated contour interpolation tool, segmentation involves drawing a relatively large number of contours to get a desired accuracy in segmentation.

[0026] Another challenge with certain manual contouring techniques can involve a physician producing a binary input for an individual pixel: 0% or 100% related to a specified anatomical structure. This can inhibit or excludes non-binary physician input, e.g., in regions on the image where the physician feels uncertain if those pixels are related to the specified anatomical structure. This can create a lack of uniformity in contours done by different physicians and cause the segmentation data to vary since the technique enables a great amount of subjective physician influence in discerning objectively difficult contours. Since this segmentation data is often indirectly included in a dose calculation, it would be desirable to develop a segmentation technique that mitigates possible judgment error of an individual physician and to receive confidence or uncertainty information therefrom relating to their manual input.

[0027] The present disclosure relates to techniques to aid or replace certain manual contour segmentation tasks. For example, the user interface 100 can include a contouring mode for receiving non-binary image contouring, such as selectable via the toolbar 106. For example, the non-binary image contouring can include a multi-level image contouring (e.g., fill-in contouring for segmentation). In an example, the non-binary image contouring can include a confidence contouring.

[0028] In an example, the user interface 100 can include a multi-level image contouring, or fill-in contouring mode, e.g., selectable via the toolbar 106. In this mode, a physician can draw a contour on the image with a chosen brush size and intensity. The brush size can range from 0 to a maximum brush size, with a selectable brush size, e.g., specifiable via a slider. The brush intensity can range from 0 to a maximum brush intensity, with a selectable brush intensity, e.g., specifiable via a slider. The brush intensity can be the same or different than the brush size. The brush intensity can represent the degree of confidence of the physician that the pixels being contoured belong to the specified anatomical structure. This can be represented in a variety of different ways, including, but not limited to, a numerical value, a graphical representation, or a color-coded representation. The multi-level image contouring or fill-in contouring mode can also enable a physician to indicate a degree of confidence by the number of passes over a particular pixel with the cursor or brush. In an example, the toolbar 106 can also include a contouring mode for automatically detecting, e.g., via interpolation, an outside body contour of a particular anatomical structure.

[0029] The non-binary image contouring can be used, e.g., to generate a probability map. In an example, a machine learning (ML) model can be trained using the non-binary image contouring or one or more probability maps produced via the non-binary image contouring. Such an ML model can be output and can be used, e.g., to establish or adjust an image contour on a different tissue image. For example, the ML model can be used to adjust a particular physician’s contouring such as to conform to average or typical contouring habits of a pool of physicians. This can be helpful in that the ML model can help “translate” a particular physician’s contouring into a more objective characterization amongst skilled physicians than if the particular physician was to estimate and indicate risk alone and with binary segmentation. Such more-objective contour segmentation can help physicians communicate by affording an individual physician tools to express concern to a downstream user interpreting, e.g., a treatment plan generated in-part using data from the contour segmentation.

[0030] By using a population-based contouring approach (e.g., considering or factoring the differences on how different radiation oncologists are contouring a structure or using one or more other methods to establish the associated uncertainty) a probability map can be created. The probablility map can represent the estimated probability of every pixel of belonging to a certain structure in an image. In an example, the probability map can include a 2D display, such include a graduation of varying colors. For example, a prostate can be indicated, e.g., by 6 a green color and rectum can be indicated, e.g., by a red color on the display. Here, a border in between the green color and the red color can be generated and displayed in a third color, e.g., orange, if there are pixels with 50%-50% probability, or in one or more other shades of green or red.

[0031] Such non-binary probability, which can extend to a gradation of a number of probabilities beyond just the binary 0% an 100% that would otherwise be available in a “binary” pixel assignment approach to contouring can be now become a non-binary measure such as which can vary from 0% to 100%, such as in 1%, 2%, 5%, 10%, 20%, 25%, or other increments. For each pixel, a table of probabilities can be associated, and such probabilities can include probabilities to the same organ, or to different organs.

[0032] For example: Pixel X,Y 145,67 Body: 100% Rectum: 80% Prostate: 20% CTV: 10% Bladder: 0%

[0033] One or more uncertainties can be included in the probability map, such as organ motion, image distortion, or general image deformations, such as obtained from a deformable image registration (DIR). In an example, the probability map can be registered to a Megavoltage (MV) imaging isocenter or other isocenter and can help estimate of the probability including a plurality of different uncertainties. Tissue Complication Probability (TPC) or Normal Tissue Complication Probability (NTCP) can also be calculated using the plurality of different uncertainties from contouring, such as to be used to help indicate uncertainty in contouring. Also, as explained further below, the probability map can be used to train a ML model such as to auto-contour a tissue image before or without manual contouring.

[0034] FIG. 2A, FIG. 2B, FIG. 2C, and FIG. 2D depict an example of segmented tissue images and a corresponding dose volume histogram. In an example, a plurality of tissue images, such as 202A, 202B, and 202C, can be obtained. The plurality of tissue images can include different views of a target area, e.g., a patient’s head as depicted in FIG. 2A-FIG. 2C, for administering radiotherapy. In an example of contour segmentation of the tissue images 7 202, a user interface 100 (depicted in FIG. 1) can be used such as to receive a plurality of nonbinary segmentation values for a particular anatomical structure represented by a particular tissue image of the tissue images 202. The user interface 100 can also receive a plurality of non-binary segmentation values for a plurality of anatomical structures 204, 206, and 208, as indicated by respective contours. In an example, a particular anatomical structure (e.g., structure 204) can be similarly represented across multiple corresponding specimen images 202 and distinct amongst other different anatomical structures (e.g., structures 206 &208).

[0035] The non-binary segmentation values corresponding to one or more anatomical structures can be used to help generate an initial probability map, such as an input probability map. For example, the probability map can include probabilities of a particular pixel of being associated with a particular anatomical structure. The initial probability map can then be used as an input to a ML model. When trained and then exposed to a different image, the ML model can produce an augmented or adjusted probability map that can, e.g., be analyzed for certain dose metrics assessments. In an example, the trained ML model can output a dose-volume histogram (DVH) indicating non-binary, non-single curve contour segmentation for one or more anatomical structures.

[0036] FIG. 2D depicts a dose-volume histogram (DVH) visually representing non-binary segmentation. In an example, volume data of the DVH corresponding to an individual dosage can include a range of values based on the non-binary segmentation values. In an example, the DVH can pool data from a plurality of tissue images 202 (e.g., image 202A, image 202B, and image 202C), to generate respective plots (e.g., plot 214, plot 216, and plot 218) for particular anatomical structures (e.g., structure 204, structure 206, and structure 208). Here, the plots 214, 216, and 218 can include planar representations alternatively or additionally to linear dosevolume plots. In an example, an individual plot (e.g., plot 218) can include an area 219 surrounding a linear plot line. An individual planar representation can include a plurality of distinct sections each corresponding to an individual accuracy probability of the non-binary segmentation values. For example, individual sections can be represented by different colors or shadings.

[0037] The trained ML model can be relied upon during radiotherapy dosage calculation based in part on the plurality of confirmed or corrected non-binary segmentation values. For example, a plurality of radiotherapy dosage calculations can be performed, an individual dosage calculation corresponding to an individual pixel or voxel of an individual tissue image based on the output of the ML, such as adjusted segmentation values or a probability map.

[0038] FIG. 3 illustrates a flowchart of a method 300 for training a model to be used in estimating a probability map using image segmentation.

[0039] In an example, at 310, the method can include obtaining a first set of images of at least one patient. For example, the first set of images can include a magnetic resonance (MR) image, or a cone beam computed tomography (CBCT) image or other diagnostic imaging modalities. Multiple corresponding images can be obtained of a particular anatomical structure, such as including different views of the same structure or tissue area.

[0040] At 320, the method can include receiving a set of non-binary segmentation values, each value in the set within a range of possible segmentation values, the set of non-binary segmentation values assigning a probability that an individual pixel or voxel of one of the first set of images corresponds to a specified anatomical structure of the at least one patient. The method can also include automatically detecting a peripheral contour of the specified anatomical structure. Also, receiving the set of non-binary segmentation values can include receiving non-binary segmentation values selected by at least two different radiation oncologists for a single image in the set of images. In an example, the set of non-binary segmentation values can include non-binary segmentation values for a plurality of anatomical structures in each image of the first set of images, the anatomical structures including at least one target and at least one organ at risk.

[0041] At 330, the method can include training a machine learning model using the set of non-binary segmentation values paired with respective images of the first set of images, the machine learning model trained to output a first probability map indicating non-binary segmentation values for pixels or voxels of a provided image. Also, training the machine learning model can include using additional probabilities of individual pixels corresponding to the specified anatomical structure of the at least one patient, the additional probabilities based on a deformable image registration (DIR).

[0042] At 340, the method can include outputting the trained machine learning model. In an example, the trained machine learning model can be used to help create a second probability map for treatment planning. For example, outputting the trained machine learning model can include saving machine learning model data to a database, the database accessible to a physician for use in planning a radiation treatment. Also, an augmented or generated second probability map output by the trained machine learning model can indicate a probability that an individual pixel of an image of a second set of images represents a contour of the specified anatomical structure of the at least one patient. In an example, the second probability map can include additional probabilities of individual pixels corresponding to a portion of anatomy of 9 the patient, the additional probabilities based on at least one of a deformable image registration (DIR), a measured organ motion over time, a measured organ deformation over time, or an image distortion.

[0043] In an example, the method can include, generating, using the second probability map, a visual representation of a contour of a portion of anatomy of the patient. The visual representation can include, e.g., a dose-volume histogram (DVH) generated using the patient probability map, the dose calculated from a treatment planning system, the DVH representing a range of dosage values for the portion of anatomy, or the range of dosage values based on the patient probability map. In an example, the DVH can include a plurality of visually distinguishable sections each corresponding to a range of probabilities from the patient probability map. Also, the DVH can include a tumor control probability (TCP) or a normal tissue complication probability (NTCP), the TCP and NTCP calculated based on the patient probability map.

[0044] FIG. 4 illustrates an exemplary radiotherapy system adapted for generating a probability map for a provided image to support radiation treatment planning. This probability map can be used to enable the radiotherapy system to provide radiation therapy to a patient based on specific aspects of captured medical imaging data. These radiotherapy plan processing operations are performed to enable the radiotherapy system 400 to provide radiation therapy to a patient based on specific aspects of captured medical imaging data and therapy dose calculations or radiotherapy machine configuration parameters. Specifically, the following processing operations can be implemented as part of the segmentation processing logic 420. It will be understood, however, that many variations and use cases of the following trained models and segmentation processing logic 420 can be provided, including in data verification, visualization, and other medical evaluative and diagnostic settings.

[0045] The radiotherapy system can include an image processing computing system 410 which hosts segmentation processing logic 420. The image processing computing system 410 can be connected to a network, and such network can be connected to the Internet. For instance, a network can connect the image processing computing system 410 with one or more medical information sources (e.g., a radiology information system (RIS), a medical record system (e.g., an electronic medical record (EMR) / electronic health record (EHR) system), an oncology information system (OIS)), one or more image data sources 450, an image acquisition device 470, and a treatment device 480 (e.g., a radiation therapy device). As an example, the image processing computing system 410 can be included such as to 10 perform image patient state operations by executing instructions or data from the segmentation processing logic 420, as part of operations to generate a probability map for use in supporting a treatment plan to be used by the treatment device 480, and a treatment data source 460.

[0046] The image processing computing system 410 can include processing circuitry 412, memory 414, a storage device 416, and other hardware and software-operable features such as a user interface 440, communication interface, and the like. The storage device 416 can store computer-executable instructions, such as an operating system, radiation therapy treatment plans (e.g., original treatment plans, adapted treatment plans, or the like), software programs (e.g., radiotherapy treatment plan software, artificial intelligence implementations such as deep learning models, machine learning models, and neural networks, etc.), and any other computer-executable instructions to be executed by the processing circuitry 412.

[0047] In an example, the processing circuitry 412 can include a processing device, such as one or more general-purpose processing devices such as a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), or the like. More particularly, the processing circuitry 412 can be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction Word (VLIW) microprocessor, a processor implementing other instruction sets, or processors implementing a combination of instruction sets. The processing circuitry 412 can also be implemented by one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a System on a Chip (SoC), or the like. As would be appreciated by those skilled in the art, in some examples, the processing circuitry 412 can be a special-purpose processor, rather than a general-purpose processor. The processing circuitry 412 can include one or more known processing devices, such as a microprocessor from the Pentium™, Core™, Xeon™, or Itanium® family manufactured by Intel™, the Turion™, Athlon™, Sempron™, Opteron™, FX™, Phenom™ family manufactured by AMD™, or any of various processors manufactured by Sun Microsystems. The processing circuitry 412 can also include graphical processing units such as a GPU from the GeForce®, Quadro®, Tesla® family manufactured by Nvidia™, GMA, Iris™ family manufactured by Intel™, or the Radeon™ family manufactured by AMD™. The processing circuitry 412 can also include accelerated processing units such as the Xeon Phi™ family manufactured by Intel™. The disclosed examples are not limited to any type of processor(s) otherwise included such as to meet the computing demands of identifying, analyzing, maintaining, generating, and / or providing large amounts of data or manipulating such data to perform the methods disclosed herein. In addition, the term “processor” can include more than one processor, for example, a multi-core design or a plurality of processors each having a multi-core design. The processing circuitry 412 can execute sequences of computer program instructions, stored in memory 414, and accessed from the storage device 416, to perform various operations, processes, methods that will be explained in greater detail below.

[0048] The memory 414 can comprise read-only memory (ROM), a phase-change random access memory (PRAM), a static random access memory (SRAM), a flash memory, a random access memory (RAM), a dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM), an electrically erasable programmable read-only memory (EEPROM), a static memory (e.g., flash memory, flash disk, static random access memory) as well as other types of random access memories, a cache, a register, a compact disc readonly memory (CD-ROM), a digital versatile disc (DVD) or other optical storage, a cassette tape, other magnetic storage device, or any other non-transitory medium that can be used to store information including image, data, or computer executable instructions (e.g., stored in any format) capable of being accessed by the processing circuitry 412, or any other type of computer device. For instance, the computer program instructions can be accessed by the processing circuitry 412, read from the ROM, or any other suitable memory location, and loaded into the RAM for execution by the processing circuitry 412.

[0049] The storage device 416 can constitute a drive unit that can include a machine-readable medium on which is stored one or more sets of instructions and data structures (e.g., software) embodying or utilized by any one or more of the methodologies or functions described herein (including, in various examples, the segmentation processing logic 420 and the user interface 440). The instructions can also reside, completely or at least partially, within the memory 414 and / or within the processing circuitry 412 during execution thereof by the image processing computing system 410, with the memory 414 and the processing circuitry 412 also constituting machine-readable media.

[0050] The memory device 414 or the storage device 416 can constitute a non-transitory computer-readable medium. For example, the memory device 414 or the storage device 416 can store or load instructions for one or more software applications on the computer-readable medium. Software applications stored or loaded with the memory device 414 or the storage device 416 can include, for example, an operating system for common computer systems as well as for software-controlled devices. The image processing computing system 410 can 12 also operate a variety of software programs comprising software code for implementing the segmentation processing logic 420 and the user interface 440. Further, the memory device 414 and the storage device 416 can store or load an entire software application, part of a software application, or code or data that is associated with a software application, which is executable by the processing circuitry 412. In a further example, the memory device 414 or the storage device 416 can store, load, or manipulate one or more radiation therapy treatment plans, imaging data, patient state data, dictionary entries, artificial intelligence model data, labels and mapping data, etc. It is contemplated that software programs can be stored not only on the storage device 416 and the memory 414 but also on a removable computer medium, such as a hard drive, a computer disk, a CD-ROM, a DVD, a HD, a Blu-Ray DVD, USB flash drive, a SD card, a memory stick, or any other suitable medium; such software programs can also be communicated or received over a network.

[0051] The image processing computing system 410 can include a communication interface, network interface card, and communications circuitry. An example communication interface can include, for example, a network adaptor, a cable connector, a serial connector, a USB connector, a parallel connector, a high-speed data transmission adaptor (e.g., such as fiber, USB 3.0, thunderbolt, and the like), a wireless network adaptor (e.g., such as a IEEE 602.11 / Wi-Fi adapter), a telecommunication adapter (e.g., to communicate with 3G, 4G / LTE, and 5G, networks and the like), and the like. Such a communication interface can include one or more digital and / or analog communication devices that permit a machine to communicate with other machines and devices, such as remotely located components, via a network. The network can provide the functionality of a local area network (LAN), a wireless network, a cloud computing environment (e.g., software as a service, platform as a service, infrastructure as a service, etc.), a client-server, a wide area network (WAN), and the like. For example, network can be a LAN or a WAN that can include other systems (including additional image processing computing systems or image-based components associated with medical imaging or radiotherapy operations).

[0052] In an example, the image processing computing system 410 can obtain image data 452 from the image data source 450, for hosting on the storage device 416 and the memory 414. In an example, the software programs operating on the image processing computing system 410 can convert medical images of one format (e.g., MRI) to another format (e.g., CT), such as by producing synthetic images, such as a pseudo-CT image. In another example, the software programs can register or associate a patient medical image (e.g., a CT image or an MR image) with that patient’s dose distribution of radiotherapy treatment (e.g., also 13 represented as an image) so that corresponding image voxels and dose voxels are appropriately associated. In yet another example, the software programs can substitute functions of the patient images such as signed distance functions or processed versions of the images that emphasize some aspect of the image information. Such functions might emphasize edges or differences in voxel textures, or other structural aspects. In another example, the software programs can visualize, hide, emphasize, or de-emphasize some aspect of anatomical features, patient measurements, patient state information, or dose or treatment information, within medical images. The storage device 416 and memory 414 can store and host data to perform these purposes, including the image data 452, patient data, and other data involved to create and implement a radiation therapy treatment plan and associated patient state estimation operations.

[0053] The processing circuitry 412 can be communicatively coupled to the memory 414 and the storage device 416, and the processing circuitry 412 can be included such as to execute computer executable instructions stored thereon from either the memory 414 or the storage device 416. The processing circuitry 412 can execute instructions to cause medical images from the image data 452 to be received or obtained in memory 414, and processed using the segmentation processing logic 420. For example, the image processing computing system 410 can receive image data 452 from the image acquisition device 470 or image data sources 450 via a communication interface and network to be stored or cached in the storage device 416. The processing circuitry 412 can also send or update medical images stored in memory 414 or the storage device 416 via a communication interface to another database or data store (e.g., a medical facility database). In some examples, one or more of the systems can form a distributed computing / simulation environment that uses a network to collaboratively perform the examples described herein. In addition, such network can be connected to internet to communicate with servers and clients that reside remotely on the internet.

[0054] In further examples, the processing circuitry 412 can utilize software programs (e.g., a treatment planning software) along with the image data 452 and other patient data to create a radiation therapy treatment plan. In an example, the image data 452 can include 2D or 3D images, such as from a CT or MR. In addition, the processing circuitry 412 can utilize software programs to generate an estimated patient state from a dictionary of measurements and corresponding patient states, such as using a correspondence motion model and a machine learning algorithm (e.g., a regression algorithm).

[0055] Further, such software programs can utilize patient segmentation logic 420 to implement an image segmentation workflow 430, using the techniques further discussed herein. The processing circuitry 412 can subsequently then transmit the executable radiation therapy treatment plan via a communication interface and the network to the treatment device 480, where the radiation therapy plan will be used to treat a patient with radiation via the treatment device, consistent with results of the image segmentation workflow 430. Other outputs and uses of the software programs and the image segmentation workflow 430 can occur with using the image processing computing system 410.

[0056] As discussed herein (e.g., with reference to the patient state estimation discussed herein, the processing circuitry 412 can execute software programs that invokes the patient segmentation logic 420 to implement functions including generation of a preliminary motion model, creation of a dictionary, training a patient state generator using machine learning, patient state estimation, and other aspects of automatic processing and artificial intelligence. For instance, the processing circuitry 412 can execute software programs that estimate a patient state using a machine learning trained system.

[0057] In an example, the image data 452 can include one or more MRI images (e.g., 2D MRI, 3D MRI, 2D streaming MRI, 4D MRI, 4D volumetric MRI, 4D cine MRI, etc ), functional MRI images (e.g., fMRI, DCE-MRI, diffusion MRI), Computed Tomography (CT) images (e.g., 2D CT, Cone beam CT, 3D CT, 4D CT), ultrasound images (e.g., 2D ultrasound, 3D ultrasound, 4D ultrasound), Positron Emission Tomography (PET) images, X-ray images, fluoroscopic images, radiotherapy portal images, Single-Photo Emission Computed Tomography (SPECT) images, computer generated synthetic images (e.g., pseudo-CT images) and the like. Further, the image data 452 can also include or be associated with medical image processing data, for instance, training images, and ground truth images, contoured images, and dose images. In an example, the image data 452 can be received from the image acquisition device 470 and stored in one or more of the image data sources 450 (e.g., a Picture Archiving and Communication System (PACS), a Vendor Neutral Archive (VNA), a medical record or information system, a datawarehouse, etc.). Accordingly, the image acquisition device 470 can comprise an MRI imaging device, a CT imaging device, a PET imaging device, an ultrasound imaging device, a fluoroscopic device, a SPECT imaging device, an integrated Linear Accelerator and MRI imaging device, or other medical imaging devices for obtaining the medical images of the patient. The image data 452 can be received and stored in any type of data or any type of format (e.g., in a Digital Imaging and Communications in Medicine (DICOM) format) that the image acquisition device 470 and the image processing computing system 410 can use to perform operations consistent with the disclosed examples.

[0058] In an example, the image acquisition device 470 can be integrated with the treatment device 480 as a single apparatus (e.g., an MRI device combined with a linear accelerator, also referred to as an “MR-linac”, as shown and described in FIG. 6 below). Such an MR-linac can be used, for example, to precisely determine a location of a target organ or a target tumor in the patient, so as to direct radiation therapy accurately according to the radiation therapy treatment plan to a predetermined target. For instance, a radiation therapy treatment plan can provide information about a particular radiation dose to be applied to each patient. The radiation therapy treatment plan can also include other radiotherapy information, such as beam angles, dose-histogram-volume information, the number of radiation beams to be used during therapy, the dose per beam, and the like.

[0059] The image processing computing system 410 can communicate with an external database through a network to send / receive a plurality of various types of data related to image processing and radiotherapy operations. For example, an external database can include machine data that is information associated with the treatment device 480, the image acquisition device 470, or other machines relevant to radiotherapy or medical procedures. Machine data information can include radiation beam size, arc placement, beam on and off time duration, machine parameters, segments, multi-leaf collimator (MLC) configuration, gantry speed, MRI pulse sequence, and the like. The external database can be a storage device and can be equipped with appropriate database administration software programs. Further, such databases or data sources can include a plurality of devices or systems located either in a central or a distributed manner.

[0060] The image processing computing system 410 can collect and obtain data, and communicate with other systems, via a network using one or more communication interfaces, which are communicatively coupled to the processing circuitry 412 and the memory 414. For instance, a communication interface can provide communication connections between the image processing computing system 410 and radiotherapy system components (e.g., permitting the exchange of data with external devices). For instance, the communication interface can in some examples have appropriate interfacing circuitry from an output device 442 or an input device 444 to connect to the user interface 440, which can be a hardware keyboard, a keypad, or a touch screen through which a user can input information into the radiotherapy system.

[0061] As an example, the output device 442 can include a display device which outputs a representation of the user interface 440 and one or more aspects, visualizations, or 16 representations of the medical images. The output device 442 can include one or more display screens that display medical images, interface information, treatment planning parameters (e.g., contours, dosages, beam angles, labels, maps, etc.) treatment plans, a target, localizing a target or tracking a target, patient state estimations (e.g., a 3D image), or any related information to the user. The input device 444 connected to the user interface 440 can be a keyboard, a keypad, a touch screen or any type of device that a user can input information to the radiotherapy system. Alternatively, the output device 442, the input device 444, and features of the user interface 440 can be integrated into a single device such as a smartphone or tablet computer, e.g., Apple iPad®, Lenovo Thinkpad®, Samsung Galaxy®, etc.

[0062] Furthermore, any and all components of the radiotherapy system can be implemented as a virtual machine (e.g., via VMWare, Hyper-V, and the like virtualization platforms). For instance, a virtual machine can be software that functions as hardware. Therefore, a virtual machine can include at least one or more virtual processors, one or more virtual memories, and one or more virtual communication interfaces that together function as hardware. For example, the image processing computing system 410, the image data sources 450, or like components, can be implemented as a virtual machine or within a cloud-based virtualization environment.

[0063] The segmentation processing logic 420 or other software programs can cause the computing system to communicate with the image data sources 450 to read images into memory 414 and the storage device 416, or store images or associated data from the memory 414 or the storage device 416 to and from the image data sources 450. For example, the image data source 450 can be included such as to store and provide a plurality of images (e.g., 3D MRI, 4D MRI, 2D MRI slice images, CT images, 2D Fluoroscopy images, X-ray images, raw data from MR scans or CT scans, Digital Imaging and Communications in Medicine (DICOM) metadata, etc.) that the image data source 450 hosts, from image sets in image data 452 obtained from one or more patients via the image acquisition device 470. The image data source 450 or other databases can also store data to be used by the segmentation processing logic 420 when executing a software program that performs patient state estimation operations, or when creating radiation therapy treatment plans. Further, various databases can store the data produced by the preliminary motion model (such as the dictionary), the correspondence motion model, or machine learning models, including the network parameters constituting the model learned by the network and the resulting predicted data. The image processing computing system 410 thus can obtain and / or receive the image data 452 (e.g., 2D MRI slice images, CT images, 2D Fluoroscopy images, X-ray images, 3D MRI images, 4D MRI images, etc.) from 17 the image data source 450, the image acquisition device 470, the treatment device 480 (e.g., an MRI-Linac), or other information systems, in connection with performing image patient state estimation as part of treatment or diagnostic operations.

[0064] The image acquisition device 470 can be included such as to acquire one or more images of the patient’s anatomy for a region of interest (e.g., a target organ, a target tumor or both). Each image, typically a 2D image or slice, can include one or more parameters (e.g., a 2D slice thickness, an orientation, and a location, etc.). In an example, the image acquisition device 470 can acquire a 2D slice in any orientation. For example, an orientation of the 2D slice can include a sagittal orientation, a coronal orientation, or an axial orientation. The processing circuitry 412 can adjust one or more parameters, such as the thickness and / or orientation of the 2D slice, to include the target organ and / or target tumor. In an example, 2D slices can be determined from information such as a 3D MRI volume. Such 2D slices can be acquired by the image acquisition device 470 in “real-time” while a patient is undergoing radiation therapy treatment, for example, when using the treatment device 480 (with “realtime” meaning acquiring the data in 10 milliseconds or less). In another example for some applications, real-time can include a timeframe within (e.g., up to) 500 or 600 milliseconds. In an example, real-time can include a time period fast enough for a clinical problem being solved by techniques described herein. In this example, real-time can vary depending on target speed, radiotherapy margins, lag, response time of a treatment device, etc.

[0065] The patient segmentation logic 420 in the image processing computing system 410 is depicted as implementing an image segmentation workflow 430 with various aspects of model generation and estimation processing operations. The image segmentation workflow 430 can include obtaining or receiving a set of images obtained from one or more patients, the set of images arranged in an image index 432. In an example, the set of images can include images obtained from a plurality of patients. The image segmentation workflow 430 can include receiving a set of segmentation values and arranging the segmentation values in a segmentation value index 434, an individual segmentation value assigning a probability or uncertainty that an individual pixel or voxel of the set of images in the image index 432 corresponds to a specified anatomical structure of the patient corresponding to the present image. For example, the set of segmentation values can be received from the user interface 440. The set of segmentation values can also include a plurality of subsets of segmentation values, each subset of segmentation values corresponding with segmentation data corresponding with a different user account. Here, the different user accounts can be associated with different oncologists each providing segmentation data for a same image. The image segmentation workflow 430 18 can include training a machine learning (ML) model using machine learning 436 (e.g., using a regression-based machine learning technique) using the set of segmentation values from the segmentation value index 434, the segmentation values paired with respective images of the set of images from the image index 432. The image segmentation workflow 430 can include generating a probability map 438 (e.g., an augmented probability map or an entirely autogenerated probability map) using the machine learning model.

[0066] In an example, the image segmentation workflow 430 operated by the segmentation processing logic 420 generates and trains a machine learning (ML) model using patient data, such as data arranged in the image index 432 or the segmentation index (e.g., data obtained from a patient being treated, from multiple previous patients, or the like). As described further below, the ML model can also be generated and trained using other uncertainty data, such as probabilities calculated based on a deformable image registration (DIR), a measured organ motion over time, a measured organ deformation over time, or an image distortion parameter. The ML model can be provided by a Neural Network (NN) trained as part of a NN model. One or more teacher ML models can be provided by a different entity or at an off-site facility relative to segmentation processing logic 420 and is accessible by issuing one or more queries to the off-site facility.

[0067] Supervised machine learning (ML) algorithms or ML models or techniques can be summarized as function approximation. Training data consisting of input-output pairs of some type (e.g., pairs of segmentation values corresponding with a medical image and a probability map) are acquired from, e.g., expert clinicians or prior optimization plan solvers and a function is “trained” to approximate this mapping. Some methods involve NNs. In these, a set of parametrized functions A9 are selected, where 9 is a set of parameters (e.g., convolution kernels and biases) that are selected by minimizing the average error over the training data. If the input-output pairs are denoted by (x_m, y_m), the function can be formalized by solving a minimization problem such as Equation 3: M min IM0(xm) -ym||2 m=l (3) The minimization problem of Equation 3 that is used to train the network can be based on a loss function that includes a derivative of a dose calculation.

[0068] Once the network has been trained (e.g., 9 has been selected), the function Ae can be applied to any new input. For example, a never-before-seen radiotherapy treatment plan information (e.g., an MR and / or CBCT image) can be fed into Ae, and one or more radiotherapy treatment plan parameters (e.g., an sCT image) are estimated. As another example, a never-before-seen MR, CT, sCT, CBCT image, segmentation and distance map can be fed into A0, and one or more radiotherapy device control points are estimated. As another example, a never-before-seen dose computation function parameters can be fed into A0, and one or more intermediate dose distributions are estimated.

[0069] Simple NNs consist of an input layer, a middle or hidden layer, and an output layer, each containing computational units or nodes. The hidden layer(s) nodes have input from all the input layer nodes and are connected to all nodes in the output layer. Such a network is termed “fully connected.” Each node communicates a signal to the output node depending on a nonlinear function of the sum of its inputs. For a classifier, the number of input layer nodes typically equals the number of features for each of a set of objects being sorted into classes, and the number of output layer nodes is equal to the number of classes. A network is trained by presenting it with the features of objects of known classes and adjusting the node weights to reduce the training error by an algorithm called backpropagation. Thus, the trained network can classify novel objects whose class is unknown.

[0070] Neural networks have the capacity to discover relationships between the data and classes or regression values, and under certain conditions, can emulate any function including non-linear functions. In ML, an assumption is that the training and test data are both generated by the same data-generating process, in which each sample is identically and independently distributed (i.i.d.). In ML, the goals are to minimize the training error and to make the difference between the training and test errors as small as possible. Underfitting occurs if the training error is too large; overfitting occurs when the train-test error gap is too large. Both types of performance deficiency are related to model capacity: large capacity may fit the training data very well but lead to overfitting, while small capacity may lead to underfitting.

[0071] The segmentation processing logic 120 and the image segmentation workflow 130 may be used when generating the radiation therapy treatment plan, within using software programs such as treatment planning software, such as Monaco®, manufactured by Elekta AB of Stockholm, Sweden. In order to generate the radiation therapy treatment plans, the image processing computing system 110 may communicate with the image acquisition device 170(e.g.,aCT device, an MRI device, a PET device, an X-ray device, an ultrasound device, etc.) to capture and access images of the patient and to delineate a target, such as a tumor. In some examples, the delineation of one or more organs at risk (OARs), such as healthy tissue surrounding the tumor or in close proximity to the tumor may be involved.

[0072] In order to delineate a target organ or a target tumor from the OAR, medical images, such as MRI images, CT images, PET images, fMRI images, X-ray images, ultrasound images, radiotherapy portal images, SPECT images and the like, of the patient undergoing radiotherapy may be obtained non-invasively by the image acquisition device 170 to reveal the internal structure of a body part. Based on the information from the medical images, a 3D structure of the relevant anatomical portion may be obtained. In addition, during a treatment planning process, many parameters may be taken into consideration to achieve a balance between efficient treatment of the target tumor (e.g., such that the target tumor receives enough radiation dose for an effective therapy) and low irradiation of the OAR(s) (e.g., the OAR(s) receives as low a radiation dose as possible), for example by using an estimated patient state to determine where OAR(s) may be at a given time when the patient is moving (e.g., breathing). Other parameters that may be considered include the location of the target organ and the target tumor, the location of the OAR, and the movement of the target in relation to the OAR. For example, the 3D structure may be obtained by contouring the target or contouring the OAR within each 2D layer or slice of an MRI or CT image and combining the contour of each 2D layer or slice. The contour may be generated manually (e.g., by a physician, dosimetrist, or health care worker using a program such as Monaco® manufactured by Elekta AB of Stockholm, Sweden) or automatically (e.g., using a program such as the Atlas-based auto-segmentation software, ABAS®, manufactured by Elekta AB of Stockholm, Sweden).

[0073] After the target tumor and the OAR(s) have been located and delineated, a dosimetrist, physician or healthcare worker may determine a dose of radiation to be applied to the target tumor, as well as any maximum amounts of dose that may be received by the OAR proximate to the tumor (e.g., left and right parotid, optic nerves, eyes, lens, inner ears, spinal cord, brain stem, and the like). After the radiation dose is determined for each anatomical structure (e.g., target tumor, OAR), a process known as inverse planning may be performed to determine one or more treatment plan parameters that would achieve the desired radiation dose distribution. Examples of treatment plan parameters include volume delineation parameters (e.g., which define target volumes, contour sensitive structures, etc.), margins around the target tumor and OARs, beam angle selection, collimator settings, and beam-on times. During the inverse-planning process, the physician may define dose constraint parameters that set bounds on how much radiation an OAR may receive (e.g., 21 defining full dose to the tumor target and zero dose to any OAR; defining 95% of dose to the target tumor; defining that the spinal cord, brain stem, and optic structures receive <45Gy, <55Gy and <54Gy, respectively). The result of inverse planning may constitute a radiation therapy treatment plan that may be stored. Some of these treatment parameters may be correlated. For example, tuning one parameter (e.g., weights for different objectives, such as increasing the dose to the target tumor) in an attempt to change the treatment plan may affect at least one other parameter, which in turn may result in the development of a different treatment plan. Thus, the image processing computing system 110 can generate a tailored radiation therapy treatment plan having these parameters in order for the treatment device 180 to provide suitable radiotherapy treatment to the patient.

[0074] FIG. 5 illustrates an exemplary image-guided radiotherapy device 502, that can include a radiation source, such as an X-ray source or a linear accelerator, a couch 516, an imaging detector 514, and a radiation therapy output 504. The radiation therapy device 502 can be included such as to emit a radiation beam 508 to provide therapy to a patient. The radiation therapy output 504 can include one or more attenuators or collimators, such as a multi-leaf collimator (MLC).

[0075] As an example, a patient can be positioned in a region 512, supported by the treatment couch 516 to receive a radiation therapy dose according to a radiation therapy treatment plan (e.g., a treatment plan generated by the radiotherapy system of FIG. 4). The radiation therapy output 504 can be mounted or attached to a gantry 506 or other mechanical support. One or more chassis motors (not shown) can rotate the gantry 506 and the radiation therapy output 504 around couch 516 when the couch 516 is inserted into the treatment area. In an example, gantry 506 can be continuously rotatable around couch 516 when the couch 516 is inserted into the treatment area. In another example, gantry 506 can rotate to a predetermined position when the couch 516 is inserted into the treatment area. For example, the gantry 506 can be included such as to rotate the therapy output 504 around an axis (“^4”). Both the couch 516 and the radiation therapy output 504 can be independently moveable to other positions around the patient, such as moveable in transverse direction (“7”), moveable in a lateral direction (“Z”), or as rotation about one or more other axes, such as rotation about a transverse axis (indicated as “Z”). A controller communicatively connected to one or more actuators (not shown) can control the couch 516 movements or rotations in order to properly position the patient in or out of the radiation beam 508 according to a radiation therapy treatment plan. As both the couch 516 and the gantry 506 are independently moveable from one another in multiple degrees of freedom, which allows the patient to be positioned such that the radiation beam 508 precisely can target the tumor.

[0076] The coordinate system (including axes A, Z, andZ) shown in FIG. 5 can have an origin located at an isocenter 510. The isocenter can be defined as a location where the central axis of the radiation therapy beam 508 intersects the origin of a coordinate axis, such as to deliver a prescribed radiation dose to a location on or within a patient. Alternatively, the isocenter 510 can be defined as a location where the central axis of the radiation therapy beam 508 intersects the patient for various rotational positions of the radiation therapy output 504 as positioned by the gantry 506 around the axis A.

[0077] Gantry 506 can also have an attached imaging detector 514. The imaging detector 514 is preferably located opposite to the radiation source (output 504), and in an example, the imaging detector 514 can be located within a field of the therapy beam 508.

[0078] The imaging detector 514 can be mounted on the gantry 506 preferably opposite the radiation therapy output 504, such as to maintain alignment with the therapy beam 508. The imaging detector 514 rotating about the rotational axis as the gantry 506 rotates. In an example, the imaging detector 514 can be a flat panel detector (e.g., a direct detector or a scintillator detector). In this manner, the imaging detector 514 can be used to monitor the therapy beam 508 or the imaging detector 514 can be used for imaging the patient’s anatomy, such as portal imaging. The control circuitry of radiation therapy device 502 can be integrated within the radiotherapy system or remote from it.

[0079] In an illustrative example, one or more of the couch 516, the therapy output 504, or the gantry 506 can be automatically positioned, and the therapy output 504 can establish the therapy beam 508 according to a specified dose for a particular therapy delivery instance. A sequence of therapy deliveries can be specified according to a radiation therapy treatment plan, such as using one or more different orientations or locations of the gantry 506, couch 516, or therapy output 504. The therapy deliveries can occur sequentially, but can intersect in a desired therapy locus on or within the patient, such as at the isocenter 510. A prescribed cumulative dose of radiation therapy can thereby be delivered to the therapy locus while damage to tissue nearby the therapy locus can be reduced or avoided.

[0080] Thus, FIG. 5 specifically illustrates an example of a radiation therapy device 502 operable to provide radiotherapy treatment to a patient, with a configuration where a radiation therapy output can be rotated around a central axis (e.g., an axis “^4”). Other radiation therapy output configurations can be used. For example, a radiation therapy output can be mounted to a robotic arm or manipulator having multiple degrees of freedom. In yet another example, the 23 therapy output can be fixed, such as located in a region laterally separated from the patient, and a platform supporting the patient can be used to align a radiation therapy isocenter with a specified target locus within the patient. In another example, a radiation therapy device can be a combination of a linear accelerator and an image acquisition device. In some examples, the image acquisition device can be an MRI, an X-ray, a CT, a CBCT, a spiral CT, a PET, a SPECT, an optical tomography, a fluorescence imaging, ultrasound imaging, an MR-linac, or radiotherapy portal imaging device, etc., as would be recognized by one of ordinary skill in the art.

[0081] FIG. 6 depicts an exemplary radiation therapy system 600 (e.g., known in the art as a MR-Linac) that can include combining a radiation therapy device 502 and an imaging system, such as a nuclear magnetic resonance (MR) imaging system consistent with the disclosed examples. As shown, system 600 can include a couch 610, an image acquisition device 620, and a radiation delivery device 630. System 600 delivers radiation therapy to a patient in accordance with a radiotherapy treatment plan. In an example, image acquisition device 620 can correspond to image acquisition device 470 in FIG. 4 that can acquire images.

[0082] Couch 610 can support a patient (not shown) during a treatment session. In some implementations, couch 610 can move along a horizontal, translation axis (labelled “I”), such that couch 610 can move the patient resting on couch 610 into or out of system 600. Couch 610 can also rotate around a central vertical axis of rotation, transverse to the translation axis. To allow such movement or rotation, couch 610 can have motors (not shown) enabling the couch to move in various directions and to rotate along various axes. A controller (not shown) can control these movements or rotations in order to properly position the patient according to a treatment plan.

[0083] In an example, image acquisition device 620 can include an MRI machine used to acquire 2D or 3D MRI images of the patient before, during, or after a treatment session. Image acquisition device 620 can include a magnet 621 for generating a primary magnetic field for magnetic resonance imaging. The magnetic field lines generated by operation of magnet 621 can run substantially parallel to the central translation axis I. Magnet 621 can include one or more coils with an axis that runs parallel to the translation axis I. In an example, the one or more coils in magnet 621 can be spaced such that a central window 623 of magnet 621 is free of coils. In other examples, the coils in magnet 621 can be thin enough or of a reduced density such that they are substantially transparent to radiation of the wavelength generated by radiotherapy device 630. Image acquisition device 620 can also include one or more shielding coils, which can generate a magnetic field outside magnet 621 24 of approximately equal magnitude and opposite polarity in order to cancel or reduce any magnetic field outside of magnet 621. As described below, radiation source 631 of radiotherapy device 630 can be positioned in the region where the magnetic field is cancelled, at least to a first order, or reduced.

[0084] Image acquisition device 620 can also include two gradient coils 625 and 626, which can generate a gradient magnetic field that is superposed on the primary magnetic field. Coils 625 and 626 can generate a gradient in the resultant magnetic field that allows spatial encoding of the protons so that their position can be determined. Gradient coils 625 and 626 can be positioned around a common central axis with the magnet 621, and can be displaced along that central axis. The displacement can create a gap, or window, between coils 625 and 626. In the examples where magnet 621 also can include a central window 623 between coils, the two windows can be aligned with each other.

[0085] Image acquisition is used to track tumor movement. At times, internal or external surrogates can be used. However, implanted seeds can move from their initial positions or become dislodged during radiation therapy treatment. Also, using surrogates assumes there is a correlation between tumor motion and the displacement of the external surrogate. However, there can be phase shifts between external surrogates and tumor motion, and their positions can frequently lose correlation over time. It is known that there can be mismatches between tumor and surrogates upward of 9mm. Further, any deformation of the shape of a tumor is unknown during tracking.

[0086] An advantage of magnetic resonance imaging (MRI) is in the superior soft tissue contrast that is provided to visualize the tumor in more detail. Using a plurality of intrafractional MR images allows the determination of both shape and position (e.g., centroid) of a tumor. In addition, MRI images improve any manual contouring performed by, for example, a radiation oncologist, even when auto-contouring software (e.g., ABAS®) is utilized. This is because of the high contrast between the tumor target and the background region provided by MR images.

[0087] Another advantage of using an MR-Linac system is that a treatment beam can be continuously on and thereby executing intrafractional tracking of the target tumor. For instance, optical tracking devices or stereoscopic x-ray fluoroscopy systems can detect tumor position at 30Hz by using tumor surrogates. With MRI, the imaging acquisition rates are faster (e.g., 3-6 fps). Therefore ,the centroid position of the target can be determined, artificial intelligence (e.g., neural network) software can predict a future target position. An added advantage of intrafractional tracking by using an MR-Linac is that the by being able to 25 predict a future target location, the leaves of the multi-leaf collimator (MLC) will be able to conform to the target contour a its predicted future position. Thus, predicting future tumor position using MRI occurs at the same rate as imaging frequency during tracking. By being able to track the movement of a target tumor clearly using detailed MRI imaging allows for the delivery of a highly conformal radiation dose to the moving target.

[0088] In an example, image acquisition device 620 can be an imaging device other than an MRI, such as an X-ray, a CT, a CBCT, a spiral CT, a PET, a SPECT, an optical tomography, a fluorescence imaging, ultrasound imaging, or radiotherapy portal imaging device, etc. As would be recognized by one of ordinary skill in the art, the above description of image acquisition device 620 concerns certain examples and is not intended to be limiting.

[0089] Radiotherapy device 630 can include the source of radiation 631, such as an X-ray source or a linear accelerator, and a multi-leaf collimator (MLC) 633. Radiotherapy device 630 can be mounted on a chassis 635. One or more chassis motors (not shown) can rotate chassis 635 around couch 610 when couch 610 is inserted into the treatment area. In an example, chassis 635 can be continuously rotatable around couch 610, when couch 610 is inserted into the treatment area. Chassis 635 can also have an attached radiation detector (not shown), preferably located opposite to radiation source 631 and with the rotational axis of chassis 635 positioned between radiation source 631 and the detector. Further, device 630 can include control circuitry (not shown) used to control, for example, one or more of couch 610, image acquisition device 620, and radiotherapy device 630. The control circuitry of radiotherapy device 630 can be integrated within system 600 or remote from it.

[0090] During a radiotherapy treatment session, a patient can be positioned on couch 610. System 600 can then move couch 610 into the treatment area defined by magnetic coils 621, 625, 626, and chassis 635. Control circuitry can then control radiation source 631, MLC 633, and the chassis motor(s) to deliver radiation to the patient through the window between coils 625 and 626 according to a radiotherapy treatment plan.

[0091] FIG. 7 illustrates an exemplary regression model machine learning engine 700 for use in image segmentation of a tissue image. Machine learning engine 700 utilizes a training engine 702 and an estimation engine 704. Training engine 702 inputs historical information 706 (e.g., manual contouring or one or more probability maps generated therefrom of an earlier obtained tissue image) into feature determination engine 708. The historical information 706 can be labeled to indicate the correspondence between an individual pixel / voxel and a particular anatomical structure.

[0092] Feature determination engine 708 determines one or more features 710 from this historical information 706. Stated generally, features 710 are a set of the information input and include information determined to be predictive of a particular outcome. The features 710 can be determined by hidden layers, in an example. The machine learning algorithm 712 produces a correspondence motion model 720 based upon the features 710 and the labels.

[0093] In the estimation engine 704, present information 714 (e.g., present segmentation values, such as manual contouring of a present, later obtained tissue image) can be input to the feature determination engine 716. Feature determination engine 716 can determine features of the current information 714 to estimate image segmentation of the present, later obtained tissue image. In some examples, feature determination engines 716 and 708 are the same engine. Feature determination engine 716 produces feature vector 718, which is input into the model 720 to generate one or more criteria weightings 722. The training engine 702 can operate in an offline manner to train the model 720. The estimation engine 704, however, can be designed to operate in an online manner. It should be noted that the model 720 can be periodically updated via additional training or user feedback (e.g., additional, changed, or removed measurements or patient states).

[0094] The machine learning algorithm 712 can be selected from among many different potential supervised or unsupervised machine learning algorithms. Examples of supervised learning algorithms include artificial neural networks, Bayesian networks, instance-based learning, support vector machines, decision trees (e.g., Iterative Dichotomiser 3, C4.5, Classification and Regression Tree (CART), Chi-squared Automatic Interaction Detector (CHAID), and the like), random forests, linear classifiers, quadratic classifiers, k-nearest neighbor, linear regression, logistic regression, and hidden Markov models. Examples of unsupervised learning algorithms include expectation-maximization algorithms, vector quantization, and information bottleneck method. Unsupervised models can not have a training engine 702.

[0095] In an example, a regression model is used and the model 720 is a vector of coefficients corresponding to a learned importance for each of the features in the vector of features 710, 718. The regression model is illustrated in block 724, showing an example linear regression. The machine learning algorithm 712 is trained using a dictionary generated as described herein. The machine learning algorithm 712 trains on how patient measurements correspond to patient states. In an example, the machine learning algorithm 712 implements a regression problem (e.g., linear, polynomial, regression trees, kernel density estimation, support vector regression, random forests implementations, or the like). The resulting training 27 parameters define the patient state generator as a correspondence motion model for the chosen machine learning algorithm.

[0096] In the conventional linac case, this training can be performed separately for every possible gantry angle (e.g., with a one degree increment), since x-ray acquisition orientation can be constrained to an orthogonal angle with respect to the treatment beam. In the MR-linac case, control can be given to a clinician on the 2D acquisition plane for position or orientation. Repeating cross-validation on training data with different choice of 2D planes can reveal which 2D planes yield best surrogate information for a given patient / tumor site.

[0097] In some cases, a patient measurement can be used to update the model 720. In some cases, a calculation can be performed to determine whether the patient measurement is consistent with the model 720, and pause the treatment if it is not (e.g., using a threshold on the variance for a KDE algorithm, or determining whether there is sufficient data in the dictionary in the neighborhood of the measurement). When the treatment is paused, a new model 720 can be generated, or the old model 720 can be reused if the measurement (e.g., motion) was an aberration.

[0098] In some techniques the entire real-time patient image can not be necessary, and only features of it can be useful. For example, the target centroid can be useful to make geometric corrections to multileaf collimators (MLCs), or to gate a beam on or off. In such cases, the single DVF vector connecting the center of the target in the reference image to the current target can be used rather than computing the entire 3D DVF and deforming the entire reference image at each time, which results in rendering the real-time process more efficient.

[0099] After the patient state generator has been successfully trained and the patient model 720 is aligned to the patient, the treatment beam is turned on and instantaneous partial measurements are acquired at a given frequency. For each received measurement, the process can include normalizing a 2D image of the received measurement to match the contrast of training images. The patient state generator can use the normalized measurement to infer model coefficients, and a DVF can be reconstructed using the model. The reconstructed DVF is used to warp the reference volume and treatment information to the current patient state, which can be output or saved.

[0100] In some cases, the model can not be well-aligned to the patient during treatment. This can occur if the patient moves between the 4D image and treatment, if a model from a previous day is used, or if data from other patients is used. The patient model (computed pretreatment) can then be aligned to the actual patient position by rigid registration to new patient measurements with the patient in treatment position. During this time, a CBCT or 28 MRI is acquired for coarse model-to-patient alignment. Fine alignment of patient model with multiple sample images (e.g., x-ray or 2D MRI slices) to account for couch shifts can be applied after CBCT or MRI acquisition.

[0101] Differences of contrast in synthetically generated training measurements versus actual 2D imaging acquisitions can hinder the generator’s ability to infer 3D patient states. Some intensity normalization procedure can be used to correct for this issue. For example, local or global linear normalization methods can be used. Other examples can include using a Generative Adversarial Network (GAN) for mapping the intensities of real versus synthetic images.

[0102] FIG. 8 is a flowchart illustrating example operations of the segmentation processing logic 420, as shown in FIG. 4, in performing process 800. The process 800 can be embodied in computer-readable instructions for execution by one or more processors such that the operations of the process 800 can be performed in part or in whole by the functional components of the segmentation processing logic 420; accordingly, the process 800 is described below by way of example with reference thereto. However, in other examples, at least some of the operations of the process 800 can be deployed on various other hardware configurations. The process 800 is therefore not intended to be limited to the segmentation processing logic 420 and can be implemented in whole, or in part, by any other component. Some or all of the operations of process 800 can be in parallel, out of order, or entirely omitted.

[0103] At operation 810, segmentation processing logic 420 receives training data. For example, segmentation processing logic 420 receives pairs of segmentation values corresponding with a medical image and a probability map. The segmentation values can each correspond with a region of a medical image (e.g., an organ or a tumor within the medical image). The probability map can include a probability of the region corresponding with the segmentation value being part of the medical image. The training data can be used in the training of the segmentation model. For example, constraints of the segmentation model can be adjusted based on the probability map. The constraints of the segmentation model can be related to the segmentation rules used to segment a medical image. For example, the probability map can indicate that a region of a medical image is more likely to be part of an organ than part of a tumor in the medical image. Accordingly, the segmentation model can be trained to more heavily favor segmenting the region as part of the organ than as part of the tumor. At operation 830, segmentation processing logic 420 performs training of the model. The training can be performed in a supervised or unsupervised manner.

[0104] At operation 840, segmentation processing logic 420 outputs the trained model. For example, the trained model can be output and stored in a memory or parameters of the model can be presented on a display device to a clinician.

[0105] At operation 850, segmentation processing logic 420 utilizes the trained model to generate results. For example, after each of the machine learning models Ae (sometimes referred to as A0) is trained, new data, including one or more patient input parameters (e.g., radiotherapy treatment plan information), can be received. The trained machine learning technique Ae can be applied to the new data to generate generated results including one or more estimated probability maps for a given medical image.

[0106] The above detailed description can include references to the accompanying drawings, which form a part of the detailed description. The drawings show, by way of illustration but not by way of limitation, specific examples in which the invention can be practiced. These examples are also referred to herein as “examples.” Such examples can include elements in addition to those shown or described. However, the present inventors also contemplate examples in which only those elements shown or described are provided. Moreover, the present inventors also contemplate examples using any combination or permutation of those elements shown or described (or one or more aspects thereof), either with respect to a particular example (or one or more aspects thereof), or with respect to other examples (or one or more aspects thereof) shown or described herein.

[0107] All publications, patents, and patent documents referred to in this document are incorporated by reference herein in their entirety, as though individually incorporated by reference. In the event of inconsistent usages between this document and those documents so incorporated by reference, the usage in the incorporated reference(s) should be considered supplementary to that of this document; for irreconcilable inconsistencies, the usage in this document controls.

[0108] In this document, the terms “a,” “an,” “the,” and “said” are used when introducing elements of aspects of the invention or in the examples thereof, as is common in patent documents, to include one or more than one or more of the elements, independent of any other instances or usages of “at least one” or “one or more.” In this document, the term “or” is used to refer to a nonexclusive or, such that “A or B” can include “A but not B,” “B but not A,” and “A and B,” unless otherwise indicated.

[0109] In the appended claims, the terms “including” and “in which” are used as the plain-English equivalents of the respective terms “comprising” and “wherein.” Also, in the following claims, the terms “comprising,” “including,” and “having” are intended to be open-ended to mean that there can be additional elements other than the listed elements, such that after such a term (e.g., comprising, including, having) in a claim are still deemed to fall within the scope of that claim. Moreover, in the following claims, the terms “first,” “second,” and “third,” etc., are used merely as labels, and are not intended to impose numerical requirements on their objects.

[0110] The present invention also relates to a computing system adapted, configured, or operated for performing the operations herein. This system can be specially constructed for the required purposes, or it can comprise a general purpose computer selectively activated or reconfigured by a computer program (e.g., instructions, code, etc.) stored in the computer. The order of execution or performance of the operations in examples of the invention illustrated and described herein is not essential, unless otherwise specified. That is, the operations can be performed in any order, unless otherwise specified, and examples of the invention can include additional or fewer operations than those disclosed herein. For example, it is contemplated that executing or performing a particular operation before, contemporaneously with, or after another operation is within the scope of aspects of the invention. [OlH] In view of the above, it will be seen that the several objects of the invention are achieved, and other advantageous results attained. Having described aspects of the invention in detail, it will be apparent that modifications and variations are possible without departing from the scope of aspects of the invention as defined in the appended claims. As various changes could be made in the above constructions, products, and methods without departing from the scope of aspects of the invention, it is intended that all matter contained in the above description and shown in the accompanying drawings shall be interpreted as illustrative and not in a limiting sense.

[0112] The above description is intended to be illustrative, and not restrictive. For example, the above-described examples (or one or more aspects thereof) can be used in combination with each other. In addition, many modifications can be made to adapt a particular situation or material to the teachings of the invention without departing from its scope. While the dimensions, types of materials and example parameters, functions, and implementations described herein are intended to define the parameters of the invention, they are by no means limiting and are exemplary examples. Many other examples will be apparent to those of skill in the art upon reviewing the above description. The scope of the invention should, therefore, be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

[0113] Also, in the above Detailed Description, various features can be grouped together to streamline the disclosure. This should not be interpreted as intending that an unclaimed disclosed feature is essential to any claim. Rather, inventive subject matter can lie in less than all features of a particular disclosed example. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separate example. The scope of the invention should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

[0114] Example 1 is a method for supporting radiation treatment planning, the method comprising: receiving a set of images of at least one patient; receiving a set of non-binary segmentation values, each value in the set within a range of possible segmentation values, the set of non-binary segmentation values assigning a probability that an individual pixel or voxel of one of the set of images corresponds to a specified anatomical structure of the at least one patient; training a machine learning model using the set of non-binary segmentation values paired with respective images of the set of images, the machine learning model trained to output a probability map indicating non-binary segmentation values for pixels or voxels of a provided image; and outputting the trained machine learning model.

[0115] In Example 2, the subject matter of Example 1 includes, wherein the probability map, when output by the trained machine learning model, indicates a probability that an individual pixel of an image of the set of images represents a contour of the specified anatomical structure of the at least one patient.

[0116] In Example 3, the subject matter of Examples 1-2 includes, wherein the set of non-binary segmentation values is received via radiation oncologist labels of images in the set of images.

[0117] In Example 4, the subject matter of Examples 1-3 includes, wherein the set of images includes a magnetic resonance (MR) image.

[0118] In Example 5, the subject matter of Examples 1-4 includes, wherein the set of images includes a cone beam computed tomography (CBCT) image.

[0119] In Example 6, the subject matter of Examples 1-5 includes, wherein training the machine learning model includes using additional probabilities of individual pixels corresponding to the specified anatomical structure of the at least one patient, the additional probabilities based on a deformable image registration (DIR).

[0120] In Example 7, the subject matter of Examples 1-6 includes, wherein the set of images correspond to images of a plurality of patients.

[0121] In Example 8, the subject matter of Examples 1-7 includes, automatically detecting a peripheral contour of the specified anatomical structure.

[0122] In Example 9, the subject matter of Example 8 includes, wherein automatically detecting the peripheral contour of the specified anatomical structure includes using a plurality of binary segmentation values.

[0123] In Example 10, the subject matter of Examples 1-9 includes, wherein receiving the set of non-binary segmentation values includes receiving non-binary segmentation values selected by at least two radiation oncologists for a single image in the set of images.

[0124] In Example 11, the subject matter of Examples 1-10 includes, wherein outputting the trained machine learning model includes saving machine learning model data to a database, the database accessible to a physician for use in planning a radiation treatment.

[0125] Example 12 is a method for supporting radiation treatment planning, the method comprising: receiving an input image for a patient; generating, using a machine learning model, a patient probability map corresponding to the input image for the patient, the patient probability map indicating non-binary segmentation values for pixels or voxels of the input image for the patient, and outputting the patient probability map for oncology dose planning, wherein the machine learning model is trained using training data that comprises: a set of images, and a set of corresponding non-binary segmentation values, each value in the set within a range of possible segmentation values, the set of non-binary segmentation values assigning a probability that an individual pixel or voxel of one of the set of images corresponds to a specified anatomical structure of the at one patient.

[0126] In Example 13, the subject matter of Example 12 includes, generating, using the patient probability map, a visual representation of a contour of a portion of anatomy of the patient.

[0127] In Example 14, the subject matter of Example 13 includes, wherein the visual representation includes a dose-volume histogram (DVH) generated using the patient probability map, the DVH representing a range of dosage values for the portion of anatomy, the range of dosage values based on the patient probability map.

[0128] In Example 15, the subject matter of Example 14 includes, wherein the DVH includes a plurality of visually distinguishable sections each corresponding to a range of probabilities from the patient probability map.

[0129] In Example 16, the subject matter of Examples 12-15 includes, wherein the patient probability map includes additional probabilities of individual pixels corresponding to a portion of anatomy of the patient, the additional probabilities based on at least one of a deformable image registration (DIR), a measured organ motion over time, a measured organ deformation over time, or an image distortion.

[0130] In Example 17, the subject matter of Examples 1-16 includes, wherein the set of non-binary segmentation values includes non-binary segmentation values for a plurality of anatomical structures in each image of the set of images, the anatomical structures including at least one target and at least one organ at risk.

[0131] Example 18 is at least one machine-readable medium, including instructions for supporting radiation treatment planning, which when executed by processing circuitry, causes the processing circuitry to perform operations to: receive a set of images of at least one patient; receive a set of non-binary segmentation values, each value in the set within a range of possible segmentation values, the set of non-binary segmentation values assigning a probability that an individual pixel or voxel of one of the set of images corresponds to a specified anatomical structure of the at least one patient; train a machine learning model using the set of non-binary segmentation values paired with respective images of the set of images, the machine learning model trained to output a probability map indicating non-binary segmentation values for pixels or voxels of a provided image; and output the trained machine learning model.

[0132] In Example 19, the subject matter of Example 18 includes, wherein the probability map, when output by the trained machine learning model, indicates a probability that an individual pixel of an image of the set of images represents a contour of the specified anatomical structure of the at least one patient.

[0133] In Example 20, the subject matter of Examples 18-19 includes, wherein the set of images includes a cone beam computed tomography (CBCT) image.

[0134] Example 21 is a method for planning a radiation treatment, the method comprising: providing or receiving a first tissue image of a patient; receiving, via user input, a plurality of a non-binary segmentation values within a predetermined range of possible segmentation values, an individual segmentation value associating an individual pixel or voxel of the first tissue image to a first anatomical structure of the patient; generating a probability map of the first tissue image based on the plurality of non-binary segmentation values, the probability map including a representation of user-confidence of the association of the individual pixel or voxel to the first anatomical structure; using the probability map as an 34 input to a train a machine learning (ML) model, the ML model trained to establish or adjust a threshold segmentation value for associating an individual pixel or voxel of a second tissue image to the anatomical structure of the patient.

[0135] In Example 22, the subject matter of Example 21 includes, confirming or correcting an individual user input non-binary segmentation value based on the threshold segmentation value.

[0136] In Example 23, the subject matter of Example 22 includes, using the confirmed or corrected non-binary segmentation values to determine a deformable image registration (DIR) including the first tissue image and the second tissue image.

[0137] In Example 24, the subject matter of Examples 21-23 includes, performing a radiotherapy dosage calculation based in part on the plurality of confirmed or corrected non-binary segmentation values.

[0138] In Example 25, the subject matter of Examples 21-24 includes, performing a plurality of radiotherapy dosage calculations each corresponding to an individual pixel or voxel of the first tissue image based on the plurality of confirmed or corrected respective non-binary segmentation values.

[0139] In Example 26, the subject matter of Examples 21-25 includes, wherein generating the probability map is also based on one of: measured organ motion overtime, measured organ deformation over time, or image distortion.

[0140] In Example 27, the subject matter of Examples 21-26 includes, using the non-binary segmentation values to generate a visual representation configured for planning dosage of radiation treatment.

[0141] In Example 28, the subject matter of Example 27 includes, wherein the visual representation includes a dose-volume histogram (DVH).

[0142] In Example 29, the subject matter of Example 28 includes, wherein volume data of the DVH corresponding to an individual dosage includes a range of values based on the non-binary segmentation values.

[0143] In Example 30, the subject matter of Example 29 includes, wherein the DVH includes a planar representation, including an area, of the respective ranges of values for a plurality of dosages.

[0144] In Example 31, the subject matter of Example 30 includes, wherein the planar representation includes a plurality of sections each corresponding to an individual accuracy probability of the non-binary segmentation values.

[0145] In Example 32, the subject matter of Example 31 includes, wherein individual sections are represented by different colors.

[0146] In Example 33, the subject matter of Examples 21-32 includes, automatically detecting an outside body contour of the first anatomical structure.

[0147] In Example 34, the subject matter of Examples 21-33 includes, superimposing at least one visual identifier of the first anatomical structure on the first tissue image, superimposing being based on the non-binary segmentation values.

[0148] Example 35 is at least one machine-readable medium including instructions that, when executed by processing circuitry, cause the processing circuitry to perform operations to implement of any of Examples 1-34.

[0149] Example 36 is an apparatus comprising means to implement of any of Examples 1-34.

[0150] Example 37 is a system to implement of any of Examples 1-34.

[0151] Example 38 is a method to implement of any of Examples 1-34.

[0152] Each of these non-limiting examples can stand on its own, or can be combined in various permutations or combinations with one or more of the other examples.

[0153] Method examples described herein can be machine or computer-implemented at least in part. Some examples can include a computer-readable medium or machine-readable medium encoded with instructions operable to configure an electronic device to perform methods as described in the above examples. An implementation of such methods can include code, such as microcode, assembly language code, a higher-level language code, or the like. Such code can include computer readable instructions for performing various methods. The code can form portions of computer program products. Further, in an example, the code can be tangibly stored on one or more volatile, non-transitory, or non-volatile tangible computer-readable media, such as during execution or at other times. Examples of these tangible computer-readable media can include, but are not limited to, hard disks, removable magnetic disks, removable optical disks (e.g., compact disks and digital video disks), magnetic cassettes, memory cards or sticks, random access memories (RAMs), read only memories (ROMs), and the like.

Claims

16 07 25What is claimed is:

1. A method for supporting radiation treatment planning, the method comprising:receiving a set of images of at least one patient;receiving a set of non-binary segmentation values, each value in the set within a range of possible segmentation values, the set of non-binary segmentation values assigning a probability that an individual pixel or voxel of one of the set of images corresponds to a specified anatomical structure of the at least one patient;training a machine learning model using the set of non-binary segmentation values paired with respective images of the set of images, the machine learning model trained to output a probability map indicating non-binary segmentation values for pixels or voxels of a provided image;wherein training the machine learning model includes using additional probabilities of individual pixels corresponding to the specified anatomical structure of the at least one patient, the additional probabilities based on at least one of a deformable image registration (DIR), a measured organ motion over time, a measured organ deformation over time, or an image distortion; andoutputting the trained machine learning model.

2. The method of claim 1, wherein the probability map, when output by the trained machine learning model, indicates a probability that an individual pixel of an image of the set of images represents a contour of the specified anatomical structure of the at least one patient.

3. The method of any one of claims 1-2, wherein the set of non-binary segmentation values is received via radiation oncologist labels of images in the set of images.

4. The method of any one of claims 1-3, wherein the set of images includes a magnetic resonance (MR) image.

5. The method of any one of claims 1-4, wherein the set of images includes a cone beam computed tomography (CBCT) image.16 07 256. The method of any one of claims 1-5, wherein the set of images correspond to images of a plurality of patients.

7. The method of any one of claims 1-6, further comprising automatically detecting a peripheral contour of the specified anatomical structure.

8. The method of claim 7, wherein automatically detecting the peripheral contour of the specified anatomical structure includes using a plurality of binary segmentation values.

9. The method of any one of claims 1-8, wherein receiving the set of non-binary segmentation values includes receiving non-binary segmentation values selected by at least two radiation oncologists for a single image in the set of images.

10. The method of any one of claims 1-9, wherein outputting the trained machine learning model includes saving machine learning model data to a database, the database accessible to a physician for use in planning a radiation treatment.

11. A method for supporting radiation treatment planning, the method comprising: receiving an input image for a patient;generating, using a machine learning model, a patient probability map corresponding to the input image for the patient, the patient probability map indicating non-binary segmentation values for pixels or voxels of the input image for the patient, andoutputting the patient probability map for oncology dose planning, wherein the machine learning model is trained using training data that comprises:a set of images, anda set of corresponding non-binary segmentation values, each value in the set within a range of possible segmentation values, the set of non-binary segmentation values assigning a probability that an individual pixel or voxel of one of the set of images corresponds to a specified anatomical structure of the at one patient;16 07 25wherein the patient probability map includes additional probabilities of individual pixels corresponding to a portion of anatomy of the patient, the additional probabilities based on at least one of a deformable image registration (DIR), a measured organ motion over time, a measured organ deformation over time, or an image distortion.

12. The method of claim 11, further comprising generating, using the patient probability map, a visual representation of a contour of a portion of anatomy of the patient.

13. The method of claim 12, wherein the visual representation includes a dose-volume histogram (DVH) generated using the patient probability map, the DVH representing a range of dosage values for the portion of anatomy, the range of dosage values based on the patient probability map.

14. The method of claim 13, wherein the DVH includes a plurality of visually distinguishable sections each corresponding to a range of probabilities from the patient probability map.

15. The method of any one of claims 1-14, wherein the set of non-binary segmentation values includes non-binary segmentation values for a plurality of anatomical structures in each image of the set of images, the anatomical structures including at least one target and at least one organ at risk.

16. At least one machine-readable medium, including instructions for supporting radiation treatment planning, which when executed by processing circuitry, causes the processing circuitry to perform operations according to the method of any preceding claim.

Images