Comparing healthcare provider profiles using automated tools

Abstract

Using a computer-implemented intermediary, contouring performed by two participants, such as two physicians, can be compared through the computer-implemented intermediary. First, contouring performed by each participant can be compared to contouring performed by the intermediary. Then, contouring performed by each participant can be compared through common intermediary and transfer analysis.

Description

[0001] Priority requirements

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application Serial No. 63 / 038,081, entitled “COMPARING CONTOURS OF SUBJECTS BY DIFFERENT PARTICIPANTS”, filed June 11, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This document generally relates to radiation therapy, and more specifically, but not in a limited way, to systems and methods for use in adaptive radiotherapy. Background Technology

[0004] Radiation therapy, also known as radiotherapy, is used to treat tumors and other diseases in the tissues of mammals (e.g., humans and animals). An example of radiation therapy is the application of a high-energy beam from an external source towards the patient to produce a collimated beam of radiation directed towards a target site on the patient. The target can be an area in the patient's body containing a diseased organ or tumor that will be exposed to and treated by the radiation beam. The placement and dosage of the radiation beam must be precisely controlled to ensure that the target receives the radiation dose prescribed by the physician for the patient, but to minimize damage to surrounding healthy tissue—often referred to as an organ of risk (OAR).

[0005] To plan a patient's radiation therapy treatment, one or more medical images of the patient at the intended treatment location are acquired before the radiation therapy (RT) session, and these images are typically acquired many days before treatment begins. These medical images are called planning images.

[0006] Physicians can use planning images to identify and manually map the contours of one or more targets and the OAR. A treatment contour, often called the planning target area (PTV), is created, comprising the target contour plus sufficient margins to account for microscopic lesions and treatment uncertainties. The physician specifies the radiation dose and creates a radiation therapy treatment plan that optimally delivers the prescribed dose to the PTV while minimizing the dose to the OAR and other normal tissues. Treatment plans can be generated manually by the physician or automatically using optimization techniques. Optimization techniques can be based on clinical and dosimetric goals and constraints (e.g., maximum, minimum, and average radiation doses to the tumor and OAR).

[0007] Treatment plans are developed to deliver a prescribed dose through multiple fractions, each delivered within a separate treatment session. For example, 30 to 40 fractions are typical, but five or even one fraction can be used. These fractions are typically delivered once per weekday or, in some cases, twice. In some cases, the radiation therapy plan can be altered throughout the treatment to concentrate more dose in certain areas. In each fraction, the patient is positioned on the patient support attachment of the radiation therapy device (often referred to as an "examination table") and repositioned as close as possible to the patient's position in the planned image. Attached Figure Description

[0008] In accompanying drawings that are not necessarily drawn to scale, the same reference numerals may describe similar parts in different figures. The same reference numerals with different letter suffixes may indicate different instances of similar parts. The accompanying drawings generally illustrate the various embodiments discussed in this document by way of example rather than limitation.

[0009] Figure 1 An example of a radiotherapy system for delivering radiation therapy to a patient is shown, which can be used to implement and / or perform embodiments of this disclosure.

[0010] Figure 2 An example of a radiation therapy system is shown, which may include a radiation therapy output configured to provide a therapeutic beam.

[0011] Figure 3 This is a conceptual diagram illustrating the use of a computer-implemented automatic segmentation engine based on various techniques according to this disclosure, which can compare image contour plots of two or more different healthcare provider participants without requiring different participants to perform contour plotting on a shared image dataset.

[0012] Figure 4A and Figure 4B It is a graphical representation of the comparison between two image datasets.

[0013] Figures 5A to 5D Describing the comparison Figures 4A to 4B Example of a graph depicting two image datasets.

[0014] Figure 6 A block diagram illustrating an example machine on which any or more of the techniques (e.g., methods) discussed in this article can be executed.

[0015] Figure 7This is a flowchart illustrating an example of a computer-implemented method for comparing image contour drawing by different human participants without requiring different participants to perform contour drawing on a shared image dataset.

[0016] Figure 8 These are illustrations of the components of client and server devices based on various examples.

[0017] Figure 9 This is an example of a user interface associated with a computer-implemented method for comparing image contour drawing by different human participants without requiring different participants to perform contour drawing on a shared image dataset.

[0018] Figure 10 This is another example of a user interface associated with a computer-implemented method that compares image contour drawing by different human participants without requiring different participants to draw contours on a shared image dataset. Summary of the Invention

[0019] The inventors have recognized the necessity of using a computer-implemented intermediary to compare profiling performed by two or more healthcare provider participants, such as physicians or dosimeters. In some examples, one or both of the first and second participants may be a group or cluster of participants. First, the profiling performed by each participant can be compared with the profiling performed by the intermediary. Then, through a common intermediary and transitive analysis, the profiling performed by each participant can be compared.

[0020] In some aspects, this disclosure relates to a computer-implemented method for comparing image contour plots of different human participants without requiring contour plotting of a shared image dataset by different participants. The method includes: receiving a first image dataset set comprising one or more first image datasets of one or more human or animal subjects generated by an imaging modality, and including first contour plotting data generated by a first participant; receiving a second image dataset set comprising one or more second image datasets of one or more human or animal subjects generated by an imaging modality, and including second contour plotting data generated by a second participant; automatically contour plotting each of the first and second image datasets using a computer-implemented automatic segmentation engine to obtain third contour plotting data generated by the automatic segmentation engine, without requiring manual contour plotting and without requiring identical or overlapping first and second image datasets; and comparing the contour plotting of the first participant and the contour plotting of the second participant, including comparing the first contour plotting data generated by the first participant and the third contour plotting data generated by the automatic segmentation engine to generate first comparison data, and further comparing the second contour plotting data generated by the second participant and the third contour plotting data generated by the automatic segmentation engine to generate second comparison data.

[0021] In some aspects, this disclosure relates to a computer-implemented system for comparing image contour plots of different human participants without requiring different participants to perform contour plotting on a shared image dataset. The system includes: a computer-readable medium having instructions stored thereon, the instructions, when executed by a processor, causing the system to: receive a first image dataset set, including one or more first image datasets of one or more human or animal subjects generated by an imaging modality, and including first contour plotting data generated by a first participant; receive a second image dataset set, including one or more second image datasets of one or more human or animal subjects generated by an imaging modality, including second contour plotting data generated by a second participant; perform automatic contour plotting on each of the first and second image datasets using a computer-implemented automatic segmentation engine to obtain third contour plotting data generated by the automatic segmentation engine, without requiring manual contour plotting and without requiring identical or overlapping first and second image datasets; and compare the contour plotting of the first participant with the contour plotting of the second participant, including comparing the first contour plotting data generated by the first participant with the third contour plotting data generated by the automatic segmentation engine to generate first comparison data, and also comparing the second contour plotting data generated by the second participant with the third contour plotting data generated by the automatic segmentation engine to generate second comparison data.

[0022] In some aspects, this disclosure relates to a computer-readable storage medium for comparing image contour plots of different human participants without requiring different participants to perform contour plotting on a shared image dataset. The computer-readable medium has instructions stored thereon that, when executed by a processor, cause one or more machines to: receive a first set of image datasets comprising one or more first image datasets of one or more human or animal subjects generated by an imaging modality, and including first contour plotting data generated by a first participant; receive a second set of image datasets comprising one or more second image datasets of one or more human or animal subjects generated by an imaging modality, including second contour plotting data generated by a second participant; perform automatic contour plotting on each of the first and second image datasets using a computer-implemented automatic segmentation engine to obtain third contour plotting data generated by the automatic segmentation engine, without requiring manual contour plotting and without requiring identical or overlapping first and second image datasets; and compare the contour plotting of the first participant with the contour plotting of the second participant, including comparing the first contour plotting data generated by the first participant with the third contour plotting data generated by the automatic segmentation engine to generate first comparison data, and also comparing the second contour plotting data generated by the second participant with the third contour plotting data generated by the automatic segmentation engine to generate second comparison data. Detailed Implementation

[0023] As mentioned above, physicians can use medical images to identify and manually outline one or more targets and organs at risk (OARs). However, reading and interpreting medical images can be largely subjective, with disagreements arising among multiple trained observers. This is especially true when it comes to accurately depicting organs or other critical anatomical volumes such as tumors or other lesions or injuries.

[0024] The pursuit of accuracy implies a pursuit of consensus, facilitated through a comprehensive study of discrepancies, followed by negotiation until agreement is reached among experts. This process can be laborious and may require highly planned approaches, where each team independently processes a large sample of shared datasets, then analyzes the differences qualitatively and / or visually and identifies trends. In addition to the labor involved, privacy concerns regarding patient images are also a significant consideration.

[0025] Consider the following scenario. Two radiation oncologists (Dr. A and Dr. C) specialize in pediatric brain cancer. Each of them uses input from computed tomography (CT), magnetic resonance (MR), and / or positron emission tomography (PET) images to carefully delineate the radiation target area and reserved critical organs for each patient. These anatomical delineations, or "outlines," serve as a three-dimensional (3D) blueprint to determine where to place (and where to avoid) radiation doses.

[0026] Doctor A and Doctor C have drawn outlines for hundreds of different patients in their practice, but they have never drawn an outline for the same patient. Neither Doctor A nor Doctor C (or their management) knows whether their methods of drawing anatomical outlines are similar, let alone identical, for the different organs and volumes they depict. If discrepancies exist, they must be understood, and ideally eliminated, so that both physicians can use the most accurate and consistent approach.

[0027] Doctors A and C could take on a large project of profiling each other for a shared patient pool, but they are already busy enough, plus their management doesn’t want to incur bias (e.g., different behaviors, because Doctors A and C know the profiles will be scrutinized).

[0028] To address the aforementioned problems, the inventors have recognized the necessity of using a computer-implemented intermediary (e.g., an automated tool) that can compare profiling performed by two or more healthcare provider participants, such as two or more individuals (e.g., physicians or dosimeters), one or more individual healthcare providers and one or more groups or communities of healthcare provider participants, or two or more groups or communities of healthcare provider participants. First, the profiling performed by each participant can be compared with the profiling performed by the intermediary (e.g., an automated tool). Then, through a common intermediary and transitive analysis, the profiling performed by each participant can be compared.

[0029] The techniques described in this disclosure can utilize the history of work already performed by participants in practice. These techniques do not require the establishment of controlled, cleaned, and shared datasets; the techniques can define anatomy from scratch on these datasets solely for research purposes.

[0030] By using the various techniques described in more detail below, meaningful feedback can be generated based on comparisons for each anatomical structure or substructure, organ, lesion, injury, etc. For example, differences in size / volume, left / right dimensions, anterior / posterior dimensions, and / or superior / inferior dimensions are some examples of feedback that can be generated. In some examples, these differences can be systematic.

[0031] Figure 1 An example of a radiotherapy system 10 for delivering radiotherapy to a patient is shown, which can be used to implement and / or perform embodiments of this disclosure. The radiotherapy system 10 includes an image processing device 12. The image processing device 12 can be connected to a network 20. The network 20 can be connected to the Internet 22. The network 20 can connect the image processing device 12 to one or more of a database 24, a hospital database 26, an oncology information system (OIS) 28, a radiotherapy device 30, an image acquisition device 32, a display device 34, and / or a user interface 36. The image processing device 12 can be configured to generate one or more radiotherapy treatment plans 42 to be used by the radiotherapy device 30.

[0032] Image processing apparatus 12 may include memory 16, image processor 14, and / or communication interface 18. Memory 16 may store computer-executable instructions, such as operating system 43, one or more radiation therapy treatment plans 42 (e.g., original treatment plans and / or adaptively adjusted treatment plans), software program 44 (e.g., artificial intelligence, deep learning, neural networks, and / or radiation therapy treatment planning software), and / or any other computer-executable instructions to be executed by image processor 14. In some embodiments, software program 44 may convert a medical image of one format (e.g., MRI) to another format (e.g., CT) by generating synthetic images such as pseudo-CT images. For example, software program 44 may include an image processing program to train a predictive model for converting a medical image 46 of one modality (e.g., MR image) to a synthetic image of a different modality (e.g., pseudo-CT image); alternatively, the trained predictive model may convert a CT image to an MRI image. Memory 16 may store data, including medical image 46, patient data 45, and / or other data required to create and / or implement radiation therapy treatment plan 42.

[0033] In addition to the memory 16 storing the software program 44, or alternatively the memory 16 storing the software program 44, it is conceivable that the software program 44 can be stored on a removable computer medium, such as a hard disk drive, computer disk, CD-ROM, DVD, HD, Blu-ray DVD, USB flash drive, SD card, memory stick, or any other suitable medium. The software program 44 can be executed by the image processor 14 when downloaded to it.

[0034] Image processor 14 may be communicatively coupled to memory 16, and image processor 14 may be configured to execute computer-executable instructions stored thereon. Image processor 14 may send medical images 46 to or receive medical images 46 from memory 16. For example, image processor 14 may receive medical images 46 from image acquisition device 32 or another image acquisition device via communication interface 18 and network 18 for storage in memory 16. Image processor 14 may also send medical images 46 stored in memory 16 to network 20 via communication interface 18 for storage in database 24 and / or hospital database 26.

[0035] Furthermore, the image processor 14 can utilize software program 44 (e.g., treatment planning software) and medical images 46 and / or patient data 45 to create and / or modify radiation therapy treatment plans 42. Medical images 46 may include, for example, imaging data associated with segmentation data of patient anatomical regions, organs, or volumes of interest. Imaging data may include information about anatomical regions and organs such as, but not limited to, the lungs, liver, pelvic region, heart, and prostate. Patient data 45 may include, for example: (1) functional organ modeling data (e.g., serial vs. parallel organs, appropriate dose response models, etc.); (2) radiation dose data (e.g., dose-volume histogram (DVH) information); or (3) other clinical information about the patient and treatment process (e.g., other surgeries, chemotherapy, previous radiation therapy, etc.).

[0036] Furthermore, the image processor 14 can utilize software programs to generate intermediate data, such as updated parameters used by a neural network model, or to generate intermediate 2D or 3D images, which can then be stored in memory 16. The image processor 14 can then send an executable radiation therapy treatment plan 42 to the radiation therapy device 30 via a communication interface 18 with network 20, which can execute the radiation therapy treatment plan 42 to treat the patient using radiation. Additionally, the image processor 14 can execute software programs 44 to perform functions such as image transformation, image segmentation, deep learning, neural networks, and / or artificial intelligence. For example, the image processor 14 can execute software programs 44 for training medical images and / or drawing contours of medical images. Such software programs 44, when executed, can train boundary detectors and / or utilize shape dictionaries.

[0037] The image processor 14 may be a processing device, including, for example, one or more general-purpose processing devices such as microprocessors, central processing units (CPUs), graphics processing units (GPUs), and / or accelerated processing units (APUs). More specifically, in some embodiments, the image processor 14 may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processor implementing other instruction sets, or a processor implementing a combination of instruction sets. The image processor 14 may also be implemented by one or more special-purpose processing devices, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), system-on-a-chip (SoCs), or other suitable processors. As those skilled in the art will understand, in some embodiments, the image processor 14 may be a special-purpose processor rather than a general-purpose processor. The image processor 14 may include one or more known processing devices, such as those from Intel. TM Manufactured Pentium TM Core TM Xeon TM or Itanium TM series, by AMD TM Turion manufactured TM Athlon TM Sempron TM Opteron TM FX TM Phenom TM The graphics processor 14 may include any processor from the Sun Microsystems family or any of the various processors manufactured by Sun Microsystems. The graphics processor 14 may also include a graphics processing unit, such as a GPU from the GeForce®, Quadro®, or Tesla® family manufactured by Nvidia™, the GMA or Iris™ family manufactured by Intel™, or the Radeon™ family manufactured by AMD™. The graphics processor 14 may also include an accelerated processing unit, such as one from AMD… TM The Desktop A-4 (6, 8) series manufactured by or by Intel TM Xeon Phi manufactured TM Series. The disclosed implementations are not limited to any type of processor otherwise configured to meet the computational needs of identifying, analyzing, maintaining, generating and / or providing large amounts of data or manipulating such data to perform the methods disclosed herein.

[0038] Furthermore, the term "processor" can include more than one processor, such as a multi-core design or multiple processors each having a multi-core design. The image processor 14 can be configured to execute a sequence of computer program instructions, such as those stored in memory 16, to perform various operations, processes, and methods according to examples of this disclosure.

[0039] The memory 16 may store medical images 46. In some embodiments, the medical images 46 may include, for example, one or more MR images (e.g., 2D MRI, 3D MRI, 2D streaming MRI, 4D MRI, 4D volumetric MRI, 4D cinematic MRI, etc.), functional MRI images (e.g., fMRI, DCE-MRI, diffusion MRI), CT images (e.g., 2D CT, CBCT, 3D CT, 4D CT), ultrasound images (e.g., 2D ultrasound, 3D ultrasound, 4D ultrasound), PET images, X-ray images, fluoroscopic images, radiotherapy field images, SPECT images, and / or computer-generated composite images (e.g., pseudo-CT images). Furthermore, the medical images 46 may include medical image data, such as training images, ground truth images, and / or contour images. Images stored in the memory 16 may include registered and / or unregistered images, and the images may have been pre-processed or may be raw, unprocessed images. In some embodiments, the medical images 46 may be received from the image acquisition device 32. Accordingly, the image acquisition device 32 may include an MR imaging device, a CT imaging device, a PET imaging device, an ultrasound imaging device, a fluorescence fluoroscopy device, a SPECT imaging device, an integrated linear accelerator and MR imaging device, or other medical imaging devices for acquiring medical images of a patient. The image processing device 12 may receive and store medical images 46 using any type of data or any type of format to perform operations conforming to the disclosed embodiments.

[0040] Memory 16 may be a non-transitory computer-readable medium, such as read-only memory (ROM), phase-change random access memory (PRAM), static random access memory (SRAM), flash memory, random access memory (RAM), dynamic random access memory (DRAM) (such as synchronous DRAM (SDRAM)), electrically erasable programmable read-only memory (EEPROM), static memory (e.g., flash memory, flash disk, static random access memory), or any other suitable type of random access memory, such as cache memory, registers, compact disk read-only memory (CD-ROM), digital universal disk (DVD) or other optical storage devices, magnetic tape, other magnetic storage devices, or any other non-transitory medium that can be used to store images, data, or computer-executable instructions (e.g., stored in any format) that can be accessed by image processor 14 or any other type of computer device. Computer program instructions may be accessed by image processor 14, read from ROM or any other suitable memory location, and loaded into RAM for execution by image processor 14. For example, memory 16 may store one or more software applications. The software application stored in memory 16 may include, for example, an operating system 43 for a public computer system and for a device used for software control. Furthermore, memory 16 may store the entire software application or only the portion of the software application that can be executed by image processor 14. For example, memory 16 may store one or more radiation therapy treatment plans 42.

[0041] Image processing device 12 can communicate with network 20 via communication interface 18, which can be communicatively coupled to image processor 14 and memory 16. Communication interface 18 provides a communication connection between image processing device 12 and components of radiotherapy system 10 (e.g., allowing data exchange with external devices). For example, in some embodiments, communication interface 18 may have suitable interface circuitry to connect to user interface 36, which may be, for example, a hardware keyboard, keypad, and / or touchscreen through which a user can input information into radiotherapy system 10.

[0042] Communication interface 18 may include one or more of the following: a network adapter, cable connector, serial connector, USB connector, parallel connector, high-speed data transmission adapter (e.g., fiber optic, USB 3.0, Thunderbolt), wireless network adapter (e.g., WiFi adapter), telecommunications adapter (e.g., 3G, 4G / LTE), or other suitable interfaces. Communication interface 18 may include one or more digital and / or analog communication devices that allow image processing device 12 to communicate with other machines and devices, such as remotely located components, via network 20.

[0043] Network 20 may provide functionality such as a local area network (LAN), wireless network, cloud computing environment (e.g., Software as a Service, Platform as a Service, Infrastructure as a Service, etc.), client server, or wide area network (WAN). For example, network 20 may be a LAN or WAN, and may include other systems S1 (38), S2 (40), and S3 (41). Systems S1, S2, and S3 may be the same as image processing device 12 or may be different systems. In some embodiments, one or more systems in network 20 may form a distributed computing / simulation environment that can collaboratively perform the embodiments described herein. In some embodiments, one or more systems S1, S2, and S3 may include a CT scanner that acquires CT images (e.g., medical image 46). Furthermore, network 20 may be connected to the Internet 22 to communicate with servers and clients residing remotely on the Internet.

[0044] Therefore, network 20 allows data transmission between image processing device 12 and multiple different other systems and devices such as OIS 28, radiation therapy device 30, and / or image acquisition device 32. Furthermore, data generated by OIS 28 and / or image acquisition device 32 can be stored in memory 16, database 24, and / or hospital database 26. As needed, data can be sent / received via communication interface 18 through network 20 for access by image processor 14.

[0045] Image processing device 12 can communicate with database 24 via network 20 to send / receive various types of data stored on database 24. For example, database 24 may include machine data containing information associated with radiation therapy device 30, image acquisition device 32, and / or other machines / devices related to radiation therapy. Machine data information may include radiation beam size, arc arrangement, beam on and off durations, control points, segments, MLC configuration, gantry speed, MRI pulse sequences, and / or other suitable information. Database 24 may be a storage device. Those skilled in the art will understand that database 24 may include multiple devices located in a centralized or distributed manner.

[0046] In some examples, database 24 may include processor-readable storage media. While a processor-readable storage medium in some embodiments may be a single medium, the term "processor-readable storage medium" should be understood to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) storing one or more sets of computer-executable instructions or data. The term "processor-readable storage medium" should also be understood to include any medium capable of storing and / or encoding a set of instructions for execution by a processor and causing the processor to perform any one or more methods of this disclosure. Accordingly, the term "processor-readable storage medium" should be understood to include, but is not limited to, solid-state memory, optical, and magnetic media. For example, a processor-readable storage medium may be one or more volatile, non-transitory, or non-volatile tangible computer-readable media.

[0047] Image processor 14 can communicate with database 24 to read images into memory 16 and / or store images from memory 16 into database 24. For example, database 24 can be configured to store multiple images (e.g., 3D MRI, 4D MRI, 2D MRI slice images, CT images, 2D fluoroscopy images, X-ray images, raw data from MR or CT scans, Medical Digital Imaging and Communications (DIMCOM) data, etc.) received by database 24 from image acquisition device 32 or other image acquisition devices. Database 24 can store data to be used by image processor 14 when executing software program 44 and / or when creating radiation therapy treatment plan 42. Image processing device 12 can receive medical images 46 (e.g., 2D MRI slice images, CT images, 2D fluoroscopy images, X-ray images, 3D MR images, 4D MR images, etc.) from database 24, radiation therapy device 30 (e.g., MRI-linear accelerator), and / or image acquisition device 32 to generate treatment plan 42.

[0048] In this example, the radiotherapy system 100 may include an image acquisition device 32 configured to acquire medical images of the patient (e.g., MR images such as 3D MRI, 2D streaming MRI, or 4D volumetric MRI, CT images, CBCT, PET images, functional MR images (e.g., fMRI, DCE-MRI, and diffusion MRI), X-ray images, fluoroscopy images, ultrasound images, radiotherapy field images, SPECT images, etc.). The image acquisition device 32 may be, for example, an MR imaging device, a CT imaging device, a PET imaging device, an ultrasound device, a fluoroscopy device, a SPECT imaging device, or any other suitable medical imaging device for acquiring one or more medical images of the patient. The images acquired by the image acquisition device 32 may be stored in a database 24 as imaging data and / or test data. As an example, the images acquired by the image acquisition device 32 may also be stored in a memory 16 by an image processing device 12 as medical image data 46.

[0049] In some implementations, for example, the image acquisition device 32 may be integrated with the radiation therapy device 30 as a single device (e.g., an MRI device combined with a linear accelerator, also referred to as an "MRI-linear accelerator"). Such an MRI-linear accelerator may, for example, be used to determine the location of a target organ or target tumor in a patient in order to accurately guide radiation therapy to a predetermined target according to the radiation therapy treatment plan 42.

[0050] Image acquisition device 32 can be configured to acquire one or more images of a patient's anatomy at a region of interest (e.g., a target organ, a target tumor, or both). Each image is typically a 2D image or slice and may include one or more parameters (e.g., 2D slice thickness, orientation, location, etc.). In some embodiments, image acquisition device 32 can acquire 2D slices of any orientation. For example, the orientation of a 2D slice may include sagittal orientation, coronal orientation, or axial orientation. Image processor 14 can adjust one or more parameters, such as the thickness and / or orientation of the 2D slice, to include the target organ and / or target tumor. In an exemplary embodiment, 2D slices may be determined based on information such as 3D MRI volume. Such 2D slices can be acquired by image acquisition device 32 "in real time" while the patient is undergoing radiation therapy, for example, using radiation therapy device 30. "Real time" may mean acquiring data in milliseconds (e.g., 500 milliseconds or 300 milliseconds) or less.

[0051] The image processing device 12 can generate and store radiation therapy treatment plans 42 for one or more patients. The radiation therapy treatment plan 42 can provide information about the specific radiation dose to be applied to each patient. The radiation therapy treatment plan 42 may also include other radiation therapy information, such as beam angle, dose histogram volume information, the number of radiation beams to be used during therapy, the dose of each beam, or other suitable information or combinations thereof.

[0052] Image processor 14 can generate a radiation therapy treatment plan 42 using software program 44, such as treatment planning software (e.g., Monaco® manufactured by Elekta AB, Stockholm, Sweden). To generate the radiation therapy treatment plan 42, image processor 14 can communicate with image acquisition device 32 (e.g., CT, MRI, PET, X-ray, ultrasound, etc.) to access images of the patient and delineate targets such as tumors. In some embodiments, it may be necessary to delineate one or more organs of risk (OARs), such as healthy tissue surrounding or adjacent to the tumor. Therefore, segmentation of the OAR can be performed if it is close to the target tumor. Furthermore, if the target tumor is close to an OAR (e.g., the prostate gland adjacent to the bladder and rectum), by segmenting the OAR from the tumor, treatment planning device 110 can study not only the dose distribution in the target but also the dose distribution in the OAR.

[0053] To delineate the target organ or tumor from the OAR, medical images of the patient undergoing radiotherapy, such as MR images, CT images, PET images, fMR images, X-ray images, ultrasound images, radiotherapy field images, SPECT images, or other medical images, can be acquired by the image acquisition device 32 to reveal the internal structure of the body part. Based on the information from the medical images, the 3D structure of the relevant anatomical part can be obtained. Furthermore, during the treatment planning process, multiple parameters can be considered to achieve a balance between effective treatment of the target tumor (e.g., ensuring the target tumor receives a sufficient radiation dose for an effective therapy) and low exposure of the OAR (e.g., ensuring the OAR receives the lowest possible radiation dose). Other parameters that can be considered include the location of the target organ and target tumor, the location of the OAR, and / or the movement of the target relative to the OAR. For example, the 3D structure can be obtained by outlining the target or the OAR within each 2D layer or slice of the MRI or CT image and combining the outlines of each 2D layer or slice. The contour can be generated manually (e.g., by a physician, dosimeter, or healthcare professional) or automatically (e.g., using a program such as ABASTM, an Atlas-based automated segmentation software manufactured by Elekta AB in Stockholm, Sweden). In some implementations, the 3D structure of the target tumor or OAR can be automatically generated by treatment planning software.

[0054] After the target tumor and OAR have been located and mapped, a dosimeter, physician, or healthcare professional can determine the radiation dose to be applied to the target tumor, as well as any maximum dose that adjacent OARs (e.g., left and right parotid glands, optic nerves, eyes, lens, inner ear, spinal cord, brainstem, or other anatomical structures) can receive. After the radiation dose has been determined for the relevant anatomical structures (e.g., the target tumor, the OAR), a process known as inverse planning can be performed to determine one or more treatment planning parameters that will achieve the desired radiation dose distribution. Examples of treatment planning parameters include volumetric mapping parameters (e.g., those defining the target volume, contour-sensitive structures, etc.), margins around the target tumor and OAR, beam angle selection, collimator settings, and / or beam-on time.

[0055] During the reverse planning process, the physician can define dose constraint parameters that set limits on how much radiation an OAR can receive (e.g., limiting the full dose to the tumor target and zero dose to any OAR; limiting 95% of the dose to the target tumor; limiting the spinal cord, brainstem, and visual structures to ≤45 Gy, ≤55 Gy, and <54 Gy, respectively). The results of the reverse planning can form a radiation therapy treatment plan 42 that can be stored in memory 16 or database 24. Some of these treatment parameters can be related. For example, attempting to change the treatment plan by adjusting one parameter (e.g., the weight of different targets, such as increasing the dose to the target tumor) may affect at least one other parameter, which in turn may lead to the development of different treatment plans. Therefore, the image processing device 12 can generate a customized radiation therapy treatment plan 42 with these parameters so that the radiation therapy device 30 can deliver radiation therapy to the patient.

[0056] In addition, the radiotherapy system 10 may include a display device 34 and a user interface 36. The display device 34 may include one or more displays configured to display medical images, interface information, treatment planning parameters (e.g., contour, dose, beam angle, etc.), treatment plans, targets, target localization and / or target tracking, or any suitable information to a user. The user interface 36 may be a keyboard, keypad, touchscreen, or any type of device through which a user can input information into the radiotherapy system 10. Alternatively, the display device 34 and user interface 36 may be integrated into a device, such as a smartphone, computer, or tablet computer like an Apple iPad®, Lenovo Thinkpad®, or Samsung Galaxy.

[0057] Furthermore, any and all components of the radiotherapy system 10 can be implemented as virtual machines (e.g., VMware, Hyper-V, etc.). For example, a virtual machine can be software used as hardware. Therefore, a virtual machine can include at least one or more virtual processors, one or more virtual memories, and / or one or more virtual communication interfaces used together as hardware. For example, the image processing device 12, OIS 28, and / or image acquisition device 32 can be implemented as virtual machines. Considering the available processing power, memory, and computing power, the entire radiotherapy system 10 can be implemented as a virtual machine.

[0058] Figure 2 An example of a radiation therapy device 202 is shown, which may include a radiation source such as an X-ray source or a linear accelerator, an examination table 216, an imaging detector 214, and a radiation therapy output 204. The radiation therapy device 202 may be configured to emit a radiation beam 208 to provide therapy to a patient. The radiation therapy output 204 may include one or more attenuators or collimators, such as a multi-leaf collimator (MLC).

[0059] Return to reference Figure 2 The patient can be positioned in area 212 supported by treatment table 216 to receive a radiation therapy dose according to the radiation therapy treatment plan. Radiation therapy output 204 can be mounted or attached to table 206 or other mechanical supports. When table 216 is inserted into the treatment area, one or more chassis motors (not shown) can rotate table 206 and radiation therapy output 204 about table 216. In one embodiment, table 206 can be continuously rotatable about table 216 when table 216 is inserted into the treatment area. In another embodiment, table 206 can be rotated to a predetermined position when table 216 is inserted into the treatment area. For example, table 206 can be configured to rotate the therapy output 204 about an axis (“A”).

[0060] Both the examination table 216 and the radiation therapy output unit 204 can be moved independently to other locations around the patient, for example, by moving laterally (“T”), laterally (“L”), or rotating about one or more other axes, such as about the transverse axis (denoted as “R”). A controller (not shown) communicatively connected to one or more actuators can control the movement or rotation of the examination table 216 to properly position the patient inside or outside the radiation beam 208 according to the radiation therapy treatment plan. When both the examination table 216 and the gantry 206 can move independently of each other in multiple degrees of freedom, this allows the patient to be positioned so that the radiation beam 208 can be precisely targeted at the tumor.

[0061] Figure 2The coordinate system shown (including axes A, T, and L) may have an origin located at an isocenter 210. The isocenter 210 may be defined as the location where the central axis of the radiation therapy beam 208 intersects the origin of the coordinate axes, for example, at the location where a predetermined radiation dose is delivered to or within the patient. Alternatively, the isocenter 210 may be defined as the location where, for each rotational position of the radiation therapy output section 204 positioned by the platform 206 about axis A, the central axis of the radiation therapy beam 208 intersects the patient.

[0062] The gantry 206 may also have an attached imaging detector 214. The imaging detector 214 is preferably located opposite the radiation source 204, and in an embodiment, the imaging detector 214 may be located within the field of the therapy beam 208.

[0063] The imaging detector 214 is preferably mounted on the stage 206 opposite to the radiation therapy output section 204 to maintain alignment with the therapy beam 208. The imaging detector 214 rotates about a rotation axis as the stage 206 rotates. In embodiments, the imaging detector 214 may be a flat panel detector (e.g., a direct detector or a scintillator detector). In this way, the imaging detector 214 can be used to monitor the therapy beam 208, or it can be used to image the patient's anatomy, such as through field imaging. The control circuitry of the radiation therapy device 202 may be integrated within the system 100 or located remotely from the system 100.

[0064] In the illustrative embodiment, one or more of the examination table 216, therapy output unit 204, or gantry 206 can be automatically positioned, and the therapy output unit 204 can establish a therapy beam 208 according to a specified dose for a particular therapy delivery instance. The sequence of therapy delivery can be specified according to a radiation therapy treatment plan, for example, using one or more different orientations or positions of the gantry 206, examination table 216, or therapy output unit 204. Therapy deliveries can occur sequentially, but can intersect at desired therapy sites on or within the patient, for example, at isocenter 210. Thus, a prescribed cumulative dose of radiation therapy can be delivered to the therapy site while minimizing or avoiding damage to tissues near the therapy site.

[0065] As mentioned above, the inventors have recognized the necessity of using a computer-implemented intermediary to compare contour drawings performed by two participants, such as two individuals (e.g., physicians). In some examples, one or both of the first and second participants may be a group or community of participants. First, the contour drawing performed by each participant can be compared with the contour drawing performed by the intermediary. Then, through a common intermediary and transitive analysis, the contour drawing performed by each participant can be compared. Figure 3 The diagram presents a conceptual illustration.

[0066] Figure 3 This is a conceptual diagram illustrating the use of a computer-implemented automatic segmentation engine that can compare image contour plots of two or more different healthcare provider participants without requiring different participants to perform contour plots on a shared image dataset. Figure 3 Two image dataset sets are illustrated: a first image dataset set 300 and a second image dataset set 302. Each image dataset set may include one or more image datasets of one or more human or animal subjects generated from imaging modalities (e.g., CT images, MR images, and / or PET images), and includes contour drawing data generated by the corresponding healthcare provider participant (e.g., participant A or participant C). Participants may be individuals (e.g., physicians or dosimeters) or groups or communities of participants.

[0067] The first image dataset set 300 may include a collection of work performed by observer or participant A, such as an image library of participant A and contour drawings or depictions created by participant A, and the second image dataset set 302 may include a collection of work performed by observer or participant C, such as an image library of participant C and contour drawings or depictions created by participant C. For example, contour drawings or depictions may include anatomical structures or substructures, organs, lesions and / or injuries.

[0068] In some examples, at least one of the first image dataset set 300 and the second image dataset set 302 may include metadata in addition to contour drawing data generated by human participants. This metadata can be provided as input to the automatic segmentation engine 304.

[0069] The metadata may include at least one of the following: an indication of at least one imaging modality type or other imaging modality parameters used to generate images in the corresponding image dataset; an indication of the features of the organ or other target structure to be targeted for treatment; an indication of the features of the organ or other structure at risk of treatment; an indication of the patient demographic features of the human or animal subject corresponding to the images in the corresponding image dataset; an indication of the disease features associated with the human or animal subject corresponding to the images in the corresponding image dataset; an indication of the treatment features associated with the human or animal subject corresponding to the images in the corresponding image dataset; and an indication of the expected treatment outcome associated with the human or animal subject corresponding to the images in the corresponding image dataset.

[0070] Each observer or participant can provide a sufficient set of patient datasets they have processed, such as anonymized samples of their past patients. Each set of image datasets may include, for example, a series of images including patient volumes, such as a series of axial medical digital imaging and communication (DICOM) images, specific imaging modalities (e.g., CT images, MR images, and / or PET images), and / or a set of standard anatomical structures with contours drawn on each image set (e.g., provided as a DICOM RT structure set).

[0071] For example, each set of image datasets can represent a specific combination of imaging modalities (such as CT images, MR images, and / or PET images), body parts, and anatomical volume sets.

[0072] The computer-implemented automatic segmentation engine 304 (automatic segmentation engine "B") can receive a first image dataset set 300, including one or more first image datasets of one or more human or animal subjects generated by the imaging modality, and including first contour drawing data generated by the first participant A. For example, as Figure 3 As shown, the automatic segmentation engine 304 can receive a contour drawing medical image library of A (shown as input 306).

[0073] The automatic segmentation engine 304 can be a computer program that takes an image set as input and generates anatomical outlines of anatomical structures and substructures (e.g., substructures of the heart, organs, lesions, and injuries) without user intervention. The automatic segmentation engine 304 does not need to be 100% accurate, nor does it need to be consistent with any or all observers. However, robust and consistent behavior is expected across many different image sets of the same imaging modality and body part. In some examples, the automatic segmentation engine 304 can be a previously trained machine learning model trained using a high-quality library of image and structure examples.

[0074] Similarly, a computer-implemented automatic segmentation engine 304 (automatic segmentation engine "B") can receive a second image dataset set 302, including one or more second image datasets of one or more human or animal subjects generated by the imaging modality, including second contour drawing data generated by the second participant B. For example, as Figure 3 As shown, the automatic segmentation engine 304 can receive a medical image library of B contour plots (shown as input 308).

[0075] Without requiring manual contour drawing, and without needing identical or overlapping first and second image datasets, the computer-implemented automatic segmentation engine 304 can then automatically draw contours for each of the first and second image datasets received from participants A and B to obtain third contour drawing data generated by the automatic segmentation engine. In some examples, the first and second image datasets may overlap.

[0076] In some examples, the automatic segmentation engine 304 can be configured to perform automatic contour drawing by including or using an atlas-based model. In some examples, the automatic segmentation engine 304 can be configured to perform automatic contour drawing by including or using a trained model. For example, the trained model can be trained using at least one or more of statistical learning, artificial intelligence, machine learning, neural networks, generative adversarial networks, or deep learning. The model can be trained independently using different and independent training image datasets. In some examples, the different and independent training image datasets can include images from a group of human or animal subjects that overlap with at least one of one or more first image datasets and second image datasets from human or animal subjects.

[0077] The computer-implemented first anatomical comparison engine 310 can compare first contour drawing data (shown as input 312) generated by first participant A with third contour drawing data (shown as input 314) generated by automatic segmentation engine 304 to generate first comparison data 315. The first comparison data 315 can provide meaningful feedback for each anatomical structure, organ, lesion, injury, etc. For example, differences in size / volume, left / right dimensions, anterior / posterior dimensions, and / or upper / lower dimensions (e.g., systematic differences) can be generated between participant A and the automatic contour drawing performed by automatic segmentation engine 304 based on the first comparison data.

[0078] In some examples, comparing the first contour drawing data generated by the first participant A with the third contour drawing data generated by the automatic segmentation engine 304 may include voxel analysis of the human contour drawing region compared to the automatically drawn region. For example, voxel analysis may include analyzing the distance between (1) one or more mismatched voxels between the human contour drawing region and the automatically drawn region, and (2) the closest matching voxel location between the human contour drawing region and the automatically drawn region. In some examples, a quality metric may be generated based on the analyzed distances.

[0079] As another example, voxel analysis may alternatively or additionally include analyzing the orientation between (1) one or more mismatched voxels between the human contour drawing region and the automatic contour drawing region and (2) the closest matching voxel location between the human contour drawing region and the automatic contour drawing region. In some examples, a quality metric may be generated based on the orientation of the analysis.

[0080] In some examples, voxel analysis may include generating a statistical representation of the difference vector between (1) one or more mismatched voxels between the human contour drawing region and the automatically drawn region, and (2) the closest matching voxel location between the human contour drawing region and the automatically drawn region. In some examples, a quality metric may be generated based on the statistical representation of the difference vector.

[0081] The first comparison data 315 can provide meaningful feedback for each anatomical structure or substructure, organ, lesion, injury, etc. For example, differences in size / volume, left / right dimensions, front / back dimensions, and / or top / bottom dimensions can be generated between participant A and the automatic contour drawing performed by the automatic segmentation engine 304, based on the first comparison data 315. In some examples, these differences can be systematic.

[0082] Similarly, the computer-implemented second anatomical comparison engine 316 can compare the second contour drawing data (shown as input 318) generated by the second participant C with the third contour drawing data (shown as input 320) generated by the automatic segmentation engine to generate second comparison data 322. In some examples, comparing the second contour drawing data generated by the second participant C with the third contour drawing data generated by the automatic segmentation engine 304 may include voxel analysis of the human contour drawing region compared to the automatically drawn region.

[0083] In some examples, the first anatomical comparison engine 310 and the second anatomical comparison engine 316 can be the same engine.

[0084] The second comparison data 322 can provide meaningful feedback for each anatomical structure or substructure, organ, lesion, injury, etc. For example, systematic differences in size / volume, left / right dimensions, front / back dimensions, and / or top / bottom dimensions between participant B and the automatic contour drawing performed by the automatic segmentation engine 304 are some examples of feedback that can be generated based on the second comparison data 322.

[0085] The computer-implemented first anatomical comparison engine 310 and second anatomical comparison engine 316 can be computer programs, for example, capable of performing comprehensive 3D analysis on any two sets of anatomical structures from a common image set. As an example, the anatomical comparison engines can perform voxelization of the outlined anatomical structures into sufficiently high-resolution volumetric elements (e.g., 0.001 cc, or 1 mm per side) and perform analysis of all matching (common), missing, and extra voxels between the first and second sets.

[0086] The Anatomy Comparison Engines 310 and 316 can perform difference vector determination on a voxel basis. These difference vectors are 3D "vector-protocol" lines connecting unmatched (missing or extra) voxels to the nearest point on the surface of another volume. In some examples, the difference distance is the total length of the difference vector. Matching voxels can have a difference of 0 mm, missing voxels can have a negative difference distance based on how far they are from the surface of the other set, and extra voxels can have a positive difference distance.

[0087] For each anatomical volume in each dataset, comparisons yield the following results: 1) a comparison of absolute volumes (e.g., size); 2) the distribution of missing, matched, and extra volumes in set 1 versus set 2, plotted as a histogram of volume versus difference distance; 3) histograms of the x-components of the difference vector (typically left-right of the patient) and the mean and standard deviation of the difference in that dimension; 4) histograms of the y-components of the difference vector (typically top-bottom of the patient) and the mean and standard deviation of the difference in that dimension; and / or 5) histograms of the z-components of the difference vector (typically front-back of the patient) and the mean and standard deviation of the difference in that dimension.

[0088] Comparisons can be presented using a user interface, as shown below. Figure 9 and Figure 10 Shown and described.

[0089] In some examples, the comparisons can be repeated across all datasets in the set, and the results and statistics can be plotted against the distribution of the results (e.g., across all datasets) to determine whether the difference between the analysis of variance (ANOVA) technique and the null hypothesis (no significant difference) is significant, and to determine whether the difference is systematic or random.

[0090] It should be noted that the anatomical comparison engines 310 and 316 can work directly for any multiple participants or observers performing contour drawing on a shared image dataset. However, the techniques of this disclosure can compare image contour drawings by different human participants without requiring different participants to perform contour drawing on a shared image dataset.

[0091] Finally, the computer-implemented technique of this disclosure can compare the contour drawings of the first participant and the second participant by performing a transfer analysis comparison, comparing each of the contour drawings of the first participant and the second participant with a reference provided by third contour drawing data generated by an automatic segmentation engine. For example, the computer-implemented transfer engine p can use transfer attributes to compare the contour drawing of the first participant A with the contour drawing of the second participant B. The computer-implemented transfer engine 324 can be a computer program capable of performing comprehensive 3D analysis on the dataset. For example, the computer-implemented transfer engine 324 can compare first comparison data 315 and second comparison data 322 to determine statistically significant trends in systematic differences between the first participant A and the second participant B in terms of size / volume, left / right dimensions, front / back dimensions, and / or top / bottom dimensions.

[0092] As mentioned above, the first participant and / or the second participant can be a group or cluster of participants (e.g., physicians). Therefore, in some examples, comparing the profile drawing of the first participant A and the profile drawing of the second participant C can include comparing the profile drawing of the first participant A and the profile drawing of the individual or aggregated members of the group or cluster of the second participant C. In some examples, comparing the profile drawing of the first participant and the profile drawing of the second participant can include comparing the profile drawing of the individual or aggregated members of the group or cluster of the first participant and the profile drawing of the individual or aggregated members of the group or cluster of the second participant.

[0093] It should be noted that although these techniques are described as comparing the contour drawing of a first participant A with the contour drawing of a second participant C, the techniques of this disclosure are not limited to the first and second participants. Rather, the techniques of this disclosure can be iterative, comparing the second participant C with a third participant D, or further iterated to extend to one or more additional participants. In some examples, the comparison can be performed on a cloud-based anonymized image dataset, and the results of the comparison can be delivered to a cloud-based location. For example, a computer server device (e.g., Figure 8 The computer server device 602 can, for example, automatically anonymize the image dataset after it has been uploaded to the computer server device.

[0094] In some examples, the delivery engine 324 can generate quality metrics based on a comparison between the contour drawing of first participant A and the contour drawing of second participant C. The delivery engine 324 can generate quality metrics regardless of whether the contour drawing was performed by an individual participant, by an individual participant and a group or community of participants, or by two groups or communities of participants.

[0095] Quality metrics may include profiling information, which is associated with at least one of treatment planning information, treatment safety information, treatment efficacy information, or treatment outcome information regarding one or more treatments performed using the profiling information. In some examples, at least one of the treatment safety information, efficacy information, or outcome information may include toxicity information, including at least one of toxicity indication, toxicity prediction, or toxicity risk indication. At least one of the treatment safety information, efficacy information, or outcome information may include toxicity information localized to anatomical structures, substructures, or regions.

[0096] In some examples, the computer-implemented methods described in this disclosure may include selecting healthcare provider participants for profiling in clinical studies of human or animal subjects, at least in part, based on quality metrics. For example, healthcare provider participants (such as physicians or dosimeters) may be associated with at least one of a hospital affiliation, a physician practice affiliation, or a geographic location.

[0097] For example, the profiling of individuals or aggregates of a group or community of a second participant can provide a gold standard benchmark for comparison with the profiling of individuals or aggregates of a group or community of a first participant.

[0098] In some examples, the transfer engine 324 may generate indications of one or more systematic differences between the contour drawing of the first participant and the contour drawing of the second participant. In some examples, for instance, the transfer engine 324 may normalize the indications of systematic differences relative to parameters of the contours generated based on the automatic segmentation engine. These parameters may include volumetric or size dimensions, for example, larger than 20%, and / or one or more directional dimensions, including but not limited to left / right, top / bottom, and / or front / back.

[0099] For example, the transfer engine 324 can generate an indication of systematic differences in the volume of the contour drawing. As another example, the transfer engine 324 can generate an indication of systematic differences in the dimensions of the side contour drawing. In some examples, the transfer engine 324 can generate an indication of systematic differences in the dimensions of the front or rear contour drawing.

[0100] In some examples, the delivery engine 324 may generate indications of systematic differences in the upper or lower contour drawing dimensions. The volume or size of the contour drawing may include at least one of the following: the volume or size of the contour-drawn organ, the volume or size of the contour-drawn organ substructure, the volume or size of the contour-drawn anatomical structure, or the volume or size of the injured or diseased area structure.

[0101] The delivery engine 324 can, for example, perform dataset-by-dataset comparisons between participant A and the entire population of automatic segmentation engine 304 (“B”) and participant C. The delivery engine 324 can receive systematic trends calculated for participant A and automatic segmentation engine 304 (“B”) (e.g., via first comparison data 315) and systematic trends calculated for participant C and automatic segmentation engine (e.g., via second comparison data 320). Using the common automatic segmentation engine 304 (“B”) as an intermediary, the delivery engine 324 can determine statistically significant trends or differences between participant A and participant C.

[0102] As mentioned above, Figure 3 A computer-implemented method is graphically depicted that uses a computer-implemented automatic segmentation engine to automatically draw contours of two image datasets, for example, where each of the image datasets was previously contoured by a corresponding participant (e.g., a physician). The computer-implemented method then compares the contour drawing performed by a first participant (e.g., a first physician (Doctor A)) with the contour drawing performed by a second participant (e.g., a second physician (Doctor C)). For example, the computer-implemented method can compare the contour drawing data generated by the first participant with the contour drawing data generated by the automatic segmentation engine to generate first comparison data, and can also compare the contour drawing data generated by the second participant with the contour drawing data generated by the automatic segmentation engine to generate second comparison data.

[0103] Using the first comparison data, the computer-implemented method can identify the systematic trend between the contour drawing data generated by the first participant and the contour drawing data generated by the automatic segmentation engine, as well as the systematic trend between the contour drawing data generated by the second participant and the contour drawing data generated by the automatic segmentation engine. Then, using the common contour drawing data generated by the automatic segmentation engine, the systematic trend between the contour drawing data generated by the first participant and the contour drawing data generated by the second participant can be determined.

[0104] The engines described in this disclosure, such as the automatic segmentation engine 304, the dissection and comparison engine 310, 316, and the transfer engine 324, can be used... Figure 6The automatic segmentation engine 304, the dissection and comparison engine 310, 316, and the transfer engine 324 are implemented using instructions 424 executed by processor 402. In some examples, the automatic segmentation engine 304, the dissection and comparison engine 310, 316, and the transfer engine 324 may be the same engine. The automatic segmentation engine 304, the dissection and comparison engine 310, 316, and the transfer engine 324 may reside together on a single machine or in different machines.

[0105] Figure 4A and Figure 4B It is a graphical representation of the comparison between two image datasets. Figure 4A An axial view depicting the contours of two image datasets, and Figure 4B A coronal view depicting the outlines of two image datasets.

[0106] In some examples, for the purpose of comparison, an anatomical comparison engine such as Figure 3 The anatomical comparison engines 310 and 316 can render specific contours from both set 1 and set 2 into structured voxels to a common, high-resolution 3D mesh and analyze each voxel in the mesh.

[0107] If a voxel exists in both structures, it is a matched voxel, and the distance difference vector is zero in length across all dimensions. The anatomy comparison engine can then classify this voxel as a matched voxel.

[0108] If a voxel is in set 2 but not in set 1, then the voxel is an additional voxel (e.g., set 2 has the voxel but set 1 does not). The Anatomy Comparison Engine can compute the 3D difference vector (3D) from the voxel to the nearest point on the surface of set 1. The Anatomy Comparison Engine can statistically analyze the distance (e.g., the sign is positive) and split it into directional components of the X, Y, and Z coordinates.

[0109] If a voxel is in set 1 but not in set 2, then the voxel is a missing voxel (e.g., the voxel is not present in set 2 but is present in set 1). The Anatomy Comparison Engine can calculate the 3D difference vector (3D) from the voxel to the nearest point on the surface of set 1. The Anatomy Comparison Engine can statistically analyze the distance (e.g., with a negative sign) and split it into directional components for the X, Y, and Z coordinates.

[0110] Using these techniques, the Anatomy Comparison Engine can accumulate histograms of the difference vectors and the lengths of the 3D components, with signs. Furthermore, the Anatomy Comparison Engine can compute basic metrics (e.g., absolute volume and Dice coefficient) for similarity comparisons.

[0111] Figures 5A to 5D Describing the comparison Figures 4A to 4BExample of a graph depicting two image datasets.

[0112] Figure 5A This plots the error relative to both set 1 and set 2. The y-axis represents the error in cubic centimeters (cc), and the x-axis represents the distance in millimeters (mm). The anatomical comparison engine is as follows: Figure 3 Anatomical comparison engines 310 and 316 determined that the structure of set 2 is significantly larger than that of set 1.

[0113] Figure 5B This plots the distances between vector X and sets 1 and 2. The y-axis represents vector X in cubic centimeters (cc), and the x-axis represents the distance in millimeters (mm). The comparison engine is described as follows: Figure 3 Anatomical comparison engines 310 and 316 determined that set 2, compared to set 1, has a systematic small offset in the -X direction (right).

[0114] Figure 5C This plots the distances between vector Y and sets 1 and 2. The y-axis represents vector Y in cubic centimeters (cc), and the x-axis represents the distance in millimeters (mm). The comparison engine is described as follows: Figure 3 The anatomical comparison engines 310 and 316 determined that set 2, compared to set 1, has a systematic large offset in the +Y direction (up).

[0115] Figure 5D This plots the distances between vector Z and sets 1 and 2. The y-axis represents vector Z in cubic centimeters (cc), and the x-axis represents the distance in millimeters (mm). The comparison engine is described as follows: Figure 3 The anatomical comparison engines 310 and 316 determined that set 2, compared to set 1, has a systematic small offset in the -Z direction (back).

[0116] A non-restrictive list of measurement examples used by the Anatomy Comparison Engine is shown in Table 1 below.

[0117]

[0118] For illustrative purposes, the following describes a non-restrictive hypothesis. Cancer Center XYZ has two radiation oncologists, Dr. A and Dr. C. Clinicians want to know if these two physicians differ in how they outline key anatomical structures in lung cancer cases.

[0119] Dr. A collected data (images and anatomical profiles) of all lung cancer patients she treated last year (N1=21). Dr. C collected data (images and anatomical profiles) of all lung cancer patients she treated last year (N2=45).

[0120] For the trachea, the contour drawn by Dr. A tends to be 32% larger in total volume than that drawn by the automatic segmentation engine, with large systematic offsets in the +Y (upper) and +Z (front) directions. For the trachea, the contour drawn by Dr. C tends to be 5% smaller in total volume than that drawn by the automatic segmentation engine, with small systematic offsets in the +X (left side of the patient) direction.

[0121] As stated above, using a computer-implemented transfer engine, for the trachea, Dr. A tends to draw it 37% larger than Dr. C. Furthermore, for the trachea, compared to Dr. C, Dr. A has a large systematic offset in the +Y (upper) and +Z (front) directions, and a small systematic offset in the -X (patient's right) direction.

[0122] In this way, the various techniques of this disclosure can provide meaningful feedback on each anatomical structure, organ, lesion, injury, etc., which can be generated based on comparisons, such as systematic differences in size / volume, left / right dimensions, anterior / posterior dimensions, and / or upper / lower dimensions.

[0123] The above-mentioned technology can be used as described below and Figure 6 The machine 400 shown is used to implement this.

[0124] Figure 6 A block diagram illustrating an example machine on which any or more of the techniques (e.g., methods) discussed herein can be performed. In alternative embodiments, machine 400 may operate as a standalone device or be connected to other machines (e.g., networked to other machines). In a networked deployment, machine 400 may operate as a server machine, a client machine, or both in a server-client network environment. In the example, machine 400 may act as a peer in a peer-to-peer (P2P) (or other distributed) network environment. Machine 400 is a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), mobile phone, smartphone, web application, network router, switch or bridge, server computer, database, conference room equipment, or any machine capable of executing (sequentially or otherwise) instructions specifying actions to be taken by that machine. In various embodiments, machine 400 may perform one or more of the processes described above. Furthermore, although only a single machine is shown, the term "machine" should also be considered as including any set of machines that individually or collectively execute a set (or more sets) of instructions to perform any or more of the methods discussed herein, such as cloud computing, Software as a Service (SaaS), and other computer cluster configurations.

[0125] Examples as described herein may include logic or components, modules, or mechanisms (all referred to as “modules” below), or may operate on logic or components, modules, or mechanisms. A module is a tangible entity (e.g., hardware) capable of performing a specified operation and is configured or arranged in a certain way. In the examples, circuitry is arranged as a module in a specified manner (e.g., internally or relative to external entities such as other circuitry). In the examples, all or part of one or more computer systems (e.g., stand-alone computer systems, client computer systems, or server computer systems) or one or more hardware processors are configured by firmware or software (e.g., instructions, application portions, or applications) to operate to perform the specified operation. In the examples, the software may reside on a non-transitory computer-readable storage medium or other machine-readable medium. In the examples, the software causes the hardware to perform the specified operation when executed by the underlying hardware of the module.

[0126] Therefore, the term "module" is understood to encompass tangible entities, which are entities that are physically constructed, specifically configured (e.g., hardwired), or temporarily (e.g., provisionally) configured (e.g., programmed) to operate or perform any of the operations described herein in a specified manner. Considering the example where modules are provisionally configured, each of the modules does not need to be instantiated at any given time. For example, in the case where the modules include a general-purpose hardware processor configured using software, the general-purpose hardware processor is configured as various different modules at different times. Thus, the software can configure the hardware processor to constitute a specific module at one time instance and different modules at different time instances.

[0127] Machine (e.g., computer system) 400 may include a hardware processor 402 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), main memory 404, and static memory 406, some or all of which may communicate with each other via interconnect 408 (e.g., a bus). Machine 400 may also include a display unit 410, an alphanumeric input device 412 (e.g., a keyboard), and a user interface (UI) navigation device 414 (e.g., a mouse). In this example, the display unit 410, the input device 412, and the UI navigation device 414 are touchscreen displays. Machine 400 may additionally include a storage device (e.g., a drive unit) 416, a signal generation device 418 (e.g., a speaker), a network interface device 420, and one or more sensors 421, such as a global positioning system (GPS) sensor, a compass, an accelerometer, or other sensors. Machine 400 may include output controller 428, for example, serial (e.g., Universal Serial Bus (USB), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection, to communicate with or control one or more peripheral devices (e.g., printers, card readers, etc.).

[0128] Storage device 416 may include machine-readable medium 422 on which one or more sets of data structures or instructions 424 (e.g., software) are stored for implementing any one or more of the techniques or functions described herein. Instructions 424 may also reside wholly or at least partially in main memory 404, static memory 406, or in hardware processor 402 during execution of instructions by machine 400. In this example, one or any combination of hardware processor 402, main memory 404, static memory 406, or storage device 416 may constitute the machine-readable medium.

[0129] Although machine-readable medium 422 is shown as a single medium, the term "machine-readable medium" can include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) configured to store one or more instructions 424.

[0130] The term "machine-readable medium" can include any medium capable of storing, encoding, or carrying instructions for use by machine 400 to execute any one or more of the techniques of this disclosure, or any medium capable of storing, encoding, or carrying data structures used in or associated with such instructions. Non-limiting examples of machine-readable media can include solid-state memory, as well as optical and magnetic media. Specific examples of machine-readable media can include: non-volatile memory, such as semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; random access memory (RAM); solid-state drives (SSDs); and CD-ROMs and DVD-ROMs. In some examples, machine-readable media can include non-transitory machine-readable media. In some examples, machine-readable media can include machine-readable media that are not transient propagating signals.

[0131] Instruction 424 can also be sent or received via a transmission medium through a communication network 426 via a network interface device 420. Machine 400 can communicate with one or more other machines using any of a variety of transmission protocols, such as Frame Relay, Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), etc. Example communication networks may include local area networks (LANs), wide area networks (WANs), packet data networks (e.g., the Internet), mobile phone networks (e.g., cellular networks), simple old-style telephone (POTS) networks, and wireless data networks (e.g., the IEEE 802.11 family of standards known as Wi-Fi®, the IEEE 802.16 family of standards known as WiMax®), the IEEE 802.15.4 family of standards, the Long Term Evolution (LTE) family of standards, the Universal Mobile Telecommunications System (UMTS) family of standards, peer-to-peer (P2P) networks, etc. In the example, network interface device 420 may include one or more physical jacks (e.g., Ethernet, coaxial, or telephone jacks) or one or more antennas for connecting to communication network 426. In the example, network interface device 420 may include multiple antennas for wireless communication using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) technology. In some examples, network interface device 420 may use multi-user MIMO technology for wireless communication.

[0132] Examples as described herein may include logic or components, modules, or mechanisms, or may operate on logic or components, modules, or mechanisms. A module is a tangible entity (e.g., hardware) capable of performing a specified operation and configured or arranged in a certain way. In the examples, circuitry may be arranged as a module in a specified manner (e.g., internally or relative to external entities such as other circuitry). In the examples, all or part of one or more computer systems (e.g., standalone computer systems, client computer systems, or server computer systems) or one or more hardware processors may be configured by firmware or software (e.g., instructions, application portions, or applications) to operate to perform a specified operation. In the examples, the software may reside on a machine-readable medium. In the examples, the software causes the hardware to perform the specified operation when executed by the underlying hardware of the module.

[0133] Therefore, the term "module" is understood to encompass tangible entities, which are entities that are physically constructed, specifically configured (e.g., hardwired), or temporarily (e.g., provisionally) configured (e.g., programmed) to operate or perform any of the operations described herein in a specified manner. Considering the example where modules are provisionally configured, each of the modules does not need to be instantiated at any given time. For example, in the case where the modules include a general-purpose hardware processor configured using software, the general-purpose hardware processor is configured as various different modules at different times. Thus, the software can configure the hardware processor to constitute a specific module at one time instance and different modules at different time instances.

[0134] Each implementation is implemented wholly or partially in software and / or firmware. This software and / or firmware may take the form of instructions contained in or on a non-transitory computer-readable storage medium. These instructions can then be read and executed by one or more processors to enable the operations described herein to be performed. The instructions may be in any suitable form, such as, but not limited to, source code, compiled code, interpreted code, executable code, static code, dynamic code, etc. Such computer-readable media may include any tangible non-transitory medium for storing information in a form readable by one or more computers, such as, but not limited to, read-only memory (ROM), random access memory (RAM), disk storage media, optical storage media, flash memory, etc.

[0135] Figure 7This is a flowchart illustrating an example of a computer-implemented method 500 for comparing image contour plots from different human participants without requiring different participants to perform contour plotting on a shared image dataset. At box 502, the method may include receiving a first set of image datasets, including one or more first image datasets from one or more human or animal subjects generated by an imaging modality, and including first contour plotting data generated by a first participant. For example, a computer-implemented automatic segmentation engine, such as... Figure 3 The computer-implemented automatic segmentation engine 304 can receive a first image dataset set 300, including one or more first image datasets of one or more human or animal subjects generated by the imaging modality, and including first contour drawing data generated by the first participant A.

[0136] At box 504, the method may include receiving a second image dataset set, including one or more second image datasets of one or more human or animal subjects generated by the imaging modality, including second contour drawing data generated by the second participant. For example, a computer-implemented automatic segmentation engine, such as Figure 3 The computer-implemented automatic segmentation engine 304 can receive a second image dataset set 302, including one or more second image datasets of one or more human or animal subjects generated by the imaging modality, including second contour drawing data generated by the second participant B.

[0137] At box 506, the method may include automatically drawing contours on each of the first and second image datasets using a computer-implemented automatic segmentation engine to obtain third contour drawing data generated by the automatic segmentation engine, without requiring manual contour drawing or identical or overlapping first and second image datasets. For example, in the absence of manual contour drawing and identical or overlapping first and second image datasets, the computer-implemented automatic segmentation engine 304 may automatically draw contours on each of the first and second image datasets received from participant A and participant B to obtain third contour drawing data generated by the automatic segmentation engine.

[0138] At box 508, the method may include comparing the contour drawing of a first participant with the contour drawing of a second participant, including comparing first contour drawing data generated by the first participant with third contour drawing data generated by an automatic segmentation engine to generate first comparison data, and further comparing second contour drawing data generated by the second participant with third contour drawing data generated by the automatic segmentation engine to generate second comparison data. In other words, the method may be based on the contour drawing of the first participant and the contour drawing of the second participant with the third contour drawing data generated by the automatic segmentation engine, such as... Figure 3The automatic segmentation engine 304 performs contour drawing analysis to obtain a comparison of the contour drawings of the first participant and the second participant. As mentioned above, the technology of this disclosure does not directly compare the first participant (A) and the second participant (C) on the same dataset, but is based on the first participant (A) and the computer-implemented automatic segmentation engine "B" as... Figure 3 The computer-implemented automatic segmentation engine 304 analyzes the second participant (C) and B, comparing the first participant (A) and the second participant (C) for different datasets.

[0139] For example, the first anatomical comparison engine implemented by a computer, such as Figure 3 The anatomical comparison engine 310 can compare the first contour drawing data generated by the first participant A with the third contour drawing data generated by the automatic segmentation engine 304 to generate first comparison data 315. Similarly, a computer-implemented second anatomical comparison engine, such as... Figure 3 The anatomical comparison engine 316 can compare the second contour drawing data generated by the second participant C with the third contour drawing data generated by the automatic segmentation engine to generate second comparison data 322.

[0140] Then, the computer-implemented delivery engine, for example Figure 3 The computer-implemented transfer engine 324 can compare the contour drawing of the first participant A with the contour drawing of the second participant B using transfer attributes. For example, the computer-implemented transfer engine 324 can compare first comparison data 315 and second comparison data 322 to determine statistically significant trends in systematic differences between the first participant A and the second participant B in terms of size / volume, left / right dimensions, front / back dimensions, and / or top / bottom dimensions.

[0141] In some implementations, the operations described above for implementing automatic partitioning techniques, such as storage, access, and computation, can be implemented using a single computing device (e.g., a server, FPGA, ASIC, SOC, virtual server, etc.). In other implementations, at least some of the operations described above for implementing automatic partitioning techniques, such as storage, access, and computation, can be distributed across multiple computing devices (e.g., server clusters, cloud computing platforms, virtual server clusters, etc.), for example, implemented in the "cloud" using computer server devices (or "servers"). Figure 8 It describes various components that can be used to implement technologies using computer server equipment.

[0142] For example, a computer server device, such as server device 602, located in a cloud computing platform, can perform a comparison of the contour drawing of the first participant and the contour drawing of the second participant. The computer server device can compare the first contour drawing data generated by the first participant with the third contour drawing data generated by the automatic segmentation engine to generate first comparison data, and can also compare the second contour drawing data generated by the second participant with the third contour drawing data generated by the automatic segmentation engine to generate second comparison data.

[0143] As another example, a computer server device can generate an indication of one or more differences between the contour drawings of a first participant and those of a second participant, and send that indication to another computing device, such as... Figure 8 The client device 606 or another computer server device.

[0144] In some examples, the first set of image datasets may be received by a computer server device, while the second set of image datasets may be received by a computer server device.

[0145] As described above, the technology of this disclosure can determine a first quality metric based on one or more differences between the contour drawing of a first participant and the contour drawing of a second participant, and provide the first quality metric to at least one of a first hospital affiliate, a first physician practice affiliate, or a first geographical location. It can also determine a second quality metric based on one or more differences between the contour drawing of the first participant and the contour drawing of the second participant, and provide the second quality metric to at least one of a second hospital affiliate, a second physician practice affiliate, or a second geographical location. In some examples, a computer server device can determine the first and second quality metrics.

[0146] In some examples, the computer server device may determine a quality metric based on one or more differences between the profile drawing of a first participant and the profile drawing of a second participant, and provide the quality metric to at least one of a first hospital affiliate, a first physician practice affiliate, or a first geographic location.

[0147] In some examples, computer server devices, such as Figure 8 The computer server device 602 can, for example, automatically anonymize the image dataset after it has been uploaded to the computer server device.

[0148] In some examples, a computer server device can send data to another computing device, such as a client device. Figure 8 The client device 606 sends information and receives information from it.

[0149] Figure 8 These are illustrations of the components of client and server devices based on various examples. Figure 8 It includes server device 602, web server 604, client device 606, web client 608, processing system 610, access permission component 612, value update component 614, search interface component 616, application logic 618, application programming interface (API) 620, user management component 622, data owner interface 624, key sending component 626, data storage 630, and data 632.

[0150] Client device 606 may be a computing device, which may be, but is not limited to, a smartphone, tablet computer, laptop computer, multiprocessor system, microprocessor-based or programmable consumer electronics, game console, set-top box, or other device used by a user to communicate over a network. In various examples, the computing device includes a display module (not shown) to display information (e.g., in the form of a specially configured user interface). In some implementations, the computing device may include one or more of a touchscreen, camera, keyboard, microphone, or Global Positioning System (GPS) device. Client device 606 may be associated with one or more entities that interact with server device 602. In various examples, an entity may be an individual, a group of individuals, or a company.

[0151] Client device 606 and server device 602 can communicate via a network (not shown). This network may include a local area network (LAN), a wide area network (WAN), a wireless network (such as 802.11 or a cellular network), a public switched telephone network (PSTN), an ad hoc network, a cellular network, a personal area network (PAN), or a point-to-point network (such as Bluetooth®, Wi-Fi Direct), or other combinations or arrangements of network protocols and network types. The network may include a single LAN or WAN, or a combination of LAN or WAN such as the Internet.

[0152] Client device 606 and server device 602 can transmit data 632 over a network. Data 632 may be, but is not limited to, search requests, search results, data item requests (discussed in more detail below), data items, decryption keys, and data inheritance details.

[0153] In some examples, communication can occur using an Application Programming Interface (API) such as API 620. An API provides a method for exchanging data during computation. A web-based API (e.g., API 620) can allow communication between two or more computing devices (e.g., client and server). An API can define a set of HTTP calls based on Representational State Transfer (RESTful) practices. For example, a RESTful API can define various GET, PUT, POST, and DELETE methods to create, replace, update, and delete data stored in a database (e.g., data store 630).

[0154] APIs can also be defined within the framework provided by the operating system (OS) to access data that applications within an application are not regularly permitted to access. For example, the OS can define API calls to obtain the current location of a mobile device on which the OS is installed. In another example, an application provider can use API calls to request a user to authenticate using biometric sensors on their mobile device. This allows the user to access information stored in data items. The risk of unauthorized biometric data transfer can be mitigated by isolating any basic biometric data—for example, through the use of a secure element.

[0155] Server device 602 is shown as a set of individual elements (e.g., components, logic, etc.). However, the functionality of a single element can be performed by that single element. An element may represent computer program code executable by processing system 610. The program code may be stored on a storage device (e.g., data storage 630) and loaded into the memory of processing system 610 for execution. Portions of the program code may be executed in parallel on multiple processing units of processing system 610 (e.g., the core of a general-purpose computer processor, a graphics processing unit, an application-specific integrated circuit, etc.). Execution of the code may be performed on a single device or distributed across multiple devices. In some examples, the program code may be executed on a cloud platform (e.g., MICROSOFT AZURE® and AMAZON EC2®) using a shared computing infrastructure.

[0156] Server device 602 may include web server 604 to enable data exchange with client device 606 via web client 608. Although generally discussed in the context of providing web pages via Hypertext Transfer Protocol (HTTP), other network protocols can be utilized by web server 604 (e.g., File Transfer Protocol, Telnet, Secure Shell, etc.). Users can enter a Uniform Resource Identifier (URI) into web client 608 (e.g., Microsoft's Internet Explorer® or Apple's Safari®), which corresponds to the logical location of web server 604 (e.g., an Internet Protocol address). In response, web server 604 can send a web page to be displayed on the client device (e.g., a mobile phone, desktop computer, etc.).

[0157] Furthermore, web server 604 enables users to interact with one or more web applications provided in the sent web page. The web application can provide user interface (UI) components that are presented on the display device of client device 606. Users can interact with the UI components (e.g., select, move, enter text), and based on this interaction, the web application can update one or more parts of the web page. The web application can execute entirely or partially locally on client device 606. The web application can populate the UI components with data from external or internal sources (e.g., data storage 630) in various examples.

[0158] For example, server device 602 can provide users with a web application (e.g., search interface component 616) to search for metadata of a data store (e.g., data store 630) for data items. For example, an input box can be presented to receive text input by the user. This text can be formatted as a database query (e.g., SQL) and published to the database to retrieve data items that have data in fields matching the text input by the user. The web application (e.g., data owner interface 624 and access permission component 612) can also provide a user interface for the owner user to edit access permissions for their data items, etc. For example, user interface elements can be presented to add or revoke access permissions to the payload of a data item.

[0159] This interface can also allow users to group data items based on characteristics in metadata, such as type, data source, etc. For example, a user can submit a search query to retrieve all data items corresponding to a user's movie preferences. The interface can include options to select one or more results and create a group of the selected results. Data storage 630 can store indications that the selected data items are part of the same group in various examples. Web applications can also allow access to edit the group, rather than requiring users to edit each data item individually.

[0160] The web application can be executed according to application logic 618. Application logic 618 can implement the web application using various elements of server device 602. For example, application logic 618 can issue API calls to retrieve or store data from data storage 630 and send it for display on client device 606. Similarly, data entered by the user into a UI component can be sent back to web server 604 using API 620. Application logic 618 can use other elements of server device 602 (e.g., access permission component 612, value update component 614, search interface component 616, application logic 618, etc.) to perform functions associated with the web application as further described herein.

[0161] Data storage 630 can store data used by server device 602 (e.g., user profile 628, data items, decryption keys, etc.). Data storage 630 is described as a single element, but can actually be multiple data storages. The specific storage layout and model used by data storage 630 can take many forms—in fact, data storage 630 can utilize a variety of models. Data storage 630 can be, but is not limited to, relational databases (e.g., SQL), non-relational databases (NoSQL), flat file databases, object models, document detail models, graph databases, shared ledgers (e.g., blockchain), or file system hierarchies. Data storage 630 can store data on one or more storage devices (e.g., hard drives, random access memory (RAM), etc.). Storage devices can be independent arrays, part of one or more servers, and can be located in one or more geographical regions.

[0162] User profile 628 can store profiles of users interacting with server device 602. The user profile may include a user identifier (e.g., username) and a password. The user profile may also include user roles. A user can have multiple roles. Roles may include owner roles, purchaser roles (e.g., those who wish to purchase access to data items), data aggregator roles (e.g., data item providers), etc. When a user logs into server device 602 (e.g., via web server 604), functionality related to the user's role can be provided. For example, under the purchaser role, server device 602 can present a search interface to find data items for purchase. Under the owner role, server device 602 can present a user interface to edit the user's access permissions for data items.

[0163] Value update component 614 can generate or update the values ​​of data items. Data items can have initial values ​​determined in a variety of ways. For example, an owner user can enter values ​​for their data items (e.g., quantitative values ​​such as $0.50 or qualitative values ​​such as high, medium). If the owner does not enter values, server device 602 can assign values ​​to data items based on their metadata. For example, weighted calculations can be performed using data types (e.g., location data has a default value, user preferences have default values) when generating data items (recent data is assigned a higher value, etc.). In some examples, the calculated values ​​can be presented to the owner for confirmation before being stored as part of the data item.

[0164] Periodically, the value update component 614 can iterate over one or more data items stored in the data storage 630 and update the values ​​of the data items. For example, formulas—such as those similar to or the same as the weighted calculation described above—can be stored to calculate the value of a data item based on its metadata. In other examples, adjustment calculations can be used to decrease or increase the value based on changes in one or more metadata fields or changes in the subject user of the data item.

[0165] For example, consider a data item indicating a subject user's (e.g., a user involved in the data payload) preference for a specific coffee shop in a specific postcode. Now, consider that the subject user no longer lives near that postcode. The change in location information can be based on location information received from the subject user's computing device by server device 602—assuming the subject user granted such location access. Therefore, if the subject user's current location is within a radius of X miles from the postcode and / or has not been within that radius in the past month, the data item may become less valuable. Accordingly, value update component 614 can reduce the value of the data item by a certain amount or percentage.

[0166] In another example, server device 602 may receive information that the subject user has become incapacitated or has died (e.g., via API 620). In this case, value update component 614 may set the values ​​of all data items associated with the subject user to 0 (or some other nominally low value).

[0167] Access permission component 612 can be configured to update data items and / or data storage 630 to indicate which users can access the data item. For example, a data item may include a tag containing a list of identifiers corresponding to users who can access the data item. Access permission component 612 can update the tag to include or remove user identifiers. Identifiers can be added based on a user's purchase of access to the data item. If a user is granted access to the data item, key sending component 626 can send a decryption key for the data payload to the purchasing user. In various examples, access to the data item is limited. For example, a data item can only be accessed for a certain period of time. Therefore, once that period has passed, the tag can be updated to remove access.

[0168] In some examples, it may be desirable to present a user interface to users, such as physicians, clinicians, or other healthcare providers, that displays data representing any missing, common, and / or extra volumes in a comparison between a first contour drawing generated by a first participant and a second contour drawing generated by a second participant. Figure 9 and Figure 10 An example of the user interface is shown in the figure.

[0169] Figure 9 This is an example of a user interface associated with a computer-implemented method for comparing image contour drawing by different human participants without requiring different participants to perform contour drawing on a shared image dataset.

[0170] As mentioned above, the anatomical comparison engine can voxelize the anatomical structures in the contour drawing into sufficiently high-resolution volumetric elements (e.g., 0.001 cc, or 1 mm per side) and analyze all matching (common), missing, and extra voxels between the first and second sets. Matching (or common) voxels are those shared by the first contour drawing generated by the first participant (e.g., healthcare provider participant A) and the second contour drawing generated by the second participant (e.g., healthcare provider participant B). Missing voxels are those that are not present in the first contour drawing but are present in the second contour drawing (missing from participant A's perspective). Extra voxels are those that are present in the first contour drawing but not in the second contour drawing (extra from participant A's perspective).

[0171] Figure 3The anatomical comparison engines 310 and 316 can perform the determination of difference vectors on a voxel basis. These difference vectors are 3D "vector-protocol" lines that connect unmatched (missing or extra) voxels to the nearest point on another volume surface.

[0172] In some examples, the difference distance is the total length of the difference vector. Matched voxels can have a difference of 0 mm, missing voxels can have a negative difference distance based on how far they are from the surface of the other set, and extra voxels can have a positive difference distance.

[0173] For each anatomical volume in each dataset, comparisons yield the following results: 1) a comparison of absolute volumes (e.g., size); 2) the distribution of missing, matched, and extra volumes in set 1 versus set 2, plotted as a histogram of volume versus difference distance; 3) histograms of the x-components of the difference vector (typically left-right of the patient) and the mean and standard deviation of the difference in that dimension; 4) histograms of the y-components of the difference vector (typically top-bottom of the patient) and the mean and standard deviation of the difference in that dimension; and / or 5) histograms of the z-components of the difference vector (typically front-back of the patient) and the mean and standard deviation of the difference in that dimension.

[0174] Figure 9 The user interface 700 shown can be presented, for example, transmitted for display or displayed by a video display, for example by... Figure 6 The video display 410 shows an example of a user interface for physicians, clinicians, healthcare providers, or other users. The user interface 700 may include a first portion 702 configured to display data of a first representation 704 of a volume that is not present in a first contour drawing of a target or risk organ of a first group of subjects generated by a first participant, but is present in a second contour drawing of a target or risk organ of a second group of subjects generated by a second participant. In some examples, the first group of subjects may include at least some subjects not included in the second group of subjects. In other examples, subjects in the first group of subjects may be completely different from subjects in the second group of subjects. In some examples, subjects in the first group of subjects may be the same subjects as subjects in the second group of subjects.

[0175] In some examples, the first group of subjects may include 10 or more subjects, such as 20 or more subjects, or 100 subjects. Similarly, the second group of subjects may include 10 or more subjects, such as 20 or more subjects, or 100 subjects. In some examples, the target or risk organ of the first subject and the target or risk organ of the second subject may be similar target or risk organs, such as the liver, lungs, prostate, etc.

[0176] The first representation of volume, 704, can represent "missing" voxels, which represent volumes (e.g., cubic centimeters), by... Figure 3 Anatomical comparison of engines 310, 316 and Figure 3 The delivery engine 324 determines, through aggregation on multiple subjects, such as 10, 20, 100, or more subjects, to provide a statistically significant trend regarding how participant A contours the subject's target or risk organ at the sample size, compared to how participant B contours the subject's target or risk organ at the sample size. The distribution of error is captured by the x-axis, which shows the distance error of the "missing" volume. The user interface 700 may include a fourth part 714 configured to display data representing the error distribution between the first contouring and the second contouring.

[0177] User interface 700 may include a second portion 706 configured to display data for a second representation of volume 708, representing a volume (e.g., cubic centimeters) shared by both the first and second contour drawings. The data for the second representation of volume 708 may represent "additional" voxels, which are... Figure 3 Anatomical comparison of engines 310, 316 and Figure 3 The delivery engine 324 determines that aggregation on multiple subjects, such as 10, 20, or 100 subjects, is used to provide a statistically significant trend regarding how participant A profiles on the sample size, compared to how participant B profiles on the sample size.

[0178] User interface 700 may include a third portion 710 configured to display data of a third representation 712 of volume, representing a volume (e.g., cubic centimeters) present in the first contour drawing but not in the second contour drawing. The data of the third representation 712 of volume may represent "additional" voxels, which are... Figure 3 Anatomical comparison of engines 310, 316 and Figure 3 The delivery engine 324 determines that aggregation across multiple subjects, such as 10, 20, or 100 subjects, provides a statistically significant trend regarding how participant A contours at the sample size compared to how participant B contours at that sample size. The distribution of error is captured by the x-axis, which shows the distance error for the “extra” volume.

[0179] exist Figure 9In the user interface 700, portions 702, 706, and 710 do not display anatomical images. Instead, the user interface 700 graphically presents the trend of differences (e.g., systematic differences) in profiling performed by two different participants at a statistically significant subject size, such as 10 or more subjects. That is, the user interface 700 can present the manifestation of systematic differences between profiling performed by two participants. The systematic differences shown can indicate degree, direction, and identifiers, such as in what direction "extra," "missing," or "common" by how much. The processor, such as... Figure 6 The processor 402 can determine the direction of the difference and its location within a volume region or subregion, sum the differences, and then average them. For example, some of the "extra" volumes are located in bins between 19.5 mm and 20.5 mm.

[0180] In some examples, the first designation 704, the second designation 706, and the third designation 712 can be visually distinguished from each other, for example, by different colors or grayscale shapes.

[0181] In some examples, to gain a deeper understanding of the orientation of the entire volume or a sub-region, the user can select a sub-region of the volume and only display the results for volume elements within that region. Examples include, but are not limited to, the following:

[0182] Volume half-region: left half-region or right half-region, upper half-region or lower half-region, front half-region or rear half-region;

[0183] Volume quadrants: left anterior quadrant, right anterior quadrant, left posterior quadrant, right posterior quadrant; and

[0184] Volume 1 / 8: Left-front-top, Left-front-bottom, Left-back-top, Left-back-bottom, Right-front-top, Right-front-bottom, Right-back-top, Right-back-bottom. For any selected volume sub-region, it can be displayed... Figure 9 The graphic shown.

[0185] In some examples, Figure 9 The graphic shown can be a two-dimensional shape. In other examples, Figure 9 The graphic shown can be a three-dimensional shape.

[0186] Figure 10 This is another example of a user interface associated with a method for comparing image contour plots by different human participants without requiring different participants to perform contour plotting on a shared image dataset. For any selected volumetric sub-region, it can display... Figure 10 The diagram shown. In some examples, Figure 10 The graphic shown can be a three-dimensional shape.

[0187] Figure 10 The user interface 800 shown can be presented, for example, transmitted for display or displayed by a video display, for example by... Figure 6 The video display 410 shows an example of a user interface for physicians, clinicians, healthcare providers, or other users. The user interface 800 can display trends, for example, for selected sub-areas. Figure 10 The diagram shows multiple sub-regions 802 to 814. For example, region 802 could represent the right-back-bottom region (out of 8 regions), and region 804 could represent the left-front-top region (out of 8 regions). Regions 802 to 814 can be visually distinguished from each other, for example, by using different colors or grayscale shapes.

[0188] User interface 800 may include a first portion, such as portion 816, which is configured to display data representing volume, which is present in a first contour drawing of a target or risk organ of a first group of subjects, but not in a second contour drawing of a target or risk organ of a second group of subjects (“additional” voxels).

[0189] User interface 800 may include a second part, such as part 818, which is configured to display data (“missing” voxels) of another representation of a volume that is not present in the first contour drawing but is present in the second contour drawing.

[0190] As an example, the first color of region 802, such as blue, can indicate that there is a large portion of "missing" volume in the selected sub-region when comparing the outline drawing of the first participant with that of the second participant. As another example, the second color of region 804, such as red, can indicate that there is a large portion of "extra" volume in the selected sub-region when comparing the outline drawing of the first participant with that of the second participant.

[0191] In this way, the user interface 800 can display a trend summary for a specific sub-region. If the volume of the outline drawn by participant A in a specific sub-region is much larger than that of participant B, the user interface 800 can display a first color for that sub-region; and if the volume of the outline drawn by participant A in a specific sub-region is much smaller than that of participant B, the user interface 800 can display a second color for that sub-region.

[0192] processors such as Figure 6 The processor 402 can perform statistical calculations on a certain area, further reduce the data, and use various colors, for example, to visually distinguish molecular regions and display trends. For instance, if the processor determines that the average difference between the contour drawing of participant A and the contour drawing of participant B is 3 mm "extra", it can display an associated color, such as red, for that specific sub-region.

[0193] For example, the lightness or intensity of a color can provide an indication of the relative error in terms of extra or missing volumes. For instance, light blue as in region 812 or light red as in region 814 can respectively indicate that the relative error in these regions is less than in regions 802 and 804. As an example, dark red can represent significantly more “extra” voxels than “missing” voxels in a particular sub-region, and / or significantly more “extra” voxels at larger distances. Figure 9 The amount of “extra” volume in a 2D image and Figure 9 If the amount of "missing" volume in the 2D image is equal, then Figure 10 You can display another color, such as white, to indicate that there is no trend in the data.

[0194] By selection Figure 10 One of the sub-regions 802 to 814 shown may indicate additional information, such as Figure 9 As depicted in the text. For example, Figure 9 The user interface 700 shown can represent Figure 10 Subregion 804.

[0195] Various annotations

[0196] Each of the non-limiting aspects or examples described herein may exist independently, or may be combined with one or more of the other examples in various permutations or combinations.

[0197] The above detailed description includes reference to the accompanying drawings, which form part of the detailed description. The drawings illustrate the detailed description by way of illustration, in which the invention can be practiced. These descriptions are also referred to herein as “examples.” Such examples may include elements other than those shown or described. However, the inventors also contemplate examples that provide only those elements shown or described. Furthermore, the inventors contemplate examples using any combination or arrangement of those elements (or one or more aspects of those elements) shown or described with respect to a particular example (or one or more aspects of that particular example) or with respect to other examples shown or described herein (or one or more aspects of those other examples).

[0198] In the event of any inconsistency between the usage in this document and any other document incorporated by reference, the usage in this document shall prevail.

[0199] In this document, as is common in patent literature, the terms “a” and “an” are used to include one or more, unrelated to any other instance or use of “at least one” or “one or more.” In this document, unless otherwise stated, the term “or” is used to refer to non-exclusivity, or to make “A or B” include “A but not B,” “B but not A,” and “A and B.” In this document, the terms “including” and “in which” are used as common English equivalents to the corresponding terms “comprising” and “wherein.” Furthermore, in the appended claims, the terms “including” and “comprising” are open-ended, meaning that a system, apparatus, article, combination, formulation, or process that includes elements other than those listed after such terms in the claims is still considered to fall within the scope of the claims. Additionally, in the appended claims, the terms “first,” “second,” and “third,” etc., are used merely as designations and are not intended to impose numerical requirements on their objects.

[0200] The methods described herein may be implemented, at least in part, by a machine or computer. Some examples may include computer-readable or machine-readable media encoded with instructions operable to configure electronic devices to perform the methods described in the examples above. Implementations of these methods may include code such as microcode, assembly language code, higher-level language code, etc. Such code may include computer-readable instructions for performing various methods. The code may form part of a computer program product. Furthermore, in the examples, the code may be tangibly stored on one or more volatile, non-transitory, or non-volatile tangible computer-readable media, for example, during execution or at other times. Examples of such tangible computer-readable media may include, but are not limited to, hard disks, removable disks, removable optical discs (e.g., compact discs and digital video disks), magnetic tape cartridges, memory cards or memory sticks, random access memory (RAM), read-only memory (ROM), etc.

[0201] The above description is intended to be illustrative and not restrictive. For example, the examples above (or one or more aspects thereof) may be used in combination with each other. Other embodiments may be used, for example, by those skilled in the art upon review of the above description. An abstract is provided to conform to 37 CFR §1.72(b) to allow the reader to quickly determine the nature of the technical disclosure. An abstract is provided and it is understood that the abstract will not be used to interpret or limit the scope or meaning of the claims. Furthermore, in the specific embodiments described above, various features may be combined to organize the present disclosure. This should not be construed as meaning that all unclaimed disclosed features are necessary for any claim. Rather, the subject matter of the invention may lie in fewer than all features of a particular disclosed embodiment. Thus, the appended claims are incorporated herein as examples or embodiments, each of which exists as a separate embodiment, and it is conceivable that such embodiments may be combined with each other in various combinations or arrangements. The scope of the invention should be determined with reference to the appended claims and the full scope of the equivalents conferred by such claims.

Claims

1. A computer-implemented method for comparing image contour drawing by different human participants without requiring the different participants to perform contour drawing on a shared image dataset, the method comprising: Receive a first set of image datasets, including one or more first image datasets of one or more human or animal subjects generated by an imaging modality, and including first contour drawing data generated by a first participant; Receive a second image dataset set, including one or more second image datasets of one or more human or animal subjects generated by the imaging modality, including second contour drawing data generated by a second participant; A computer-implemented automatic segmentation engine is used to automatically draw contours for each of the first image dataset and the second image dataset to obtain third contour drawing data generated by the automatic segmentation engine, without the need for manual contour drawing and without the need for identical or overlapping first and second image datasets. Comparing the contour drawing of the first participant and the contour drawing of the second participant includes comparing the first contour drawing data generated by the first participant and the third contour drawing data generated by the automatic segmentation engine to generate first comparison data, and further comparing the second contour drawing data generated by the second participant and the third contour drawing data generated by the automatic segmentation engine to generate second comparison data. as well as Based on the first comparison data and the second comparison data, an indication of one or more differences between the contour drawing of the first participant and the contour drawing of the second participant is generated.

2. The method according to claim 1, wherein, Indications for generating one or more differences between the contour drawing of the first participant and the contour drawing of the second participant include indications for generating one or more systematic differences between the contour drawing of the first participant and the contour drawing of the second participant.

3. The method according to claim 2, wherein, Indications for generating one or more systematically poor indicators include generating at least one of the following: An indication of systematic poorness in the volume of contour drawing; Indicators of poor systematicity in the dimensional representation of the side profile; An indication of systematic poorness in the dimensions of the front or rear contour drawing; or Indicator of systematic poorness in the dimensions of the upper or lower contour drawing.

4. The method according to claim 3, wherein, The contour drawing volume or size includes at least one of the following: The volume or size of the organ drawn in outline; The volume or size of organ substructures drawn from outlines; The volume or dimensions of the anatomical structure drawn in outline; or The volume or size of the injured or diseased area.

5. The method according to claim 2, wherein, The indication of systematic poorness is normalized relative to the parameters of the contours generated based on the automatic segmentation engine.

6. The method according to claim 1, wherein, The comparison between the contour drawing of the first participant and the contour drawing of the second participant includes: A transitive analysis comparison of the contour drawings of the first participant and the second participant is performed by comparing each of their contour drawings with a reference provided by the third contour drawing data generated by the automatic segmentation engine.

7. The method according to claim 1, wherein, At least one of the first image dataset set and the second image dataset set includes, in addition to contour drawing data generated by the participants, metadata, wherein the metadata is provided as input to the automatic segmentation engine.

8. The method according to claim 7, wherein, The metadata includes at least one of the following: An indication of at least one imaging modality parameter used to generate images in the corresponding image dataset; Indications of the characteristics of the target structure to be targeted for treatment; Indicators of the characteristics of risk structures that should be avoided from treatment; Indications of patient demographic characteristics of human or animal subjects corresponding to images in the corresponding image dataset; Indications of disease characteristics associated with human or animal subjects corresponding to images in the corresponding image dataset; Indications of treatment characteristics associated with human or animal subjects corresponding to images in the corresponding image dataset; as well as Indicators of expected treatment outcomes associated with human or animal subjects corresponding to images in the relevant image dataset.

9. The method according to claim 8, wherein, The imaging modality parameters include the imaging modality type; The target structure includes a target organ; and / or The risk structure includes risk organs.

10. The method according to claim 1, wherein, The automatic segmentation engine is configured to perform the automatic contour drawing by including or using atlas-based models.

11. The method according to claim 1, wherein, The automatic segmentation engine is configured to perform the automatic contour drawing by including or using a trained model, the trained model being trained using at least one or more of statistical learning, artificial intelligence, machine learning, neural networks, generative adversarial networks, or deep learning.

12. The method according to claim 11, wherein, The model was trained independently using different and independent training image datasets.

13. The method according to claim 12, wherein: The distinct and independent learning image datasets include images from human or animal subject groups that overlap with at least one of the first and second image datasets.

14. The method according to claim 1, wherein, At least one of the following comparisons is included: comparing the first contour drawing data generated by the first participant with the third contour drawing data generated by the automatic segmentation engine, or comparing the second contour drawing data generated by the second participant with the third contour drawing data generated by the automatic segmentation engine: Voxel analysis was performed on the manually drawn contour region compared with the automatically drawn contour region.

15. The method of claim 14, further comprising: Analysis (1) the distance between one or more mismatched voxels between the artificial contour drawing area and the automatic contour drawing area and (2) the closest matching voxel position between the artificial contour drawing area and the automatic contour drawing area.

16. The method of claim 15, further comprising: Analysis (1) the direction between one or more mismatched voxels between the artificial contour drawing area and the automatic contour drawing area and (2) the closest matching voxel position between the artificial contour drawing area and the automatic contour drawing area.

17. The method of claim 14, further comprising: Generate a statistical representation of the difference vector between (1) one or more mismatched voxels between the artificial contour drawing area and the automatic contour drawing area and (2) the closest matching voxel position between the artificial contour drawing area and the automatic contour drawing area.

18. The method according to claim 1, wherein, Comparing the contour drawing of the first participant with the contour drawing of the second participant includes comparing the contour drawing of the first participant with the contour drawing of an individual or aggregate of the group or community of the second participant.

19. The method according to claim 1, wherein, The comparison of the contour drawing of the first participant and the contour drawing of the second participant includes comparing the contour drawing of the individual or aggregate of the group or community of the first participant with the contour drawing of the individual or aggregate of the group or community of the second participant.

20. The method according to any one of claims 1, 14 to 16, 18 or 19, further comprising generating a quality metric based on comparison.

21. The method according to any one of claim 18 or 19, wherein, The outline drawing of the individuals or aggregates of the second participant's group or community provides a gold standard benchmark for comparison with the outline drawing of the individuals or aggregates of the first participant's group or community.

22. The method of claim 20, comprising selecting human participants for profiling in clinical studies of human or animal subjects based at least in part on the quality metric.

23. The method according to any one of claims 1, 18, or 19, wherein, The first participant and / or the second participant includes a healthcare provider associated with at least one of the hospital affiliates, physician practice group affiliates, or geographic locations.

24. The method according to any one of claims 1, 18, or 19, wherein, The comparison between the contour drawing of the first participant and the contour drawing of the second participant is iterated to compare the second participant and the third participant, or further iterated to extend to one or more additional participants.

25. The method according to claim 20, wherein, The quality metric includes contour plotting information, which is associated with at least one of treatment plan information, treatment safety information, treatment efficacy information, or treatment outcome information regarding one or more treatments performed using the contour plotting information.

26. The method of claim 25, wherein, At least one of the treatment safety information, treatment efficacy information, or treatment outcome information includes toxicity information, which includes at least one of toxicity indication, toxicity prediction, or toxicity risk indication.

27. The method according to claim 25, wherein, At least one of the treatment safety information, treatment efficacy information, or treatment outcome information includes toxicity information located at an anatomical structure, substructure, or region.

28. A computer-implemented system comprising a processor configured with instructions that can be executed to perform the method according to any one of claims 1 to 27.

29. A computer-readable medium comprising instructions executable to perform the method according to any one of claims 1 to 27.

30. The computer-readable medium according to claim 29, wherein, The computer-readable medium is non-transitory.

31. A computer-implemented method for comparing image contour drawing by different human participants without requiring the different participants to perform contour drawing on a shared image dataset, the method comprising: Receive a first set of image datasets, including one or more first image datasets of one or more human or animal subjects generated by an imaging modality, and including first contour drawing data generated by a first participant; Receive a second image dataset set, including one or more second image datasets of one or more human or animal subjects generated by the imaging modality, including second contour drawing data generated by a second participant; A computer-implemented automatic segmentation engine is used to automatically draw contours for each of the first image dataset and the second image dataset to obtain third contour drawing data generated by the automatic segmentation engine, without the need for manual contour drawing and without the need for identical or overlapping first and second image datasets. Comparing the contour drawing of the first participant and the contour drawing of the second participant includes comparing the first contour drawing data generated by the first participant and the third contour drawing data generated by the automatic segmentation engine to generate first comparison data, and further comparing the second contour drawing data generated by the second participant and the third contour drawing data generated by the automatic segmentation engine to generate second comparison data. as well as Based on the first comparison data and the second comparison data, a quality metric is generated based on one or more differences between the contour drawing of the first participant and the contour drawing of the second participant.

32. The method according to claim 31, wherein, The quality metric includes at least one of contour plotting information, which is associated with at least one of treatment plan information, treatment safety information, treatment efficacy information, or treatment outcome information regarding one or more treatments performed using the contour information.

33. A computer-readable storage device, comprising instructions that, when executed by at least one processor, configure the at least one processor to perform the following operations: Receive a first set of image datasets, including one or more first image datasets of one or more human or animal subjects generated by an imaging modality, and including first contour drawing data generated by a first participant; Receive a second image dataset set, including one or more second image datasets of one or more human or animal subjects generated by the imaging modality, including second contour drawing data generated by a second participant; A computer-implemented automatic segmentation engine is used to automatically draw contours for each of the first image dataset and the second image dataset to obtain third contour drawing data generated by the automatic segmentation engine, without the need for manual contour drawing and without the need for identical or overlapping first and second image datasets. Comparing the contour drawing of the first participant and the contour drawing of the second participant includes comparing the first contour drawing data generated by the first participant and the third contour drawing data generated by the automatic segmentation engine to generate first comparison data, and further comparing the second contour drawing data generated by the second participant and the third contour drawing data generated by the automatic segmentation engine to generate second comparison data. as well as Based on the first comparison data and the second comparison data, an indication of one or more differences between the contour drawing of the first participant and the contour drawing of the second participant is generated.

34. A system for comparing image contour drawing by different human participants without requiring the different participants to perform contour drawing on a shared image dataset, the system comprising: At least one processor; as well as A computer-readable storage device includes instructions that, when executed by at least one processor, configure the at least one processor to perform the following operations: Receive a first set of image datasets, including one or more first image datasets of one or more human or animal subjects generated by an imaging modality, and including first contour drawing data generated by a first participant; Receive a second image dataset set, including one or more second image datasets of one or more human or animal subjects generated by the imaging modality, including second contour drawing data generated by a second participant; A computer-implemented automatic segmentation engine is used to automatically draw contours for each of the first image dataset and the second image dataset to obtain third contour drawing data generated by the automatic segmentation engine, without the need for manual contour drawing and without the need for identical or overlapping first and second image datasets. Comparing the contour drawing of the first participant and the contour drawing of the second participant includes comparing the first contour drawing data generated by the first participant and the third contour drawing data generated by the automatic segmentation engine to generate first comparison data, and further comparing the second contour drawing data generated by the second participant and the third contour drawing data generated by the automatic segmentation engine to generate second comparison data. as well as Based on the first comparison data and the second comparison data, an indication of one or more differences between the contour drawing of the first participant and the contour drawing of the second participant is generated.

Images