System and method for quality assurance of medical image segmentation - Patents.com

Abstract

A method and system are described for reviewing contouring of medical images in a contouring system, the method comprising the steps of providing at least one medical image showing a structure to be contoured, generating a contour of the structure on the at least one medical image, determining whether the generated contour complies with guidelines for the structure being contoured, providing feedback on the quality of the contour in response to a conformance determination, and continuing to generate the contour based on the provided feedback. A method and system are also described for reviewing previously contoured medical images.

Description

[Technical field]

[0001] The present invention relates to the field of medical imaging and medical image processing, and more particularly to the review of medical image contours, particularly in the field of radiation therapy planning. [Background technology]

[0002] In many scenarios, it is necessary for clinicians to contour anatomical structures on medical images. For example, in radiation treatment planning, organs, tumors, and target volumes are typically contoured by a radiation oncologist on a patient's medical images. The patient images are usually acquired with a computed tomography (CT) scan, but may also be acquired with other imaging modalities, such as magnetic resonance imaging (MRI). These images typically comprise a stack of 2D images of cross sections of the patient, which together form a 3D volumetric image.

[0003] Patient medical images are contoured by clinicians to indicate which areas should be irradiated and which healthy tissues and organs should be avoided; overtreatment can cause harm while undertreatment can lead to cancer recurrence. Thus, when preparing treatment, organs-at-risk (OAR) and target volumes need to be contoured in 'planning' images to irradiate the tumor while sparing healthy tissue. Planning software can then calculate a treatment plan that maximizes the radiation dose to the target volume and tumor while minimizing the dose to the surrounding healthy tissue. This process of contouring is known as contouring, a term that can be used to indicate the process of contouring in either 2D or 3D to define the boundaries of anatomical structures. This contouring is time-consuming and skill-intensive, leading to variability in the contour and the resulting treatment effect. Similarly, the term auto-contouring is sometimes used to indicate that the contours are generated automatically by a system.

[0004] Manual contouring of patient cases by human operators is time consuming and subject to variability in contouring anatomical structures [1]. Such variability is due to intra- and inter-operator variability, as well as variability between different institutions or departments. To reduce this variability, institutions and professional organizations have developed contouring guidelines, e.g., [2], and atlases, e.g., [3]. The use of guidelines has been found to reduce inter-observer variability [4, 5].

[0005] Guidelines take the form of textual descriptions detailing how structures should be drawn, whereas an atlas, in this context, is a "gold standard" set of contours to be drawn on example patient images. Typically, guidelines and / or atlases are formed by consensus among experts.

[0006] Although the clinical team is trained and educated on the guidelines, variability may again occur over time as staff become complacent [6] or as guidelines are updated. This variability, and the risk of errors in contouring, can be mitigated through having another senior clinician review and correct the contours, a process called peer review. Thus, peer review is often suggested as a method to check the contours, ensure their accuracy, and remind staff of the contouring guidelines. Peer review of contours to ensure quality is highly recommended [7, 8, 9].

[0007] However, this process is resource intensive, and in practice, contours are often not checked against the guidelines or peer reviewed due to resource and time constraints. Therefore, a system that automatically checks conformance of contouring to the guidelines is desirable.

[0008] Guidelines for contouring are usually: Anatomical boundaries between structures, Other anatomical structures, or other anatomical features, , to describe the correct contour for the structure being contoured.

[0009] These reference points are called the reference anatomy. The structure being contoured is referred to as the target structure (it may not be the target of radiation therapy, but this term is used only to refer to the intended purpose of the user's contouring).

[0010] The guidelines can refer to delineating the target structure in a relative orientation to a reference anatomy, such as, for example, “superiorly, the entire heart begins just inferior to the left pulmonary artery”

[16] , where the “left pulmonary artery” is considered the reference anatomy and the relative orientation is “just inferior.”

[0011] Guidelines may refer to delineating target structures at a distance from a reference anatomy, such as, for example, "the lymph node CTV was defined as the area encompassed by a 7 mm margin around the applicable pelvic vessels (arteries and veins)"

[17] , where the "applicable pelvic vessels" are the reference anatomy (further defined in the guidelines) and the distance is a 7 mm extension margin. The "CTV" is the region known as the clinical target volume, i.e., the area where tumor cells are expected to be. Such regions are often an extension around visible structures to address non-visible tumor spread.

[0012] Guidelines can refer to contouring target structures by including or excluding reference anatomy, such as, for example, “the CTV was modified to exclude bones and muscles”

[17] , where “bones and muscles” are the reference anatomy that is excluded from the contour of the target structure, the “CTV.”

[0013] Several technical solutions have been proposed to assess the quality of the contours using automated methods to identify errors. Various techniques have been published that are used to assess the contours of targets and organs at risk.

[0014] One class of approaches evaluates patient contours against the distribution of parameters of previously contoured cases

[10] . However, such approaches only flag outliers to the distribution and do not provide quality assurance against the contouring guidelines. Failure modes can be communicated to the user, but the user is still required to interpret the contour against the guidelines. Furthermore, the system does not detect contours that violate the guidelines when the contour has the expected patient distribution. For example, by comparing the heart volume to previously contoured heart volumes, one may infer that an abnormally large contour may be wrong and therefore requires review. However, the patient may have a large heart relative to the population and the contour may be correct. Conversely, the heart may be inaccurately under-contoured and the contour may have an “acceptable” volume that is seemingly within the normal distribution. Such systems do not relate failure modes to published contouring guidelines.

[0015] More complex machine learning approaches have also been applied to group features and determine potential erroneous contours.

[11] However, such approaches still fall into the category of detecting anomalous contours relative to population distributions and do not correlate back to the flagging of cases for review to guidelines, as is done in human-performed peer review.

[0016] Another class of approaches is the comparison of contours with automatically generated contours (autocontours)

[12] . In this approach, the autocontour is assumed to be acceptable and therefore any deviations from it are flagged as errors. Such approaches have also been used to review autocontours by comparison with an independent system

[13] . If the two systems disagree, the case is flagged for editing. Such systems do not give a meaningful description of the type of error, but simply report differences. Also, it is possible for both systems to agree and both to be incorrect. Thus, this approach does not provide the same feedback linked to guidelines as is done in peer review.

[0017] A further approach that has been taken is to simulate an image from the contours drawn, and then compare the simulated image to the original medical image.

[14] Again, differences are flagged to indicate contours that may require review. Such an approach to comparing images also precludes the contours themselves from requiring visual review of the "error map," and therefore cannot provide the user with human-interpretable feedback about the nature of the contouring errors.

[0018] Machine learning systems have also been applied to predict segmentation quality scores

[15] . However, the concept of a quality score does not exist in delineation guidelines, and a low quality score cannot give a human-interpretable indication as to what violations of the guidelines may have occurred.

[0019] It is therefore observed that none of these systems have a way to associate potential errors with natural language explanations. None of these systems can suggest changes based on guidelines. Such systems may highlight poor contours, but do not give the user insight into why those contours are poor.

[0020] Accordingly, although automated contour quality inspection systems have been proposed, no solution can be fully used as a form of automated "peer review" to provide human-interpretable, or natural language, feedback on where contouring guidelines have been violated. [1] Brouwer CL, Steenbakkers RJ, van den Heuvel E, Duppen JC, Navran A, Bijl HP, Chouvalova O, Burlage FR, Meertens H, Langendijk JA, van't Veld AA. 3D variation in delineation of head and neck organs at risk. Radiation Oncology. 2012 Dec;7(1):1-0. [2] Brouwer CL, Steenbakkers RJ, Bourhis J, Budach W, Grau C, Gregoire V, Van Herk M, Lee A, Maingon P, Nutting C, O'Sullivan B. CT-based delineation of organs at risk in the head and neck region: DAHANCA, EORTC, GORTEC, HKNPCSG, NCIC CTG, NCRI, NRG Oncology and TROG consensus guidelines. Radiotherapy and Oncology. 2015 Oct 1;117(1):83-90. [3] https: / / www.nrgoncology.org / About-Us / Center-for-Innovation-in-Radiation-Oncology / Head-and-Neck / Head-and-Neck-Atlases [4] Sun KY, Hall WH, Mathai M, Dublin AB, Gupta V, Purdy JA, Chen AM. Validating the RTOG-endorsed brachial plexus contouring atlas: an evaluation of reproducibility among patients treated by intensity-modulated radiotherapy for head-and-neck cancer. International Journal of Radiation Oncology* Biology* Physics. 2012 Mar 1;82(3):1060-4. [5] Vinod SK, Min M, Jameson MG, Holloway LC. A review of interventions to reduce inter‐observer variability in volume delineation in radiation oncology. Journal of medical imaging and radiation oncology. 2016 Jun;60(3):393-406. [6] Brouwer CL, Boukerroui D, Oliveira J, Looney P, Steenbakkers RJ, Langendijk JA, Both S, Gooding MJ. Assessment of manual adjustment performed in clinical practice following deep learning contouring for head and neck organs at risk in radiotherapy. Physics and imaging in radiation oncology. 2020 Oct 1;16:54-60. [7] The Royal College of Radiologists. Radiotherapy target volume definition and peer review RCR guidance. Clin Oncol. 2017; [8] Rooney KP, McAleese J, Crockett C, Harney J, Eakin RL, Young VAL, et al. The impact of colleague peer review on the radiotherapy treatment planning process in the radical treatment of lung cancer. Clin Oncol. 2015;27(9):514-8. [9] Abrams RA, Winter KA, Regine WF, Safran H, Hoffman JP, Lustig R, et al. Failure to adhere to protocol specified radiation therapy guidelines was associated with decreased survival in RTOG 9704 - A phase III trial of adjuvant chemotherapy and chemoradiotherapy for patients with resected adenocarcinoma of the pancreas. Int J Radiat Oncol Biol Phys. 2012;82(2):809-16.

[10] Altman MB, Kavanaugh JA, Wooten HO, Green OL, DeWees TA, Gay H, Thorstad WL, Li H, Mutic S. A framework for automated contour quality assurance in radiation therapy including adaptive techniques. Physics in Medicine & Biology. 2015 Jun 17;60(13):5199.

[11] McIntosh C, Svistoun I, Purdie TG. Groupwise conditional random forests for automatic shape classification and contour quality assessment in radiotherapy planning. IEEE transactions on medical imaging. 2013 Mar 6;32(6):1043-57.

[12] Men K, Geng H, Biswas T, Liao Z, Xiao Y. Automated quality assurance of OAR contouring for lung cancer based on segmentation with deep active learning. Frontiers in Oncology. 2020 Jul 3;10:986.

[13] Rhee DJ, Cardenas CE, Elhalawani H, McCarroll R, Zhang L, Yang J, Garden AS, Peterson CB, Beadle BM, Court LE. Automatic detection of contouring errors using convolutional neural networks. Medical physics. 2019 Nov;46(11):5086-97.

[14] Brusini I, Padilla DF, Barroso J, Skoog I, Smedby O, Westman E, Wang C. A deep learning-based pipeline for error detection and quality control of brain MRI segmentation results. arXiv preprint arXiv:2005.13987. 2020 May 28.

[15] Chen X, Men K, Chen B, Tang Y, Zhang T, Wang S, Li Y, Dai J. CNN-based quality assurance for automatic segmentation of breast cancer in radiotherapy. Frontiers in Oncology. 2020 Apr 28;10:524.

[16] Feng M, Moran JM, Koelling T, Chughtai A, Chan JL, Freedman L, Hayman JA, Jagsi R, Jolly S, Larouere J, Soriano J. Development and validation of a heart atlas to study cardiac exposure to radiation following treatment for breast cancer. International Journal of Radiation Oncology* Biology* Physics. 2011 Jan 1;79(1):10-8.

[17] Toita T, Ohno T, Kaneyasu Y, Uno T, Yoshimura R, Kodaira T, Furutani K, Kasuya G, Ishikura S, Kamura T, Hiraoka M. A consensus-based guideline defining the clinical target volume for pelvic lymph nodes in external beam radiotherapy for uterine cervical cancer. Japanese journal of clinical oncology. 2010 May 1;40(5):456-63. Summary of the Invention

[0021] Thus, the following problem has been solved by the present invention: It is therefore desirable to have a system and method for checking contours on medical images according to contouring guidelines and providing feedback to a user on the contour quality, preferably in natural language (human interpretable) feedback.

[0022] According to the present invention, there is provided a method for reviewing contours of medical images in a contouring system, the method comprising the steps of: providing at least one medical image showing a structure to be contoured, generating a contour of the structure on the at least one medical image, determining whether the generated contour complies with guidelines for the structure being contoured, providing feedback on the quality of the contour in response to the determination of compatability, and continuing to generate the contour based on the provided feedback. Preferably, the method may further comprise repeating the steps of determining, providing feedback, and contouring until the structure on the medical image is completely contoured.

[0023] In a further embodiment of the present invention, there is provided a method of reviewing previously contoured images in a contouring system, the method comprising the steps of loading a medical image having one or more contours, determining whether at least one of the contours conforms to guidelines for contoured structures, and providing feedback on the quality of the contours on the medical image in response to the determination of conformance with guidelines, preferably these steps being repeated until all of the contours on the medical image have been reviewed.

[0024] In a preferred embodiment of the invention, the method further comprises the step of editing the contour in response to the feedback about the quality of the contour.

[0025] Preferably, the generated contour is generated manually or semi-automatically.

[0026] In an embodiment of the invention, the method further comprises the step of outputting the contours of the contoured structures in the medical images. More preferably, the method steps are repeated such that multiple structures in the at least one medical image are contoured.

[0027] In a preferred embodiment of the invention, the method further comprises displaying one or more contours on the medical image before the user begins generating manual contours for the medical image. Preferably, the method further comprises displaying the one or more contours on the medical image before the system determines whether the one or more contours conform to the guidelines.

[0028] Preferably, the method also comprises the step of determining at least one violation zone from said guidelines, said at least one violation zone being used to determine areas of said scanned image where a contour of said contoured structure should not be placed. Preferably, said at least one violation zone is determined using machine learning on example medical images. In a preferred embodiment of the invention, the method comprises the step of providing additional spatial feedback on the position of said contour relative to the position of said at least one violation zone. Preferably, said additional spatial feedback for each of said at least one violation zone is specific to the image area covered by said at least one violation zone. In a preferred embodiment of the invention, said additional spatial feedback is provided by highlighting areas on the image where said contour violates said violation zone. In yet a further embodiment of the invention, said additional spatial feedback is provided as soon as a violation of said violation zone by said contour occurs.

[0029] More preferably, the guidelines for contouring refer to one or more of anatomical boundaries between structures, other anatomical structures, and other anatomical features.

[0030] In one embodiment of the invention, a reference anatomy on the medical image is identified using at least one of atlas-based automatic contouring, machine learning methods, and algorithmic approaches.

[0031] Preferably, a comparison of the relative position of the contour to the reference anatomy in the medical image is used to determine compliance with a guideline. More preferably, a tolerance is applied to the contour to the reference anatomy. In an embodiment of the invention, the tolerance is determined using at least one of an isotropic margin or a directional margin.

[0032] In an embodiment of the invention, the at least one medical image is one of a 2D image, a 3D image, or a time series of medical images, more preferably the at least one medical image is at least one of a CT scan, a CBCT scan, a PET scan, a SPECT scan, or an MRI scan.

[0033] In one embodiment of the invention, the feedback is natural language or human interpretable feedback. Preferably, the feedback is provided as a report. In a further embodiment of the invention, the feedback is provided by annotation of the contour to indicate the type of error in the contour. In one embodiment of the invention, the annotation is one or more of: symbolic annotation, text annotation linked to a line, line weight change, line color change, line style change, shading / coloring around the line, shading / coloring within the violations in the contour.

[0034] Preferably, the user can select an appropriate guideline according to the structure being delineated or the delineated structure in a pre-delineated image.

[0035] More preferably, the guidelines provided are determined automatically according to the structure being outlined.

[0036] Preferably, the method for determining whether a generated contour fits the guidelines is derived from a database of previously contoured images, more preferably said database includes previous feedback on whether / how previous contours have varied from said guidelines.

[0037] According to the present invention there is also provided a system for contouring at least one medical image, the system comprising a display for displaying at least one medical image to be contoured by a user, and a processor for determining when the user has started contouring a structure on the medical image, the processor determining whether a generated contour complies with guidelines for the structure being contoured, and in response to the determination of contour compatibility with the guidelines providing feedback to the user on the quality of the contour, thereby enabling the user to adjust the contour to take into account the provided feedback.

[0038] Preferably, the generated contour is generated manually or semi-automatically.

[0039] In a further embodiment of the present invention, there is provided a system for analysing medical images having contoured structures, the system having an input for receiving at least one contoured medical image and a processor for determining whether at least one of the contours conforms to guidelines for the contoured structure, the processor providing feedback on the quality of the contour on the medical image in response to the determination of conformance with the guidelines.

[0040] Preferably, the system further comprises a display for displaying said at least one contoured medical image. [Brief description of the drawings]

[0041] Further details, aspects and embodiments of the present invention are illustrated, by way of example only, with reference to the drawings, in which like reference numbers are used to identify like or functionally similar elements, and in which elements are illustrated for simplicity and clarity and have not necessarily been drawn to scale. [Figure 1] 2 is a flow chart illustrating a method according to one embodiment of the present invention. [Diagram 2]4 is a flow chart illustrating a method according to an alternative embodiment of the present invention. [Diagram 3] FIG. 3(a) shows a medical scan image with a first contour, FIG. 3(b) shows a medical scan image with a violation zone, and FIG. 3(c) shows a medical scan image showing potential areas of error. [Figure 4] 1 shows a medical scan image having contours that do not meet contour delineation guidelines. [Diagram 5] 1 shows a medical scan image alongside a report to provide feedback on image contours. [Figure 6] 1 shows a simplified block diagram of an example medical imaging system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0042] The disclosed invention is a system and method that can provide guideline-related feedback for contours on a medical image. Feedback can be provided for review of existing contours or for newly generated contours on a medical image. Feedback can be related to the quality of the contour. Additional feedback can also be provided, including feedback on spatial properties of the contour, such as the location of the contour on the medical image. Preferably, the feedback is natural language feedback or some other human interpretable feedback. In some cases, the feedback can be visual or auditory feedback. Alternatively, the feedback can be provided as a report. In one embodiment of the invention, the system and method can provide feedback via an interface while the user is drawing or editing the contour or after the user has completed drawing the contour, depending on the user's preferences. In one embodiment of the invention, the contours are generated either manually or semi-automatically. Preferably, the interface is a visual interface that is provided to the user as part of the system. The system and method identifies structures of interest within a medical scan and verifies that the contours of the structures of interest comply with guidelines specific to each structure. In a preferred embodiment of the present invention, the guidelines for contouring refer to one or more of anatomical boundaries between structures, other anatomical structures, and other anatomical features. Preferably, a user can select appropriate guidelines according to the structure being contoured or the contoured structures in a pre-contoured image. In one embodiment of the present invention, the guidelines provided are automatically determined according to the structure being contoured.

[0043] Preferably, the medical image to be contoured or for contour review is a CT scan, although other scans, such as MRI, CBCT, PET, or SPECT, may be used in other embodiments of the invention. In one embodiment of the invention, the at least one medical image is one of a 2D image, a 3D image, or a time series of medical images. If it is determined that the contoured structure does not conform to the appropriate guidelines for the structure, the user is informed of the nature of the violation. The system and method determine which guidelines are appropriate for the body region and structure of interest, evaluate the contours on the medical scan against the guidelines, and provide appropriate feedback, preferably natural language or human interpretable feedback, of potential violations of the guidelines. In one embodiment of the present invention, the system and method may indicate to the user the nature of the potential contour error in a variety of ways, including but not limited to one or more of the following: a report, a screenshot, an annotation of the contour, highlighting the contour with color, fill or shading, pointing out the area with color, fill or shading, an on-screen text annotation of the contour or other part of the image, a tooltip on the contour, audio feedback from the system, pointing out the location or type of violation on the anatomical diagram, highlighting the relevant paragraph of the guideline, preferably an automatic highlighting, displaying the relevant guideline rule. The feedback may link multiple forms of feedback, for example a report may reference a screenshot.

[0044] Referring now to FIG. 6, there is shown a simplified block diagram of an example of a medical imaging system 600 configured to enable a user to display medical images for contouring or review of existing contours as used in the method of the present invention. Preferably, the system comprises at least a display for displaying at least one medical image to be contoured by a user, and a processor for determining that a user has initiated contouring of a structure or contour review of an existing contoured structure on the medical image. Preferably, the processor determines whether the generated contour fits the guidelines for the structure being contoured, and in response to determining the contour fit with the guidelines, provides feedback to the user regarding the quality of the contour, allowing the user to adjust the contour to take into account the provided feedback. Alternatively, the processor determines whether at least one of the contours on the structure having an existing contour fits the guidelines for the contoured structure, and in response to determining the fit with the guidelines, the processor provides feedback regarding the quality of the contour on the medical image.

[0045] In the illustrated example, the medical imaging system 600 comprises one or more user terminals 601, e.g. comprising a workstation or the like, configured to access medical images stored, e.g. in a database 602 or other data storage device. In one embodiment of the invention, the system also comprises an input for receiving at least one contoured medical image or an image to be contoured. In the illustrated example, a single database 602 is shown. However, it will be understood that the user terminal 601 may be configured to access medical images from more than one data storage device. Also, in the illustrated example, the database 602 is shown as being external to the user terminal 601. However, it will be understood that the user terminal 601 may be equally configured to access medical images stored locally, e.g. in a local storage module, indicated at 610, on one or more internal storage elements, e.g. a memory element indicated at 603 or a disk element indicated at 609. The user terminal 601 further comprises one or more signal processing modules, e.g. a signal processing module, indicated generally at 604. The signal processing module(s) are configured to execute computer program code, e.g., stored in the local storage module 609. In the illustrated example, the signal processing module(s) 604 are configured to execute computer program code having one or more automatic contour quality check components, shown in this example as contour fit to contouring guidelines component 605. The signal processing module 604 in the illustrated example is further configured to execute computer program code having one or more image display components 606 configured to display to a user, e.g., on a display screen 607 or the like, an image having the generated or loaded contours and feedback on the quality of the contours, such as those generated by the contour fit to contouring guidelines component 605.The medical imaging system 600 may further include one or more user input devices, such as those generally indicated at 608, to enable a user to interact with computer program code, etc., executing on the signal processing module(s) 604.

[0046] One embodiment of the method of the present invention is shown in Fig. 1. In step 101, the system loads at least one medical image. The medical image can be a 2D image or a 3D image, or a time series of medical images, which is a sequence of images of the same anatomical structure acquired in the same scanning session, i.e., while the patient remains in the scanner throughout; for example, a 4D image typically has a series of 3D images, each representing a different phase (a particular point in time in the time series) of the respiratory or cardiac cycle. In this embodiment of the present invention, the medical image is a CT image, but other medical images may be used as mentioned above. This image loading process may be user-initiated or automatic. In step 102, the system may also optionally load one or more existing contours for review by the user and / or for editing by the user if necessary after review. Preferably, the editing of the contour is interactive (manual or semi-automatic). The loaded / received contours may be generated by any contour drawing method. Contour editing / review may be for all ROIs (regions of interest) on an image or just a portion of the ROIs, depending on the user's expertise.

[0047] It is expected that such a system for contour review will have the usual display capabilities of a medical image visualization tool, and the user may choose to change the view. That is, the system will have the ability to visualize the image (with the contours) using different layouts of any combination of views (axial, sagittal, coronal, possibly 3D rendering). Such a system will also likely have the ability to window / level the image, change contrast, navigate, zoom, and pan within the 3D image. In one embodiment of the present invention, one or more contours are displayed on the medical image before the user begins to generate a manual contour for the medical image. At some point, the user begins the contouring process at step 103 using a manual or semi-automatic contouring tool available in the system to generate a contour for the loaded medical image. Such tools typically include click-based polygon drawing tools, pen-like freehand drawing tools, and / or brush-like area fill tools. The system then determines at step 104 whether the contour generated or loaded on the image complies with the contouring guidelines for the structure being contoured, and preferably this is done automatically by the system. If the generated contour, or part of the generated contour, is determined to violate the guidelines for the structure being contoured, the system presents this information to the user in step 105 as feedback on the guideline violation in a user interface, preferably in a human interpretable format. In a preferred embodiment of the invention, the method for determining whether the generated contour complies with the guidelines is derived from a database of previously contoured images. Preferably, the database includes previous feedback on whether / how previous contours have changed from the guidelines. In one embodiment of the invention, feedback is provided on the quality of the generated contour in response to the determination of compliance with the guidelines. Preferably, the user then continues to generate the contour based on the feedback provided.The user may then continue to contour / edit the contour(s) by repeating these steps until the contour(s) are satisfied, taking into account the feedback to ensure that the best possible contour(s) are generated and that the structures on the medical images are completely contoured. Upon completing the contouring of the medical images at step 106, the system saves, stores, or outputs the contour(s) at step 107. The output of the contour(s) of the contoured structures of the medical images is typically in a format commonly used within the medical device community, such as the DICOM format. DICOM is also an image transfer protocol, so that the data can be transferred to another device, such as a Treatment Planning System (TPS) or a Picture Archiving and Communications System (PACS). Alternatively, the contour(s) may be stored in a local or network file system. In a preferred embodiment of the present invention, the steps of FIG. 1 may be repeated such that one or more structures in at least one medical image are contoured.

[0048] An alternative embodiment of the operation of the present invention is illustrated by FIG. 2. This embodiment does not involve the generation of new contours on a medical image, but rather the review of existing contours on previously contoured images for conformance with appropriate guidelines. The system loads at least one medical image (which may be 2D or 3D, or time-series medical images) and a set of one or more associated contours at step 201. This image loading process may be user-initiated or automatic. In one embodiment of the present invention, the method may also include the step of displaying one or more contours on the medical image before the system determines whether the one or more contours conform to the guidelines. The system then determines at step 202 whether the one or more contours for review conform to the guidelines for the contoured structure, preferably an automatic determination. In response to the determination of conformance with the guidelines, the system provides feedback regarding the quality of the contours on the medical image. The system may also generate a report detailing potential violations of one or more guidelines by the reviewed contour. Preferably, the system also suggests possible improvements that may be made to the contours based on the feedback provided. Preferably, improvements are suggested in natural language or human interpretable form in step 203. The system may also make a report available to the user in step 204. In one embodiment of the invention, the report can be exported / stored in any document format (PDF, DOC, etc.) including DICOM. Alternatively, the report may either be displayed directly / immediately on the screen or stored (in DICOM or any other format) for later retrieval, or it may be emailed or made available as a download, or even sent directly to a printer if a hard copy is required. In one embodiment of the invention, the method may also include a step of editing the contours in response to feedback on the quality of the contours. In a preferred embodiment of the invention, the steps of FIG. 2 may be repeated until all of the contours on the medical image have been reviewed.

[0049] In some embodiments of the present invention, the system may also provide feedback based on experience learned from peer review (preferably manual peer review) of the contours or with one or more examples of incorrect contours. Machine learning methods may be used to learn the correspondence between one or more contours, their location on the medical image, and previous human-interpretable explanations / knowledge of how the contour violates the guidelines or can be improved. To do so, a database of previous contours is needed along with information on how the contour violates the guidelines or how the contour can be improved. Optionally, this database of previous contours also includes the patient's medical images so that the relationship between the contour and the images can be learned. A machine learning system is then trained with this database of contours and optional medical images to develop a model that can predict what feedback is appropriate to provide for a given contour and optionally associated patient medical images. Such an approach may be trained per body region, or per guideline, or for several body regions and / or guidelines. Once the model is developed, the system uses the trained model to analyze the user contours (step 104 in FIG. 1 and step 202 in FIG. 2). Because the system has been trained with examples of human-interpretable feedback, the system can provide feedback about the user's contour(s) in a human-interpretable form, for example, the feedback can be provided as information displayed on a screen, annotations of the contour, or some other means.

[0050] In some embodiments of the present invention, "violation zones" are defined on the example medical image with additional linked spatial feedback. Typically, an image will have at least one violation zone. These violation zones are areas of the medical image where the contour of the structure being contoured should not be placed. For example, when the guidelines state "superiorly, the entire heart starts just below the left pulmonary artery", the area above the left pulmonary artery can be identified and automatically annotated on the example medical image as a "violation zone" where the heart contour should not be placed. This violation zone can also be linked with feedback that "the guidelines state that the heart should start below the pulmonary artery. The contour drawn appears to extend too far upwards". This feedback can be provided by an on-screen display, as an audio message to the user. These one or more violation zones are then mapped or detected / predicted on the patient medical image being contoured or evaluated by the user. In a preferred embodiment of the present invention, the violation zone does not have to be visible to the user. Preferably, areas of the image where the contour violates the violation zone can be clearly identified to the user. In one embodiment of the present invention, the method further comprises determining at least one violation zone from the guidelines, with which the at least one violation zone is used to determine areas of the scanned image where the contour of the contoured structure should not be placed. Preferably, the at least one violation zone is determined using machine learning on the example medical images. The area of ​​the contour that violates the violation zone is clearly identifiable based on the color, shading, or line weighting of the border of the contour. Preferably, this mapping of the violation zone can be done by finding a mapping between the patient medical image and the example medical image using image registration. Typically, image registration finds a mapping between the patient image and the example medical image by a transformation that maximizes a similarity measure between the two images after a transformation has been applied, whether local or global. The use of multiple local transformations at different points in the image is called deformable registration.The global transformation of the image is called rigid registration or affine registration. In a further embodiment of the present invention, the violation zones can also be learned from example medical images by generating a machine learning model using machine learning methods. This machine learning model is then applied to the patient medical images to predict violation zones on the patient medical images for structures or regions of interest to be contoured. Other approaches for identifying regions of interest on medical images will be known to those skilled in the art, such as classical algorithmic approaches including, but not limited to, thresholding, watershed methods, level set methods, and graph cut methods

[18] . Upon analyzing the patient medical images (step 104 in FIG. 1 and step 202 in FIG. 2), the system compares the user-generated contour with the violation zones for the contour provided from the guidelines. The system can then provide specific spatial feedback regarding the location of the contour and other spatial parameters relative to the location of at least one violation zone. If the user-generated contour overlaps with a violation zone, additional linked spatial feedback is provided to the user, either visually or using audio feedback or some other feedback system. In one embodiment of the invention, the spatial feedback is feedback regarding the position of the contour relative to the position of the at least one violation zone. More preferably, the additional spatial feedback for each of the at least one violation zone is specific to the image area covered by the at least one violation zone. Preferably, the spatial feedback is provided immediately as soon as a violation of the violation zone by the contour occurs, i.e. as soon as the generated contour encounters the violation zone, while the user is contouring or editing. However, in some cases the system may wait until the contour is completed before providing spatial feedback of the contour's possible overlap with the violation zone. In one embodiment of the invention, the additional spatial feedback is provided by highlighting areas on the image where the contour violates the violation zone.The system may also highlight potentially erroneous portions of the contour to the user by highlighting them in some manner on the contoured image or by notifying the user in some other manner. In some embodiments of the invention, the evaluation of the contour may be done retroactively on previously generated contours. In one such embodiment, the feedback can be reported in a user interface that displayed the images and contours to the user. In an alternative embodiment, the retroactive evaluation of the contours is done automatically in the system and the feedback is provided to the user as a report, for example sent by email or downloadable from a web page.

[0051] One such embodiment of the present invention is shown in Figure 3. Figure 3(a) shows a patient image 301 on which a user has drawn a contour 302. This is typically drawn either manually or semi-automatically. In Figure 3(b) an automatic "violation zone" 303 has been mapped by the system onto the patient image 304. Note that 301 and 304 represent the same patient image at different stages of processing. In Figure 3(c) the system graphically indicates to the user the area of ​​potential error 305 on the patient image 306, as well as providing an explanation 307 of the potential guideline violation linked to the violation area 308. As shown, this indication and explanation is provided visually, but may be provided in different ways, for example as audio feedback. Note that 308 represents the same as 303, and 306 represents the same image as 304, but at a different stage of processing. As mentioned above, the violation zone is not necessarily displayed to the user, and preferably is not shown at all. In a preferred embodiment of the invention, areas where the contour violates the violation zone may be highlighted using a different color, annotation, line thickness or line style on the portion of the contour that is within the violation zone, or by shading or chopping up the area in or around the portion of the contour that is within the violation zone.

[0052] In another embodiment of the invention, the system and method are configured to apply one or more rules to detect possible guideline violations on the generated contour. As mentioned in the background, many guidelines use a relationship to a reference anatomy to define the target structure. In this embodiment of the invention, the system automatically determines the location of the reference anatomy. Preferably, a comparison of the relative location of the contour to the reference anatomy in the medical image is used to determine compliance with the guideline. The system applies one or more rules derived from the guidelines for contouring. These rules can be pre-configured in the system, loaded from a database, configured by the user, or determined using natural language processing of the guidelines. The system identifies whether one or more of the rules have been violated. If the system determines that a rule has been violated, it reports the violation back to the user in a human-interpretable format, such as, for example, a textual display or highlighting the problem area of ​​the contour with a different color, shade, or weight.

[0053] Such an embodiment of the invention is shown in Fig. 4. The system shows a user interface 401 on a display screen, such as a computer monitor, or a tablet device, or any other display device suitable for use in contouring. Within the user interface, a patient medical image 402 is displayed, which is preferably loaded in response to a user action. Preferably, the image is a CT image, but other imaging techniques may be used to provide the medical image to be contoured. The medical image shows a cross-section of the patient, showing the patient's skin contour 403, left lung 404, right lung 405, and heart 406. The user draws the right breast contour 407. The contour is drawn manually or semi-automatically. The system has identified the reference anatomy of the patient's skin contour 403, left lung 404, right lung 405, and heart 406. Preferably, this identification is done automatically by the system. The system has applied one or more rules from the guidelines: the right breast contour 407 should not overlap with lung tissue, and the breast contour 407 should be 5 mm from the skin border. The system provides this feedback in a human interpretable form, preferably through the use of annotations 408 and 409 on the contour along with keys 410 and 411 to provide an interpretation of the annotations in the user interface. The keys can be provided on the screen, or in a separate look-up table, or alternatively as audio output, or the keys may be provided separately from the user interface, for example in a user manual. In a preferred embodiment of the invention, the annotations to the contour may be one or more of: symbolic annotations, text annotations linked to the line, line weighting changes, line color changes, line style changes, shading / coloring around the line, shading / coloring within violations of the contour.

[0054] Methods for detecting reference anatomy include methods known to those skilled in the art

[18] and may include (but are not limited to) atlas-based contour delineation, image registration from a reference example, machine learning based contour identification, or classical algorithmic methods including, but not limited to, thresholding, watershed methods, level set methods, and graph cut methods.

[0055] The system may apply one or more rules including, but not limited to, applying a relative position to a reference anatomy (i.e., must be above / below / left / right / front / back relative to...), checking for inclusion or exclusion of the reference anatomy in the contour, taking a distance measurement to the reference anatomy, applying a tolerance of the contour to the reference anatomy, for example by adding an isotropic or directional margin, etc. Further rules may consider the structure itself without comparison to the reference anatomy, for example "the target structure should be a single continuous volume" or "the target structure should not contain holes".

[0056] The system may indicate the nature of the potential contour error to the user in a variety of ways, including but not limited to one or more of the following: a report, a screenshot, an annotation on the contour, highlighting the contour with color, fill or shading, pointing out the area with color, fill or shading, on-screen text annotation, a tooltip on the contour, audio feedback from the system, pointing out the location or type of violation on the anatomical diagram, automatically highlighting the relevant paragraph of the guideline, displaying the relevant guideline rule. The feedback may link multiple forms of feedback, for example a report may reference a screenshot.

[0057] FIG. 5 illustrates the use of reports to provide feedback to the user. In this embodiment of the invention, the report and images are displayed on the user display, but as previously mentioned, the report can also be provided in any document format (PDF, DOC, etc.), including DICOM. Alternatively, the report can either be displayed directly / immediately on the screen or stored (in DICOM or any other format) for later retrieval, or the report can be emailed or made available as a download, or even sent directly to a printer if a hard copy is required. The report is generated by the system and made available to the user. The report is shown at 501. One error is encapsulated in 502, 503, 504, along with screenshot 505. Another error is encapsulated in 506, 507, 508, and screenshot 509. In one embodiment of the present invention, the <human readable errors> identified at 504 and 508 may take the form of words such as "Breast contour is too close to the skin border. Guidelines state that the contour should be 5 mm from the skin" or "Breast contour overlaps lung tissue". Additional dots shown at 510, 511, and 512 are used to indicate that any number of potential errors may be identified, with at least one potential error being reported. The slice 502 or slice 506 related to the detected potential error is shown. One or more geometric locations representing the contour that may be in error are shown at 503 and 507. A classification of the human-interpretable errors is included in the reports 504 and 508. Such reports may be directly human readable or may be stored in a machine readable format to allow the report to be visualized by the user in text or graphics within a user interface. The reports may also be saved for later analysis and to aid in updating the guidelines or other work.

[0058] The system preferably applies the appropriate guideline(s) to the contour. There are various ways this can be implemented to allow the correct guideline(s) to be applied, and the user can select the appropriate guideline(s) to apply, which can be, for example, via a menu item, drop-down selection, or button in the user interface, or by activating the system with a particular configuration. The system can be configured to detect which guideline(s) to apply based on the designation of the structure(s) to be contoured, detection of the region of the anatomical structure, or by comparison with a template image example and a set of contours (known as an atlas). Preferably, this can be an automatic detection by the system. Detection of the region of the anatomical structure can be performed using a machine learning model trained to classify images into relevant anatomical structures using previous examples. Comparison with the template image example and a set of contours known as an atlas can be performed using deformable image registration to map the contours between the patient image and the atlas image. This process is well known to those skilled in the art

[19] . The distance or overlap between the contours can be used to determine the quality of the match and thus whether the anatomical structures are the same. For example, the atlas may contain regions for all of the structures expected to be contoured. These are then mapped to the patient image using deformable image registration. These contours are not shown to the user, rather the user's contours are compared to each of the atlas contours. The region of the anatomical structure that the user is contouring can be determined by finding the atlas region with the greatest percentage of overlap, or by determining the region with the smallest average distance from the user contour to the closest point on the mapped atlas contour. An intensity-based image similarity measure, such as mutual information, correlation coefficient, or sum of squared differences, can be used to assess the quality of the match between the atlas image and the patient image to determine the region of the anatomical structure, where multiple available atlas images represent different regions of the anatomical structure. Such similarity measures are well known to those skilled in the art

[20] .Different atlases for different anatomical regions can be used to determine the most similar anatomical region. For example, if two atlases are available, representing the pelvic and thoracic regions, respectively, the sum of squared differences after deformable image registration will be lower for the thoracic atlas than for the pelvic atlas when the patient case is a thoracic image. Thus, since the sum of squared differences similarity measure is lower, it can be determined that the user is contouring in the thoracic region and therefore the thoracic contouring guidelines should be applied as a reference. A similar approach can be taken with other similarity measures that consider whether the similarity measure is maximized or minimized for similar images. The reference guidelines for contouring or contour review can be determined from a lookup table, database, or other storage based on the target structure being evaluated by the user. In one embodiment of the present invention, the user can select the appropriate guideline according to the structure being contoured or the contoured structure in the pre-contoured image. Alternatively, the provided guideline can be automatically determined according to the structure being contoured. The user can choose to evaluate a single target structure or multiple target structures in the image.

[18] Sharp G, Fritscher KD, Pekar V, Peroni M, Shusharina N, Veeraraghavan H, Yang J. Vision 20 / 20: perspectives on automated image segmentation for radiotherapy. Medical physics. 2014 May;41(5):050902.

[19] Rohlfing T, Brandt R, Menzel R, Russakoff DB, Maurer CR. Quo vadis, atlas-based segmentation?. InHandbook of biomedical image analysis 2005 (pp. 435-486). Springer, Boston, MA.

[20] Penney GP, Weese J, Little JA, Desmedt P, Hill DL. A comparison of similarity measures for use in 2-D-3-D medical image registration. IEEE transactions on medical imaging. 1998 Aug;17(4):586-95.

[0059] Examples of the invention may be applied to any or all of the following: Picture Archiving and Communication Systems (PACS), advanced visualization workstations, image acquisition workstations, web-based or cloud-based medical information and imaging systems, radiation treatment planning systems (TPS), radiation therapy linear accelerator consoles, and radiation therapy proton beam consoles.

[0060] The present invention has been described with reference to the accompanying drawings. It is to be understood, however, that the present invention is not limited to the specific examples described herein and shown in the accompanying drawings. Also, the illustrated embodiments of the present invention can be implemented for the most part using electronic components and circuits known to those skilled in the art, and will not be described in detail to a degree greater than is deemed necessary as illustrated above for the understanding and appreciation of the concepts underlying the present invention, and in order not to obscure or detract from the teachings of the present invention.

[0061] The invention may be implemented in a computer program for execution on a computer system, comprising at least code portions for performing the steps of a method according to the invention when executed on a programmable apparatus, such as a computer system, or for enabling the programmable apparatus to perform the functions of a device or system according to the invention.

[0062] A computer program is a listing of instructions, such as, for example, a particular application program and / or an operating system. A computer program may include, for example, one or more of a subroutine, a function, a procedure, an object method, an object implementation, an executable application, an applet, a servlet, source code, object code, shared library / dynamic load library, and / or other sequence of instructions designed for execution on a computer system. Thus, some examples describe a non-transitory computer program product that stores executable program code for automatic contouring of cone beam CT images.

[0063] The computer program may be stored internally on a tangible non-transitory computer readable storage medium or transmitted to the computer system via a computer readable transmission medium. All or a portion of the computer program may be provided on a computer readable medium that is permanently, removably, or remotely coupled to an information processing system. The tangible non-transitory computer readable medium may include, for example, without limitation, any number of the following: magnetic storage media, including disk and tape storage media; optical storage media, such as compact disk media (e.g., CD-ROM, CD-R, etc.) and digital video disk storage media; non-volatile memory storage media, including semiconductor-based memory units, such as FLASH memory, EEPROM, EPROM, ROM, etc.; ferromagnetic digital memory; MRAM; volatile storage media, including registers, buffers or caches, main memory, RAM, etc.

[0064] A computer process typically includes an executing (running) program or part of a program, current program values ​​and state information, and resources used by an operating system to manage the execution of the process. An operating system (OS) is software that manages the sharing of a computer's resources and provides programmers with the interface used to access those resources. An operating system processes system data and user input, and responds by allocating and managing tasks and internal system resources as services to the system's users and programs.

[0065] A computer system may include, for example, at least one processing unit, associated memory, and a number of input / output (I / O) devices. When executing a computer program, the computer system processes information in accordance with the computer program and generates resulting output information via the I / O devices.

[0066] In the foregoing specification, the invention has been described with reference to specific examples of its embodiments, but it will be apparent that various modifications and changes may be made thereto without departing from the scope of the invention as set forth in the appended claims, which are not to be construed as limited to the specific examples described above.

[0067] Those skilled in the art will recognize that the boundaries between logical blocks are merely exemplary, and that alternative embodiments may merge logical blocks or circuit elements or impose alternative decompositions of functionality on the various logical blocks or circuit elements. Thus, it should be understood that the architectures depicted herein are merely exemplary, and that in fact many other architectures may be implemented which achieve the same functionality.

[0068] Any arrangement of components to achieve the same functionality is effectively "associated" such that the desired functionality is achieved. Thus, any two components herein that combine to achieve a particular functionality can be viewed as being "associated" with one another such that the desired functionality is achieved, regardless of architecture or intervening components. Likewise, any two components so associated can also be viewed as being "operably connected" or "operably coupled" with one another such that the desired functionality is achieved.

[0069] Those skilled in the art will also recognize that the boundaries between the operations described above are merely exemplary, that operations may be combined into a single operation, that a single operation may be distributed among additional operations, and that operations may be performed with at least partial overlap in time, that alternative embodiments may include multiple instances of a particular operation, and that the order of operations may be changed in various other embodiments.

[0070] However, other modifications, variations, and alternatives are possible, and the specification and drawings are accordingly to be regarded in an illustrative rather than a restrictive sense.

[0071] In the claims, any reference signs placed in parentheses shall not be construed as limiting the claim. The term 'comprising' does not exclude the presence of elements or steps other than those recited in the claim. Moreover, the terms 'a' or 'an', as used herein, are defined as one or more than one. Moreover, the use of introductory phrases such as 'at least one' and 'one or more' in the claims shall not be construed as implying that the introduction of another claim element with the indefinite article 'a' or 'an' limits any particular claim containing the claim element so introduced to an invention containing only one such element, even if the same claim contains the introductory phrase 'one or more' or 'at least one' and an indefinite article such as 'a' or 'an'. The same applies to the use of definite articles. Unless otherwise indicated, terms such as 'first' and 'second' are used to draw arbitrary distinctions between the elements that such terms describe. Thus, these terms are not necessarily intended to indicate a temporal or other prioritization of such elements.The mere fact that certain measures are recited in mutually different claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. 1. A method for reviewing contouring of a medical image in a contouring system, comprising: providing at least one medical image showing a structure to be contoured; generating a contour of a structure on the at least one medical image; determining whether the generated contour conforms to guidelines for the structure being contoured; providing feedback on the quality of the contour in response to said determination of suitability; continuing to generate the contour based on the provided feedback; A method having the following.

2. 1. A method for reviewing previously contoured images in a contouring system, comprising: loading a medical image having one or more contours; determining whether at least one of the contours conforms to guidelines for the contoured structure; providing feedback on the quality of the contour on the medical image in response to the determination of conformance with guidelines; A method having the following.

3. The method of claim 2 , further comprising editing the contour in response to the feedback about the quality of the contour.

4. The method of claim 1 , further comprising repeating the steps of determining, feedback, and delineating until the structure on the medical image is completely delineated.

5. The method of claim 1 or 4, wherein the generated contour is generated manually or semi-automatically.

6. The method of claim 1 or 4, further comprising the step of outputting the contours of the contoured structures of the medical image.

7. The method of claim 1 or 4, wherein the steps of the method are repeated so that multiple structures in the at least one medical image are delineated.

8. The method of claim 2 or 3, wherein the steps of the method are repeated until all of the contours on the medical image have been reviewed.

9. displaying one or more contours on the medical image before a user begins generating a manual contour for the medical image; The method of claim 1 or 4, further comprising:

10. displaying one or more contours on the medical image before the system determines whether the one or more contours conform to the guidelines; 4. The method of claim 2 or 3, further comprising:

11. 5. The method according to claim 1, further comprising the step of determining at least one violation zone from the guidelines, the at least one violation zone being used to determine an area of ​​the scanned image in which the contour of the structure being contoured should not be placed.

12. The method of claim 11 , wherein the at least one violation zone is determined using machine learning on example medical images.

13. The method of claim 11 , further comprising providing additional spatial feedback about the position of the contour relative to the position of the at least one violation zone.

14. The method of claim 13 , wherein the additional spatial feedback for each of the at least one violation zone is specific to an image area covered by the at least one violation zone.

15. The method of claim 13 , wherein the additional spatial feedback is provided by highlighting areas on the scanned image where the contour violates the violation zone.

16. The method of claim 13 , wherein the additional spatial feedback is provided immediately upon violation of the violation zone by the contour.

17. The method of any one of claims 1 to 4, wherein the guidelines for contouring refer to one or more of anatomical boundaries between structures, other anatomical structures, and other anatomical features.

18. The reference anatomy on the medical image is Atlas-based automatic contouring, machine learning methods, algorithmic approaches, The method of claim 17, wherein the target is identified using at least one of:

19. The method of claim 18, wherein a comparison of the relative position of the contour to the reference anatomy in the medical image is used to determine compliance with a guideline.

20. The method of claim 18 , wherein a tolerance is applied to the contour relative to the reference anatomy.

21. 21. The method of claim 20, wherein the tolerance is determined using at least one of an isotropic margin or a directional margin.

22. The method of any one of claims 1 to 4, wherein the at least one medical image is one of a 2D image, a 3D image, or a time series of medical images.

23. The method of any one of claims 1 to 4, wherein the at least one medical image is at least one of a CT scan, a CBCT scan, a PET scan, a SPECT scan, or an MRI scan.

24. The method of any one of claims 1 to 4, wherein the feedback is natural language or human interpretable feedback.

25. The method of claim 24 , wherein the feedback is provided as a report.

26. The method of any one of claims 1 to 4, wherein the feedback is provided by annotation of the contour to indicate a type of error in the contour.

27. 27. The method of claim 26, wherein the annotation is one or more of a symbol annotation, a text annotation linked to a line, a line weight change, a line color change, a line style change, shading / coloring around a line, and shading / coloring within a violation portion of a contour.

28. The method according to any one of claims 1 to 4, wherein a user can select an appropriate guideline according to the structure being outlined or according to the outlined structure in a pre-outlined image.

29. The method according to any one of claims 1 to 4, wherein the guidelines to be provided are determined automatically according to the structure being outlined.

30. A method according to any one of claims 1 to 4, wherein the method for determining whether a generated contour complies with a guideline is derived from a database of previously contoured images.

31. 31. The method of claim 30, wherein the database includes previous feedback on whether / how previous contours have varied from the guidelines.

32. 1. A system for contouring of at least one medical image, comprising: a display for displaying at least one medical image to be outlined by a user; a processor for determining that the user has begun outlining a structure on the medical image; and the processor determines whether the generated contour conforms to guidelines for the structure being contoured, and in response to the determination of contour conformance with the guidelines, provides feedback to the user about the quality of the contour, thereby enabling the user to adjust the contour to take into account the provided feedback. system.

33. 33. The system of claim 32, wherein the generated contour is generated manually or semi-automatically.

34. 1. A system for analyzing a medical image having contoured structures, comprising: an input for receiving at least one contoured medical image; a processor that determines whether at least one of the contours conforms to guidelines for the contoured structure; and the processor providing feedback on the quality of the contour on the medical image in response to the determination of conformance with the guidelines. system.

35. 35. The system of claim 34, further comprising a display for displaying the at least one outlined medical image.

Images