Knowledge-based quality assurance of organ-at-risk contours in radiotherapy planning

Abstract

Knowledge-based quality assurance (QA) of organ-at-risk contours in radiotherapy planning assists with contour review based on features obtained from the contours. These contour-based features are input to one or more computational models to predict, determine, or otherwise identify erroneous contours in the radiotherapy plan. The computational models may include statistical models, distance metric-based models, and / or machine learning models.

Description

KNOWLEDGE-BASED QUALITY ASSURANCE OF ORGAN-AT-RISK CONTOURS IN RADIOTHERAPY PLANNINGCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 515,125, filed on July 23, 2023, and entitled “KNOWLEDGE-BASED QUALITY ASSURANCE OF ORGAN- AT-RISK CONTOURS IN RADIOTHERAPY PLANNING." which is herein incorporated by reference in its entirety.BACKGROUND

[0002] Safe and effective radiotherapy relies on the creation and review of contours, which identify the presence of organs-at-risk (OARs) on CT image sets. This can be a time consuming process. Ensuring accurate OAR contouring is a vital and time-consuming component of creating safe and effective radiotherapy plans. Statistical and deep learning models have demonstrated initial potential in detecting improperly delineated contours and may enhance the efficiency and safety of radiotherapy planning. Both the performance of models and the ability' to interpret their results are beneficial for contour qualify assurance (QA). Nevertheless, models with the best performance are often less interpretable.SUMMARY OF THE DISCLOSURE

[0003] It is an aspect of the present disclosure to provide a method for knowledgebased qualify assurance of organ-at-nsk contours in a radiotherapy plan. The method includes accessing radiotherapy plan contour data with a computer system, where the radiotherapy plan contour data include at least organ-at-risk (OAR) contours. Contour-based feature are generated from the radiotherapy plan contour data using the computer system. One or more computational models are accessed with the computer system, where the computational model is constructed to receive contour-based feature data as an input to predict errors in OAR contours. The contour-based feature data are input to the one or more computational models with the computer system, generating contour error data as an output. The contour error data are then outputted via the computer system.

[0004] It is another aspect of the present disclosure to provide a method for knowledgebased qualify- assurance of contours in medical image data. Medical image contour data are accessed with a computer system, where the medical image contour data include at leastcontours generated based on medical images. Contour-based feature data are generated from the medical image contour data using the computer system. A computational model is also accessed with the computer system, where the computational model is constructed to receive contour-based feature data as an input to predict errors in medical image contours. The contourbased feature data are then input to the computational model with the computer system, generating contour error data as an output. The contour error data can then be outputted via the computer system.BRIEF DESCRIPTION OF THE DRAWINGS

[0005] FIG. l is a flowchart setting forth the steps of an example method for generating contour error data from radiotherapy plan contour data using computational models to analyze contour-based features.

[0006] FIG. 2 illustrates an example CCR matrix. Rows correspond to the contour that is evaluated, columns correspond to comparison contours. CCRs to check are indicated in orange and marked with a 1. CCRs were selected by physician and physicist experience and institutional guidelines. The number of checks per OAR are displayed in parenthesis after the evaluated OAR name.

[0001] FIG. 3 illustrates an example of combining contour error data generated from multiple different computational model types. The workflow diagram illustrates automated OAR QA during model validation. Single-contour feature models are trained initially with the training set. The validation set is used to identify the optimal threshold for the single-contour feature models iteratively. Thresholds for the CCR and connectedness models are set by evaluating performance on the validation set.

[0002] FIG. 4 illustrates another example workflow diagram for a knowledge-based QA framework.

[0003] FIG. 5 is a flowchart setting forth the steps of an example method for training a computational model, such as a neural network or other machine learning or statistical model.

[0004] FIG. 6 is a block diagram of an example system for knowledge-based quality assurance of OAR contours.

[0005] FIG. 7 is a block diagram of example components that can implement the system of FIG. 6.DETAILED DESCRIPTION

[0006] Described here are systems and methods for knowledge-based quality assurance (QA) of organ-at-risk contours in radiation therapy planning. In general, the disclosed systems and methods assist with contour review based on statistical and / or machine learning techniques based on single-contour and / or contour-to-contour relationship (CCR) features used for model training. In some other examples, features obtained from connectedness models (e.g., features checking the number of distinct parts that are in a single contour) may also be incorporated to detect and identify outlier features and contours. Advantageously, the disclosed systems and methods both decrease contour review time and improve contour review quality.

[0007] According to some examples, contour-based features (e.g., single-contour features, CCR features, connectedness model features) may be used to identify incorrect or otherwise erroneous organ-at-risk (OAR) contours. In some implementations, the systems and methods can be used for contour review in head and neck (HN) radiotherapy plans, though they may also be used for contour review of other anatomical locations. Individual contour features may be analyzed using a suitable model. For example, a model may include a statistical model based on a z-score, a model based on a Mahalanobis distance (MD) or other distance metric, and / or an autoencoder (AE) model or other machine learning model. A statistical model (e.g., a z-score model) can identify outlier features and contours, providing improved interpretability. Additionally or alternatively, an MD model or AE model may be designed to detect outlier contours only.

[0008] Accordingly, the disclosed systems and methods provide for knowledge-based quality assurance (QA) of OAR contours. The discloses systems and methods utilize contourbased features (e.g., single-contour features, CCR features, connectedness features) to identify incorrect contours for OARs in a radiotherapy plan. In some embodiments, multiple models can be used and combined, both in general and for each OAR type. Additionally or alternatively, models with varying degrees of interpretability may be used to analyze the contour-based features.

[0009] Referring now to FIG. 1, a flowchart is illustrated as setting forth the steps of an example method for generating classified feature data using a suitably trained neural network or other machine learning algorithm. As will be described, the neural network or other machine learning algorithm takes radiotherapy plan contour data as input data and generates contour error data as output data. As an example, the contour error data can be indicative of contours in the radiotherapy plan contour data that may expose OARs to radiation dose levels that are unacceptable or otherwise undesirable. The contour error data can thus be used toreview and revise the contours in the radiotherapy plan. Additionally or alternatively, the neural network or other machine learning algorithm may receive contour data other than radiotherapy plan contour data in order to generate contour error data indicative of contours that are unacceptable or otherwise undesirable. For instance, the neural network or other machine learning algorithm may receive medical image contour data that includes one or more contours based on medical images of a subject (e.g.. contours based on segmented anatomy, contours based on functional imaging data, contours based on quantitative parameters generated from medical images, etc.). The other contour data may also include other patient health data contours based on other types of patient health data. Quality assurance can be performed on these contours using the disclosed systems and methods.

[0010] The method includes accessing radiotherapy plan contour data with a computer system, as indicated at step 102. Accessing the radiotherapy plan contour data may include retrieving such data from a memory or other suitable data storage device or medium. Additionally or alternatively, accessing the radiotherapy plan contour data may include generating such data with radiotherapy planning system and transferring or otherwise communicating the data to the computer system, which may be a part of the radiotherapy planning system.

[0011] Additionally or alternatively, other contour data can be accessed, including medical image contour data. Accessing the medical image contour data may include retrieving such data from a memory or other suitable data storage device or medium. Additionally or alternatively, accessing the medical image contour data may include generating such data by processing medical images of a subject and transferring or otherwise communicating the resulting medical image contour data to the computer system.

[0012] Contour-based feature data are then generated from the radiotherapy plan contour data (or other contour data), as indicated at step 104. As described above, the contourbased feature data may generally include features that are obtained from radiotherapy plan contours, medical image contours, or other patient health data contours, including contour-to- contour features, single-contour features, connectedness features, and so on.

[0013] As a non-limiting example, contour-to-contour relationship (CCR) features may include the minimum distance between two contours and / or the fractional volume of overlap of one contour with another. A matrix can be generated to identify the CCRs to calculate and analyze. Rows in this matrix can be associated with a selected contour, while columns can be associated with a comparison contour. An additional column can be added to allow comparisonto a body contour. Anatomically meaningful CCRs may be selected, for example, based on input from a physician, medical physicist, or other clinician using anatomical and clinical knowledge. Statistical thresholds for each CCR may be identified for each OAR type using a training and validation set.

[0014] The minimum distance feature data can be fit to a gamma distribution ranging from zero to infinity’, while the fractional overlap volume feature data can be fit to a beta distribution ranging from zero to one. As one non-limiting example, starting points for upper and lower outlier cutoffs can be selected by taking the upper and lower 99th percentile boundaries of the data set. These cutoffs can be expanded (e.g., by 0.02 for fractional volume and 2 mm for minimum distance) to minimize identification of errors that may be present but small enough to not be clinically relevant. Expanding the cutoffs can be advantageous for abutting structures, where fractional volume of overlap and minimum distance metrics for the training data both have small deviations from zero. Any features outside of these thresholds identified a contour as erroneous according to the CCR model.

[0015] The CCR model can determine overlap, abutment, or separation between two contours. This is advantageous in clinical settings for at least two reasons. First, contour overlap, abutment, and separation should be consistent with anatomy. Additionally, gaps in contours for abutting OARs may result in unreported high doses to the OAR. Using the systems and methods described in the present disclosure, CCRs that have consistent anatomical relationships or are close to each other can be utilized; however, the techniques can be extended to any CCRs. The high specificity of the CCR model allows for easy deployment as a contour review7tool, either on its own or in conjunction with other models.

[0016] As a non-limiting example, features for a single-contour model can include features describing contour shape, volume, location, orientation, and / or CT number. The Pearson correlation coefficient can be used to identify and remove correlated features across all OARs, and a feature set may be optimized for performance on the validation set. After feature determination, the same feature set can be used for all contour types and model types (Table 1).Table 1. Example list of features used for single-contour, connectedness, and CCR models

[0017] The centroid features of a contour in x, y, and z coordinates (e.g., lateral, vertical, and longitudinal directions) can be calculated as the difference between the contour's centroid and the centroid of a relevant anatomical landmark (e.g., the brainstem for head and neck applications). For brainstem contours, the centroid features may be calculated as the difference in centroid locations between the brainstem contour and the pituitary' contour’s centroid. Calculating these differences accounts for variations in image coordinates between CT images.

[0018] Features including the centroid, volume, and orientation features may also be analyzed with Procrustes analysis. For centroid features, Procrustes analysis superimposes different patients over one another using rotation, translation, and / or scaling to best align the contour sets from different patients with one another and to eliminate variation in patient size and setup orientation (e g., headfirst supine, feet first prone, etc.). This approach improves the accuracy of error identification as well as providing a unified coordinate system for contour centroids across all different anatomical sites. As an example, contour locations can be generated by rotating, translating, and / or scaling individual contours to overlap with training contours in a given coordinate system. This rotating, translating, and scaling can be performed using a Procrustes analysis.

[0019] The extent in x, y, and z can be calculated as the difference between the largest and smallest pixel coordinate values for a given contour. Principal component analysis may be performed to obtain the eigenvectors and eigenvalues of the principal components (PC) of a contour’s pixel coordinates. The x, y, and z components of the first and second PC eigenvectorsmay be used as orientation features, while the ratio of the PC eigenvalues (X) may be used as shape features.

[0020] The orientation of a PC eigenvector can be arbitrarily positive or negative. To standardize the orientation of PCI or PC2 vectors for a given OAR, a representative eigenvector r0ARis identified from the training set using the following equation: r0AR = argmax(1);Vi VoAR

[0021] where V0ARis the set of all PCI or PC2 eigenvectors for a given OAR in the training set with number n and v is a single PC eigenvector. After identification of r0AR, the orientation of all eigenvectors in the training, validation, and test set (V0AR*) may be oriented either positive or negative to maximize the dot product between r0ARand each eigenvectorv eoAR ■

[0022] One or more individual models can be trained for each OAR. These individual OAR models may be a statistical model (e.g., z-score model, MD model) or a machine learning model (e.g., an AE model). A z-score model calculates individual feature z-scores using:

[0023] where / z and cr are the mean and standard deviation of feature values in the training set. After calculation, the maximum value of z across all the features is output from the model. An MD model calculates the Mahalanobis distance of a contour's features with respect to the training dataset as the model output:

[0024] where / is a vector containing the mean feature values and E-1is the inverse of a covariance matrix calculated from the training data set.

[0025] The output metric for the AE network can be the mean squared difference between reconstructed features and input features for a given contour. As an example, an AE network may include a single hidden layer with 18 neurons and a cost function with a single L2 regularization term. Features may be normalized to their z-score values before being input into the AE model. In an example implementation, the number of epochs was limited to a maximum of 7000, and the L2 weight regularization coefficient was set to 0.005. The number of hidden layers and L2 regularization coefficient were optimized by evaluating model performance on the validation set across a range of values.

[0026] Contour connectedness can be estimated by analyzing the number of distinct parts within an individual contour. To establish the maximum number of allowable parts, a statistical threshold (e.g., a threshold of 99.95%) is set using a distribution (e.g., a gamma distribution) fitted to the training data. The threshold may be optimized by evaluating performance on the validation dataset.

[0027] For the CCR model, an example objective is to determine a set of features that can quantify varying degrees of contour-to-contour overlap and separation. To do this, the CCR model can utilize the minimum distance between two contours and the fractional volume of overlap of one contour with another as its features. The combination of both features yields information that can be used to quantify these relationships. In an example implementation, a Boolean matrix with 42 rows and 43 columns was generated to select the CCRs to include in the CCR model. Rows were associated with the selected contour, while columns were associated wi th the comparison contour. An additional column was added to allow comparison to the body contour. An example CCR matrix is illustrated in FIG. 2. In the illustrated example, the selected CCRs primarily focused on OARs that were close to each other. This included OAR types with distinct anatomical boundaries (e.g., cord and brain stem) and cases where one OAR was a subset of another (e.g., brain stem and brain). Well-defined contours in these cases exhibit consistent anatomical boundaries with each other. In contrast, contours that are not in close proximity to each other may have more uncertainty in their relationship, making them susceptible to false positives.

[0028] The minimum distance feature data can be fit to a gamma distribution ranging from zero to infinity, while the fractional overlap volume feature data can be fit to a beta distribution ranging from zero to one. Distribution types can be selected to have the same upper and lower input domains as their representative features and can follow the probability distribution of the CCR features. As a non-limiting example, initial upper and lower outlier cutoffs can be determined by taking the upper and lower 99thpercentile boundaries of the fitted distribution. The percentile boundaries can be set manually to minimize the number of false positives detected by the CCR model in the validation set. In some implementations, the determined percentile boundary cutoffs can be expanded (e.g., by 0.02 for fractional volume and 2 mm for minimum distance) to minimize identification of errors that may be present, but small enough to not be clinically relevant.

[0029] The CCR model is a tool that can identify incorrect amounts of overlap or separation between two contours. This is advantageous in clinical settings for two reasons:first, overlap and separation should be consistent with actual anatomy, and second, gaps between contours that are anatomically touching may result in unreported high doses to the OAR. In the example described above OCRs that had consistent anatomical relationships or were close to each other were selected, but the technique can be extended to any OCRs. The high specificity of the CCR model allows for easy deployment as a contour review tool, either on its own or in conjunction with other models.

[0030] In some implementations, to improve clarity for potential human reviewers using the systems and methods described in the present disclosure, the connectedness feature can be separated from the single-contour feature models. In these instances, a separate model including only the number of connected parts in a contour can be constructed and used. Using a separate model for connectedness enables easier reporting of this feature to reviewers. As a non-limiting example, to establish the maximum number of allowable parts, a statistical threshold of 99.95% can be set using a gamma distribution fitted to the training data. The threshold can be optimized by evaluating performance on a validation dataset and selected to minimize false positives. A statistical threshold can be used instead of setting a predetermined cutoff as some contours may be allowed to have multiple parts anatomically (e.g., thyroid) and other contours may have multiple parts due to CT image-related scan truncation (e.g., left and right brachial plexus).

[0031] One or more computational models is then accessed with the computer system, as indicated at step 106. In general, the computational model(s) may include statistical models (e g., z-score-based models), distance metric-based models (e.g., models based on a Mahalanobis distance metric), and / or machine learning models (e.g., neural network models such as autoencoder models). Accessing the computational model(s) may include accessing model parameters (e.g.. weights, biases, etc.) that have been constructed, optimized, or otherwise estimated by training the computational model(s) on training data. In some instances, retrieving the computational model(s) can also include retrieving, constructing, or otherwise accessing the particular model architecture to be implemented. For instance, data pertaining to the layers in a neural network architecture (e.g., number of layers, type of layers, ordering of layers, connections between layers, hyperparameters for layers) may be retrieved, selected, constructed, or otherwise accessed; data pertaining to the distance metrics to be used in a computational model based on a distance metric may be retrieved, selected, constructed, or otherwise accessed; and / or data pertaining to the statistical measures to be used in a statistical model may be retrieved, selected, constructed, or otherwise accessed.

[0032] Regarding the use of a neural network, in general, a neural network is trained, or has been trained, on training data in order to identify contour errors amongst radiotherapy plan contour data. An artificial neural network generally includes an input layer, one or more hidden layers (or nodes), and an output layer. Typically, the input layer includes as many nodes as inputs provided to the artificial neural network. The number (and the type) of inputs provided to the artificial neural network may vary based on the particular task for the artificial neural network.

[0033] The input layer connects to one or more hidden layers. The number of hidden layers varies and may depend on the particular task for the artificial neural network. Additionally, each hidden layer may have a different number of nodes and may be connected to the next layer differently. For example, each node of the input layer may be connected to each node of the first hidden layer. The connection between each node of the input layer and each node of the first hidden layer may be assigned a weight parameter. Additionally, each node of the neural network may also be assigned a bias value. In some configurations, each node of the first hidden layer may not be connected to each node of the second hidden layer. That is, there may be some nodes of the first hidden layer that are not connected to all of the nodes of the second hidden layer. The connections between the nodes of the first hidden layers and the second hidden layers are each assigned different weight parameters. Each node of the hidden layer is generally associated with an activation function. The activation function defines how the hidden layer is to process the input received from the input layer or from a previous input or hidden layer. These activation functions may vary and be based on the type of task associated with the artificial neural network and also on the specific type of hidden layer implemented.

[0034] Each hidden layer may perform a different function. For example, some hidden layers can be convolutional hidden layers which can, in some instances, reduce the dimensionality of the inputs. Other hidden layers can perform statistical functions such as max pooling, which may reduce a group of inputs to the maximum value; an averaging layer; batch normalization; and other such functions. In some of the hidden layers each node is connected to each node of the next hidden layer, which may be referred to then as dense layers. Some neural networks including more than, for example, three hidden layers may be considered deep neural networks.

[0035] The last hidden layer in the artificial neural network is connected to the output layer. Similar to the input layer, the output layer typically has the same number of nodes as the possible outputs.

[0036] The contour-based feature data are then input to the one or more computational models, generating output as contour error data, as indicated at step 108. For example, the contour error data may indicate contours in the radiotherapy plan contour data that will result in unacceptable or otherwise undesirable radiation dose levels being imparted to one or more OARs. The contour error data may include a classification of a contour as being erroneous. Additionally or alternatively, the contour error data may include a quantitative score indicating a percentage or likelihood of a contour being erroneous.

[0037] In some implementations, the outputs from multiple computational models can be generated and combined. For example, to combine computational models, predictions from the connectedness and CCR models may be combined with single-contour feature model predictions at various single-contour feature model thresholds, as illustrated in FIG. 2. If any models identify a contour as erroneous, then the contour is classified as such. The optimal thresholds for the single-contour feature models may be determined by maximizing balanced accuracy (BA), defined as the average of the sensitivity and specificity.

[0038] Alternatively, the workflow illustrated in FIG. 3 may be implemented. In this example, AE refers to an autoencoder, MD refers to the Mahalanobis distance, CCR again refers to contour-to-contour relationship, and MSRE refers to mean squared reconstruction error. First, a training dataset (e g., acceptable contours only) can be separated by OAR type (e.g., brain, left eye, esophagus). Then, the desired features can be calculated for each contour and model training performed. Features dependent on only one contour can be assigned to each of the three single-contour feature model types (i.e., AE, MD. and Z-score in the illustrated example), while features involving the relationship between two contours can be assigned to the CCR model type. A feature counting the number of disconnected parts in a single contour can be included as an additional statistical check outside of the other models as the connectedness model. Separate single-contour feature, CCR, and connectedness models can be trained for each OAR type. All models generate a single output metric that indicates the likelihood of a contour being erroneous. Output metrics from models of the same model type can be thresholded using a single value to obtain classifications, as noted above. After model training, the validation set, which includes both acceptable and erroneous contours, can be used to evaluate the model’s performance, select input features, and determine output metricthresholds. Classifications obtained from the single-contour feature models, CCR models, and connectedness models can be combined to form a final classification. To ensure no overfitting, the models can be evaluated on a test set.

[0039] To obtain combined classifications, if any individual model identified a contour as erroneous, it can be classified as such. Thresholds for the connectedness model and CCR model output metrics can be set manually and the single-contour features model thresholds can be tuned to maximize balanced accuracy for the combined classifications. Balanced accuracy can be defined as the average of sensitivity and specificity. The values of prevalence and relative severity can be adjusted, resulting in different optimal thresholds for future clinical use.

[0040] The contour error data generated by inputting the contour-based feature data to the computational model(s) can then be displayed to a user, stored for later use or further processing, or both, as indicated at step 110. As a non-limiting example, the contour error data can be used to refine or otherwise update the radiotherapy plan for the patient, such as by revising the erroneous contours to reduce the dose imparted to the one or more OARs. As another example, the contour error data can be used to refine or otherwise update processing of medical images (e.g., to adjust segmentation of medical images, etc.).

[0041] In some implementations, a report indicating the contour errors is generated and output to a user. For example, the workstation or other computer system runs the model(s) and generates the contour error data, which may then be incorporated into a report that provides information to the end user regarding contour errors. The report allows the end user to review the contours flagged by the model(s) and mark whether they are acceptable or not. The review by the end user can then be used for QA purposes, or may be input back into the model(s) for re-training.

[0042] Referring now to FIG. 5, a flowchart is illustrated as setting forth the steps of an example method for training one or more neural networks (or other suitable machine learning algorithms or computational models) on training data, such that the one or more neural networks are trained to receive contour-based feature data as input data in order to generate contour error data as output data.

[0043] In general, the neural network(s) can implement any number of different neural network architectures. For instance, the neural network(s) could implement a convolutional neural network, a residual neural network, an autoencoder network, or the like. Alternatively, the neural network(s) could be replaced with other suitable machine learning or artificialintelligence algorithms, such as those based on supervised learning, unsupervised learning, deep learning, ensemble learning, dimensionality reduction, and so on.

[0044] The method includes accessing training data with a computer system, as indicated at step 502. Accessing the training data may include retrieving such data from a memory' or other suitable data storage device or medium. In general, the training data can include radiotherapy plan contours that have been labeled by a user. For example, a large, highly curated head and neck OAR dataset may be used for model development. The training dataset may include existing contours that have been labeled as erroneous, or may include augmented data generated by fabricating erroneous contours from existing non-erroneous contours. In some embodiments, the training data may include radiotherapy plan contours data that have been labeled (e.g., labeled as acceptable, labeled as unacceptable and / or erroneous). Additionally, the training data may include other data, such as CT images or other medical images.

[0045] The method can include assembling training data from radiotherapy plan contour data using a computer system. This step may include assembling the radiotherapy’ plan contour data into an appropriate data structure on which the neural network or other machine learning algorithm can be trained. Assembling the training data may include assembling radiotherapy plan contour data and other relevant data. For instance, assembling the training data may include generating labeled data and including the labeled data in the training data. Labeled data may include radiotherapy plan contour data or other relevant data that have been labeled as belonging to, or otherwise being associated with, one or more different classifications or categories. For instance, labeled data may include radiotherapy plan contour data that have been labeled as being associated with erroneous contours.

[0046] As a non-limiting example of generating augmented data, erroneous contours can be generated and assembled as part of the training data. For instance, gold-standard acceptable contours can be edited to mimic clinically observed errors encountered during both manual contouring and autocontouring processes. In some implementations, manually generated erroneous contours can be added directly to validation and test sets after creation. Each erroneous contour error can additionally be categorized as moderate or major by the contour editor, providing the ability to assess how the clinical severity of errors influences the performance of automated outlier detection. Errors categorized as moderate may or may not be clinically relevant depending on clinical context, such as the treatment planning approach andthe relationship with the target, while errors categorized as major can be relevant in nearly all clinical contexts.

[0047] In a non-limiting example, to train and evaluate computational models, a set of 490 gold standard HN patient structure sets with a total of 19,813 contours was used. Specialized HN radiotherapy physicians manually delineated the gold standard contours with a focus on standardization, which were then meticulously reviewed by certified physicists and dosimetry staff. All typical OARs included in HN radiotherapy plans were contoured, amounting to forty -two HN OARs per patient plan. The gold standard structure sets were split into 80% training, 10% validation, and 10% test sets. In addition, 227 erroneous, or outlier, contours were created by a medical physicist by manually editing gold-standard contours.

[0048] For each OAR. a minimum of four or five erroneous contours were created, depending on whether the OAR included left and right counterparts or not, respectively. These ad hoc errors included partially contoured OARs, extra volume contoured on one or several slices, inaccurately delineated organ boundaries, missing slices, mislabeled structures, small disconnected contoured regions (ditzels), and improper anatomical identification according to institutional guidelines. The edits varied in scope from small changes on a single slice to complete recontouring of an organ. As such, these errors were intended to present as typical random errors made by humans or auto-segmentation tools in the clinical OAR delineation context. One-hundred ninety outlier contours were allocated to the validation set and 37 to the test set.

[0049] Since one-class novelty detection approaches are sensitive to outliers in the training data, the contours were grouped based on their OAR and the median absolute deviation of the median (MAD) was calculated for single-contour features. Any contours that had features with more than twelve MAD for each OAR type were removed from the training set. This resulted in the removal of 0% to 3.6% of contours from the training set for each OAR. The threshold of twelve MAD was determined by evaluating the number of contours removed for each OAR and the impact of contour removal on model performance for the validation dataset.

[0050] One or more neural networks (or other suitable machine learning algorithms or computational models) are trained on the training data, as indicated at step 504. In general, the neural network can be trained by optimizing network parameters (e.g., weights, biases, or both) based on minimizing a loss function. As one non-limiting example, the loss function may be a mean squared error loss function.

[0051] Training a neural network may include initializing the neural network, such as by computing, estimating, or otherwise selecting initial network parameters (e.g.. weights, biases, or both). During training, an artificial neural network receives the inputs for a training example and generates an output using the bias for each node, and the connections between each node and the corresponding weights. For instance, training data can be input to the initialized neural network, generating output as contour error data. The artificial neural network then compares the generated output with the actual output of the training example in order to evaluate the quality of the contour error data. For instance, the contour error data can be passed to a loss function to compute an error. The current neural network can then be updated based on the calculated error (e g., using backpropagation methods based on the calculated error). For instance, the current neural network can be updated by updating the network parameters (e.g., weights, biases, or both) in order to minimize the loss according to the loss function. The training continues until a training condition is met. The training condition may correspond to, for example, a predetermined number of training examples being used, a minimum accuracy threshold being reached during training and validation, a predetermined number of validation iterations being completed, and the like. When the training condition has been met (e.g., by determining whether an error threshold or other stopping criterion has been satisfied), the current neural network and its associated network parameters represent the trained neural network. Different types of training processes can be used to adjust the bias values and the weights of the node connections based on the training examples. The training processes may include, for example, gradient descent, Newton's method, conjugate gradient, quasi-Newton, Levenberg-Marquardt, among others.

[0052] The artificial neural network can be constructed or otherwise trained based on training data using one or more different learning techniques, such as supervised learning, unsupervised learning, reinforcement learning, ensemble learning, active learning, transfer learning, training using a generalized adversarial network, or other suitable learning techniques for neural networks. As an example, supervised learning involves presenting a computer system with example inputs and their actual outputs (e.g.. categorizations). In these instances, the artificial neural network is configured to learn a general rule or model that maps the inputs to the outputs based on the provided example input-output pairs.

[0053] The one or more trained neural networks (or other machine learning algorithms or computational models) are then stored for later use, as indicated at step 506. Storing the neural network(s) may include storing network parameters (e.g.. weights, biases, or both).which have been computed or otherwise estimated by training the neural network(s) on the training data. Storing the trained neural network(s) may also include storing the particular neural network architecture to be implemented. For instance, data pertaining to the layers in the neural network architecture (e.g., number of layers, type of layers, ordering of layers, connections between layers, hyperparameters for layers) may be stored.

[0054] Thus, systems and methods for knowledge-based detection of erroneous OAR contours that are clinically relevant in radiotherapy have been described. Models based on single-contour features, CCR features, and / or connectedness features may be used. Classification performance using only single-contour features models may be significantly improved using combinations with other models, such as a non-interpretable AE model. Additionally or alternatively, combining models that take single-contour features as an input with models that take CCR and / or connectedness features as an input can significantly improves performance for all models. For instance, an individual AE model may lack the ability' to identify specific failed features. On the other hand, z-score and MD-based models provide output metrics related to statistical probability’, making them more interpretable. Thus as one example, an AE model may be used to detect outliers and a separate z-score model may be used identify feature outliers, post-hoc.

[0055] Referring now to FIG. 6, an example of a system 600 for automated review and quality assurance of radiotherapy plan contours (e.g., OAR contours) in accordance with some embodiments of the systems and methods described in the present disclosure is shown. As shown in FIG. 6, a computing device 650 can receive one or more types of data (e.g., radiotherapy plan contour data) from data source 602. In some embodiments, computing device 650 can execute at least a portion of a knowledge-based OAR contour quality assurance system 604 to evaluate OAR and other radiotherapy plan contours from data received from the data source 602.

[0056] Additionally or alternatively, in some embodiments, the computing device 650 can communicate information about data received from the data source 602 to a server 652 over a communication network 654, which can execute at least a portion of the knowledgebased OAR contour quality assurance system 604. In such embodiments, the server 652 can return information to the computing device 650 (and / or any other suitable computing device) indicative of an output of the know ledge-based OAR contour quality assurance system 604.

[0057] In some embodiments, computing device 650 and / or server 652 can be any suitable computing device or combination of devices, such as a desktop computer, a laptopcomputer, a smartphone, a tablet computer, a wearable computer, a server computer, a virtual machine being executed by a physical computing device, and so on. The computing device 650 and / or server 652 can also reconstruct images from the data.

[0058] In some embodiments, data source 602 can be any suitable source of data (e.g., radiotherapy plan data, measurement data, images reconstructed from measurement data, processed image data), such as a radiotherapy planning system, another computing device (e.g., a server storing radiotherapy plan contour data, measurement data, images reconstructed from measurement data, processed image data), and so on. In some embodiments, data source 602 can be local to computing device 650. For example, data source 602 can be incorporated with computing device 650 (e.g., computing device 650 can be configured as part of a device for measuring, recording, estimating, acquiring, or otherwise collecting or storing data). As another example, data source 602 can be connected to computing device 650 by a cable, a direct wireless link, and so on. Additionally or alternatively, in some embodiments, data source 602 can be located locally and / or remotely from computing device 650, and can communicate data to computing device 650 (and / or server 652) via a communication network (e.g., communication netw ork 654).

[0059] In some embodiments, communication network 654 can be any suitable communication network or combination of communication networks. For example, communication netw ork 654 can include a Wi-Fi network (which can include one or more wireless routers, one or more switches, etc.), a peer-to-peer network (e.g.. a Bluetooth network), a cellular network (e.g., a 3G network, a 4G network, etc., complying with any suitable standard, such as CDMA, GSM, LTE, LTE Advanced, WiMAX, etc.), other types of wireless netw ork, a wired netw ork, and so on. In some embodiments, communication network 654 can be a local area network, a wide area network, a public network (e.g., the Internet), a private or semi-private network (e.g., a corporate or university intranet), any other suitable type of netw ork, or any suitable combination of networks. Communications links show n in FIG. 6 can each be any suitable communications link or combination of communications links, such as wired links, fiber optic links, Wi-Fi links. Bluetooth links, cellular links, and so on.

[0060] Referring now to FIG. 7, an example of hardware 700 that can be used to implement data source 602, computing device 650, and server 652 in accordance with some embodiments of the systems and methods described in the present disclosure is shown.

[0061] As shown in FIG. 7, in some embodiments, computing device 650 can include a processor 702, a display 704, one or more inputs 706, one or more communication systems708, and / or memory 710. In some embodiments, processor 702 can be any suitable hardware processor or combination of processors, such as a central processing unit (“CPU”), a graphics processing unit (“GPU”), and so on. In some embodiments, display 704 can include any suitable display devices, such as a liquid crystal display (“LCD”) screen, a light-emitting diode (“LED”) display, an organic LED (“OLED”) display, an electrophoretic display (e.g., an “e- ink” display), a computer monitor, a touchscreen, a television, and so on. In some embodiments, inputs 706 can include any suitable input devices and / or sensors that can be used to receive user input, such as a keyboard, a mouse, a touchscreen, a microphone, and so on.

[0062] In some embodiments, communications systems 708 can include any suitable hardware, firmware, and / or software for communicating information over communication network 654 and / or any other suitable communication networks. For example, communications systems 708 can include one or more transceivers, one or more communication chips and / or chip sets, and so on. In a more particular example, communications systems 708 can include hardware, firmware, and / or software that can be used to establish a Wi-Fi connection, a Bluetooth connection, a cellular connection, an Ethernet connection, and so on.

[0063] In some embodiments, memory 710 can include any suitable storage device or devices that can be used to store instructions, values, data, or the like, that can be used, for example, by processor 702 to present content using display 704, to communicate with server 652 via communications system(s) 708, and so on. Memory 710 can include any suitable volatile memory, non-volatile memory, storage, or any suitable combination thereof. For example, memory 710 can include random-access memory (“RAM”), read-only memory (“ROM”), electrically programmable ROM (“EPROM”), electrically erasable ROM (“EEPROM”), other forms of volatile memory, other forms of non-volatile memory, one or more forms of semi-volatile memory, one or more flash drives, one or more hard disks, one or more solid state drives, one or more optical drives, and so on. In some embodiments, memory 710 can have encoded thereon, or otherwise stored therein, a computer program for controlling operation of computing device 650. In such embodiments, processor 702 can execute at least a portion of the computer program to present content (e.g., images, user interfaces, graphics, tables), receive content from server 652, transmit information to server 652, and so on. For example, the processor 702 and the memory 710 can be configured to perform the methods described herein (e.g., the method of FIG. 1, the method of FIG. 5).

[0064] In some embodiments, server 652 can include a processor 712. a display 714, one or more inputs 716. one or more communications systems 718. and / or memory 720. Insome embodiments, processor 712 can be any suitable hardware processor or combination of processors, such as a CPU, a GPU, and so on. In some embodiments, display 714 can include any suitable display devices, such as an LCD screen, LED display, OLED display, electrophoretic display, a computer monitor, a touchscreen, a television, and so on. In some embodiments, inputs 716 can include any suitable input devices and / or sensors that can be used to receive user input, such as a keyboard, a mouse, a touchscreen, a microphone, and so on.

[0065] In some embodiments, communications systems 718 can include any suitable hardware, firmware, and / or software for communicating information over communication network 654 and / or any other suitable communication networks. For example, communications systems 718 can include one or more transceivers, one or more communication chips and / or chip sets, and so on. In a more particular example, communications systems 718 can include hardware, firmware, and / or software that can be used to establish a Wi-Fi connection, a Bluetooth connection, a cellular connection, an Ethernet connection, and so on.

[0066] In some embodiments, memory' 720 can include any suitable storage device or devices that can be used to store instructions, values, data, or the like, that can be used, for example, by processor 712 to present content using display 714, to communicate with one or more computing devices 650, and so on. Memory 720 can include any suitable volatile memory7, non-volatile memory', storage, or any suitable combination thereof. For example, memory 720 can include RAM, ROM, EPROM, EEPROM, other ty pes of volatile memory, other ty pes of non-volatile memory, one or more types of semi-volatile memory, one or more flash drives, one or more hard disks, one or more solid state drives, one or more optical drives, and so on. In some embodiments, memory' 720 can have encoded thereon a server program for controlling operation of server 652. In such embodiments, processor 712 can execute at least a portion of the server program to transmit information and / or content (e.g., data, images, a user interface) to one or more computing devices 650, receive information and / or content from one or more computing devices 650, receive instructions from one or more devices (e.g., a personal computer, a laptop computer, a tablet computer, a smartphone), and so on.

[0067] In some embodiments, the server 652 is configured to perform the methods described in the present disclosure. For example, the processor 712 and memory 720 can be configured to perform the methods described herein (e.g., the method of FIG. 1, the method of FIG. 5).

[0068] In some embodiments, data source 602 can include a processor 722, one or more data acquisition systems 724, one or more communications systems 726. and / or memory 728.In some embodiments, processor 722 can be any suitable hardware processor or combination of processors, such as a CPU, a GPU. and so on. In some embodiments, the one or more data acquisition systems 724 are generally configured to acquire radiotherapy plan contour data, image data, images, or both, and can include a radiotherapy planning system, a medical imaging system, etc. Additionally or alternatively, in some embodiments, the one or more data acquisition systems 724 can include any suitable hardware, firmware, and / or software for coupling to and / or controlling operations of a radiotherapy planning system, a medical imaging system, and the like. In some embodiments, one or more portions of the data acquisition system(s) 724 can be removable and / or replaceable.

[0069] Note that, although not shown, data source 602 can include any suitable inputs and / or outputs. For example, data source 602 can include input devices and / or sensors that can be used to receive user input, such as a keyboard, a mouse, a touchscreen, a microphone, a trackpad, a trackball, and so on. As another example, data source 602 can include any suitable display devices, such as an LCD screen, an LED display, an OLED display, an electrophoretic display, a computer monitor, a touchscreen, a television, etc., one or more speakers, and so on.

[0070] In some embodiments, communications systems 726 can include any suitable hardware, firmware, and / or software for communicating information to computing device 650 (and, in some embodiments, over communication network 654 and / or any other suitable communication networks). For example, communications systems 726 can include one or more transceivers, one or more communication chips and / or chip sets, and so on. In a more particular example, communications systems 726 can include hardware, firmware, and / or software that can be used to establish a wired connection using any suitable port and / or communication standard (e.g., VGA, DVI video, USB, RS-232, etc ), Wi-Fi connection, a Bluetooth connection, a cellular connection, an Ethernet connection, and so on.

[0071] In some embodiments, memory 728 can include any suitable storage device or devices that can be used to store instructions, values, data, or the like, that can be used, for example, by processor 722 to control the one or more data acquisition systems 724, and / or receive data from the one or more data acquisition systems 724; to generate images from data; present content (e.g., data, images, a user interface) using a display; communicate with one or more computing devices 650; and so on. Memory 728 can include any suitable volatile memory, non-volatile memory', storage, or any suitable combination thereof. For example, memory 728 can include RAM, ROM, EPROM, EEPROM, other ty pes of volatile memory, other types of non-volatile memory, one or more types of semi-volatile memory, one or moreflash drives, one or more hard disks, one or more solid state drives, one or more optical drives, and so on. In some embodiments, memory 728 can have encoded thereon, or otherwise stored therein, a program for controlling operation of data source 602. In such embodiments, processor 722 can execute at least a portion of the program to generate images, transmit information and / or content (e.g., data, images, a user interface) to one or more computing devices 650, receive information and / or content from one or more computing devices 650, receive instructions from one or more devices (e.g., a personal computer, a laptop computer, a tablet computer, a smartphone, etc.), and so on.

[0072] In some embodiments, any suitable computer-readable media can be used for storing instructions for performing the functions and / or processes described herein. For example, in some embodiments, computer-readable media can be transitory or non-transitory. For example, non-transitory computer-readable media can include media such as magnetic media (e.g., hard disks, floppy disks), optical media (e.g., compact discs, digital video discs, Blu-ray discs), semiconductor media (e.g., RAM, flash memory', EPROM, EEPROM), any suitable media that is not fleeting or devoid of any semblance of permanence during transmission, and / or any suitable tangible media. As another example, transitory computer- readable media can include signals on networks, in wires, conductors, optical fibers, circuits, or any suitable media that is fleeting and devoid of any semblance of permanence during transmission, and / or any suitable intangible media.

[0073] As used herein in the context of computer implementation, unless otherwise specified or limited, the terms “component,” “system,” “module,” “framework,” and the like are intended to encompass part or all of computer-related systems that include hardware, software, a combination of hardware and software, or software in execution. For example, a component may be, but is not limited to being, a processor device, a process being executed (or executable) by a processor device, an object, an executable, a thread of execution, a computer program, or a computer. By way of illustration, both an application running on a computer and the computer can be a component. One or more components (or system, module, and so on) may reside within a process or thread of execution, may be localized on one computer, may be distnbuted between two or more computers or other processor devices, or may be included within another component (or system, module, and so on).

[0074] In some implementations, devices or sy stems disclosed herein can be utilized or installed using methods embodying aspects of the disclosure. Correspondingly, description herein of particular features, capabilities, or intended purposes of a device or system isgenerally intended to inherently include disclosure of a method of using such features for the intended purposes, a method of implementing such capabilities, and a method of installing disclosed (or otherwise known) components to support these purposes or capabilities. Similarly, unless otherwise indicated or limited, discussion herein of any method of manufacturing or using a particular device or system, including installing the device or system, is intended to inherently include disclosure, as embodiments of the disclosure, of the utilized features and implemented capabilities of such device or system.

[0075] The present disclosure has described one or more preferred embodiments, and it should be appreciated that many equivalents, alternatives, variations, and modifications, aside from those expressly stated, are possible and within the scope of the invention.

Claims

CLAIMS1. A method for knowledge-based quality assurance of organ-at-risk contours in a radiotherapy plan, the method comprising:(a) accessing radiotherapy plan contour data with a computer system, wherein the radiotherapy plan contour data comprise at least organ-at-risk (OAR) contours;(b) generating contour-based feature data from the radiotherapy plan contour data using the computer system;(c) accessing a computational model with the computer system, wherein the computational model is constructed to receive contour-based feature data as an input to predict errors in OAR contours;(d) inputting the contour-based feature data to the computational model with the computer system, generating contour error data as an output; and(e) outputting the contour error data via the computer system.

2. The method of claim 1. wherein the contour-based feature data are generated using a contour model.

3. The method of claim 1. wherein the contour-based feature data comprise single-contour features obtained from individual contours in the radiotherapy plan contour data.

4. The method of claim 3, wherein the single-contour features comprise at least one of contour shape, contour volume, contour location, contour orientation, or computed tomography (CT) number.

5. The method of claim 4, wherein the single-contour features comprise contour locations generated by computing a difference between a contour centroid and a centroid of an anatomical landmark.

6. The method of claim 4, wherein the single-contour features comprise contour locations generated by at least one of rotating, translating, or scaling the individual contoursin the radiotherapy plan contour data to overlap with training contours in a given coordinate system.

7. The method of claim 6, wherein the contour locations generated by at least one of rotating, translating, or scaling the individual contours in the radiotherapy plan contour data using a Procrustes analysis.

8. The method of claim 4, wherein the single-contour features are generated based on at least one of eigenvalues or eigenvectors of principal components generated by a principal component analysis of the radiotherapy plan contour data.

9. The method of claim 8, wherein the single-contour features comprise contour shapes generated as a ratio between eigenvalues of the principal components.

10. The method of claim 8. wherein the single-contour features comprise contour orientations generated as eigenvectors of the principal components.

11. The method of claim 1 , wherein the contour-based feature data comprise contour-to-contour relationship (CCR) features obtained between contours in the radiotherapy plan contour data.

12. The method of claim 11, wherein the CCR features are indicative of at least one of overlap between contours, abutment of contours, or separation between contours.

13. The method of claim 11, wherein the CCR features comprise at least one of a minimum distance between contours or a fractional volume of overlap between contours.

14. The method of claim 1. wherein the contour-based feature data comprise connectedness features indicative of a number of disconnected parts of contours in the radiotherapy plan contour data.

15. The method of claim 1. comprising generating updated radiotherapy plan contour data with the computer system using the contour error data to correct OAR contours in the radiotherapy plan contour data.

16. A method for knowledge-based quality assurance of contours in medical image data, the method comprising:(a) accessing medical image contour data with a computer system, wherein the medical image contour data comprise at least contours generated based on medical images;(b) generating contour-based feature data from the medical image contour data using the computer system;(c) accessing a computational model with the computer system, wherein the computational model is constructed to receive contour-based feature data as an input to predict errors in medical image contours;(d) inputting the contour-based feature data to the computational model with the computer system, generating contour error data as an output; and(e) outputting the contour error data to a user via the computer system.

17. The method of claim 16, wherein the contour-based feature data are generated using a contour model.

18. The method of claim 16, wherein the contour-based feature data comprise single-contour features obtained from individual contours in the medical image contour data.

19. The method of claim 18, wherein the single-contour features comprise at least one of contour shape, contour volume, contour location, or contour orientation.

20. The method of claim 19, wherein the single-contour features comprise contour locations generated by computing a difference between a contour centroid and a centroid of an anatomical landmark.

21. The method of claim 19, wherein the single-contour features comprise contour locations generated by at least one of rotating, translating, or scaling the individual contoursin the radiotherapy plan contour data to overlap with training contours in a given coordinate system.

22. The method of claim 21, wherein the contour locations generated by at least one of rotating, translating, or scaling the individual contours in the radiotherapy plan contour data using a Procrustes analysis.

23. The method of claim 19, wherein the single-contour features are generated based on at least one of eigenvalues or eigenvectors of principal components generated by a principal component analysis of the radiotherapy plan contour data.

24. The method of claim 23, wherein the single-contour features comprise contour shapes generated as a ratio between eigenvalues of the principal components.

25. The method of claim 23, wherein the single-contour features comprise contour orientations generated as eigenvectors of the principal components.

26. The method of claim 16, wherein the contour-based feature data comprise contour-to-contour relationship (CCR) features obtained between contours in the medical image contour data.

27. The method of claim 26, wherein the CCR features are indicative of at least one of overlap between contours, abutment of contours, or separation between contours.

28. The method of claim 26, wherein the CCR features comprise at least one of a minimum distance between contours or a fractional volume of overlap between contours.

29. The method of claim 16, wherein the contour-based feature data comprise connectedness features indicative of a number of disconnected parts of contours in the medical image contour data.

30. The method of claim 1 or 16. wherein the computational model comprises a statistical model.

31. The method of claim 30, wherein the statistical model comprises a z-score model.

32. The method of claim 1 or 16, wherein the computational model is based on a distance metric computed between the contour-based feature data and a training dataset.

33. The method of claim 32, wherein the distance metric is a Mahalanobis distance.

34. The method of claim 1 or 16, wherein the computational model is comprises a machine learning model.

35. The method of claim 34, wherein the machine learning model comprises a neural network.

36. The method of claim 35, wherein the neural network comprises an autoencoder network.

37. The method of claim 1 or 16, wherein a plurality of computational models is accessed by the computer system, the contour-based feature data are input to the plurality of computational models to generate a corresponding plurality of contour error data, and presenting the contour error data to the user comprises selectively combining the plurality of contour error data generated by the plurality' of computational models.

Images