Quality Assurance Process for Radiation Therapy Treatment Planning

Abstract

A computer-implemented method enables testing and evaluation of an expert human reviewer or an artificial intelligence (AI) error detection engine associated with a radiation therapy treatment planning process. Intentional errors are introduced 303 into the output of a software module or AI engine that performs a step in the radiation therapy treatment planning process. The efficacy of the human or AI reviewer in detecting the errors can then be evaluated or tested by determining 306 whether the human or AI reviewer detects the introduced errors.

Description

Background Art

[0001] Unless otherwise indicated herein, the approaches described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.

[0002] Radiation therapy is a localized treatment directed to a specific target tissue (planned target volume) such as a cancerous tumor. Ideally, radiation therapy is delivered to the planned target volume, which avoids surrounding normal tissue from receiving a dose exceeding a specified tolerance, thereby minimizing the risk of damage to healthy tissue. Prior to the delivery of radiation therapy, an imaging system is typically employed to provide a three-dimensional image of the target tissue and surrounding area. From such imaging, the size and mass of the target tissue can be estimated, and an appropriate treatment plan can be generated and the planned target volume determined.

[0003] Due to the geometric complexity of applying radiation therapy to a three-dimensional target that may be in close proximity to one or more organs at risk, treatment planning is a complex process that may involve the participation of multiple highly trained medical professionals and analysis using sophisticated software. Thus, the generation of a treatment plan that properly doses the total tumor volume without overdosing nearby organs at risk is a multi-step process, each of which can be very time consuming and is typically only performed by expert personnel. Furthermore, because many steps in the treatment planning process can include subjective assessments and judgment calls, the details of a particular treatment plan can vary depending on the specific personnel involved in the treatment planning process. Summary of the invention

[0004] In one aspect, the invention provides a computer implemented method for quality assurance testing of a treatment planning process as defined in claim 1. Optional features are defined in the dependent claims.

[0005] A computer-implemented method for quality assurance testing of a treatment planning process may include: automatically generating output data for a module of a treatment process, wherein the output data is based on input data for the module and includes at least one error; automatically submitting the output data to a reviewer for error checking; automatically receiving an error check response associated with the reviewer; and when the error check response indicates that at least one error failed to be detected, automatically executing a corrective action associated with the reviewer.

[0006] In another aspect, the invention provides a non-transitory computer readable storage medium comprising instructions as defined in claim 18. Optional features are defined in the dependent claims.

[0007] In accordance with at least some embodiments of the present disclosure, a method for quality assurance (QA) testing of a radiation therapy planning process enables testing and evaluation of an expert human reviewer or an artificial intelligence (AI) error detection engine. Specifically, intentional errors can be introduced into the output of a software module or AI engine, hereinafter referred to as a "review module," which performs a certain step in the radiation therapy planning process. The effectiveness of the human or AI reviewer in detecting errors can then be evaluated or tested by determining whether the human or AI reviewer detects the introduced errors. In some embodiments, such a determination is made based on an error check response from the reviewer, such as an error detection output from an AI engine reviewer or a user input made by a human reviewer.

[0008] The foregoing summary is illustrative only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features will become apparent by reference to the drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The foregoing and other features of the present disclosure will become more apparent from the following description and appended claims, taken in conjunction with the accompanying drawings. These drawings depict only a few embodiments according to the present disclosure and therefore should not be considered as limiting the scope thereof. The present disclosure will be described with additional specificity and detail through the use of the accompanying drawings.

[0010] Figure 1 is a block diagram illustrating an example treatment planning process 100 .

[0011] Figure 2 is a block diagram illustrating a quality assurance process configured to implement one or more embodiments of the present disclosure.

[0012] Figure 3 A flow chart illustrating an example quality assurance process for a treatment planning process according to one or more embodiments of the present disclosure

[0013] Figure 4 is a block diagram illustrating a training process 400 for training a machine learning process 401 according to various embodiments of the present disclosure.

[0014] Figure 5 is an illustration of a computing device configured to perform various embodiments of the present disclosure.

[0015] Figure 6 is a block diagram of an illustrative embodiment of a computer program product 600 for implementing a method for segmenting an image according to one or more embodiments of the present disclosure. DETAILED DESCRIPTION

[0016] In the following detailed description, reference is made to the accompanying drawings, which form a part thereof. In the drawings, similar symbols generally identify similar components unless the context indicates otherwise. The illustrative embodiments described in the detailed description, drawings, and claims are not meant to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein. It will be readily understood that aspects of the present disclosure, as generally described herein and illustrated in the accompanying drawings, may be arranged, substituted, combined, and designed in a variety of different configurations, all of which are expressly contemplated and constitute part of the present disclosure.

[0017] As described above, the process of developing a treatment plan for a particular patient is a multi-step process that may include analysis performed by both expert humans and sophisticated software applications. Figure 1 An exemplary treatment planning process is shown.

[0018] Figure 1 1 is a block diagram illustrating an example treatment planning process 100. The treatment planning process 100 includes a plurality of steps 101-110 that are performed to generate a treatment plan for a particular patient. The treatment planning process 100 is performed in response to a diagnosis of the patient that indicates that the patient will be treated via external beam radiation therapy. The diagnosis typically indicates external beam radiation therapy (rather than internal radiation therapy) based on a variety of factors, including: the type of cancer tumor detected, the size of the detected tumor, the location of the tumor in the body, the proximity of the tumor to organs at risk (OARs) or other normal tissues that are sensitive to radiation, the patient's general health and medical history, the presence of other types of cancer in the patient's body, the patient's age, certain medical conditions of the patient, and the like.

[0019] As shown, the treatment planning process 100 includes one or more of the following steps: a treatment determination step 101, a patient consultation step 102, a treatment plan computerized tomography (CT) step 103, a treatment plan indication step 104, a target segmentation step 105, an OAR segmentation step 106, a plan optimization step 107, a quantitative plan quality assurance (QA) step 108, a plan review and approval step 109, and a plan delivery QA step 110. As described in more detail below, some of the steps 101-110 are performed by appropriately trained medical professionals, some of the steps 101-110 are completed by appropriately trained medical professionals performing certain software-based analyses, and in some cases, one or more of the steps 101-110 may be performed entirely by an appropriately designed software application.

[0020] In treatment determination step 101, patient imaging, tumor pathology, and patient diagnosis are reviewed and one or more possible treatment plans or prescriptions for radiation therapy are determined. In some cases, step 101 is performed by a radiation oncologist, who may be employed at the clinical site where the radiation therapy will ultimately be performed. In some cases, the radiation oncologist may be assisted by a software application that is configured to generate possible treatment plans based on the patient's diagnosis and imaging information. In this case, the radiation oncologist selects and / or modifies one or more treatment plans provided by the software application.

[0021] In a patient consultation step 102, the radiation oncologist consults with the patient about potential treatment options. As noted in the treatment determination step 101, in some cases, the radiation oncologist may select and / or modify one or more potential treatment plans from one or more treatment plans initially proposed by the software application.

[0022] In the treatment planning CT step 103, a treatment planning CT displaying the tumor and the anatomical structure region surrounding the tumor is specified and performed. There is generally an inherent trade-off between the image quality / noise level in the treatment planning CT and the radiation dose received by the patient during the treatment planning CT. Therefore, in some cases, in step 103, certain parameters of the treatment planning CT scan are manipulated to optimize safer imaging of the patient while maintaining sufficient diagnostic image quality, such as X-ray detector configuration, tube current, tube potential, what reconstruction algorithm is used, patient positioning, scan range, thickness of reconstructed slices, etc.

[0023] In some cases, the radiation oncologist specifies some or all parameters of the treatment planning CT scan. Alternatively, in some cases, the radiation oncologist may be assisted by a software application configured to specify potential parameters of the treatment planning CT. For example, such software may specify parameters of the treatment planning CT based on the diagnosis, imaging information previously acquired for the patient, and the specific treatment plan selected in step 102. The treatment planning CT is then generated, for example, by scanning the patient during a clinical visit.

[0024] In the treatment plan instructions step 104, a treatment plan instruction is generated based on the treatment plan CT. The treatment plan instruction generally describes an image study of the treatment site, including target tissue structures and normal tissue structures defined via the imaging study. These targets and normal tissue structures are then used for the treatment plan. In order to score multiple treatment plans during the optimization process, the treatment plan instruction may also specify an extension of the target tissue structure and the normal tissue structure. Therefore, in addition to the gross tumor volume (GTV), the treatment plan instruction may also include a clinical target volume (CTV), an internal target volume (ITV), a planned target volume (PTV), an OAR and / or a planned organ volume at risk (PRV), etc. The treatment plan instruction may further specify radiation therapy prescription guidelines, planning recommendations and / or special orders.

[0025] In some cases, the radiation oncologist generates some or all of the treatment planning instructions, e.g., based on local clinical standards, the patient's specific medical conditions, etc. Alternatively, in some cases, the radiation oncologist may be assisted by a software application configured to suggest some or all of the information included in the treatment planning instructions.

[0026] In the target segmentation step 105, the target of the planned radiation therapy is depicted based on the treatment plan CT and the information included in the treatment plan instructions. For example, one or more of GTV, CTV, ITV and / or PTV are depicted on a two-dimensional slice of the reconstructed volume imaged in the planned treatment CT. In some cases, the radiation oncologist manually performs one or more of these depictions via a display screen. Typically, the radiation oncologist depicts various target volumes based on visual cues in each two-dimensional slice of the reconstructed volume and personal training and experience. Alternatively, in some cases, a software application configured to automatically generate a segmentation of GTV, PTV, etc. can help the radiation oncologist segment one or more target areas (GTV, CTV, ITV, PTV, etc.). Due to the algorithmic complexity of the segmentation process, such software applications may include one or more machine learning models that are trained to approximate human logic or decision-making when searching for hidden structures, patterns or features in two-dimensional slices. In this case, the radiation oncologist can then view and / or modify this automatically generated segmentation. In some cases, multiple segmentations for a specific target area can be generated by a software application, and the radiation oncologist selects the most appropriate segmentation.

[0027] In the OAR segmentation step 106, the OARs defined in the treatment plan instructions are depicted (i.e., segmented). In some cases, a radiation oncologist or dosimetrist manually performs one or more of the OAR segmentations via a display screen. Alternatively, in some cases, an automatic segmentation software application configured to automatically segment one or more of the defined OARs can generate some or all of the OAR segmentations, and the radiation oncologist can view and / or modify such segmentations. In this case, the automatic segmentation software application can generate multiple segmentations for a particular OAR based on various factors, including, for example, user input, information included in the treatment plan instructions (such as clinical margins), and / or other factors. As described above, due to the algorithmic complexity of the segmentation process, such software applications may include one or more machine learning models.

[0028] In the plan optimization step 107, one or more plans for the planned treatment are optimized. That is, one or more beam geometries for implementing the planned treatment are determined and the dose distribution of each beam geometry is optimized. In some cases, a qualified dosimetrist using a software application determines one or more beam geometries and optimizes the associated dose distribution, and then the doctor in charge of the patient usually approves one or more plans. In some cases, the software application is approved by the Food and Drug Administration (FDA) for such use. Alternatively, in some cases, a software application including one or more machine learning models can generate one or more beam geometries and associated optimized dose distributions in the plan optimization step 107, and the dosimetrist can select and / or modify and / or review the output of the software application. Typically, in this case, the plan generated by the FDA-approved software application still requires doctor approval. Alternatively, in some cases, certain machine learning software applications can generate plans including beam geometries and associated optimized dose distributions without the need for dosimetrist review. However, in this case, the approval of the responsible doctor is still generally required.

[0029] In the quantitative plan QA step 108, the one or more plans optimized in step 107 are evaluated against specified thresholds or other metrics to determine whether each plan meets the clinical goals specified in the treatment plan instructions. For example, the thresholds may include institution-defined thresholds, physician-defined thresholds, thresholds included in the treatment plan instructions, dose thresholds established for a particular patient, etc. Each plan evaluated in the quantitative plan QA step 108 is compared to the above-mentioned quantitative thresholds. In some cases, the physician responsible for the patient evaluates each plan approved in the plan optimization step 107. In other cases, a machine learning-based software application can be used to detect and mark plans that exceed one or more of the specified thresholds. In this case, the physician responsible for the patient only reviews the marked plans.

[0030] In the plan review and approval step 109, the physician responsible for the patient reviews the available plans and selects the best plan for implementing the treatment specified in the treatment plan instructions. In some cases, the physician may request a change in the selected plan, in which case the GTV segmentation, OAR segmentation step 106 and / or plan optimization step 107 are performed again.

[0031] In the plan delivery QA step 110, the plan reviewed and selected in the plan review and approval step 109 is delivered on a suitable radiation therapy system and the predicted dose is measured. The patient is then treated with the selected plan.

[0032] like Figure 1 As shown, radiotherapy planning is a complex process that may involve the participation of multiple trained medical professionals and the analysis of complex software. In order to reduce the cost of radiotherapy planning and improve the uniformity of treatment plans, considerable efforts have been made to apply machine learning to the process of radiotherapy planning. Specifically, appropriately trained artificial intelligence (AI) engines, neural networks, or other machine learning software applications can be used to automatically perform certain radiotherapy planning tasks that currently require many hours of manpower to complete. For example, automatic segmentation algorithms have been developed for depicting OARs and other structures on CT, positron emission tomography (PET), MRI images, or images generated with other imaging modes for certain areas of the patient's anatomical structure. As a result, OARs can be segmented without the time-consuming task of manually depicting one or more OARs by multiple two-dimensional slices across the reconstruction volume by radiation oncologists, dosimetrists, or other doctors. In addition, in addition to the automatic segmentation of OARs, efforts are also made to automate other steps in the treatment planning process 100.

[0033] However, when fully automated steps are incorporated into the radiation therapy treatment planning process 100, there is a danger that even a robustly trained AI engine or neural network may not perform the step correctly for 100% of patients. This may be caused by outliers in the patient population, weaknesses in the AI ​​engine, or other factors. Although it is possible to create automated tools that flag potential abnormal cases for manual review, given the high-risk nature of radiation therapy, especially hypofractionated radiation therapy, patient plans will require final review by experts for the foreseeable future. This review can be comprehensive and include segmentation, beam trajectory, three-dimensional dose, etc. As a particular AI engine becomes more robust in performing the steps of the radiation therapy planning process 100, detectable errors become relatively rare. Therefore, there is a risk that the expert human reviewer responsible for checking the output of the AI ​​engine may become complacent and fail to detect infrequent or subtle errors.

[0034] According to various embodiments, a method for quality assurance (QA) testing of a radiation therapy planning process enables testing and evaluating an expert human reviewer or an Al error detection engine. Specifically, intentional errors are introduced into the output of a software module or AI engine, hereinafter referred to as a "review module," which performs a certain step in the radiation therapy planning process. The effectiveness of the human or AI reviewer in detecting errors can then be evaluated or tested by determining whether the human or AI reviewer has detected the introduced errors. In some embodiments, such a determination is made based on an error check response from the reviewer, such as an error detection output from an AI engine reviewer or a user input made by a human reviewer.

[0035] Alternatively, according to some embodiments, the methods described herein for QA testing of the radiation therapy treatment planning process can be applied to end-to-end inspections rather than to the review of a single treatment planning module. Therefore, in such embodiments, the QA testing methods described herein can be used to check the errors in the doctor's inspection of the optimized plan that has undergone the complete treatment planning process 100. In addition, in some embodiments, the software-based radiation therapy treatment planning process can generate an optimized radiation therapy plan based on the patient CT, wherein the software-based radiation therapy treatment planning process generates the optimized plan in a single step rather than in the multiple steps 101-110 shown in the treatment planning process 100. In such embodiments, the doctor's inspection of errors in such an optimized plan can be checked by the QA testing methods described herein.

[0036] In some embodiments, the magnitude, significance, and clinical severity of the introduced errors may vary as appropriate. For example, in some embodiments, the magnitude, significance, and / or clinical severity of the introduced errors may vary based on the current reviewer rating, certain clinical criteria specific to the current step of the treatment planning process 100, etc. In some embodiments, the QA process is configured to implement QA testing of one or more steps of the radiation therapy treatment planning process 100. Figure 2 One such embodiment is described.

[0037] Figure 2is a block diagram illustrating a QA system 200 configured to implement one or more embodiments of the present disclosure. The QA system 200 may be implemented as a digital tool or other software application that runs on one or more computing devices that are associated with the execution of one or more steps 101-110 of the treatment planning process 100. For example, in some embodiments, the QA system 200 may include a software application running on a computing device that is configured to perform one or more of the automated target segmentation step 105, the automated OAR segmentation step 106, the dose optimization step 107, etc. Alternatively or additionally, the QA system 200 may include a software application running on a computing device that is configured to facilitate an expert human reviewer to perform a review of one or more automated target segmentation steps 105, automated OAR segmentation steps 106, the review step of dose optimization 107, etc. Alternatively or additionally, the QA system 200 may include a software application running on a computing device that is configured to perform error checking or other automated review of one or more steps of the treatment planning process 100, such as via a suitably trained AI engine. Thus, the QA system 200 can be implemented in a clinical setting or as part of a software-as-a-service platform that can interface with multiple different clinical sites. Figure 2 In the illustrated embodiment, the QA system 200 includes, but is not limited to, one or more error introduction modules 201 , an auditor monitoring module 202 , a performance database 203 , an error database 204 , and a planning module 205 .

[0038] The QA system 200 may include one or more error introduction modules 201. Each error introduction module 201 is configured to generate output data for a different one or more of the steps 101-110 of the treatment planning process 100, wherein the output data includes at least one error. That is, for a specific review module that performs a step of the treatment planning process 100, the corresponding error introduction module 201 is configured to generate output data including at least one error for the specific review module.

[0039] In some embodiments, the error introduction module 201 is configured to modify the output data generated by the corresponding review module, for example, by changing one or more specific portions of the output data and / or by adding error information to the output data. Thus, in such embodiments, the error introduction module 201 introduces one or more intentional errors into the output of a particular review module that performs a step of the treatment planning process 100.

[0040] For example, in an embodiment in which an automatic segmentation process is performed in the target segmentation step 105, one of the error introduction modules 201 can be configured to introduce one or more segmentation errors into the output of the automatic segmentation process. In such an embodiment, the output of the automatic segmentation process may include a segmentation of one or more of the GTV, CTV, ITV and / or PTV. Therefore, in this embodiment, the error introduction module 201 is configured to sufficiently modify or change one or more such segmentations so that one or more clinical boundaries associated with the (multiple) segmentations are exceeded. For example, in one or more two-dimensional slices of the patient's anatomy, the boundaries of the GTV segmented by the error introduction module 201 may erroneously cover too little actual tumor tissue, too much non-tumor tissue, or a combination of both. As a result, radiotherapy based on one or more modified segmentations may result in a clinically significant reduction in the efficacy of the radiotherapy or associated side effects to the patient. Therefore, the modifications introduced to the (multiple) segmentations are of sufficient magnitude that they should be detected by the reviewer rather than being passed to the next step in the treatment planning process 100 (e.g., the OAR segmentation step 106).

[0041] The one or more segmentation errors or modifications introduced into the output segmentation of the target segmentation step 105 may include a translation, rotation, and / or other offset from the original position of the output segmentation. Alternatively or additionally, the one or more segmentation errors or modifications may include distortion of some or all of the output segmentation (e.g., compression or expansion), addition and / or subtraction of specific segmentation regions in the output segmentation, etc. Alternatively or additionally, the one or more introduced segmentation errors or modifications may be implemented by any other technically feasible program or algorithmic process, such as an AI process or a machine learning process. In an alternative embodiment, a generative adversarial network (GAN) may be used to generate the error, as shown below in combination with Figure 4 described.

[0042] In some embodiments, the one or more segmentation errors or modifications introduced by the error introduction module 201 may be selected from an error database 204, which includes a plurality of previously generated errors for one or more of steps 101-110. Such previously generated errors may include errors associated with specific steps of the treatment planning process 100. Furthermore, such previously generated errors may include errors that are customized for specific regions of the patient's anatomy, specific scan parameters, specific X-ray detector configurations, specific reconstruction algorithms used for the original CT scan, etc. Thus, the previously generated errors selected by the error introduction module 201 from the error database 204 generally correspond to errors that are generally associated with the specific treatment planning scenario in which the error introduction module 201 is being employed. For example, in an embodiment of the radiation therapy treatment planning process 100 for a pancreatic tumor, the one or more previously generated segmentation errors introduced by 201 after the target segmentation step 105 may be selected from segmentation errors that are generally associated with automatic segmentation of the pancreas.

[0043] In some embodiments, the error introduction module 201 is configured to generate output data including one or more intentional errors from a particular step of the treatment planning process 100. In such embodiments, the error introduction module 201 generates output data in place of a corresponding review module, wherein the generated output is based on normal input data of the review module. Thus, when a reviewer of a review module of the treatment planning process 100 is to be tested, the error introduction module 201 generates output data for a particular step of the treatment planning process 100, rather than the review module that normally generates output data for that step of the treatment planning process 100. In addition, the one or more intentional errors included in the output data generated by the error introduction module 201 cause one or more clinical limits associated with the output data to be exceeded. As a result, a radiation therapy based on the output data generated by the error introduction module 201 may result in a clinically significant reduction in the efficacy of the proposed radiation therapy treatment or an increase in patient side effects.

[0044] For example, in an embodiment in which an automatic segmentation process is performed in the target segmentation step 105, one of the error introduction modules 201 can be configured to perform the automatic segmentation process when the reviewer of the output of the target segmentation step 105 is to be tested. Similar to the review module that generally generates output for the target automatic segmentation, the error introduction module 201 generates a specific output (e.g., one or more of the GTV, CTV, ITV and / or PTV) based on the normal input to the review module (e.g., a CT scan of the patient's anatomical structure area). However, unlike the review module that normally generates output for the target automatic segmentation, the error introduction module 201 generates an output including one or more intentional errors, such as segmentation errors. In one such embodiment, the error introduction module 201 may include one or more AI engines or other machine learning models trained to perform automatic segmentation of CT images. Such an AI engine or machine learning model is further trained to generate output data including one or more errors that are specific to the current step of the treatment planning process 100. In addition, the one or more errors are configured to have sufficient magnitude and / or be located so that one or more clinical boundaries associated with the output of the target segmentation step 105 are exceeded. For example, the boundary between the GTV and the CTV may be smaller than the range defined in the treatment planning instructions associated with the current treatment planning process. In addition, in some embodiments, the AI ​​engine or machine learning model is trained to generate errors that may be more difficult for the reviewer to detect. Figure 4 A method of training such an AI engine according to an embodiment is described.

[0045] The reviewer monitoring module 202 is configured to determine whether an error introduction module 201 is employed after each of the steps 101-110 of the treatment planning process 100. That is, the reviewer monitoring module 202 determines whether a particular reviewer responsible for reviewing a module associated with that particular step of the treatment planning process 100 is qualified for a QA review. When the reviewer monitoring module 202 determines that a reviewer is qualified for a QA review, then the appropriate error introduction module 201 is employed to generate output data for that particular step of the treatment planning process 100, wherein the output data includes at least one error. In this manner, the reviewer monitoring module 202 ensures that each individual reviewer is evaluated or tested at a minimum and maximum frequency. In some embodiments, the evaluation or testing frequency of a particular reviewer may be automatically adjusted based on the performance of the reviewer during a previous evaluation or test.

[0046] In some embodiments, the reviewer monitoring module 202 determines the reviewer's eligibility for a QA inspection based on the time interval since a particular reviewer has undergone a QA inspection. Therefore, in such embodiments, when it is determined that the time interval has been exceeded, it is determined that the reviewer is eligible for a QA inspection. Alternatively or additionally, in some embodiments, the reviewer monitoring module 202 determines such eligibility based on the number of times the reviewer has reviewed the review module since the previous QA inspection. Therefore, in such embodiments, when the number exceeds a predetermined threshold, it is determined that the reviewer is eligible for a QA inspection. Alternatively or additionally, in some embodiments, the reviewer monitoring module 202 determines such eligibility based on the reviewer's current credibility score, which can be stored in the performance database 203. Therefore, in such embodiments, when the credibility score is less than a predetermined threshold value, it is determined that the reviewer is eligible or more likely to be eligible for a QA inspection. Alternatively or additionally, in some embodiments, the reviewer monitoring module 202 determines such eligibility based on a random determination process or any other technically feasible determination process. Therefore, in such embodiments, it can be randomly determined that the reviewer is eligible for a QA inspection.

[0047] The performance database 203 includes a confidence score or other quantitative measure of the past performance of one or more reviewers. As described above, the reviewer can be a professional human user or a suitably trained machine learning entity, such as an AI engine or a neural network. In some embodiments, the reviewer for whom the performance database 203 stores confidence scores is a radiation oncologist who reviews and / or modifies the automatically generated segmentation in the target segmentation step 105. In some embodiments, such a reviewer is a radiation oncologist who reviews and / or modifies the automatically generated OAR segmentation generated by the AI ​​engine or other software application in the OAR segmentation step 106. In some embodiments, such a reviewer is a physician who approves one or more of the plans generated in the plan optimization step 107, wherein one or more plans are generated by an FDA-approved software application or by a dosimetrist who employs such a software application. In some embodiments, such a reviewer is a dosimetrist who reviews the output of such a software application in the plan optimization step 107. In some embodiments, such a reviewer is a physician who performs a quantitative plan QA step 108 and evaluates the plan approved in the plan optimization step 107. In some embodiments, the reviewer for which the performance database 203 stores confidence scores is an A1 engine, a neural network, or other machine learning entity that is trained to perform any of the above review processes in the target segmentation step 105, the OAR segmentation step 106, the plan optimization step 107, and / or the quantitative plan QA step 108.

[0048] In some embodiments, the performance database 203 may include a credibility score for each specific instance of a specific machine learning entity. For example, in one such embodiment, an AI engine may be employed to perform an automatic segmentation process in the target segmentation step 105, wherein different instances of the AI ​​engine are trained for different corresponding regions of the patient's anatomy. Thus, in this embodiment, the performance database 203 may include different credibility scores for each instance of the AI ​​engine that is trained to perform the automatic segmentation process in the target segmentation step 105.

[0049] In some embodiments, some or all of the credibility scores included in the performance database 203 may include multiple values ​​for each reviewer. For example, in some embodiments, such a multi-value credibility score includes the reviewer's success rate in detecting intentionally included errors. Alternatively or additionally, in some embodiments, the multi-value credibility score includes the reviewer's different success rates for each of different categories of intentionally included errors. Alternatively or additionally, in some embodiments, the multi-value credibility score includes how recently the reviewer had an opportunity to detect an intentionally included error. Alternatively or additionally, in some embodiments, the multi-value credibility score includes the frequency with which the reviewer has an opportunity to detect intentionally included errors. Therefore, the qualification of a particular reviewer can be based on the reviewer's error detection success rate, how recently the reviewer had an opportunity to detect an intentionally included error, the frequency with which the reviewer has an opportunity to detect an intentionally included error, etc.

[0050] The error database 204 includes a plurality of previously generated errors for one or more of the steps 101-110 of the treatment planning process 100. Typically, each of the previously generated errors is associated with a specific one of the steps 101-110 of the treatment planning process 100. Thus, errors associated with the target segmentation step 105 include errors in image interpretation, such as incorrect depiction of the PTV. In contrast, errors associated with different steps of the treatment planning process 100 include different types of errors. For example, errors associated with the plan optimization step 107 may include incorrectly planned beam geometry or dose distribution. In some embodiments, the previously generated errors may include human-generated errors. Alternatively or additionally, in some embodiments, the previously generated errors may include software-generated errors and / or errors extracted from an actual planned treatment process.

[0051] The planning module 205 is configured to track each radiation therapy treatment plan that includes one or more intentional errors. Thus, the planning module 205 ensures that no plan that includes one or more intentional errors is delivered to the patient. In some embodiments, the planning module 205 interacts with one or more plan databases 299 to indicate that a particular plan is not intended for any patient.

[0052] Figure 3 A flow chart of an example QA process for a treatment planning process according to one or more embodiments of the present disclosure is set forth. The method may include one or more operations, functions, or actions as shown in one or more of blocks 301-330. Although the blocks are shown in a sequential order, the blocks may be executed in parallel and / or in a different order than those described herein. In addition, various blocks may be combined into fewer blocks, divided into additional blocks, and / or eliminated based on the desired implementation. Although combined Figure 1-2 The method is described with reference to processes and systems, but those skilled in the art will appreciate that any suitably configured QA process performed with respect to the treatment planning process is within the scope of the present disclosure.

[0053] The QA process 300 begins at step 301, when the QA system 200 receives input data from a software module ("review module") that performs one of the steps 101-110 of the treatment planning process 100 in an automated manner. For example, in one embodiment, the input data to the review module for performing the target segmentation step 105 may include a CT scan of the patient's anatomical region and applicable information from the patient's treatment plan instructions. In another embodiment, the input data to the review module for performing the OAR segmentation step 106 may include a CT scan of the patient's anatomical region, applicable information from the patient's treatment plan instructions, and a delineation of one or more target regions (GTV, CTV, ITV, PTV, etc.) in the anatomical region. In another embodiment, the input data to the review module for performing the plan optimization step 107 may include applicable information from the patient's treatment plan instructions and a delineation of one or more target regions (GTV, CTV, ITV, PTV, etc.) in the region. In another embodiment, the input data to the review module for performing the quantitative plan QA step 108 may include applicable information from the patient's treatment plan instructions and one or more approved plans generated in the plan optimization step 107.

[0054] In step 302, QA system 200 determines whether the auditor of the audited module is qualified to conduct a QA inspection, for example, via auditor monitoring module 202. If not, QA process 300 proceeds to step 320; if yes, QA process 300 proceeds to step 303.

[0055] In step 303, the QA system 200 generates output data for the review module determined to be eligible for QA inspection in step 302, for example, via a suitable error introduction module 201. Therefore, the output data so generated includes one or more intentional errors. In some embodiments, the error introduction module 201 introduces one or more intentional errors into the output of the review module. In other embodiments, instead of the review module generating output data, the error introduction module 201 generates output data including one or more intentional errors.

[0056] In step 304, the QA system 200 submits the output data generated in step 303 to the reviewer. Typically, the QA system 200 sends the output data to the reviewer in the same manner as normal output data from the module being reviewed is sent to the reviewer. Thus, in embodiments where the reviewer is an expert human, the QA system 200 may send the output data to a computing device employed by the reviewer during the review process. In such embodiments, the output data generated in step 303 is indistinguishable in content and format from normal output data from the module being reviewed. In embodiments where the reviewer is a suitably trained machine learning entity, the QA system 200 sends the output data to the A1 reviewer normally.

[0057] In step 305, the QA system 200 receives an error check response associated with the reviewer. For example, in embodiments where the reviewer is an expert human, the QA system 200 may receive user input indicating that the human reviewer detected one or more errors or that the output is error-free. In embodiments where the reviewer is a properly trained machine learning entity, the QA system 200 may receive a signal from the machine learning entity indicating that one or more errors were detected or that the output was error-free. In some embodiments, the error check response may also include information indicating the nature of the one or more errors detected, such as the type, severity, and / or location of the detected error(s).

[0058] In step 306 , the QA system 200 determines whether the auditor correctly identified one or more errors. If so, the QA process 300 proceeds to step 311 ; if not, the QA process 300 proceeds to step 307 .

[0059] In step 310, performed in response to QA system 200 determining that the auditor correctly identified one or more errors, QA system 200 updates the confidence score and / or other performance statistics associated with the auditor in performance database 203. QA process 300 then proceeds to step 320 and terminates.

[0060] In response to the QA system 200 determining that (1) the reviewer of the reviewed module is not qualified to perform the QA check, or (2) the reviewer of the reviewed module correctly identified one or more intentional errors, step 320 is performed. In step 320, the QA system sends the input data received in step 301 to the reviewed module to perform the treatment planning step normally. The QA process 300 then terminates.

[0061] In response to the QA system 200 determining that the reviewer did not correctly identify all intentional errors included in the output of the reviewed module, step 330 is performed. In step 330, the QA system 200 marks the reviewer and takes one or more corrective actions. For example, in an embodiment where the reviewer is an expert, the corrective action may include updating the user's credibility rating and modifying the appropriate value stored in the performance database 203. Alternatively or additionally, in some embodiments, the corrective action may include causing an indication to be displayed, indicating to the reviewer that the reviewer failed to detect at least one error. As a result of this corrective action, the current performance of the reviewed module is tracked and the future performance of the reviewer can be improved.

[0062] In an embodiment where the reviewer is a machine learning process, the QA system 200 may generate an identifier associated with the machine learning process as a corrective action, wherein the identifier indicates that the machine learning process is eligible for retraining. In such an embodiment, the identifier may further indicate that the machine learning process is not eligible for use prior to retraining. Alternatively or additionally, in some embodiments, the corrective action may include updating the credibility rating of the current instance of the machine learning process and modifying appropriate values ​​stored in the performance database 203. Alternatively or additionally, in some embodiments, the corrective action may include retraining or modifying the machine learning process based on one or more errors that the machine learning process failed to detect. As a result of such corrective action, future performance of the machine learning process may be improved based on increasingly subtle errors in the output of the review module. For example, in some embodiments, the machine learning process may be retrained (as well as initially trained) as a discriminator function for a GAN. The following is combined with Figure 4 One such embodiment is described.

[0063] Figure 4 400 for training a machine learning process 401 according to various embodiments of the present disclosure. The training process 400 includes one or more input objects (module input 402 and additional error 403) and expected output (curated output 404). In the training process 400, the one or more input objects are used as input to the training machine learning process 401, which acts as a generator function of the GAN 420. In addition, the expected output (curated output 404) is used as input to the discriminator function 421 of the GAN 420.

[0064] During GAN training of the machine learning process 401, the machine learning process 401 learns to generate an error-modified output 405. The error-modified output 405 is similar to the output of the review module that performs a particular step of the treatment planning process 100, except that the error-modified output 405 also includes one or more intentional errors (e.g., the added error 403). Thus, after such training, the machine learning process 401 can be used in the QA system 200 as an error-introducing module 201 that is configured to generate output data for a particular step of the treatment planning process 100, wherein the output data includes at least one intentional error.

[0065] Module inputs 402 are used as input objects in the training process 400 and may include typical input instances of the module under review that the machine learning process 401 is trained to simulate. Added errors 403 are specific errors or error types that may occur when the module under review performs steps of the treatment planning process 100, such as structures that are incorrectly delineated in OAR or target region segmentation, areas of overdose / underdose that may result from the treatment plan, etc.). In some embodiments, the added errors 403 may be selected from the error database 204. Curated outputs 404 are used as the expected output (or "ground truth") of the training process 400 and are input into the discriminator function 421. Because the purpose of the training process 400 is to teach the machine learning process 401 to generate output data that includes one or more added errors 403, each curated output 404 input into the discriminator function 421 corresponds to a specific instance of the module input 402 and a specific added error 403 that has been incorporated into that instance of the module input 402. In addition, each curated output 404 includes a different error that may typically occur in the output from the review module. Thus, each curated output 404 enables the machine learning process 401 to learn to include different errors into the error-modified output 405.

[0066] In some embodiments, the various examples of curated output 404 and added errors 403 are manually generated examples of errors that occurred in module output 402. In some embodiments, the various examples of curated output 404 and added errors 403 are based on actual examples of errors that have been detected when module output 402 was processed by a human reviewer. In some embodiments, the curated output 404 and added errors 403 can be based on actual examples of errors that have been detected by a human or Al reviewer. Figure 3One or more intentional errors included in the output of the review module that were not detected during iterations of the QA process 300 in the embodiment. In such an embodiment, the training process 400 is employed to retrain a particular Al reviewer. Thus, in such an embodiment, intentional errors that have proven difficult to detect can be used to increase the sensitivity of the Al reviewer to such errors.

[0067] GAN420 is a machine learning system that typically relies on unsupervised learning to attempt to approximate human logic or decision making when searching for hidden structures, patterns, or features in input objects. Thus, GAN420 includes a machine learning process 401 (as a generator function) and a discriminator function 421.

[0068] The machine learning process 401 may be a neural network or other suitable machine learning model configured to generate an error modified output 405 based on a particular module input 402. For example, the machine learning process 401 may be configured to generate an error modified output 405 for a review module that performs one of the steps 101-110 of the treatment planning process 100. In addition, during the training process 400, the machine learning process 401 is configured to improve the quality of the generated errors based on feedback 422 from the discriminator function 421. For example, during the training process 400, the machine learning process 401 is configured to modify the algorithm parameters 423 so that the discriminator function 421 cannot detect the errors included in the error modified output 405. More specifically, the algorithm parameters 423 are modified through an iterative process included in the training process 400. In this way, the machine learning process 401 can learn and / or be retrained to generate an error modified output 405 that includes errors that are more difficult for the discriminator function 421 to detect. The machine learning process 401 may then generate another error-corrected output 405 using the new modified values ​​of the algorithm parameters 423 .

[0069] Discriminator function 421 may be a neural network or other suitable machine learning model that is configured to detect errors in a particular error-modified output 405 generated by machine learning process 401. In some embodiments, during training process 400, discriminator function 421 is configured to improve its performance in detecting such errors. In such embodiments, discriminator function 421 may be configured to improve its performance based on a comparison of an expected output (or "ground truth") such as curated output 404 for training process 400 with the corresponding error-modified output 405. For example, during training process 400, discriminator function 421 is configured to determine whether error-modified output 405 corresponding to a particular instance of module input 402 is error-free; compare error-modified output 405 to a suitable instance of curated output 404; and, based on the comparison, modify algorithm parameters 424 so that discriminator function 421 can more effectively detect errors in subsequent outputs generated by the module under scrutiny. Additionally, during the training process 400, the discriminator function 421 is also configured to generate feedback 422, which informs the machine learning process 401 which errors were detected in the error-modified output 405. Based on the feedback 422, the machine learning process 401 can modify the algorithm parameters 423 and then use the new modified values ​​of the algorithm parameters 423 to generate another error-modified output 405.

[0070] Therefore, through the iterative process included in the training process 400, the discriminator function 421 and the machine learning process 401 interact in a double feedback loop: the discriminator function 421 is in a feedback loop with the machine learning process 401, where the difference between the ground truth of the curated output 404 and the error-modified output 405 serves as feedback to the discriminator function 421; and the machine learning process 401 is in a feedback loop with the discriminator function 421, where the feedback 422 serves as feedback to the machine learning process 401.

[0071] In some embodiments, the iterative process included in the training process 400 typically includes the application of certain success criteria. In some embodiments, one success criterion is that the magnitude and / or significance of one or more errors included in the error-modified output 405 is not detected (or difficult to detect) by the discriminator function 421, but is still clinically significant. In some embodiments, the clinical significance of the errors included in the error-modified output 405 can be determined based on quantifiable dimensions and / or numerical magnitude tests, such as dose-volume histograms, relative bioeffectiveness, predicted clinical outcomes or side effect profiles, OAR toxicity, etc.

[0072] According to various embodiments, the training method 400 may be employed multiple times in the training of a single machine learning process 401. Specifically, the training method 400 may be performed for multiple different module inputs 402 and / or added errors 403. That is, the training method 400 may be used multiple times to train a single machine learning process 401, each time using different module inputs 402, added errors 403, and corresponding curated outputs 404.

[0073] Figure 5 is an illustration of a computing device 500 configured to perform various embodiments of the present disclosure. The computing device 500 may be a desktop computer, a laptop computer, a smart phone, or any other type of computing device suitable for practicing one or more embodiments of the present disclosure. In operation, the computing device 500 is configured to execute the QA system 200, the QA process 300, the training process 400, and / or the GAN 420, as described herein. Note that the computing devices described herein are exemplary, and any other technically feasible configurations fall within the scope of the present disclosure.

[0074] As shown, computing device 500 includes, but is not limited to, interconnect (bus) 540, which connects processing unit 550, input / output (I / O) device interface 560 coupled to input / output (I / O) device 580, memory 510, storage device 530, and network interface 570. Processing unit 550 can be any suitable processor implemented as the following: central processing unit (CPU), graphics processing unit (GPU), application specific integrated circuit (ASIC), field programmable gate array (FPGA), any other type of processing unit, or a combination of different processing units, such as a CPU configured to operate in conjunction with a GPU or a digital signal processor (DSP). In general, processing unit 550 can be any technically feasible hardware unit capable of processing data and / or executing software applications, including QA system 200, QA process 300, training process 400, and / or GAN 420.

[0075] I / O devices 580 may include devices capable of providing input, such as a keyboard, a mouse, a touch-sensitive screen, etc., and devices capable of providing output, such as a display device and the like. In addition, I / O devices 580 may include devices capable of receiving input and providing output, such as a touch screen, a universal serial bus (USB) port, etc. I / O devices 580 may be configured to receive various types of input from an end user of computing device 500, and also provide various types of output, such as a displayed digital image or digital video, to an end user of computing device 500. In some embodiments, one or more of I / O devices 580 are configured to couple computing device 500 to a network.

[0076] The memory 510 may include a random access memory (RAM) module, a flash memory unit, or any other type of memory unit or a combination thereof. The processing unit 550, the I / O device interface 560, and the network interface 570 are configured to read data from and write data to the memory 510. The memory 510 includes various software programs executable by the processor 550 and application data associated with the software programs, including the QA system 200, the QA process 300, the training process 400, and / or the GAN 420.

[0077] Figure 6 6 is a block diagram of an illustrative embodiment of a computer program product 600 for implementing a method for segmenting an image according to one or more embodiments of the present disclosure. The computer program product 600 may include a signal bearing medium 605. The signal bearing medium 604 may include one or more sets of executable instructions 602, which when executed by, for example, a processor of a computing device, may provide at least the above-mentioned Figure 1-5 Describes the functionality.

[0078] In some implementations, the signal bearing medium 604 may include a non-transitory computer readable medium 608, such as but not limited to a hard drive, a compact disk (CD), a digital video disk (DVD), a digital tape, a memory, etc. In some implementations, the signal bearing medium 604 may include a recordable medium 610, such as but not limited to a memory, a read / write (R / W) CD, a R / W DVD, etc. In some implementations, the signal bearing medium 604 may include a communication medium 606, such as but not limited to a digital and / or analog communication medium (e.g., an optical cable, a waveguide, a wired communication link, a wireless communication link, etc.). The computer program product 600 may be recorded on the non-transitory computer readable medium 608 or another similar recordable medium 610.

[0079] In summary, embodiments of the present disclosure enable testing and evaluation of an expert human reviewer or an AI error detection engine associated with a radiation therapy treatment planning process. Intentional errors are introduced into the output of a software module or artificial intelligence (AI) engine that performs a certain step in the radiation therapy treatment planning process. The effectiveness of the human or AI reviewer in detecting the errors can then be evaluated or tested by determining whether the human or AI reviewer detects the introduced errors.

[0080] The description of the various embodiments has been presented for purposes of illustration, but is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.

[0081] Aspects of the present embodiment may be embodied as a system, method, or computer program product. Therefore, aspects of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microcode, etc.), or an embodiment of a combination of software and hardware aspects, which may be generally referred to as "circuit", "module" or "system" herein. In addition, aspects of the present disclosure may take the form of a computer program product embodied in one or more computer-readable media having a computer-readable program code embodied thereon.

[0082] Any combination of one or more computer-readable media may be utilized. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer-readable storage media would include the following: an electrical connection with one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium that may include or store a program used by or in conjunction with an instruction execution system, device, or apparatus.

[0083] While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and not limitation, with the true scope and spirit being indicated by the following claims.

Claims

1. A computer-implemented method for quality assurance testing of a treatment planning process, the method comprising: receiving output data from a module of the treatment planning process, wherein the output data is based on input data of the module; introducing at least one intentional error into the output data; submitting the output data having the at least one intentional error to a reviewer for error checking, wherein the reviewer comprises a machine learning process configured to check the output of the module for errors; receiving an error check response associated with the examiner; as well as When the error checking response indicates a failure to detect the at least one intentional error, the machine learning process is retrained based on the at least one intentional error and normal output data from the module generated from the input data.

2. The computer-implemented method of claim 1 , further comprising: When the error checking response indicates a failure to detect the at least one intentional error, an identifier is generated that is associated with the machine learning process and indicates that the machine learning process is eligible for retraining.

3. The computer-implemented method of claim 1 , wherein retraining the machine learning process comprises modifying the machine learning process based on the at least one error.

4. The computer-implemented method of claim 1 , wherein retraining the machine learning process based on the normal output data from the module comprises processing the normal output data into expected outputs using a discriminator function of the machine learning process.

5. A computer-implemented method according to claim 1, wherein retraining the machine learning process based on the normal output data from the module includes processing the output data as an input object with a generator function of the machine learning process, the output data is based on the input data of the module and includes at least one error.

6. The computer-implemented method of claim 1 , introducing the at least one intentional error into the output data comprising: retrieving the at least one intentional error from an error database; as well as The at least one intentional error is merged into the normal output data.

7. A computer-implemented method according to claim 1, wherein introducing the at least one intentional error into the output data comprises processing the normal output data with a machine learning process, wherein the machine learning process is configured to incorporate appropriate errors into the output data from the module.

8. The computer-implemented method of claim 7, wherein processing the normal output data with the machine learning process comprises processing the normal output data with a generator function of the machine learning process.

9. The computer-implemented method of claim 1 , wherein introducing at least one intentional error into the output data comprises: receiving the input data of the module; as well as The output data is generated by processing the input data with an error generating module other than the module of the treatment planning process.

10. The computer-implemented method of claim 9, wherein the error generation module comprises a machine learning process configured to generate the output data having errors associated with the module of the treatment planning process.

11. The computer-implemented method of claim 1 , further comprising updating a credibility rating of a user in the reviewer when the error-checking response indicates that the reviewer detected the at least one intentional error.

12. A non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processors, configure the one or more processors to perform the following steps: receiving output data from a module of a treatment planning process, wherein the output data is based on input data of the module; introducing at least one intentional error into the output data; submitting the output data having the at least one intentional error to a reviewer for error checking, wherein the reviewer comprises a machine learning process configured to check the output of the module for errors; receiving an error check response associated with the examiner; as well as When the error checking response indicates a failure to detect the at least one intentional error, the machine learning process is retrained based on the at least one intentional error and normal output data from the module generated from the input data.

13. The non-transitory computer readable storage medium of claim 12, further comprising instructions that, when executed by one or more processors, configure the one or more processors to perform the following steps: When the error checking response indicates a failure to detect the at least one intentional error, an identifier is generated that is associated with the machine learning process and indicates that the machine learning process is eligible for retraining.

14. The non-transitory computer-readable storage medium of claim 12, wherein retraining the machine learning process comprises modifying the machine learning process based on the at least one error.

15. The non-transitory computer-readable storage medium of claim 12, wherein retraining the machine learning process based on the normal output data from the module comprises processing the normal output data into an expected output with a discriminator function of the machine learning process.

16. A non-transitory computer-readable storage medium according to claim 12, wherein retraining the machine learning process based on the normal output data from the module includes processing the output data as an input object using a generator function of the machine learning process, the output data is based on the input data of the module and includes at least one error.

17. The non-transitory computer-readable storage medium of claim 12, wherein introducing the at least one intentional error into the output data comprises: retrieving the at least one intentional error from an error database; as well as The at least one intentional error is merged into the normal output data.

18. The non-transitory computer-readable storage medium of claim 12, wherein introducing the at least one intentional error into the output data comprises processing the normal output data with a machine learning process configured to incorporate appropriate errors into the output data from the module.

19. The non-transitory computer-readable storage medium of claim 12, wherein introducing at least one intentional error into the output data comprises: receiving the input data of the module; as well as The output data is generated by processing the input data with an error generating module other than the module of the treatment planning process.

20. The non-transitory computer readable storage medium of claim 19, wherein the error generation module comprises a machine learning process configured to generate the output data having errors associated with the module of the treatment planning process.

Images