A method to facilitate risk-based quality control practices in clinical laboratories

Abstract

The invention relates to a computer-based quality control method for clinical laboratories comprising the implementation of quality control strategies based on the risk posed by analytical performance affecting the patient, statistical quality control, risk management and analytical performance evaluation.

Description

[0001] A METHOD TO FACILITATE RISK-BASED QUALITY CONTROL PRACTICES IN CLINICAL LABORATORIES

[0002] Technical Field

[0003] The invention relates to a computer-based quality control method for clinical laboratories comprising the implementation of quality control strategies based on the risk posed by analytical performance affecting the patient, statistical quality control, risk management and analytical performance evaluation.

[0004] Prior Art

[0005] In Zhang Y et al. (2023), which is one of the similar studies on risk-based quality control method, quality control parameters can be comprehensively obtained according to the number of quality control materials to be studied and the maximum expected number of unreliable patient results (Max E(Nuf)) (1). The calculations in the publication by Zhang et al. included only 8 multiple quality control scenarios (1). On the other hand, the calculation of the so-called risk management index (2), recommended by the Clinical and Laboratory Standards Institute (CLSI) in document EP23, is not available in this article. While basic information on how to calculate the risk management index can be found in the CLSI document EP23 (2), in the state of the art the only platform where this calculation can be done is the Bio-Rad Mission: Control program from Biorad.

[0006] Another example of a risk-based quality control application is the publication by Westgard JO et al. (2018), in which nomogram-based calculations are made, risk management index-based calculations cannot be made, some of the results can be obtained semi-quantitatively, and the power graphs of quality control rules are based on a limited number of simulations made 30 years ago (3-5). Laboratory Information Management Systems (LIMS) are an integral part of the integration of quality control and assurance practices, thus significantly improving laboratory operations (6). Furthermore, middleware solutions are used to facilitate the interaction between analytical instruments and LIMS (7). These solutions often include modules for real-time monitoring of quality control applications and analysis of moving averages (8). However, this field is currently struggling with significant challenges, particularly the prevalence of security and privacy concerns. This is compounded by the limited availability of prototypes and the scarcity of middleware solutions that cover a comprehensive range of applications (9).

[0007] As noted in CLSI C24ed4, procedures that exhibit relatively good analytical performance in medical contexts can be monitored using relatively simple Quality Control rules such as 4-ls and l-3s. In contrast, procedures with only marginal performance may require the application of a wide range of quality control (QC) rules (10). Moreover, while QC rules such as 2-2s and 1 -3 s are effective in detecting significant changes, trend detection rules such as Exponentially Weighted Moving Average (EWMA) and Cumulative Sum (CUSUM) are effective in detecting smaller changes and trends (10). However, trend detection approaches such as CUSUM and EWMA have been adopted in a limited capacity in clinical chemistry, primarily due to the complexity associated with their calculation (11).

[0008] The sigma metric, which measures analytical performance, is used for managing risk-based internal quality control. However, there is a considerable amount of discussion in the literature regarding sigma-metric calculations (12, 13) and the choice between total error and MU utilization (14). From a general industry perspective, sigma-metric calculation is basically based on the ratio of the tolerance limit to process variability (12). In the field of laboratory medicine, the traditional formula undergoes a notable change: the tolerance limit is replaced by allowable total error (TEa) minus Bias (15). An alternative formulation replaces TEa-Bias with the intraindividual biological variation (CVI) parameter (16) and maintains the same denominator (CVA) for both approaches (15, 16). The maximum allowable expanded uncertainty (MAU) documented in the EFLM biological variation database has been suggested to be equivalent to CVI at the 95% confidence level (17). Furthermore, MAU corresponds to twice the APS value for standard MU, which can be derived from also the studies based on indirect results (18).

[0009] Traditional statistical quality control practice is based on the selection of quality control rules targeting a probability of error detection (Ped) value of 90%, but applies primarily to procedures with high sigma metric values (19). For procedures with lower sigma metrics (~3), achieving 90% Ped becomes more difficult and adjustments to maximum run size (MRS: also means number of patient samples run between two QC events) are required to achieve an acceptable Maximum expected number of unreliable final patient results (E(Nuf)) value (Max E(Nuf)) (19). In this context, the nomogram-based MRS determination approach of Westgard et al. emerges as a relevant reference (3, 4). However, QC rule power curves appear to be based on old and limited simulations from 35 years ago and do not take advantage of contemporary computing capabilities for more comprehensive and accurate predictions (5). On the other hand, a recent paper by Zhang et al. provided a quantitative MRS calculation based on Max E(Nuf) and patient risk factors (1). In this study, the Max E(Nuf) value was evaluated according to the patient risk factor (Max E(Nuf) < patient risk factor) (1). The important role of the patient risk factor requires that it be determined objectively (20) and should also be linked to the relative harm caused by an erroneous test result (21).

[0010] Max E(Nuf) only expresses the expected increase in the number of erroneous results during an out-of-control error scenario and does not provide an accurate depiction of the overall situation of erroneous results in situations involving analytical processes with low sigma values (19). Furthermore, Max E(Nuf) serves as a proxy measure for patient risk (22). The likelihood of harm to patients due to reported erroneous results for a given test depends not only on the amount of reported erroneous results, but also on associated risk factors that encompass the likelihood and severity of harm associated with an erroneous result for that test in relation to a patient (22). This points to the imperative need for additional risk management parameters to effectively evaluate a risk-based quality control strategy. The recent CLSI EP23 document helps laboratory professionals to evaluate quality control strategies in terms of the potential likelihood of harm from an erroneous test result (2). However, the document does not clearly set out the complex calculations of probabilities. Furthermore, EP 23 ed2 does not directly help to determine the maximum run size (2).

[0011] Due to the fact that the methods and studies described above have different limitations on their own, there was a need to develop a computer-based quality control method that enables the implementation of quality control strategies (statistical quality control, risk management and analytical performance evaluation) that keep the risk of analytical performance in clinical laboratories at an acceptable level. The methods used within the scope of the applied computer-based quality control application provide MRS calculations and recommendations based on Max E (Nuf) and probability of false rejection (Pfr), as well as RMI (Risk Management Index) calculation according to EP23 ed2, which assesses the appropriateness of the quality control strategy. As a result, users have the ability to comprehensively evaluate QC plans considering various parameters such as Pfr, Max E(Nuf), RMI and other relevant QC parameters.

[0012] The Objects of the Invention

[0013] The object of the present invention is to develop a computer-based quality control method for the comprehensive implementation of risk-based quality control in clinical laboratories.

[0014] Another object of the invention is to ensure that effective statistical quality control practices, in particular risk-based quality control (QC) strategies, are implemented while keeping the risk to patients at an acceptable level.

[0015] Detailed Description of the Invention The invention relates to a computer-based quality control method for clinical laboratories, considering clinical impact of the error realated to the rest results, statistical quality control, risk management and analytical performance evaluation, comprising these process steps;

[0016] Selecting the the quality control rule with the number of quality control measurements by users,

[0017] - Entering the tolerance limit such as TEa (%) or MAU (%) values, the analytical coefficient of variation (CVA (%)) expressing the precision of the analytical method used, in the users interface,

[0018] Calculating the probability of false rejection (Pfir) using the cumulative distribution function (CDF) of the normal distribution in line with the entered values and quality control rules,

[0019] - Listing the potential probable errors expressed in units of standard deviation,

[0020] Calculating the error detection probabilities (Ped) for each potential defect, taking into account the specified quality control (QC) rules and the number of quality control (QC) measurements,

[0021] Calculating the average run length (the average number of QC samples that need to be analyzed before a specific error is detected by the QC rule for each potential error),

[0022] Calculating the expected number of patient test results produced between the occurrence of an out-of-control condition and the next quality control event,

[0023] Calculating the average number of patient samples analysed from the onset of an out-of-control condition until its detection by the QC application,

[0024] Calculating the probability of an erroneous result occurring,

[0025] Calculating the expected number of unreliable results for each potential error scenario,

[0026] Calculating of the percentage of unreliable results, Calculating the expected number of unreliable final patient results for each error scenario,

[0027] Calculating the expected number of correctable unreliable results, Calculating the maximum study size (MRS) (the maximum number of patient samples that can be analysed between two QC events), taking into account the severity of patient harm caused by a potentially erroneous result, Calculating the risk management index (RMI) described in CLSI EP23 ed2 document,

[0028] - Presenting the base QC parameters in table form, based on these calculated QC parameters, including probability of false rejection (Pfir), sigma-metric value, MaxE(Nuf), systematic error in MaxE(Nuf) and MRS,

[0029] - Recommendation for using the different QC rules or reducing the number of QC measurements, if the PFR value is above 5%,

[0030] - Reporting the maximum number of patient samples that can be analysed between two QC events (MRS),

[0031] Accepting the QC strategy as appropriate if RMI < 1,.

[0032] Within the scope of the invention, basic quality control parameters are taken as input from the users for the risk-based quality control application and suggestions are provided to guide the user. In this context; for a single rule scheme, users are enabled to select a quality control rule using the number of quality control (QC) measurements ranging from 1 to 12 and standard deviation multiples ranging from 2.0 to 5.0 as control limits. The software user interface allows users to enter the total allowable error (TEa (%)) or maximum allowable uncertainty (MAU (%)) values and the analytical coefficient of variation (CVA (%)), which expresses the imprecision of the analytical method used. Using the cumulative distribution function (CDF) of the normal distribution from the statistical functions module of SciPy version 1.12.0 (23), the probability of false rejection (Pfr: probability of false rejection) for the specified QC rule is calculated to assess the risk of false rejection of QC results under controlled conditions. To assess potential errors, various multiples of TEa (or MAU) are listed in percentage units. These error percentages are then converted into standard deviation units by dividing by the CVA% value entered earlier by the user. For each potential error, expressed in units of standard deviation, the probability of error detection (Ped) is calculated using the CDF of the normal distribution, taking into account the QC rule chosen and the number of QC measurements. The average run length (the average number of QC samples that must be analyzed before a given error is detected by the QC rule for each potential error) (ARLed) corresponding to each Ped value is then calculated (ARLed = 1 / Ped). Taking the expected number of patient samples between QC measurements (E(NB)) as input from the user, the expected number of patient test results (E(N0)) produced between the occurrence of an out-of-control condition and the next QC application is calculated (E(N0) = E(NB) / 2) (24). Then, for each potential error scenario, the average number of patient samples processed from the onset of an out- of-control condition until its detection by the QC event is calculated as (E(NP) = E(0) + E(NB) x (1 / Ped - 1)). The probabilities and of an erroneous result exceeding TEa (or MAU) during the analytical process in-control (P(TEa-in-control) and out- of-control analytical process P(TEa-out-of-control) are calculated using the cumulative distribution function (CDF). For each potential error scenario, the expected number of unreliable results (E(Nu)) is calculated (E(Nu) = E(NP) x (APE)) (APE represents the increase in the probability of producing a result exceeding TEa (or MAU) due to the presence of an error condition in the out-of- control process). The percentage of unreliable results (UnR%) is calculated for each scenario by dividing E(Nu) by the average number of patient samples processed until error detection (E(NP)) and multiplying by 100. The expected number of unreliable final patient results (E(Nuf)) for each error scenario is then determined using the formula proposed by Parvin (25) (E(Nuf) = APE x [(ARLed- 1) x E(NB) - (1-Ped) x (E(NB) - E(N0))]). The expected number of correctable unreliable results (E(Nuc)) for all possible error levels is calculated according to the method of Zhang et al. (E(Nuc) = (Ped x E(N0)+(l-Ped) x E(NB))xAPE) (1). The maximum value of E(Nuf) (Max E(Nuf)) and the corresponding degree of systematic error are determined. The maximum run size (MRS: the maximum number of patient samples that can be studied between two QC events) is calculated taking into account the severity of harm caused by a possible erroneous result with the following formula:

[0033] MRS =(E(NB)) / (Max E(Nuf)) x (6-severity of harm score))

[0034] The severity of harm caused by a potentially erroneous result is classified according to ISO 14971 as negligible, minor, serious, critical and catastrophic, corresponding to severity of harm scores of 1, 2, 3, 4 and 5, respectively (26, 27).

[0035] The risk management index (RMI) recommended in CLSI EP23 ed2 is calculated.

[0036] The following formulas are used in the calculation (2, 28):

[0037] Predicted Pu

[0038] RMI = - — - -

[0039] Acceptabie PH

[0040] Predicted PH= PEx P^u

[0041] For each error scenario, the values ofP(TEa in-control), E(Nuf) andARLed are included in the calculation of the probability of producing an erroneous result under out-of-control conditions (PE)(29). The average of the PE values calculated for each failure scenario is then assigned as the final PE value (29).

[0042] The average number of days between device failures and the average number of patients tested per day are entered by the user. The mean number of patients between failures (MPBF) is calculated by multiplying these two input variables (28). Another user input factor is the probability of harm given an unacceptable result (Ph\u), which represents the probability that an erroneous result will harm the patient (2). Acceptable PH values for negligible, minor, serious, critical and catastrophic harm severity categories are 0.01, 0.001, 0.0001, 0.00001 and 0.000001, respectively (2). Based on these calculated QC parameters, key QC parameters are tabulated, including probability of false rejection (Pfr), sigma-metric value, MaxE(Nuf), systematic error in MaxE(Nuf) and MRS. If the Pfr value is above 5%, it is recommended to use different QC rules or to reduce the number of QC measurements. The MRS value is presented, indicating the maximum number of patient samples that can be studied between two QC applications. If RMI < 1, the QC strategy is considered to have an acceptable risk according to the EP23 ed2 document (2).

[0043] The next stage of the method involves additional quality control parameters calculated based on the systematic error in MaxE(Nuf). Flexibility is offered to users through a checkbox option to enter another desired custom systematic error. If the checkbox is selected, the user is prompted to enter a value; otherwise, the default systematic error value based on MaxE(Nuf) is used, whether default or custom, the corresponding systematic error is converted to a multiple of CVA. The tool then calculates the same risk-based quality control parameters mentioned above in relation to the systematic error of interest.

[0044] A comprehensive table of QC parameters is presented showing calculated values such as Ped, ARLed, E(NP), P(TEa in-control), P(TEa out-of-control), APE, E(Nu), UnR%, E(Nuf) and E(Nuc). These metrics guide decisions on QC frequency, error handling, balancing patient risk and resource utilization.

[0045] In addition, a power function plot is created that visualizes the relationship between the magnitude of a systematic error and the probability of its detection using the selected quality control rules.

[0046] The calculations for the multi-rule QC scenarios are similar except for the estimation of the probability of error detection (Ped). A Monte Carlo simulation was used to determine Ped and Pfr values for eight different multi-rule cases under varying conditions. These conditions included systematic error ranging from 0-10 in increments of 0.01 and the number of quality control measurements ranging from 1-6. For the simulation, 50,000 iterations were performed for each degree of error, taking into account each specified number of quality control measurements. Subsequent calculations in the multi-rule scheme closely followed the methodology of the single-rule scheme, with the important difference of using Ped and Pfir values obtained directly from the simulation.

[0047] The computer-based implementation of the inventive method was developed using Python version 3.11 (30) and Microsoft Visual Studio Code version 1. 77 .3. Data processing and some calculations within the application were performed using Pandas version 1.3.5 (31) and NumPy version 1.21.2 (32). The graphs in the application were created using Plotly version 5.15.0 (33). The computer-based application was developed using Streamlit version 1.29.0 (34).

[0048] References:

[0049] 1. Zhang Y, Ren B, Zou G, Yang L. A spreadsheet tool for designing statistical quality control programs based on patient risk parameters. Clin Biochem. 2023;116:52-8.

[0050] 2. Clinical and Laboratory Standards Institute. EP23-ED2:2023 - Laboratory Quality Control Based on Risk Management. USA2023.

[0051] 3. Westgard JO, Bayat H, Westgard SA. Planning Risk-Based SQC Schedules for Bracketed Operation of Continuous Production Analyzers. Clin Chem. 2018;64(2):289-96.

[0052] 4. Bayat H, Westgard SA, Westgard JO. Planning Risk-Based Statistical Quality Control Strategies: Graphical Tools to Support the New Clinical and Laboratory Standards Institute C24-Ed4 Guidance. J Appl Lab Med. 2017;2(2):211-21.

[0053] 5. Parvin CA. Planning Statistical Quality Control to Minimize Patient Risk: It's About Time. Clin Chem. 2018;64(2):249-50.

[0054] 6. Sepulveda JL, Young DS. The ideal laboratory information system. Arch Pathol Lab Med. 2013; 137(8): 1129-40. 7. Roland K, Yakimec J, Markin T, Chan G, Hudoba M. Customized middleware experience in a tertiary care hospital hematology laboratory. J Pathol Inform. 2022; 13: 100143.

[0055] 8. Riben M. Laboratory Automation and Middleware. Surg Pathol Clin. 2015;8(2): 175-86.

[0056] 9. Li X, Eckert M, Martinez JF, Rubio G. Context Aware Middleware Architectures: Survey and Challenges. Sensors (Basel). 2015;15(8):20570-607.

[0057] 10. Clinical and Laboratory Standards Institute. C24-ED4:2016 Statistical Quality Control for Quantitative Measurement Procedures: Principles and Definitions, 4th Edition. Wayne, PA2016.

[0058] 11. Cervinski MA, Bietenbeck A, Katayev A, Loh TP, van Rossum HH, Badrick T. Advances in clinical chemistry patient-based real-time quality control (PBRTQC). Adv Clin Chem. 2023;117:223-61.

[0059] 12. Coskun A, Serteser M, Unsal I. Sigma metric revisited: True known mistakes. Biochem Med (Zagreb). 2019;29(l):010902.

[0060] 13. Westgard S, Bayat H, Westgard JO. Mistaken assumptions drive new Six Sigma model off the road. Biochem Med (Zagreb). 2019;29(l):010903.

[0061] 14. Panteghini M. Reply to Westgard et al.: 'Keep your eyes wide ... as the present now will later be past'. Clin Chem Lab Med. 2022;60(9):e202-e3.

[0062] 15. Westgard JO, Westgard SA. Six Sigma Quality Management System and Design of Risk-based Statistical Quality Control. Clin Lab Med. 2017;37(l):85-96.

[0063] 16. Oosterhuis WP, Coskun A. Sigma metrics in laboratory medicine revisited: We are on the right road with the wrong map. Biochem Med (Zagreb). 2018;28(2):020503.

[0064] 17. Aarsand A, Fernandez-Calle P, Webster C, Coskun A, Gonzales-Lao E, Diaz-Garzon J, et al. The EFLM Biological Variation Database [updated April 16, 2020. Available from: https: / / biologicalvariation.eu / .

[0065] 18. ubukgu HC, Vanstapel F, Thelen M, van Schrojenstein Lantman M, Bemabeu-Andreu FA, Mesko Brguljan P, et al. APS calculator: a data-driven tool for setting outcome-based analytical performance specifications for measurement uncertainty using specific clinical requirements and population data. Clin Chem Lab Med. 2023.

[0066] 19. Yago M, Alcover S. Selecting Statistical Procedures for Quality Control Planning Based on Risk Management. Clin Chem. 2016;62(7):959-65.

[0067] 20. Yago M, Lopez-Escribano H. Establishing Commonsense-Based Statistical Quality Control Practices. Am J Clin Pathol. 2019;151(4):350-2.

[0068] 21. Westgard SA, Bayat H, Westgard JO. A multi-test planning model for risk based statistical quality control strategies. Clin Chim Acta. 2021;523:216-23.

[0069] 22. Yago M. Risk-Based Statistical Quality Control Planning Should Be Based More on Patient Risk. J Appl Lab Med. 2018;2(6):970-l.

[0070] 23. Virtanen P, Gommers R, Oliphant TE, Haberland M, Reddy T, Cournapeau D, et al. SciPy 1.0: fundamental algorithms for scientific computing in Python. Nat Methods. 2020;17(3):261-72.

[0071] 24. Yundt-Pacheco J, Parvin CA. Validating the performance of QC procedures. Clin Lab Med. 2013 ;33( 1 ):75-88.

[0072] 25. Parvin CA. Assessing the impact of the frequency of quality control testing on the quality of reported patient results. Clin Chem. 2008;54(12):2049-54.

[0073] 26. International Organization for Standardization. ISO 14971 :2019 Medical devices - Application of risk management to medical devices. 2019.

[0074] 27. International Organization for Standardization. ISO 24971 :2020 Medical devices — Guidance on the application of ISO 14971. 2020.

[0075] 28. Kamutsch D, Occhipinti F, Tumiatti D, Mueller T. Evaluation of the Impact of Changing Quality Control Rules and Frequency on the Risk Management Index: Results from the Clinical Routine of a Medical Laboratory. Lab Med. 2021;52(3):211-8.

[0076] 29. Parvin CA, Baumann NA. Assessing Quality Control Strategies for HbAlc Measurements From a Patient Risk Perspective. J Diabetes Sci Technol. 2018;12(4):786-91.

[0077] 30. Van Rossum G, Drake FL. Python 3 Reference Manual. Scotts Valley, CA: CreateSpace; 2009 2009. 31. Team TPD. pandas-dev / pandas: Pandas, latest ed: Zenodo; 2020.

[0078] 32. Harris CR, Millman KJ, van der Walt SJ, Gommers R, Virtanen P, Coumapeau D, et al. Array programming with NumPy. Nature. 2020;585(7825):357-62. 33. Plotly Technologies Inc. Collaborative data science Publisher: Plotly Technologies Inc. Montreal, QC, 2015 [cited 2023 June 21, 2023], Available from: https: / / plot.ly.

[0079] 34. Streamlit • A faster way to build and share data apps [cited 2023 April 12, 2023], Available from: https: / / streamlit.io / .

Claims

CLAIMS1. A computer-based quality control method for clinical laboratories comprising implementation of patient-impact quality control strategies, statistical quality control, risk management and analytical performance evaluation, characterized in that the method comprises the following steps;Selecting the the quality control rule with the number of quality control measurements by users,- Entering the total permissible error (TEa (%)) or maximum permissible uncertainty (MAU (%)) values, the coefficient of analytical variation (CVA (%)) expressing the precision of the analytical method used, in the users interface,Calculating the probability of false rejection (Pfir) using the cumulative distribution function (CDF) of the normal distribution in line with the entered values and quality control rules,- Listing the potential probable errors expressed in units of standard deviation,Calculating the defect detection probabilities (Ped) for each potential defect, taking into account the specified quality control (QC) rules and the number of quality control (QC) measurements,Calculating the average run length (the average number of QC samples that need to be analyzed before a specific error is detected by the QC rule for each potential error),Calculating the expected number of patient test results produced between the occurrence of an out-of-control condition and the next quality control exercise,Calculating the average number of patient samples worked from the onset of an out-of-control condition until its detection by the QC application,Calculating the probability of an erroneous result occurring,Calculating the expected number of unreliable outcomes for each potential failure scenario,Calculating of the percentage of unreliable results,Calculating the expected number of unreliable final patient outcomes for each error scenario,Calculating the expected number of correctable unreliable results, Calculating the maximum study size (MRS) (the maximum number of patient samples that can be studied between two QC applications), taking into account the severity of patient harm caused by a potentially erroneous result,Calculating the risk management index (RMI) recommended in CLSI EP23 ed2,- Presenting the base QC parameters in table form, based on these calculated QC parameters, including probability of false rejection (Pfir), sigma-metric value, MaxE(Nuf), systematic error in MaxE(Nuf) and MRS,- Using the different QC rules or recommending reducing the number of QC measures, if the pfr value is above 5%,- Reporting the maximum number of patient samples that can be studied between two QC applications by presenting the MRS value,Accepting the QC strategy as appropriate if RMI < 1,.

Images