Assay performance monitoring

Abstract

Multiple instances of the assay may be run to obtain multiple concentration data points. Significant change points may then be identified that correspond to locations in the plurality of concentration data points where one or more statistical properties of the plurality of concentration data points change by more than a threshold. The significant change points may be correlated with one or more assay parameters associated with the assay by identifying instances of running the assay that correspond to the locations of the significant change points in the plurality of concentration data points. Based on the correlation, a cause for the change in one or more statistical properties of the plurality of concentration data points at the identified significant change points may be determined.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 433,598, entitled "Monitoring Assay Performance," filed December 19, 2022, the disclosure of which is incorporated herein by reference in its entirety. [Technical Field]

[0002] The present disclosure relates generally to techniques for monitoring and evaluating the performance of analytical biological procedures. More specifically, the present disclosure relates to methods for identifying and evaluating the significance of changes in data collected during analytical biological procedures. [Background technology]

[0003] Taking repeated measurements of a process or one or more aspects thereof over time has the potential to provide valuable insight into changes occurring within the process, however, techniques for analyzing such data to covertly identify process changes are currently lacking.

[0004] There are many sources of process data for analyzing process variations. For example, in the manufacturing of biological therapeutics such as antibodies, immunoassays are useful for tracking the concentration of the biological therapeutic between different manufacturing runs. The manufacturing of biological therapeutics is highly complex, involving numerous reagents and components (such as live cells), instruments, manufacturing and testing environments, and human operators, all of which are subject to change over time. Careful monitoring of the biological therapeutic manufacturing process, including monitoring assays using controls, is essential for the manufacturing quality control of such therapeutics. A further example can be found in the analysis of digital biomarkers, such as human body measurements taken over time from medical and / or consumer smart devices, such as smartwatches. Analysis of these biomarkers generates complex multiparameter data, including linked information on heart rate, blood glucose levels, blood oxygen content, GPS coordinates, gyroscope data, and environmental conditions.

[0005] Current techniques for evaluating such data involve tedious manual processes that are irregularly spaced, noisy, multidimensional, often contain limited ground truth, and lack established metrics for identifying process outliers. Furthermore, current techniques only allow for identification of process deviations well after the process has been running, thus making correction of problems undesirably slow. Summary of the Invention

[0006] As previously mentioned, process data often includes repeated measurements of a time series of a process. A method is provided for determining the cause of statistical variation in process data by identifying locations in the process data where significant changes in the data occur. The method may enable a laboratory analyst to efficiently identify periods during which important shifts in the data occurred. Once the relevant periods are identified, the analyst can investigate potential causes of the shifts in the data by evaluating parameters associated with measurements made during the identified periods.

[0007] In some embodiments, the described methods can identify locations in the process data where changes in the data occur and verify the significance of the changes relative to the entire data set. Once potential locations have been algorithmically identified, the significance of each potential location may be verified by quantifying the statistical change in the data surrounding that location. This quantification of statistical change can be used to determine whether the change occurring at that location is significant, and therefore potentially indicative of measurement error or other causes worth evaluating, or whether it is not significant (e.g., the result of random statistical fluctuations). By verifying the significance of identified changes in the process data, the methods provided herein can, in some embodiments, help analysts spend their time investigating important changes that are likely to affect the outcome of the process being measured.

[0008] An example of a method for determining a cause of significant statistical change in a plurality of concentration data points obtained from an assay includes: running multiple instances of the assay to obtain a plurality of concentration data points; receiving assay information including the plurality of concentration data points and a plurality of assay parameters, where each assay parameter of the plurality of assay parameters is associated with one of the multiple instances of running the assay; identifying significant change points corresponding to locations in the plurality of concentration data points at which one or more statistical properties of the plurality of concentration data points change by more than a threshold; correlating one or more of the plurality of assay parameters with the identified significant change points by identifying one of the multiple instances of running the assay that corresponds to the location of the significant change point in the plurality of concentration data points; and determining a cause of change in the one or more statistical properties of the plurality of concentration data points at the identified significant change points based on the correlation between the one or more assay parameters and the significant change points.

[0009] In some embodiments of the method, the assay is configured to measure the concentration of an analyte in the sample.

[0010] In some embodiments of the method, the analyte is a therapeutic analyte.

[0011] In some embodiments of the method, the analyte is a therapeutic polypeptide.

[0012] In some embodiments of the method, the analyte is an antibody or a fragment thereof.

[0013] In some embodiments of the method, the sample is a cell culture sample or a derivative thereof.

[0014] In some embodiments of the method, the assay is an immunoassay.

[0015] In some embodiments of the method, the assay is a competitive assay.

[0016] In some embodiments of the method, the assay is a non-competitive assay.

[0017] In some embodiments of the method, the assay is a heterogeneous assay.

[0018] In some embodiments of the method, the assay is a homogeneous assay.

[0019] In some embodiments of the method, the assay is an ELISA assay.

[0020] In some embodiments of the method, the ELISA assay is a direct ELISA assay.

[0021] In some embodiments of the method, the ELISA assay is a sandwich ELISA assay.

[0022] In some embodiments of the method, the ELISA assay is a competitive ELISA.

[0023] In some embodiments of the method, running a plurality of instances of the assay comprises running two or more of the plurality of instances two or more times.

[0024] In some embodiments of the method, the two or more times comprise a time course of at least about one week.

[0025] In some embodiments of the method, running multiple instances of the assay includes running two or more of the multiple instances simultaneously.

[0026] In some embodiments of the method, the plurality of concentration data points includes data points relating to the concentration of a target analyte.

[0027] In some embodiments of the method, the plurality of concentration data points includes a data point for a control concentration.

[0028] In some embodiments of the method, the control is a negative control, a non-specific binding control, a blank control, a detection antibody control, a negative matrix control, or a positive control.

[0029] In some embodiments of the method, the plurality of concentration data points includes data points relating to solution concentration.

[0030] In some embodiments of the method, the plurality of concentration data points includes data points relating to the absolute amount of the target analyte.

[0031] In some embodiments of the method, the plurality of concentration data points includes data points for measurements associated with concentrations.

[0032] In some embodiments of the method, the measurement associated with concentration is an optical density (OD) measurement.

[0033] In some embodiments of the method, the plurality of concentration data points includes a data point for the mean, lowest standard deviation mean, highest standard deviation mean, or median control concentration.

[0034] In some embodiments of the method, the significant change point reflects inter-assay variability.

[0035] In some embodiments of the method, the significant change point reflects intra-assay variability.

[0036] In some embodiments of the method, two or more concentration data points of the plurality of concentration data points are in the same format.

[0037] In some embodiments of the method, the one or more assay parameters correlated with significant change points include an environmental factor.

[0038] In some embodiments of the method, the environmental factors include temperature, humidity, light, or pollutants.

[0039] In some embodiments of the methods, the one or more assay parameters correlated with significant change points include instrument factors.

[0040] In some embodiments of the method, the instrument factor comprises a reagent change, reagent aging, reagent expiration date, reagent contamination, reagent failure, hardware change, hardware aging, hardware contamination, hardware failure, meter change, meter failure, or meter calibration.

[0041] In some embodiments of the methods, the one or more assay parameters correlated with significant change points include human factors associated with one or more humans who performed or assisted in performing the assay.

[0042] In some embodiments of the method, the human factor comprises performance variation, performance error, or operator change.

[0043] In some embodiments of the method, identifying significant change-points includes determining an expected change-point population in the plurality of concentration data points.

[0044] In some embodiments of the method, identifying a significant change point includes selecting a first segment of concentration data points from the plurality of concentration data points; determining a first median value associated with the first segment of concentration data points; selecting a second segment of concentration data points from the plurality of concentration data points, wherein the concentration data points in the second segment are contiguous to the concentration data points in the first segment; determining a second median value associated with the second segment of concentration data points; comparing the first median value to the second median value; and determining whether a candidate change point is located between the first segment and the second segment based on the comparison between the first median value and the second median value.

[0045] In some embodiments of the method, the first segment and the second segment include at least a threshold number of concentration data points.

[0046] In some embodiments, the method includes receiving a threshold number of concentration data points from a user.

[0047] In some embodiments of the method, the threshold number of concentration data points is determined based on the assay.

[0048] In some embodiments, the method includes generating one or more average values ​​for the first segment and the second segment; generating one or more data point clusters, each data point cluster associated with one of the one or more average values ​​and including a concentration data point in the segment closest to the associated average value; updating a mean value in each data point cluster of the one or more data point clusters, wherein updating a mean value in a data point cluster includes identifying a centroid of the data point cluster; and iteratively repeating the steps of generating the one or more data point clusters and updating the mean value in each data point cluster until the mean value in each data point cluster no longer changes.

[0049] In some embodiments, the method includes generating one or more data point clusters within each of the first segment and the second segment, wherein the concentration data points within each data point cluster are normally distributed and have a unique mean value and a unique standard deviation value.

[0050] In some embodiments, the method includes identifying a principal data point cluster of the one or more data point clusters for the first segment and the second segment, the principal data point cluster comprising at least a threshold percentage of the total number of concentration data points in the segment.

[0051] In some embodiments, the method includes determining a deviation value for the candidate change point, the deviation value being a statistical difference between a primary data point cluster in the first segment and a primary data point cluster in the second segment.

[0052] In some embodiments of the method, the deviation value is a Jensen-Shannon deviation.

[0053] In some embodiments, the method includes determining a median change value for the candidate change points, the median change value being the difference between a first median value associated with the first segment and a second median value associated with the second segment.

[0054] In some embodiments, the method includes determining whether one or more statistical characteristics of the plurality of concentration data points vary by more than a threshold value by determining a weighted combination of a deviation value and a median variation value.

[0055] In some embodiments of the method, the weighted combination of the deviation value and the median change value is characterized by a weight parameter.

[0056] In some embodiments, the method includes receiving a weight parameter from a user.

[0057] In some embodiments of the method, the weighting parameters are determined based on an assay.

[0058] In some embodiments, the method includes performing a second plurality of instances of the assay after determining the cause of the change in one or more statistical properties of the plurality of concentration data points at the identified significant change point.

[0059] In some embodiments of the method, a second plurality of instances of the assay is performed using assay parameters that match one or more assay parameters correlated with the identified significant change points.

[0060] In some embodiments of the method, a second plurality of instances of the assay are run using assay parameters that match assay parameters associated with instances of running the assay that occurred before the identified significant change point.

[0061] In some embodiments, the method includes deleting one or more concentration data points from the plurality of concentration data points that correspond to instances from the plurality of instances of performing the assay that occurred after the identified significant change point.

[0062] An example system for determining a cause of a significant statistical change in a plurality of concentration data points obtained from an assay may comprise one or more processors configured to receive assay information including a plurality of concentration data points and a plurality of assay parameters obtained by running a plurality of instances of the assay; each assay parameter of the plurality of assay parameters is associated with one of the plurality of instances of running the assay; identify significant change-points corresponding to locations in the plurality of concentration data points at which one or more statistical properties of the plurality of concentration data points change by more than a threshold; correlate one or more of the plurality of assay parameters with the identified significant change-points by identifying one of the plurality of instances of running the assay that corresponds to the location of the significant change-point in the plurality of concentration data points; and determine a cause of the change in the one or more statistical properties of the plurality of concentration data points at the identified significant change-points based on the correlation between the one or more assay parameters and the significant change-points.

[0063] In some embodiments of the system, the assay is configured to measure the concentration of an analyte in a sample.

[0064] In some embodiments of the system, the analyte is a therapeutic analyte.

[0065] In some embodiments of the system, the analyte is a therapeutic polypeptide.

[0066] In some embodiments of the system, the analyte is an antibody or a fragment thereof.

[0067] In some embodiments of the system, the sample is a cell culture sample or a derivative thereof.

[0068] In some embodiments of the system, the assay is an immunoassay.

[0069] In some embodiments of the system, the assay is a competitive assay.

[0070] In some embodiments of the system, the assay is a non-competitive assay.

[0071] In some embodiments of the system, the assay is a heterogeneous assay.

[0072] In some embodiments of the system, the assay is a homogeneous assay.

[0073] In some embodiments of the system, the assay is an ELISA assay.

[0074] In some embodiments of the system, the ELISA assay is a direct ELISA assay.

[0075] In some embodiments of the system, the ELISA assay is a sandwich ELISA assay.

[0076] In some embodiments of the system, the ELISA assay is a competitive ELISA.

[0077] In some embodiments of the system, running multiple instances of an assay includes running two or more of the multiple instances two or more times.

[0078] In some embodiments of the system, the two or more times constitute a time course of at least about one week.

[0079] In some embodiments of the system, running multiple instances of an assay includes running two or more of the multiple instances simultaneously.

[0080] In some embodiments of the system, the plurality of concentration data points includes data points relating to the concentration of a target analyte.

[0081] In some embodiments of the system, the plurality of concentration data points includes data points for the concentration of a control.

[0082] In some embodiments of the system, the control is a negative control, a non-specific binding control, a blank control, a detection antibody control, a negative matrix control, or a positive control.

[0083] In some embodiments of the system, the plurality of concentration data points includes data points relating to solution concentration.

[0084] In some embodiments of the system, the plurality of concentration data points includes data points relating to the absolute amount of the target analyte.

[0085] In some embodiments of the system, the plurality of concentration data points includes data points for measurements associated with concentrations.

[0086] In some embodiments of the system, the concentration-related measurement is an optical density (OD) measurement.

[0087] In some embodiments of the system, the plurality of concentration data points includes a data point for the mean, lowest standard deviation mean, highest standard deviation mean, or median control concentration.

[0088] In some embodiments of the system, the significant change point reflects inter-assay variability.

[0089] In some embodiments of the system, the significant change point reflects intra-assay variability.

[0090] In some embodiments of the system, two or more concentration data points of the plurality of concentration data points are in the same format.

[0091] In some embodiments of the system, the one or more assay parameters correlated with significant change points include environmental factors.

[0092] In some embodiments of the system, the environmental factors include temperature, humidity, light, or pollutants.

[0093] In some embodiments of the system, the one or more assay parameters correlated with significant change points include instrument factors.

[0094] In some embodiments of the system, the instrument factor includes a reagent change, reagent aging, reagent expiration date, reagent contamination, reagent failure, hardware change, hardware aging, hardware contamination, hardware failure, meter change, meter failure, or meter calibration.

[0095] In some embodiments of the system, one or more assay parameters correlated with significant change points include human factors associated with one or more humans who performed or assisted in performing the assay.

[0096] In some embodiments of the system, the human factor includes performance variation, performance error, or operator change.

[0097] In some embodiments of the system, identifying significant change-points includes determining an expected change-point population in the plurality of concentration data points.

[0098] In some embodiments of the system, identifying a significant change point includes selecting a first segment of concentration data points from the plurality of concentration data points; determining a first median value associated with the first segment of concentration data points; selecting a second segment of concentration data points from the plurality of concentration data points, wherein the concentration data points in the second segment are contiguous to the concentration data points in the first segment; determining a second median value associated with the second segment of concentration data points; comparing the first median value to the second median value; and determining whether a candidate change point is located between the first segment and the second segment based on the comparison between the first median value and the second median value.

[0099] In some embodiments of the system, the first segment and the second segment include at least a threshold number of concentration data points.

[0100] In some embodiments of the system, the one or more processors are configured to receive a threshold number of concentration data points from a user.

[0101] In some embodiments of the system, the threshold number of concentration data points is determined based on the assay.

[0102] In some embodiments of the system, the one or more processors are configured to generate one or more average values ​​for the first segment and the second segment, generate one or more data point clusters, each data point cluster being associated with one of the one or more average values ​​and including a concentration data point within the segment that is closest to the associated average value, update the average value in each data point cluster of the one or more data point clusters, where updating the average value in a data point cluster includes identifying a centroid of the data point cluster, and iteratively repeat the steps of generating the one or more data point clusters and updating the average value in each data point cluster until the average value in each data point cluster no longer changes.

[0103] In some embodiments of the system, the one or more processors are configured to generate one or more data point clusters within each of the first segment and the second segment, wherein the concentration data points within each data point cluster are normally distributed and have a unique mean value and a unique standard deviation value.

[0104] In some embodiments of the system, the one or more processors are configured to identify a principal data point cluster of the one or more data point clusters for the first segment and the second segment, the principal data point cluster comprising at least a threshold percentage of the total number of concentration data points in the segment.

[0105] In some embodiments of the system, the one or more processors are configured to determine a deviation value for the candidate change point, the deviation value being a statistical difference between a primary data point cluster in the first segment and a primary data point cluster in the second segment.

[0106] In some embodiments of the system, the deviation value is a Jensen-Shannon deviation.

[0107] In some embodiments of the system, the one or more processors are configured to determine a median change value for the candidate change points, the median change value being the difference between a first median value associated with the first segment and a second median value associated with the second segment.

[0108] In some embodiments of the system, the one or more processors are configured to determine whether one or more statistical characteristics of the plurality of concentration data points change by determining a weighted combination of the deviation value and the median change value.

[0109] In some embodiments of the system, the weighted combination of the deviation value and the median change value is characterized by a weight parameter.

[0110] In some embodiments of the system, the one or more processors are configured to receive weighting parameters from a user.

[0111] In some embodiments of the system, the weighting parameters are determined based on the assay.

[0112] In some embodiments of the system, the one or more processors are configured to delete one or more concentration data points of the plurality of concentration data points that correspond to instances of the plurality of instances of performing the assay that occurred after the identified significant change point.

[0113] An example of a non-transitory computer-readable storage medium may store instructions for determining a cause of a significant statistical change in a plurality of concentration data points obtained from an assay, the instructions being configured to be executed by one or more processors of an electronic device to cause the device to: receive assay information including a plurality of concentration data points and a plurality of assay parameters obtained by running a plurality of instances of the assay; identify significant change-points corresponding to locations in the plurality of concentration data points at which one or more statistical properties of the plurality of concentration data points change beyond a threshold; correlate one or more of the plurality of assay parameters with the identified significant change-points by identifying the one or more instances of running the assay that corresponds to the location of the significant change-point in the plurality of concentration data points; and determine a cause of the change in the one or more statistical properties of the plurality of concentration data points at the identified significant change-points based on the correlation between the one or more assay parameters and the significant change-points.

[0114] In some embodiments of the non-transitory computer-readable storage medium, the assay is configured to measure the concentration of an analyte in a sample.

[0115] In some embodiments of the non-transitory computer-readable storage medium, the analyte is a therapeutic analyte.

[0116] In some embodiments of the non-transitory computer-readable storage medium, the analyte is a therapeutic polypeptide.

[0117] In some embodiments of the non-transitory computer-readable storage medium, the analyte is an antibody or a fragment thereof.

[0118] In some embodiments of the non-transitory computer-readable storage medium, the sample is a cell culture sample or a derivative thereof.

[0119] In some embodiments of the non-transitory computer-readable storage medium, the assay is an immunoassay.

[0120] In some embodiments of the non-transitory computer-readable storage medium, the assay is a competitive assay.

[0121] In some embodiments of the non-transitory computer-readable storage medium, the assay is a non-competitive assay.

[0122] In some embodiments of the non-transitory computer-readable storage medium, the assay is a heterogeneous assay.

[0123] In some embodiments of the non-transitory computer-readable storage medium, the assay is a homogeneous assay.

[0124] In some embodiments of the non-transitory computer-readable storage medium, the assay is an ELISA assay.

[0125] In some embodiments of the non-transitory computer-readable storage medium, the ELISA assay is a direct ELISA assay.

[0126] In some embodiments of the non-transitory computer-readable storage medium, the ELISA assay is a sandwich ELISA assay.

[0127] In some embodiments of the non-transitory computer-readable storage medium, the ELISA assay is a competitive ELISA assay.

[0128] In some embodiments of the non-transitory computer-readable storage medium, running a plurality of instances of the assay comprises running two or more of the plurality of instances two or more times.

[0129] In some embodiments of the non-transitory computer-readable storage medium, two or more times constitute a time course of at least about one week.

[0130] In some embodiments of the non-transitory computer-readable storage medium, running multiple instances of the assay includes running two or more of the multiple instances simultaneously.

[0131] In some embodiments of the non-transitory computer-readable storage medium, the plurality of concentration data points includes data points relating to the concentration of a target analyte.

[0132] In some embodiments of the non-transitory computer-readable storage medium, the plurality of concentration data points includes data points relating to the concentration of a control.

[0133] In some embodiments of the non-transitory computer-readable storage medium, the control is a negative control, a non-specific binding control, a blank control, a detection antibody control, a negative matrix control, or a positive control.

[0134] In some embodiments of the non-transitory computer-readable storage medium, the plurality of concentration data points includes data points related to solution concentration.

[0135] In some embodiments of the non-transitory computer-readable storage medium, the plurality of concentration data points includes data points related to absolute amounts of the target analytes.

[0136] In some embodiments of the non-transitory computer-readable storage medium, the plurality of concentration data points includes data points for measurements associated with concentrations.

[0137] In some embodiments of the non-transitory computer-readable storage medium, the measurement associated with the concentration is an optical density (OD) measurement.

[0138] In some embodiments of the non-transitory computer-readable storage medium, the plurality of concentration data points includes a data point for the mean, lowest standard deviation mean, highest standard deviation mean, or median control concentration.

[0139] In some embodiments of the non-transitory computer-readable storage medium, the significant change point reflects inter-assay variability.

[0140] In some embodiments of the non-transitory computer-readable storage medium, the significant change point reflects intra-assay variability.

[0141] In some embodiments of the non-transitory computer-readable storage medium, two or more concentration data points of the plurality of concentration data points are in the same format.

[0142] In some embodiments of the non-transitory computer-readable storage medium, the one or more assay parameters correlated with significant change points include an environmental factor.

[0143] In some embodiments of the non-transitory computer-readable storage medium, the environmental factors include temperature, humidity, light, or pollutants.

[0144] In some embodiments of the non-transitory computer-readable storage medium, the one or more assay parameters correlated with significant change points include an instrument factor.

[0145] In some embodiments of the non-transitory computer-readable storage medium, the instrument factor comprises a reagent change, reagent aging, reagent expiration date, reagent contamination, reagent failure, hardware change, hardware aging, hardware contamination, hardware failure, meter change, meter failure, or meter calibration.

[0146] In some embodiments of the non-transitory computer-readable storage medium, the one or more assay parameters correlated with significant change points include human factors associated with one or more humans who performed or assisted in performing the assay.

[0147] In some embodiments of the non-transitory computer-readable storage medium, the human factor includes a performance variation, a performance error, or an operator change.

[0148] In some embodiments of the non-transitory computer-readable storage medium, identifying significant change-points includes determining an expected change-point population in the plurality of concentration data points.

[0149] In some embodiments of the non-transitory computer-readable storage medium, identifying a significant change point includes selecting a first segment of concentration data points from the plurality of concentration data points; determining a first median value associated with the first segment of concentration data points; selecting a second segment of concentration data points from the plurality of concentration data points, wherein the concentration data points in the second segment are contiguous to the concentration data points in the first segment; determining a second median value associated with the second segment of concentration data points; comparing the first median value to the second median value; and determining whether a candidate change point is located between the first segment and the second segment based on the comparison between the first median value and the second median value.

[0150] In some embodiments of the non-transitory computer-readable storage medium, the first segment and the second segment include at least a threshold number of concentration data points.

[0151] In some embodiments of the non-transitory computer-readable storage medium, the instructions, when executed by one or more processors of the electronic device, are configured to cause the electronic device to receive a threshold number of concentration data points from a user.

[0152] In some embodiments of the non-transitory computer-readable storage medium, a threshold number of concentration data points is determined based on the assay.

[0153] In some embodiments of the non-transitory computer-readable storage medium, the instructions, when executed by one or more processors of the electronic device, are configured to cause the device to generate one or more average values ​​for the first segment and the second segment, generate one or more data point clusters, each data point cluster being associated with one of the one or more average values ​​and including a concentration data point in the segment that is closest to the associated average value, update the average value in each data point cluster of the one or more data point clusters, wherein updating the average value in the data point cluster includes identifying a centroid of the data point cluster, and iteratively repeat the steps of generating the one or more data point clusters and updating the average value in each data point cluster until the average value in each data point cluster no longer changes.

[0154] In some embodiments of the non-transitory computer-readable storage medium, the instructions, when executed by one or more processors of the electronic device, are configured to cause the device to generate one or more data point clusters within each of the first segment and the second segment, wherein the concentration data points within each data point cluster are normally distributed and have a unique mean value and a unique standard deviation value.

[0155] In some embodiments of the non-transitory computer-readable storage medium, the instructions, when executed by one or more processors of the electronic device, are configured to cause the device to identify a principal data point cluster of the one or more data point clusters for the first segment and the second segment, the principal data point cluster comprising at least a threshold percentage of the total number of concentration data points in the segment.

[0156] In some embodiments of the non-transitory computer-readable storage medium, the instructions, when executed by one or more processors of the electronic device, are configured to cause the device to determine a deviation value for the candidate change point, the deviation value being a statistical difference between a primary data point cluster in a first segment and a primary data point cluster in a second segment.

[0157] In some embodiments of the non-transitory computer-readable storage medium, the deviation value is a Jensen-Shannon deviation.

[0158] In some embodiments of the non-transitory computer-readable storage medium, the instructions, when executed by one or more processors of the electronic device, are configured to cause the device to determine a median change value for the candidate change points, the median change value being a difference between a first median value associated with the first segment and a second median value associated with the second segment.

[0159] In some embodiments of the non-transitory computer-readable storage medium, the instructions, when executed by one or more processors of the electronic device, are configured to cause the device to determine whether one or more statistical characteristics of the plurality of concentration data points change by more than a threshold value by determining a weighted combination of a deviation value and a median change value.

[0160] In some embodiments of the non-transitory computer-readable storage medium, the weighted combination of the deviation value and the median change value is characterized by a weight parameter.

[0161] In some embodiments of the non-transitory computer-readable storage medium, the instructions, when executed by one or more processors of an electronic device, are configured to cause the device to receive weight parameters from a user.

[0162] In some embodiments of the non-transitory computer-readable storage medium, the weighting parameters are determined based on an assay.

[0163] In some embodiments of the non-transitory computer-readable storage medium, the instructions, when executed by one or more processors of the electronic device, are configured to cause the device to delete one or more concentration data points of the plurality of concentration data points that correspond to instances of the plurality of instances of performing the assay that occurred after the identified significant change point. [Brief explanation of the drawings]

[0164] The following figures illustrate various systems and methods for identifying and validating change points in concentration data obtained from an assay. The systems and methods illustrated in the figures may, in some embodiments, have any one or more of the features described herein.

[0165] [Figures 1A-1C] Figures 1A-1C show schematic diagrams of ELISA assay formats for obtaining analyte concentration data. Figure 1A shows a direct ELISA format, Figure 1B shows a sandwich ELISA format, and Figure 1C shows a competitive ELISA format.

[0166] [Figure 2] FIG. 2 shows an example of concentration data points obtained from the assay.

[0167] [Figure 3] FIG. 3 shows an example of the deviation in concentration data obtained from the assay.

[0168] [Figure 4]Figure 4 shows an example of change points in a dataset.

[0169] [Figure 5] FIG. 5 illustrates a method for determining the cause of significant statistical variation in multiple concentration data points obtained from an assay.

[0170] [Figure 6] FIG. 6 illustrates a method for identifying candidate change-points using binary segmentation.

[0171] [Figure 7] FIG. 7 shows the change points in the signal identified using binary segmentation.

[0172] [Figures 8A-8C] Figures 8A-8C illustrate methods for validating the significance of candidate change points. Figure 8A illustrates a method for generating data point clusters using a k-means algorithm. Figure 8B illustrates a method for generating data point clusters using a Gaussian mixture model. Figure 8C illustrates a method for determining whether statistical characteristics of multiple concentration data points surrounding a candidate change point change beyond a threshold.

[0173] [Figure 9A-9B] 9A-9B show an example segment in the concentration data and an example cluster of data points in the concentration data.

[0174] [Figures 10A-10C] 10A-10C show exemplary contour plots of functions that can be used to generate loss values ​​at candidate change points.

[0175] [Figure 11] FIG. 11 illustrates a method for proceeding after a cause of significant statistical variation in multiple concentration data points obtained from an assay has been identified.

[0176] [Figure 12] FIG. 12 illustrates a system for determining the cause of significant statistical variation in multiple concentration data points obtained from an assay.

[0177] [Figure 13] FIG. 13 illustrates an example of a computing system.

[0178] [Figures 14A-14J] 14A-14J show examples of significant change points identified in multiple concentration data points using the disclosed method. DETAILED DESCRIPTION OF THE INVENTION

[0179] The following disclosure describes a method for identifying and determining causes of statistical changes in process data by determining locations in the data where significant changes in the data occur. These locations, known as "change points," are typically times when statistical shifts occur in the time series data. The provided method allows laboratory analysts to correlate change points in the process data with changes in measured parameters that may have occurred at the time of the identified change points. This may enable efficient extraction of the root causes of changes in process data even when process ground truth data is limited or unavailable.

[0180] In this disclosure, the methods are described in the context of an assay, i.e., an investigative process that may be used to assess the presence of an analyte, such as a drug, cell, or chemical. This context is not intended to limit the disclosure, and the provided methods may be used to assess any data set that includes multiple repeated measurements.

[0181] Obtaining concentration data from the assay An assay is a process that can be used to determine the presence of an analyte. There are numerous assay types and formats known in the art. For example, as shown in Figures 1A-1C, a variety of enzyme-linked immunosorbent assay (ELISA) formats are available, including the direct ELISA format (Figure 1A), the sandwich ELISA format (Figure 1B), and the competitive ELISA format (Figure 1C).

[0182] In a direct ELISA format (FIG. 1A), a sample 108 is coated onto a solid phase, such as well 106 of a plate 104, and the desired antigen can be detected by an antibody 112. Negative and positive controls are available for the direct ELISA format. Such controls include wavelength correction, a blank control (e.g., a dry well or well containing ELISA buffer), an S0 negative control (no standard or any form of analyte, e.g., sample, added to the well), a negative matrix control, and a positive control such as B0 (to assess maximum color development). In some embodiments, a standard curve can also be utilized for quantification purposes.

[0183] In a sandwich ELISA format (FIG. 1B), a capture antibody 116 is coated onto a solid phase, such as a well 106 of a plate 104. The capture antibody 116 specifically binds to an analyte 110 of interest. A sample 108, if present, is then added to the well 106 to allow the capture antibody 120 to bind to the analyte. A detection antibody 112 is then added to the well to allow analyte detection. Negative and positive controls are available for sandwich ELISA formats. These controls include wavelength correction, a blank control (e.g., a dry well or well containing ELISA buffer), an S0 negative control (no standard or any form of analyte, e.g., sample, added to the well), a negative matrix control, and a positive control, such as B0 (to assess maximum color development). In some embodiments, a standard curve may also be utilized for quantification purposes.

[0184] In a competitive ELISA format (FIG. 1C), a secondary capture antibody 118 is coated onto a solid phase, such as a well 106 of a plate 104. The secondary capture antibody 118 specifically binds to the capture antibody 116, which in turn specifically binds to the analyte 110 of interest. To the well 106 containing the secondary capture antibody 118, the sample 108, the conjugated analyte 114, and the capture antibody 116 are added, allowing competitive binding to occur. The more analyte 110 it contains, the less of the conjugated analyte 114 remains bound to the capture antibody 116. Detection can then proceed to collect concentration data. Negative and positive controls are available for competitive ELISA formats. These controls include wavelength correction, a blank control (e.g., a dry well or well containing ELISA buffer), a nonspecific binding (NSB) control, a negative matrix control, and a positive control, such as B0 (to assess maximum color development). In some embodiments, a standard curve can be used for quantification purposes.

[0185] Concentration data, such as concentration data for antibodies produced from cell cultures, can be obtained using assays such as those shown in Figures 1A-1C. The concentration data can include multiple concentration data points, which may have been collected over a period of time. In some situations, the concentration data can include data directly related to the concentration of an analyte, control, or solution. In other scenarios, the concentration data can include data related to measurements that are correlated with the concentration of an analyte, control, or solution. For example, as shown in Figure 2, the concentration data can be a time series of measurements of average optical density.

[0186] Many parameters can affect the data obtained from an assay. These assay parameters may relate to the environment in which the assay is performed, the equipment used to perform the assay, or the people involved in performing the assay. Ideally, the assay parameters remain unchanged throughout the period in which the assay is used to obtain concentration data. However, when a large number of concentration data points are obtained from an assay over an extended period of time (e.g., over weeks or months), changes in some assay parameters may be unavoidable. In some cases, these changes may cause significant fluctuations in the concentration data.

[0187] Figure 3 shows an example of deviations in concentration data obtained from an assay caused by changes in assay parameters, specifically, changes in laboratory analysts. As shown, concentration data obtained by one of the analysts during the period from mid-October 2014 to December 2014 deviate significantly from the range of the rest of the concentration data. Such outlier data may suggest potential systemic error related to the assay parameters (in this case, error systematically propagated by the analyst collecting the external data).

[0188] To ensure that concentration data obtained from an assay are accurate, it may be necessary to identify changes in the concentration data and evaluate the underlying cause of the change. In many cases, changes in the concentration data may be the result of changes in assay parameters, as illustrated by the exemplary concentration data shown in Figure 3. In other cases, changes in the concentration data may be the result of a process or reaction that should be further investigated. Regardless of the underlying cause, efficient location and validation of the significance of change points in the concentration data may be an essential step in the analysis of concentration data.

[0189] Method overview The described method can identify change points in multiple concentration data points obtained from an assay. As mentioned above, a change point can be a point at which a statistical change occurs in the concentration data. An example of a change point in a data set is shown in Figure 4.

[0190] Changes in the measured variable are reflected in the data shown in FIG. 4 at a first time point 402, a second time point 404, and a third time point 406. After each of time points 402-406, the statistical characteristics of the data (e.g., median, mean, etc.) appear to change. The methods provided herein can be used to algorithmically identify such change points within a dataset, such as a dataset containing multiple concentration data points obtained from an assay. After the locations of the change points within the dataset have been identified, the method can verify the significance of each change point (block 410). In other words, the method can determine whether the difference between the statistical distribution of the multiple data points before the change point and the statistical distribution of the multiple data points after the change point is large enough to indicate that an error or other event occurred at the time of the change point that merits further investigation. If a change point is determined to be significant, personnel associated with the data (e.g., the laboratory analyst who collected the data) can be notified (block 412) so that the underlying cause of the change in the data can be evaluated. The underlying causes of the changes in the data may include one or more of human factors (block 414), equipment factors (block 416), environmental factors (block 418), and the like.

[0191] 5 shows an example of a method 500 for determining the cause of significant statistical variation in concentration data obtained from an assay. In some embodiments, one or more steps of method 500 may be performed by one or more processors in a system configured to identify and verify change points in a dataset (e.g., by a processor of a computer belonging to a laboratory analyst who assisted in performing the assay).

[0192] As shown, method 500 may include a first step 502 in which multiple instances of an assay (or, in some embodiments, multiple instances of multiple assays) may be performed to obtain multiple concentration data points. The assay performed may be of any type or format known in the art. For example, the assay may be an immunoassay, a competitive assay, a non-competitive assay, a heterogeneous assay, or a homogeneous assay. Optionally, the assay may be an ELISA assay, such as a direct ELISA assay, a sandwich ELISA assay, or a competitive ELISA assay (see FIGS. 1A-1C for a schematic diagram of an ELISA assay).

[0193] The assay may be configured to measure the concentration of an analyte in a sample. Optionally, the analyte may be a therapeutic analyte, a therapeutic polypeptide, an antibody, or a fragment of an antibody. The sample may be a cell culture sample or a derivative thereof.

[0194] In some embodiments, performing multiple instances of an assay may include performing two or more of the multiple instances two or more times. Two or more times may constitute a time course of at least about one day, at least about one week, at least about one month, at least about six months, at least about one year, or at least about five years. In other words, two or more of the multiple concentration data points obtained in step 502 may be obtained at two or more different times in a day, on two or more different days in a week, on two or more different days in a month, or on two or more different days in a year. In some embodiments, performing multiple instances of an assay may include performing two or more of the multiple instances in parallel (i.e., simultaneously).

[0195] The plurality of concentration data points may include data points for the concentration of the target analyte, the absolute amount of the target analyte, the concentration of a control (e.g., a negative control, a non-specific binding control, a blank control, a detection antibody control, a negative matrix control, or a positive control), and / or a solution concentration. In some aspects of method 500, the plurality of concentration data points may include data points for a measurement associated with concentration, such as an optical density (OD) measurement. Optionally, the plurality of concentration data points may include data points for an average, a low reference sample, a medium reference sample, a high reference sample, a low control, a medium control, and a high control. In some embodiments, two or more of the plurality of concentration data points may be in the same format or different formats.

[0196] After obtaining multiple concentration data points from the assay in step 502, method 500 may proceed to step 504, where assay information may be received. The assay information may include multiple concentration data points as well as multiple assay parameters. The assay parameter information may be recorded during each instance of running the assay, and each assay parameter of the multiple assay parameters may be associated with one of the multiple instances of running the assay and may indicate a characteristic or attribute of the assay or a characteristic or attribute of a factor associated with the assay in that instance. For example, assay parameters associated with an instance of running the assay may include environmental factors associated with the environment in which the assay was run (e.g., temperature, humidity, light, or the presence of one or more contaminants), equipment factors associated with the instrument used to run the assay (e.g., reagent change, reagent aging, reagent expiration date, reagent contamination, reagent failure, hardware change, hardware aging, hardware contamination, hardware failure, instrument change, instrument failure, or instrument calibration), or human factors associated with one or more operators who ran or assisted in running the assay (e.g., inter-operator performance variation, performance error by one or more operators, or operator change). In some embodiments, several assay parameters of the plurality of assay parameters may be associated with a single instance of running the assay.

[0197] After the assay information is received in step 504, the method 500 may proceed to step 506, where a significant change point may be identified. A significant change point may correspond to a location in the plurality of concentration data points where one or more statistical characteristics of the plurality of concentration data points change beyond a threshold. A significant change point may be a location between the plurality of concentration data points where the median, mean, variance, and / or correlation of the plurality of concentration data points changes, or where an anomaly in the plurality of concentration data points is identified.

[0198] To identify significant change points, one or more candidate change points can be first identified. The validity of each candidate change point can then be determined by quantifying the statistical change in the concentration data points before and after each candidate change point. If the statistical change in the concentration data points before and after the candidate change point is determined to be significant (e.g., if the quantification of the change exceeds a cutoff value), the candidate change point can be identified as a significant change point.

[0199] Once a significant change-point is identified in step 506, method 500 may move to step 508, where one or more assay parameters of the plurality of assay parameters may be correlated with the identified significant change-point. In some embodiments, one or more assay parameters may be correlated with the significant change-point by identifying an instance of the plurality of instances of performing the assay that corresponds to the location of the significant change-point in the plurality of concentration data points. For example, the significant change-point may be identified at a location in the plurality of concentration data points that corresponds to a date (e.g., day, month, year). The instance of running the assay may be identified based on this date. After the instance of running the assay is identified, one or more assay parameters associated with that instance may be correlated with the significant change-point.

[0200] After one or more assay parameters have been correlated with the identified significant change-point in step 508, method 500 may proceed to step 510, where a cause of the change in one or more statistical properties of the plurality of concentration data points at the significant change-point may be determined based on the correlation between the one or more assay parameters and the significant change-point. For example, if one or more assay parameters indicate that an operator change occurred at or near the time of the significant change-point, determining the cause of the change in one or more statistical properties of the plurality of concentration data points may include determining that an operator who performed the assay before the significant change-point or an operator who performed the assay after the significant change-point may have made an error. Once the root cause of the change has been determined, the root cause of the change may be efficiently and accurately verified.

[0201] The following sections provide additional description of the step of identifying significant change points (step 506 of method 500), specifically, how candidate change points may be identified and how the significance of the candidate change points may be verified.

[0202] Identifying candidate change points After multiple concentration data points have been obtained by running multiple instances of the assay (step 502 of method 500), significant change-points may be identified (step 506 of method 500). As described in the previous section, identifying significant change-points may include identifying one or more candidate change-points. Figure 6 shows an example of a method for identifying one or more candidate change-points using a binary segmentation algorithm.

[0203] The binary segmentation algorithm begins by selecting a first segment of concentration data points from the plurality of concentration data points (step 602). In some embodiments, the first segment of concentration data points may include at least a threshold number (N) of concentration data points. The threshold number of concentration data points may help ensure that only change points that are likely to be significant are identified. In other words, the threshold number of concentration data points may reduce the sensitivity of the binary segmentation algorithm to small, random fluctuations in the plurality of concentration data points that do not indicate significant changes. The threshold number of concentration data points may be provided by a user. Optionally, the threshold number of concentration data points may be determined based on the assay used to obtain the plurality of concentration data points. A first median (m1) associated with the first segment of concentration data points may be determined after the first segment of concentration data points is selected (step 604).

[0204] Next, in step 606, a second segment of concentration data points is selected from the plurality of concentration data points. The concentration data points in the second segment may be contiguous to the concentration data points in the first segment. Like the first segment, the second segment of concentration data points may include at least a threshold number of concentration data points (N) to reduce the sensitivity of the binary segmentation algorithm to insignificant variations. After the second segment of concentration data points is selected, a second median value (m2) associated with the second segment of concentration data points may be determined (step 608).

[0205] Once the first median and second median have been determined, the first median may be compared to the second median to determine whether the candidate change-point is located between the first and second segments of concentration data points (step 610). Comparing the first median to the second median determines whether the difference between the first median and the second median is greater than or equal to the difference between the first median and the second median. The method may involve determining whether the coefficient of variation (CV) of the first segment scaled by a scaling parameter (k), where the coefficient of variation of the first segment is the ratio of the standard deviation (σ) of the first segment to the mean (μ) of the first segment. The scaling parameter (k) may be provided by a user or may depend on the assay being performed to obtain the multiple concentration data points. The scaling parameter may be about 0.1, about 0.2, about 0.3, about 0.4, about 0.5, about 0.6, about 0.7, about 0.8, or about 0.9. In some embodiments, the scaling parameter may be greater than 0.1, greater than 0.2, greater than 0.3, greater than 0.4, greater than 0.5, greater than 0.6, greater than 0.7, greater than 0.8, or greater than 0.9. In some embodiments, the scaling parameter may be less than 0.1, less than 0.2, less than 0.3, less than 0.4, less than 0.5, less than 0.6, less than 0.7, less than 0.8, or less than 0.9.

[0206] If a candidate change-point is identified between the first and second segments in step 610, steps 602-610 may be repeated for the concentration data points in the first segment and the concentration data points in the second segment to determine if additional candidate change-points exist within the first and / or second segments. Several iterations of the binary segmentation method are shown in FIG. 7. As shown, an entire signal 702 (e.g., multiple concentration data points obtained from an assay) is segmented into a first segment of data points 704 and a second segment of data points 706, as described in step 602 shown in FIG. 6. A candidate change-point 712 is identified between the first segment 704 and the second segment 706, as described in steps 604-610 of FIG. 6. The binary segmentation process is then repeated for the data points within the first segment 704. In other words, the first segment 704 is divided into two segments of data 708, 710, and a second candidate change point 714 is identified between the segments 708, 710.

[0207] The binary segmentation process may continue to iterate until one or more stopping conditions are met, for example, until a maximum number of iterations have been performed or until a threshold number of candidate change-points have been identified. Once a stopping condition is met, the binary segmentation process may stop.

[0208] Verifying the significance of candidate change points Once one or more candidate change points have been identified in the plurality of concentration data points (e.g., using the binary segmentation techniques described in FIGS. 6-7), the validity of each candidate change point may be determined. In some embodiments, validating the significance of a candidate change point may require identifying data point clusters (e.g., statistical patterns) in the segment of concentration data points before and after each candidate change point.

[0209] There are many possible methods for generating such data point clusters. Figure 8A describes a first method using a k-means algorithm to generate data point clusters for multiple concentration data points. If candidate change points are identified between a first segment of concentration data points and a second segment of concentration data points (e.g., as described in method 600 shown in Figure 6), one or more mean values ​​may be generated for each segment (step 802). In some embodiments, mean values ​​for a segment may be generated by randomly assigning one or more concentration data points within the segment to be mean values. After the one or more mean values ​​are generated, one or more data point clusters may be generated (step 804). Each of the one or more data point clusters may be associated with the mean value of the one or more mean values ​​generated in step 802. The data point cluster associated with a given mean value may include the concentration data points within the respective segment that are closest to that mean value. The distance of the concentration data points to the mean value may be determined using a norm (e.g., the Euclidean norm). After one or more data point clusters are generated in step 804, the mean value in each data point cluster may be updated by identifying the centroid (i.e., the central concentration data point) of the data point cluster and assigning the centroid to an updated mean value. Steps 804-806 may be repeated iteratively for each segment until the mean value no longer changes during the update step.

[0210] 8B describes an alternative method for generating data point clusters utilizing a Gaussian mixture model. If candidate change points are identified between a first segment of concentration data points and a second segment of concentration data points (e.g., as described in method 600 shown in FIG. 6), one or more data point clusters may be identified for each segment (step 808). In this case, each of the one or more data point clusters may be normally distributed and may have a unique mean and a unique variance. In other words, the one or more data point clusters within each segment may be generated by dividing the concentration data points within each segment into separate normally distributed groups of concentration data points.

[0211] In some embodiments, identifying data point clusters within a segment of concentration data points (e.g., using the method illustrated in FIGS. 8A-8B) may include determining the expected number of data point clusters within each segment. For example, techniques such as the Bayesian Information Criterion may be used to find the number of data point clusters within a segment of concentration data points.

[0212] Exemplary segments and exemplary data point clusters for a plurality of concentration data points are provided in Figures 9A-9B. As shown in Figure 9A, a plurality of concentration data points 902 can be divided into segments 904 separated by candidate change points 906. Within each segment, data point clusters 908a, 908b, and 908c can be generated. Figure 9B illustrates how, when data point clusters 908a-c are generated using the method shown in Figure 8B, each cluster can be uniquely normally distributed, i.e., normally distributed with a unique mean (μ) and a unique standard deviation (σ).

[0213] Once one or more data point clusters within each segment of the plurality of concentration data points have been identified (e.g., by using the k-means algorithm shown in FIG. 8A or the Gaussian mixture model method shown in FIG. 8B), the significance of each candidate change point may be verified. Verifying the significance of a candidate change point may include determining whether one or more statistical characteristics of the plurality of concentration data points change beyond a threshold, as shown in FIG. 8C. First, for each segment of concentration data points, a principal data point cluster of one or more data point clusters within the segment may be identified (step 810). A principal data point cluster in a segment may be a data point cluster that includes at least a threshold percentage of the concentration data points in the segment. For example, a principal data point cluster in a segment may be a data point cluster that includes at least 5%, at least 10%, at least 15%, at least 20%, at least 25%, or at least 30% of the concentration data points in the segment.

[0214] After the main data point clusters in each segment are identified, a deviation value at each candidate change point may be determined (step 812). For a candidate change point located between a first segment and a second segment of concentration data points, the deviation value may be the statistical difference between the main data point cluster in the first segment and the main data point cluster in the second segment. Specifically, the deviation value may be the difference in cluster membership probability distribution between data point cluster membership in the first segment and data point cluster membership in the second segment. The deviation value at a candidate change point may be determined using a method for measuring the similarity between probability distributions. Jensen-Shannon deviation (JSD) is an example of such a method. If P is a probability distribution representing the main data point cluster in the segment located immediately before the candidate change point (first segment) and Q is a probability distribution representing the main data point cluster in the segment immediately after the candidate change point (second segment), the JSD at the candidate change point is defined as follows: TIFF2026505230000003.tif10170 formula: TIFF2026505230000004.tif10170 and TIFF2026505230000005.tif7170 This is the Kullback-Leibler deviation.

[0215] In addition to the deviation values, a median change value may be determined for each candidate change point (step 814). The median change value at a change point may be the difference between the median of the segment located immediately before the candidate change point (e.g., the first median associated with the first segment as described in step 604 of method 600 shown in FIG. 6) and the median of the segment located immediately after the candidate change point (e.g., the second median associated with the second segment as described in step 608 of method 600 shown in FIG. 6). In some embodiments, the median of the segment may be the median of all concentration data points within the segment or the median of the concentration data points within a main data point cluster within the segment.

[0216] The change in the statistical properties before and after each candidate change-point may then be quantified to determine whether one or more statistical properties of the plurality of concentration data points change by more than a threshold (step 816). Quantifying the change in the statistical properties of the plurality of concentration data points before and after the candidate change-point may include determining a loss value that includes a weighted combination of a deviation value at the change-point and a median change value at the change-point. The weighted combination may be characterized by a weight parameter (λ). This weight parameter may be received from a user or may depend on the assay performed to obtain the plurality of concentration data points.

[0217] As previously mentioned, the deviation value at a candidate change point may be determined using the Jensen-Shannon deviation (JSD), in which case the loss value at a candidate change point may be defined as:

number

[0218] If the loss value at the candidate change-point exceeds a threshold, the candidate change-point may be identified as a significant change-point. The threshold may be given by a cut-off parameter (ε). This cut-off parameter may characterize change-point detection sensitivity. The specific circumstances of the assay (e.g., the motivation for performing the assay) may determine the type of change in the multiple concentration data points that the user wants to analyze. For example, if the user wants to investigate only large variations in the multiple concentration data points, the user may use a large cut-off parameter. Alternatively, if the user wants to investigate less obvious variations in the multiple concentration data points, the user may use a smaller cut-off parameter.

[0219] Determining the cause of change After a significant change-point is identified (step 506 of method 500), a cause for the change in one or more statistical properties of the plurality of concentration data points at the identified significant change-point may be determined based on a correlation between one or more assay parameters and the significant change-point (step 510 of method 500). Figure 11 illustrates an optional method for proceeding after a cause for significant statistical change in a plurality of concentration data points obtained from an assay has been identified.

[0220] As shown, in some embodiments, a second plurality of instances of the assay after determining the cause of the change in one or more statistical properties of the plurality of concentration data points at the identified significant change-point (step 1102) is performed. The second plurality of instances of the assay may be performed using assay parameters that match assay parameters associated with an instance of performing the assay before or after the identified significant change-point. In some embodiments, one or more concentration data points corresponding to instances of the plurality of instances of performing the assay before or after the identified significant change-point may be removed from the plurality of concentration data points (step 1104). Step 1104 may be preferred in situations where the cause of the change is determined to be error (e.g., systematic error in performing the assay).

[0221] System for identifying and validating change points One or more steps of the described methods may be performed by a system or device configured to identify and verify change points in a plurality of concentration data points obtained from an assay. An example of a system 1200 for determining the cause of significant statistical change in a plurality of concentration data points obtained from an assay is shown in Figure 12. The system 1200 may comprise a memory 1202 coupled to one or more processors 1204. The memory 1202 may store instructions that, when executed by the processor 1204, cause the processor 1204 to identify and verify change points in the plurality of concentration data points obtained from the assay.

[0222] System 1200 may be configured to receive a plurality of concentration data points from concentration data storage 1206 (e.g., from one or more computers used by a laboratory analyst to store a plurality of concentration data points obtained from an assay). Additionally, system 1200 may be configured to receive assay parameter data from assay parameter storage 1208. The assay parameter data may include information about assay parameters (e.g., environmental factors, instrument factors, and / or human factors) associated with a plurality of concentration data points obtained from an assay.

[0223] In some embodiments, system 1200 may be coupled to a user interface 1210. Optionally, user interface 1210 may be a component of system 1200. Once processor 1204 identifies and verifies a significant change-point in the plurality of concentration data points, processor 1204 may cause user interface 1210 to output information regarding the significant change-point. For example, processor 1204 may be configured to cause user interface 1210 to display one or more plots of the plurality of concentration data points and indicate the location of the significant change-point on the plot. If assay parameter data corresponding to the significant change-point is available, processor 1204 may be configured to cause user interface 1210 to provide the corresponding assay parameter data to a user so that the user can use the assay parameter data to evaluate the underlying causes of the changes in the plurality of concentration data points.

[0224] In some embodiments, a system for identifying and validating change points in a plurality of concentration data points may be or may comprise a computer system. FIG. 13 illustrates an example of a computing system. A computer 1300 may be involved in performing one or more of the methods described herein. The computer 1300 may be a host computer connected to a network. The computer 1300 may be a client computer or a server. As illustrated in FIG. 13, the computer 1300 may be any suitable type of microprocessor-based device, such as a personal computer, a workstation, a server, or a handheld computing device such as a phone or tablet. The computer may include, for example, one or more of a processor 1310, an input device 1320, an output device 1330, storage 1340, and a communication device 1360. The input device 1320 and the output device 1330 may correspond to those described above and may be connectable to or integrated with the computer.

[0225] The input device(s) 1320 may be any suitable device that provides input, such as a touchscreen or monitor, a keyboard, a mouse, or a voice recognition device. The output device(s) 1330 may be any suitable device that provides output, such as a touchscreen, a monitor, a printer, a disk drive, or a speaker.

[0226] Storage 1340 may be any suitable device providing storage, such as electrical, magnetic, or optical memory, including random access memory (RAM), cache, hard drive, CD-ROM drive, tape drive, or removable storage disk. Communication device 1360 may include any suitable device capable of sending and receiving signals over a network, such as a network interface chip or device. Computer components may be connected in any suitable manner, such as by a physical bus or wirelessly. Storage 1340 may be a non-transitory computer-readable storage medium containing one or more programs that, when executed by one or more processors, such as processor 1310, cause the one or more processors to perform methods described herein.

[0227] Software 1350, which may be stored in storage 1340 and executed by processor 1310, may include, for example, programming that embodies the functionality of the present disclosure (e.g., as implemented in the systems, computers, servers, and / or devices described above). In one or more examples, software 1350 may include a combination of servers, such as an application server and a database server.

[0228] The software 1350 may also be stored and / or carried in any computer-readable storage medium for use by or in connection with an instruction execution system, apparatus, or device, such as those described above, that can fetch and execute instructions associated with the software from the instruction execution system, apparatus, or device. In the context of this disclosure, a computer-readable storage medium may be any medium, such as storage 1340, that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0229] The software 1350 may also be propagated in any transmission medium for use by or in connection with an instruction execution system, apparatus, or device, such as those described above, that can fetch and execute instructions associated with the software from the instruction execution system, apparatus, or device. In the context of this disclosure, a transmission medium may be any medium that can communicate, propagate, or transmit programming for use by or in connection with an instruction execution system, apparatus, or device. Transmission-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, or infrared wired or wireless propagation media.

[0230] The computer 1300 may be connected to a network, which may be any suitable type of interconnected communication system. The network may implement any suitable communication protocol and may be protected by any suitable security protocol. The network may include any suitable arrangement of network links capable of implementing the transmission and reception of network signals, such as a wireless network connection (T1 or T3 line), a cable network, DSL, or telephone lines.

[0231] Computer 1300 may implement any operating system suitable for operating on a network. Software 1350 may be written in any suitable programming language, such as C, C++, Java, or Python. In various embodiments, application software embodying functionality of the present disclosure may be deployed in different configurations, such as in a client / server configuration, or via a web browser as a web-based application or web service, etc.

[0232] Examples of significant change points Various examples of significant change points in a plurality of concentration data points identified using the provided methods are provided in Figures 14A-14J. As shown, significant change points in a plurality of concentration data points can be locations in the plurality of concentration data points where the statistical characteristics of the distributed concentration data points change or shift significantly. The disclosed methods allow for efficient and accurate location of such change points in concentration data obtained from an assay, even when the data is noisy and irregular.

[0233] Illustrative Embodiments Embodiments disclosed herein may include the following. 1. A method for determining the cause of significant statistical variation in multiple concentration data points obtained from an assay, comprising: running multiple instances of the assay to obtain multiple concentration data points; receiving assay information including a plurality of concentration data points and a plurality of assay parameters, each assay parameter of the plurality of assay parameters being associated with one instance of a plurality of instances of performing the assay; identifying significant change points corresponding to locations in the plurality of concentration data points where one or more statistical characteristics of the plurality of concentration data points change by more than a threshold; identifying one instance of the plurality of instances of performing the assay that corresponds to the location of a significant change-point in the plurality of concentration data points, thereby correlating one or more assay parameters of the plurality of assay parameters with the identified significant change-point; determining a cause for the change in one or more statistical properties of the plurality of concentration data points at the identified significant change-points based on a correlation between the one or more assay parameters and the significant change-points; A method comprising: 2. The method of embodiment 1, wherein the assay is configured to measure the concentration of an analyte in the sample. 3. The method of embodiment 2, wherein the analyte is a therapeutic analyte. 4. The method of embodiment 2 or 3, wherein the analyte is a therapeutic polypeptide. 5. The method of any one of embodiments 2 to 4, wherein the analyte is an antibody or a fragment thereof. 6. The method of any one of embodiments 2 to 5, wherein the sample is a cell culture sample or a derivative thereof. 7. The method of any one of embodiments 1 to 7, wherein the assay is an immunoassay. 8. The method of any one of embodiments 1 to 7, wherein the assay is a competitive assay. 9. The method of any one of embodiments 1 to 7, wherein the assay is a non-competitive assay. 10. The method of any one of embodiments 1 to 7, wherein the assay is a non-homogeneous assay. 11. The method of any one of embodiments 1 to 7, wherein the assay is a homogeneous assay. 12. The method of any one of embodiments 1 to 10, wherein the assay is an ELISA assay. 13. The method of embodiment 12, wherein the ELISA assay is a direct ELISA assay. 14. The method of embodiment 12, wherein the ELISA assay is a sandwich ELISA assay. 15. The method of embodiment 12, wherein the ELISA assay is a competitive ELISA assay. 16. The method of any one of embodiments 1 to 15, wherein performing a plurality of instances of the assay comprises performing two or more of the plurality of instances two or more times. 17. The method of embodiment 16, wherein the two or more times constitute a time course of at least about one week. 18. The method of any one of embodiments 1 to 17, wherein running multiple instances of the assay comprises running two or more of the multiple instances simultaneously. 19. The method of any one of embodiments 1 to 18, wherein the plurality of concentration data points comprises data points relating to the concentration of a target analyte. 20. The method of any one of embodiments 1 to 19, wherein the plurality of concentration data points includes a data point for a control concentration. 21. The method of embodiment 20, wherein the control is a negative control, a non-specific binding control, a blank control, a detection antibody control, a negative matrix control, or a positive control. 22. The method of any one of embodiments 1 to 21, wherein the plurality of concentration data points comprises data points relating to solution concentration. 23. The method of any one of embodiments 1 to 22, wherein the plurality of concentration data points comprises data points relating to the absolute amount of the target analyte. 24. The method of any one of embodiments 1 to 23, wherein the plurality of concentration data points includes data points relating to measurements associated with concentrations. 25. The method of embodiment 24, wherein the measurement associated with concentration is an optical density (OD) measurement. 26. The method of any one of embodiments 1 to 25, wherein the plurality of concentration data points comprises a data point for the mean, lowest standard deviation mean, highest standard deviation mean, or median control concentration. 27. The method of any one of embodiments 1-26, wherein the significant change point reflects inter-assay variability. 28. The method of any one of embodiments 1 to 27, wherein the significant change point reflects intra-assay variability. 29. The method of any one of embodiments 1 to 28, wherein two or more concentration data points of the plurality of concentration data points are in the same format. 30. The method of any one of embodiments 1 to 29, wherein the one or more assay parameters correlated with significant change-points include environmental factors. 31. The method of embodiment 30, wherein the environmental factors include temperature, humidity, light, or pollutants. 32. The method of any one of embodiments 1 to 31, wherein the one or more assay parameters correlated with significant change points include instrument factors. 33. The method of embodiment 32, wherein the equipment factor comprises a reagent change, reagent aging, reagent expiration date, reagent contamination, reagent failure, hardware change, hardware aging, hardware contamination, hardware failure, meter change, meter failure, or meter calibration. 34. The method of any one of embodiments 1 to 33, wherein the one or more assay parameters correlated with the significant change-points include human factors associated with one or more humans who performed or assisted in performing the assay. 35. The method of embodiment 34, wherein the human factors include performance variation, performance error, or operator change. 36. The method of any one of embodiments 1 to 35, wherein identifying significant change-points comprises determining an expected change-point population in the plurality of concentration data points. 37. Identifying significant change points is selecting a first segment of concentration data points from the plurality of concentration data points; determining a first median value associated with a first segment of concentration data points; selecting a second segment of concentration data points from the plurality of concentration data points, wherein the concentration data points in the second segment are contiguous with the concentration data points in the first segment; determining a second median value associated with a second segment of concentration data points; comparing the first median to the second median; determining whether the candidate change-point is located between the first segment and the second segment based on a comparison between the first median and the second median; 37. The method of any one of embodiments 1 to 36, comprising: 38. The method of embodiment 37, wherein the first segment and the second segment include at least a threshold number of concentration data points. 39. The method of embodiment 38, comprising receiving a threshold number of concentration data points from a user. 40. The method of embodiment 38, wherein the threshold number of concentration data points is determined based on the assay. 41. With respect to the first and second segments: generating one or more average values; generating one or more data point clusters, each data point cluster associated with one of the one or more average values ​​and including a concentration data point in the segment that is closest to the associated average value; updating a mean value in each data point cluster of one or more data point clusters, where updating the mean value in a data point cluster includes identifying a centroid of the data point cluster; iteratively repeating the steps of generating one or more data point clusters and updating the mean value in each data point cluster until the mean value in each data point cluster no longer changes; 41. The method of any one of embodiments 37 to 40, comprising: 42. The method of any one of embodiments 37 to 40, comprising generating one or more data point clusters within each of the first segment and the second segment, wherein the concentration data points within each data point cluster are normally distributed and have a unique mean value and a unique standard deviation value. 43. The method of embodiment 41 or 42, comprising identifying a principal data point cluster of the one or more data point clusters for the first segment and the second segment, the principal data point cluster comprising at least a threshold percentage of the total number of concentration data points in the segment. 44. In some embodiments, the method of embodiment 43 includes determining a deviation value for the candidate change point, the deviation value being a statistical difference between a main data point cluster in the first segment and a main data point cluster in the second segment. 45. The method of embodiment 44, wherein the deviation value is a Jensen-Shannon deviation. 46. ​​The method of any one of embodiments 43 to 45, comprising determining a median change value for the candidate change points, the median change value being the difference between a first median value associated with the first segment and a second median value associated with the second segment. 47. The method of embodiment 46, comprising determining whether one or more statistical characteristics of the plurality of concentration data points change by more than a threshold value by determining a weighted combination of a deviation value and a median change value. 48. The method of embodiment 47, wherein the weighted combination of the deviation value and the median change value is characterized by a weight parameter. 49. The method of embodiment 48, comprising receiving weight parameters from a user. 50. The method of embodiment 48 or 49, wherein the weighting parameters are determined based on an assay. 51. The method of any one of embodiments 1 to 50, comprising performing a second plurality of instances of the assay after determining the cause of the change in one or more statistical properties of the plurality of concentration data points at the identified significant change-point. 52. The method of embodiment 50, wherein the second plurality of instances of the assay are performed using assay parameters that match one or more assay parameters correlated with the identified significant change points. 53. The method of embodiment 50, wherein the second plurality of instances of the assay are performed using assay parameters that match assay parameters associated with instances of performing the assay that occurred before the identified significant change point. 54. The method of any one of embodiments 1 to 53, comprising deleting one or more concentration data points of the plurality of concentration data points corresponding to instances of the plurality of instances of performing the assay that occurred after the identified significant change point. 55. A system for determining the cause of significant statistical variation in a plurality of concentration data points obtained from an assay, comprising one or more processors, the processors comprising: receiving assay information including a plurality of concentration data points and a plurality of assay parameters obtained by performing a plurality of instances of the assay, each assay parameter of the plurality of assay parameters being associated with one instance of the plurality of instances of performing the assay; identifying significant change points corresponding to locations within the plurality of concentration data points where one or more statistical characteristics of the plurality of concentration data points change by more than a threshold; identifying one instance of the plurality of instances of performing the assay that corresponds to the location of the significant change-point in the plurality of concentration data points, thereby correlating one or more assay parameters of the plurality of assay parameters with the identified significant change-point; determining a cause for the change in one or more statistical properties of the plurality of concentration data points at the identified significant change-points based on the correlation between the one or more assay parameters and the significant change-points; The system is configured as follows: 56. The system of embodiment 55, wherein the assay is configured to measure the concentration of an analyte in a sample. 57. The system of embodiment 56, wherein the analyte is a therapeutic analyte. 58. The system of embodiment 56 or 57, wherein the analyte is a therapeutic polypeptide. 59. The system of any one of embodiments 56 to 58, wherein the analyte is an antibody or a fragment thereof. 60. The system of any one of embodiments 56 to 59, wherein the sample is a cell culture sample or a derivative thereof. 61. The system of any one of embodiments 55 to 60, wherein the assay is an immunoassay. 62. The system of any one of embodiments 55 to 60, wherein the assay is a competitive assay. 63. The system of any one of embodiments 55 to 60, wherein the assay is a non-competitive assay. 64. The system of any one of embodiments 55 to 60, wherein the assay is a heterogeneous assay. 65. The system of any one of embodiments 55 to 60, wherein the assay is a homogeneous assay. 66. The system of any one of embodiments 55 to 65, wherein the assay is an ELISA assay. 67. The system of embodiment 66, wherein the ELISA assay is a direct ELISA assay. 68. The system of embodiment 66, wherein the ELISA assay is a sandwich ELISA assay. 69. The system of embodiment 66, wherein the ELISA assay is a competitive ELISA. 70. The system of any one of embodiments 55 to 69, wherein running multiple instances of the assay comprises running two or more of the multiple instances two or more times. 71. The system of embodiment 70, wherein the two or more times constitute a time course of at least about one week. 72. The system of any one of embodiments 55 to 71, wherein running multiple instances of the assay includes running two or more of the multiple instances simultaneously. 73. The system of any one of embodiments 55 to 72, wherein the plurality of concentration data points includes data points relating to the concentration of a target analyte. 74. The system of any one of embodiments 55 to 73, wherein the plurality of concentration data points includes a data point relating to a control concentration. 75. The system of embodiment 74, wherein the control is a negative control, a non-specific binding control, a blank control, a detection antibody control, a negative matrix control, or a positive control. 76. The system of any one of embodiments 55 to 75, wherein the plurality of concentration data points includes data points relating to solution concentration. 77. The system of any one of embodiments 55 to 76, wherein the plurality of concentration data points includes data points relating to the absolute amount of the target analyte. 78. The system of any one of embodiments 55 to 77, wherein the plurality of concentration data points includes data points relating to measurements associated with concentrations. 79. The system of embodiment 78, wherein the measurement associated with concentration is an optical density (OD) measurement. 80. The system of any one of embodiments 55 to 79, wherein the plurality of concentration data points includes a data point for the mean, lowest standard deviation mean, highest standard deviation mean, or median control concentration. 81. The system of any one of embodiments 55 to 80, wherein the significant change point reflects inter-assay variability. 82. The system of any one of embodiments 55 to 81, wherein the significant change point reflects intra-assay variability. 83. The system of any one of embodiments 55 to 82, wherein two or more concentration data points of the plurality of concentration data points are in the same format. 84. The system of any one of embodiments 55 to 83, wherein the one or more assay parameters correlated with significant change points include environmental factors. 85. The system of embodiment 84, wherein the environmental factors include temperature, humidity, light, or pollutants. 86. The system of any one of embodiments 55 to 85, wherein the one or more assay parameters correlated with significant change points include instrument factors. 87. The system of embodiment 86, wherein the equipment factor includes a reagent change, reagent aging, reagent expiration date, reagent contamination, reagent failure, hardware change, hardware aging, hardware contamination, hardware failure, meter change, meter failure, or meter calibration. 88. The system of any one of embodiments 55 to 87, wherein the one or more assay parameters correlated with the significant change point include human factors associated with one or more humans who performed or assisted in performing the assay. 89. The system of embodiment 88, wherein the human factors include performance variation, performance error, or operator change. 90. The system of any one of embodiments 55 to 89, wherein identifying significant change-points includes determining an expected change-point population in the plurality of concentration data points. 91. Identifying significant change points selecting a first segment of concentration data points from the plurality of concentration data points; determining a first median value associated with a first segment of concentration data points; selecting a second segment of concentration data points from the plurality of concentration data points, wherein the concentration data points in the second segment are contiguous with the concentration data points in the first segment; determining a second median value associated with said second segment of concentration data points; comparing the first median value to the second median value; determining whether the candidate change-point is located between the first segment and the second segment based on a comparison between the first median and the second median; 91. The system of any one of embodiments 55 to 90, comprising: 92. The system of embodiment 91, wherein the first segment and the second segment include at least a threshold number of concentration data points. 93. The system of embodiment 92, wherein the one or more processors are configured to receive a threshold number of concentration data points from a user. 94. The system of embodiment 93, wherein the threshold number of concentration data points is determined based on the assay. 95. One or more processors may, with respect to the first segment and the second segment: Generate one or more average values, generating one or more data point clusters, each data point cluster associated with one of the one or more mean values ​​and including the concentration data point in the segment closest to the associated mean value; updating a mean value for each data point cluster of the one or more data point clusters, and updating a mean value for a data point cluster includes identifying a centroid of the data point cluster; iteratively repeating the steps of generating one or more data point clusters and updating the mean value for each data point cluster until the mean value for each data point cluster no longer changes; 95. The system of any one of embodiments 91 to 94, configured as follows: 96. The system of any one of embodiments 91 to 94, wherein the one or more processors are configured to generate one or more data point clusters within each of the first segment and the second segment, and the concentration data points within each data point cluster are normally distributed and have a unique mean value and a unique standard deviation value. 97. The system of embodiment 95 or 96, wherein the one or more processors are configured to identify a principal data point cluster of the one or more data point clusters with respect to the first segment and the second segment, the principal data point cluster comprising at least a threshold percentage of the total number of concentration data points in the segment. 98. The system of embodiment 97, wherein the one or more processors are configured to determine a deviation value for the candidate change point, the deviation value being a statistical difference between a primary data point cluster in the first segment and a primary data point cluster in the second segment. 99. The system of embodiment 98, wherein the deviation value is a Jensen-Shannon deviation. 100. The system of any one of embodiments 97 to 99, wherein the one or more processors are configured to determine, for the candidate change points, a median change value, the median change value being a difference between a first median value associated with the first segment and a second median value associated with the second segment. 101. The system of embodiment 100, wherein the one or more processors are configured to determine whether one or more statistical characteristics of the plurality of concentration data points change beyond a threshold by determining a weighted combination of a deviation value and a median change value. 102. The system of embodiment 101, wherein the weighted combination of the deviation value and the median change value is characterized by a weight parameter. 103. The system of embodiment 102, wherein the one or more processors are configured to receive weight parameters from a user. 104. The system of embodiment 102 or 103, wherein the weighting parameters are determined based on an assay. 105. The system of any one of embodiments 55 to 104, wherein the one or more processors are configured to delete one or more concentration data points of the plurality of concentration data points corresponding to instances of the plurality of instances of performing the assay that occurred after the identified significant change point. 106. A non-transitory computer-readable storage medium storing instructions for determining causes of significant statistical variation in a plurality of concentration data points obtained from an assay, the instructions being executable by one or more processors of an electronic device to cause the device to: receiving assay information including a plurality of concentration data points and a plurality of assay parameters obtained by performing a plurality of instances of the assay, each assay parameter of the plurality of assay parameters being associated with one instance of the plurality of instances of performing the assay; identifying significant change points corresponding to locations in the plurality of concentration data points where one or more statistical characteristics of the plurality of concentration data points change by more than a threshold; identifying one instance of the plurality of instances of performing the assay that corresponds to the location of the significant change-point in the plurality of concentration data points, thereby correlating one or more assay parameters of the plurality of assay parameters with the identified significant change-point; A non-transitory computer-readable storage medium for determining a cause for a change in one or more statistical properties of a plurality of concentration data points at an identified significant change-point based on a correlation between one or more assay parameters and the significant change-point. 107. The non-transitory computer-readable storage medium of embodiment 106, wherein the assay is configured to measure the concentration of an analyte in the sample. 108. The non-transitory computer-readable storage medium of embodiment 107, wherein the analyte is a therapeutic analyte. 109. The non-transitory computer-readable storage medium of embodiment 107 or 108, wherein the analyte is a therapeutic polypeptide. 110. The non-transitory computer-readable storage medium of any one of embodiments 107 to 109, wherein the analyte is an antibody or a fragment thereof. 111. The non-transitory computer-readable storage medium of any one of embodiments 107 to 110, wherein the sample is a cell culture sample or a derivative thereof. 112. The non-transitory computer-readable storage medium of any one of embodiments 106 to 111, wherein the assay is an immunoassay. 113. The non-transitory computer-readable storage medium of any one of embodiments 106 to 111, wherein the assay is a competitive assay. 114. The non-transitory computer-readable storage medium of any one of embodiments 106 to 111, wherein the assay is a non-competitive assay. 115. The non-transitory computer-readable storage medium of any one of embodiments 106 to 111, wherein the assay is a non-homogeneous assay. 116. The non-transitory computer-readable storage medium of any one of embodiments 106 to 111, wherein the assay is a homogeneous assay. 117. The non-transitory computer-readable storage medium of any one of embodiments 106 to 116, wherein the assay is an ELISA assay. 118. The non-transitory computer-readable storage medium of embodiment 117, wherein the ELISA assay is a direct ELISA assay. 119. The non-transitory computer-readable storage medium of embodiment 117, wherein the ELISA assay is a sandwich ELISA assay. 120. The non-transitory computer-readable storage medium of embodiment 117, wherein the ELISA assay is a competitive ELISA assay. 121. The non-transitory computer-readable storage medium of any one of embodiments 106 to 120, wherein running multiple instances of the assay includes running two or more of the multiple instances two or more times. 122. The non-transitory computer-readable storage medium of embodiment 121, wherein the two or more times constitute a time course of at least about one week. 123. The non-transitory computer-readable storage medium of any one of embodiments 106 to 122, wherein running multiple instances of the assay includes running two or more of the multiple instances simultaneously. 124. The non-transitory computer-readable storage medium of any one of embodiments 106 to 123, wherein the plurality of concentration data points includes data points relating to the concentration of a target analyte. 125. The non-transitory computer-readable storage medium of any one of embodiments 106 to 124, wherein the plurality of concentration data points includes data points relating to the concentration of a control. 126. The non-transitory computer-readable storage medium of embodiment 125, wherein the control is a negative control, a non-specific binding control, a blank control, a detection antibody control, a negative matrix control, or a positive control. 127. The non-transitory computer-readable storage medium of any one of embodiments 106 to 126, wherein the plurality of concentration data points includes data points relating to solution concentration. 128. The non-transitory computer-readable storage medium of any one of embodiments 106 to 127, wherein the plurality of concentration data points includes data points relating to absolute amounts of the target analytes. 129. The non-transitory computer-readable storage medium of any one of embodiments 106 to 128, wherein the plurality of concentration data points includes data points relating to measurements associated with concentrations. 130. The non-transitory computer-readable storage medium of embodiment 129, wherein the measurement associated with the concentration is an optical density (OD) measurement. 131. The non-transitory computer-readable storage medium of any one of embodiments 106 to 130, wherein the plurality of concentration data points comprises a data point for the mean, lowest standard deviation mean, highest standard deviation mean, or median control concentration. 132. The non-transitory computer-readable storage medium of any one of embodiments 106 to 131, wherein the significant change point reflects inter-assay variability. 133. The non-transitory computer-readable storage medium of any one of embodiments 106 to 132, wherein the significant change point reflects intra-assay variability. 134. The non-transitory computer-readable storage medium of any one of embodiments 106 to 133, wherein two or more concentration data points of the plurality of concentration data points are in the same format. 135. The non-transitory computer-readable storage medium of any one of embodiments 106 to 134, wherein the one or more assay parameters correlated with the significant change-point include an environmental factor. 136. The non-transitory computer-readable storage medium of embodiment 135, wherein the environmental factors include temperature, humidity, light, or pollutants. 137. The non-transitory computer-readable storage medium of any one of embodiments 106 to 136, wherein the one or more assay parameters correlated with the significant change-point include an instrument factor. 138. The non-transitory computer-readable storage medium of embodiment 137, wherein the equipment factor includes a reagent change, reagent aging, reagent expiration date, reagent contamination, reagent failure, hardware change, hardware aging, hardware contamination, hardware failure, meter change, meter failure, or meter calibration. 139. The non-transitory computer-readable storage medium of any one of embodiments 106 to 138, wherein one or more assay parameters correlated with significant change points include human factors associated with one or more humans who performed or assisted in performing the assay. 140. The non-transitory computer-readable storage medium of embodiment 139, wherein the human factors include performance variations, performance errors, or operator changes. 141. The non-transitory computer-readable storage medium of any one of embodiments 106 to 140, wherein identifying significant change-points includes determining an expected change-point population in the plurality of concentration data points. 142. Identifying significant change points selecting a first segment of concentration data points from the plurality of concentration data points; determining a first median value associated with a first segment of concentration data points; selecting a second segment of concentration data points from the plurality of concentration data points, wherein the concentration data points in the second segment are contiguous with the concentration data points in the first segment; determining a second median value associated with said second segment of concentration data points; comparing the first median value to the second median value; determining whether the candidate change-point is located between the first segment and the second segment based on a comparison between the first median and the second median; 142. The non-transitory computer-readable storage medium of any one of embodiments 106 to 141, comprising: 143. The non-transitory computer-readable storage medium of embodiment 142, wherein the first segment and the second segment include at least a threshold number of concentration data points. 144. The non-transitory computer-readable storage medium of embodiment 143, wherein the instructions, when executed by one or more processors of the electronic device, are configured to cause the electronic device to receive a threshold number of concentration data points from a user. 145. The non-transitory computer-readable storage medium of embodiment 144, wherein a threshold number of concentration data points is determined based on the assay. 146. The instructions, when executed by one or more processors of an electronic device, cause the device to: generating one or more average values; generating one or more data point clusters, each data point cluster associated with one of the one or more mean values ​​and including a concentration data point within the segment closest to the associated mean value; updating a mean value for each data point cluster of one or more data point clusters, wherein updating the mean value for a data point cluster includes identifying a centroid for the data point cluster; repeatedly repeating the steps of generating one or more data point clusters and updating the mean value for each data point cluster until the mean value for each data point cluster no longer changes; 146. The non-transitory computer-readable storage medium of any one of embodiments 142 to 145. 147. The non-transitory computer-readable storage medium of any one of embodiments 142 to 145, wherein the instructions, when executed by one or more processors of the electronic device, are configured to cause the device to generate one or more data point clusters within each of the first segment and the second segment, and the concentration data points within each data point cluster are normally distributed and have a unique mean value and a unique standard deviation value. 148. The non-transitory computer-readable storage medium of embodiment 146 or 147, wherein the instructions, when executed by one or more processors of the electronic device, are configured to cause the device to identify, for the first segment and the second segment, a principal data point cluster of the one or more data point clusters, the principal data point cluster comprising at least a threshold percentage of the total number of concentration data points in the segment. 149. The non-transitory computer-readable storage medium of embodiment 148, wherein the instructions, when executed by one or more processors of the electronic device, are configured to cause the device to determine a deviation value for the candidate change point, the deviation value being a statistical difference between a primary data point cluster in a first segment and a primary data point cluster in a second segment. 150. The non-transitory computer-readable storage medium of embodiment 149, wherein the deviation value is a Jensen-Shannon deviation. 151. The non-transitory computer-readable storage medium of any one of embodiments 148 to 150, wherein the instructions, when executed by one or more processors of an electronic device, are configured to cause the device to determine a median change value for the candidate change points, the median change value being a difference between a first median value associated with the first segment and a second median value associated with the second segment. 152. The non-transitory computer-readable storage medium of embodiment 151, wherein the instructions, when executed by one or more processors of an electronic device, are configured to cause the device to determine whether one or more statistical characteristics of the plurality of concentration data points change beyond a threshold by determining a weighted combination of a deviation value and a median change value. 153. The non-transitory computer-readable storage medium of embodiment 152, wherein the weighted combination of the deviation value and the median change value is characterized by a weight parameter. 154. The non-transitory computer-readable storage medium of embodiment 153, wherein the instructions, when executed by one or more processors of an electronic device, are configured to cause the device to receive weight parameters from a user. 155. The non-transitory computer-readable storage medium of embodiment 153 or 154, wherein the weighting parameters are determined based on an assay. 156. The non-transitory computer-readable storage medium of any one of embodiments 106 to 155, wherein the instructions, when executed by one or more processors of the electronic device, are configured to cause the device to delete one or more concentration data points of the plurality of concentration data points that correspond to instances of the plurality of instances of performing the assay that occurred after the identified significant change point.

[0234] The description presents only preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the description of preferred exemplary embodiments provides those skilled in the art with an enabling description for practicing various embodiments. It will be understood that various changes can be made in the function and arrangement of elements without departing from the spirit and scope as set forth in the appended claims.

[0235] In the description, specific details are set forth to provide a thorough understanding of the embodiments. However, it will be understood that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order to avoid obscuring the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.

[0236] conclusion The foregoing description has been set forth in connection with specific embodiments and / or examples for purposes of explanation. However, the illustrative discussion above is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. The embodiments were chosen and described to best explain the principles of the technology and their practical applications so as to enable others skilled in the art to best utilize the technology and various embodiments with various modifications suited to the particular use contemplated.

[0237] Although the present disclosure and examples have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will become apparent to those skilled in the art. Such changes and modifications are to be understood as being included within the scope of the present disclosure and examples as defined by the claims. Finally, the entire disclosures of the patents and publications referenced in this application are incorporated herein by reference.

[0238] Any of the systems, methods, techniques, and / or features disclosed herein may be combined, in whole or in part, with any other system, method, technique, and / or feature disclosed herein.

Claims

1. 1. A system for determining the cause of significant statistical variation in a plurality of concentration data points obtained from an assay, comprising: one or more processors, said processors comprising: receiving assay information including the plurality of concentration data points and a plurality of assay parameters obtained by performing a plurality of instances of the assay, each assay parameter of the plurality of assay parameters being associated with one instance of the plurality of instances of performing the assay; identifying significant change points corresponding to locations in the plurality of concentration data points where one or more statistical characteristics of the plurality of concentration data points change by more than a threshold; correlating one or more assay parameters of the plurality of assay parameters with the identified significant change-point by identifying one instance of the plurality of instances of performing the assay that corresponds to the location of the significant change-point in the plurality of concentration data points; determining the cause of the change in the one or more statistical properties of the plurality of concentration data points at the identified significant change-points based on the correlation between the one or more assay parameters and the significant change-points; The system is configured as follows:

2. The system of claim 1 , wherein the assay is configured to measure the concentration of an analyte in a sample.

3. 3. The system of claim 1 or claim 2, wherein the one or more assay parameters correlated with the significant change-points include environmental factors including temperature, humidity, light, or contaminants.

4. 4. The system of claim 1, wherein the one or more assay parameters correlated with significant change points include an instrument factor.

5. 5. The system of claim 4, wherein the equipment factor comprises a reagent change, reagent aging, reagent expiration date, reagent contamination, reagent failure, hardware change, hardware aging, hardware contamination, hardware failure, meter change, meter failure, or meter calibration.

6. 6. The system of claim 1, wherein the one or more assay parameters correlated with the significant change points include human factors associated with one or more humans who performed or assisted in performing the assay.

7. The system of claim 6 , wherein the human factors include performance variations, performance errors, or operator changes.

8. 8. The system of claim 1, wherein identifying the significant change-points comprises determining an expected change-point population in the plurality of concentration data points.

9. identifying the significant change points selecting a first segment of concentration data points from the plurality of concentration data points; determining a first median value associated with the first segment of concentration data points; selecting a second segment of concentration data points from the plurality of concentration data points, wherein the concentration data points in the second segment are contiguous with the concentration data points in the first segment; determining a second median value associated with the second segment of concentration data points; comparing the first median value to the second median value; determining whether a candidate change-point is located between the first segment and the second segment based on the comparison between the first median value and the second median value; The system of any one of claims 1 to 8, comprising:

10. The system of claim 9 , wherein the first segment and the second segment include at least a threshold number of concentration data points.

11. The system of claim 10 , wherein the one or more processors are configured to receive the threshold number of concentration data points from a user.

12. The system of claim 11 , wherein the threshold number of concentration data points is determined based on the assay.

13. The one or more processors, with respect to the first segment and the second segment: generating one or more average values; generating one or more data point clusters, each data point cluster being associated with one of the one or more average values ​​and including a concentration data point within the segment that is closest to the associated average value; updating the mean value for each data point cluster of the one or more data point clusters, and updating the mean value for a data point cluster includes identifying a centroid of the data point cluster; iteratively repeating the steps of generating one or more data point clusters and updating the mean value for each data point cluster until the mean value for each data point cluster no longer changes; 13. The system according to any one of claims 9 to 12, configured to:

14. 13. The system of claim 9, wherein the one or more processors are configured to generate one or more data point clusters within each of the first segment and the second segment, the concentration data points within each data point cluster being normally distributed and having a unique mean value and a unique standard deviation value.

15. 15. The system of claim 13 or claim 14, wherein the one or more processors are configured to identify, for the first segment and the second segment, a principal data point cluster of the one or more data point clusters, the principal data point cluster comprising at least a threshold percentage of a total number of concentration data points in the segment.

16. 16. The system of claim 15, wherein the one or more processors are configured to determine a deviation value for the candidate change point, the deviation value being a statistical difference between the primary data point cluster in the first segment and the primary data point cluster in the second segment.

17. The system of claim 16 , wherein the deviation value is a Jensen-Shannon deviation.

18. 18. The system of claim 15, wherein the one or more processors are configured to determine a median change value for the candidate change points, the median change value being a difference between the first median value associated with the first segment and the second median value associated with the second segment.

19. 20. The system of claim 18, wherein the one or more processors are configured to determine whether the one or more statistical characteristics of the plurality of concentration data points vary by more than a threshold value by determining a weighted combination of the deviation value and the median change value.

20. 20. The system of claim 19, wherein the weighted combination of the deviation value and the median change value is characterized by a weight parameter.

21. The system of claim 20 , wherein the one or more processors are configured to receive the weighting parameters from a user.

22. 22. The system of claim 20 or claim 21, wherein the weighting parameters are determined based on the assay.

23. 23. The system of any one of claims 1-22, wherein the one or more processors are configured to delete one or more concentration data points from the plurality of concentration data points that correspond to instances from the plurality of instances of performing the assay that occurred after the identified significant change point.

Images