Monitoring assay performance

Abstract

Multiple instances of the assay may be made to obtain multiple concentration data points. Next, a significant change point may be identified that corresponds to a location in the plurality of concentration data points at which one or more statistical characteristics of the plurality of concentration data points change beyond a threshold. The point of significant change in the plurality of concentration data points may be correlated with one or more determination parameters associated with the determination by identifying an instance of performing the determination that corresponds to the location of the point of significant change. Based on the correlation, a cause of a change in the one or more statistical characteristics of the plurality of concentration data points at the identified points of significant change may be determined.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 433,598, filed on Dec. 19, 2022, entitled “Monitoring Assay Performance,” the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure generally relates to techniques for monitoring and evaluating the performance of analytical biological procedures. More specifically, the present disclosure relates to methods for identifying and evaluating the significance of changes in data collected during analytical biological procedures. Background Art

[0004] Over time, making repeated measurements of a process or one or more aspects thereof has the potential to provide valuable insights into changes occurring in the process. However, there is currently a lack of techniques for analyzing such data to confidentially identify process changes.

[0005] There are many sources of process data for analyzing process changes. For example, in the production of biotherapeutic agents such as antibodies, immunoassays can be used to track the concentration of biologic agents between different production runs. The production of biotherapeutic agents is very complex and involves many reagents and components (such as living cells), instruments, manufacturing and testing environments, and human operators, all of which are prone to change over time. Careful monitoring of the biotherapeutic agent manufacturing process, including using control monitoring assays, is crucial for the production quality control of such agents. Additional examples can be found in the analysis of digital biomarkers, such as anthropometric measurements taken over time from medical and / or consumer smart devices (e.g., smartwatches). The analysis of these biomarkers generates complex multi - parameter data, including associated information about heart rate, glucose level, blood oxygen content, GPS coordinates, gyroscope data, and environmental conditions.

[0006] Current techniques for evaluating such data, which is typically irregularly spaced, noisy, multi - dimensional, and contains limited ground truth, involve cumbersome manual processes and lack established metrics for identifying process outliers. In addition, current techniques are only able to identify process deviations long after the process has occurred, thus making the correction of problems undesirably slow. Summary of the Invention

[0007] As described above, process data typically includes a time series of repeated measurements of a process. A method is provided for determining the cause of a statistical change in process data by identifying the locations in the process data where significant changes in the data have occurred. The method can allow a laboratory analyst to effectively identify the time periods during which significant shifts in the data have occurred. Once the relevant time periods have been identified, the analyst can investigate the potential causes of the data shift by evaluating the parameters associated with the measurements taken during the identified time periods.

[0008] In some embodiments, the described method can both identify the locations in the process data where changes in the data have occurred and verify the significance of the changes relative to the dataset as a whole. Once the potential locations have been precisely located by an algorithm, the significance of each potential location can be verified by quantifying the statistical change in the data surrounding that location. This quantification of the statistical change can be used to determine whether the change that occurred at that location is significant—and thus, potentially indicative of a measurement error or other cause worthy of evaluation—or not significant (e.g., the result of random statistical fluctuations). By verifying the significance of the identified changes in the process data, in some embodiments, the method provided herein can help an analyst spend time investigating important changes that are likely to affect the outcome of the process being measured.

[0009] An example of a method for determining the cause of a significant statistical change in a plurality of concentration data points obtained from an assay can include performing a plurality of instances of the assay to obtain a plurality of concentration data points, receiving assay information including the plurality of concentration data points and a plurality of assay parameters, wherein each assay parameter of the plurality of assay parameters is associated with one of the plurality of instances of the assay; identifying a significant change point corresponding to a location in the plurality of concentration data points at which one or more statistical properties of the plurality of concentration data points change by more than a threshold; correlating one or more of the assay parameters with the identified significant change point by identifying the instance of the plurality of instances of the assay corresponding to the location of the significant change point in the plurality of concentration data points; and determining the cause of the change in one or more statistical properties of the plurality of concentration data points at the identified significant change point based on the correlation between one or more of the assay parameters and the significant change point.

[0010] In some embodiments of the method, the assay is configured to measure the concentration of an analyte in a sample.

[0011] In some embodiments of the method, the analyte is a therapeutic analyte.

[0012] In some embodiments of the method, the analyte is a therapeutic polypeptide.

[0013] In some embodiments of the present method, the analyte is an antibody or a fragment thereof.

[0014] In some embodiments of the method, the sample is a cell culture sample or a derivative thereof.

[0015] In some embodiments of the method, the assay is an immunoassay.

[0016] In some embodiments of the method, the assay is a competitive assay.

[0017] In some embodiments of the method, the assay is a non-competitive assay.

[0018] In some embodiments of the method, the assay is a heterogeneous assay.

[0019] In some embodiments of the method, the assay is a homogeneous assay.

[0020] In some embodiments of the method, the assay is an ELISA assay.

[0021] In some embodiments of the method, the ELISA assay is a direct ELISA assay.

[0022] In some embodiments of the method, the ELISA assay is a sandwich ELISA assay.

[0023] In some embodiments of the method, the ELISA assay is a competitive ELISA

[0024] In some embodiments of the method, multiple instances of performing the assay include performing two or more of the instances in the multiple instances two or more times.

[0025] In some embodiments of the method, two or more times constitute a time course of at least about one week.

[0026] In some embodiments of the method, multiple instances of performing the assay include performing two or more of the instances in the multiple instances simultaneously.

[0027] In some embodiments of the method, the multiple concentration data points include data points related to the concentration of the target analyte.

[0028] In some embodiments of the method, the multiple concentration data points include data points related to the concentration of a control.

[0029] In some embodiments of the method, the control is a negative control, a non-specific binding control, a blank control, a detection antibody control, a negative matrix control, or a positive control.

[0030] In some embodiments of the method, the multiple concentration data points include data points related to the concentration of a solution.

[0031] In some embodiments of the method, the plurality of concentration data points include data points related to the absolute amount of the target analyte.

[0032] In some embodiments of the method, the plurality of concentration data points include data points related to measurements associated with the concentration.

[0033] In some embodiments of the method, the measurement associated with the concentration is an optical density (OD) measurement.

[0034] In some embodiments of the method, the plurality of concentration data points include data points related to an average value, a lowest standard deviation average value, a highest standard deviation average value, or an intermediate control concentration.

[0035] In some embodiments of the method, the significant change points reflect between-assay variability.

[0036] In some embodiments of the method, the significant change points reflect within-assay variability.

[0037] In some embodiments of the method, two or more of the plurality of concentration data points are in the same format.

[0038] In some embodiments of the method, one or more assay parameters related to the significant change points include environmental factors.

[0039] In some embodiments of the method, the environmental factors include temperature, humidity, light, or contaminants.

[0040] In some embodiments of the method, one or more assay parameters related to the significant change points include equipment factors.

[0041] In some embodiments of the method, the equipment factors include reagent replacement, reagent aging, reagent expiration, reagent contamination, reagent failure, hardware replacement, hardware aging, hardware contamination, hardware failure, instrument replacement, instrument failure, or instrument calibration.

[0042] In some embodiments of the method, one or more assay parameters related to the significant change points include human factors associated with one or more persons performing or assisting in performing the assay.

[0043] In some embodiments of the method, the human factors include performance variability, performance error, or operator replacement.

[0044] In some embodiments of the method, identifying the significant change points includes determining a population of expected change points among the plurality of concentration data points.

[0045] In some embodiments of the method, identifying significant change points includes: selecting a first segment of concentration data points among a plurality of concentration data points; determining a first median associated with the first segment of concentration data points; selecting a second segment of concentration data points among the plurality of concentration data points, wherein the concentration data points in the second segment are consecutive with the concentration data points in the first segment; determining a second median associated with the second segment of concentration data points; comparing the first median with the second median; and based on the comparison between the first median and the second median, determining whether a candidate change point is located between the first segment and the second segment.

[0046] In some embodiments of the method, the first segment and the second segment include at least a threshold number of concentration data points.

[0047] In some embodiments, the method includes receiving a threshold number of concentration data points from a user.

[0048] In some embodiments of the method, the threshold number of concentration data points is determined based on an assay.

[0049] In some embodiments, the method includes: generating one or more averages for the first segment and the second segment; generating one or more clusters of data points, wherein each cluster of data points is associated with an average among the one or more averages and includes the concentration data points in the segment that are closest to the associated average; updating the average of each cluster of data points in the one or more clusters of data points, wherein updating the average of the cluster of data points includes identifying the centroid of the cluster of data points; and iteratively repeating the steps of generating one or more clusters of data points and updating the average of each cluster of data points until the average of each cluster of data points no longer changes.

[0050] In some embodiments, the method includes generating one or more clusters of data points within each of the first segment and the second segment, wherein the concentration data points in each cluster of data points are normally distributed and have a unique average and a unique standard deviation value.

[0051] In some embodiments, the method includes identifying a main cluster of data points among the one or more clusters of data points for the first segment and the second segment, wherein the main cluster of data points includes at least a threshold percentage of the total number of concentration data points in the segment.

[0052] In some embodiments, the method includes determining a divergence value for the candidate change point, wherein the divergence value measures the statistical difference between the main cluster of data points in the first segment and the main cluster of data points in the second segment.

[0053] In some embodiments of the method, the divergence value is the Jensen-Shannon divergence.

[0054] In some embodiments, the method includes determining a median change value for a candidate change point, where the median change value measures the difference between a first median associated with a first segment and a second median associated with a second segment.

[0055] In some embodiments, the method includes determining whether one or more statistical properties of a plurality of concentration data points have changed beyond a threshold by determining a weighted combination of a divergence value and a median change value.

[0056] In some embodiments of the method, the weighted combination of the divergence value and the median change value is characterized by a weight parameter.

[0057] In some embodiments, the method includes receiving a weight parameter from a user.

[0058] In some embodiments of the method, the weight parameter is determined based on an assay.

[0059] In some embodiments, the method includes performing a second plurality of instances of an assay after determining the cause of a change in one or more statistical properties of a plurality of concentration data points at an identified significant change point.

[0060] In some embodiments of the method, the second plurality of instances of the assay are performed using assay parameters that match one or more assay parameters associated with the identified significant change point.

[0061] In some embodiments of the method, the second plurality of instances of the assay are performed using assay parameters that match assay parameters associated with instances of the assay that occurred prior to the identified significant change point.

[0062] In some embodiments, the method includes deleting one or more concentration data points from the plurality of concentration data points, where the one or more concentration data points correspond to instances of the assay that occurred after the identified significant change point among the plurality of instances of the assay.

[0063] Examples of systems for determining the cause of significant statistical variations in a plurality of concentration data points obtained from an assay can include one or more processors configured to receive assay information including a plurality of concentration data points obtained through a plurality of instances of performing the assay and a plurality of assay parameters, where each assay parameter of the plurality of assay parameters is associated with an instance of the plurality of instances of performing the assay, identify a significant change point corresponding to a position among the plurality of concentration data points at which one or more statistical characteristics of the plurality of concentration data points change by more than a threshold; associate one or more of the assay parameters of the plurality of assay parameters with the identified significant change point by identifying an instance among the plurality of instances of performing the assay corresponding to the position of the significant change point among the plurality of concentration data points, and determine the cause of the change in one or more statistical characteristics of the plurality of concentration data points at the identified significant change point based on the correlation between one or more of the assay parameters and the significant change point.

[0064] In some embodiments of the system, the assay is configured to measure the concentration of an analyte in a sample.

[0065] In some embodiments of the system, the analyte is a therapeutic analyte.

[0066] In some embodiments of the system, the analyte is a therapeutic polypeptide.

[0067] In some embodiments of the system, the analyte is an antibody or a fragment thereof.

[0068] In some embodiments of the system, the sample is a cell culture sample or a derivative thereof.

[0069] In some embodiments of the system, the assay is an immunoassay.

[0070] In some embodiments of the system, the assay is a competitive assay.

[0071] In some embodiments of the system, the assay is a non - competitive assay.

[0072] In some embodiments of the system, the assay is a heterogeneous assay.

[0073] In some embodiments of the system, the assay is a homogeneous assay.

[0074] In some embodiments of the system, the assay is an ELISA assay.

[0075] In some embodiments of the system, the ELISA assay is a direct ELISA assay.

[0076] In some embodiments of the system, the ELISA assay is a sandwich ELISA assay.

[0077] In some embodiments of the system, the ELISA assay is a competitive ELISA

[0078] In some embodiments of the system, multiple instances of the assay include two or more of the multiple instances being performed two or more times.

[0079] In some embodiments of the system, two or more times constitute a time course of at least about one week.

[0080] In some embodiments of the system, multiple instances of the assay include two or more of the multiple instances being performed simultaneously.

[0081] In some embodiments of the system, the multiple concentration data points include data points related to the concentration of the target analyte.

[0082] In some embodiments of the system, the multiple concentration data points include data points related to the concentration of a control.

[0083] In some embodiments of the system, the control is a negative control, a non-specific binding control, a blank control, a detection antibody control, a negative matrix control, or a positive control.

[0084] In some embodiments of the system, the multiple concentration data points include data points related to the concentration of a solution.

[0085] In some embodiments of the system, the multiple concentration data points include data points related to the absolute amount of the target analyte.

[0086] In some embodiments of the system, the multiple concentration data points include data points related to measurements associated with the concentration.

[0087] In some embodiments of the system, the measurement associated with the concentration is an optical density (OD) measurement.

[0088] In some embodiments of the system, the multiple concentration data points include data points related to an average value, an average of minimum standard deviations, an average of maximum standard deviations, or an intermediate control concentration.

[0089] In some embodiments of the system, the significant change points reflect inter-assay variability.

[0090] In some embodiments of the system, the significant change points reflect intra-assay variability.

[0091] In some embodiments of the system, two or more of the multiple concentration data points are in the same format.

[0092] In some embodiments of the system, one or more assay parameters related to the significant change points include environmental factors.

[0093] In some embodiments of the system, environmental factors include temperature, humidity, light, or contaminants.

[0094] In some embodiments of the system, one or more measured parameters associated with significant change points include equipment factors.

[0095] In some embodiments of the system, equipment factors include reagent replacement, reagent aging, reagent expiration, reagent contamination, reagent failure, hardware replacement, hardware aging, hardware contamination, hardware failure, instrument replacement, instrument failure, or instrument calibration.

[0096] In some embodiments of the system, one or more measured parameters associated with significant change points include human factors associated with one or more individuals performing or assisting in performing the measurement.

[0097] In some embodiments of the system, human factors include performance variability, performance error, or operator replacement.

[0098] In some embodiments of the system, identifying significant change points includes determining an expected population of change points among a plurality of concentration data points.

[0099] In some embodiments of the system, identifying significant change points includes: selecting a first segment of concentration data points among a plurality of concentration data points; determining a first median associated with the first segment of concentration data points; selecting a second segment of concentration data points among a plurality of concentration data points, wherein the concentration data points in the second segment are consecutive with the concentration data points in the first segment; determining a second median associated with the second segment of concentration data points; comparing the first median with the second median; and based on the comparison between the first median and the second median, determining whether a candidate change point lies between the first segment and the second segment.

[0100] In some embodiments of the system, the first segment and the second segment include at least a threshold number of concentration data points.

[0101] In some embodiments of the system, one or more processors are configured to receive a threshold number of concentration data points from a user.

[0102] In some embodiments of the system, the threshold number of concentration data points is determined based on the measurement.

[0103] In some embodiments of the system, one or more processors are configured to, for a first segment and a second segment: generate one or more average values; generate one or more clusters of data points, where each cluster of data points is associated with an average value among the one or more average values and includes concentration data points in the segment that are closest to the associated average value; update the average value of each cluster of data points among the one or more clusters of data points, where updating the average value of the cluster of data points includes identifying the centroid of the cluster of data points; and iteratively repeat the steps of generating one or more clusters of data points and updating the average value of each cluster of data points until the average value of each cluster of data points no longer changes.

[0104] In some embodiments of the system, one or more processors are configured to generate one or more clusters of data points within each of the first segment and the second segment, where the concentration data points in each cluster of data points are normally distributed and have a unique average value and a unique standard deviation value.

[0105] In some embodiments of the system, one or more processors are configured to identify a primary cluster of data points among the one or more clusters of data points for the first segment and the second segment, where the primary cluster of data points includes at least a threshold percentage of the total number of concentration data points in the segment.

[0106] In some embodiments of the system, one or more processors are configured to determine a divergence value for a candidate change point, where the divergence value measures the statistical difference between the primary cluster of data points in the first segment and the primary cluster of data points in the second segment.

[0107] In some embodiments of the system, the divergence value is the Jensen-Shannon divergence.

[0108] In some embodiments of the system, one or more processors are configured to determine a median change value for a candidate change point, where the median change value measures the difference between a first median associated with the first segment and a second median associated with the second segment.

[0109] In some embodiments of the system, one or more processors are configured to determine whether one or more statistical characteristics of a plurality of concentration data points change by more than a threshold by determining a weighted combination of the divergence value and the median change value.

[0110] In some embodiments of the system, the weighted combination of the divergence value and the median change value is characterized by a weight parameter.

[0111] In some embodiments of the system, one or more processors are configured to receive the weight parameter from a user.

[0112] In some embodiments of the system, the weight parameter is determined based on an assay.

[0113] In some embodiments of the system, one or more processors are configured to delete one or more of the plurality of concentration data points that correspond to instances among the plurality of instances of the assay that occur after the identified significant change point.

[0114] Examples of non - transitory computer - readable storage media may store instructions for determining the cause of a significant statistical change in a plurality of concentration data points obtained from an assay, where the instructions are configured to be executed by one or more processors of an electronic device to cause the device to receive assay information that includes a plurality of concentration data points obtained through a plurality of instances of the assay and a plurality of assay parameters, where each assay parameter among the plurality of assay parameters is associated with an instance among the plurality of instances of the assay, identify a significant change point corresponding to a position among the plurality of concentration data points at which one or more statistical characteristics of the plurality of concentration data points change by more than a threshold; correlate one or more of the assay parameters with the identified significant change point by identifying the instance among the plurality of instances of the assay that corresponds to the position of the significant change point among the plurality of concentration data points, and determine the cause of the change in one or more statistical characteristics of the plurality of concentration data points at the identified significant change point based on the correlation between the one or more assay parameters and the significant change point.

[0115] In some embodiments of the non - transitory computer - readable storage media, the assay is configured to measure the concentration of an analyte in a sample.

[0116] In some embodiments of the non - transitory computer - readable storage media, the analyte is a therapeutic analyte.

[0117] In some embodiments of the non - transitory computer - readable storage media, the analyte is a therapeutic polypeptide.

[0118] In some embodiments of the non - transitory computer - readable storage media, the analyte is an antibody or a fragment thereof.

[0119] In some embodiments of the non - transitory computer - readable storage media, the sample is a cell - culture sample or a derivative thereof.

[0120] In some embodiments of the non - transitory computer - readable storage media, the assay is an immunoassay.

[0121] In some embodiments of the non - transitory computer - readable storage media, the assay is a competitive assay.

[0122] In some embodiments of the non - transitory computer - readable storage media, the assay is a non - competitive assay.

[0123] In some embodiments of the non - transitory computer - readable storage media, the assay is a non - homogeneous assay.

[0124] In some embodiments of the non-transitory computer-readable storage medium, the assay is a homogeneous assay.

[0125] In some embodiments of the non-transitory computer-readable storage medium, the assay is an ELISA assay.

[0126] In some embodiments of the non-transitory computer-readable storage medium, the ELISA assay is a direct ELISA assay.

[0127] In some embodiments of the non-transitory computer-readable storage medium, the ELISA assay is a sandwich ELISA assay.

[0128] In some embodiments of the non-transitory computer-readable storage medium, the ELISA assay is a competitive ELISA assay.

[0129] In some embodiments of the non-transitory computer-readable storage medium, multiple instances of the assay include performing two or more of the multiple instances two or more times.

[0130] In some embodiments of the non-transitory computer-readable storage medium, two or more times constitute a time course of at least about one week.

[0131] In some embodiments of the non-transitory computer-readable storage medium, multiple instances of the assay include performing two or more of the multiple instances simultaneously.

[0132] In some embodiments of the non-transitory computer-readable storage medium, the multiple concentration data points include data points related to the concentration of the target analyte.

[0133] In some embodiments of the non-transitory computer-readable storage medium, the multiple concentration data points include data points related to the concentration of a control.

[0134] In some embodiments of the non-transitory computer-readable storage medium, the control is a negative control, a non-specific binding control, a blank control, a detection antibody control, a negative matrix control, or a positive control.

[0135] In some embodiments of the non-transitory computer-readable storage medium, the multiple concentration data points include data points related to the concentration of a solution.

[0136] In some embodiments of the non-transitory computer-readable storage medium, the concentration data points include data points related to the absolute amount of the target analyte.

[0137] In some embodiments of the non-transitory computer-readable storage medium, the multiple concentration data points include data points related to measurements associated with the concentration.

[0138] In some embodiments of the non - transitory computer - readable storage medium, the measurement associated with the concentration is an optical density (OD) measurement.

[0139] In some embodiments of the non - transitory computer - readable storage medium, the plurality of concentration data points include data points related to an average value, an average of minimum standard deviations, an average of maximum standard deviations, or an intermediate control concentration.

[0140] In some embodiments of the non - transitory computer - readable storage medium, the significant change points reflect between - assay variability.

[0141] In some embodiments of the non - transitory computer - readable storage medium, the significant change points reflect within - assay variability.

[0142] In some embodiments of the non - transitory computer - readable storage medium, two or more of the plurality of concentration data points are in the same format.

[0143] In some embodiments of the non - transitory computer - readable storage medium, one or more assay parameters related to the significant change points include environmental factors.

[0144] In some embodiments of the non - transitory computer - readable storage medium, the environmental factors include temperature, humidity, light, or contaminants.

[0145] In some embodiments of the non - transitory computer - readable storage medium, one or more assay parameters related to the significant change points include equipment factors.

[0146] In some embodiments of the non - transitory computer - readable storage medium, the equipment factors include reagent replacement, reagent aging, reagent expiration, reagent contamination, reagent failure, hardware replacement, hardware aging, hardware contamination, hardware failure, instrument replacement, instrument failure, or instrument calibration.

[0147] In some embodiments of the non - transitory computer - readable storage medium, one or more assay parameters related to the significant change points include human factors associated with one or more individuals performing or assisting in performing the assay.

[0148] In some embodiments of the non - transitory computer - readable storage medium, the human factors include performance changes, performance errors, or operator replacement.

[0149] In some embodiments of the non - transitory computer - readable storage medium, identifying the significant change points includes determining a population of expected change points among the plurality of concentration data points.

[0150] In some embodiments of the non-transitory computer-readable storage medium, identifying significant change points includes: selecting a first segment of concentration data points among a plurality of concentration data points; determining a first median associated with the first segment of concentration data points; selecting a second segment of concentration data points among the plurality of concentration data points, wherein the concentration data points in the second segment are consecutive with the concentration data points in the first segment; determining a second median associated with the second segment of concentration data points; comparing the first median with the second median; and based on the comparison between the first median and the second median, determining whether a candidate change point is located between the first segment and the second segment.

[0151] In some embodiments of the non-transitory computer-readable storage medium, the first segment and the second segment include at least a threshold number of concentration data points.

[0152] In some embodiments of the non-transitory computer-readable storage medium, the instructions, when executed by one or more processors of an electronic device, are configured to cause the electronic device to receive a threshold number of concentration data points from a user.

[0153] In some embodiments of the non-transitory computer-readable storage medium, the threshold number of concentration data points is determined based on an assay.

[0154] In some embodiments of the non-transitory computer-readable storage medium, the instructions, when executed by one or more processors of an electronic device, are configured to cause the device, for the first segment and the second segment: generate one or more averages; generate one or more clusters of data points, wherein each cluster of data points is associated with an average among the one or more averages and includes the concentration data points in the segment that are closest to the associated average; update the average of each cluster of data points in the one or more clusters of data points, wherein updating the average of the cluster of data points includes identifying the centroid of the cluster of data points; and iteratively repeat the steps of generating one or more clusters of data points and updating the average of each cluster of data points until the average of each cluster of data points no longer changes.

[0155] In some embodiments of the non-transitory computer-readable storage medium, the instructions, when executed by one or more processors of an electronic device, are configured to cause the device to generate one or more clusters of data points within each of the first segment and the second segment, wherein the concentration data points in each cluster of data points are normally distributed and have a unique average and a unique standard deviation value.

[0156] In some embodiments of the non-transitory computer-readable storage medium, the instructions, when executed by one or more processors of an electronic device, are configured to cause the device to identify a primary cluster of data points among the one or more clusters of data points for the first segment and the second segment, wherein the primary cluster of data points includes at least a threshold percentage of the total number of concentration data points in the segment.

[0157] In some embodiments of the non - transitory computer - readable storage medium, the instructions, when executed by one or more processors of an electronic device, are configured to cause the device to determine a divergence value for a candidate change point, where the divergence value measures the statistical difference between a cluster of primary data points in a first segment and a cluster of primary data points in a second segment.

[0158] In some embodiments of the non - transitory computer - readable storage medium, the divergence value is the Jensen - Shannon divergence.

[0159] In some embodiments of the non - transitory computer - readable storage medium, the instructions, when executed by one or more processors of an electronic device, are configured to cause the device to determine a median change value for a candidate change point, where the median change value measures the difference between a first median associated with the first segment and a second median associated with the second segment.

[0160] In some embodiments of the non - transitory computer - readable storage medium, the instructions, when executed by one or more processors of an electronic device, are configured to cause the device to determine whether one or more statistical characteristics of a plurality of concentration data points change beyond a threshold by determining a weighted combination of the divergence value and the median change value.

[0161] In some embodiments of the non - transitory computer - readable storage medium, the weighted combination of the divergence value and the median change value is characterized by a weight parameter.

[0162] In some embodiments of the non - transitory computer - readable storage medium, the instructions, when executed by one or more processors of an electronic device, are configured to cause the device to receive a weight parameter from a user.

[0163] In some embodiments of the non - transitory computer - readable storage medium, the weight parameter is determined based on an assay.

[0164] In some embodiments of the non - transitory computer - readable storage medium, the instructions, when executed by one or more processors of an electronic device, are configured to cause the device to delete one or more concentration data points among the plurality of concentration data points, where the one or more concentration data points correspond to instances among the plurality of instances of the assay that occur after the identified significant change point. BRIEF DESCRIPTION OF THE DRAWINGS

[0165] The following figures illustrate various systems and methods for identifying and validating change points in concentration data obtained from an assay. In some embodiments, the systems and methods shown in the figures may have any one or more of the characteristics described herein.

[0166] Figures 1A - 1C A schematic representation of an ELISA assay format for obtaining concentration data of an analyte is shown. Figure 1A A direct ELISA format is shown. Figure 1BShows a sandwich ELISA format. Figure 1C Shows a competitive ELISA format.

[0167] Figure 2 Shows an example of concentration data points obtained from an assay.

[0168] Figure 3 Shows an example of the deviation of concentration data obtained from an assay.

[0169] Figure 4 Shows an example of a change point in a data set.

[0170] Figure 5 Shows a method for determining the cause of a significant statistical change among multiple concentration data points obtained from an assay.

[0171] Figure 6 Shows a method for using binary segmentation to identify candidate change points.

[0172] Figure 7 Shows a change point in a signal that has been identified using binary segmentation.

[0173] Figures 8A - 8C Shows a method for verifying the significance of candidate change points. Figure 8A Shows a method for using the k-means algorithm to generate clusters of data points. Figure 8B Shows a method for using a Gaussian mixture model to generate clusters of data points. Figure 8C Shows a method for determining whether the statistical characteristics of multiple concentration data points around a candidate change point change beyond a threshold.

[0174] Figures 9A - 9B Shows exemplary segments and exemplary clusters of data points in concentration data.

[0175] Figures 10A - 10C Shows an exemplary contour plot of a function that can be used to generate a loss value for a candidate change point.

[0176] Figure 11 Shows a method performed after the cause of a significant statistical change among multiple concentration data points obtained from an assay has been identified.

[0177] Figure 12 Shows a system for determining the cause of a significant statistical change among multiple concentration data points obtained from an assay.

[0178] Figure 13 Shows an example of a computing system.

[0179] Figures 14A - 14J Shows various examples of significant change points identified among multiple concentration data points using the disclosed method. Detailed implementation mode

[0180] The following disclosure describes a method for identifying and determining the cause of statistical changes in process data by determining the locations in the data where significant changes in the data have occurred. These locations (referred to as "change points") are typically the time points at which a statistical shift has occurred in the time series of the data. The method provided can allow laboratory analysts to relate the change points in the process data to changes in the measurement parameters that may have occurred at the identified change points. This can enable the effective extraction of the root cause of the changes in the process data, even when the underlying true data of the process is limited or unavailable.

[0181] In the present disclosure, the method is explained in the context of an assay (i.e., an investigative process that can be used to assess the presence of an analyte such as a drug, cell, or chemical). This context is not intended to limit the present disclosure; the method provided can be used to evaluate any data set that contains multiple repeated measurements.

[0182] Concentration data obtained from the measurement

[0183] An assay is a process that can be used to determine the presence of an analyte. There are many assay types and formats well known in the art. For example, as Figures 1A - 1C shown, various enzyme-linked immunosorbent assay (ELISA) formats are available, including direct ELISA formats ( Figure 1A ), sandwich ELISA formats ( Figure 1B ), and competitive ELISA formats ( Figure 1C ).

[0184] In the direct ELISA format ( Figure 1A ), the sample 108 is coated onto a solid phase (such as the wells 106 of the plate 104), and the desired antigen can be detected by the antibody 112. Negative and positive controls are available for the direct ELISA format. Such controls include wavelength correction, blank controls (e.g., dry wells or wells containing ELISA buffer), S0 negative controls (no standard or any form of analyte, e.g., sample, is added to the wells), negative matrix controls, and positive controls such as B0 (to evaluate maximum color development). In some embodiments, a standard curve can also be used for quantification purposes.

[0185] In the sandwich ELISA format ( Figure 1B) In it, the capture antibody 116 is coated onto a solid phase (such as the well 106 of the plate 104). The capture antibody 116 specifically binds to the analyte 110 of interest. Then, the sample 108 is added to the well 106 to allow the capture antibody 120 to bind to the analyte (if present). After that, the detection antibody 112 is added to the well to allow analyte detection. Negative and positive controls are available for the sandwich ELISA format. Such controls include wavelength correction, blank controls (e.g., dry wells or wells containing ELISA buffer), S0 negative controls (no standard or any form of analyte is added to the well, e.g., the sample), negative matrix controls, and positive controls such as B0 (to evaluate maximum color development). In some embodiments, a standard curve can also be used for quantification purposes.

[0186] In the competitive ELISA format ( Figure 1C ) In it, the secondary capture antibody 118 is coated onto a solid phase (such as the well 106 of the plate 104). The secondary capture antibody 118 specifically binds to the capture antibody 116, and the capture antibody 118 specifically binds to the analyte 110 of interest. The sample 108, the conjugated analyte 114, and the capture antibody 116 are added to the well 106 containing the secondary capture antibody 118, and competitive binding occurs. The more analyte 110 contained therein, the less the conjugated analyte 114 will remain bound to the capture antibody 116. Then, the detection can continue to collect concentration data. Negative and positive controls are available for the competitive ELISA format. Such controls include wavelength correction, blank controls (e.g., dry wells or wells containing ELISA buffer), non-specific binding (NSB) controls, negative matrix controls, and positive controls such as B0 (to evaluate maximum color development). In some embodiments, a standard curve can also be used for quantification purposes.

[0187] Concentration data, such as the concentration data of antibodies produced by cell cultures, can be obtained using assays such as Figure 1A - those shown in Assay 1C. The concentration data can include multiple concentration data points that may be collected over a period of time. In some cases, the concentration data can include data directly related to the concentration of the analyte, control, or solution. In other scenarios, the concentration data can include data related to measurements associated with the concentration of the analyte, control, or solution. For example, as Figure 2 shown, the concentration data can be a time series of measurements of the average optical density.

[0188] Many parameters can affect the data obtained from an assay. These assay parameters can be related to the environment in which the assay is performed, the equipment used to perform the assay, or the person involved in performing the assay. Ideally, the assay parameters remain constant throughout the time period in which the assay is used to obtain concentration data. However, when many concentration data points are obtained from an assay over an extended time period (e.g., over several weeks or months), some variation in the assay parameters may be inevitable. In some cases, these variations can result in significant fluctuations in the concentration data.

[0189] Figure 3 An example of the deviation of concentration data obtained from an assay due to a change in assay parameters, particularly a change in laboratory analyst, is shown. As shown, the concentration data obtained by one of the analysts during the time period between mid - October 2014 and December 2014 significantly exceeds the range of the remaining concentration data. Such anomalous data may imply a potential systematic error associated with the assay parameters (in this case, an error systematically propagated by the analyst who collected the anomalous data).

[0190] Ensuring that the concentration data obtained from an assay is accurate may require identifying changes in the concentration data and evaluating the root cause of the changes. Generally, changes in the concentration data may be the result of changes in the assay parameters, as demonstrated by the exemplary concentration data shown in Figure 3 In other cases, changes in the concentration data may be the result of a process or reaction that warrants further investigation. Regardless of the root cause, the effective location and validation of the significance of change points in the concentration data may be an important step in the analysis of the concentration data.

[0191] Method overview

[0192] The described method can identify change points among multiple concentration data points obtained from an assay. As previously discussed, a change point can be a point at which a statistical change occurs in the concentration data. An example of a change point in a data set is shown in Figure 4 The change in the measured variable is reflected at a first time point 402, a second time point 404, and a third time point 406 in

[0193] Figure 4In the data shown. After each of time points 402 - 406, the statistical characteristics (e.g., median, mean, etc.) of the data appear to change. The method provided herein can be used to algorithmically identify such change points in a data set (such as a data set including multiple concentration data points obtained from an assay). After the location of the change point in the data set has been identified, the method can verify the significance of each change point (block 410). In other words, the method can determine whether the difference between the statistical distribution of multiple data points before the change point and the statistical distribution of multiple data points after the change point is large enough to indicate that an error or other event worthy of further investigation occurred at the time of the change point. If it is determined that the change point is significant, the person associated with the data (e.g., the laboratory analyst who collected the data) can be notified so that the root cause of the data change can be evaluated (block 412). The root cause of the data change can include one or more of human factors (block 414), equipment factors (block 416), environmental factors (block 418), etc.

[0194] Figure 5 An example of a method 500 for determining the cause of a significant statistical change in concentration data obtained from an assay is shown. In some embodiments, one or more steps of method 500 can be performed by one or more processors in a system configured to identify and verify change points in a data set (e.g., by a processor in a computer belonging to a laboratory analyst who assisted with the assay).

[0195] As shown, method 500 can include a first step 502, in which multiple instances of an assay (or in some embodiments, multiple instances of multiple assays) can be performed to obtain multiple concentration data points. The assays performed can be of any type or format known in the art. For example, the assay can be an immunoassay, a competitive assay, a non - competitive assay, a non - homogeneous assay, or a homogeneous assay. Optionally, the assay can be an ELISA assay, such as a direct ELISA assay, a sandwich ELISA assay, or a competitive ELISA assay (for a schematic representation of an ELISA assay, see Figures 1A - 1C )

[0196] The assay can be configured to measure the concentration of an analyte in a sample. Optionally, the analyte can be a therapeutic analyte, a therapeutic polypeptide, an antibody, or a fragment of an antibody. The sample can be a cell culture sample or a derivative thereof.

[0197] In some embodiments, multiple instances of the assay may include two or more of the multiple instances being performed two or more times. The two or more times may constitute a time course of at least about one day, at least about one week, at least about one month, at least about six months, at least about one year, or at least about five years. In other words, two or more of the multiple concentration data points obtained in step 502 may be obtained at two or more different times of the day, on two or more different days of the week, on two or more different days of the month, or on two or more different days of the year. In some embodiments, multiple instances of the assay may include two or more of the multiple instances being performed in parallel (i.e., simultaneously).

[0198] The multiple concentration data points may include data points related to the concentration of the target analyte, the absolute amount of the target analyte, the concentration of a control (e.g., negative control, non-specific binding control, blank control, detection antibody control, negative matrix control, or positive control), and / or the solution concentration. In some aspects of method 500, the multiple concentration data points may include data points related to measurements associated with the concentration, such as optical density (OD) measurements. Optionally, the multiple concentration data points may include data points related to a mean, a low reference sample, a medium reference sample, a high reference sample, a low control, a medium control, and a high control. In some embodiments, two or more of the multiple concentration data points may be in the same format or in different formats.

[0199] After multiple concentration data points have been obtained from the assay in step 502, method 500 may proceed to step 504, where assay information may be received. The assay information may include the multiple concentration data points and multiple assay parameters. The assay parameter information may be recorded during each instance of the assay; each of the multiple assay parameters may be associated with one of the multiple instances of the assay and may indicate a characteristic or property of the assay during that instance or a characteristic or property of a factor associated with the assay. For example, assay parameters associated with an instance of the assay may include environmental factors associated with the environment in which the assay is performed (e.g., temperature, humidity, light, or the presence of one or more contaminants), equipment factors associated with the equipment used to perform the assay (e.g., reagent replacement, reagent aging, reagent expiration, reagent contamination, reagent failure, hardware replacement, hardware aging, hardware contamination, hardware failure, instrument replacement, instrument failure, or instrument calibration), or human factors associated with one or more operators performing or assisting in the performance of the assay (e.g., performance differences between operators, performance errors of one or more operators, or operator replacement). In some embodiments, multiple ones of the multiple assay parameters may be associated with a single instance of the assay.

[0200] After receiving the measurement information in step 504, method 500 may proceed to step 506, where significant change points may be identified. A significant change point may correspond to a position among a plurality of concentration data points at which one or more statistical characteristics of the plurality of concentration data points change by more than a threshold. A significant change point may be a position among the plurality of concentration data points at which the median, mean, variance, and / or correlation of the plurality of concentration data points has changed, or at which an anomaly among the plurality of concentration data points has been identified.

[0201] To identify significant change points, one or more candidate change points may first be identified. Then, the validity of each candidate change point may be determined by quantifying the statistical change of the concentration data points before and after each candidate change point. If the statistical change of the concentration data points before and after a candidate change point is determined to be significant (e.g., if the quantification of the change exceeds a cut-off value), then the candidate change point may be identified as a significant change point.

[0202] Once significant change points have been identified in step 506, method 500 may move to step 508, where one or more of the plurality of measurement parameters may be associated with the identified significant change points. In some embodiments, one or more of the measurement parameters may be associated with the significant change points by identifying the instances among the plurality of instances in which the measurement is performed that correspond to the positions of the significant change points among the plurality of concentration data points. For example, a significant change point may be identified at a position among the plurality of concentration data points corresponding to a date (e.g., day, month, and year). The instance in which the measurement is performed may be identified based on this date. After determining the instance in which the measurement is performed, one or more of the measurement parameters associated with that instance may be associated with the significant change point.

[0203] After associating one or more of the measurement parameters with the identified significant change points in step 508, method 500 may proceed to step 510, where the cause of the change in one or more statistical characteristics of the plurality of concentration data points at the significant change point may be determined based on the correlation between one or more of the measurement parameters and the significant change points. For example, if one or more of the measurement parameters indicate a change of operator at or near the time of the significant change point, determining the cause of the change in one or more statistical characteristics of the plurality of concentration data points may include determining that the operator who performed the measurement before the significant change point or the operator who performed the measurement after the significant change point may have made an error. Once the potential source of the change has been determined, the root cause of the change may be verified efficiently and accurately.

[0204] The following sections provide additional descriptions of the steps for identifying significant change points (step 506 of method 500) - specifically, descriptions of how candidate change points may be identified and how the significance of candidate change points may be verified.

[0205] Identifying candidate change points

[0206] After obtaining a plurality of concentration data points through a plurality of instances in which measurements are made (step 502 of method 500), significant change points can be identified (step 506 of method 500). As described in the previous section, identifying significant change points may involve identifying one or more candidate change points. Figure 6 An example of a method for using a binary segmentation algorithm to identify one or more candidate change points is shown.

[0207] The binary segmentation algorithm begins by selecting a first segment of concentration data points among the plurality of concentration data points (step 602). In some embodiments, the first segment of concentration data points may include at least a threshold number (N) of concentration data points. The threshold number of concentration data points may help ensure that only change points with a high likelihood of significance are identified. In other words, the threshold number of concentration data points can reduce the sensitivity of the binary segmentation algorithm to small random fluctuations in the plurality of concentration data points that do not indicate a significant change. The threshold number of concentration data points can be provided by the user. Optionally, the threshold number of concentration data points can be determined based on the measurements used to obtain the plurality of concentration data points. A first median (m1) associated with the first segment of concentration data points can be determined after selecting the first segment of concentration data points (step 604).

[0208] Next, in step 606, a second segment of concentration data points among the plurality of concentration data points is selected. The concentration data points in the second segment can be consecutive with the concentration data points in the first segment. Similar to the first segment, the second segment of concentration data points can include at least a threshold number (N) of concentration data points in order to reduce the sensitivity of the binary segmentation algorithm to unimportant fluctuations. After selecting the second segment of concentration data points, a second median (m2) associated with the second segment of concentration data points can be determined (step 608).

[0209] Once the first median and the second median have been determined, the first median can be compared with the second median to determine whether the candidate change point lies between the first segment and the second segment of the concentration data points (step 610). Comparing the first median with the second median can involve determining whether the difference between the first median and the second average value (e.g., Δ = |m1 - m2|) exceeds the coefficient of variation of the first segment (CV1) scaled by a scaling parameter (k), where the coefficient of variation of the first segment is the ratio of the standard deviation (σ1) of the first segment to the average value (μ1) of the first segment. The scaling parameter (k) can be provided by the user and can depend on the determination made to obtain a plurality of concentration data points. The scaling parameter can be about 0.1, about 0.2, about 0.3, about 0.4, about 0.5, about 0.6, about 0.7, about 0.8, or about 0.9. In some embodiments, the scaling parameter can be greater than 0.1, greater than 0.2, greater than 0.3, greater than 0.4, greater than 0.5, greater than 0.6, greater than 0.7, greater than 0.8, or greater than 0.9. In some embodiments, the scaling parameter can be less than 0.1, less than 0.2, less than 0.3, less than 0.4, less than 0.5, less than 0.6, less than 0.7, less than 0.8, or less than 0.9.

[0210] If a candidate change point is identified between the first segment and the second segment in step 610, steps 602 - 610 can be repeated for the concentration data points in the first segment and the concentration data points in the second segment to determine whether additional candidate change points lie in the first segment and / or the second segment. Figure 7 Several iterations of the binary segmentation method are shown. As shown, the entire signal 702 (e.g., a plurality of concentration data points obtained from a determination) is divided into a first segment 704 of data points and a second segment 706 of data points, as Figure 6 described in step 602 as shown. A candidate change point 712 is identified between the first segment 704 and the second segment 706, as Figure 6 described in steps 604 - 610. Then the binary segmentation process is repeated for the data points in the first segment 704. In other words, the first segment 704 is divided into two segments 708 and 710 of data points, and a second candidate change point 714 is identified between segments 708 and 710.

[0211] The binary segmentation process can continue to iterate until one or more stop conditions are met, such as until a maximum number of iterations have been performed or until a threshold number of candidate change points have been identified. Once the stop conditions are met, the binary segmentation process may stop.

[0212] Verifying the significance of candidate change points

[0213] Once one or more candidate change points have been identified in the plurality of concentration data points (e.g., usingFigures 6 - 7 (the binary segmentation technique described in), the validity of each candidate change point can be determined. In some embodiments, verifying the significance of a candidate change point may require identifying data point clusters (e.g., statistical patterns) in segments of concentration data points before and after each candidate change point.

[0214] There are many possible methods for generating such data point clusters. Figure 8A A first method is described, which uses the k - means algorithm to generate data point clusters in multiple concentration data points. If a candidate change point is identified between a first segment of concentration data points and a second segment of concentration data points (e.g., as described in method 600 as shown in Figure 6 ), then for each segment, one or more averages can be generated (step 802). In some embodiments, the average for a segment can be generated by randomly assigning one or more concentration data points within the segment as the average. After generating one or more averages, one or more data point clusters can be generated (step 804). Each data point cluster in the one or more data point clusters can be associated with an average among the one or more averages generated in step 802. The data point cluster associated with a given average can include the concentration data points in the corresponding segment that are closest to that average. The distance of a concentration data point to the average can be determined using a norm (e.g., Euclidean norm). After generating one or more data point clusters in step 804, the average of each data point cluster can be updated by identifying the centroid (i.e., the central concentration data point) of the data point cluster and assigning the centroid as the updated average. Steps 804 - 806 can be iteratively repeated for each segment until the average no longer changes in the update step.

[0215] Figure 8B An alternative method for generating data point clusters using a Gaussian mixture model is described. If a candidate change point is identified between a first segment of concentration data points and a second segment of concentration data points (e.g., as described in method 600 as shown in Figure 6 ), then one or more data point clusters can be identified for each segment (step 808). In this case, each data point cluster in the one or more data point clusters can be normally distributed and can have a unique average and a unique variance value. In other words, one or more data point clusters in each segment can be generated by dividing the concentration data points in each segment into different, normally - distributed groups of concentration data points.

[0216] In some embodiments, identifying data point clusters in a segment of concentration data points (e.g., using Figures 8A - 8BThe method shown in) may include determining the expected number of data point clusters within each segment. For example, techniques such as the Bayesian Information Criterion can be used to find the number of data point clusters within a segment of concentration data points.

[0217] In Figures 9A - 9B exemplary segments and exemplary data point clusters among a plurality of concentration data points are provided. As Figure 9A shown in, a plurality of concentration data points 902 can be divided into segments 904 separated by candidate change points 906. Within each segment, data point clusters 908a, 908b, and 908c can be generated. Figure 9B Shows how each cluster can be uniquely normally distributed, i.e., how it can be normally distributed with a unique mean (μ) and a unique standard deviation (σ), when generating data point clusters 908a-c using the method shown in Figure 8B .

[0218] Once one or more data point clusters in each segment among a plurality of concentration data points have been identified (e.g., by using the k-means algorithm shown in Figure 8A , the Gaussian mixture model method shown in Figure 8B ), the significance of each candidate change point can be verified. Verifying the significance of a candidate change point can involve determining whether one or more statistical properties of the plurality of concentration data points change by more than a threshold, as shown in Figure 8C . First, for each segment of concentration data points, the primary data point cluster of one or more data point clusters within the segment can be identified (step 810). The primary data point cluster for a segment can be a data point cluster that contains at least a threshold percentage of the concentration data points in the segment. For example, the primary data point cluster for a segment can be a data point cluster that contains at least 5%, at least 10%, at least 15%, at least 20%, at least 25%, or at least 30% of the concentration data points in the segment.

[0219] After identifying the primary data point cluster for each segment, the divergence value of each candidate change point can be determined (step 812). For a candidate change point located between the first segment and the second segment of concentration data points, the divergence value can measure the statistical difference between the primary data point cluster in the first segment and the primary data point cluster in the second segment. Specifically, the divergence value can measure the difference in the probability distribution of cluster membership between the data point cluster membership in the first segment and the data point cluster membership in the second segment. A method for measuring the similarity between probability distributions can be used to determine the divergence value of the candidate change point. Jensen-Shannon divergence (JSD) is an example of such a method. If P is the probability distribution representing the primary data point cluster in the segment (the first segment) immediately before the candidate change point, and Q is the probability distribution representing the primary data point cluster in the segment (the second segment) immediately after the candidate change point, then the JSD of the candidate change point is defined as:

[0220]

[0221] Wherein:

[0222]

[0223] And D(P||M) and D(Q||M) are Kullback-Leibler divergences.

[0224] In addition to the divergence values, a median change value can be determined for each candidate change point (step 814). The median change value of a change point can measure the median of the segment immediately before the candidate change point (e.g., the first median associated with the first segment as described in step 604 of method 600 as shown in Figure 6 and the median of the segment immediately after the candidate change point (e.g., the second median associated with the second segment as described in step 608 of method 600 as shown in Figure 6 ). In some embodiments, the median of a segment can be the median of all concentration data points in the segment or the median of the concentration data points in the main data point cluster in the segment.

[0225] Then, the change in the statistical characteristics before and after each candidate change point can be quantified to determine whether one or more statistical characteristics of a plurality of concentration data points change beyond a threshold (step 816). Quantifying the change in the statistical characteristics of a plurality of concentration data points before and after a candidate change point can include determining a loss value, which includes a weighted combination of the divergence value of the change point and the median change value of the change point. The weighted combination can be characterized by a weight parameter (λ). The weight parameter can be received from the user and can depend on the measurements made to obtain the plurality of concentration data points.

[0226] As previously described, the divergence value of a candidate change point can be determined using Jensen-Shannon divergence (JSD). In this case, the loss value of a candidate change point can be defined as follows:

[0227]

[0228] Where MC is the median change value of the candidate change point, and JSD is the divergence value of the candidate change point. The dependence of the function on the weight parameter (λ) is shown in the contour plot of the function provided in Figures 10A - 10C .

[0229] If the loss value of a candidate change point exceeds a threshold, the candidate change point can be identified as a significant change point. The threshold can be given by a cut-off parameter (ε). This cut-off parameter can characterize the change point detection sensitivity. The particular background of the measurement (e.g., the motivation for performing the measurement) can determine the type of change among the plurality of concentration data points that the user wishes to analyze. For example, if the user only wishes to investigate the major fluctuations among the plurality of concentration data points, the user can employ a large cut-off parameter. Alternatively, if the user wishes to investigate the less obvious fluctuations among the plurality of concentration data points, the user can employ a smaller cut-off parameter.

[0230] Determining the cause of the change

[0231] After identifying the significant change points (step 506 of method 500), the reason for the change in one or more statistical characteristics of the plurality of concentration data points at the identified significant change points can be determined based on the correlation between one or more measurement parameters and the significant change points (step 510 of method 500). Figure 11 An optional method is shown that is performed after the reason for the significant statistical change in the plurality of concentration data points obtained from the measurement has been identified.

[0232] As shown, in some embodiments, after determining the reason for the change in one or more statistical characteristics of the plurality of concentration data points at the identified significant change points, a second plurality of instances of the measurement (step 1102) are performed. The second plurality of instances of the measurement can be performed using measurement parameters that match the measurement parameters associated with the instances of the measurement that occurred before or after the identified significant change points. In some embodiments, one or more concentration data points can be removed from the plurality of concentration data points, the one or more concentration data points corresponding to the instances of the plurality of instances of the measurement that occurred before or after the identified significant change points (step 1104). Step 1104 may be preferred in cases where the reason for the change is determined to be an error (e.g., a systematic error during the measurement).

[0233] System for identifying and verifying change points

[0234] One or more steps of the described method can be performed by a system or device configured to identify and validate change points in a plurality of concentration data points obtained from a measurement. In Figure 12 An example of a system 1200 for determining the reason for a significant statistical change in a plurality of concentration data points obtained from a measurement is shown. The system 1200 can include a memory 1202 coupled to one or more processors 1204. The memory 1202 can store instructions that, when executed by the processor 1204, cause the processor 1204 to identify and validate change points in a plurality of concentration data points obtained from a measurement.

[0235] System 1200 can be configured to receive a plurality of concentration data points from a concentration data storage 1206 (e.g., one or more computers used by a laboratory analyst to store a plurality of concentration data points obtained from assays). Additionally, system 1200 can be configured to receive assay parameter data from an assay parameter storage device 1208. The assay parameter data can include information regarding assay parameters (e.g., environmental factors, equipment factors, and / or human factors) associated with the plurality of concentration data points obtained from an assay.

[0236] In some embodiments, system 1200 can be coupled to a user interface 1210. Optionally, user interface 1210 can be a component of system 1200. When processor 1204 identifies and validates a significant change point among the plurality of concentration data points, processor 1204 can cause user interface 1210 to output information regarding the significant change point. For example, processor 1204 can be configured to cause user interface 1210 to display one or more graphs of the plurality of concentration data points and indicate the location of the significant change point on the graph. If assay parameter data corresponding to the significant change point is available, processor 1204 can be configured to cause user interface 1210 to provide the corresponding assay parameter data to the user such that the user can use the assay parameter data to evaluate the root cause of the change in the plurality of concentration data points.

[0237] In some embodiments, the system for identifying and validating change points among a plurality of concentration data points can be a computer system or can include a computer system. Figure 13 An example of a computing system is shown. Computer 1300 can participate in performing one or more of the methods described herein. Computer 1300 can be a host computer connected to a network. Computer 1300 can be a client computer or a server. As Figure 13 shown, computer 1300 can be any suitable type of microprocessor-based device, such as a personal computer, a workstation, a server, or a handheld computing device, such as a phone or a tablet. The computer can include, for example, one or more of a processor 1310, an input device 1320, an output device 1330, a memory 1340, and a communication device 1360. Input device 1320 and output device 1330 can correspond to the devices described above and can be connected to or integrated with the computer.

[0238] Input device 1320 can be any suitable device that provides input, such as a touchscreen or monitor, a keyboard, a mouse, or a voice recognition device. Output device 1330 can be any suitable device that provides output, such as a touchscreen, a monitor, a printer, a disk drive, or a speaker.

[0239] The storage device 1340 can be any suitable device that provides storage, such as an electrical memory, a magnetic memory, or an optical memory, including random access memory (RAM), cache, hard disk drive, CD-ROM drive, tape drive, or removable storage disk. The communication device 1360 can include any suitable device capable of transmitting and receiving signals over a network, such as a network interface chip or card. The components of the computer can be connected in any suitable manner, such as via a physical bus or a wireless connection. The storage device 1340 can be a non-transitory computer-readable storage medium including one or more programs that, when executed by one or more processors, such as the processor 1310, cause the one or more processors to perform the methods described herein.

[0240] The software 1350 that can be stored in the storage device 1340 and executed by the processor 1310 can include, for example, programming embodying the functionality of the present disclosure (e.g., as embodied in the systems, computers, servers, and / or devices described above). In one or more instances, the software 1350 can include a combination of servers such as application servers and database servers.

[0241] The software 1350 can also be stored in and / or transmitted within any computer-readable storage medium for use by or in connection with an instruction execution system, apparatus, or device, such as those described herein, which can fetch and execute instructions associated with the software from the instruction execution system, apparatus, or device. In the context of the present disclosure, a computer-readable storage medium can be any medium, such as the storage device 1340, that can contain or store programming for use by or in connection with an instruction execution system, apparatus, or device.

[0242] The software 1350 can also be propagated within any transmission medium for use by or in connection with an instruction execution system, apparatus, or device, such as those described above, which can fetch and execute instructions associated with the software from the instruction execution system, apparatus, or device. In the context of the present disclosure, a transmission medium can be any medium that can communicate, propagate, or transmit programming for use by or in connection with an instruction execution system, apparatus, or device. The transmission-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, or infrared wired or wireless propagation media.

[0243] The computer 1300 can be connected to a network, which can be any suitable type of interconnected communication system. The network can implement any suitable communication protocol and can be protected by any suitable security protocol. The network can include network links arranged in any suitable manner that can send and receive network signals, such as wireless network connections, T1 or T3 lines, cable networks, DSL, or telephone lines.

[0244] The computer 1300 can implement any operating system suitable for operating on the network. The software 1350 can be written in any suitable programming language, such as C, C++, Java, or Python. In various embodiments, the application software embodying the functionality of the present disclosure can be deployed in different configurations, such as, for example, in a client / server arrangement or as a web-based application or web service via a web browser.

[0245] Examples of significant change points

[0246] In Figures 14A - 14J various instances of significant change points among the multiple concentration data points identified using the provided method are presented. As shown, a significant change point among the multiple concentration data points can be a location among the multiple concentration data points where the statistical characteristics of the distributed concentration data points change significantly or deviate. The disclosed method allows such change points to be effectively and accurately located in the concentration data obtained from the assay, even if the data is noisy and irregular.

[0247] Exemplary embodiment

[0248] Embodiments disclosed herein may include:

[0249] 1. A method for determining the cause of a significant statistical change among multiple concentration data points obtained from an assay, the method comprising:

[0250] performing multiple instances of the assay to obtain the multiple concentration data points;

[0251] receiving assay information including the multiple concentration data points and multiple assay parameters, where

[0252] each of the multiple assay parameters is associated with one of the multiple instances of performing the assay;

[0253] identifying a significant change point corresponding to a location among the multiple concentration data points where one or more statistical characteristics of the multiple concentration data points change by more than a threshold;

[0254] Correlating one or more of the plurality of measurement parameters with the identified significant change point by identifying the instances among the plurality of instances of the measurement that correspond to the location of the significant change point among the plurality of concentration data points; and

[0255] Determining the cause of the change in one or more statistical characteristics of the plurality of concentration data points at the identified significant change point based on the correlation between the one or more measurement parameters and the significant change point.

[0256] 2. The method according to embodiment 1, wherein the measurement is configured to measure the concentration of an analyte in a sample.

[0257] 3. The method according to embodiment 2, wherein the analyte is a therapeutic analyte.

[0258] 4. The method according to embodiment 2 or 3, wherein the analyte is a therapeutic polypeptide.

[0259] 5. The method according to any one of embodiments 2 to 4, wherein the analyte is an antibody or a fragment thereof.

[0260] 6. The method according to any one of embodiments 2 to 5, wherein the sample is a cell culture sample or a derivative thereof.

[0261] 7. The method according to any one of embodiments 1 to 7, wherein the measurement is an immunoassay.

[0262] 8. The method according to any one of embodiments 1 to 7, wherein the measurement is a competitive assay.

[0263] 9. The method according to any one of embodiments 1 to 7, wherein the measurement is a non-competitive assay.

[0264] 10. The method according to any one of embodiments 1 to 7, wherein the measurement is a heterogeneous assay.

[0265] 11. The method according to any one of embodiments 1 to 7, wherein the measurement is a homogeneous assay.

[0266] 12. The method according to any one of embodiments 1 to 10, wherein the measurement is an ELISA assay.

[0267] 13. The method according to embodiment 12, wherein the ELISA assay is a direct ELISA assay.

[0268] 14. The method according to embodiment 12, wherein the ELISA assay is a sandwich ELISA assay.

[0269] 15. The method according to embodiment 12, wherein the ELISA assay is a competitive ELISA assay.

[0270] 16. The method according to any one of embodiments 1 to 15, wherein the plurality of instances in which the assay is performed includes two or more of the plurality of instances being performed two or more times.

[0271] 17. The method according to embodiment 16, wherein the two or more times constitute a time course of at least about one week.

[0272] 18. The method according to any one of embodiments 1 to 17, wherein the plurality of instances in which the assay is performed includes two or more of the plurality of instances being performed simultaneously.

[0273] 19. The method according to any one of embodiments 1 to 18, wherein the plurality of concentration data points includes data points related to the concentration of the target analyte.

[0274] 20. The method according to any one of embodiments 1 to 19, wherein the plurality of concentration data points includes data points related to the concentration of a control.

[0275] 21. The method according to embodiment 20, wherein the control is a negative control, a non-specific binding control, a blank control, a detection antibody control, a negative matrix control, or a positive control.

[0276] 22. The method according to any one of embodiments 1 to 21, wherein the plurality of concentration data points includes data points related to the concentration of a solution.

[0277] 23. The method according to any one of embodiments 1 to 22, wherein the plurality of concentration data points includes data points related to the absolute amount of the target analyte.

[0278] 24. The method according to any one of embodiments 1 to 23, wherein the plurality of concentration data points includes data points related to measurements associated with the concentration.

[0279] 25. The method according to embodiment 24, wherein the measurement associated with the concentration is an optical density (OD) measurement.

[0280] 26. The method according to any one of embodiments 1 to 25, wherein the plurality of concentration data points includes data points related to an average value, an average value of the lowest standard deviation, an average value of the highest standard deviation, or an intermediate control concentration.

[0281] 27. The method according to any one of embodiments 1 to 26, wherein the significant change points reflect variability between assays.

[0282] 28. The method according to any one of embodiments 1 to 27, wherein the significant change point reflects within-assay variability.

[0283] 29. The method according to any one of embodiments 1 to 28, wherein two or more of the plurality of concentration data points are in the same format.

[0284] 30. The method according to any one of embodiments 1 to 29, wherein the one or more assay parameters associated with the significant change point include environmental factors.

[0285] 31. The method according to embodiment 30, wherein the environmental factors include temperature, humidity, light, or contaminants.

[0286] 32. The method according to any one of embodiments 1 to 31, wherein the one or more assay parameters associated with the significant change point include equipment factors.

[0287] 33. The method according to embodiment 32, wherein the equipment factors include reagent replacement, reagent aging, reagent expiration, reagent contamination, reagent malfunction, hardware replacement, hardware aging, hardware contamination, hardware malfunction, instrument replacement, instrument malfunction, or instrument calibration.

[0288] 34. The method according to any one of embodiments 1 to 33, wherein the one or more assay parameters associated with the significant change point include human factors associated with one or more individuals performing or assisting in performing the assay.

[0289] 35. The method according to embodiment 34, wherein the human factors include performance variability, performance error, or operator replacement.

[0290] 36. The method according to any one of embodiments 1 to 35, wherein identifying the significant change point includes determining a population of expected change points among the plurality of concentration data points.

[0291] 37. The method according to any one of embodiments 1 to 36, wherein identifying the significant change point includes:

[0292] selecting a first segment of concentration data points among the plurality of concentration data points;

[0293] determining a first median associated with the first segment of concentration data points;

[0294] selecting a second segment of concentration data points among the plurality of concentration data points, wherein the concentration data points in the second segment are consecutive with the concentration data points in the first segment;

[0295] Determine a second median associated with the second segment of the concentration data points;

[0296] Compare the first median with the second median; and

[0297] Based on the comparison between the first median and the second median, determine whether the candidate change point is located between the first segment and the second segment.

[0298] 38. The method according to embodiment 37, wherein the first segment and the second segment include at least a threshold number of concentration data points.

[0299] 39. The method according to embodiment 38, including receiving the threshold number of concentration data points from a user.

[0300] 40. The method according to embodiment 38, wherein the threshold number of concentration data points is determined based on the assay.

[0301] 41. The method according to any one of embodiments 37 to 40, which includes, for the first segment and the second segment:

[0302] Generate one or more averages;

[0303] Generate one or more clusters of data points, where each cluster of data points is associated with an average among the one or more averages and includes the concentration data points in the segment that are closest to the associated average;

[0304] Update the average of each cluster of data points among the one or more clusters of data points, where updating the average of a cluster of data points includes identifying the centroid of the cluster of data points; and

[0305] Iteratively repeat the steps of generating one or more clusters of data points and updating the average of each cluster of data points until the average of each cluster of data points no longer changes.

[0306] 42. The method according to any one of embodiments 37 to 40, which includes generating one or more clusters of data points within each of the first segment and the second segment, where the concentration data points in each cluster of data points are normally distributed and have a unique average and a unique standard deviation value.

[0307] 43. The method according to embodiment 41 or 42, including identifying a primary cluster of data points among the one or more clusters of data points for the first segment and the second segment, where the primary cluster of data points includes at least a threshold percentage of the total number of concentration data points in the segment.

[0308] 44. The method according to embodiment 43 includes determining a divergence value for the candidate change point, where the divergence value measures the statistical difference between the cluster of master data points in the first segment and the cluster of master data points in the second segment.

[0309] 45. The method according to embodiment 44, where the divergence value is Jensen-Shannon divergence.

[0310] 46. The method according to any one of embodiments 43 to 45 includes determining a median change value for the candidate change point, where the median change value measures the difference between the first median associated with the first segment and the second median associated with the second segment.

[0311] 47. The method according to embodiment 46 includes determining whether one or more statistical characteristics of the plurality of concentration data points change beyond a threshold by determining a weighted combination of the divergence value and the median change value.

[0312] 48. The method according to embodiment 47, where the weighted combination of the divergence value and the median change value is characterized by a weight parameter.

[0313] 49. The method according to embodiment 48 includes receiving the weight parameter from a user.

[0314] 50. The method according to embodiment 48 or 49, where the weight parameter is determined based on the assay.

[0315] 51. The method according to any one of embodiments 1 to 50 includes performing a second plurality of instances of the assay after determining the cause of the change in one or more statistical characteristics of the plurality of concentration data points at the identified significant change point.

[0316] 52. The method according to embodiment 50, where the second plurality of instances of the assay are performed using assay parameters that match one or more assay parameters associated with the identified significant change point.

[0317] 53. The method according to embodiment 50, where the second plurality of instances of the assay are performed using assay parameters that match assay parameters associated with instances of the assay that occurred before the identified significant change point.

[0318] 54. The method according to any one of embodiments 1 to 53 includes deleting one or more concentration data points from the plurality of concentration data points, where the one or more concentration data points correspond to instances of the plurality of instances of the assay that occurred after the identified significant change point.

[0319] 55. A system for determining the cause of a significant statistical change among a plurality of concentration data points obtained from an assay, the system comprising one or more processors configured to:

[0320] Receive assay information, the assay information including the plurality of concentration data points obtained by performing a plurality of instances of the assay and a plurality of assay parameters, wherein each assay parameter of the plurality of assay parameters is associated with one instance of the plurality of instances of the assay;

[0321] Identify a significant change point corresponding to a position among the plurality of concentration data points, at which position one or more statistical characteristics of the plurality of concentration data points change by more than a threshold;

[0322] Associate one or more of the assay parameters with the identified significant change point by identifying the instance among the plurality of instances of the assay corresponding to the position of the significant change point among the plurality of concentration data points; and

[0323] Determine the cause of the change in the one or more statistical characteristics of the plurality of concentration data points at the identified significant change point based on the correlation between the one or more assay parameters and the significant change point.

[0324] 56. The system according to embodiment 55, wherein the assay is configured to measure the concentration of an analyte in a sample.

[0325] 57. The system according to embodiment 56, wherein the analyte is a therapeutic analyte.

[0326] 58. The system according to embodiment 56 or 57, wherein the analyte is a therapeutic polypeptide.

[0327] 59. The system according to any one of items 56 to 58, wherein the analyte is an antibody or a fragment thereof.

[0328] 60. The system according to any one of embodiments 56 to 59, wherein the sample is a cell culture sample or a derivative thereof.

[0329] 61. The system according to any one of embodiments 55 to 60, wherein the assay is an immunoassay.

[0330] 62. The system according to any one of embodiments 55 to 60, wherein the assay is a competitive assay.

[0331] 63. The system according to any one of embodiments 55 to 60, wherein the assay is a non-competitive assay.

[0332] 64. The system according to any one of embodiments 55 to 60, wherein the assay is a heterogeneous assay.

[0333] 65. The system according to any one of embodiments 55 to 60, wherein the assay is a homogeneous assay.

[0334] 66. The system according to any one of embodiments 55 to 65, wherein the assay is an ELISA assay.

[0335] 67. The system according to embodiment 66, wherein the ELISA assay is a direct ELISA assay.

[0336] 68. The system according to embodiment 66, wherein the ELISA assay is a sandwich ELISA assay.

[0337] 69. The system according to embodiment 66, wherein the ELISA assay is a competitive ELISA assay.

[0338] 70. The system according to any one of embodiments 55 to 69, wherein the plurality of instances in which the assay is performed includes two or more of the plurality of instances performed two or more times.

[0339] 71. The system according to embodiment 70, wherein the two or more times constitute a time course of at least about one week.

[0340] 72. The system according to any one of embodiments 55 to 71, wherein the plurality of instances in which the assay is performed includes two or more of the plurality of instances performed simultaneously.

[0341] 73. The system according to any one of embodiments 55 to 72, wherein the plurality of concentration data points includes data points related to the concentration of a target analyte.

[0342] 74. The system according to any one of embodiments 55 to 73, wherein the plurality of concentration data points includes data points related to the concentration of a control.

[0343] 75. The system according to embodiment 74, wherein the control is a negative control, a non-specific binding control, a blank control, a detection antibody control, a negative matrix control, or a positive control.

[0344] 76. The system according to any one of embodiments 55 to 75, wherein the plurality of concentration data points includes data points related to the concentration of a solution.

[0345] 77. The system according to any one of embodiments 55 to 76, wherein the plurality of concentration data points includes data points related to the absolute amount of a target analyte.

[0346] 78. The system according to any one of embodiments 55 to 77, wherein the plurality of concentration data points include data points related to measurements associated with the concentration.

[0347] 79. The system according to embodiment 78, wherein the measurement associated with the concentration is an optical density (OD) measurement.

[0348] 80. The system according to any one of embodiments 55 to 79, wherein the plurality of concentration data points include data points related to an average value, a minimum standard deviation average value, a maximum standard deviation average value, or an intermediate control concentration.

[0349] 81. The system according to any one of embodiments 55 to 80, wherein the significant change points reflect between-assay variability.

[0350] 82. The system according to any one of embodiments 55 to 81, wherein the significant change points reflect within-assay variability.

[0351] 83. The system according to any one of embodiments 55 to 82, wherein two or more of the plurality of concentration data points are in the same format.

[0352] 84. The system according to any one of embodiments 55 to 83, wherein the one or more assay parameters related to the significant change points include environmental factors.

[0353] 85. The system according to embodiment 84, wherein the environmental factors include temperature, humidity, light, or contaminants.

[0354] 86. The system according to any one of embodiments 55 to 85, wherein the one or more assay parameters related to the significant change points include equipment factors.

[0355] 87. The system according to embodiment 86, wherein the equipment factors include reagent replacement, reagent aging, reagent expiration, reagent contamination, reagent failure, hardware replacement, hardware aging, hardware contamination, hardware failure, instrument replacement, instrument failure, or instrument calibration.

[0356] 88. The system according to any one of embodiments 55 to 87, wherein the one or more assay parameters related to the significant change points include human factors associated with one or more persons performing or assisting in performing the assay.

[0357] 89. The system according to embodiment 88, wherein the human factors include performance variability, performance error, or operator replacement.

[0358] 90. The system according to any one of embodiments 55 to 89, wherein identifying the significant change points includes determining a population of expected change points among the plurality of concentration data points.

[0359] 91. The system according to any one of embodiments 55 to 90, wherein identifying the significant change points includes:

[0360] selecting a first segment of concentration data points among the plurality of concentration data points;

[0361] determining a first median associated with the first segment of concentration data points;

[0362] selecting a second segment of concentration data points among the plurality of concentration data points, wherein the concentration data points in the second segment are consecutive with the concentration data points in the first segment;

[0363] determining a second median associated with the second segment of concentration data points;

[0364] comparing the first median with the second median; and

[0365] based on the comparison between the first median and the second median, determining whether a candidate change point is located between the first segment and the second segment.

[0366] 92. The system according to embodiment 91, wherein the first segment and the second segment include at least a threshold number of concentration data points.

[0367] 93. The system according to embodiment 92, wherein the one or more processors are configured to receive the threshold number of concentration data points from a user.

[0368] 94. The system according to embodiment 93, wherein the threshold number of concentration data points is determined based on the assay.

[0369] 95. The system according to any one of embodiments 91 to 94, wherein the one or more processors are configured to, for the first segment and the second segment:

[0370] generate one or more averages;

[0371] generate one or more clusters of data points, wherein each cluster of data points is associated with an average among the one or more averages and includes the concentration data points in the segment that are closest to the associated average;

[0372] update the average of each cluster of data points among the one or more clusters of data points, wherein updating the average of the cluster of data points includes identifying the centroid of the cluster of data points; and

[0373] Iteratively repeat the steps of generating one or more clusters of data points and updating the mean of each cluster of data points until the mean of each cluster of data points no longer changes.

[0374] 96. The system according to any one of embodiments 91 to 94, wherein the one or more processors are configured to generate one or more clusters of data points within each of the first segment and the second segment, wherein the concentration data points in each cluster of data points are normally distributed and have a unique mean and a unique standard deviation value.

[0375] 97. The system according to embodiment 95 or 96, wherein the one or more processors are configured to identify a main cluster of data points among the one or more clusters of data points for the first segment and the second segment, wherein the main cluster of data points includes at least a threshold percentage of the total number of concentration data points in the segment.

[0376] 98. The system according to embodiment 97, wherein the one or more processors are configured to determine a divergence value for the candidate change point, wherein the divergence value measures the statistical difference between the main cluster of data points in the first segment and the main cluster of data points in the second segment.

[0377] 99. The system according to embodiment 98, wherein the divergence value is the Jensen-Shannon divergence.

[0378] 100. The system according to any one of embodiments 97 to 99, wherein the one or more processors are configured to determine a median change value for the candidate change point, wherein the median change value measures the difference between the first median associated with the first segment and the second median associated with the second segment.

[0379] 101. The system according to embodiment 100, wherein the one or more processors are configured to determine whether one or more statistical characteristics of the plurality of concentration data points change by more than a threshold by determining a weighted combination of the divergence value and the median change value.

[0380] 102. The system according to embodiment 101, wherein the weighted combination of the divergence value and the median change value is characterized by a weight parameter.

[0381] 103. The system according to embodiment 102, wherein the one or more processors are configured to receive the weight parameter from a user.

[0382] 104. The system according to embodiment 102 or 103, wherein the weight parameter is determined based on the assay.

[0383] 105. The system according to any one of embodiments 55 to 104, wherein the one or more processors are configured to delete one or more of the plurality of concentration data points corresponding to instances among the plurality of instances of performing the assay that occur after the identified significant change point.

[0384] 106. A non - transitory computer - readable storage medium storing instructions for determining the cause of a significant statistical change in a plurality of concentration data points obtained from an assay, wherein the instructions are configured to be executed by one or more processors of an electronic device to cause the device to:

[0385] Receive assay information, the assay information including the plurality of concentration data points obtained through a plurality of instances of performing the assay and a plurality of assay parameters, wherein each assay parameter among the plurality of assay parameters is associated with one instance among the plurality of instances of performing the assay;

[0386] Identify a significant change point corresponding to a position among the plurality of concentration data points at which one or more statistical characteristics of the plurality of concentration data points change by more than a threshold;

[0387] Associate one or more of the plurality of assay parameters with the identified significant change point by identifying the instance among the plurality of instances of performing the assay corresponding to the position of the significant change point among the plurality of concentration data points; and

[0388] Determine the cause of the change in one or more statistical characteristics of the plurality of concentration data points at the identified significant change point based on the correlation between the one or more assay parameters and the significant change point.

[0389] 107. The non - transitory computer - readable storage medium according to embodiment 106, wherein the assay is configured to measure the concentration of an analyte in a sample.

[0390] 108. The non - transitory computer - readable storage medium according to embodiment 107, wherein the analyte is a therapeutic analyte.

[0391] 109. The non - transitory computer - readable storage medium according to embodiment 107 or 108, wherein the analyte is a therapeutic polypeptide.

[0392] 110. The non - transitory computer - readable storage medium according to any one of embodiments 107 to 109, wherein the analyte is an antibody or a fragment thereof.

[0393] 111. The non - transitory computer - readable storage medium according to any one of embodiments 107 to 110, wherein the sample is a cell - culture sample or a derivative thereof.

[0394] 112. The non - transitory computer - readable storage medium according to any one of embodiments 106 to 111, wherein the assay is an immunoassay.

[0395] 113. The non - transitory computer - readable storage medium according to any one of embodiments 106 to 111, wherein the assay is a competitive assay.

[0396] 114. The non - transitory computer - readable storage medium according to any one of embodiments 106 to 111, wherein the assay is a non - competitive assay.

[0397] 115. The non - transitory computer - readable storage medium according to any one of embodiments 106 to 111, wherein the assay is a heterogeneous assay.

[0398] 116. The non - transitory computer - readable storage medium according to any one of embodiments 106 to 111, wherein the assay is a homogeneous assay.

[0399] 117. The non - transitory computer - readable storage medium according to any one of embodiments 106 to 116, wherein the assay is an ELISA assay.

[0400] 118. The non - transitory computer - readable storage medium according to embodiment 117, wherein the ELISA assay is a direct ELISA assay.

[0401] 119. The non - transitory computer - readable storage medium according to embodiment 117, wherein the ELISA assay is a sandwich ELISA assay.

[0402] 120. The non - transitory computer - readable storage medium according to embodiment 117, wherein the ELISA assay is a competitive ELISA assay.

[0403] 121. The non - transitory computer - readable storage medium according to any one of embodiments 106 to 120, wherein the plurality of instances in which the assay is performed includes two or more of the plurality of instances performed two or more times.

[0404] 122. The non - transitory computer - readable storage medium according to embodiment 121, wherein the two or more times constitute a time course of at least about one week.

[0405] 123. The non-transitory computer-readable storage medium according to any one of embodiments 106 to 122, wherein the multiple instances for which measurements are made include two or more of the multiple instances being performed simultaneously.

[0406] 124. The non-transitory computer-readable storage medium according to any one of embodiments 106 to 123, wherein the multiple concentration data points include data points related to the concentration of a target analyte.

[0407] 125. The non-transitory computer-readable storage medium according to any one of embodiments 106 to 124, wherein the multiple concentration data points include data points related to the concentration of a control.

[0408] 126. The non-transitory computer-readable storage medium according to embodiment 125, wherein the control is a negative control, a non-specific binding control, a blank control, a detection antibody control, a negative matrix control, or a positive control.

[0409] 127. The non-transitory computer-readable storage medium according to any one of embodiments 106 to 126, wherein the multiple concentration data points include data points related to the concentration of a solution.

[0410] 128. The non-transitory computer-readable storage medium according to any one of embodiments 106 to 127, wherein the multiple concentration data points include data points related to the absolute amount of a target analyte.

[0411] 129. The non-transitory computer-readable storage medium according to any one of embodiments 106 to 128, wherein the multiple concentration data points include data points related to measurements associated with the concentration.

[0412] 130. The non-transitory computer-readable storage medium according to embodiment 129, wherein the measurement associated with the concentration is an optical density (OD) measurement.

[0413] 131. The non-transitory computer-readable storage medium according to any one of embodiments 106 to 130, wherein the multiple concentration data points include data points related to an average value, an average of minimum standard deviations, an average of maximum standard deviations, or an intermediate control concentration.

[0414] 132. The non-transitory computer-readable storage medium according to any one of embodiments 106 to 131, wherein the significant change points reflect variability between assays.

[0415] 133. The non-transitory computer-readable storage medium according to any one of embodiments 106 to 132, wherein the significant change points reflect within-assay variability.

[0416] 134. The non - transitory computer - readable storage medium according to any one of embodiments 106 to 133, wherein two or more of the plurality of concentration data points are in the same format.

[0417] 135. The non - transitory computer - readable storage medium according to any one of embodiments 106 to 134, wherein the one or more measurement parameters associated with the significant change point include environmental factors.

[0418] 136. The non - transitory computer - readable storage medium according to embodiment 135, wherein the environmental factors include temperature, humidity, light, or contaminants.

[0419] 137. The non - transitory computer - readable storage medium according to any one of embodiments 106 to 136, wherein the one or more measurement parameters associated with the significant change point include equipment factors.

[0420] 138. The non - transitory computer - readable storage medium according to embodiment 137, wherein the equipment factors include reagent replacement, reagent aging, reagent expiration, reagent contamination, reagent malfunction, hardware replacement, hardware aging, hardware contamination, hardware malfunction, instrument replacement, instrument malfunction, or instrument calibration.

[0421] 139. The non - transitory computer - readable storage medium according to any one of embodiments 106 to 138, wherein the one or more measurement parameters associated with the significant change point include human factors associated with one or more persons performing or assisting in performing the measurement.

[0422] 140. The non - transitory computer - readable storage medium according to embodiment 139, wherein the human factors include performance variability, performance error, or operator replacement.

[0423] 141. The non - transitory computer - readable storage medium according to any one of embodiments 106 to 140, wherein identifying the significant change point includes determining a group of expected change points among the plurality of concentration data points.

[0424] 142. The non - transitory machine - readable storage medium according to any one of embodiments 106 to 141, wherein identifying the significant change point includes:

[0425] selecting a first segment of concentration data points among the plurality of concentration data points;

[0426] determining a first median associated with the first segment of concentration data points;

[0427] Select a second segment of concentration data points among the plurality of concentration data points, wherein the concentration data points in the second segment are contiguous with the concentration data points in the first segment;

[0428] Determine a second median associated with the second segment of concentration data points;

[0429] Compare the first median with the second median; and

[0430] Based on the comparison between the first median and the second median, determine whether a candidate change point lies between the first segment and the second segment.

[0431] 143. The non-transitory computer-readable storage medium according to embodiment 142, wherein the first segment and the second segment include at least a threshold number of concentration data points.

[0432] 144. The non-transitory computer-readable storage medium according to embodiment 143, wherein the instructions, when executed by the one or more processors of the electronic device, are configured to cause the electronic device to receive the threshold number of concentration data points from a user.

[0433] 145. The non-transitory computer-readable storage medium according to embodiment 144, wherein the threshold number of concentration data points is determined based on the assay.

[0434] 146. The non-transitory computer-readable storage medium according to any one of embodiments 142 to 145, wherein the instructions, when executed by the one or more processors of the electronic device, are configured to cause the device, for the first segment and the second segment:

[0435] Generate one or more averages;

[0436] Generate one or more clusters of data points, wherein each cluster of data points is associated with an average among the one or more averages and includes the concentration data points in the segment that are closest to the associated average;

[0437] Update the average of each cluster of data points among the one or more clusters of data points, wherein updating the average of a cluster of data points includes identifying the centroid of the cluster of data points; and

[0438] Iteratively repeat the steps of generating one or more clusters of data points and updating the average of each cluster of data points until the average of each cluster of data points no longer changes.

[0439] 147. The non-transitory computer-readable storage medium according to any one of embodiments 142 to 145, wherein the instructions, when executed by the one or more processors of the electronic device, are configured to cause the device to generate one or more clusters of data points within each of the first segment and the second segment, wherein the concentration data points within each data point cluster are normally distributed and have a unique mean value and a unique standard deviation value.

[0440] 148. The non-transitory computer-readable storage medium according to embodiment 146 or 147, wherein the instructions, when executed by the one or more processors of the electronic device, are configured to cause the device to identify a main data point cluster among the one or more data point clusters for the first segment and the second segment, wherein the main data point cluster includes at least a threshold percentage of the total number of concentration data points in the segment.

[0441] 149. The non-transitory computer-readable storage medium according to embodiment 148, wherein the instructions, when executed by the one or more processors of the electronic device, are configured to cause the device to determine a divergence value for the candidate change point, wherein the divergence value measures the statistical difference between the main data point cluster in the first segment and the main data point cluster in the second segment.

[0442] 150. The non-transitory computer-readable storage medium according to embodiment 149, wherein the divergence value is the Jensen-Shannon divergence.

[0443] 151. The non-transitory computer-readable storage medium according to any one of embodiments 148 to 150, wherein the instructions, when executed by the one or more processors of the electronic device, are configured to cause the device to determine a median change value for the candidate change point, wherein the median change value measures the difference between the first median associated with the first segment and the second median associated with the second segment.

[0444] 152. The non-transitory computer-readable storage medium according to embodiment 151, wherein the instructions, when executed by the one or more processors of the electronic device, are configured to cause the device to determine whether one or more statistical characteristics of the plurality of concentration data points change beyond a threshold by determining a weighted combination of the divergence value and the median change value.

[0445] 153. The non-transitory computer-readable storage medium according to embodiment 152, wherein the weighted combination of the divergence value and the median change value is characterized by a weight parameter.

[0446] 154. The non-transitory computer-readable storage medium according to embodiment 153, wherein the instructions are configured to cause the device to receive the weight parameter from a user when executed by the one or more processors of the electronic device.

[0447] 155. The non-transitory computer-readable storage medium according to embodiment 153 or 154, wherein the weight parameter is determined based on the determination.

[0448] 156. The non-transitory computer-readable storage medium according to any one of embodiments 106 to 155, wherein the instructions are configured to cause the device to delete one or more of the plurality of concentration data points when executed by the one or more processors of the electronic device, the one or more concentration data points corresponding to instances among the plurality of instances in which the determination is made that occur after the identified significant change point.

[0449] This description only provides preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. On the contrary, the description of the preferred exemplary embodiments will provide those skilled in the art with a feasible description for implementing various embodiments. It should be understood that various changes can be made to the functions and arrangements of the elements without departing from the spirit and scope set forth in the appended claims.

[0450] Specific details are given in this description to provide a thorough understanding of the embodiments. However, it should be understood that the embodiments can be practiced without these specific details. For example, circuits, systems, networks, processes, and other components can be shown as components in block diagram form so as not to obscure the embodiments in unnecessary details. In other instances, well-known circuits, processes, algorithms, structures, and techniques can be shown without unnecessary details so as not to obscure the embodiments.

[0451] Conclusion

[0452] For purposes of explanation, the foregoing description has been made with reference to specific embodiments and / or examples. However, the above illustrative discussion is not intended to be exhaustive or to limit the invention to the precise form disclosed. Given the above teachings, many modifications and variations are possible. The embodiments are chosen and described in order to best explain the principles of the technology and its practical application. Thus, others skilled in the art will be able to best utilize the technology with various modifications suitable for the particular purposes contemplated.

[0453] Although the present disclosure and examples have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will become apparent to those skilled in the art. Such changes and modifications should be understood to be included within the scope of the present disclosure and embodiments as defined by the claims. Finally, the entire disclosures of the patents and publications mentioned in this application are hereby incorporated by reference.

[0454] Any system, method, technique, and / or feature disclosed herein may be combined, in whole or in part, with any other system, method, technique, and / or feature disclosed herein.

Claims

1. A system for determining the cause of a significant statistical change among a plurality of concentration data points obtained from an assay, the system comprising one or more processors configured to: Receive assay information, the assay information including the plurality of concentration data points obtained by performing a plurality of instances of the assay and a plurality of assay parameters, wherein each assay parameter of the plurality of assay parameters is associated with one instance of the plurality of instances of performing the assay; Identify a significant change point corresponding to a position among the plurality of concentration data points at which one or more statistical characteristics of the plurality of concentration data points change by more than a threshold; Associate one or more of the assay parameters with the identified significant change point by identifying the instance among the plurality of instances of performing the assay corresponding to the position of the significant change point among the plurality of concentration data points; and Determine the cause of the change in the one or more statistical characteristics of the plurality of concentration data points at the identified significant change point based on the correlation between the one or more assay parameters and the significant change point.

2. The system according to claim 1, wherein the assay is configured to measure the concentration of an analyte in a sample.

3. The system according to claim 1 or claim 2, wherein the one or more assay parameters associated with the significant change point include environmental factors, the environmental factors including temperature, humidity, light, or contaminants.

4. The system according to any one of claims 1 to 3, wherein the one or more assay parameters associated with the significant change point include equipment factors.

5. The system according to claim 4, wherein the equipment factors include reagent replacement, reagent aging, reagent expiration, reagent contamination, reagent failure, hardware replacement, hardware aging, hardware contamination, hardware failure, instrument replacement, instrument failure, or instrument calibration.

6. The system according to any one of claims 1 to 5, wherein the one or more assay parameters associated with the significant change point include human factors associated with one or more persons performing or assisting in performing the assay.

7. The system according to claim 6, wherein the human factors include performance variability, performance error, or operator replacement.

8. The system according to any one of claims 1 to 7, wherein identifying the significant change point includes determining a population of expected change points among the plurality of concentration data points.

9. The system according to any one of claims 1 to 8, wherein identifying the significant change point includes: Selecting a first segment of concentration data points among the plurality of concentration data points; Determining a first median associated with the first segment of concentration data points; Selecting a second segment of concentration data points among the plurality of concentration data points, wherein the concentration data points in the second segment are contiguous with the concentration data points in the first segment; Determining a second median associated with the second segment of concentration data points; Comparing the first median with the second median; And Based on a comparison between the first median and the second median, determine whether a candidate change point is between the first segment and the second segment.

10. The system of claim 9, wherein the first segment and the second segment include at least a threshold number of concentration data points.

11. The system of claim 10, wherein the one or more processors are configured to receive the threshold number of concentration data points from a user.

12. The system of claim 11, wherein the threshold number of concentration data points is determined based on the assay.

13. The system of any one of claims 9-12, wherein the one or more processors are configured to, for the first segment and the second segment: generate one or more averages; generate one or more clusters of data points, wherein each cluster of data points is associated with an average among the one or more averages and includes the concentration data points in the segment that are closest to the associated average; update the average of each cluster of data points among the one or more clusters of data points, wherein updating the average of a cluster of data points includes identifying the centroid of the cluster of data points; and iteratively repeat the steps of generating one or more clusters of data points and updating the average of each cluster of data points until the average of each cluster of data points no longer changes.

14. The system of any one of claims 9-12, wherein the one or more processors are configured to generate one or more clusters of data points within each of the first segment and the second segment, wherein the concentration data points in each cluster of data points are normally distributed and have a unique average and a unique standard deviation value.

15. The system of claim 13 or claim 14, wherein the one or more processors are configured to identify a primary cluster of data points among the one or more clusters of data points for the first segment and the second segment, wherein the primary cluster of data points includes at least a threshold percentage of the total number of concentration data points in the segment.

16. The system of claim 15, wherein the one or more processors are configured to determine a divergence value for the candidate change point, wherein the divergence value measures a statistical difference between the primary cluster of data points in the first segment and the primary cluster of data points in the second segment.

17. The system of claim 16, wherein the divergence value is the Jensen-Shannon divergence.

18. The system of any one of claims 15-17, wherein the one or more processors are configured to determine a median change value for the candidate change point, wherein the median change value measures the difference between the first median associated with the first segment and the second median associated with the second segment.

19. The system of claim 18, wherein the one or more processors are configured to determine whether one or more statistical characteristics of the plurality of concentration data points change by more than a threshold by determining a weighted combination of the divergence value and the median change value.

20. The system according to claim 19, wherein the weighted combination of the divergence value and the median change value is characterized by a weight parameter.

21. The system according to claim 20, wherein the one or more processors are configured to receive the weight parameter from a user.

22. The system according to claim 20 or claim 21, wherein the weight parameter is determined based on the determination.

23. The system according to any one of claims 1 to 22, wherein the one or more processors are configured to delete one or more concentration data points among the plurality of concentration data points corresponding to the instances that occur after the identified significant change points in the plurality of instances for performing the determination.

Images