qPCR curve detection

Abstract

Disclosed are embodiments of the present invention that provide improved computer systems, computerized methods, and computer program products for generating and evaluating automated predictions regarding whether a particular amplification curve from a qPCR assay indicates the presence of a target molecule in a sample. In some embodiments, the predictions are generated using deep learning networks. In some embodiments, curve quality predictions are generated and used to assess whether an amplification prediction can be made reliably from a particular amplification curve, or whether the curve reflects an anomaly in the qPCR assay. In various embodiments, prediction confidence data is also generated and used with the prediction data in an electronic user interface to improve the qPCR measurement.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to and the benefit of U.S. Provisional Application No. 63 / 320,213, filed March 15, 2022. To the extent permitted in the applicable jurisdiction, the entire contents of that application are incorporated herein by reference.

[0002] FIELD OF THEINVENTION The present disclosure relates generally to techniques for determining whether a target is present in a sample. [Background technology]

[0003] Amplification curves obtained from real-time (also known as quantitative) polymerase chain reaction (qPCR) experiments can be used to determine whether a target is present in a biological sample (e.g., blood or food sample). In a typical qPCR experiment, the fluorescence of the sample is measured after each thermal cycle of the experiment. The set of fluorescence values ​​versus cycle number associated with a particular assay for a sample forms an amplification curve. Traditionally, an algorithm and / or a human reviewer analyzes the amplification curves and determines whether the associated sample has amplified based on a visual or other analysis of the characteristics of the curve, which in turn indicates whether the target molecule was present in the sample. A typical algorithmic technique for determining whether the associated sample has amplified involves determining whether the associated amplification curve has crossed a threshold, which is either fixed or calculated based on characteristics of the amplification curve. If the threshold is crossed, the curve is determined to represent amplification. If the threshold is not crossed, the curve is determined to represent non-amplification. Summary of the Invention [Problem to be solved by the invention]

[0004] Automated determination of amplification is important for increasing the throughput of sample analysis, which in turn can advance scientific research and improve the provision of time-sensitive clinically important information. Existing methods for automatically determining amplification have relied on a combination of techniques and parameters to improve accuracy. Machine learning techniques, including deep learning networks such as artificial neural networks, can help improve accuracy. Improved machine learning techniques are needed to further improve accuracy and reduce the cases where human review is required. At the same time, it is important to effectively and optimally determine when human review is required, and the criteria for triggering human review of amplification curves evaluated by a computer system may vary depending on the application context. Improving the overall call rate accuracy of an amplification calling machine learning system does not necessarily fully address the problem of knowing which individual amplification curve auto-call results are at the highest risk of being wrong and therefore require human review. [Means for solving the problem]

[0005] Embodiments of the present invention address aspects of this "which curve" problem by using an improved amplification calling machine learning system (in some embodiments, an improved artificial neural network) in parallel with a machine learning system (e.g., a deep learning network such as an artificial neural network) to assess the quality of the amplification curves. This can help identify potential problematic curves. In some embodiments, an alternative amplification calling algorithm is used to help determine which curves called by the machine learning (e.g., neural network) amplification calling system should be further evaluated for potential curve quality issues that may require an invalidation result.

[0006] In some embodiments, evidence learning techniques are used to extract call confidence data from both the amplification calling neural network and the curve quality calling neural network. The confidence data can be used to further evaluate when machine-generated amplification calls should be reviewed by a human.

[0007] Some embodiments also provide improved amplification calling and curve quality calling networks based on specific types, variations and arrangements of neural network elements, pre-processing and processing techniques, and / or engineered features used in processing amplification curves from qPCR assays, further details of these embodiments are more fully disclosed herein. [Brief description of the drawings]

[0008] [Figure 1] 1 illustrates a PCR testing and analysis system according to an embodiment of the present invention. [Diagram 2] FIG. 1 is a high-level block architecture diagram of a computer system according to an embodiment of the present disclosure. [Diagram 3] 1 illustrates the pre-processing performed in one embodiment of the present disclosure. [Figure 4] FIG. 1 is a high-level block diagram of an architecture for an Amp Calling artificial neural network, according to one embodiment of the present disclosure. [Diagram 5] 5 is a block diagram illustrating the structure of FIG. 4 in further detail, according to one embodiment of the present disclosure. [Figure 6] FIG. 1 is a block diagram of a curve quality calling artificial neural network architecture according to one embodiment of the present disclosure. [Figure 7] 1 illustrates a computer process for performing amp and curve quality call evaluation according to one embodiment of the present disclosure. [Figure 8] FIG. 1 is a high-level block diagram of an architecture for an Amp Calling artificial neural network, according to one embodiment of the present disclosure. [Figure 9]9 is a block diagram illustrating the structure of FIG. 8 in further detail, according to one embodiment of the present disclosure. [Figure 10] FIG. 1 is a block diagram of a curve quality calling artificial neural network architecture according to one embodiment of the present disclosure. [Figure 11] 1 illustrates a computer process for performing trust processing according to one embodiment of the present disclosure. [Figure 12] An exemplary computer system configurable with a computer program product for implementing embodiments of the present invention is illustrated. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Although the invention has been described with reference to the above-mentioned drawing figures, the drawing figures are intended to be illustrative and other embodiments are consistent with the spirit and scope of the invention.

[0010] Various embodiments will now be described in more detail below with reference to the accompanying drawings, which form a part of this specification and which show for the purpose of illustrating specific examples in which the embodiments can be practiced. However, this specification may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this specification will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. In particular, this specification may be embodied as a method or device. Thus, any of the various embodiments herein may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Thus, the following specification should not be construed in a limiting sense.

[0011] 1 illustrates a system 1000 according to an exemplary embodiment of the invention. The system 1000 includes a polymerase chain reaction ("PCR") instrument 101, one or more computers 103, and a user device 107.

[0012] Instructions for implementing the amplification ("amp") curve analysis system 102 reside in a computer program product 104 stored in a storage device 105, the instructions being executable by a processor 106. When the processor 106 is executing instructions of the computer program product 104, the instructions, or portions thereof, are typically loaded into a working memory 109 from which the instructions are readily accessed by the processor 106. In the illustrated embodiment, the computer program product 104 is stored in the storage device 105 or another non-transitory computer readable medium (which may include being distributed among media in different devices and locations). In an alternative embodiment, the storage medium is transitory.

[0013] In one embodiment, the processor 106 actually includes multiple processors that may include additional working memory (additional processors and memory not separately shown), including a graphics processing unit (GPU) that includes at least several thousand arithmetic logic units to support massively parallel computations. GPUs are frequently utilized in deep learning applications because they can perform the associated processing tasks more efficiently than general purpose processors (CPUs). Other embodiments include one or more specialized processing units that include systolic arrays and / or other hardware configurations that support efficient parallel processing. In some embodiments, such specialized hardware works in conjunction with the CPU and / or GPU to perform various operations described herein. In some embodiments, such specialized hardware includes application specific integrated circuits, etc. (which may refer to a portion of an application specific integrated circuit), field programmable gate arrays, etc., or combinations thereof. However, in some embodiments, a processor such as the processor 106 may be implemented as one or more general purpose processors (preferably having multiple cores) without necessarily departing from the spirit and scope of the present invention.

[0014] The user device 107 includes a display 108 for displaying the results of the processing performed by the amplifier curve analysis system 102. In alternative embodiments, the amplifier curve analysis system 102, or portions thereof, may be stored in a storage device and executed by one or more processors residing in the PCR instrument 101 and / or the user device 107. Such alternatives do not depart from the scope of the present invention.

[0015] 2 is a high-level block architecture diagram of a computer system 2000 according to one embodiment of the present disclosure. The computer system 2000 includes a pre-processing block 201, one or more amplification calling neural networks 202, one or more curve quality calling neural networks 203, one or more alternative amplification calling algorithms 204, and an amplification and quality calling evaluation block 205.

[0016] The pre-processing block 201 is typically configured to receive one or more amplification curves corresponding to amplification data obtained from a 40-cycle or 50-cycle qPCR assay. In alternative embodiments, amplification curves obtained from PCR experiments with other cycle amounts may be processed by the illustrated embodiment using various techniques. A collection of discrete data points, e.g., 40 fluorescence values ​​for a 40-cycle PCR experiment (or 50 for a 50-cycle experiment), is referred to herein as a "curve" even though it is not continuous data. At the same time, a best-fit continuous curve may or may not be fitted to the data for easier visual display and / or analysis purposes. In any case, the term amplification "curve" herein is used generally to refer to a set of discrete amplification data obtained and analyzed for an assay of a biological sample, unless the context suggests otherwise.

[0017] The pre-processing block 201 processes the amplification curve to generate pre-processed curves and engineered features. The term "engineered" features refers to various features resulting from predetermined calculations on the amplification curve, and is distinguished from "learned" features resulting from submitting the amplification curve to a neural network or network portion (e.g., a convolutional layer of a neural network). In some embodiments of the present disclosure, the engineered features may be obtained during pre-processing and submitted to the neural network along with the pre-processed amplification curve, which may improve the learning speed and / or accuracy over submitting only the pre-processed curve itself. In some embodiments of the present disclosure, the engineered features include one or more of curve derivatives (e.g., first, second, and / or other higher order derivatives), time series features, and other features such as features from various PCR analysis algorithms, including, for example, PCRedux features and / or cycle relative threshold features, as further defined and described elsewhere herein.

[0018] In addition to processing the received amplification curves to obtain the designed characteristics, pre-processing block 201 also processes the amplification curves to remove baseline signals, stretch or shorten the curves to a uniform length (e.g., 40 or 50 cycles), and normalize the curves. These operations are used to generate the pre-processed amplification curves that are output by block 201.

[0019] The amplification ("amp") calling neural network 202, the alternative amp calling algorithm 204, and the curve quality calling neural network 203 receive preprocessed amplification curves. In the illustrated embodiment, the amp calling neural network 202 receives designed features from the preprocessing block 201, and the alternative amp calling algorithm 204 receives a portion of the designed features related to implementing a particular alternative amp calling algorithm. In an alternative embodiment, the curve quality calling neural network 203 also receives designed features from a preprocessing block such as the preprocessing block 201.

[0020] The amplification and curve quality calling evaluation block 205 receives amplified call data from the amplifier calling neural network 202. In the illustrated embodiment, the amplified call data includes class probabilities for at least the amplified and unamplified classes, and further includes confidence data generated from evidence learning techniques. However, in alternative embodiments, the confidence data is not generated and is not used. The block 205 also receives amplified call data from an alternative amplifier calling algorithm 204. The block 205 receives curve quality call data from the curve quality calling neural network 203. In the illustrated embodiment, the curve quality call data includes class probabilities for at least the "clean" and anomalous ("problem") curve classes, and further includes confidence data generated from evidence learning techniques. However, in alternative embodiments, the confidence data is not generated and is not used.

[0021] As described further below in connection with FIG. 7, the evaluation block 205 receives the amplification call and curve quality call information and uses it to evaluate which, if any, neural network based amp calls should be overridden. In some embodiments, the indication of which neural network (or other ML based) amp calls should be overridden is based on user selected settings in an electronic graphical user interface (GUI). In some embodiments, the GUI allows a user to selectively overridden calls based on visual inspection of corresponding amplification curves that have been flagged for review by the evaluation block 205. In some embodiments, the GUI allows a user to set parameters (such as confidence requirements, curve quality probability requirements, and / or other requirements) that the evaluation block 205 uses to determine which neural network based amp calls should be overridden and / or flagged for further review by the user.

[0022] FIG. 3 illustrates the pre-processing 3000 performed by the pre-processing block 201 of FIG. 2 in one embodiment of the present disclosure. Step 301 receives an amplification curve, for example, obtained from a qPCR assay performed on a qPCR instrument. Step 302 removes the baseline from the curve, which in this example corresponds to the baseline signal generated by the instrument in the absence of template. The baseline includes noise before a sufficient signal-to-noise ratio is reached. Those skilled in the art will appreciate that there are various ways in which baseline estimation can be performed. In one embodiment, the baseline is estimated by a linear least squares fit through the points of the curve selected as the baseline region. In one embodiment, the baseline is estimated by an average of the points of the curve selected as the baseline region. Step 303 determines whether the received amplification curve is less than 50 cycles. If so, the process proceeds to step 304 to extend the data to 50 cycle data. Those skilled in the art will appreciate various ways in which the extension can be performed. In one embodiment, constant padding is used, which simply involves repeating the last cycle value (e.g., the value corresponding to the 40th cycle) until the data is 50 cycles long. The stretched 50 cycle curve resulting from step 304 is provided to a derivative calculation step 305 and a normalization step 306. If the outcome of step 303 is no, the 50 cycle curve is provided directly to steps 305 and 306 without stretching. In an alternative embodiment, the data can be normalized to 40 cycle data, and other curves greater than 50 or 40 cycles can be truncated or interpolated to obtain 40 cycle data, as described, for example, in U.S. Patent Application Serial No. 17,346,147, filed June 11, 2021, Publication No. U.S. Patent Application Publication No. 2021 / 0391033, the entire contents of which are incorporated herein by reference.

[0023] In the illustrated embodiment, the derivative calculation step 305 calculates derivatives from the unnormalized amplification curve of 50 qPCR cycles. It calculates both the first and second derivatives. In one example, the derivative calculation is performed using a Savitzky-Golay filter and the following Savitzky-Golay parameters: polynomial order=3; and window length=9. The first and second derivatives may be normalized using maximum normalization. The resulting normalized derivative values ​​represent a portion of the designed features output by preprocessing 3000.

[0024] Step 306 normalizes the amplifier curves using cycle threshold ("Ct") values ​​for a particular assay on a particular instrument. Normalization may be performed by dividing each value of the unnormalized curve by the Ct value corresponding to the assay from which the curve was obtained. In alternative embodiments, different values ​​may be used for normalization.

[0025] The normalized curve is used by step 308 to calculate additional engineered features. In one embodiment, the additional engineered features include, for example, one or more of the following: features associated with the Cycle Relative Threshold ("Crt") algorithm, the version of which is described in U.S. Patent Application Publication No. 2016 / 0110495(A1) of U.S. Patent Application Serial No. 14 / 921,948 ("Crt Patent Application Publication"), features from PCRedux (see PCRedux: A Data Mining and Machine Learning Toolkit for qPCR Experiments at biorxiv.org / content / 10.1101 / 2021.03.31.437921v1), time series features (e.g., shapelets, wavelets), and features associated with various anomalies (e.g., waterfalls, bubbles, wavy baselines, and creep). Both of the above-referenced publications are incorporated herein by reference in their entirety.

[0026] In one embodiment, one or more of the features from the Crt algorithm listed and described below in Table 1 are calculated and used. Cross-references to the above-referenced Crt patent application publications are also provided below, where applicable.

[0027] [Table 1]

[0028] "AmpStatus" is the predicted result of the Crt algorithm. It can be obtained using software from commercially available products from Thermo Fisher Scientific, including, among others, one or more of the following products: QuantStudio™ 12k Flex v1.5, QuantStudio™ 6 and 7 Flex v1.7.2, Relative Quantification v4.3 (Cloud App), Standard Curve v4.0 (Cloud App), and Presence Absence Analysis v1.9 (Cloud App).

[0029] In one embodiment, one or more of the PCRedux features listed in Table 2 are determined and used.

[0030] [Table 2]

[0031] In one embodiment, step 308 normalizes the additional designed features after they are calculated. In one embodiment, a quantile normalizer is used. The quantile normalizer transforms the features to follow a normal distribution. The calculated quantile is equal to 1000. The scaler used for normalization is taken from a representative training set selected for a given model.

[0032] Step 307 outputs the normalized derivatives generated by step 305, the normalized curves generated by step 306, and the additional normalized engineered features generated by step 308. These values ​​may be output, for example, by pre-processing block 201 shown in Figure 2, and may be used by various portions of the amplification analysis system, as described further below.

[0033] FIG. 4 shows a high-level block diagram of an architecture 4000 of one of the amplifier calling neural networks 202 of FIG. 2 according to one embodiment of the present disclosure. Note that in one embodiment, the amplifier calling network 202 uses an ensemble approach that includes multiple (e.g., four, five, or some other number) neural networks that are similarly structured but trained differently. Various techniques (e.g., averaging, majority voting, etc.) can be used to combine the results of the individual networks in the ensemble to return class probabilities for a given curve generated from a particular assay for a sample. Also, various techniques can be used to train the networks of the ensemble differently (e.g., using different training data, ordering the training data differently, initializing weights / filters to different values, etc.). FIG. 4 shows the structure used for each neural network of the ensemble of neural networks 202 according to one embodiment of the present disclosure.

[0034] The structure 4000 includes a first portion 401 of a neural net (an "artificial" i.e., computerized neural network as implied throughout when referring to a "neural net" or "neural network" herein; other deep learning networks may be used in alternative embodiments, and it will be assumed that "deep learning network" as used herein also implies a "computerized deep learning network"), a second portion 402 of a neural net, and a third portion 403 of a neural net. In the illustrated embodiment, the second portion 402 of the neural net receives a pre-processed amplitude curve (from pre-processing block 201 of FIG. 2) along with selected designed features. In this example, the selected designed features received by the second portion 402 of the neural net include the normalized first and second derivatives calculated in step 305, as shown and described above in connection with FIG. 3. However, in alternative embodiments, other designed features may be received by second portion 402 in addition to or instead of the derivative value, or in some embodiments, a pre-processed amp curve may be received and used by second portion 402 without designed features.

[0035] Additional designed features are received by the first portion of the neural net 401. In the illustrated embodiment, the additional designed features include one or more of the features described above in connection with step 308 of Figure 3 (e.g., Crt algorithm features, PCRedux features, time series features, known anomaly features, etc.). However, in alternative embodiments, different designed features may be received by the first portion of the neural net 401.

[0036] The first portion of the neural net 401 and the second portion of the neural net 402 process their respective inputs. Their respective outputs are combined by a concatenation function 404, which provides the concatenated output as an input to the third portion of the neural net 403. The third portion of the neural net 403 processes the concatenated outputs of the first portion 401 and the second portion 402 and produces call and confidence data as an output. In an alternative embodiment, only the amplified call data is determined and output by the third portion of the neural net 403.

[0037] 5 is a block diagram illustrating the neural network structure 4000 of FIG. 4 in more detail, according to one embodiment of the present disclosure. As shown, the first portion 401 of the neural network includes a fully connected layer 501, a Leaky ReLU layer (activation function) 502, a fully connected layer 503, and a Leaky ReLU layer (activation function) 504. The second portion 402 of the neural network includes a separable convolutional layer 505, a Leaky ReLU layer (activation function) 506, a separable convolutional layer 507, a Leaky ReLU layer (activation function) 508, and a smoothing layer 509. As described above in connection with FIG. 4, a concatenation operation 404 concatenates the outputs of the first portion 401 and the second portion 402 and provides the concatenated output to an input of the third portion 403. The third part 403 includes a fully connected layer 511, a Leaky ReLU layer (activation function) 512, a fully connected layer 513, a ReLU layer (activation function) 514, and an evidence processing module 515.

[0038] The above structure will now be described with further details regarding the layer input / output size and other characteristics of certain embodiments. The details disclosed have been found to work well for the purposes of the amplification curve analysis application described herein. However, those skilled in the art will understand that many of these details may vary from those referenced below without necessarily departing from the spirit and scope of the present disclosure.

[0039] In one embodiment, the fully connected layer 501 has input data of 9 units (1×9) and output data of 16 single units (1×16). The Leaky ReLU layer 502 is an activation function that operates on the output of the fully connected layer 501 and provides a 1×16 output to the fully connected layer 503, which then provides a 1×16 output that is processed by the Leaky ReLU layer 504 to provide a 1×16 output to the concatenation function 404.

[0040] Turning to further details of the second portion 402 of the neural net, a separable convolutional layer 505, in this embodiment, receives three channels of data, with each channel containing 50 units, i.e., 3×50 data. Specifically, for a given cycle of the 50 cycle amp curve data, three values ​​are provided as inputs to the convolutional layer 505: the normalized fluorescence value, the first derivative, and the second derivative of the preprocessed amp curve at that cycle.

[0041] In the illustrated embodiment, the separable convolutional layer 505 uses 16 filters, each of size 3×3. A stride of 2 is used and sufficient padding is applied to the input array to obtain the desired output feature map dimensions, which for the separable convolutional layer 505 is 16×26. In this embodiment, this means that the 3×50 input array is separated into three “sub” arrays, each of size 1×50. Similarly, each 3×3 filter is separated into three “sub” filters, each of size 1×3. Each respective sub-array is convolved with a respective sub-filter to create a respective 1D feature map. The resulting feature maps are “stacked” to provide a 2D feature map. A pointwise convolution is performed on the 2D feature map and convolved with a 1×3 filter. This pointwise step outputs a 1×26 array, which is the same size feature map as would result from a normal convolution of a 3×3 filter with a 50×3 input array, including the necessary padding. In the illustrated embodiment, 16 filters are used, so the resulting output from the separable convolutional layer 505 is a 16x26 feature map.

[0042] In alternative embodiments, more or fewer filters may be used in each of the separable convolutional layers described herein. The number of filters shown in the illustrated embodiment is preferred, but may differ in alternative embodiments without necessarily departing from the spirit and scope of the present disclosure. In the illustrated embodiment, the separable convolutional layer 505 uses a stride of 2 and uses duplication padding. Duplication padding creates a copy of the sequence in the input array to be padded, reverses it, and then uses the reversed sequence to pad either end of the sequence. The padding allows the convolution process to create feature maps with the desired length dimension. Alternative embodiments use different types of padding, e.g., "same" padding, without necessarily departing from the spirit and scope of the present disclosure.

[0043] The other separable convolution layers in the illustrated embodiment use the same filter size, stride, padding type, and depthwise followed by pointwise convolution. These properties are assumed and will not be repeated further below, but these properties may vary in alternative embodiments.

[0044] The resulting 16×26 output from the separable convolutional layer 505 is processed by a Leaky ReLU activation function, represented here by Leaky ReLU layer 506, and the result is provided as 16×26 data to the separable convolutional layer 507. The separable convolutional layer 507 uses eight filters. The resulting 8×13 output data is processed by a Leaky ReLU layer 508, and the result is provided as 8×13 data to a flattening layer 509. The flattening layer 509 converts the 8×13 data into a single column (i.e., one-dimensional) array of length 104, and provides the resulting 1×104 output to the concatenation operation 404.

[0045] The 1×16 output of the Leaky ReLU 504 in the first part 401 and the 1×104 output of the flattening layer 509 in the second part 402 are concatenated in block 404, and the resulting 1×120 data is provided to the fully connected layer 511 in the third part 403.

[0046] The fully connected layer 511 receives 1×120 concatenated data and provides 1×16 output data, which is processed by the Leaky ReLU layer 512, which provides a 1×16 output to the fully connected layer 513, which provides an output from a two-node final layer providing a two-unit output (the output dimension corresponds to the number of categories, i.e., two categories, amplified and unamplified, in the illustrated embodiment).

[0047] In the illustrated embodiment, the output of the fully connected layer 513 is processed by a ReLU layer 514. The illustrated embodiment implements evidence learning. The output of the ReLU layer 514 is used as an evidence vector by an evidence processing block 515. The evidence processing block 515 uses evidence learning techniques to process the received evidence vectors to obtain class probability decisions corresponding to the amplifier and non-amplifier classifications for each amplification curve, along with confidence measures for the corresponding classification data. One example of the evidence learning technique used by block 515 in the illustrated embodiment is described in Sensoy et al., "Evidential Deep Learning to Quantify Classification Uncertainty," arXiv:1806.01768v3 [cs.LG] 31 Oct 2018, https: / / arxiv.org / pdf / 1806.01768.pdf, which is incorporated herein by reference in its entirety.

[0048] In an alternative embodiment, the ReLU layer 514 and the evidence processing block 515 are replaced by a softmax layer that simply converts the output of the fully connected layer 513 into class probabilities and does not generate confidence data.

[0049] Figure 6 is a block diagram of an architecture 6000 of one of the curve quality calling neural networks 203 of Figure 2, according to one embodiment of the present disclosure. Note that in one embodiment, the curve quality calling network 203 uses an ensemble approach that includes multiple (e.g., four, five, or some other number) similarly structured but differently trained neural networks (as described above in connection with the amp calling neural network 202). Figure 6 shows the structure used for each neural network of the ensemble of neural networks 203, according to one embodiment of the present disclosure.

[0050] In the illustrated embodiment, the same preprocessed amp curve and selected designed features (first and second derivatives) submitted to the second portion of the neural net 402 (see Figures 4 and 5 and accompanying text) are also submitted to the first portion 601 of the curve quality calling neural net illustrated in Figure 6. In an alternative embodiment, only the preprocessed amp curve is submitted. In other alternative embodiments, one or more additional designed features are also presented along with the preprocessed amp curve.

[0051] The first portion 601 of the neural net may be understood as operating to extract learned features (along with selected designed features of those curves) from the pre-processed amplifier curves it receives. The optimized parameter values ​​(e.g., convolution filter values) used to extract those features from each curve are learned during training of the neural network. The second portion 602 may be understood as a classification network used to classify the curves as "clean" or "problematic."

[0052] As shown, the first portion 601 of the neural net includes a separable convolutional layer 605, a Leaky ReLU layer 606, a separable convolutional layer 607, a Leaky ReLU layer 608, and a smoothing layer 609. In the illustrated embodiment, more detailed characteristics (e.g., input / output data dimensions, number of filters, filter size, stride, padding type) of the separable convolutional layer 605, the Leaky ReLU layer 606, the separable convolutional layer 607, the Leaky ReLU layer 608, and the smoothing layer 609 are the same as those described above for the separable convolutional layer 505, the Leaky ReLU layer 506, the separable convolutional layer 507, the Leaky ReLU layer 508, and the smoothing layer 509, respectively, and therefore those descriptions will not be repeated here.

[0053] Thus, the input provided from the flattening layer 609 to the fully connected layer 611 is of size 1×104. The fully connected layer 611 outputs 1×16 data for processing by the Leaky ReLU layer 612. The resulting 1×16 data is input to the fully connected layer 613, which outputs 1×2 data to the ReLU layer 614. The resulting evidence vectors are processed by the evidence processing block 615 in a similar manner as described above in connection with the evidence processing block 515. The evidence processing block 615 outputs class probabilities and corresponding confidence measures for the curve quality classification of “clean” or “problem”.

[0054] In an alternative embodiment, the ReLU layer 614 and the evidence processing block 615 are replaced by a softmax layer that simply converts the output of the fully connected layer 613 into class probabilities and does not generate confidence data.

[0055] 7 illustrates a process flow 7000 performed by the amp and curve quality call evaluation block 205. Process flow 7000 receives amp call data and curve quality call data generated by a curve analysis algorithm as described earlier in this specification. The flow of process flow 7000 provides one example of a process flow for curve analysis result processing that may be used to support presentation of results to a user interacting with a graphical user interface (GUI) on a user device that implements or is coupled to another computer that implements an embodiment of a curve analysis system consistent with the present disclosure.

[0056] Process 7000 starts at step 701. Step 702 receives an amp call from an alternative amp calling algorithm, such as one or more of those described above (or from other alternative amp calling algorithms). Step 703 receives an amp call from a neural network / machine learning based algorithm, such as, for example, neural network 202 of FIG. 2. Step 704 determines whether the results received in steps 702 and 703 match. If yes, processing proceeds to step 707 where a curve quality indicator (CQI) flag is not set. If the result of step 704 is no, step 705 evaluates the call received from a curve quality processing algorithm, such as, for example, neural network 203 of FIG. 2. Step 706 determines whether the curve quality call was "clean" (i.e., not "problematic"). If no, processing proceeds to step 708 where a CQI flag is set. If yes, processing proceeds to step 707 where a CQI flag is not set for the curve. In either case, processing continues to step 709, which determines whether CQI analysis is complete for all curves generated from the particular sample well for which curves generated for one or more tests are currently being analyzed. If the result of step 709 is no, processing returns to steps 702 and 703 for the next curve. If the result of step 709 is yes, processing continues to step 710.

[0057] Step 710 determines whether any curves corresponding to the currently analyzed well have a CQI flag. If no, processing proceeds directly to steps 712 and 716 and none of the tests associated with the current well being analyzed are disabled. If yes, processing proceeds to step 711 to determine whether CQI override is currently enabled. In some embodiments, this is a user-selected setting. In alternative embodiments, whether CQI override is enabled may be determined automatically based on various user-selected or predefined factors and / or may be enabled by default. If CQI override is not enabled, the result of step 711 is no and processing proceeds to steps 712 and 716 and none of the tests associated with the current well being analyzed are disabled. If CQI override is enabled, the result of step 711 is yes and processing proceeds to step 713 to determine whether only calls from individual tests associated with CQI flags in a given well should be disabled or all calls for tests in the well should be disabled. If the result of step 713 is "calls", only test results associated with CQI flags are disabled. If the result of step 713 is "well", then all results for that well are voided, regardless of whether the individual test is associated with a CQI-flagged amplification curve.

[0058] Those skilled in the art will appreciate that in various embodiments, different factors can be used to determine whether to disable all tests in a well or only tests associated with calls for CQI-flagged curves. In some embodiments, this determination can be made from available data based on user-set criteria. For example, a user may set criteria based on the number or percentage of CQI-flagged calls in a given well to determine whether to disable all calls in the well. Once user-based criteria are set, results can be automatically implemented based on call and CQI flag data. If confidence measures are available, the user can set criteria based on confidence scores that meet or do not meet certain thresholds, in some embodiments, at the level of individual calls and / or across all calls in the well. Conditional criteria based on various combinations of call and curve quality data can also be implemented. Various alternatives consistent with this disclosure will be apparent to those skilled in the art.

[0059] 8 illustrates a high-level block diagram of an alternative architecture 8000 of one of the amplifier calling neural networks 202 of FIG. 2, according to one embodiment of the present disclosure. The illustrated structure 8000 is an alternative to the structure 4000 illustrated in FIG.

[0060] As previously described in connection with FIG. 4, in one embodiment, the amplifier calling network 202 uses an ensemble approach that includes multiple (e.g., four, five, or some other number) similarly structured but differently trained neural networks. Various techniques (e.g., averaging, majority voting, etc.) can be used to combine the results of the individual networks in the ensemble to return class probabilities for a given curve generated from a particular assay for a sample. Also, various techniques can be used to train the networks of the ensemble differently (e.g., using different training data, ordering the training data differently, initializing weights / filters to different values, etc.). FIG. 8 illustrates the structure used for each neural network of the ensemble of neural networks 202, according to an alternative embodiment of the present disclosure.

[0061] The structure 8000 includes a first portion of a neural net 801, a second portion of a neural net 802, and a third portion of a neural net 803. In the illustrated embodiment, the second portion of the neural net 802 receives a preprocessed amplitude curve (from preprocessing block 201 of FIG. 2) along with selected designed features. In this example, the selected designed features received by the second portion of the neural net 802 include the normalized first and second derivatives calculated in step 305 as shown and described above in connection with FIG. 3. However, in alternative embodiments, other designed features may be received by the second portion of the neural net 802 in addition to or instead of the derivative values, or in some embodiments the preprocessed amplitude curve may be received and used by the second portion of the neural net 802 without the designed features.

[0062] Additional designed features are received by the first portion of the neural net 801. In the illustrated embodiment, the additional designed features include one or more of the features described above in connection with step 308 of Figure 3 (e.g., Crt algorithm features, PCRedux features, time series features, known anomaly features, etc.). However, in alternative embodiments, different designed features may be received by the first portion of the neural net 801.

[0063] The first neural net portion 801 and the second neural net portion 802 process their respective inputs. Their respective outputs are combined by a concatenation function 804, which provides the concatenated output as an input to the third neural net portion 803. The third neural net portion 803 processes the concatenated output of the first neural net portion 801 and the second neural net portion 802 and generates as an output, call amplified call data. In one example, such call data includes class probabilities (sometimes referred to herein as predicted scores) for one or more classifications, such as amplified and / or non-amplified.

[0064] As further shown in FIG. 8, the class probabilities output by the third portion of the neural net 803 are also provided to a reliability processing block 805. The reliability processing block 805 also receives quantification cycle (Cq) data (sometimes referred to as cycle threshold or Ct data) including Cq values ​​for each amplification curve from which the amplifier class probabilities are generated. The reliability processing block 805 generates the reliability data. One embodiment for performing the processing of the reliability processing block 805 is further described below in conjunction with FIG. 11.

[0065] FIG. 9 is a block diagram illustrating the neural network structure 8000 of FIG. 8 in more detail, according to one embodiment of the present disclosure. As shown, the first portion of the neural network 801 includes FCwLD processing blocks 901 and 902. Each FCwLD processing block includes a fully connected layer 921 followed by a Leaky ReLU layer (activation function) 922 and a drop layer 923. The second portion of the neural network 802 includes a separable convolutional layer 903, a Leaky ReLU layer (activation function) 904, a separable convolutional layer 905, a Leaky ReLU layer (activation function) 906, and a flattening layer 907. As described above in connection with FIG. 8, a concatenation operation 804 concatenates the outputs of the first portion of the neural network 801 and the second portion of the neural network 802, and provides the concatenated output to an input of the third portion 803. The third portion 803 of the neural network includes FCwLD blocks 908, 909, 910 followed by a fully connected layer 911 and a softmax layer 912. Similar to the FCwLD blocks 901 and 902, the FCwLD blocks 908, 909, 910 each include a fully connected layer 921 followed by a leaky ReLU layer 922 and a dropout layer 923.

[0066] In the illustrated embodiment, each dropout layer 923 in the FCwLD block uses a dropout rate of 0.5. However, other rates may be used without necessarily departing from the spirit and scope of this disclosure. In the illustrated embodiment, the separable convolutional layers 903 and 905 operate in a manner similar to that described for the separable convolutional layers 505 and 507 of FIG.

[0067] In the illustrated embodiment, the softmax layer 912 outputs class probabilities (prediction scores) for use by the computer user interface and for use by the reliability processing block 805.

[0068] In an alternative embodiment, the softmax layer 912 and confidence processing block 805 are replaced by a Leaky ReLU layer and evidence learning block as described above which generate amplified call data and confidence data.

[0069] FIG. 10 illustrates a high-level block diagram of an alternative architecture 10000 of one of the curve quality calling neural networks 203 of FIG. 2, according to one embodiment of the present disclosure.

[0070] As shown, a first portion 1001 of the neural network includes a separable convolutional layer 1003, a Leaky ReLU layer (activation function) 1004, a separable convolutional layer 1005, a Leaky ReLU layer (activation function) 1006, and a flattening layer 1007. A second portion 1002 of the neural network includes FCwLD blocks 1008, 1009, and 1010, followed by a fully connected layer 1011 and a softmax layer 1012. Similar to FCwLD blocks 901 and 902, FCwLD blocks 1008, 1009, and 1010 each include a fully connected layer 921, followed by a Leaky ReLU layer 922 and a dropout layer 923.

[0071] In the illustrated embodiment, each dropout layer 923 in the FCwLD block uses a dropout rate of 0.5. However, other rates may be used without necessarily departing from the spirit and scope of this disclosure. In the illustrated embodiment, the separable convolutional layers 1003 and 1005 operate in a manner similar to that described for the separable convolutional layers 605 and 607 of FIG.

[0072] In the illustrated embodiment, the softmax layer 1012 outputs class probabilities (prediction scores) for use by the computer user interface and for use by the reliability processing block 1025.

[0073] In an alternative embodiment, the softmax layer 1012 and confidence processing block 10255 are replaced by a Leaky ReLU layer and evidence learning block as described above which generate the curve quality call data and confidence data.

[0074] Figure 11 is a flow diagram illustrating a process 1100 performed by amplifier predictive reliability processing block 805 shown in Figures 8 and 9, according to one embodiment of the present disclosure. Similar processing steps may be used by curve quality predictive reliability processing block 1025 shown in Figure 10. Process 1100 will first be described in conjunction with the amplifier predictive reliability processing performed by block 805.

[0075] Process 1100 begins at step 1101, which calculates an ensemble confidence for each predicted score. As previously described, some embodiments of the present disclosure use an ensemble of similarly structured but differently trained deep learning networks. In related embodiments using process 1100, each deep learning amp calling network 202 in the ensemble of such networks generates a predicted score for a particular amp curve. As one skilled in the art will appreciate, a variety of statistical techniques can be used to generate a confidence metric using a set of data points drawn from an underlying population.

[0076] In some embodiments, a 95% confidence interval is calculated for the ensemble prediction score using all individual prediction scores generated by the amplifier calling network 202 in the ensemble, assuming a normal distribution. The difference between the upper and lower bounds of that confidence interval is then used as a confidence metric, and an empirical threshold is used to assign and output an intuitive confidence level to the prediction score result (e.g., "low", "medium", or "high" confidence). For example, the confidence interval for the prediction score of the ensemble may have a lower bound (LB) of 0.75 and an upper bound (UB) of 0.85. The resulting confidence metric (UB-LB) is 0.10. For example, using a threshold of 0.15 such that anything less than 0.15 is "high" confidence, the process assigns a high confidence level to the corresponding amplifier prediction score of the ensemble as a whole (the overall prediction score of the ensemble may be taken as the average prediction score of all scores generated for a given curve by the amplifier calling networks of the ensemble, for example).

[0077] Step 1102 determines whether the calculated confidence metric is less than a first lower threshold. If the result of step 1102 is yes, the process proceeds to step 1103 and outputs a "high" confidence level. If the result of step 1102 is no, the process proceeds to step 1104 and determines whether the confidence metric (e.g., the UB-LB of the confidence interval) is less than a next higher threshold. If the result of step 1104 is yes, the process proceeds to step 1105 and outputs a "medium" confidence level. If the result of step 1104 is no, the process proceeds to step 1106 and outputs a "low" confidence level.

[0078] In one embodiment, the following thresholds are used: if the UB-LB of the 95% confidence interval is less than 0.25, the confidence is determined to be "high"; if the UB-LB of the 95% confidence interval is equal to or greater than 0.25 but less than 0.6, the confidence level is determined to be "medium"; if the UB-LB of the 95% confidence interval is equal to or greater than 0.6, the confidence level is determined to be "low". However, a person skilled in the art will understand that the appropriate threshold for confidence determination depends on the dataset and context. And, different thresholds may be used depending on the statistics of the underlying dataset.

[0079] Also, the threshold used may vary based on which Cq value is associated with the amplification curve being processed for prediction. For example, a narrower range of prediction scores may be expected across some Cq ranges. As an example, if Cq values ​​correspond to cycles in the range of 12-20, the amplified class probabilities are expected to be consistently high (closer to 1), and a narrower confidence interval range (lower threshold for UB-LB) may be required to assign a "high" confidence value to the class probability (prediction score) (e.g., 0.2 or 0.15 instead of 0.25). Thus, in some embodiments, the first (lower) threshold depends on the Cq value. For example, in step 1103, a first threshold lower than 0.25 (e.g., 0.15, 0.2) may be used when the corresponding amplification curve has a relatively low Cq value (e.g., 12-20), and some other value higher than 0.25 or 0.2 may be used for curves with higher Cq values. In some embodiments, the second (higher) threshold also depends on the Cq value.

[0080] In some embodiments, process 1100 (or a similar process) is used by block 1025 of FIG. 10 to assign a confidence value to a prediction score for a curve quality prediction. For curve quality prediction, the appropriate threshold for assigning low, medium, or high confidence also depends on the particular data set and context. However, the process may be similar to that described above in connection with the amp prediction confidence process. For example, in some embodiments using process 1100 for curve quality prediction confidence process, a 95% confidence interval is calculated for the ensemble prediction score using all individual prediction scores generated by the curve quality calling network 203 in the ensemble, assuming a normal distribution. The difference between the upper and lower limits of that confidence interval is then used as a confidence metric, and an empirical threshold is used to assign an intuitive confidence level (e.g., “low”, “medium”, or “high” confidence) to the prediction score result. In some embodiments, 0.25 is used as the first (lower) threshold, and if the ensemble confidence metric is less than 0.25, “high” confidence is assigned. If the ensemble confidence metric is greater than or equal to 0.25 but less than 0.6, then a "medium" confidence is assigned. If the ensemble confidence metric is not less than 0.6, then a "low" confidence is assigned.

[0081] Although Cq values ​​may be less relevant for assessing the reliability of curve quality predictions than for assessing the reliability of amplification predictions, in some embodiments, other context-dependent factors may be used to modify the thresholds for curve quality reliability processing. As noted above, those skilled in the art will understand that appropriate thresholds for assigning confidence levels may depend on the context of a particular data set. The above thresholds are provided merely as examples.

[0082] As one skilled in the art will appreciate, in alternative embodiments, the confidence metric itself and / or other underlying statistics for the associated data set may be output via a computer user interface without necessarily assigning an intuitive confidence value such as "low," "medium," or "high." Sophisticated users may prefer to assess confidence based on the underlying confidence interval metric rather than have the assessment generated automatically.

[0083] FIG. 12 illustrates an exemplary computer system that may be configured with a computer program product to carry out embodiments of the present invention.

[0084] In this example, the computer system 1200 may provide one or more of the components of an automated qPCR curve analysis system configured to implement one or more logic modules and artificial neural networks, as well as related components for a computer-implemented qPCR automated analysis system and associated interactive graphical user interface. The computer system 1200 executes instruction codes included in a computer program product 1260. The computer program product 1260 includes executable code in an electronically readable medium that instructs one or more computers, such as the computer system 1200, to perform operations to accomplish the exemplary method steps performed by the embodiments referenced herein. The electronically readable medium may be any non-transitory medium that electronically stores information and may be accessed locally or remotely, for example, via a network connection. In alternative embodiments, the medium may be transitory. The medium may include multiple geographically distributed media, each configured to store different portions of the executable code at different locations or different times. The executable instruction codes in the electronically readable medium direct the illustrated computer system 1200 to perform various exemplary tasks described herein. Executable code for directing the execution of the tasks described herein is typically implemented in software. However, those skilled in the art will appreciate that a computer or other electronic device may utilize hardware-implemented code to perform many or all of the identified tasks without departing from the invention. Those skilled in the art will appreciate that many variations on executable code implementing the exemplary methods may be found within the spirit and scope of the invention.

[0085] Code or copies of code included in computer program product 1460 may reside in one or more storage persistent media (not shown separately) communicatively coupled to computer system 1200 for loading and storage in persistent storage device 1270 and / or memory 1210 for execution by processor 1220. Computer system 1200 also includes I / O subsystem 1230 and peripheral devices 1240. I / O subsystem 1230, peripheral devices 1240, processor 1220, memory 1210, and persistent storage device 1270 are coupled via bus 1250. Like persistent storage device 1270 and any other persistent storage that may comprise computer program product 1260, memory 1210 is a non-transitory medium (even if implemented as a typical volatile computer memory device). Further, those skilled in the art will appreciate that in addition to storing the computer program product 1260 for performing the processes described herein, the memory 1210 and / or the persistent storage device 1270 may be configured to store various data elements referenced and illustrated herein.

[0086] Those skilled in the art will appreciate that computer system 1200 illustrates just one example of a system in which a computer program product according to an embodiment of the present invention may be implemented. As an example of an alternative embodiment, storage and execution of instructions included in a computer program product according to an embodiment of the present invention may be distributed across multiple computers, such as, for example, computers in a distributed computing network.

Claims

1. A computerized method for processing the output of a quantitative polymerase chain reaction (qPCR) instrument, wherein the qPCR instrument is configured to generate amplification data referred to herein as amplification curves, the amplification curves comprising data corresponding to fluorescence measurements obtained for multiple thermal cycles of the qPCR instrument used to perform a qPCR assay on a biological sample, the method using one or more computer processors, The process involves preprocessing multiple amplification curves generated by the qPCR instrument to obtain the designed features and the preprocessed amplification curve for one of the multiple amplification curves. One or more amplification calling deep learning networks process the preprocessed amplification curve and one or more of the designed features to generate amplification call data. One or more curve-quality calling deep learning networks process the pre-processed amplification curves to generate curve-quality call data. A method, which is carried out by performing a process that includes providing an interactive electronic graphical user interface (GUI) configured to facilitate user evaluation of the output of the qPCR instrument using the amplification call data and the curve quality call data.

2. The method according to claim 1, wherein using the amplification call data and the curve quality data includes determining whether the amplification call data should be invalidated.

3. The method according to claim 2, wherein determining whether the amplified call data should be invalidated is based on one or more user settings in the GUI.

4. The method according to claim 2, wherein determining whether the amplified call data should be invalidated is based on the fact that the curve quality call data does not meet a predefined criterion.

5. The method according to claim 4, wherein the predefined criteria are set by the user via the GUI.

6. The method according to claim 1, wherein the amplified call data includes at least class and / or class probability data corresponding to amplified classes and / or non-amplified classes.

7. The method according to claim 1, wherein the amplified call data includes call reliability data.

8. The method according to claim 1, wherein the curve quality call data includes at least class and / or class probability data corresponding to clean curve classes and / or problem curve classes.

9. The method according to claim 1, wherein the curve quality call data includes call reliability data.

10. The method according to claim 1, wherein the amplification calls generated by the one or more amplification calling deep learning networks are compared with amplification results generated by a predetermined non-neural network-based amplification calling algorithm performed on the same preprocessed amplification curve to determine whether to evaluate the curve quality call data to determine whether to invalidate the amplification calls generated by the one or more amplification calling deep learning networks.

11. The method according to claim 10, further comprising evaluating curve quality call data for a plurality of curves corresponding to results from a sample well to determine whether to disable all amplification calls associated with the sample well.

12. The method according to claim 10, wherein the non-neural network-based amplification calling algorithm includes generating the amplification curve by comparing a cycle threshold (Ct) value corresponding to the amplification curve with a Ct value assigned to the qPCR assay performed on the qPCR instrument.

13. The method according to claim 10, wherein the non-neural network-based amplification calling algorithm includes a cycle-relative threshold (Crt) algorithm.

14. The method according to claim 1, wherein one or more of the designed features corresponding to the application curve are processed together with the preprocessed amplification curve by one or more curve quality calling deep learning networks to generate the curve quality call data.

15. The method according to claim 1, wherein the designed feature includes one or more features selected from the group consisting of curve derivative features, time series features, PCRedux features, and cycle relative threshold (CRT) algorithm features.

16. The method according to claim 1, wherein the one or more amplified calling deep learning networks comprises an ensemble of similarly structured but differently trained deep learning networks.

17. The method according to claim 1, wherein the one or more curve-quality calling deep learning networks comprises an ensemble of similarly structured but differently trained deep learning networks.

18. A non-temporary computer-readable medium for storing instructions that, when executed by one or more computer processors, perform processing according to the method described in any one of claims 1 to 17.

19. A computer system comprising one or more processors configured to perform processing according to the method described in any one of claims 1 to 17.

Images