QPCR curve detection

Through machine learning systems and evidence-based learning technology, the amplification calling system is improved, and the problems of insufficient accuracy and difficulty in error identification in the existing technology are solved, thus achieving higher accuracy and automated calling results. reliability.

CN118891679APending Publication Date: 2024-11-01LIFE TECHNOLOGIES CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380027870.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-15
Filing Date
2023-03-15
Publication Date
2024-11-01

AI Technical Summary

Technical Problem

The existing automatic determination amplification method relies on the combination of technology and parameters, resulting in insufficient accuracy and difficulty in effectively identifying amplification curves that require manual viewing, resulting in errors in the automated call results not being discovered in time.

Method used

Using machine learning systems, including deep learning networks and evidence-based learning techniques, the confidence data is extracted through parallel improved amplification calling systems to evaluate the quality of the amplification curve, determine when manual viewing is required, combined with specific types of neural network elements and pretreatment techniques to process amplification curves determined by qPCR.

Benefits of technology

It improves the accuracy of the amplified call system, reduces the need for manual viewing, ensures the identification of potential problem curves and the reliability of automated call results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118891679A_ABST
    Figure CN118891679A_ABST
Patent Text Reader

Abstract

Embodiments of the invention are disclosed that provide improved computer systems, computerized methods, and computer program products for generating and assessing an automated prediction as to whether a specific amplification curve from a qPCR assay is indicative of the presence of a target molecule in a sample. In some embodiments, a prediction is generated using a deep learning network. In some embodiments, a curve quality prediction is generated and used to assess whether amplification prediction can be reliably made from a particular amplification curve or whether the curve reflects anomalies in a qPCR assay. In various embodiments, prediction confidence data is also generated and used with prediction data in an electronic user interface to improve qPCR measurements.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 320,213, filed on March 15, 2022. The entire contents of that application are incorporated herein by reference to the extent permitted by applicable jurisdictions. Background Art

[0003] The present disclosure generally relates to techniques for determining whether a target is present in a sample.

[0004] Amplification curves obtained from real-time (also referred to as quantitative) polymerase chain reaction (qPCR) experiments can be used to determine whether a target is present in a biological sample (e.g., blood or food sample). In a typical qPCR experiment, the fluorescence of the sample is measured after each thermal cycle of the experiment. The group of fluorescence values ​​and cycle numbers associated with a specific determination of the sample forms an amplification curve. Traditionally, algorithmic analysis and / or manual viewing of the amplification curve, and based on visual or other analysis of the characteristics of the curve, determine whether the relevant sample is amplified, which in turn indicates whether the target molecule is present in the sample. Typical algorithmic techniques for determining that a relevant sample has been amplified involve determining whether the associated amplification curve has crossed a threshold, which is fixed or calculated based on the characteristics of the amplification curve. If the threshold is crossed, it is determined that the curve represents amplification; if the threshold is not crossed, it is determined that the curve represents non-amplification. Summary of the invention

[0005] Automated determination of amplification is very important for improving the throughput of sample analysis, which in turn can advance scientific research and improve the provision of time-sensitive clinically important information. Existing methods for automatically determining amplification have relied on a combination of techniques and parameters to improve accuracy. Machine learning techniques (including deep learning networks such as artificial neural networks) can help improve accuracy. Improved machine learning techniques are needed to further improve accuracy and reduce the need for manual review. At the same time, it is important to effectively and optimally determine when manual review is required, and the criteria for triggering manual review of amplification curves that have been evaluated by a computer system can vary depending on the application scenario. Improving the accuracy of the total call rate of amplification call machine learning systems does not necessarily completely solve the problem of knowing which individual amplification curve automated call results are wrong in most risks and therefore require manual review.

[0006] Embodiments of the present invention address aspects of this "which curve" problem by using an improved amplification calling machine learning system (in some embodiments, an improved artificial neural network) in parallel with a machine learning system (e.g., a deep learning network such as an artificial neural network) for evaluating the quality of amplification curves. This can help identify potential problem curves. In some embodiments, an alternative amplification calling algorithm is used to help determine which curves called by the machine learning (e.g., neural network) amplification calling system should be further evaluated for potential curve quality issues that may require invalidating the results.

[0007] In some embodiments, evidential learning techniques are used to extract call confidence data from both the amplification calling neural network and the curve quality calling neural network. The confidence data can be used to further assess when machine-generated amplification calls should be reviewed by a human.

[0008] Some embodiments also provide improved amplification calling and curve quality calling networks based on specific types, variations and arrangements of neural network elements, preprocessing and processing techniques, and / or engineered features for processing amplification curves from qPCR assays. Further details of these embodiments are more fully disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A PCR testing and analysis system according to one embodiment of the present disclosure is shown.

[0010] Figure 2 is a high-level framework diagram of a computer system according to one embodiment of the present disclosure.

[0011] Figure 3 The pre-processing implemented in one embodiment of the present disclosure is shown.

[0012] Figure 4 is a high-level block diagram of the architecture of an augmented call artificial neural network according to one embodiment of the present disclosure.

[0013] Figure 5 According to one embodiment of the present disclosure, Figure 4 Block diagram of the structure.

[0014] Figure 6 is a block diagram of the architecture of a curve quality call artificial neural network according to one embodiment of the present disclosure.

[0015] Figure 7 A computer process for implementing amplification and curve quality calling assessment according to one embodiment of the present disclosure is shown.

[0016] Figure 8is a high-level block diagram of the architecture of an augmented call artificial neural network according to one embodiment of the present disclosure.

[0017] Fig. 9 According to one embodiment of the present disclosure, Figure 8 Block diagram of the structure.

[0018] Fig.10 is a block diagram of the architecture of a curve quality call artificial neural network according to one embodiment of the present disclosure.

[0019] Fig.11 A computer process for implementing confidence processing according to one embodiment of the present disclosure is shown.

[0020] Fig.12 An exemplary computer system that can be configured by a computer program product to implement embodiments of the present disclosure is shown.

[0021] Although the present invention is described with reference to the above drawings, the drawings are intended to be illustrative and other embodiments are consistent with the spirit of the invention and are within the scope of the invention. DETAILED DESCRIPTION

[0022] Various embodiments will now be described more fully below with reference to the accompanying drawings, which form a part of the present invention and are used as illustrations to show specific examples of practical embodiments. However, this specification can be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein; on the contrary, these embodiments are provided to make this specification more thorough and complete and to fully convey the scope of the invention to those skilled in the art. In addition, this specification can be embodied as a method or an apparatus. Therefore, any of the various embodiments herein may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Therefore, the following description should not be considered to be limiting.

[0023] Figure 1 A system 1000 is shown according to an exemplary embodiment of the present disclosure. The system 1000 includes a polymerase chain reaction ("PCR") instrument 101, one or more computers 103, and a user device 107.

[0024] Instructions for implementing the amplification ("amp") curve analysis system 102 reside in a computer program product 104 stored in a storage device 105, and these instructions can be executed by a processor 106. When the processor 106 is executing the instructions of the computer program product 104, the instructions or a portion thereof are typically loaded into a working memory 109, from which the processor 106 can easily access the instructions. In the illustrated embodiment, the computer program product 104 is stored in a storage device 105 or another non-transitory computer-readable medium (which may include a distribution of media across different devices and different locations). In an alternative embodiment, the storage medium is transient.

[0025] In one embodiment, processor 106 actually includes multiple processors, which may include additional working memory (additional processors and memory are not shown separately), including a graphics processing unit (GPU), which includes at least thousands of arithmetic logic units that support large-scale parallel computing. GPUs are commonly used for deep learning applications because GPUs can perform related processing tasks more efficiently than typical general-purpose processors (CPUs). Other embodiments include one or more special-purpose processing units that include systolic arrays and / or other hardware arrangements that support efficient parallel processing. In some embodiments, such special-purpose hardware works with CPUs and / or GPUs to perform various processes described herein. In some embodiments, such special-purpose hardware includes application-specific integrated circuits, etc. (which may refer to a part of an application-specific integrated circuit), field programmable gate arrays, etc., or a combination thereof. However, in some embodiments, processors such as processor 106 can be implemented as one or more general-purpose processors (preferably with multiple cores) without departing from the spirit and scope of the present invention.

[0026] The user device 107 includes a display 108 for displaying the results of the processing performed by the amplification curve analysis system 102. In alternative embodiments, the amplification curve analysis system 102 or a portion thereof may be stored in a storage device and executed by one or more processors resident on the PCR instrument 101 and / or the user device 107. Such alternatives do not depart from the scope of the present invention.

[0027] Figure 2 2 is a high-level framework diagram of a computer system 2000 according to one embodiment of the present disclosure. The computer system 2000 includes a pre-processing block 201, one or more amplification calling neural networks 202, one or more curve quality calling neural networks 203, one or more alternative amplification calling algorithms 204, and an amplification and quality calling assessment block 205.

[0028] Pre-processing block 201 is usually configured to receive one or more amplification curves corresponding to the amplification data obtained from 40 cycles or 50 cycles of qPCR assays. In an alternative embodiment, the amplification curve produced by the PCR experiment with other number of cycles can be processed by the illustrated embodiment using various techniques. The set of discrete data points (e.g., 40 fluorescence values ​​for 40 cycle PCR experiments (or 50 fluorescence values ​​for 50 cycle experiments)) is referred to as "curve" in this article, even if it is not continuous data. Meanwhile, for easier visual display and / or analysis purposes, the best fit continuous curve may or may not fit data. In any case, unless the context implies another meaning, the term amplification "curve" herein is usually used to refer to a set of discrete amplification data obtained and analyzed for the determination of biological samples.

[0029] The preprocessing block 201 processes the amplification curve to generate a preprocessed curve and an engineered feature. The term "engineered" feature refers to various features obtained from a predetermined calculation of the amplification curve, and is distinguished from the "learned" features generated by submitting the amplification curve to a neural network or a network portion (e.g., a convolutional layer of a neural network). In some embodiments of the present disclosure, the engineered features can be obtained during preprocessing and submitted to the neural network together with the preprocessed amplification curve, and the learning speed and / or accuracy can be improved relative to submitting only the preprocessed curve itself. In some embodiments of the present disclosure, the engineered features include one or more of the following: curve derivatives (e.g., first-order, second-order, and / or other higher-order derivatives); time series features; and other features, such as, for example, features from various PCR analysis algorithms including PCRedux features and / or cycle relative threshold features, as further defined and explained elsewhere herein.

[0030] In addition to processing the received amplification curve to obtain the engineered features, the pre-processing block 201 also processes the amplification curve to remove the baseline signal, extend or shorten the curve to a uniform length (e.g., 40 cycles or 50 cycles) and normalize the curve. These operations are used to generate the pre-processed amplification curve output by the block 201.

[0031] Amplification ("amp") calling neural network 202, alternative amplification calling algorithm 204, and curve quality calling neural network 203 receive pre-processed amplification curves. In the illustrated embodiment, amplification calling neural network 202 receives engineered features from pre-processing block 201, and alternative amplification calling algorithm 204 receives a portion of the engineered features associated with executing a particular alternative amplification calling algorithm. In alternative embodiments, curve quality calling neural network 203 also receives engineered features from a pre-processing block such as pre-processing block 201.

[0032] Amplification and curve quality call assessment block 205 receives amplification call data from amplification call neural network 202. In the illustrated embodiment, the amplification call data includes the class probabilities of at least amplification categories and non-amplification categories, and also includes confidence data generated from evidential learning techniques. However, in alternative embodiments, confidence data is not generated and used. Block 205 also receives amplification call data from alternative amplification call algorithm 204. Block 205 receives curve quality call data from curve quality call neural network 203. In the illustrated embodiment, the curve quality call data includes the class probabilities for at least "clean" curve categories and abnormal ("problem") curve categories, and also includes confidence data generated from evidential learning techniques. However, in alternative embodiments, confidence data is not generated and used.

[0033] As will be Figure 7 As further described in the context of , the evaluation block 205 receives amplification calls and curve quality call information and uses it to evaluate which neural network-based amplification calls (if any) should be invalidated. In some embodiments, the indication that a neural network (or other ML-based) amplification call should be invalidated is based on a user-selected setting in an electronic graphical user interface (GUI). In some embodiments, the GUI allows the user to selectively invalidate the call based on a visual inspection of the corresponding amplification curve marked for review by the evaluation block 205. In some embodiments, the GUI allows the user to set parameters (such as confidence requirements, curve quality probability requirements, and / or other requirements) that the evaluation block 205 uses to determine which neural network-based amplification calls should be invalidated and / or which neural network-based amplification calls should be marked for further review by the user.

[0034] Figure 3 In one embodiment of the present disclosure, Figure 2The pre-processing block 201 of the embodiment of the present invention is implemented by the pre-processing block 201 of the present invention. Step 301 receives the amplification curve obtained from the qPCR assay, for example, running on a qPCR instrument. Step 302 removes the baseline from the curve, in this example, the baseline corresponds to the baseline signal generated by the instrument in the absence of a template. Before reaching a sufficient signal-to-noise ratio, the baseline contains noise. It will be appreciated by those skilled in the art that there are various ways in which baseline estimation can be implemented. In one embodiment, the baseline is estimated by linear least squares through the point of the curve selected as the baseline region. In one embodiment, the baseline is estimated by the average value of the point of the curve selected as the baseline region. Step 303 determines whether the received amplification curve is less than 50 cycles. If so, the process proceeds to step 304, and the data is extended to 50 cycle data. It will be appreciated by those skilled in the art that various extensions can be implemented. In one embodiment, a constant fill is used, which simply involves repeating the last cycle value (for example, the value corresponding to the 40th cycle) until the data is 50 cycles long. The extended 50 cycle curve obtained from step 304 is provided to a derivative calculation step 305 and to a normalization step 306. If the result of step 303 is no, the 50 cycle curve is directly provided to steps 305 and 306 without extension. In an alternative embodiment, the data can be normalized to 40 cycle data, and the 50 cycle or other curves greater than 40 cycles can be truncated or interpolated to obtain 40 cycle data, as described, for example, in U.S. application serial number 17,346,147, filed on June 11, 2021, and published as US2021 / 0391033, the entire contents of which are incorporated herein by reference.

[0035] In the illustrated embodiment, the derivative calculation step 305 calculates the derivative from the unnormalized amplification curve of 50 qPCR cycles. It calculates the 1st and 2nd order derivatives. In one example, the derivative calculation is performed using a Savitzky-Golay filter and the following Savitzky-Golay parameters: polynomial order = 3; and window length = 9. The 1st and 2nd order derivatives can be normalized using maximum normalization. The resulting normalized derivative values ​​represent a portion of the engineered features output by preprocessing 3000.

[0036] Step 306 uses the cycle threshold ("Ct") value for a particular assay on a particular instrument to normalize the amplification curve. Normalization can be performed by dividing each value of the unnormalized curve by the Ct value corresponding to the assay from which the curve was obtained. In alternative embodiments, different values ​​can be used for normalization.

[0037] Step 308 uses the normalized curve to calculate additional engineered features. In one embodiment, the additional engineered features include, for example, one or more of the following: features associated with the cycle relative threshold ("Crt") algorithm, a version of which is described in US 2016 / 0110495A1 ("Crt patent publication") of US patent application Ser. No. 14 / 921,948; features from PCRedux (see PCRedux: A Data Mining and Machine Learning Toolkit for qPCR Experiments, URL: biorxiv.org / content / 10.1101 / 2021.03.31.437921v1 ); time series features (e.g., shapelets, wavelets); and features associated with various anomalies (e.g., waterfalls, bubbles, wavy baselines, and creep). The entire texts of the two publications cited above are hereby incorporated by reference herein.

[0038] In one embodiment, one or more of the features from the Crt algorithm listed and described below in Table 1 are calculated and used. Cross references to the above-mentioned Crt patent publications are also provided below (if applicable):

[0039] Table 1

[0040]

[0041]

[0042] "AmpStatus" is the predicted result of the Crt algorithm. It can be obtained using software from commercially available products from ThermoFisher Scientific, which commercially available products include one or more of the following products: QuantStudio TM 12k Flex v1.5; QuantStudio TM 6 and 7Flex v1.7.2; Relative Quantification v4.3 (cloud application); Standard Curve v4.0 (cloud application); and Presence and Absence Analysis v1.9 (cloud application).

[0043] In one embodiment, one or more of the PCRedux features listed in Table 2 are determined and used:

[0044] Table 2

[0045]

[0046]

[0047] In one embodiment, step 308 normalizes the additional engineered features after they are calculated. In one embodiment, a quantile normalizer is used. The quantile normalizer transforms the features to follow a normal distribution. The number of quantiles calculated is equal to 1000. The scalar to be used for normalization is obtained from a representative training set selected for a given model.

[0048] Step 307 outputs the normalized derivatives generated by step 305, the normalized curves generated by step 306, and the additional normalized engineered features generated by step 308. These values ​​are determined by, for example, Figure 2 The output of the pre-processing block 201 is shown and used by various parts of the amplification analysis system, as further described below.

[0049] Figure 4 An embodiment according to the present disclosure is shown Figure 2 A high-level block diagram of the architecture 4000 of one of the augmented call neural networks 202 of FIG. 4000 is shown. Note that in one embodiment, the augmented call network 202 uses an ensemble approach that includes multiple (e.g., 4, 5, or some other number) of similarly structured but differently trained neural networks. Various techniques can be used to combine the results of the individual networks in the ensemble (e.g., averaging, majority voting, etc.) to return the class probabilities for a given curve generated for a particular assay of a sample. In addition, various techniques can be used to train the networks of the ensemble differently (e.g., using different training data, ordering the training data differently, initializing weights / filters to different values, etc.). Figure 4 The structure of each neural network for the set of neural networks 202 is shown according to one embodiment of the present disclosure.

[0050] Structure 4000 includes a neural network (when reference is made herein to "neural network", an "artificial" (i.e., computerized) neural network is always implied; other deep learning networks may be used in alternative embodiments, and "deep learning network" as used herein will also be assumed to imply a "computerized deep learning network") first portion 401, a neural network second portion 402, and a neural network third portion 403. In the illustrated embodiment, neural network second portion 402 (from Figure 2 The preprocessing block 201 of the neural network receives the preprocessed amplification curve and the selected engineered features. In this example, the selected engineered features received by the neural network second part 402 include the normalized first and second order derivatives calculated at step 305, as previously described in Figure 3However, in alternative embodiments, other engineered features may be received by the second portion 402 in addition to or in lieu of derivative values, or in some embodiments, a pre-processed amplification curve may be received by the second portion 402 without any engineered features and used.

[0051] The additional engineered features are received by the neural network first portion 401. In the illustrated embodiment, the additional engineered features include Figure 3 One or more of the features described in the context of step 308 of (e.g., Crt algorithm features, PCRedux features, time series features, known anomaly features, etc.). However, in alternative embodiments, different engineered features may be received by the neural network first portion 401.

[0052] Neural network first part 401 and neural network second part 402 process their respective inputs. Their respective outputs are connected by a connection function 404, which provides the connected outputs as input to neural network third part 403. Neural network third part 403 processes the connected outputs of first part 401 and second part 402 and generates call and confidence data as output. In an alternative embodiment, only the amplification call data is determined and output by neural network third part 403.

[0053] Figure 5 According to one embodiment of the present disclosure, Figure 4 4000. As shown in the figure, the first part 401 of the neural network includes a fully connected layer 501, a leaky ReLU layer (activation function) 502, a fully connected layer 503 and a leaky ReLU layer (activation function) 504. The second part 402 of the neural network includes a separable convolution layer 505, a leaky ReLU layer (activation function) 506, a separable convolution layer 507, a leaky ReLU layer (activation function) 508 and a flattening layer 509. As described above in Figure 4 As described in the context of , the connection operation 404 connects the outputs of the first part 401 and the second part 402 and provides the connected output to the input of the third part 403. The third part 403 includes a fully connected layer 511, a leaky ReLU layer (activation function) 512, a fully connected layer 513, a ReLU layer (activation function) 514 and an evidence processing module 515.

[0054] The above structure will now be described in further detail about layer input / output size and other characteristics of a particular embodiment. The disclosed details have been found to work well for the purposes of the amplification curve analysis applications described herein. However, it will be appreciated by those skilled in the art that many of these details may be different from those referenced below without necessarily departing from the spirit and scope of the present disclosure.

[0055] In one embodiment, the fully connected layer 501 has 9 units (1x9) input data and 16 units (1x16) output data. The leaky ReLU layer 502 is an activation function that operates on the output of the fully connected layer 501, and it provides a 1x16 output to the fully connected layer 503, which in turn provides a 1x16 output processed by the leaky ReLU layer 504 to provide a 1x16 output to the connection function 404.

[0056] Turning to further details of the second portion 402 of the neural network, in this embodiment, the separable convolutional layer 505 receives 3-channel data, each channel including 50 units, i.e., 3x50 data. Specifically, for a given cycle of the 50-cycle amplification curve data, three values ​​are provided as input to the convolutional layer 505: the normalized fluorescence value, the first derivative, and the second derivative of the preprocessed amplification curve at that cycle.

[0057] In the illustrated embodiment, the separable convolution layer 505 uses 16 filters, each of size 3x3. Using a stride of 2, and applying sufficient padding to the input array to obtain the desired output feature map size, the output feature map size is 16x26 for the separable convolution layer 505. In this embodiment, this means that the 3X50 input array is separated into three "sub" arrays, each of size 1X50. Similarly, each 3X3 filter is separated into three "sub" filters, each of size 1X3. Each corresponding subarray is convolved with the corresponding subfilter to create a corresponding 1D feature map. The generated feature maps are "stacked" to provide a 2D feature map. Point-by-point convolution is performed on the 2D feature map, where it is convolved with the 1X3 filter. This point-by-point step outputs a 1x26 array, which is the same size as the feature map that would be produced by a normal convolution of a 3X3 filter with a 50X3 input array, including the necessary padding. In the illustrated embodiment, where 16 filters are used, the resulting output from the separable convolutional layer 505 is a 16x26 feature map.

[0058] In alternative embodiments, more or fewer filters may be used in each of the separable convolutional layers described herein. The number of filters shown in the current exemplified embodiment is preferred, but may be different in alternative embodiments without departing from the spirit and scope of the present disclosure. In the exemplified embodiment, the separable convolutional layer 505 uses a stride of 2 and uses copy padding. Copy padding causes a copy of the sequence in the input array to be padded, reversed, and then the reversed sequence is used to fill either end of the sequence. Padding allows the convolution process to create a feature map with a desired length size. Alternative embodiments use different types of padding, such as "same" padding, without departing from the spirit and scope of the present disclosure.

[0059] The other separable convolutional layers in the illustrated embodiment use the same filter size, stride, padding type, and depth direction, followed by point-wise convolution; those features will be assumed and not repeated further below, although they may vary in alternative embodiments.

[0060] The resulting 16x26 output from the separable convolution layer 505 is processed by a leaky ReLU activation function (represented here by a leaky ReLU layer 506), and the result is provided as 16x26 data to the separable convolution layer 507. The separable convolution layer 507 uses 8 filters. The resulting 8x13 output data is processed by a leaky ReLU layer 508, and the result is provided as 8x13 data to the flattening layer 509. The flattening layer 509 converts the 8x13 data into a single column (i.e., one-dimensional) array of length 104, and provides the resulting 1x104 output to the concatenation operation 404.

[0061] The 1x16 output of the leaky ReLU 504 of the first part 401 and the 1x104 output of the flatten layer 509 of the second part 402 are connected at box 404, and the resulting 1x120 data of the connection is provided to the fully connected layer 511 of the third part 403.

[0062] The fully connected layer 511 receives the 1x120 connected data and provides 1x16 output data, which is processed by the leaky ReLU layer 512, which in turn provides the 1x16 output to the fully connected layer 513, which in turn provides the output from the 2-node final layer providing 2-unit output (the output dimension corresponds to the number of categories used for the illustrated embodiment, i.e., two categories: augmented and non-augmented).

[0063] In the illustrated embodiment, the output of the fully connected layer 513 is processed by the ReLU layer 514. The illustrated embodiment implements evidential learning. The output of the ReLU layer 514 is used as an evidence vector by the evidential processing block 515. The evidential processing block 515 processes the received evidence vector using evidential learning techniques and obtains class probability determinations corresponding to the amplification and non-amplification classifications for each amplification curve and confidence measurements for the corresponding classification data. In Sensoy et al., “Evidential Deep Learning to Quantify Classification Uncertainty”, arXiv:1806.01768v3 [cs.LG] 31 Oct 2018, https: / / arxiv.org / pdf / 1806.01768.pdf An example of an evidential learning technique used by block 515 in the illustrated embodiment is described in , which is incorporated herein by reference in its entirety.

[0064] In an alternative embodiment, the ReLU layer 514 and the evidential processing block 515 are replaced by a softmax layer that simply converts the output of the fully connected layer 513 into class probabilities and does not generate confidence data.

[0065] Figure 6 According to one embodiment of the present disclosure Figure 2 6000 of the architecture of one of the curve quality call neural networks 203. Note that in one embodiment, the curve quality call neural network 203 uses an ensemble approach that includes multiple (e.g., 4, 5, or some other number) structurally similar but differently trained neural networks (as previously described in the context of augmenting the call neural network 202). Figure 6 The structure of each neural network for the set of neural networks 203 is shown according to one embodiment of the present disclosure.

[0066] In the illustrated embodiment, the neural network is submitted to the second portion 402 (see Figure 4 and Figure 5 The same pre-processed amplification curves and selected engineered features (first and second derivatives) of Figure 6 The curve quality shown calls the first part of the neural network 601. In an alternative embodiment, only the pre-processed amplification curve is submitted. In other alternative embodiments, one or more additional engineering features are also submitted together with the pre-processed amplification curve.

[0067] The first neural network part 601 can be understood as operating to extract learned features from the pre-processed amplification curves (and selected engineered features of those curves) it receives. Optimized parameter values ​​(e.g., convolution filter values) for extracting those features from each curve are learned during the training of the neural network. The second part 602 can be understood as a classification network for classifying curves as "clean" or "problematic".

[0068] As shown, the neural network first part 601 includes a separable convolution layer 605, a leaky ReLU layer 606, a separable convolution layer 607, a leaky ReLU layer 608, and a flattening layer 609. In the illustrated embodiment, the more detailed characteristics of the separable convolution layer 605, the leaky ReLU layer 606, the separable convolution layer 607, the leaky ReLU layer 608, and the flattening layer 609 (e.g., input / output data dimensions, number of filters, filter size, stride, padding type) are the same as those previously described for the separable convolution layer 505, the leaky ReLU layer 506, the separable convolution layer 507, the leaky ReLU layer 508, and the flattening layer 509, respectively, and therefore these descriptions are not repeated here.

[0069] Thus, the input size provided from the flattening layer 609 to the fully connected layer 611 is 1x104. The fully connected layer 611 outputs 1x16 data for processing by the leaky ReLU layer 612. The resulting 1x16 data is input to the fully connected layer 613, which outputs 1x2 data to the ReLU layer 614. The resulting evidence vector is processed by the evidential processing block 615 in a manner similar to that described above in the context of the evidential processing block 515. The evidential processing block 615 outputs class probabilities and corresponding confidence measures for the following curve quality classifications: "clean" or "problematic".

[0070] In an alternative embodiment, the ReLU layer 614 and the evidential processing block 615 are replaced by a softmax layer that simply converts the output of the fully connected layer 613 into class probabilities and does not generate confidence data.

[0071] Figure 7 A processing flow 7000 implemented by the amplification and curve quality call evaluation block 205 is shown. The processing flow 7000 receives amplification call data and curve quality call data generated by a curve analysis algorithm such as those previously described herein. The processing flow 7000 provides an example of a processing flow for curve analysis result processing that can be used to support presentation of results to a user interacting with a graphical user interface (GUI) on a user device that implements an embodiment of a curve analysis system consistent with the present disclosure or is coupled to other computers that implement an embodiment of a curve analysis system consistent with the present disclosure.

[0072] Process 7000 is at step 701. Step 702 receives an amplification call from an alternative amplification call algorithm such as one or more of those previously described (or from other alternative amplification call algorithms). Step 703 receives an amplification call from a neural network / machine learning based algorithm (such as, for example, Figure 2 The neural network 202 of FIG. 700 receives an augmentation call. Step 704 determines whether the results received at steps 702 and 703 match. If so, processing proceeds to step 707 and the curve quality indicator (CQI) flag is not set. If the result of step 704 is no, step 705 evaluates the curve quality processing algorithm (such as, for example, Figure 2 706 determines if the curve quality call is "clean" (i.e., has no "problems"). If not, processing proceeds to step 708 and the CQI flag is set. If yes, processing proceeds to step 707 and the CQI flag is not set for the curve. In either case, processing proceeds to step 709, which determines if the CQI analysis is complete for all curves generated from the particular sample well whose curves generated for one or more tests are currently being analyzed. If the result of step 709 is no, processing returns to steps 702 and 703 for the next curve. If the result of step 709 is yes, processing proceeds to step 710.

[0073] Step 710 determines whether any curve corresponding to the hole currently being analyzed has a CQI flag. If not, processing proceeds directly to steps 712 and 716, and none of the tests associated with the current hole being analyzed are invalidated. If yes, processing proceeds to step 711 to determine whether CQI invalidation is currently enabled. In some embodiments, this is a user-selected setting. In alternative embodiments, whether CQI invalidation is enabled can be automatically determined based on various user-selected or predetermined factors and / or can be enabled by default. If CQI invalidation is not enabled, the result of step 711 is no, and processing proceeds to steps 712 and 716, and none of the tests associated with the current hole being analyzed are invalidated. If CQI invalidation is enabled, the result of step 711 is yes, and processing proceeds to step 713 to determine whether calls from only individual tests associated with the CQI flag in a given hole should be invalidated, or whether all calls for tests in the hole should be invalidated. If the result of step 713 is "Call", only the test result associated with the CQI marker is invalidated. If the result of step 713 is "Well", all results for that well are invalidated, regardless of whether the individual test is associated with a CQI-labeled amplification curve.

[0074] Those skilled in the art will appreciate that in various embodiments, different factors may be used to determine whether to invalidate all tests in a well or only the tests associated with the call for the CQI-tagged curve. In some embodiments, this determination may be made from available data based on user-set criteria. For example, a user may set criteria based on the number or percentage of CQI-tagged calls in a given well to determine whether to invalidate all calls in a well. Once user-based criteria are set, the result may be automatically achieved based on the call and CQI tag data. If a confidence measurement result is available, in some embodiments, the user may set criteria based on confidence scores that meet or do not meet a specific threshold at the level of a separate call and / or on all calls in a well. Conditional criteria based on various combinations of call and curve quality data may also be implemented. Various alternatives consistent with the present disclosure will be apparent to those skilled in the art.

[0075] Figure 8 An embodiment according to the present disclosure is shown Figure 2 A high-level block diagram of an alternative architecture 8000 for one of the augmented call neural networks 202. The illustrated structure 8000 is Figure 4 An alternative to structure 4000 is shown.

[0076] As previously in Figure 4 In the context of, in one embodiment, the amplification call network 202 uses an ensemble approach including multiple (e.g., 4, 5, or some other number) neural networks of similar structure but different training. Various techniques can be used to combine the results of the individual networks in the ensemble (e.g., averaging, majority voting, etc.) to return the class probability of a given curve generated for a particular determination of the sample. In addition, various techniques can be used to train the networks of the ensemble differently (e.g., using different training data, sorting the training data differently, initializing weights / filters to different values, etc.). Figure 8 The structure of each neural network for the set of neural networks 202 is shown according to an alternative embodiment of the present disclosure.

[0077] Structure 8000 includes a first neural network portion 801, a second neural network portion 802, and a third neural network portion 803. In the illustrated embodiment, the second neural network portion 802 (from Figure 2 The preprocessing block 201 of the neural network receives the preprocessed amplification curve and the selected engineered features. In this example, the selected engineered features received by the neural network second part 802 include the normalized first and second order derivatives calculated at step 305, as previously described in Figure 3However, in alternative embodiments, other engineered features may be received by the neural network second portion 802 in addition to or in lieu of derivative values, or in some embodiments, a pre-processed amplification curve may be received and used by the neural network second portion 802 without any engineered features.

[0078] The additional engineered features are received by the neural network first portion 801. In the illustrated embodiment, the additional engineered features include Figure 3 One or more of the features described in the context of step 308 of (e.g., Crt algorithm features, PCRedux features, time series features, known anomaly features, etc.). However, in alternative embodiments, different engineered features may be received by the neural network first portion 801.

[0079] Neural network first part 801 and neural network second part 802 process their respective inputs. Their respective outputs are connected by connection function 804, which provides the connected output as input to neural network third part 803. Neural network third part 803 processes the connected output of neural network first part 801 and neural network second part 802, and generates call amplification call data as output. In one example, such call data includes class probabilities (sometimes referred to herein as prediction scores) for one or more classifications (such as amplification / or non-amplification).

[0080] like Figure 8 As further shown, the class probabilities output by the third portion 803 of the neural network are also provided to a confidence processing block 805. The confidence processing block 805 also receives quantified cycle (Cq) data (sometimes referred to as cycle threshold or Ct data), which includes Cq values ​​for each amplification curve for which amplification class probabilities are generated. The confidence processing block 805 generates confidence data. Fig.11 One embodiment of the processing for implementing the confidence processing block 805 is further described in the context of .

[0081] Fig. 9 According to one embodiment of the present disclosure, Figure 8 8000. As shown, the first part 801 of the neural network includes FCwLD processing blocks 901 and 902. Each FCwLD processing block includes a fully connected layer 921, followed by a leaky ReLU layer (activation function) 922 and a dropout layer 923. The second part 802 of the neural network includes a separable convolution layer 903, a leaky ReLU layer (activation function) 904, a separable convolution layer 905, a leaky ReLU layer (activation function) 906 and a flattening layer 907. As described above in Figure 8As described in the context of , the connection operation 804 connects the outputs of the first neural network part 801 and the second neural network part 802 and provides the connected output to the input of the third neural network part 803. The third neural network part 803 includes FCwLD blocks 908, 909, 910, followed by a fully connected layer 911 and a softmax layer 912. Similar to FCwLD blocks 901 and 902, FCwLD blocks 908, 909, 910 each include a fully connected layer 921, followed by a leaky ReLU layer 922 and a dropout layer 923.

[0082] In the illustrated embodiment, each dropout layer 923 in the FCwLD block uses a dropout rate of 0.5. However, other rates may be used without departing from the spirit and scope of the present disclosure. In the illustrated embodiment, the convolutional layers 903 and 905 may be separated to be similar to the Figure 5 The separable convolutional layers 505 and 507 in FIG. 5 operate in the manner described above.

[0083] In the illustrated embodiment, the softmax layer 912 outputs class probabilities (prediction scores) for use by the computer user interface and for use by the confidence processing block 805.

[0084] In an alternative embodiment, the softmax layer 912 and confidence processing block 805 are replaced by a leaky ReLU layer and an evidential learning block that generate augmented call data and confidence data as previously described.

[0085] Fig.10 An embodiment according to the present disclosure is shown Figure 2 A high-level block diagram of an alternative architecture 10000 of one of the curve quality calls neural networks 203.

[0086] As shown, the first part 1001 of the neural network includes a separable convolution layer 1003, a leaky ReLU layer (activation function) 1004, a separable convolution layer 1005, a leaky ReLU layer (activation function) 1006, and a flattening layer 1007. The second part 1002 of the neural network includes FCwLD blocks 1008, 1009, and 1010, followed by a fully connected layer 1011 and a softmax layer 1012. Similar to FCwLD blocks 901 and 902, FCwLD blocks 1008, 1009, and 1010 each include a fully connected layer 921, followed by a leaky ReLU layer 922 and a dropout layer 923.

[0087] In the illustrated embodiment, each dropout layer 923 in the FCwLD block uses a dropout rate of 0.5. However, other rates may be used without departing from the spirit and scope of the present disclosure. In the illustrated embodiment, the convolutional layers 1003 and 1005 may be separated to be similar to the Figure 6 The separable convolutional layers 605 and 607 in FIG. 1 are operated in the manner described above.

[0088] In the illustrated embodiment, the softmax layer 1012 outputs class probabilities (prediction scores) for use by a computer user interface and for use by a confidence processing block 1025 .

[0089] In an alternative embodiment, the softmax layer 1012 and confidence processing block 10255 are replaced by a leaky ReLU layer and an evidential learning block that generate curve quality recall data and confidence data as previously described.

[0090] Fig.11 is a diagram showing an embodiment of the present disclosure. Figure 8 and Fig. 9 A flowchart of the process 1100 implemented by the amplification prediction confidence processing block 805 is shown. Fig.10 Similar processing steps may be used for the curve quality prediction confidence processing block 1025 shown. Process 1100 will first be described in the context of the amplification prediction confidence processing performed by block 805.

[0091] Process 1100 begins at step 1101, which calculates a collection confidence for each prediction score. As previously explained, some embodiments of the present disclosure use a collection of deep learning networks that are similar in structure but trained differently. In a related embodiment using process 1100, each deep learning amplification call network 202 in a collection of such networks generates a prediction score for a specific amplification curve. As will be appreciated by those skilled in the art, various statistical techniques can be used to generate a confidence metric using a set of data points obtained from a base population.

[0092] In some embodiments, all individual prediction scores generated by the amplification call network 202 in the set and assuming a normal distribution are used to calculate a 95% confidence interval for the set prediction score. Then, the difference between the upper and lower limits of the confidence interval is used as a confidence metric, and an empirical threshold is used to assign and output an intuitive confidence level (e.g., "low," "medium," or "high" confidence) to the prediction score result. For example, the confidence interval for the prediction score of the set may have a lower limit (LB) of 0.75 and an upper limit (UB) of 0.85. The resulting confidence metric (UB-LB) will be 0.10. Using a threshold value of, for example, 0.15, any value below 0.15 is a "high" confidence, and the process will assign a high confidence level to the corresponding amplification prediction score of the set as a whole (the total prediction score of the set may, for example, be taken as the average prediction score of all scores generated by the amplification call network of the set for a given curve).

[0093] Step 1102 determines whether the calculated confidence metric is below a first lower threshold. If the result of step 1102 is yes, the process proceeds to step 1103, which outputs a "high" confidence level. If the result of step 1102 is no, the process proceeds to step 1104, which determines whether the confidence metric (e.g., UB-LB of the confidence interval) is below the next higher threshold. If the result of step 1104 is yes, the process proceeds to step 1105, which outputs a "medium" confidence level. If the result of step 1104 is no, the process proceeds to step 1106, which outputs a "low" confidence level.

[0094] In one embodiment, the following thresholds are used: If the UB-LB of the 95% confidence interval is less than 0.25, the confidence is determined to be "high". If the UB-LB of the 95% confidence interval is equal to or greater than 0.25 but less than 0.6, the confidence level is determined to be "medium". If the UB-LB of the 95% confidence interval is 0.6 or greater, the confidence level is determined to be "low". However, those skilled in the art will understand that the appropriate threshold for confidence determination will depend on the data set and context. And different thresholds can be used depending on the statistics of the underlying data set.

[0095] Moreover, the threshold value used can be changed based on what Cq value is associated with the amplification curve being processed for prediction. For example, in some Cq ranges, a prediction score of a tighter range can be expected. As an example, if the Cq value corresponds to a cycle in the range of 12-20, the amplification category probability will be expected to be consistently high (close to 1) and a tighter confidence interval range (lower threshold of UB-LB) may be required (for example, 0.2 or 0.15 instead of 0.25) in order to assign a "high" confidence value to the category probability (prediction score). Therefore, in some embodiments, the first (lower) threshold value depends on the Cq value. For example, when the corresponding amplification curve has a relatively low Cq value (for example, 12-20), a first threshold value lower than 0.25 (for example, 0.15, 0.2) can be used in step 1103, and 0.25 or some other values ​​higher than 0.2 can be used for a curve with a higher Cq value. In some embodiments, the second (higher) threshold value also depends on the Cq value.

[0096] In some embodiments, Fig.10The box 1025 of uses the process 1100 (or similar process) to assign confidence values ​​to the prediction scores of the curve quality prediction. For the curve quality prediction, the appropriate threshold for assigning low, medium or high confidence will also depend on the specific data set and context. But the process can be similar to the process described above in the context of the amplification prediction confidence process. For example, in some embodiments of the process 1100 used for the curve quality prediction confidence process, all the individual prediction scores generated by the curve quality call network 203 in the set and assuming a normal distribution are used to calculate the 95% confidence interval for the set prediction score. Then, the difference between the upper and lower limits of the confidence interval is used as a confidence metric, and the empirical threshold is used to assign intuitive confidence levels (e.g., "low", "medium" or "high" confidence) to the prediction score results. In some embodiments, 0.25 is used as the first (lower) threshold, and if the set confidence metric is lower than 0.25, a "high" confidence is assigned. If the set confidence measure is equal to or greater than 0.25 but below 0.6, a "medium" confidence is assigned. If the set confidence measure is not below 0.6, a "low" confidence is assigned.

[0097] Although Cq values ​​may be less relevant to assessing the confidence of curve quality predictions than to assessing the confidence of amplification predictions, in some embodiments, other context-dependent factors may be used to modify the thresholds used for curve quality confidence processing. As described above, those skilled in the art will appreciate that appropriate thresholds for assigning confidence levels may depend on the context of a particular data set. The above thresholds are set forth by way of example only.

[0098] Those skilled in the art will appreciate that, in alternative embodiments, the confidence metric itself and / or other underlying statistics about the relevant data set may be output via a computer user interface without assigning an intuitive confidence value, such as "low," "medium," or "high." Experienced users may prefer to assess confidence based on an underlying confidence interval metric rather than automatically generating an assessment.

[0099] Fig.12 An exemplary computer system is shown that can be configured by a computer program product to implement embodiments of the present invention.

[0100] In this example, the computer system 1200 may provide one or more of the components of the automated qPCR curve analysis system, which is configured to implement one or more logic modules and artificial neural networks and associated components for a computer-implemented qPCR automated analysis system and an associated interactive graphical user interface. The computer system 1200 executes the instruction code contained in the computer program product 1260. The computer program product 1260 includes executable code in an electronically readable medium, which can instruct one or more computers (such as computer system 1200) to perform the processing of the exemplary method steps performed by the embodiments cited herein. The electronically readable medium can be any non-transient medium that stores information electronically and can be accessed locally or remotely, for example, via a network connection. In an alternative embodiment, the medium can be transient. The medium may include multiple geographically dispersed media, each of which is configured to store different parts of the executable code at different locations or at different times. The executable instruction code in the electronically readable medium instructs the illustrated computer system 1200 to implement various exemplary tasks described herein. The executable code for indicating the implementation of the tasks described herein will usually be implemented in software. However, it will be appreciated by those skilled in the art that a computer or other electronic device may utilize code implemented in hardware to perform many or all of the identified tasks without departing from the present invention.It will be appreciated by those skilled in the art that many variations of the exemplary methods within the spirit and scope of the present invention may be found to implement executable code.

[0101] The code or a copy of the code contained in the computer program product 1460 may reside in one or more permanent storage media (not separately shown) communicatively coupled to the computer system 1200 for loading and storage in the permanent storage device 1270, and / or reside in the memory 1210 for execution by the processor 1220. The computer system 1200 also includes an I / O subsystem 1230 and peripherals 1240. The I / O subsystem 1230, the peripherals 1240, the processor 1220, the memory 1210, and the permanent storage device 1270 are coupled via a bus 1250. Similar to the permanent storage device 1270 and any other permanent storage device that may contain the computer program product 1260, the memory 1210 is a non-transitory medium (even if implemented as a typical volatile computer storage device). In addition, those skilled in the art will appreciate that in addition to storing the computer program product 1260 for implementing the processes described herein, the memory 1210 and / or the permanent storage device 1270 may be configured to store various data elements mentioned and illustrated herein.

[0102] Those skilled in the art will appreciate that computer system 1200 illustrates only one example of a system in which a computer program product according to an embodiment of the present invention may be implemented. To cite only one example of an alternative embodiment, the storage and execution of instructions contained in a computer program product according to an embodiment of the present invention may be distributed across multiple computers, such as, for example, computers of a distributed computing network.

Claims

1. A method of computerized processing of the output of a quantitative polymerase chain reaction (qPCR) instrument, the qPCR instrument being configured to generate amplification data, referred to herein as an amplification curve, the amplification curve comprising data corresponding to fluorescence measurements obtained for a plurality of thermal cycles of the qPCR instrument for performing a qPCR assay on a biological sample, the method being implemented by using one or more computer processors to perform a process comprising: preprocessing a plurality of amplification curves generated by the qPCR instrument to obtain an engineered feature and a preprocessed amplification curve for an amplification curve in the plurality of amplification curves; processing the preprocessed amplification curves and one or more of the engineered features by one or more amplification calling deep learning networks to generate amplification calling data; processing the pre-processed amplification curves by one or more curve quality calling deep learning networks to generate curve quality calling data; as well as The amplification call data and the curve quality call data are used to provide an interactive electronic graphical user interface (GUI) configured to facilitate user evaluation of the output of the qPCR instrument.

2. The method of claim 1, wherein using the amplification call data and the curve quality data comprises determining whether the amplification call data should be invalidated. 3 . The method of claim 2 , wherein determining whether the augmentation call data should be invalidated is based on one or more user settings in the GUI.

4. The method of claim 2, wherein determining whether the amplification call data should be invalidated is based on failure of the curve quality call data to meet a predefined criterion. The method of claim 4 , wherein the predefined criteria are set by a user via the GUI.

6. The method according to any one of claims 1 to 5, wherein the amplification call data comprises class and / or class probability data corresponding to at least an amplification class and / or a non-amplification class.

7. A method according to any one of claims 1 to 6, wherein the amplification call data includes call confidence data.

8. The method according to any one of claims 1 to 7, wherein the curve quality call data comprises class and / or class probability data corresponding to at least a clean curve class and / or a problem curve class.

9. The method of any one of claims 1 to 8, wherein the curve quality call data comprises call confidence data.

10. A method according to any one of claims 1 to 9, wherein the amplification calls generated by the one or more amplification call deep learning networks are compared with the amplification results generated by a predetermined non-neural network-based amplification calling algorithm performed on the same preprocessed amplification curve to determine whether to evaluate the curve quality call data for determining whether to invalidate the amplification calls generated by the one or more amplification call deep learning networks.

11. The method of claim 10, further comprising evaluating curve quality call data for a plurality of curves corresponding to results from a sample well to determine whether to invalidate all amplification calls associated with the sample well.

12. The method of any one of claims 10 to 11, wherein the non-neural network-based amplification calling algorithm comprises comparing a cycle threshold (Ct) value corresponding to the amplification curve with a Ct value assigned to the qPCR assay run on the qPCR instrument to generate the amplification curve.

13. The method of any one of claims 10 to 11, wherein the non-neural network-based amplification calling algorithm comprises a cycle relative threshold (Crt) algorithm.

14. The method of any one of claims 1 to 13, wherein one or more of the engineered features corresponding to the amplification curves are processed together with the pre-processed amplification curves by the one or more curve quality calling deep learning networks to generate the curve quality calling data.

15. The method of any one of claims 1 to 14, wherein the engineered features include one or more features selected from the group consisting of: a curve derivative feature, a time series feature, a PCRedux feature, and a cycle relative threshold (Crt) algorithm feature.

16. The method of any one of claims 1 to 15, wherein the one or more augmented call deep learning networks comprise a collection of deep learning networks that are structurally similar but trained differently.

17. The method of any one of claims 1 to 16, wherein the one or more curve quality-recalling deep learning networks comprises a collection of deep learning networks that are structurally similar but trained differently.

18. A non-transitory computer readable medium storing instructions that, when executed by one or more computer processors, perform processing according to the method of any one of claims 1 to 17.

19. A computer system comprising one or more processors configured to perform processing according to the method of any one of claims 1 to 17.

20. A method of computerized processing of the output of a quantitative polymerase chain reaction (qPCR) instrument, the qPCR instrument being configured to generate amplification data, referred to herein as an amplification curve, the amplification curve comprising data corresponding to fluorescence measurements obtained for a plurality of thermal cycles of the qPCR instrument for performing a qPCR assay on a biological sample, the method being accomplished by using one or more computer processors to perform a process comprising: preprocessing a plurality of amplification curves generated by the qPCR instrument to obtain an engineered feature and a preprocessed amplification curve for an amplification curve in the plurality of amplification curves; processing the preprocessed amplification curves and one or more of the engineered features by one or more amplification calling deep learning networks to generate amplification calling data; processing the pre-processed amplification curves by one or more curve quality calling deep learning networks to generate curve quality calling data; as well as The curve quality call data is used to determine whether to invalidate any of the amplification call data.

21. The method according to claim 20, further comprising: An interactive electronic graphical user interface (GUI) configured to facilitate user evaluation of the output of the qPCR instrument is provided, wherein the GUI includes interface elements for a user to select settings and view output related to the amplification call data and the curve quality call data.

22. The method of any one of claims 20 to 21, wherein determining whether the amplification call data should be invalidated is based on the curve quality call data failing to meet a predefined criterion.

23. A method according to any one of claims 20 to 22, wherein the amplification call data comprises class and / or class probability data corresponding to at least an amplification class and / or a non-amplification class.

24. A method according to any one of claims 20 to 23, wherein the amplified call data comprises call confidence data.

25. A method according to any one of claims 20 to 24, wherein the curve quality call data comprises class and / or class probability data corresponding to at least a clean curve class and a problem curve class.

26. A method according to any one of claims 20 to 25, wherein the curve quality call data comprises call confidence data.

27. A method according to any one of claims 20 to 27, wherein the amplification calls generated by the one or more amplification call deep learning networks are compared with the amplification results generated by a predetermined non-neural network-based amplification calling algorithm performed on the same amplification calls to determine whether to evaluate the curve quality call data for determining whether to invalidate the amplification calls generated by the one or more amplification call deep learning networks.

28. A computer program product, comprising executable code stored in a non-transitory computer readable medium, the executable code being executable on one or more computer processors to implement the process according to any one of claims 20 to 27.

29. A computer system comprising one or more processors configured to perform processing according to the method of any one of claims 20 to 27.

30. A computer program product comprising executable code stored in a non-transitory computer readable medium, the executable code executable on one or more computer processors to implement one or more amplification calling deep learning networks for use in processing an output of a quantitative polymerase chain reaction (qPCR) instrument, the qPCR instrument configured to generate amplification data, referred to herein as an amplification curve, the amplification curve comprising data corresponding to fluorescence measurements obtained for a plurality of thermal cycles of the qPCR instrument for performing a qPCR assay on a biological sample, an amplification calling deep learning network of the one or more amplification calling deep learning networks comprising: a first part configured to receive and process one or more first engineered features obtained from preprocessing the amplification curve; a second part configured to extract learned features by receiving and processing a preprocessed amplification curve corresponding to the amplification curve or optionally by processing the preprocessed amplification curve and one or more second engineered features obtained from preprocessing the amplification curve and / or the preprocessed amplification curve; and A third part is configured to process the outputs of the first part and the second part together to generate amplification calling data.

31. The computer program product of claim 30, wherein the engineered features from which the one or more first engineered features and the one or more second engineered features are derived comprise features selected from the group consisting of: a curve derivative feature, a time series feature, a PCRedux feature, and a cycle relative threshold (CRT). RT )Algorithm characteristics.

32. The computer program product of any one of claims 30 to 31, wherein the one or more second engineered features comprise one or more derivative values, or optionally at least two derivative values.

33. A computer program product according to any one of claims 30 to 32, wherein the second part comprises one or more convolutional layers.

34. A computer program product according to any one of claims 30 to 33, wherein the second portion comprises one or more leaky ReLU layers, each leaky ReLU layer following a convolutional layer in the one or more convolutional layers.

35. The computer program product of any one of claims 33 to 34, wherein the second portion further comprises a flattened layer.

36. The computer program product of any one of claims 30 to 35, wherein the first portion comprises one or more fully connected layers.

37. A computer program product according to any one of claims 30 to 36, wherein the first portion comprises one or more leaky ReLU layers, each leaky ReLU layer following a fully connected layer in the one or more fully connected layers.

38. A computer program product according to any one of claims 30 to 37, wherein the third portion comprises one or more fully connected layers.

39. A computer program product according to any one of claims 30 to 38, wherein the third portion comprises one or more leaky ReLU layers, each leaky ReLU layer following a fully connected layer in the one or more fully connected layers of the third portion.

40. A computer program product according to any one of claims 30 to 39, wherein the third part includes a ReLU layer and an evidential learning processing block, and the evidential learning processing block is configured to use the output of one or more previous layers of the third part to generate the augmented call data including class probabilities and confidence data.

41. A computer program product according to any one of claims 30 to 39, wherein the third part includes a softmax layer, which is configured to use the output of one or more previous layers of the third part to generate the augmented call data to include class probabilities.

42. The computer program product of any one of claims 30 to 41, wherein the one or more augmented call deep learning networks comprise a collection of structurally similar but differently trained deep learning networks.

43. A computer system comprising one or more processors configured to execute instructions of the computer program product according to any one of claims 30 to 42.

44. A computer program product comprising executable code stored in a non-transitory computer readable medium, the executable code executable on one or more computer processors to implement one or more curve quality calling deep learning networks for use in processing an output of a quantitative polymerase chain reaction (qPCR) instrument, the qPCR instrument being configured to generate amplification data, referred to herein as an amplification curve, the amplification curve comprising data corresponding to fluorescence measurements obtained for a plurality of thermal cycles of the qPCR instrument for performing a qPCR assay on a biological sample, the curve quality calling deep learning network of the one or more curve quality calling deep learning networks comprising: a first portion configured to extract learned features by receiving and processing a preprocessed amplification curve corresponding to the amplification curve or, optionally, by processing the preprocessed amplification curve and one or more engineered features obtained from the amplification curve; and A second part is configured to process the output of the first part to generate curve quality call data.

45. A computer program product according to any one of Claims 44, wherein the first portion comprises one or more convolutional layers.

46. ​​A computer program product according to any one of claims 44 to 45, wherein the first portion comprises one or more leaky ReLU layers.

47. The computer program product of any one of claims 44 to 46, wherein the first portion further comprises a flattened layer.

48. A computer program product according to any one of claims 44 to 47, wherein the first portion comprises one or more fully connected layers.

49. A computer program product according to any one of claims 44 to 48, wherein the second portion comprises one or more fully connected layers.

50. A computer program product according to any one of claims 44 to 49, wherein the second portion comprises one or more leaky ReLU layers.

51. A computer program product according to any one of claims 44 to 50, wherein the second part includes a ReLU layer and an evidential learning processing block, and the evidential learning processing block is configured to use the output of one or more previous layers of the first part to generate the curve quality recall data to include class probability and confidence data.

52. A computer program product according to any one of claims 44 to 51, wherein the second part includes a softmax layer configured to use the output of one or more previous layers of the second part to generate the curve quality recall data to include class probabilities.

53. The computer program product of any one of claims 44 to 52, wherein the one or more curve quality call deep learning networks comprises a collection of structurally similar but differently trained deep learning networks.

54. A computer program product according to any one of claims 44 to 53, wherein the one or more engineered features comprise one or more derivative values, or optionally at least two derivative values.

55. A computer system comprising one or more processors configured to execute instructions of the computer program product according to any one of claims 44 to 54.

56. A computer program product comprising executable code stored in a non-transitory computer readable medium, the executable code executable on one or more computer processors to implement one or more amplification calling deep learning networks for use in processing an output of a quantitative polymerase chain reaction (qPCR) instrument, the qPCR instrument configured to generate amplification data, referred to herein as an amplification curve, the amplification curve comprising data corresponding to fluorescence measurements obtained for a plurality of thermal cycles of the qPCR instrument for performing a qPCR assay on a biological sample, an amplification calling deep learning network of the one or more amplification calling deep learning networks comprising: a first part configured to receive and process one or more first engineered features obtained from preprocessing the amplification curve; a second part configured to extract learned features by receiving and processing a preprocessed amplification curve corresponding to the amplification curve or optionally by processing the preprocessed amplification curve and one or more second engineered features obtained from preprocessing the amplification curve and / or the preprocessed amplification curve; and a third part configured to process outputs of the first part and the second part together to generate amplification call data; Wherein at least one of the first portion and the third portion comprises one or more exit layers.

57. The computer program product of claim 56, wherein the first portion comprises one or more processing blocks, each of the one or more processing blocks comprising a fully connected layer, a leaky ReLU layer, and a dropout layer.

58. A computer program product according to any one of claims 56 to 57, wherein the third part comprises one or more processing blocks, each of which comprises a fully connected layer, a leaky ReLU layer, and a dropout layer.

59. The computer program product of any one of claims 56 to 58, wherein the engineered features from which the one or more first engineered features and the one or more second engineered features are derived are selected from the group consisting of: a curve derivative feature, a time series feature, a PCRedux feature, and a cycle relative threshold (CRT). RT )Algorithm characteristics.

60. The computer program product of any one of claims 56 to 59, wherein the one or more second engineered features comprise one or more derivative values, or optionally at least two derivative values.

61. A computer program product according to any one of claims 56 to 60, wherein the second portion comprises one or more convolutional layers.

62. A computer program product according to any one of claims 56 to 60, wherein the second portion comprises one or more convolutional layers and one or more leaky ReLU layers following each of the one or more convolutional layers.

63. The computer program product of any one of claims 61 to 62, wherein the second portion further comprises a flattened layer.

64. A computer program product according to any one of claims 56 to 63, wherein the third part includes a softmax layer, which is configured to use the output of one or more previous layers of the third part to generate the augmented call data to include class probabilities.

65. The computer program product of any one of claims 56 to 64, wherein the one or more augmented call deep learning networks comprise a collection of structurally similar but differently trained deep learning networks.

66. A computer system comprising one or more processors configured to execute instructions of the computer program product according to any one of claims 56 to 65.

67. A computer program product comprising executable code stored in a non-transitory computer readable medium, the executable code executable on one or more computer processors to implement one or more curve quality calling deep learning networks for use in processing an output of a quantitative polymerase chain reaction (qPCR) instrument, the qPCR instrument configured to generate amplification data, referred to herein as an amplification curve, the amplification curve comprising data corresponding to fluorescence measurements obtained for a plurality of thermal cycles of the qPCR instrument for performing a qPCR assay on a biological sample, the curve quality calling deep learning network of the one or more curve quality calling deep learning networks comprising: a first portion configured to extract learned features by receiving and processing a preprocessed amplification curve corresponding to the amplification curve or, optionally, by processing the preprocessed amplification curve and one or more engineered features obtained from the amplification curve; and A second portion is configured to process an output of the first portion to generate curve quality call data, wherein the second portion includes one or more dropout layers.

68. The computer program product of claim 67, wherein the second portion comprises one or more processing blocks, each of the one or more processing blocks comprising a fully connected layer, a leaky ReLU layer, and a dropout layer.

69. A computer program product according to any one of claims 67 to 68, wherein the first part comprises one or more convolutional layers.

70. The computer program product of any one of claims 67 to 68, wherein the first portion comprises one or more convolutional layers and one or more leaky ReLU layers following each of the one or more convolutional layers.

71. The computer program product of any one of claims 69 to 70, wherein the first portion further comprises a flattened layer.

72. A computer program product according to any one of claims 67 to 71, wherein the second part includes a softmax layer, which is configured to use the output of one or more previous layers of the second part to generate the augmented call data to include class probabilities.

73. The computer program product of any one of claims 67 to 72, wherein the one or more curve quality-recall deep learning networks comprises a collection of structurally similar but differently trained deep learning networks.

74. A computer program product according to any one of claims 67 to 73, wherein the one or more engineered features include one or more derivative values, or optionally at least two derivative values.

75. A computer system comprising one or more processors configured to execute instructions of a computer program product according to any one of claims 67 to 74.

76. A method of computerized processing of the output of a quantitative polymerase chain reaction (qPCR) instrument, the qPCR instrument being configured to generate amplification data, referred to herein as an amplification curve, the amplification curve comprising data corresponding to fluorescence measurements obtained for a plurality of thermal cycles of the qPCR instrument for performing a qPCR assay on a biological sample, the method being accomplished by using one or more computer processors to perform a process comprising: preprocessing a plurality of amplification curves generated by the qPCR instrument to obtain a preprocessed amplification curve for an amplification curve in the plurality of amplification curves; processing the preprocessed amplification curves by an ensemble of amplification calling deep learning networks to generate amplification calling data; processing the amplification call data to generate a set amplification prediction and first confidence data corresponding to the set amplification prediction; as well as An interactive electronic graphical user interface (GUI) configured to facilitate user evaluation of the output of the qPCR instrument is provided using the collective amplification predictions and the first confidence data.

77. The method of claim 76, further comprising: processing the preprocessed amplification curves by an ensemble of curve quality calling deep learning networks to generate curve quality calling data; processing the curve quality call data to generate an aggregate curve quality prediction and second confidence data corresponding to the aggregate curve quality prediction; as well as The interactive electronic graphical user interface (GUI) configured to facilitate user evaluation of the output of the qPCR instrument is provided using the collective curve quality prediction and the second confidence data and the collective amplification prediction and the first confidence data.

78. The method of any one of claims 76 to 77, wherein: The amplification calling data includes a plurality of amplification prediction scores generated by the set of amplification calling networks and corresponding to the amplification curves; and The first confidence data is obtained by calculating confidence intervals corresponding to the plurality of amplification prediction scores.

79. The method of any one of claims 77 to 78, wherein: The curve quality calling data comprises a plurality of curve quality prediction scores generated by the set of curve quality calling networks and corresponding to the amplification curves; and The second confidence data is obtained by calculating confidence intervals corresponding to the plurality of curve quality prediction scores.

80. The method of any one of claims 1 to 6, 8, 10 to 17, 20 to 23, 25, and 27, wherein the one or more augmentation-calling deep learning networks comprises a collection of augmentation-calling deep learning networks, and the one or more curve-quality-calling deep learning networks comprises a collection of curve-quality-calling deep learning networks, the method further comprising: processing the amplification call data to generate a set amplification prediction and first confidence data corresponding to the set amplification prediction; as well as The curve quality call data is processed to generate an aggregate curve quality prediction and second confidence data corresponding to the aggregate curve quality prediction.

81. The method of claim 80, wherein: The amplification calling data includes a plurality of amplification prediction scores generated by the set of amplification calling networks and corresponding to the amplification curves; and The first confidence data is obtained by calculating confidence intervals corresponding to the plurality of amplification prediction scores.

82. The method of any one of claims 80 to 81, wherein: The curve quality calling data comprises a plurality of curve quality prediction scores generated by the set of curve quality calling networks and corresponding to the amplification curves; and The second confidence data is obtained by calculating confidence intervals corresponding to the plurality of curve quality prediction scores.

83. A computer program product comprising executable code stored in a non-transitory computer readable medium, the executable code being executable on one or more computer processors to implement processing according to any one of claims 76 to 82.

84. A bioanalysis system, comprising: a quantitative polymerase chain reaction (qPCR) instrument configured to generate amplification data, referred to herein as an amplification curve, comprising data corresponding to fluorescence measurements obtained for a plurality of thermal cycles of the qPCR instrument used to perform a qPCR assay on a biological sample; and One or more computer processors configured to perform processing according to the executable code of any one of claims 28, 30 to 32, 44 to 54, 56 to 65, 67 to 74 and 83.