A method and system for evaluating the performance of SNP detection algorithms based on latent class models

Abstract

The present invention relates to a method and system for evaluating the performance of a SNP detection algorithm based on a latent class model, which belongs to the technical field of gene sequencing and solves the problem of inaccurate performance evaluation of a SNP detection algorithm when there is no gold standard. The method comprises: after obtaining the detection results of multiple SNP detection algorithms, integrating the detection results into binary type results; according to the binary type results of all SNP detection algorithms, evaluating the sensitivity and specificity of each SNP detection algorithm based on the latent class model under different assumptions; calculating the correlation residuals of paired SNP detection algorithms under different assumptions, determining the optimal assumption based on the correlation residuals, and obtaining the sensitivity and specificity ranking results of each algorithm under the optimal assumption. Accurate performance evaluation of the SNP detection algorithm is achieved.

Description

Technical Field

[0001] The present invention relates to the field of gene sequencing technology, and in particular to a method and system for evaluating the performance of a SNP detection algorithm based on a latent class model. Background Art

[0002] Single nucleotide variations, also known as single nucleotide polymorphisms (SNPs), are usually binary polymorphisms. The development of second-generation sequencing and third-generation sequencing has accelerated the development of DNA detection methods, but due to the incompleteness of genome sequences and SNP gold standard datasets, it is still challenging to evaluate the accuracy of detection methods.

[0003] Existing studies with a "gold standard" use a benchmark dataset as the gold standard and use standard definitions of sensitivity, accuracy, and positive predictive value to calculate these indicators. However, there are two problems with these studies: the first is that the benchmark dataset is not completely equivalent to the gold standard. Even in the case of sufficient sequencing, the dataset can only approximate the gold standard and cannot be used completely as a gold standard. The second problem is that these studies only focus on positive detection, that is, SNPs, and do not care about the detection results of normal bases, which leads to the lack of "negative" results and the inability to estimate specificity. Although the detection method can be evaluated by sensitivity, accuracy, and positive predictive value, accuracy and positive predictive value will be affected by class imbalance. Class imbalance refers to the situation where the number of positive and negative results in the study is inconsistent. The degree of class imbalance directly affects the estimated values ​​of accuracy and positive predictive value.

[0004] Existing studies in the absence of a gold standard use Bayesian methods to estimate sensitivity and specificity, but they do not consider the impact of possible correlations between base sites on the sensitivity and specificity of the detection algorithm, which can lead to biased sensitivity and specificity estimates, and therefore inaccurate performance evaluation of the SNP detection algorithm. Summary of the invention

[0005] In view of the above analysis, the embodiments of the present invention aim to provide a method and system for evaluating the performance of a SNP detection algorithm based on a latent class model, so as to solve the problem of inaccurate performance evaluation of a SNP detection algorithm when there is no gold standard.

[0006] On the one hand, an embodiment of the present invention provides a method for evaluating the performance of a SNP detection algorithm based on a latent class model, comprising the following steps:

[0007] After obtaining the test results of multiple SNP detection algorithms, the test results are integrated into binary type results;

[0008] According to the dichotomous results of all SNP detection algorithms, the sensitivity and specificity of each SNP detection algorithm were evaluated based on the latent class model under different assumptions;

[0009] The correlation residuals of the paired SNP detection algorithms under different assumptions were calculated, and the optimal assumption was determined based on the correlation residuals to obtain the sensitivity and specificity ranking results of each algorithm under the optimal assumption.

[0010] Based on the further improvement of the above method, the sensitivity and specificity of each SNP detection algorithm were evaluated in the following way under different assumptions:

[0011] Construct latent class models under different assumptions;

[0012] Obtaining a likelihood equation based on the latent class model;

[0013] The EM algorithm is used to iteratively solve the likelihood equation to obtain the maximum likelihood estimates of the sensitivity and specificity of each SNP detection algorithm.

[0014] Based on the further improvement of the above method, the different assumptions include the conditional independence assumption, and the latent class model under the conditional independence assumption is expressed as:

[0015]

[0016] The likelihood equation under the conditional independence assumption is:

[0017]

[0018] Among them, ρ represents the probability of the existence of SNP in the target gene, N represents the number of sites, I represents the number of SNP detection algorithms, Se i represents the sensitivity of the i-th SNP detection algorithm, Sp i represents the specificity of the ith SNP detection algorithm, t ik represents the variation detection result of the i-th SNP detection algorithm at the k-th site, T1, T2, …, T I Represents the test result variables of each SNP detection algorithm, t1, t2, …, t I represents the detection result value of each SNP detection algorithm, D represents the true mutation status, and P(·) represents the probability.

[0019] Based on the further improvement of the above method, the different assumptions include the random effect assumption; the latent class model under the random effect assumption is expressed as:

[0020]

[0021] The joint probability function corresponding to the latent class model under the random effect assumption is:

[0022]

[0023] The likelihood equation under the random effects assumption is:

[0024]

[0025]

[0026] Where ρ represents the probability of the existence of SNP in the target gene, N represents the number of loci, I represents the number of SNP detection algorithms, and t ik represents the variation detection result of the i-th SNP detection algorithm at the k-th site, T1, T2, …, T I Represents the test result variables of each SNP detection algorithm, t1, t2, …, t I represents the detection result value of each SNP detection algorithm, P(·) represents the probability, μ i1 , μ i0 , σ i1 and σ i0 is the parameter to be solved, Z k represents the heterogeneity variable, z represents the variable quantifying the heterogeneity, and Φ is the cumulative distribution of the standard normal distribution.

[0027] Based on the further improvement of the above method, the random effect assumption is used to solve the likelihood equation using the EM algorithm to obtain the maximum likelihood estimates of the sensitivity and specificity of each SNP detection algorithm, including:

[0028] The EM algorithm is used to iteratively solve the likelihood equation under the random effect assumption to obtain the parameter μ i1 , μ i0 , σ i1 and σ i0 The maximum likelihood estimate of ;

[0029] The maximum likelihood estimates of sensitivity and specificity were calculated using the following formula:

[0030]

[0031] Among them, Se i represents the sensitivity of the i-th SNP detection algorithm, Sp i Represents the specificity of the i-th detection algorithm.

[0032] Based on the further improvement of the above method, the following formula is used to calculate the correlation residual of the pairwise SNP detection algorithm:

[0033]

[0034] Among them, Residual ij represents the correlation residual between the i-th SNP detection algorithm and the j-th SNP detection algorithm, represents the correlation coefficient between the observations of the i-th SNP detection algorithm and the j-th SNP detection algorithm, Represents the correlation coefficient between the expected values ​​of the i-th SNP detection algorithm and the j-th SNP detection algorithm.

[0035] Based on the further improvement of the above method, the following formula is used to calculate the correlation coefficient between the observations of the i-th SNP detection algorithm and the j-th SNP detection algorithm:

[0036]

[0037] Among them, P O (T i =1) represents the probability that the mutation detection result of the i-th SNP detection algorithm is 1, P O (T j =1) represents the probability that the mutation detection result of the j-th SNP detection algorithm is 1, P O (T i =1,T j =1) represents the probability that the variation detection result of the i-th SNP detection algorithm is 1 and the variation detection result of the j-th SNP detection algorithm is 1, t ik represents the variation detection result of the i-th SNP detection algorithm at the k-th site, N represents the number of sites, and t jk Represents the mutation detection result of the j-th algorithm at the k-th site.

[0038] Based on the further improvement of the above method, the following formula is used to calculate the correlation coefficient between the expected values ​​of the i-th SNP detection algorithm and the j-th SNP detection algorithm:

[0039]

[0040] P E (T i =1) = (1-ρ)(1-Sp i )+ρSe i

[0041] P E (T j =1) = (1-ρ)(1-Sp j )+ρSe j

[0042] Under the conditional independence assumption, P E (T i =1,T j =1) = (1-ρ)(1-Spi )(1-Sp j )+ρSe i Se j ;

[0043] Under the random effects assumption,

[0044] Among them, P E (T i =1) represents the probability that the expected result of the mutation detection of the i-th SNP detection algorithm is 1, P E (T j =1) represents the probability that the expected result of the variation detection of the j-th SNP detection algorithm is 1, P E (T i =1,T j =1) represents the probability that the expected result of the variation detection of the i-th SNP detection algorithm is 1 and the expected result of the variation detection of the j-th SNP detection algorithm is 1, ρ represents the probability that the target gene has a SNP, Se i represents the sensitivity of the i-th SNP detection algorithm, Sp i represents the specificity of the i-th SNP detection algorithm, μ i1 , μ i0 , σ i1 and σ i0 represents the parameters solved under the random effects assumption, z represents the variable quantifying heterogeneity, and Φ is the cumulative distribution of the standard normal distribution.

[0045] Based on the further improvement of the above method, the hypothesis with the correlation residual value of the paired SNP detection algorithm closest to 0 is the optimal hypothesis.

[0046] On the other hand, an embodiment of the present invention provides a SNP detection algorithm performance evaluation system based on a latent class model, comprising the following modules:

[0047] A test result integration module is used to obtain the test results of multiple SNP detection algorithms and integrate the test results into binary type results;

[0048] The indicator calculation module is used to evaluate the sensitivity and specificity of each SNP detection algorithm based on the latent class model under different assumptions according to the binary type results of all SNP detection algorithms;

[0049] The performance evaluation module calculates the correlation residuals of the paired SNP detection algorithms under different assumptions, determines the optimal assumption based on the correlation residuals, and obtains the sensitivity and specificity ranking results of each algorithm under the optimal assumption.

[0050] Compared with the prior art, the SNP detection algorithm performance evaluation method based on the latent class model provided in this embodiment obtains the detection results of multiple SNP detection algorithms and integrates them into binary classification results. The present invention not only uses the latent class model to perform performance evaluation under the conditional independence assumption, but also evaluates the sensitivity and specificity of each SNP detection algorithm based on the latent class model under different assumptions, determines the optimal assumption through the correlation residual, and then obtains more accurate performance indicator evaluation results under a reasonable correlation assumption.

[0051] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings are only used for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the entire drawings, the same reference symbols represent the same components;

[0053] Figure 1 It is a flow chart of a method for evaluating the performance of a SNP detection algorithm based on a latent class model according to an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of the SNP detection results of an embodiment of the present invention;

[0055] Figure 3 A schematic diagram of correlation residuals according to an embodiment of the present invention;

[0056] Figure 4 4 is a block diagram of a SNP detection algorithm performance evaluation system based on a latent class model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0058] Existing methods often only evaluate the performance of SNP detection algorithms under the conditional independence assumption. However, Mendelian randomization studies believe that there is correlation between sites that are no more than 10,000 base pairs apart. Since existing methods do not consider the correlation between sites and cannot quantify the impact of the correlation between sites on sensitivity and specificity, they cannot accurately evaluate the performance of SNP detection algorithms.

[0059] A specific embodiment of the present invention discloses a method for evaluating the performance of a SNP detection algorithm based on a latent class model, such as Figure 1 As shown, the following steps are included:

[0060] S1. After obtaining the test results of multiple SNP detection algorithms, the test results are integrated into binary type results;

[0061] S2. Based on the binary results of all SNP detection algorithms, the sensitivity and specificity of each SNP detection algorithm were evaluated based on the latent class model under different assumptions.

[0062] S3. Calculate the correlation residuals of paired SNP detection algorithms under different assumptions, determine the optimal assumption based on the correlation residuals, and obtain the sensitivity and specificity ranking results of each algorithm under the optimal assumption.

[0063] It should be noted that the hypothesis that the correlation residual value of the paired SNP detection algorithm is closest to 0 is the optimal hypothesis, which is most consistent with the actual data results. Therefore, the estimated values ​​of sensitivity and specificity under the optimal hypothesis are the most accurate.

[0064] Compared with the prior art, the SNP detection algorithm performance evaluation method based on the latent class model provided in this embodiment obtains the detection results of multiple SNP detection algorithms and integrates them into binary classification results. The present invention not only uses the latent class model to perform performance evaluation under the conditional independence assumption, but also evaluates the sensitivity and specificity of each SNP detection algorithm based on the latent class model under different assumptions, determines the optimal assumption through the correlation residual, and then obtains more accurate performance indicator evaluation results under reasonable affinity assumptions.

[0065] It should be noted that the present invention is mainly used for estimating the sensitivity and specificity of the detection algorithm when there is no SNP gold standard. It is assumed that there is only one variation, namely SNP, and the research unit is the base site. Figure 2 The results of SNP detection in two situations, gold standard and no gold standard, are shown. The difference between the gold standard dataset and the normal reference genome is that the former includes not only normal bases (blue) but also known SNPs (highlighted in red) ( Figure 2 (a)). In actual analysis, sites with the same base type as the normal reference genome are usually excluded from the SNP gold standard, and the remaining sites are regarded as the SNP gold standard. Once the SNP gold standard is defined, the four indicators in the 2×2 contingency table can be defined and used to calculate various performance indicators, such as sensitivity, specificity, positive predictive value, etc. These indicators are uniformly defined as follows:

[0066] (1) Number of true positives (n TP ): the number of sites correctly marked as SNPs;

[0067] (2) False negative number (n FN ): The number of SNP sites that were incorrectly marked as normal sites;

[0068] (3) Number of false positives (n FP ): the number of normal sites that were incorrectly marked as SNPs;

[0069] (4) Number of true negatives (n TN ): The number of normal sites correctly labeled as normal sites.

[0070] Under the perfect gold standard, sensitivity and specificity can be expressed as n TP / (n TP +n FN ) and n TN / (n FP +n TN ) to calculate.

[0071] However, in the real world, the actual detection results are often Figure 2 As shown at the bottom of (a) (filled with purple). SNPs can be homozygous mutations or heterozygous mutations, where heterozygous mutations can be divided into two cases. The first is that the base type of the SNP does not match the gold standard, as shown in positions 9, 10, 16, and 17. The second is that the base type of the SNP has two types, separated by commas, as shown in positions 11, 13, and 14. For the latter, we will not assign weights to each type, nor will we divide the base type error (BE) into two SNPs to calculate sensitivity or specificity, but only regard it as one BE. Otherwise, it can only be regarded as a negative test, which will increase the number of false negatives and reduce the number of true positives, resulting in an underestimate of sensitivity.

[0072] To consider all the above mentioned situations, the present invention provides a more relaxed indicator definition, which allows a wider range of SNPs to be considered. Starting from the definition of SNP, the base type of each SNP is not considered. Figure 2 The test results of (a) can be divided into the following five categories, and the results are shown in Table 1 when there is a gold standard:

[0073] (I) Sequencing bases that completely match the base type in the SNP gold standard (called TP);

[0074] (II) Sequencing bases that do not completely match the base type in the SNP gold standard (called BE);

[0075] (III) false positive sequencing bases (sites marked as SNPs but not included in the SNP gold standard, called FPs);

[0076] (IV) false negative sequencing bases (SNPs were not detected by the detection algorithm but actually existed in the gold standard, referred to as FN);

[0077] (V) No variants were detected by sequencing and these sites were not included in the SNP gold standard (called TN).

[0078] Table 1 Detection results of the detection algorithm when there is a SNP gold standard

[0079]

[0080] In the actual data processing process of the present invention, the variation with incomplete base type matching in category (II) is not distinguished, but is regarded as TP. With this setting, the SNP detected at the site is marked as a positive result, otherwise, the SNP not detected at the site is considered to be a negative result, that is, a normal site. Therefore, by adding the positive and negative results defined in Table 1 at each site, a 2×2 contingency table shown in Table 2 can be obtained, so that the calculation formula for sensitivity and specificity can be defined as:

[0081]

[0082] The corresponding false positive rate (FPR) and false negative rate (FNR) calculation formulas are defined as FPR = n FP / (n FP +n TN ) and FNR = n FN / (n TP +n BE +n FN ). Ideally, a perfect detection algorithm would accurately identify the presence of a SNP at each site and determine its base type, resulting in zero BE.

[0083] Table 2 2×2 contingency table of binary SNP detection results

[0084]

[0085] However, once the SNP gold standard is unavailable or does not exist, that is, the gold standard is imperfect or there is no gold standard, it is impossible to calculate sensitivity and specificity according to the above two definitions, especially when the results of multiple detection algorithms are inconsistent, it is difficult to make an accurate judgment, for example, Figure 2 Positions 8, 9, 10, 11, etc. in (b).

[0086] The present invention uses a binary outcome to represent the true variation status of the site and the detection results of different SNP detection algorithms.

[0087] The base type test result is as follows Figure 2 As shown in the detection results of different SNP detection algorithms in (b), the base type result of each site is one or more of A, C, G, and T. Since there is no gold standard at this time, the detection result of each SNP algorithm is a positive result, recorded as 1, and the site not detected by the algorithm is a negative result, recorded as 0.

[0088] Assume that I detection algorithms are used to detect N sites, and D is the actual mutation state of the site. D = 1 indicates that there is a SNP at the site, and D = 0 indicates that there is no SNP (i.e., a normal site). The probability that the target gene has a SNP is expressed as ρ = P (D = 1), which represents the probability of the SNP existing. Let T i =1 and T i =0 respectively represent the positive and negative detection results of the ith algorithm. The presence of SNP at the site is called a "positive" result, while the absence of SNP at the site is a "negative" result. The sensitivity of the ith algorithm refers to the ability of the ith algorithm to correctly detect SNP at the site where SNP exists. It is defined as Se i =P(T i =1|D=1), which is the probability that the result of the ith algorithm is positive when the true variant state is SNP. The specificity of the ith algorithm refers to its ability to correctly detect sites without SNP as normal sites, which is defined as Sp i =P(T i =0|D=0), which is the probability that the result of algorithm i is negative when the true mutation state is that the SNP does not exist.

[0089] Afterwards, the detection results of all SNP detection algorithms were summarized into a binary data table, and the sensitivity and specificity of each SNP detection algorithm were estimated based on the latent class model under different assumptions.

[0090] Specifically, the sensitivity and specificity of each SNP detection algorithm were evaluated in the following way under different assumptions:

[0091] S21. Construct latent class models under different assumptions;

[0092] S22, obtaining a likelihood equation based on the latent class model;

[0093] S23. Using the EM algorithm to iteratively solve the likelihood equation, obtain the maximum likelihood estimation value of the sensitivity and specificity of each SNP detection algorithm.

[0094] When implementing, different assumptions include the conditional independence assumption (CIA) and the random effects assumption. Under the conditional independence assumption, Ti ⊥T j |D, that is, under the true mutation state of a given site, the results of the i-th detection algorithm and the j-th detection algorithm do not have the same tendency to detect errors. Under the random effect assumption, T i ⊥T j |Z k ,D, that is, after the true variation state of a given site and the heterogeneity measurement between sites, there is no tendency for the same detection error between the results of the i-th detection algorithm and the j-th detection algorithm.

[0095] The latent class model under the conditional independence assumption is expressed as:

[0096]

[0097] The likelihood equation under the conditional independence assumption is:

[0098]

[0099] Among them, ρ represents the probability of the existence of SNP in the target gene, N represents the number of sites, I represents the number of SNP detection algorithms, Se i represents the sensitivity of the i-th SNP detection algorithm, Sp i represents the specificity of the ith SNP detection algorithm, t ik represents the variation detection result of the i-th SNP detection algorithm at the k-th site, T1, T2, …, T I Represents the test result variables of each SNP detection algorithm, t1, t2, …, t I represents the detection result value of each SNP detection algorithm, D represents the true mutation status, and P(·) represents the probability.

[0100] d,t i are the true mutation state and the result of the i-th detection algorithm, d,t i ,t ik ∈{0,1}. The parameters to be identified in the likelihood equation are 2I+1 (the parameters to be identified include π, Se i ,Sp i ), the degrees of freedom of the data are 2 I -1, when I ≥ 3, all parameters can be estimated.

[0101] During implementation, the EM algorithm is used to iteratively solve the likelihood equation to obtain the maximum likelihood estimates of the sensitivity and specificity of each SNP detection algorithm.

[0102] During implementation, π (m) , and are the values ​​after the mth iteration of the EM algorithm, and the estimated values ​​after the m+1th iteration are:

[0103]

[0104] in:

[0105]

[0106] Repeat the above iterative process until the parameters ρ, Se i ,Sp i Converge to obtain the maximum likelihood estimates of each parameter.

[0107] However, the actual data may not meet the CIA assumption. Therefore, the dependence between the detection algorithm results is modeled through random effects. When implementing, the site-based symbols and parameters are introduced to establish a latent class model with random effects. The detection result variable of the i-th detection algorithm at the k-th site is represented as T ik , where T ik =1 indicates a positive result, i.e., a SNP is detected at site k, otherwise T ik = 0 indicates a negative result, i.e. no SNP was detected at site k. k represents a summary measure of the heterogeneity between sites that may affect site detection, assuming that it follows a standard normal distribution and is similar to D k If the true variant state of site k and the summary measure of heterogeneity are independent, the probability that the i-th detection algorithm detects site k as a SNP, that is, the random effect model, can be expressed as:

[0108]

[0109] where d k =0,1 represents the true variation state of site k, Φ is the cumulative distribution of the standard normal distribution, Z k ~N(0,1) is used to explain the heterogeneity variable between sites, z represents the variable that quantifies the heterogeneity, is the parameter to be solved, that is, the unknown real number that needs to be estimated.

[0110] according to In different situations, the random effect assumption is divided into the constant random effect assumption and the variable random effect assumption. The constant random effect assumption, that is, for all detection algorithms, is a constant; the random effect of variation is assumed, that is, for each detection algorithm, σ id =σ i It changes with the algorithm detection results. The random effects model then degenerates into a latent class model under the basic CIA assumption.

[0111] Under both the constant random effects assumption and the varying random effects assumption, sensitivity and specificity can be evaluated in the following ways.

[0112] After integrating z from (10), we can get the sensitivity and specificity:

[0113]

[0114] The joint probability function under the random effect assumption is expressed as:

[0115]

[0116]

[0117] The likelihood equation under the random effects assumption is:

[0118]

[0119] Among them, ρ represents the probability of the existence of SNP in the target gene, N represents the number of sites, I represents the number of SNP detection algorithms, Se i represents the sensitivity of the i-th SNP detection algorithm, Sp i represents the specificity of the i-th detection algorithm, t ik represents the variation detection result of the i-th SNP detection algorithm at the k-th site, T1, T2, …, T I Represents the test result variables of each SNP detection algorithm, t1, t2, …, t I represents the detection result value of each SNP detection algorithm, P(·) represents the probability, μ i1 , μ i0 , σ i1 and σ i0 is the parameter to be solved, Z k represents the heterogeneity variable, z represents the variable quantifying the heterogeneity, and Φ is the cumulative distribution of the standard normal distribution.

[0120] Since the integral of equation (14) is not easy to solve, the integral is calculated by using the Gauss-Hermite quadrature method, that is, the integral is replaced by the summation of a finite number of points. Assuming that the number of points is J, the above equation can be rewritten as:

[0121]

[0122] Among them, w j Represents the quality of the j-th point, that is, the weight of the j-th point.

[0123] Then the EM algorithm is used to iteratively solve ρ, and The maximum likelihood estimate is obtained. Then, the sensitivity and specificity estimates of each algorithm are obtained according to equations (11) and (12).

[0124] Through the above steps, the sensitivity and specificity of each algorithm under different assumptions when there is no gold standard can be quickly and accurately obtained, and no other assistance is required to determine whether the SNP exists. Since the Bayesian estimation method is not used, it is not affected by the prior distribution, and the user does not need to input the tolerance for false positives and false negatives. Therefore, it is not affected by the user's acceptance of the error rate, and the estimation of sensitivity and specificity is more accurate.

[0125] Since different data sets meet different types of assumptions, if the influence of correlation is not considered for data sets that do not meet the conditional independence assumption, the accuracy of the performance indicators of the detection algorithm evaluated based on the data will be biased. Therefore, the present invention estimates the sensitivity and specificity of each algorithm under the conditional independence assumption, the constant random effect assumption, and the variable random effect assumption, and then calculates the correlation residuals of the paired SNP detection algorithms under different assumptions, determines the optimal assumption based on the correlation residuals, and obtains the sensitivity and specificity ranking results of each algorithm under the optimal assumption, indicating the necessity of incorporating correlation for algorithm sensitivity and specificity estimation. Thereby providing statistical support for the recommendation of the detection method, revealing the influence of inter-site correlation on the performance ranking of the detection method. The technology of the present invention can be widely used in the performance evaluation of multiple (≥4) SNP detection methods.

[0126] Specifically, the following formula is used to calculate the correlation residual of the paired SNP detection algorithm:

[0127]

[0128] Among them, Residual ij represents the correlation residual between the i-th SNP detection algorithm and the j-th SNP detection algorithm, represents the correlation coefficient between the observations of the i-th SNP detection algorithm and the j-th SNP detection algorithm, Represents the correlation coefficient between the expected values ​​of the i-th SNP detection algorithm and the j-th SNP detection algorithm.

[0129] It should be noted that the observed value is the detection result value of the SNP detection algorithm.

[0130] Specifically, the correlation coefficient between the observed values ​​of the i-th SNP detection algorithm and the j-th SNP detection algorithm is calculated using the following formula:

[0131]

[0132] Among them, P O (T i=1) represents the probability that the mutation detection result of the i-th SNP detection algorithm is 1, P O (T j =1) represents the probability that the mutation detection result of the j-th SNP detection algorithm is 1, P O (T i =1,T j =1) represents the probability that the variation detection result of the i-th SNP detection algorithm is 1 and the variation detection result of the j-th SNP detection algorithm is 1, t ik represents the variation detection result of the i-th SNP detection algorithm at the k-th site, N represents the number of sites, and t jk Represents the mutation detection result of the j-th algorithm at the k-th site.

[0133] Specifically, the correlation coefficient between the expected values ​​of the i-th SNP detection algorithm and the j-th SNP detection algorithm is calculated using the following formula:

[0134]

[0135] P E (T i =1) = (1-ρ)(1-Sp i )+ρSe i (twenty two)

[0136] P E (T j =1) = (1-ρ)(1-Sp j )+ρSe j (twenty three)

[0137] Under the conditional independence assumption, P E (T i =1,T j =1) = (1-ρ)(1-Sp i )(1-Sp j )+ρSe i Se j (twenty four);

[0138] Under the random effects assumption,

[0139] Among them, P E (T i =1) represents the probability that the expected result of the mutation detection of the i-th SNP detection algorithm is 1, P E (T j =1) represents the probability that the expected result of the variation detection of the j-th SNP detection algorithm is 1, P E (T i =1,T j=1) represents the probability that the expected result of the variation detection of the i-th SNP detection algorithm is 1 and the expected result of the variation detection of the j-th SNP detection algorithm is 1, ρ represents the probability that the target gene has a SNP, Se i represents the sensitivity of the i-th SNP detection algorithm, Sp i represents the specificity of the i-th SNP detection algorithm, μ i1 , μ i0 , σ i1 and σ i0 represents the parameters solved under the random effects assumption, z represents the variable quantifying heterogeneity, and Φ is the cumulative distribution of the standard normal distribution.

[0140] By calculating the correlation residuals between paired SNP detection algorithms, for example, comparing 4 SNP detection algorithms, two algorithms are a pair, then 6 correlation residual values ​​of paired algorithms are obtained under each hypothesis. The closer the correlation residual value is to 0, the closer the hypothesis is to the actual data result.

[0141] During implementation, two benchmark datasets NA12878 are taken as examples to show their correlation residual graphs, such as Figure 3 . Figure 3 (a) is the correlation residual graph of the PrecisionFDA data set. Figure 3 (b) is the correlation residual graph of the 1000Genomes dataset. These two datasets are commonly used benchmark datasets in the field of gene sequencing and are used as "gold standards" in performance evaluation research. B--BCFtools, D--DeepVariant, F--FreeBayes and G--GATK represent four different detection algorithms, LCMC is the latent class model under the conditional independence assumption, LCMR represents the latent class model under the variable random effect, and LCR is the latent class model under the constant random effect assumption. From the residual graphs of the two datasets, we can see that the variable random effect assumption is more consistent with the two datasets, but the existing research based on this data uses non-perfect gold standards and conditional independence assumptions to estimate the sensitivity and specificity of different detection algorithms. Our calculations (Table 3) show that the use of non-perfect gold standards and incorrect conditional assumptions may result in biased sensitivity and false positive rate estimates, and thus in incorrect performance ranking of detection algorithms.

[0142] Table 3 Sensitivity, false positive rate and corresponding 95% confidence interval of four detection algorithms on PrecisionFDA data and 1000Genomes data

[0143]

[0144]

[0145] Existing studies have estimated the sensitivity and specificity of different detection algorithms under the CIA assumption based on the Bayesian method, and have not included the correlation between sites into the model. On the one hand, the Bayesian model depends on the prior, and on the other hand, the correlation has an impact on the parameter estimate. The present invention introduces the concept of correlation in the field of gene sequencing for the first time, and uses random effects to model the correlation in the latent class model, thereby obtaining the sensitivity and specificity under two different conditional dependency assumptions. By determining the optimal assumption, a more accurate ranking of the detection algorithm performance is obtained. Compared with the existing Bayesian method, the present invention estimates the sensitivity and specificity based on the maximum likelihood method, which can obtain more robust estimates, and is not affected by the prior distribution and the degree of class imbalance, providing a basis for the accuracy ranking of the detection algorithm when there is no gold standard. This provides a basis for accurate biometric recognition.

[0146] The embodiment of the present invention provides a SNP detection algorithm performance evaluation system based on a latent class model, such as Figure 4 As shown, it includes the following modules:

[0147] A test result integration module is used to obtain the test results of multiple SNP detection algorithms and integrate the test results into binary type results;

[0148] The indicator calculation module is used to evaluate the sensitivity and specificity of each SNP detection algorithm based on the latent class model under different assumptions according to the binary type results of all SNP detection algorithms;

[0149] The performance evaluation module calculates the correlation residuals of the paired SNP detection algorithms under different assumptions, determines the optimal assumption based on the correlation residuals, and obtains the sensitivity and specificity ranking results of each algorithm under the optimal assumption.

[0150] The above method embodiments and system embodiments are based on the same principle, and their related parts can be used for reference, and can achieve the same technical effect. The specific implementation process refers to the above embodiment, which will not be repeated here.

[0151] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0152] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for evaluating the performance of a SNP detection algorithm based on a latent class model, characterized in that: The following steps are involved: After obtaining the test results of multiple SNP detection algorithms, the test results are integrated into binary type results; According to the dichotomous results of all SNP detection algorithms, the sensitivity and specificity of each SNP detection algorithm were evaluated based on the latent class model under different assumptions; Calculate the correlation residuals of the paired SNP detection algorithms under different assumptions, determine the optimal assumption based on the correlation residuals, and obtain the sensitivity and specificity ranking results of each algorithm under the optimal assumption; The different assumptions described include the conditional independence assumption, the constant random effects assumption, and the varying random effects assumption; The correlation residuals for the pairwise SNP detection algorithm were calculated using the following formula: ； in, Indicates SNP detection algorithms and The correlation residuals between SNP detection algorithms, Indicates SNP detection algorithms and The correlation coefficient between the observations of the SNP detection algorithm, Indicates SNP detection algorithms and The correlation coefficient between the expected values ​​of the SNP detection algorithms; The sensitivity and specificity of each SNP detection algorithm were evaluated under different assumptions in the following way: Construct latent class models under different assumptions; Obtaining a likelihood equation based on the latent class model; The EM algorithm is used to iteratively solve the likelihood equation to obtain the maximum likelihood estimate of the sensitivity and specificity of each SNP detection algorithm; The hypothesis with the correlation residual value of the pairwise SNP detection algorithm closest to 0 is the optimal hypothesis.

2. The method for evaluating the performance of a SNP detection algorithm based on a latent class model according to claim 1, characterized in that: The latent class model under the conditional independence assumption is expressed as: ； The likelihood equation under the conditional independence assumption is: ； in, represents the probability of the existence of SNP in the target gene, N represents the number of sites, represents the number of SNP detection algorithms, represents the sensitivity of the ith SNP detection algorithm, represents the specificity of the i-th SNP detection algorithm, represents the variation detection result of the i-th SNP detection algorithm at the k-th site, represents the detection result variable of each SNP detection algorithm, represents the detection result value of each SNP detection algorithm, D represents the true mutation state variable, represents the probability, It represents the detection result value of the i-th detection algorithm, and d represents the actual mutation state.

3. The method for evaluating the performance of a SNP detection algorithm based on a latent class model according to claim 1, wherein: The latent class model under the random effects assumption is expressed as: ； The joint probability function corresponding to the latent class model under the random effect assumption is: ； The likelihood equation under the random effects assumption is: ； in, represents the probability of the existence of SNP in the target gene, N represents the number of sites, represents the number of SNP detection algorithms, represents the variation detection result of the i-th SNP detection algorithm at the k-th site, represents the probability, and is the parameter to be solved, represents the heterogeneity variable, z represents the variable that quantifies the heterogeneity, is the cumulative distribution of the standard normal distribution, represents the test result variable of the i-th SNP algorithm at the k-th site, The value is 0 or 1, indicating the true mutation status of the kth site. and Indicates the parameter to be solved.

4. The method for evaluating the performance of a SNP detection algorithm based on a latent class model according to claim 3, characterized in that: The random effect assumption was used to solve the likelihood equation using the EM algorithm to obtain the maximum likelihood estimates of the sensitivity and specificity of each SNP detection algorithm, including: The EM algorithm is used to iteratively solve the likelihood equation under the random effect assumption to obtain the parameters to be solved. and The maximum likelihood estimate of ; The maximum likelihood estimates of sensitivity and specificity were calculated using the following formula: ； in, represents the sensitivity of the ith SNP detection algorithm, Represents the specificity of the i-th detection algorithm.

5. The method for evaluating the performance of a SNP detection algorithm based on a latent class model according to claim 1, wherein: Use the following formula to calculate the SNP detection algorithms and Correlation coefficient between observations of SNP detection algorithms: ； ； in, represents the probability that the mutation detection result of the i-th SNP detection algorithm is 1, represents the probability that the mutation detection result of the j-th SNP detection algorithm is 1, Shidi The probability that the mutation detection result of the jth SNP detection algorithm is 1 and the mutation detection result of the jth SNP detection algorithm is 1, Indicates The variation detection result of the SNP detection algorithm at the kth site, N represents the number of sites, Represents the mutation detection result of the j-th algorithm at the k-th site.

6. The method for evaluating the performance of a SNP detection algorithm based on a latent class model according to claim 1, wherein: The correlation coefficient between the expected values ​​of the i-th SNP detection algorithm and the j-th SNP detection algorithm is calculated using the following formula: ； ； Under the conditional independence assumption, ； Under the random effects assumption, ； in, represents the probability that the expected result of the mutation detection of the i-th SNP detection algorithm is 1, represents the probability that the expected result of the mutation detection of the j-th SNP detection algorithm is 1, represents the probability that the expected result of the mutation detection of the i-th SNP detection algorithm is 1 and the expected result of the mutation detection of the j-th SNP detection algorithm is 1, Indicates the probability of the existence of SNP in the target gene, represents the sensitivity of the ith SNP detection algorithm, represents the specificity of the i-th SNP detection algorithm, and represents the parameter to be solved under the random effect assumption, z represents the variable that quantifies heterogeneity, is the cumulative distribution of the standard normal distribution.

7. A SNP detection algorithm performance evaluation system based on a latent class model, characterized in that: Includes the following modules: A test result integration module is used to obtain the test results of multiple SNP detection algorithms and integrate the test results into binary type results; The indicator calculation module is used to evaluate the sensitivity and specificity of each SNP detection algorithm based on the latent class model under different assumptions according to the binary type results of all SNP detection algorithms; A performance evaluation module calculates the correlation residuals of paired SNP detection algorithms under different assumptions, determines the optimal assumption based on the correlation residuals, and obtains the sensitivity and specificity ranking results of each algorithm under the optimal assumption; The different assumptions described include the conditional independence assumption, the constant random effects assumption, and the varying random effects assumption; The correlation residuals for the pairwise SNP detection algorithm were calculated using the following formula: ； in, represents the correlation residual between the i-th SNP detection algorithm and the j-th SNP detection algorithm, represents the correlation coefficient between the observed values ​​of the i-th SNP detection algorithm and the j-th SNP detection algorithm, represents the correlation coefficient between the expected values ​​of the i-th SNP detection algorithm and the j-th SNP detection algorithm; The sensitivity and specificity of each SNP detection algorithm were evaluated under different assumptions in the following way: Construct latent class models under different assumptions; Obtaining a likelihood equation based on the latent class model; The EM algorithm is used to iteratively solve the likelihood equation to obtain the maximum likelihood estimate of the sensitivity and specificity of each SNP detection algorithm; The hypothesis with the correlation residual value of the pairwise SNP detection algorithm closest to 0 is the optimal hypothesis.

Images