Method and apparatus for data consistency verification

Abstract

A method and apparatus for checking data consistency are disclosed. A method for checking data consistency according to one embodiment may include the steps of: operating an existing system with replicated traffic generated by sampling at least a portion of the traffic to store a first log; operating a new system with the replicated traffic to store a second log; setting a first null hypothesis that inconsistency data exceeding a certain error rate is detected in the first log and the second log; performing a consistency check on the first log and the second log until inconsistency data is detected in the first log and the second log; and determining the value of the error rate based on a first probability distribution approximation of a p-value for rejecting the first null hypothesis, with the determined significance level and the number of data pairs for which the consistency check was performed as factors.

Description

Technology Field

[0001] The following embodiments relate to a data consistency check method and apparatus. Background Technology

[0002] Data consistency checks are inspections that verify the consistency and accuracy of data within a system. They are primarily utilized during database integration, system migration, or real-time data synchronization. For instance, consistency can be verified by repeatedly comparing each data item. Since consistency checks can prevent errors that may occur during data input, processing, and storage, enhance system reliability, and reduce business errors caused by incorrect data, there is a demand for the development of technologies to implement efficient data consistency check methods. The problem to be solved

[0004] Through the following embodiments, a technique can be provided to statistically significantly calculate the amount of consistency checks required to guarantee a target error rate at a confidence level.

[0005] Through the following embodiments, a technique can be provided to calculate an error rate that is statistically significant at a confidence level for the results of data consistency checks.

[0006] However, technical challenges are not limited to the technical challenges described above, and other technical challenges may exist. means of solving the problem

[0008] A method for checking data consistency according to one embodiment includes the steps of: running an existing system with replicated traffic generated by sampling at least a portion of the traffic to store a first log; running a new system with the replicated traffic to store a second log; setting a first null hypothesis that inconsistency data exceeding a random error rate is detected in the first log and the second log; performing a consistency check of the first log and the second log until inconsistency data is detected in the first log and the second log; and determining the value of the error rate based on a first probability distribution approximation of a p-value for rejecting the first null hypothesis, wherein the number of data pairs for which the consistency check was performed is a factor and a predetermined significance level.

[0009] The above-determined error rate may correspond to the probability of detecting discrepancy data between the first log and the second log at a confidence level corresponding to the significance level.

[0010] The step of performing the consistency check may include: increasing the number of data pairs for which the consistency check was performed by 1 when the first data extracted from the first log matches the second data included in the second log corresponding to the first data; and terminating the consistency check when the first data extracted from the first log does not match the second data included in the second log corresponding to the first data.

[0011] The above first probability distribution approximation formula includes the error rate as an unknown, and based on the significance level and the number of data pairs for which the consistency check was performed, the value of the error rate guaranteed at a confidence level corresponding to the significance level can be determined from the above first probability distribution approximation formula.

[0012] The above first probability distribution approximation formula can be obtained from a formula indicating a condition in which the p-value of the result of the consistency check according to the probability distribution is less than or equal to the significance level.

[0013] The above probability distribution may include at least one of a Poisson distribution and a normal distribution.

[0014] The p-value for the result of the consistency test according to the Poisson distribution is determined as the probability that the random variable following the Poisson distribution is 0, and the probability that the random variable following the Poisson distribution is 0 can be determined based on the number of data pairs for which the consistency test was performed and the arbitrary error rate.

[0016] A method for checking data consistency according to one embodiment includes: a step of setting a second null hypothesis that inconsistency data exceeding a target error rate is detected in different data sets; and a step of determining a target number of consistency checks based on a second probability distribution approximation of a p-value for rejecting the second null hypothesis, with a predetermined significance level and a target error rate as factors.

[0017] The above data consistency verification method may further include the steps of: generating duplicate traffic by sampling at least a portion of the traffic; operating an existing system with the duplicate traffic to store a first log; operating a new system with the duplicate traffic to store a second log; performing a consistency verification of the first log and the second log; and outputting the result of the consistency verification of the first log and the second log based on the target number.

[0018] The above-determined target number may correspond to the minimum number of data pairs for which the consistency check must be performed in order to guarantee the above-determined target error rate at a confidence level corresponding to the above-determined significance level.

[0019] The second probability distribution approximation formula includes the target number, which is the number of data pairs for which the consistency check must be performed, as an unknown, and the value of the target number can be determined from the second probability distribution approximation formula based on the significance level and the target error rate.

[0020] The step of performing the above consistency check may include the step of performing a consistency check between the first log and the second log until inconsistent data is detected in the first log and the second log.

[0021] The step of outputting the result of the consistency check of the first log and the second log may include a step of determining the result of the consistency check as successful if the number of data pairs for which the consistency check was performed is greater than or equal to the target number.

[0022] The step of outputting the result of the consistency check of the first log and the second log may include a step of determining the result of the consistency check as a failure if inconsistent data is detected in the first log and the second log before the number of data pairs for which the consistency check was performed reaches the target number.

[0023] The step of performing the above consistency check may include the step of terminating the consistency check if there is a discrepancy between the first data extracted from the first log and the second data included in the second log corresponding to the first data.

[0024] The step of performing the consistency check may include: increasing the number of data pairs for which the consistency check was performed by 1 when the first data extracted from the first log matches the second data included in the second log corresponding to the first data; and terminating the consistency check when the number of data pairs for which the consistency check was performed reaches the target number.

[0026] An electronic device according to one embodiment comprises one or more processors; and a memory for storing instructions, wherein the instructions, when executed by the one or more processors, cause the electronic device to perform the steps of: running an existing system with replicated traffic generated by sampling at least a portion of the traffic to store a first log; running a new system with the replicated traffic to store a second log; setting a first null hypothesis that discrepancy data exceeding a certain error rate is detected in the first log and the second log; performing a consistency check of the first log and the second log until discrepancy data is detected in the first log and the second log; and determining the value of the error rate based on a first probability distribution approximation of a p-value for rejecting the first null hypothesis, wherein the p-value has a predetermined significance level and the number of data pairs for which the consistency check was performed as factors.

[0027] The above-determined error rate may correspond to the probability of detecting discrepancy data between the first log and the second log at a confidence level corresponding to the significance level.

[0029] An electronic device according to one embodiment includes one or more processors; and a memory for storing instructions, wherein the instructions, when executed by the one or more processors, cause the electronic device to perform the steps of: setting a second null hypothesis that inconsistency data exceeding a target error rate is detected in different data sets; and determining a target number of consistency checks based on a second probability distribution approximation of a p-value for rejecting the second null hypothesis, with a predetermined significance level and a target error rate as factors. Brief explanation of the drawing

[0031] FIG. 1 is an operation flowchart of a data consistency check method according to one embodiment. FIGS. 2a and FIGS. 2b are drawings illustrating the first logarithm and the second logarithm, respectively. FIG. 3 is a drawing for specifically explaining a conformity check process according to one embodiment. Figure 4 is a diagram illustrating the p-value. FIG. 5 is an operation flowchart of a data consistency check method according to one embodiment. FIG. 6 is a drawing for specifically explaining a consistency check process according to one embodiment. FIG. 7 is a diagram illustrating a system for checking the consistency of traffic according to one embodiment. FIGS. 8A and FIGS. 8B are drawings illustrating a user interface screen for a conformity check according to one embodiment. Figures 9a and 9b are drawings illustrating a user interface screen for outputting the calculation result of a probability distribution approximation formula corresponding to a consistency check. FIG. 10 is an example diagram of the configuration of an electronic device according to one embodiment. Specific details for implementing the invention

[0032] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Accordingly, actual implementations are not limited to the specific embodiments disclosed, and the scope of this specification includes modifications, equivalents, or substitutions included in the technical concept described by the embodiments.

[0033] In relation to the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of the noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise.

[0034] In this document, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B, or C" may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof.

[0035] Terms such as “first,” “second,” or “first” or “second” may be used simply to distinguish a component from another component and do not limit the components in other aspects (e.g., importance or order). For example, a first component may be named a second component, and similarly, a second component may be named a first component.

[0036] Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0037] The singular expression includes the plural expression unless the context clearly indicates otherwise. In this specification, terms such as "comprising" or "having" are intended to specify the existence of the described features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0038] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this specification.

[0039] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are given the same reference numeral regardless of the drawing number, and redundant descriptions thereof will be omitted.

[0041] FIG. 1 is an operation flowchart of a data consistency check method according to one embodiment.

[0042] Data consistency checking may refer to verifying whether different data sets maintain consistency without errors. For example, data consistency checking may include verifying whether data sets accumulated for the same type of logs across different systems are identical or if only intended changes exist. Logs may include data items that store records of events, system operations, user activities, etc. Different systems may include, for example, existing systems and new systems. A new system may refer to a system in which a part of the existing system has been changed or modified. For example, a new system may include a system in which at least a part of the code of the existing system has been changed.

[0043] A data consistency verification method according to one embodiment may be performed in an electronic device comprising one or more processors. The specific hardware configuration of the electronic device performing the data consistency verification method is described in detail below.

[0044] A data consistency check method according to one embodiment may include a step (110) of storing a first log by operating an existing system with replication traffic and a step (120) of storing a second log by operating a new system with replication traffic.

[0045] Replicated traffic may be generated by sampling at least a portion of the traffic. Replicated traffic may be data that replicates at least a portion of the sampled traffic. A consistency check method may include the step of generating replicated traffic by sampling at least a portion of the traffic. For example, an electronic device may sample traffic by a predetermined sampling rate. For example, if the sampling rate is 0.1, 10% of the generated traffic may be sampled.

[0046] Electronic devices can generate replicated traffic by replicating sampled traffic. In other words, replicated traffic may correspond to a portion of traffic sampled for consistency checks of data accumulated in different systems. Data sets or logs obtained by running replicated traffic in different systems may be the subject of consistency checks.

[0047] The first log and the second log may be data sets of the same type. The data included in the first log and the data included in the second log may be expected to be identical to each other, except for intended changes. The data included in the first log and the second log may be subject to consistency checks. Consistency checks are described in detail below.

[0048] For example, FIGS. 2a and FIGS. 2b illustrate a first log and a second log, respectively. Referring to FIGS. 2a and FIGS. 2b, the data (221) in 'Column 5' of the second log may correspond to an intended change. In other words, the new system may correspond to a system modified to accumulate data (221) in 'Column 5' that is different from the data (211) in 'Column 5' accumulated in the existing system, based on input traffic. Meanwhile, the remaining data, excluding the data (211, 221) in 'Column 5', may be expected to be identical in the first log and the second log. In a consistency check, it may be determined whether data other than 'Column 5' matches in the first log and the second log. For example, in a consistency check, it can be determined whether the data in row 3 (212) of column 3 of the first log and the data in row 3 (222) of column 3 of the second log match.

[0050] A data consistency check method according to one embodiment may include a step (130) of setting a first null hypothesis that inconsistent data with an error rate greater than or equal to any error rate is detected in the first log and the second log. The error rate is the error rate of the consistency check of the first log and the second log, and may mean the ratio of data that is inconsistent with each other in the first log and the second log. For example, if n data are included in each of the first log and the second log, and m data are inconsistent with each other among the n data, the error rate may be m / n. Data that is inconsistent with each other in the consistency check may not include data that is inconsistent with each other due to an intended change.

[0051] The error rate can be determined as a positive real number. An arbitrary error rate may correspond to an unknown or variable that takes any value among positive real numbers and is not determined to a specific value. An arbitrary error rate is p error When denoted as such, the first null hypothesis is that in the first and second logarithms p error It can be established under the assumption that discrepancy data corresponding to a certain proportion is detected. The first alternative hypothesis, corresponding to the first null hypothesis, can be established under the assumption that there is no discrepancy data exceeding a certain error rate in the first and second logs; in other words, discrepancy data below a certain error rate is detected in the first and second logs. If the first null hypothesis is rejected, the conclusion can be drawn that the first alternative hypothesis can be accepted or that there is statistical evidence to accept the first alternative hypothesis.

[0053] A data consistency check method according to one embodiment may include the step (140) of performing a consistency check between the first log and the second log until inconsistent data is detected in the first log and the second log.

[0054] The data included in the first log and the second log may be subject to a consistency check. During the consistency check, it may be checked whether the first data included in the first log and the second data included in the second log match. The first data and the second data may correspond to each other.

[0055] For example, corresponding data may include data recorded at the same location in the first log and the second log. For example, referring to FIGS. 2a and 2b, the data included in the first log and the data included in the second log that correspond to each other may mean data recorded at the same location in the first log and the second log. For example, the data included in row 1 of 'Column 1' in the first log and the data included in row 1 of 'Column 1' in the second log may be corresponding data.

[0056] For example, corresponding data may include at least one of the data accumulated by the same traffic. For example, data accumulated by operating an existing system with the first traffic and data accumulated by operating a new system with the first traffic may be corresponding data.

[0057] The consistency check can be performed until inconsistent data is detected in the first log and the second log. In other words, the consistency check can be terminated when inconsistent data is detected in the first log and the second log.

[0058] According to one embodiment, the step (140) of performing a consistency check may include a step of increasing the number of data pairs for which a consistency check was performed by 1 when the first data extracted from the first log and the second data included in the second log corresponding to the first data match, and a step of terminating the consistency check when the first data extracted from the first log and the second data included in the second log corresponding to the first data do not match. A data pair may refer to a pair of data extracted from the first log and data extracted from the second log. When the first data and the second data match, the operation of extracting data corresponding to each other from the first log and the second log to perform a consistency check and increasing the number of data pairs for which a consistency check was performed by 1 may be repeated.

[0059] The number of data pairs for which a consistency check was performed may be recorded until the consistency check is completed. If a consistency check is performed on n data (where n is an arbitrary natural number) extracted from the first log and n data extracted from the second log, the number of data pairs for which a consistency check was performed may be recorded as n.

[0060] For example, if the data extracted up to the nth in the first log and the second log, respectively, match each other, and the data extracted at the (n+1)th is inconsistent, the number of data pairs for which consistency checks were performed can be recorded as n. Since inconsistent data was detected in the (n+1)th extracted data, the consistency check can be terminated.

[0061] For example, referring to FIG. 3, the consistency check is performed on the (i+1)th data Data1 extracted from the first log (301). i+1 Data2, the (i+1)th data extracted from (311) and the second log (302). i+1 It may include an operation (320) for determining whether (312) matches. The initial value of i may be set to 0.

[0062] Consistency check is Data1 i+1 (311) and Data2 i+1 If (312) matches, it may include an operation (330) of updating i to a value increased by 1. Data1 corresponding to the updated i. i+1 (311) and Data2 i+1 (312) can be extracted from the first log and the second log, respectively, and a consistency check can be performed.

[0063] Data1 i+1 (311) and Data2 i+1 (312) If there is a discrepancy, the consistency check may be terminated. The value of i at the time of termination may be determined by the number of data pairs for which the consistency check was performed.

[0065] Referring again to FIG. 1, a data consistency check method according to one embodiment may include a step (150) of determining the value of an error rate based on a first probability distribution approximation of a p-value for rejecting a first null hypothesis, with a predetermined significance level and the number of data pairs for which consistency check was performed as factors.

[0066] As described above, the number of data pairs for which a consistency check has been performed may correspond to the number of data pairs for which a consistency check has been performed until the detection of inconsistent data. For example, if the data extracted up to the nth time in each of the first log and the second log match each other, and the data extracted at the (n+1)th time does not match each other, the number of data pairs for which a consistency check has been performed may be n. As a result of performing a consistency check on n data extracted from each of the first log and the second log, all n data match, so the error rate may be 0.

[0067] The p-value may represent the probability that the observed data appears when the null hypothesis is assumed to be true. For example, referring to FIG. 4, the probability distribution of the data is plotted when the null hypothesis is assumed to be true. The p-value (410) may correspond to the probability or cumulative probability that the observed data (420) appears.

[0068] Referring again to FIG. 1, the observed data in the embodiment may include data indicating that no discrepancy data appears as a result of performing a consistency check on n data. In other words, the information that the number of error data, that is, the number of data pairs that do not match each other, is zero as a result of performing a consistency check on n data may correspond to the observed data.

[0069] The significance level serves as a criterion for rejecting the null hypothesis and indicates the acceptable limit of the probability of committing a Type I error, which is the error of choosing the alternative hypothesis when the null hypothesis is true. If the p-value is less than or equal to the significance level, the null hypothesis can be determined to be rejected. The significance level can be a value determined in advance as a positive real number. The significance level can be set to a value sufficiently small to reject the null hypothesis; for example, the significance level can be set to 0.05 or 0.01. A significance level of 0.05 may mean that the null hypothesis will be rejected within a 5% error probability. When the significance level is 0.05, the null hypothesis can be determined to be rejected because the probability of the observed data appearing is sufficiently small assuming the null hypothesis is true. In other words, a p-value of 0.05 or less (or less than the significance level) means that the probability of the observed data appearing when assuming the null hypothesis is true is 0.05 or less (or less than), and since the p-value is judged to be small enough to reject the null hypothesis, it can be decided that the null hypothesis is rejected.

[0070] The first probability distribution approximation of the p-value for rejecting the first null hypothesis may include the significance level and the number of data pairs for which consistency testing was performed as factors. Hereinafter, the first probability distribution approximation of the p-value for rejecting the first null hypothesis may be briefly referred to as the first probability distribution approximation.

[0071] The first probability distribution approximation may include an error rate as an unknown. As previously mentioned, the first null hypothesis may be set as "discrepancy data greater than an arbitrary error rate is detected in the first log and the second log." Through the first probability distribution approximation, the value of the arbitrary error rate included in the unspecified first null hypothesis can be determined. Based on the significance level and the number of data pairs for which consistency checks were performed, the value of the error rate guaranteed at the confidence level corresponding to the significance level can be determined from the first probability distribution approximation.

[0072] The confidence level corresponding to the significance level is a parameter that has a complement to 1 relationship with the significance level, and the confidence level can be determined as (1 - significance level). For example, if the significance level is 0.05, the confidence level can be 0.95 or 95%.

[0073] Since the significance level is predetermined and the number of data pairs for which consistency checks have been performed is determined in step (140), the significance level and the number of data pairs for which consistency checks have been performed can be substituted into the first probability distribution approximation formula to determine the error rate guaranteed at the confidence level corresponding to the significance level.

[0074] The determined error rate may correspond to the probability of detecting discrepancy data between the first log and the second log to reject the null hypothesis at a confidence level corresponding to the significance level. For example, if the significance level is 0.05 and the error rate is determined to be 0.02 from the first probability distribution approximation, the error rate between the first log and the second log can be guaranteed to be less than 0.02 at a 95% confidence level. In other words, it can be concluded that the error rate between the first log and the second log is less than 0.02 at a 95% confidence level.

[0075] The first probability distribution approximation can be obtained from a formula indicating a condition in which the p-value of the result of the consistency test according to the probability distribution is below the significance level. As previously mentioned, the condition in which the p-value of the result of the consistency test is below the significance level may correspond to a condition for rejecting the first null hypothesis. For example, the formula indicating a condition in which the p-value of the result of the consistency test according to the probability distribution is below the significance level can be expressed as Equation 1 below.

[0076]

[0077] In mathematical formula 1, ε is the p-value for the result of the consistency test according to the probability distribution, and can represent the probability that the random variable K is 0 in a probability distribution where the number of mismatched data pairs is defined as the random variable K. In Equation 1, α corresponds to the significance level. If Equation 1 holds, the first null hypothesis can be determined to be rejected.

[0078] For example, the probability distribution may include a Poisson distribution. The p-value for the result of a consistency check based on a Poisson distribution can be determined as the probability that the random variable following the Poisson distribution is 0. The probability that the random variable following the Poisson distribution is 0 can be determined based on the number of data pairs for which the consistency check was performed and a random error rate.

[0079] The probability that a random variable following a Poisson distribution is 0 It can be approximated based on the number of data pairs for which consistency checks were performed and an arbitrary error rate. Let n be the number of data pairs for which consistency checks were performed, and p be the arbitrary error rate. error When saying, silver It can be approximated as the probability that the random variable is 0. Based on the approximation, mathematical formula 1 can be transformed into mathematical formula 2 below.

[0080]

[0081] In Equation 2, n is the number of data pairs for which consistency checks were performed, determined as a specific value, and α is the significance level, also determined as a specific value. In Equation 2, p error can correspond to an unknown variable whose value is not specified.

[0082] The unknown p error To find the range of values, By taking the natural logarithm ln of both sides of the inequality in Equation 2, Equation 2 can be transformed into Equation 3.

[0083]

[0084] p error To find the range of values, By multiplying both sides of the inequality in Equation 3 by -1 / n, Equation 3 can be transformed into the first probability distribution approximation in Equation 4.

[0085]

[0086] p error is a value greater than (or exceeding) -lnα / n, p error The range of values ​​can be determined.

[0087] Error rates p of the first and second logarithms for a fixed n error If α is set to a value greater than or equal to -lnα / n, a confidence level of (1-α) can be secured.

[0088] For example, the answer to the question "A consistency check was performed for one hour on the first and second logs obtained by running the existing system and the new system, respectively, at 10% of the average hourly traffic of 1 million transactions. What is the minimum error rate guaranteed by the new system?" can be determined by the first probability distribution approximation derived from Equation 4. By substituting 0.05 for α in Equation 4 and 1,000,000 * 0.1 = 100,000 for n, p error ≥0.0000460517086 - This can be obtained, and from this It can be concluded that the new system guarantees an error rate of less than 0.000046 at a 95% level or an error rate of less than 46 out of 10,000 cases.

[0089] For example, probability distributions may include the normal distribution. The p-value for the result of a consistency check based on the normal distribution is np error The average, It can be obtained using the cumulative distribution function (CDF) of the standard normal distribution based on the standardized scores (z-scores) of zero errors, or zero discrepancies, observed data in a normal distribution with as the standard deviation. The standardized score of observed data x can be calculated as shown in Equation 5 below.

[0090]

[0091] np error The average, In a normal distribution with as the standard deviation, the standardized score of the observed data with an error of 0 is obtained by substituting 0 for x, It can be calculated as. The value of the CDF function for can be determined as the p-value of the result of the consistency check according to the normal distribution. For example, The value of the CDF function for is the standardized score obtained using the standard normal distribution table It may correspond to the probability of being less than or equal to. By substituting the values ​​of n and α into the inequality indicating that the p-value is less than or equal to the significance level α, p error The range of values ​​can be determined.

[0093] FIG. 5 is an operation flowchart of a data consistency check method according to one embodiment.

[0094] Referring to FIG. 5, a consistency check method according to one embodiment may include a step (510) of setting a second null hypothesis that discrepancy data exceeding a target error rate is detected in different data sets. The target error rate may be specified as a predetermined value. For example, the target error rate may be determined as 0.01, etc. The second null hypothesis may be set as the assumption that discrepancy data equal to the target error rate is detected in the first log and the second log. The second alternative hypothesis corresponding to the second null hypothesis may be set as the assumption that there is no discrepancy data exceeding the target error rate in the first log and the second log, that is, discrepancy data less than the target error rate is detected in the first log and the second log. If the second null hypothesis is rejected, a conclusion may be drawn that the second alternative hypothesis can be accepted or that there is a statistical basis for accepting the first alternative hypothesis.

[0095] A consistency test method according to one embodiment may include a step (520) of determining the target number of consistency tests based on a second probability distribution approximation of a p-value for rejecting a second null hypothesis, with a target significance level and a target error rate as factors.

[0096] The determined target number may correspond to the minimum number of data pairs for which consistency checks must be performed to guarantee the target error rate at a confidence level corresponding to the significance level. In other words, if consistency checks are performed for the target number of pairs and no inconsistent data are detected between different data sets, it can be concluded that the target error rate is guaranteed at a confidence level corresponding to the significance level. Guaranteeing the target error rate at a confidence level may mean that the error rate at a confidence level is guaranteed to be less than or equal to the target error rate.

[0097] The second probability distribution approximation of the p-value for rejecting the second null hypothesis may include the significance level and the target error rate as factors. Hereinafter, the second probability distribution approximation of the p-value for rejecting the second null hypothesis may be briefly referred to as the second probability distribution approximation.

[0098] The second probability distribution approximation may include a target number as an unknown, which is the number of data pairs for which consistency checks must be performed. As previously mentioned, the second null hypothesis may be set as "discrepancy data greater than the target error rate is detected in different data sets." The value of the target number to reject the second null hypothesis can be determined through the second probability distribution approximation. The value of the target number for the significance level and the target error rate can be determined from the second probability distribution approximation.

[0099] As mentioned above, the confidence level corresponding to the significance level is a parameter that has a complement to 1 relationship with the significance level, and the confidence level can be determined as (1 - significance level). For example, if the significance level is 0.05, the confidence level can be 0.95 or 95%.

[0100] The second probability distribution approximation can be obtained from a formula indicating a condition in which the p-value of the result of the consistency test according to the probability distribution is below the significance level. As previously mentioned, the condition in which the p-value of the result of the consistency test is below the significance level may correspond to a condition for rejecting the second null hypothesis. For example, the formula indicating a condition in which the p-value of the result of the consistency test according to the probability distribution is below the significance level can be expressed as Equation 1 described above.

[0101] For example, the probability distribution may include a Poisson distribution. The p-value for the result of a consistency check based on a Poisson distribution can be determined as the probability that the random variable following the Poisson distribution is 0. The probability that the random variable following the Poisson distribution is 0 can be determined based on the number of data pairs for which the consistency check was performed and a random error rate.

[0102] The probability that a random variable following a Poisson distribution is 0 It can be approximated based on the number of data pairs for which consistency checks were performed and an arbitrary error rate. Let n be the number of data pairs for which consistency checks were performed and p be the error rate. error When saying, silver It can be approximated as the probability that the random variable is 0. According to the approximation, mathematical formula 1 can be transformed into mathematical formula 2 as described above.

[0103] Unlike the first probability distribution approximation formula, in Equation 2, perror is determined as the value of the target error rate, and α is determined as the significance level, which is also a specific value. In Equation 2, n is the target number, which is the number of data pairs for which consistency checks must be performed, and may correspond to an unspecified unknown.

[0104] To find the range of values ​​for the unknown variable n, By taking the natural logarithm ln of both sides of the inequality in Equation 2, Equation 2 can be transformed into the aforementioned Equation 3.

[0105] To find the range of values ​​for the unknown variable n, -1 / p on both sides of the inequality in Equation 3 error By multiplying, mathematical formula 3 can be transformed into the second probability distribution approximation of mathematical formula 6 below.

[0106]

[0107] n is -lnα / p error The range of values ​​for n can be determined by values ​​greater than or equal to (or exceed).

[0108] In order to guarantee the target error rate at a confidence level corresponding to the significance level, the minimum number of data pairs for which consistency checks must be performed in the first log and the second log can be determined to be n.

[0109] For example, the answer to the question "What is the minimum number of data pairs that must be checked for consistency to guarantee, at a 95% confidence level, that there is no error in 1 out of 10 million cases?" can be determined by the second probability distribution approximation derived from Equation 6. By substituting 0.05 into α in Equation 6 and p error Substituting 1 / 10,000,000 into n≥46,051,708 - This can be obtained, and from this To guarantee a target error rate of 1 / 10,000,000 at a 95% confidence level, consistency checks must be performed on more than 46,051,708 data pairs. The conclusion can be determined.

[0110] For example, probability distributions may include the normal distribution. The p-value for the result of a consistency check based on the normal distribution is np error The average, It can be obtained using the cumulative distribution function (CDF) of the normal distribution based on the standardized score (z-score) of zero errors, or zero discrepancies, of the observed data in a normal distribution with as the standard deviation. The standardized score of the observed data x can be calculated as shown in Equation 5 above.

[0111] np error The average, In a normal distribution with as the standard deviation, the standardized score of the observed data with an error of 0 is obtained by substituting 0 for x, It can be calculated as. The value of the CDF function for can be determined as the p-value of the result of the consistency test according to the normal distribution. In the inequality indicating that the p-value is less than or equal to the significance level α, p error The range of values ​​for n can be determined by substituting the values ​​of and α.

[0112] A consistency check method according to one embodiment may include a step (530) of running an existing system with replication traffic to store a first log and a step (540) of running a new system with replication traffic to store a second log. The step (530) of running an existing system with replication traffic to store a first log and the step (540) of running a new system with replication traffic to store a second log may correspond to the steps (110) and (120) described above in FIG. 1.

[0113] A consistency check method according to one embodiment may include a step (550) of performing a consistency check between a first log and a second log.

[0114] According to one embodiment, the step (550) of performing a consistency check may include the step of performing a consistency check between the first log and the second log until inconsistent data is detected in the first log and the second log.

[0115] According to one embodiment, the step (550) of performing a consistency check may include the step of terminating the consistency check if there is a discrepancy between the first data extracted from the first log and the second data included in the second log corresponding to the first data.

[0116] According to one embodiment, the step (550) of performing a consistency check may include the step of increasing the number of data pairs for which a consistency check was performed by 1 when the first data extracted from the first log and the second data included in the second log corresponding to the first data match, and the step of terminating the consistency check when the number of data pairs for which a consistency check was performed reaches a target number.

[0117] In other words, the consistency check can be terminated when the first data extracted from the first log and the second data included in the second log corresponding to the first data are inconsistent, or when the number of data pairs for which the consistency check was performed reaches a target number.

[0118] For example, referring to FIG. 6, the consistency check is performed on the (i+1)th data Data1 extracted from the first log (601). i+1 Data2, the (i+1)th data extracted from (611) and the second log (602). i+1 It may include an operation (620) for determining whether (612) matches. The initial value of i may be set to 0.

[0119] Consistency check is Data1 i+1 (611) and Data2 i+1If (612) matches, the operation (630) of updating i to a value increased by 1 may be included. The updated i may represent the number of data pairs for which a consistency check has been performed or has passed a consistency check.

[0120] The consistency check may include an operation (640) of comparing the updated i with the target count. If the updated i is smaller than the target count, the Data1 corresponding to the updated i i+1 (611) and Data2 i+1 (6112) can be extracted from the first log and the second log, respectively, and a consistency check can be performed. If the updated i is greater than or equal to the target number, the consistency check can be terminated. If the consistency check is terminated because the updated i is determined to be greater than or equal to the target number, the result of the consistency check can be determined to be successful.

[0121] Data1 i+1 (611) and Data2 i+1 (612) If there is a discrepancy, the consistency check may be terminated. The value of i at the time of termination may be determined by the number of data pairs for which the consistency check was performed. If i is less than the target number and Data1 i+1 (611) and Data2 i+1 If (612) is determined to be inconsistent and the consistency check is terminated, the result of the consistency check may be determined to be a failure.

[0123] Referring again to FIG. 5, a consistency check method according to one embodiment may include a step (560) of outputting the result of a consistency check of a first log and a second log based on a target number.

[0124] According to one embodiment, the step (560) of outputting the result of the consistency check of the first log and the second log may include a step of determining the result of the consistency check as successful when the number of data pairs for which the consistency check was performed is greater than or equal to the target number. For example, when the number of data pairs for which the consistency check was performed reaches the target number and the consistency check is terminated, the result of the consistency check may be determined as successful. As a result of performing the consistency check for the target number, no data that is inconsistent with each other is detected in the first log and the second log, so the second null hypothesis is rejected, and the conclusion that the error rate of the first log and the second log is less than (or less than) the target error rate may be derived.

[0125] According to one embodiment, the step (560) of outputting the result of the consistency check of the first log and the second log may include a step of determining the result of the consistency check as a failure if inconsistent data is detected in the first log and the second log before the number of data pairs for which the consistency check was performed reaches the target number. For example, if the consistency check is terminated because the data extracted from each of the first log and the second log is inconsistent with each other, the consistency check may be terminated before the consistency check is performed to the target number. If the number of data pairs for which the consistency check was performed is less than the target number, the result of the consistency check may be determined as a failure. As a result of performing the consistency check, since inconsistent data is detected in the first log and the second log before the consistency check is performed to the target number, the second null hypothesis is not rejected, and the confidence level may not guarantee that the error rate of the first log and the second log is less than (or lower than) the target error rate.

[0127] FIG. 7 is a diagram illustrating a system for checking the consistency of traffic according to one embodiment.

[0128] Referring to FIG. 7, a system (700) according to one embodiment may include a traffic processing zone (710) and a consistency check zone (720). The traffic processing zone (710) may include an existing system (711) and a storage (712) that stores data obtained by operating the existing system (711) with received traffic.

[0129] The consistency check area (720) may include an existing system (721) and a new system (722). The existing system (721) of the consistency check area (720) may include at least a portion of the entire existing system (711). For example, if the entire existing system (711) consists of 30 servers, the existing system (721) of the consistency check area (720) may include 3 servers. The new system (722) of the consistency check area (720) may include at least a portion of the entire new system. For example, if the entire new system consists of 30 servers, the new system (722) of the consistency check area (720) may include 3 servers.

[0130] The consistency check area (720) may include a first storage (723) for storing a first log obtained by running an existing system (721) with replicated traffic and a second storage (724) for storing a second log obtained by running a new system (722) with replicated traffic. A portion (e.g., 1%) of the received traffic may be sampled and replicated for consistency checks. A first log may be obtained by running an existing system (721) with the replicated traffic of the sampled traffic. A second log may be obtained by running a new system (722) with the replicated traffic of the sampled traffic.

[0131] The first log stored in the first storage (723) and the second log stored in the second storage (724) may be subject to a consistency check (725). A consistency check (725) may be performed on the first log extracted from the first storage (723) and the second log extracted from the second storage (724).

[0133] FIGS. 8A and FIGS. 8B are drawings illustrating a user interface screen for a conformity check according to one embodiment.

[0134] The screens (801, 802) of the user interface illustrated in FIGS. 8a and 8b may be examples of screens displayed on a user terminal. The user terminal may include a user terminal requesting a data consistency check based on traffic processing of a new system compared to an existing system. The user terminal is an electronic device including a display, and may include, for example, a personal PC, a smartphone, a tablet, a wearable device, etc.

[0135] Referring to the screen (801) of FIG. 8a, input values ​​for the factors of a probability distribution approximation formula for executing a consistency check can be received through a user interface for consistency checks. For example, the user interface may include a target error rate input window (814). The value of the target error rate of the second probability distribution approximation formula can be entered through the target error rate input window (814). For example, the user interface may include a confidence level input window (815). The value of the confidence level for determining the significance level of the second probability distribution approximation formula can be entered through the confidence level input window (815). If the input value of the confidence level is 95%, the factor value of the significance level can be determined as 5% or 0.05, and if the input value of the confidence level is 99%, the factor value of the significance level can be determined as 1% or 0.01.

[0136] In addition to the second probability distribution approximation formula, input values ​​for factors required for consistency checks may be received through a user interface. For example, the user interface may include a data unit input window (811). A data calculation unit may be entered through the data unit input window (811). For example, if the value of the data unit is entered as 'daily', the value entered through the average data count input window (812) may correspond to the number of data collected on average over one day. For example, if the value of the data unit is entered as 'daily', the time required to perform consistency checks on the target number of data determined by the second probability distribution approximation formula may be output in days. For example, if the value of the data unit is entered as 'hourly', the value entered through the average data count input window (812) may correspond to the number of data collected on average over one hour. For example, if the data unit value is entered as 'hourly', the time required to perform a consistency check on the target number of data determined by the second probability distribution approximation formula can be output in hours.

[0137] For example, the user interface may include a sampling rate input window (813). A value of the sampling rate for consistency check may be entered through the sampling rate input window (813). The number of data obtained by multiplying the entered average number of data by the entered sampling rate may be the number of data subject to consistency check during a specific period (e.g., one day or one hour, etc.).

[0138] When the value of an argument for executing a consistency check is entered and an input selecting the button (816) is received, the screen (802) of FIG. 8b can be displayed.

[0139] Referring to the screen (802) of FIG. 8b, the target number of consistency checks corresponding to the value of the input factor based on the second probability distribution approximation formula can be calculated and output through the user interface. For example, the target number of consistency checks can be output as the time required to acquire the target number of data pairs based on the number of data acquired during a specific period (e.g., 1 day, 1 hour, etc.) and the sampling rate.

[0140] Referring to the screen (802) of FIG. 8b, the results of the consistency check can be output in real time through the user interface. Whenever a consistency check is performed on a data pair, the p-value and the average of the number of inconsistent data can be calculated and output.

[0142] Figures 9a and 9b are drawings illustrating a user interface screen for outputting the calculation result of a probability distribution approximation formula corresponding to a consistency check.

[0143] The screens (901, 902) of the user interface illustrated in FIGS. 9a and 9b may be examples of screens displayed on a user's terminal. The user's terminal may correspond to the user's terminal described in FIGS. 8a and 8b.

[0144] Referring to the screen (901) of FIG. 9a, the result of calculating the error rate based on the first probability distribution approximation of the p-value for rejecting the first null hypothesis can be output through the user interface. The result of calculating the error rate may be the result calculated by substituting 1% or 0.01 as the significance level value into the first probability distribution approximation and substituting the value of the number of data pairs for which consistency checks were performed.

[0145] Referring to the screen (902) of FIG. 9b, the result of calculating the target number of consistency tests based on the second probability distribution approximation of the p-value for rejecting the second null hypothesis can be output through the user interface. The result of calculating the target number of consistency tests may be the result calculated by substituting 0.000001% as the value of the target error rate into the second probability distribution approximation and substituting the value of the significance level.

[0147] FIG. 10 is an example diagram of the configuration of an electronic device according to one embodiment.

[0148] Referring to FIG. 10, the electronic device (1000) may include a processor (1001), a memory (1003), and a communication device (1005). An electronic device (1000) according to one embodiment may include an electronic device that performs the data consistency check method described above through FIG. 1 to FIG. 9b. For example, the electronic device (1000) may include an electronic device that performs the data consistency check method described above in FIG. 1. For example, the electronic device (1000) may include an electronic device that performs the data consistency check method described above in FIG. 5. For example, the electronic device (1000) may include a user terminal described above in FIG. 8a to FIG. 9b. The electronic device (1000) may include an electronic device that communicates with the user terminal described above in FIG. 8a to FIG. 9b and provides the user terminal with the result of performing the data consistency check method through the interface described above in FIG. 8a to FIG. 9b.

[0149] A processor (1001) according to one embodiment can perform at least one operation of the data consistency check method described above through FIGS. 1 to 9b.

[0150] For example, the processor (1001) may perform at least one of the following operations: operating an existing system with duplicate traffic generated by sampling at least a portion of the traffic to store a first log; operating a new system with duplicate traffic to store a second log; setting a first null hypothesis that inconsistency data greater than an arbitrary error rate is detected in the first log and the second log; performing a consistency check of the first log and the second log until inconsistency data is detected in the first log and the second log; and determining the value of the error rate based on a first probability distribution approximation of a p-value for rejecting the first null hypothesis, with a predetermined significance level and the number of data pairs for which the consistency check was performed as factors.

[0151] For example, the processor (1001) may perform at least one of the following operations: setting a second null hypothesis that inconsistency data exceeding a target error rate is detected in different data sets; determining a target number of consistency checks based on a second probability distribution approximation of a p-value for rejecting the second null hypothesis, with a predetermined significance level and target error rate as factors; generating duplicate traffic by sampling at least a portion of the traffic; operating an existing system with duplicate traffic to store a first log; operating a new system with duplicate traffic to store a second log; performing consistency checks between the first log and the second log; and outputting the result of the consistency check between the first log and the second log based on the target number.

[0152] A memory (1003) according to one embodiment may be a volatile memory or a non-volatile memory and may store data regarding the data consistency check method described above through FIGS. 1 to 9b. For example, the memory (1003) may store data generated during the execution of the data consistency check method or data necessary to perform the data consistency check method.

[0153] A communication device (1005) according to one embodiment may provide a function for an electronic device (1000) to communicate with another electronic device or another server through a network. In other words, the electronic device (1000) may be connected to an external device (e.g., a user's terminal, a server, or a network) and exchange data through the communication device (1005). For example, the electronic device (1000) may transmit and receive data with another device (e.g., a storage, an existing system, a new system, a gateway, etc.) included in a system for checking traffic consistency (e.g., the system (700) of FIG. 7) through the communication device (1005).

[0154] According to one embodiment, memory (1003) may store a program in which the data consistency check method described above through FIGS. 1 to 9b is implemented. A processor (1001) may execute the program stored in memory (1003) and control a server (800). The code of the program executed by the processor (801) may be stored in memory (803).

[0155] According to one embodiment, the memory (1003) may store instruction(s). For example, when the instruction(s) stored in the memory (1003) are executed by one or more processors (1001), the electronic device (1000) may perform the steps of: running an existing system with duplicate traffic generated by sampling at least a portion of the traffic to store a first log; running a new system with duplicate traffic to store a second log; setting a first null hypothesis that inconsistency data exceeding a random error rate is detected in the first log and the second log; performing a consistency check of the first log and the second log until inconsistency data is detected in the first log and the second log; and determining the value of the error rate based on a first probability distribution approximation of the p-value for rejecting the first null hypothesis, with the number of data pairs for which the consistency check was performed as a factor, and a predetermined significance level.

[0156] For example, when the instruction(s) stored in memory (1003) are executed by one or more processors (1001), the electronic device (1000) sets a second null hypothesis that inconsistency data exceeding a target error rate is detected in different data sets;

[0157] Based on a second probability distribution approximation of a p-value for rejecting a second null hypothesis with a predetermined significance level and target error rate as factors, the steps of determining a target number of consistency checks, generating duplicate traffic by sampling at least a portion of the traffic, operating an existing system with the duplicate traffic to store a first log, operating a new system with the duplicate traffic to store a second log, performing consistency checks between the first log and the second log, and outputting the results of the consistency checks between the first log and the second log based on the target number may be performed.

[0158] An electronic device (1000) according to one embodiment may further include other components not illustrated. For example, the electronic device (1000) may further include an input device and an output device as means for interfacing with a communication device (1005). Also, for example, the electronic device (1000) may further include other components such as a transceiver, various sensors, a database, etc.

[0160] The embodiments described above may be implemented as hardware components, software components, and / or combinations of hardware and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0161] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be stored on any type of machine, component, physical device, virtual equipment, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on computer-readable recording media.

[0162] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may store program instructions, data files, data structures, etc., either individually or in combination, and the program instructions recorded on the medium may be those specifically designed and configured for the embodiment or those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0163] The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0164] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can apply various technical modifications and variations based thereon. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0165] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

Claim 1 A method for checking the consistency of data, comprising: a step of storing a first log by running an existing system with replicated traffic generated by sampling at least a portion of the traffic; a step of storing a second log by running a new system with the replicated traffic; a step of setting a first null hypothesis that inconsistency data exceeding a certain error rate is detected in the first log and the second log; a step of performing a consistency check of the first log and the second log until inconsistency data is detected in the first log and the second log; and a step of determining the value of the error rate based on a first probability distribution approximation of a p-value for rejecting the first null hypothesis, wherein the first probability distribution approximation of a p-value has a predetermined significance level and the number of data pairs for which the consistency check was performed as factors. Claim 2 A method according to claim 1, wherein the determined error rate corresponds to the probability of detecting discrepancy data between the first log and the second log at a confidence level corresponding to the significance level. Claim 3 A method according to claim 1, wherein the step of performing the consistency check comprises: increasing the number of data pairs for which the consistency check was performed by 1 when the first data extracted from the first log matches the second data included in the second log corresponding to the first data; and terminating the consistency check when the first data extracted from the first log does not match the second data included in the second log corresponding to the first data. Claim 4 A method according to claim 1, wherein the first probability distribution approximation formula includes the error rate as an unknown, and the value of the error rate guaranteed at a confidence level corresponding to the significance level is determined from the first probability distribution approximation formula based on the significance level and the number of data pairs for which the consistency check was performed. Claim 5 A method according to claim 1, wherein the first probability distribution approximation formula is obtained from a formula indicating a condition in which the p-value of the result of the consistency test according to the probability distribution is less than or equal to the significance level. Claim 6 In paragraph 5, the method wherein the probability distribution comprises at least one of a Poisson distribution and a normal distribution. Claim 7 A method according to claim 1, wherein the p-value for the result of the consistency check according to the Poisson distribution is determined as the probability that the random variable following the Poisson distribution is 0, and the probability that the random variable following the Poisson distribution is 0 is determined based on the number of data pairs on which the consistency check was performed and the arbitrary error rate. Claim 8 A method for checking data consistency, comprising: a step of establishing a second null hypothesis that discrepancy data exceeding a target error rate is detected in different data sets; and a step of determining a target number of consistency checks based on a second probability distribution approximation of a p-value for rejecting the second null hypothesis, with a predetermined significance level and a target error rate as factors. Claim 9 A method according to claim 8, further comprising: a step of generating replicated traffic by sampling at least a portion of the traffic; a step of storing a first log by operating an existing system with said replicated traffic; a step of storing a second log by operating a new system with said replicated traffic; a step of performing a consistency check between said first log and said second log; and a step of outputting the result of the consistency check between said first log and said second log based on said target number. Claim 10 In claim 8, the method wherein the determined target number corresponds to the minimum number of data pairs for which the consistency check must be performed in order to guarantee the target error rate at a confidence level corresponding to the significance level. Claim 11 A method according to claim 8, wherein the second probability distribution approximation formula includes the target number, which is the number of data pairs for which the consistency check must be performed, as an unknown, and the value of the target number is determined from the second probability distribution approximation formula based on the significance level and the target error rate. Claim 12 In claim 9, the step of performing the consistency check comprises the step of performing a consistency check between the first log and the second log until inconsistent data is detected in the first log and the second log. Claim 13 In claim 9, the step of outputting the result of the consistency check of the first log and the second log includes the step of determining the result of the consistency check as successful when the number of data pairs for which the consistency check was performed is greater than or equal to the target number. Claim 14 In claim 9, the step of outputting the result of the consistency check of the first log and the second log includes the step of determining the result of the consistency check as a failure if inconsistent data is detected in the first log and the second log before the number of data pairs for which the consistency check was performed reaches the target number. Claim 15 In claim 9, the step of performing the consistency check includes the step of terminating the consistency check when the first data extracted from the first log and the second data included in the second log corresponding to the first data are inconsistent. Claim 16 In claim 9, the step of performing the consistency check comprises: increasing the number of data pairs for which the consistency check was performed by 1 when the first data extracted from the first log matches the second data included in the second log corresponding to the first data; and terminating the consistency check when the number of data pairs for which the consistency check was performed reaches the target number. Claim 17 A computer program stored on a computer-readable medium in combination with hardware to execute the method of any one of claims 1 through 16. Claim 18 An electronic device comprising: one or more processors; and a memory for storing instructions, wherein, when the instructions are executed by the one or more processors, the electronic device performs the steps of: running an existing system with replicated traffic generated by sampling at least a portion of the traffic to store a first log; running a new system with the replicated traffic to store a second log; setting a first null hypothesis that discrepancy data greater than a certain error rate is detected in the first log and the second log; performing a consistency check of the first log and the second log until discrepancy data is detected in the first log and the second log; and determining the value of the error rate based on a first probability distribution approximation of a p-value for rejecting the first null hypothesis, wherein the number of data pairs for which the consistency check was performed is a factor. Claim 19 An electronic device, wherein the determined error rate in paragraph 18 corresponds to the probability of detecting discrepancy data between the first log and the second log at a confidence level corresponding to the significance level. Claim 20 An electronic device comprising: one or more processors; and a memory for storing instructions, wherein, when the instructions are executed by the one or more processors, the electronic device performs the step of setting a second null hypothesis that discrepancy data exceeding a target error rate is detected in different data sets; and the step of determining a target number of consistency checks based on a second probability distribution approximation of a p-value for rejecting the second null hypothesis, with a predetermined significance level and a target error rate as factors.

Images