Comprehensive interpretable sketch performance evaluation method and device based on statistical inference method

Abstract

The invention discloses a comprehensive interpretable sketch performance evaluation method and device based on a statistical inference method, and the method aims at a summary data structure sketch in the field of network measurement, and comprises the steps: confirming the flow level characteristics needing to be collected based on a precision index set which is given by a user and is used for evaluating the sketch performance, and carrying out the flow level sampling of a working load during operation; a Bootstrap method based on a resampling principle is combined with a Monte Carlo method, and flow level feature distribution is constructed through samples, so that a precision index value and a confidence interval of the precision index value are estimated. According to the method, in the face of the evaluation requirement that comprehensiveness and interpretability need to be guaranteed at the same time, the multi-dimensional precision index can be analyzed, it is guaranteed that the middle process and the evaluation result are confidence, based on the more comprehensive precision index, performance evaluation of credible interpretation is conducted on sketch, and precise estimation of the precision index is achieved.

Description

Technical Field

[0001] This invention belongs to the field of computer systems, and relates to the fields of network measurement and system performance analysis technology, and in particular to a fully interpretable sketch performance evaluation method and apparatus based on statistical inference methods. Background Technology

[0002] Sketch, a data structure used in network measurement, is a crucial tool. Because it employs approximate estimation to balance measurement accuracy and memory overhead, it inevitably introduces some measurement error. For network measurement tasks, the accuracy of the measurement results impacts the decisions of upper-layer applications; therefore, users have high requirements for the measurement accuracy of Sketch, making performance evaluation a key objective.

[0003] Current methods for evaluating Sketch performance include theoretically provable boundaries and machine learning. Theoretically provable boundaries primarily provide an upper bound on the usable precision of a specific measurement through mathematical proof; machine learning, on the other hand, uses datasets for training to predict the precision of a specific measurement in actual operation. Neither of these methods satisfies the two major requirements of comprehensiveness and interpretability: they cannot be used to evaluate Sketch performance based on a wider range of precision metrics, and they cannot explain the intermediate processes and results of the evaluation to make them credible.

[0004] Intuitively, mirroring all traffic of the workload and directly comparing it with the measurements from the sketch yields a comprehensive and interpretable assessment, which is accurate and reliable. However, this consumes a significant amount of key-value space; even considering only the 104-bit quintuple key space in IPv4 (i.e., source address, destination address, source port, destination port, protocol number), theoretically it would require up to 2... 104 The storage requirements for these bits are inconsistent with the original intention of using sketching to reduce overhead. Therefore, a comprehensive and interpretable sketching performance evaluation method based on statistical inference is urgently needed. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a comprehensive and interpretable sketch performance evaluation method and apparatus based on statistical inference.

[0006] The objective of this invention is achieved through the following technical solution: The first aspect of this invention provides a fully interpretable sketch performance evaluation method based on statistical inference, comprising the following steps:

[0007] (1) The set of precision indicators given by the user is parsed and processed to obtain the flow level features on which each precision indicator depends and other precision indicators involved in its calculation.

[0008] (2) Maintain a sampler for each type of flow-level feature, and use the sampler to perform flow-level sampling on the runtime workload to obtain the sample set S corresponding to each type of flow-level feature; wherein, the sampler includes a full flow sampler and a reflow tracer. The full flow sampler is responsible for tracking and storing a specified number of flow key-value pairs in an unordered manner, and the reflow tracer is responsible for tracking and storing a specified number of flow key-value pairs whose corresponding values ​​are sorted from largest to smallest in an ordered manner.

[0009] (3) Use the sample set S obtained in step (2) to perform Bootstrap resampling, randomly selecting samples in each round. A subset S is constructed from the subsamples. * According to the subsample set S * Number of samples Determine whether to introduce a Bayesian weighting model, where It is the total number of samples in the sample set S, ∈ (0,1);

[0010] (4) The mean and confidence interval of the estimators of the precision index to be evaluated are calculated by Monte Carlo simulation; wherein, the estimators of the precision index to be evaluated are obtained based on the estimators of other precision indices on which it depends, and the estimators of the precision indices are based on the subsample set S obtained by resampling. * Calculated and obtained.

[0011] 2. The fully interpretable sketch performance evaluation method based on statistical inference as described in claim 1, characterized in that step (1) includes the following sub-steps:

[0012] (1.1) User-defined set of precision metrics {μ1,μ2,…,μ} used to evaluate sketch performance n ,…,μ N}, where μ n This represents the nth precision index, and N represents the total number of precision indexes;

[0013] (1.2) The accuracy index set from step (1.1) is parsed and processed, specifically as follows: for each accuracy index μ in the accuracy index set... n According to this precision index μ n The calculation formula is analyzed to obtain the precision index μ. n The flow-level characteristics on which it depends and other precision metrics involved in its calculation {μ n :({v i |1≤i≤I},{π(μ k )*μ k |1≤k≤N})}, where v i Let I represent the i-th flow level feature, and let π(μ) represent the total number of flow level features it depends on.k ) is an indicator function, which is 1 if and only if the precision index μ n The calculation of requires dependence on the precision index μ k is 1, otherwise it is 0.

[0014] 3. The comprehensive interpretable sketch performance evaluation method based on the statistical inference method according to claim 1, wherein the flow key-value pair includes the unique identifier k of the flow j [[ID=​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​ Then directly skip all subsequent conditional judgments;

[0020] (2.2.3) When neither Γ nor Γ1 has traced or cached k j , and p < Pr, and the storage capacity of Γ has not reached the maximum value, update the flow key-value pair (k j , v ij ) to the corresponding storage unit of Γ;

[0021] (2.2.4) When neither Γ nor Γ1 has traced or cached k j , and p ≥ Pr, and the storage capacity of Γ has reached the maximum value, cache the flow key-value pair (k j , v ij ) to the corresponding storage unit of Γ1;

[0022] (2.2.5) When neither Γ nor Γ1 has traced or cached k j , and p ≥ Pr, and the storage capacities of both Γ and Γ1 have reached the maximum value, record k j into the set For any flow recorded into the set , which is no longer traced or cached, set it to the eviction state;

[0023] (2.3) For a data packet carrying the flow key-value pair (k j , v ij ), when L does not contain the node related to k j , update the flow key-value pair (k j , v ij ) to L. When the storage capacity of L reaches the maximum value and does not contain the node related to k j , replace the key of the node with the smallest value in L with k j , and then update (k j , v ij ) to L;

[0024] (2.4) Combine the flow key-value pairs stored in the full-flow sampler obtained in step (2.2) and the flow key-value pairs stored in the heavy-flow tracker obtained in step (2.3) to obtain the sample set S = {(k j , v ij )|j = 1, 2,..., J} corresponding to the flow-level features of each type, where J represents the total number of samples in the sample set S.

[0025] 5. The comprehensive interpretable sketch performance evaluation method based on the statistical inference method according to claim 1, wherein in the step (3), determining whether to introduce the Bayesian weight model according to the sample quantity * of the subsample set S specifically includes:

[0026] Determine the subsample set S * Total number of samples Does it exceed the user-defined threshold for the number of subsample sets, SamplesThreshold? If the number of subsample sets S... * Total number of samples If the sample size exceeds the user-defined threshold SamplesThreshold, then a predefined Dirichlet distribution from the Bayesian weighting model is introduced. As a weight, where Update the subsample set to Otherwise, the Bayesian weighting model is not introduced.

[0027] 6. The fully interpretable sketch performance evaluation method based on statistical inference as described in claim 1, characterized in that step (4) specifically includes:

[0028] For the accuracy index μ n Calculate its estimate according to its calculation formula. Based on the subsample set S obtained through round t1 resampling * The accuracy index μ was calculated. n estimator set For those relying on other precision metrics {μ n The accuracy index μ to be evaluated is calculated for |n=1,2,...,u} h Perform t2 rounds of Monte Carlo simulations, meaning each round of Monte Carlo simulations uses the set of estimates for each dependent accuracy metric. The estimated value is randomly selected and substituted into the accuracy index μ to be evaluated. h The corresponding estimated value is obtained by performing calculations using the given formula. The t2 round Monte Carlo simulation ultimately yielded the accuracy index μ to be evaluated. h estimator set And calculate the accuracy index μ to be evaluated. h The mean and confidence interval of the estimator, where u is the precision index to be evaluated, are given. h The number of other accuracy metrics on which it is relied.

[0029] 7. The fully interpretable sketch performance evaluation method based on statistical inference as described in claim 6, characterized in that the accuracy index to be evaluated, μ h The mean and confidence interval of the estimators are obtained through the following steps:

[0030] (4.1) Based on the accuracy index μ to be evaluated h estimator set Calculate the accuracy index μ to be evaluated h mean of the estimators and standard deviation

[0031] (4.2) If the user sets the confidence level θ, then the corresponding significance level is 1-θ, where θ∈(0,1);

[0032] (4.3) Determine the accuracy index μ to be evaluated h The test checks whether the number of estimators t2 in the estimator set is greater than the sample set size threshold SamplesThreshold. If t2 > SamplesThreshold, then the critical value Z of the standard normal distribution Z is used. θ / 2 Determine the accuracy index to be evaluated, μ h confidence interval If t2 ≤ SamplesThreshold, then use the critical value of the t-distribution. Determine the accuracy index to be evaluated, μ h confidence interval Z θ / 2 It is the critical value of the standard normal distribution at confidence level θ. It is the critical value of the t distribution at confidence level θ and degrees of freedom n-1.

[0033] A second aspect of this invention provides a sketch performance evaluation apparatus for implementing the above-described fully interpretable sketch performance evaluation method based on statistical inference, comprising:

[0034] The precision index parsing module is used to parse and process the set of precision indices given by the user in order to obtain the flow-level features on which each precision index depends and other precision indices involved in its calculation.

[0035] The sampler module is used to maintain a sampler for each type of flow-level feature. The sampler performs flow-level sampling on the runtime workload to obtain the sample set S corresponding to each type of flow-level feature. The sampler includes a full-flow sampler and a reflow tracer. The full-flow sampler is responsible for tracking and storing a specified number of flow key-value pairs in an unordered manner. The reflow tracer is responsible for tracking and storing a specified number of flow key-value pairs whose corresponding values ​​are sorted from largest to smallest in an ordered manner.

[0036] The data processor module is used to perform Bootstrap resampling using the sample set S obtained through the sampler module, randomly selecting samples in each round. A subset S is constructed from the subsamples. * According to the subsample set S * Number of samples Determine whether to introduce a Bayesian weighting model, where Let S be the total number of samples in the sample set S, ∈ (0,1); and

[0037] The analyzer module is used to calculate the mean and confidence interval of the estimator of the precision index to be evaluated through Monte Carlo simulation; wherein the estimator of the precision index to be evaluated is obtained based on the estimators of other precision indices on which it depends, and the estimator of the precision index is based on a subsample set S obtained by resampling. * Calculated and obtained.

[0038] A third aspect of the present invention provides an electronic device including one or more processors and a memory, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-described fully interpretable sketch performance evaluation method based on statistical inference.

[0039] A fourth aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, is used to implement the above-described fully interpretable sketch performance evaluation method based on statistical inference.

[0040] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0041] (1) The present invention can perform performance evaluation of sketch based on more comprehensive accuracy indicators while ensuring low overhead.

[0042] (2) The present invention reconstructs the characteristic distribution of the running workload flow based on sample sizes of different scales through a statistical inference scheme, thereby achieving accurate estimation of the accuracy index. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the architecture of the fully interpretable sketch performance evaluation method based on statistical inference methods of the present invention;

[0044] Figure 2 This is a schematic diagram of the fully interpretable sketch performance evaluation device based on statistical inference methods of the present invention.

[0045] Figure 3 This is a schematic diagram of the structure of the electronic device of the present invention. Detailed Implementation

[0046] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not intended to limit this application.

[0047] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0048] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to determination," or "includes." Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process or method. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0049] The present invention will now be described in detail with reference to the accompanying drawings. Unless otherwise specified, the features of the following embodiments and implementations can be combined with each other.

[0050] This invention presents a comprehensive and interpretable sketch performance evaluation method based on statistical inference. It is a performance evaluation scheme for sketches, a summary data structure in the network measurement field. This method, based on a user-provided set of accuracy metrics for evaluating sketch performance, identifies the required flow-level features and performs flow-level sampling on the runtime workload. Using a Bootstrap method based on resampling principles, combined with Monte Carlo methods, it constructs a flow-level feature distribution through samples to estimate the accuracy metric values ​​and their confidence intervals. This invention addresses the evaluation requirement of simultaneously ensuring comprehensiveness and interpretability by enabling the analysis of multi-dimensional accuracy metrics and guaranteeing the reliability of intermediate processes and evaluation results.

[0051] See Figure 1 The Sketch performance evaluation method specifically includes the following steps:

[0052] (1) The set of precision indicators given by the user is parsed and processed to obtain the flow level features on which each precision indicator depends and other precision indicators involved in its calculation.

[0053] (1.1) User-defined set of precision metrics {μ1,μ2,…,μ} used to evaluate sketch performance n ,…,μ N}, where μ n Let N represent the nth precision index, and N represent the total number of precision indices.

[0054] (1.2) The accuracy index set from step (1.1) is parsed and processed, specifically as follows: for each accuracy index μ in the accuracy index set... n According to this precision index μ n The calculation formula is analyzed to obtain the precision index μ. n The flow-level characteristics on which it depends and other precision metrics involved in its calculation {μ n :({v i |1≤i≤I},{π(μ k )*μ k |1≤k≤N})}, where v i Let I represent the i-th flow level feature, and let π(μ) represent the total number of flow level features it depends on. k ) is an indicator function if and only if the precision index μ n The calculation relies on the precision index μ k It is 1 if it is true, otherwise it is 0.

[0055] It should be understood that the precision metrics used to evaluate sketch performance have corresponding calculation formulas. Based on the calculation formula of the precision metric, the flow level characteristics on which the precision metric depends and other precision metrics involved in its calculation can be determined.

[0056] (2) For each type of flow-level feature v i Maintain a sampler. Through the sampler, perform flow-level sampling on the runtime workload to obtain a sample set S corresponding to each type of flow-level feature. Among them, the sampler consists of two parts: a full-flow sampler (Light Sampler) and a heavy-flow tracer (Heavy Tracer). Light Sampler is responsible for tracking and disorderly saving a specified number of flow key-value pairs (k j ,v ij ), and Heavy Tracer is responsible for tracking and orderly saving the flow key-value pairs (k i ) with the corresponding values of the specified number of flow-level features v sorted in descending order. The k j ,v ij ), where k j represents the unique identifier of the j-th type of flow, and v ij represents the value of the flow-level feature v corresponding to the j-th type of flow i .

[0057] (2.1) Light Sampler is allocated two arrays Γ and Γ1 as the tracking part and the caching part respectively. Each unit of the arrays Γ and Γ1 stores a flow key-value pair, and their unit storage spaces are M and αM respectively, where M represents the maximum storage capacity of key-value pairs specified by the user, and α ∈ (0, 1); allocate a set for storing any number of keys as the eviction part; Heavy Tracer is allocated an empty linked list L. Each node in the linked list L stores a flow key-value pair to ensure that the values of the linked list nodes are arranged in order. The maximum node storage space of the linked list L is βM, where β ∈ (0, 1).

[0058] (2.2) When performing flow-level sampling on the runtime workload, receive data packets carrying flow key-value pairs (k j ,v ij ) one by one. Generate a probability value p pseudo-randomly according to the system time, and the user sets a probability threshold Pr, where p ∈ (0, 1) and Pr ∈ (0, 1); and perform the following conditional judgment:

[0059] (2.2.1) When Γ tracks k j , update the value v of the flow-level feature in the corresponding unit flow key-value pair (k j ,v ij ); if Γ1 caches k ij and p < pr and the Γ corresponding array unit that has not tracked k j has not been associated with other flow key-value pairs, then move the cached flow key-value pair (k j ) in Γ1 to the corresponding unit of Γ, and update the value v of the flow-level feature in the flow key-value pair (k j ,vij ) Update to the corresponding array unit of Γ, and clear the Γ1 cache unit, k j Transition from the cache state to the tracking state.

[0060] (2.2.2) When Then directly skip all subsequent conditional judgments.

[0061] (2.2.3) When neither Γ nor Γ1 is tracking or caching k j , and p < Pr, and the storage capacity of Γ has not reached the maximum value, update the flow key-value pair (k j , v ij ) to the corresponding storage unit of Γ.

[0062] (2.2.4) When neither Γ nor Γ1 is tracking or caching k j , and p ≥ Pr, and the storage capacity of Γ has reached the maximum value, cache the flow key-value pair (k j , v ij ) to the corresponding storage unit of Γ1.

[0063] (2.2.5) When neither Γ nor Γ1 is tracking or caching k j , and p ≥ Pr, and the storage capacities of both Γ and Γ1 have reached the maximum value, record k j into the set For any flow recorded in the set , which is no longer tracked or cached, set it to the eviction state.

[0064] It should be understood that the probability value p is used for both tracking and caching, that is, first meet the probability threshold, and then perform tracking / caching / eviction. After Light Sampler confirms that the flow key-value pair is tracked, the key is associated with the stored cell position through the hash algorithm and the key is no longer replaced. The above judgment conditions need to be executed one by one in the order described in steps (2.2.1) - step (2.2.5). Only when the previous conditions are not met will the last one of setting to the eviction state be executed.

[0065] (2.3) For the data packet carrying the flow key-value pair (k j , v ij ), when L does not contain the node related to k j , then update the flow key-value pair (k j , v ij ) to L. When the storage capacity of L reaches the maximum value and does not contain the node related to k j , replace the key of the node with the smallest value in L with k j , and then update (k j , v ij ) to L.

[0066] (2.4) Merge the stream key-value pairs stored in Light Sampler obtained in step (2.2) and the stream key-value pairs stored in Heavy Tracer obtained in step (2.3) to obtain the sample set S = {(k j ,v ij Let S = |j = 1, 2, ..., J, where J represents the total number of samples in the sample set S.

[0067] It should be noted that the evaluation device described in this invention and the sketch being evaluated are placed under the same runtime workload, and the above process is used to perform stream-level sampling on the runtime workload to obtain the sample set S = {(k j ,v ij For any sample key k j key value v ij Compared to real runtime workloads Consistent, S is A subset of.

[0068] (3) Use the sample set S obtained in step (2) to perform Bootstrap resampling, randomly selecting samples in each round. A subset is constructed from the subsamples. in The total number of samples in the sample set S, ∈ (0,1); according to the subsample set S * Number of samples Determine whether to introduce a Bayesian weighting model.

[0069] Furthermore, based on the subsample set S * Number of samples Determining whether to introduce a Bayesian weighting model specifically includes: judging the subsample set S * Total number of samples Does it exceed the user-defined threshold for the number of subsample sets, SamplesThreshold? If the number of subsample sets S... * Total number of samples If the sample size exceeds the user-defined threshold SamplesThreshold, then a predefined Dirichlet distribution from the Bayesian weighting model is introduced. As a weight, where Update the subsample set to Otherwise, the Bayesian weighting model is not introduced.

[0070] It should be understood that the Dirichlet distribution is a component of the Bayesian weighted model. This is because it effectively incorporates uncertainty into the model, generating different values.

[0071] (4) The mean and confidence interval of the estimators of the precision index to be evaluated are calculated by Monte Carlo simulation; wherein, the estimators of the precision index to be evaluated are obtained based on the estimators of other precision indices on which it depends, and the estimators of the precision indices are based on the subsample set S obtained by resampling. * Calculated and obtained.

[0072] Specifically, for the accuracy index μ n Calculate its estimate according to its calculation formula. Based on the subsample set S obtained through round t1 resampling * The accuracy index μ was calculated. n estimator set For those relying on other precision metrics {μ n The accuracy index μ to be evaluated is calculated for |n=1,2,...,u} h Perform t2 rounds of Monte Carlo simulations, meaning each round of Monte Carlo simulations uses the set of estimates for each dependent accuracy metric. The estimated value is randomly selected and substituted into the accuracy index μ to be evaluated. h The corresponding estimated value is obtained by performing calculations using the given formula. The t2 round Monte Carlo simulation ultimately yielded the accuracy index μ to be evaluated. h estimator set And calculate the accuracy index μ to be evaluated. h The mean and confidence interval of the estimator, where u is the precision index to be evaluated, are given. h The number of other accuracy metrics on which it is relied.

[0073] It should be understood that the present invention requires the measurement performance of the sketch (i.e., the estimation of a given accuracy index value). Since the accuracy index value is an estimate and may contain errors, a confidence interval needs to be given to indicate that the estimated value of the sketch's accuracy index is within a certain confidence interval to have a certain degree of credibility.

[0074] Furthermore, the accuracy index μ to be evaluated h The mean and confidence interval of the estimators are obtained through the following steps:

[0075] (4.1) Based on the accuracy index μ to be evaluated h estimator set Calculate the accuracy index μ to be evaluated h mean of the estimators and standard deviation

[0076] (4.2) If the user sets the confidence level θ, the corresponding significance level is 1-θ, where θ∈(0,1).

[0077] (4.3) Determine the accuracy index μ to be evaluated h The test checks whether the number of estimators t2 in the estimator set is greater than the sample set size threshold SamplesThreshold. If t2 > SamplesThreshold, then the critical value Z of the standard normal distribution Z is used. θ / 2 Determine the accuracy index to be evaluated, μ h confidence interval If t2 ≤ SamplesThreshold, then use the critical value of the t-distribution. Determine the accuracy index to be evaluated, μ h confidence interval Z θ / 2 It is the critical value of the standard normal distribution at confidence level θ. It is the critical value of the t distribution at confidence level θ and degrees of freedom n-1.

[0078] It should be understood that the critical value Z of the standard normal distribution Z is... θ / 2 It is well known that the critical value of the t-distribution is... Z can be obtained by looking up a table or using a publicly available software calculator. θ / 2 and The value of .

[0079] It is worth mentioning that this invention also provides a sketch performance evaluation device for implementing the fully interpretable sketch performance evaluation method based on statistical inference methods described in the above embodiments. This sketch performance evaluation device includes a precision index parsing module, a sampler module, a data processor module, and an analyzer module, such as... Figure 2 As shown.

[0080] In this embodiment, the precision index parsing module is used to parse the set of precision indices given by the user in order to obtain the flow level features on which each precision index depends and other precision indices involved in its calculation.

[0081] In this embodiment, the sampler module is used to maintain a sampler for each type of flow-level feature. The sampler performs flow-level sampling on the runtime workload to obtain the sample set S corresponding to each type of flow-level feature. The sampler includes a full-flow sampler and a reflow tracer. The full-flow sampler is responsible for tracking and storing a specified number of flow key-value pairs in an unordered manner. The reflow tracer is responsible for tracking and storing a specified number of flow key-value pairs whose values ​​are sorted from largest to smallest in an ordered manner.

[0082] In this embodiment, the data processor module is used to perform Bootstrap resampling using the sample set S obtained through the sampler module, randomly sampling in each round. A subset S is constructed from the subsamples. * According to the subsample set S * Number of samples Determine whether to introduce a Bayesian weighting model, where It is the total number of samples in the sample set S, ∈ (0,1).

[0083] In this embodiment, the analyzer module is used to calculate the mean and confidence interval of the estimated precision index to be evaluated through Monte Carlo simulation; wherein, the estimated precision index to be evaluated is obtained based on the estimated precision indices it depends on, and the estimated precision index is based on the subsample set S obtained by resampling. * Calculated and obtained.

[0084] In summary, in the evaluation device described in this invention, the sampler module effectively samples the flow-level characteristics of the workload and outputs them to the data processor module and the analyzer module for calculating accuracy indicators and their confidence intervals. The specific implementation process of the functions and roles of each unit in the above scheme is detailed in the corresponding steps of the above method, and will not be repeated here.

[0085] Corresponding to the aforementioned embodiment of the fully interpretable sketch performance evaluation method based on statistical inference, the present invention also provides an embodiment of an electronic device.

[0086] See Figure 3 An electronic device provided in this embodiment of the invention includes one or more processors and a memory, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the fully interpretable sketch performance evaluation method based on statistical inference method in the above embodiment.

[0087] The embodiments of the electronic device described in this invention can be applied to any device with data processing capabilities, such as a computer or other similar device. The electronic device embodiments can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logically defined electronic device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 3 The diagram shown is a hardware structure diagram of any device with data processing capabilities, including the electronic device described in this invention. Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0088] The specific implementation process of the functions and roles of each unit in the above electronic device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0089] For the electronic device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The electronic device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0090] This invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements the fully interpretable sketch performance evaluation method based on statistical inference methods described in the above embodiments.

[0091] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0092] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A comprehensive interpretable sketch performance evaluation method based on statistical inference method, characterized in that: The following steps are involved: (1) Analyze the accuracy index set given by the user to obtain the flow-level features on which each accuracy index depends and other accuracy indexes involved in its calculation; (2) A sampler is maintained for each type of flow-level feature, and flow-level sampling is performed on the runtime workload through the sampler to obtain a sample set S corresponding to each type of flow-level feature; wherein the sampler includes a full-flow sampler and a heavy-flow tracker, wherein the full-flow sampler is responsible for tracking and storing a specified number of flow key-value pairs in an unordered manner, and the heavy-flow tracker is responsible for tracking and storing a specified number of flow-level feature corresponding values ​​in a descending order of the flow key-value pairs; (3) Use the sample set S obtained in step (2) to perform Bootstrap resampling, randomly selecting subsamples to construct a subsample set S * , according to the subsample set S * The number of samples Determine whether to introduce the Bayesian weight model, where is the total number of samples in the sample set S, ∈∈(0,1); (4) Calculate the mean and confidence interval of the estimated value of the precision indicator to be evaluated through Monte Carlo simulation; the estimated value of the precision indicator to be evaluated is obtained based on the estimated values ​​of other precision indicators on which it depends, and the estimated value of the precision indicator is calculated based on the subsample set S obtained by resampling. * Calculate and obtain.

2. The method for comprehensively interpretable sketch performance evaluation based on statistical inference method according to claim 1, characterized in that: The step (1) comprises the following sub-steps: (1.1) A set of user-defined accuracy indicators {μ1,μ2,…,μ n ,…,μ N }, where μ n represents the nth precision index, and N represents the total number of precision indexes; (1.2) Analyze the accuracy index set of step (1.1), specifically: for each accuracy index μ in the accuracy index set n , according to the accuracy index μ n The calculation formula is analyzed to parse out the accuracy index μ n The stream-level features it relies on and other accuracy metrics involved in its calculation {μ n :({v i |1≤i≤I},{π(μ k )*μ k |1≤k≤N})}, where v i represents the i-th flow-level feature, I represents the total number of dependent flow-level features, π(μ k ) is the indicator function, if and only if the precision index μ n The calculation needs to rely on the precision index μ k 1 if the value is 0, otherwise it is 0.

3. The method for comprehensively interpretable sketch performance evaluation based on statistical inference method according to claim 1, characterized in that: The stream key-value pair includes a unique identifier k for the stream. j and its corresponding flow-level feature v i The value of v ij .

4. The method for comprehensively interpretable sketch performance evaluation based on statistical inference method according to claim 1, characterized in that: The step (2) comprises the following sub-steps: (2.1) The full stream sampler is allocated with two arrays Γ and Γ1 as the tracking part and the cache part respectively. Each cell of the arrays Γ and Γ1 stores a stream key-value pair, and their cell storage spaces are M and αM respectively, where M represents the maximum storage capacity of the key-value pair specified by the user, α∈(0,1); a set is allocated for storing any number of keys As part of the expulsion; The heavy flow tracker is allocated an empty linked list L. Each node in the linked list L stores a flow key-value pair to ensure that the values ​​of the linked list nodes are arranged in order. The maximum node storage space of the linked list L is βM, where β∈(0,1); (2.2) When performing stream-level sampling on runtime workloads, each stream key-value pair (k j ,v ij ) data message, a probability value p is pseudo-randomly generated according to the system time, and the user sets the probability threshold Pr, where p∈(0,1), Pr∈(0,1); and the following conditional judgment is performed: (2.2.1) When Γ traces k j , the value v of the flow level feature in the corresponding unit flow key-value pair (k j , v ij ) is updated; if Γ1 caches k ij and p < Pr and k j has not been traced yet and the Γ corresponding array unit of k j has not been associated with other flow key-value pairs, then update the cached flow key-value pair (k j , v ij ) in Γ1 to the corresponding array unit of Γ, clear the Γ1 cache unit, and k j changes from the cached state to the traced state; (2.2.2) When Then directly skip all subsequent condition judgments; (2.2.3) When neither Γ nor Γ1 has traced or cached k j , and p < Pr, and the storage capacity of Γ has not reached the maximum value, update the flow key-value pair (k j , v ij ) to the corresponding storage unit of Γ; (2.2.4) When neither Γ nor Γ11 tracks or caches k j , and p ≥ Pr, and the storage capacity of Γ has reached the maximum value, the stream key-value pair (k j ,v ij ) is cached to the corresponding storage unit of Γ1; (2.2.5) When neither Γ nor Γ1 tracks or caches k j , and p ≥ Pr, and the storage capacity of Γ and Γ1 has reached the maximum value, k j Record collection For the recorded collection Any flow of , which is no longer tracked or cached, is set to the evicted state; (2.3) For the stream key-value pairs (k j ,v ij ) data message, when L does not contain k j Related nodes, the stream key-value pairs (k j ,v ij ) is updated to L, when the storage capacity of L reaches the maximum value and does not contain k j Related nodes, replace the key of the node with the smallest value in L with k j , and then (k j ,v ij ) is updated to L; (2.4) The flow key-value pairs stored in the full-flow sampler obtained in step (2.2) and the flow key-value pairs stored in the heavy-flow tracker obtained in step (2.3) are merged to obtain the sample set S corresponding to each type of flow-level feature = {(k j ,v ij )|j=1,2,...,J}, where J represents the total number of samples in the sample set S.

5. The method for comprehensively interpretable sketch performance evaluation based on statistical inference method according to claim 1, characterized in that: In the step (3), the subsample set S * The number of samples Determine whether to introduce the Bayesian weight model, including: Determine the subsample set S * The total number of samples Is it greater than the sub-sample set number threshold SamplesThreshold set by the user? If the sub-sample set S * The total number of samples If the number of sub-samples is greater than the user-set threshold SamplesThreshold, the Dirichlet distribution defined in the Bayesian weight model is introduced. As weight, where 0 <d i <1, And update the subsample set to Otherwise, the Bayesian weight model is not introduced.

6. The method for comprehensively interpretable sketch performance evaluation based on statistical inference method according to claim 1, characterized in that: The step (4) specifically comprises: For the accuracy index μ n , calculate its estimate according to its calculation formula Based on the subsample set S obtained by t1 rounds of resampling * The precision index μ is calculated n The set of estimators For other precision indicators {μ n |n=1,2,...,u} to be evaluated h , perform t2 rounds of Monte Carlo simulation, that is, each round of Monte Carlo simulation starts from the set of estimates of each dependent accuracy indicator The estimated value is randomly selected and substituted into the accuracy index μ to be evaluated h The corresponding estimated value is calculated by using the calculation formula The final result of the t2 rounds of Monte Carlo simulation is the accuracy index μ to be evaluated. h The set of estimators And calculate the accuracy index μ to be evaluated h The mean and confidence interval of the estimate, where u is the accuracy index to be evaluated μ h The number of other accuracy metrics to rely on.

7. The method for comprehensively interpretable sketch performance evaluation based on statistical inference method according to claim 6, characterized in that: The accuracy index μ to be evaluated h The estimated mean and confidence interval of are calculated by the following steps: (4.1) According to the accuracy index μ to be evaluated h The set of estimators Calculate the accuracy index μ to be evaluated h The mean of the estimated and standard deviation (4.2) The user sets the confidence level θ, and the corresponding significance level is 1-θ, where θ∈(0,1); (4.3) Determine the accuracy index μ to be evaluated h Is the number of estimators in the estimator set t2 greater than the subsample set number threshold SamplesThreshold? If t2>SamplesThreshold, use the critical value Z of the standard normal distribution Z θ / 2 Determine the accuracy index μ to be evaluated h Confidence interval of If t2≤SamplesThreshold, use the critical value of the t distribution Determine the accuracy index μ to be evaluated h Confidence interval of Where Z θ / 2 is the critical value of the standard normal distribution at the confidence level θ, is the critical value of the t distribution with confidence level θ and n-1 degrees of freedom.

8. A sketch performance evaluation device for implementing the comprehensive interpretable sketch performance evaluation method based on statistical inference method according to claims 1-7, characterized in that: include: The precision index parsing module is used to parse the precision index set given by the user to obtain the flow-level features on which each precision index depends and other precision indexes involved in its calculation; A sampler module is used to maintain a sampler for each type of flow-level feature, and perform flow-level sampling on the runtime workload through the sampler to obtain a sample set S corresponding to each type of flow-level feature; wherein the sampler includes a full-flow sampler and a heavy-flow tracker, wherein the full-flow sampler is responsible for tracking and storing a specified number of flow key-value pairs in an unordered manner, and the heavy-flow tracker is responsible for tracking and storing a specified number of flow-level feature corresponding values ​​in an ordered order from large to small. The data processor module is used to perform Bootstrap resampling using the sample set S obtained by the sampler module. Each round of random sampling subsamples to construct a subsample set S * , according to the subsample set S * The number of samples Determine whether to introduce the Bayesian weight model, where is the total number of samples in the sample set S, ∈∈(0,1); and The analyzer module is used to calculate the mean and confidence interval of the estimated value of the precision index to be evaluated through Monte Carlo simulation; wherein the estimated value of the precision index to be evaluated is obtained based on the estimated values ​​of other precision indicators on which it depends, and the estimated value of the precision indicator is obtained based on the subsample set S obtained by resampling. * Calculate and obtain.

9. An electronic device comprising one or more processors and a memory, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the comprehensive interpretable sketch performance evaluation method based on statistical inference method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: A program is stored thereon, which, when executed by a processor, is used to implement the comprehensive interpretable sketch performance evaluation method based on a statistical inference method as described in any one of claims 1 to 7.

Images