Comprehensive interpretable sketch performance evaluation method and apparatus based on statistical inference methods

Abstract

The application discloses a comprehensive and explainable sketch performance evaluation method and device based on a statistical inference method. The method is aimed at a summary data structure sketch in the field of network measurement. The method confirms required collected flow level features based on a precision index set given by a user for evaluating sketch performance, and performs flow level sampling on a runtime workload. A Bootstrap method based on a resampling principle is combined with a Monte Carlo method to estimate precision index values and confidence intervals by constructing flow level feature distribution through samples. When facing the evaluation demand of simultaneously guaranteeing comprehensiveness and explainability, the application realizes multi-dimensional precision index analysis, guarantees the credibility of the intermediate process and the evaluation result, performs credible and explainable performance evaluation on the sketch based on more comprehensive precision indexes, and realizes accurate estimation of the precision indexes.

Description

Technical Field

[0001] This invention belongs to the field of computer systems, and relates to the fields of network measurement and system performance analysis technology, and in particular to a fully interpretable sketch performance evaluation method and apparatus based on statistical inference methods. Background Technology

[0002] Sketch, a data structure used in network measurement, is a crucial tool. Because it employs approximate estimation to balance measurement accuracy and memory overhead, it inevitably introduces some measurement error. For network measurement tasks, the accuracy of the measurement results impacts the decisions of upper-layer applications; therefore, users have high requirements for the measurement accuracy of Sketch, making performance evaluation a key objective.

[0003] Current methods for evaluating Sketch performance include theoretically provable boundaries and machine learning. Theoretically provable boundaries primarily provide an upper bound on the usable precision of a specific measurement through mathematical proof; machine learning, on the other hand, uses datasets for training to predict the precision of a specific measurement in actual operation. Neither of these methods satisfies the two major requirements of comprehensiveness and interpretability: they cannot be used to evaluate Sketch performance based on a wider range of precision metrics, and they cannot explain the intermediate processes and results of the evaluation to make them credible.

[0004] Intuitively, mirroring all traffic of the workload and directly comparing it with the measurements from the sketch yields a comprehensive and interpretable assessment, which is accurate and reliable. However, this consumes a significant amount of key-value space; even considering only the 104-bit quintuple key space in IPv4 (i.e., source address, destination address, source port, destination port, protocol number), theoretically it would require up to 2... 104 The storage requirements for these bits are inconsistent with the original intention of using sketching to reduce overhead. Therefore, a comprehensive and interpretable sketching performance evaluation method based on statistical inference is urgently needed. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a comprehensive and interpretable sketch performance evaluation method and apparatus based on statistical inference.

[0006] The objective of this invention is achieved through the following technical solution: The first aspect of this invention provides a fully interpretable sketch performance evaluation method based on statistical inference, comprising the following steps:

[0007] (1) The set of precision indicators given by the user is parsed and processed to obtain the flow level features on which each precision indicator depends and other precision indicators involved in its calculation.

[0008] (2) Maintain a sampler for each type of flow-level feature, and use the sampler to perform flow-level sampling on the runtime workload to obtain the sample set S corresponding to each type of flow-level feature; wherein, the sampler includes a full flow sampler and a reflow tracer. The full flow sampler is responsible for tracking and storing a specified number of flow key-value pairs in an unordered manner, and the reflow tracer is responsible for tracking and storing a specified number of flow key-value pairs whose corresponding values ​​are sorted from largest to smallest in an ordered manner.

[0009] (3) Use the sample set S obtained in step (2) to perform Bootstrap resampling, randomly selecting samples in each round. A subset S is constructed from the subsamples. * According to the subsample set S * Number of samples Determine whether to introduce a Bayesian weighting model, where It is the total number of samples in the sample set S, ∈ (0,1);

[0010] (4) The mean and confidence interval of the estimators of the precision index to be evaluated are calculated by Monte Carlo simulation; wherein, the estimator of the precision index to be evaluated is obtained based on the estimators of other precision indices on which it depends, and the estimator of the precision index is based on the subsample set S obtained by resampling. * Calculated and obtained.

[0011] Further, step (1) includes the following sub-steps:

[0012] (1.1) User-defined set of precision metrics {μ1,μ2,…,μ} used to evaluate sketch performance n ,…,μ N}, where μ n This represents the nth precision index, and N represents the total number of precision indices.

[0013] (1.2) The precision index set from step (1.1) is parsed and processed, specifically as follows: for each precision index μ in the precision index set... n According to this precision index μ n The calculation formula is analyzed to obtain the accuracy index μ. n The flow-level characteristics on which it depends and other precision metrics involved in its calculation {μ n :({v i |1≤i≤I},{π(μ k )*μ k |1≤k≤N})}, where v i Let I represent the i-th flow level feature, and let π(μ) represent the total number of flow level features it depends on. k ) is an indicator function if and only if the precision index μ nThe calculation of k is 1 when

[0014] Further, the flow key-value pair includes the unique identifier k of the flow j and its corresponding flow-level feature v i the value v ij .

[0015] Further, the step (2) includes the following sub-steps:

[0016] (2.1) The full-flow sampler is assigned two arrays Γ and Γ1 as the tracking part and the caching part respectively. Each cell of the arrays Γ and Γ1 stores a flow key-value pair, and their cell storage spaces are M and αM respectively, where M represents the maximum storage capacity of the key-value pairs specified by the user, and α ∈ (0, 1); An empty set for storing any number of keys is assigned as the eviction part; The heavy-flow tracker is assigned an empty linked list L. Each node in the linked list L stores a flow key-value pair to ensure that the values of the linked list nodes are arranged in order. The maximum node storage space of the linked list L is βM, where β ∈ (0, 1);

[0017] (2.2) When performing flow-level sampling on the runtime workload, successively receive data packets carrying flow key-value pairs (k j , v ij ). According to the system time, pseudo-randomly generate a probability value p, and the user sets a probability threshold Pr, where p ∈ (0, 1) and Pr ∈ (0, 1); And perform the following conditional judgments:

[0018] (2.2.1) When Γ tracks k j , update the value v of the flow-level feature in the corresponding cell flow key-value pair (k j , v ij ); If Γ1 caches k ij and p < Pr and the Γ corresponding array cell that has not yet tracked k j has not been associated with other flow key-value pairs, then update the cached flow key-value pair (k j ) in Γ1 to the corresponding array cell of Γ, clear the Γ1 cache cell, and k j , v ij changes from the cached state to the tracked state; j

[0019] (2.2.2) When then directly skip all subsequent conditional judgments;

[0020] (2.2.3) When both Γ and Γ1 have not tracked or cached k j , and p < Pr, and the storage capacity of Γ has not reached the maximum value, the flow key-value pair (k j ​,v ij Update to the corresponding storage unit of Γ;

[0021] (2.2.4) When neither Γ nor Γ1 is tracking or cached k j When p ≥ Pr, and the storage capacity of Γ has reached its maximum value, the stream key-value pair (k) will be... j ,v ij ) cached in the corresponding storage unit of Γ1;

[0022] (2.2.5) When neither Γ nor Γ1 is tracking or cached k j When p ≥ Pr, and the storage capacity of Γ and Γ1 has reached its maximum value, then k j Record in set For the set Any stream that is no longer tracked or cached is set to an eviction state.

[0023] (2.3) For those carrying stream key-value pairs (k j ,v ij The data message, when L does not contain k j The relevant nodes will then stream key-value pairs (k j ,v ij Update to L when the storage size of L reaches its maximum value and does not include k. j For the relevant nodes, replace the key of the node with the smallest value in L with k. j Then (k) j ,v ij Updated to L;

[0024] (2.4) Merge the flow key pairs stored in the full-flow sampler obtained in step (2.2) and the flow key pairs stored in the reflow tracer obtained in step (2.3) to obtain the sample set S = {(k j ,v ij Let S = |j = 1, 2, ..., J, where J represents the total number of samples in the sample set S.

[0025] Furthermore, in step (3), the step of basing the data on the subsample set S... * Number of samples Determining whether to introduce a Bayesian weighting model specifically includes:

[0026] Determine the subsample set S * Total number of samples Does it exceed the user-defined threshold for the number of subsample sets, SamplesThreshold? If the number of subsample sets S... * Total number of samples If the sample size exceeds the user-defined threshold SamplesThreshold, then a predefined Dirichlet distribution from the Bayesian weighting model is introduced. As a weight, where Update the subsample set to Otherwise, the Bayesian weighting model is not introduced.

[0027] Furthermore, step (4) specifically includes:

[0028] For the accuracy index μ n Calculate its estimate according to its calculation formula. Based on the subsample set S obtained through round t1 resampling * The accuracy index μ was calculated. n estimator set For those relying on other precision metrics {μ n The accuracy index μ to be evaluated is calculated for |n=1,2,...,u} h Perform t2 rounds of Monte Carlo simulations, meaning each round of Monte Carlo simulations uses the set of estimates for each dependent accuracy metric. The estimated value is randomly selected and substituted into the accuracy index μ to be evaluated. h The corresponding estimated value is obtained by performing calculations using the given formula. The t2 round Monte Carlo simulation ultimately yielded the accuracy index μ to be evaluated. h estimator set And calculate the accuracy index μ to be evaluated. h The mean and confidence interval of the estimator, where u is the precision index to be evaluated, are given. h The number of other accuracy metrics on which it is relied.

[0029] Furthermore, the accuracy index μ to be evaluated h The mean and confidence interval of the estimators are obtained through the following steps:

[0030] (4.1) Based on the accuracy index μ to be evaluated h estimator set Calculate the accuracy index μ to be evaluated h mean of the estimators and standard deviation

[0031] (4.2) If the user sets the confidence level θ, then the corresponding significance level is 1-θ, where θ∈(0,1);

[0032] (4.3) Determine the accuracy index μ to be evaluated hThe test checks whether the number of estimators t2 in the estimator set is greater than the sample set size threshold SamplesThreshold. If t2 > SamplesThreshold, then the critical value Z of the standard normal distribution Z is used. θ / 2 Determine the accuracy index to be evaluated, μ h confidence interval If t2 ≤ SamplesThreshold, then use the critical value of the t-distribution. Determine the accuracy index to be evaluated, μ h confidence interval Z θ / 2 It is the critical value of the standard normal distribution at confidence level θ. It is the critical value of the t distribution at confidence level θ and degrees of freedom n-1.

[0033] A second aspect of this invention provides a sketch performance evaluation apparatus for implementing the above-described fully interpretable sketch performance evaluation method based on statistical inference, comprising:

[0034] The precision index parsing module is used to parse and process the set of precision indices given by the user in order to obtain the flow-level features on which each precision index depends and other precision indices involved in its calculation.

[0035] The sampler module is used to maintain a sampler for each type of flow-level feature. The sampler performs flow-level sampling on the runtime workload to obtain the sample set S corresponding to each type of flow-level feature. The sampler includes a full-flow sampler and a reflow tracer. The full-flow sampler is responsible for tracking and storing a specified number of flow key-value pairs in an unordered manner. The reflow tracer is responsible for tracking and storing a specified number of flow key-value pairs whose corresponding values ​​are sorted from largest to smallest in an ordered manner.

[0036] The data processor module is used to perform Bootstrap resampling using the sample set S obtained through the sampler module, randomly selecting samples in each round. A subset S is constructed from the subsamples. * According to the subsample set S * Number of samples Determine whether to introduce a Bayesian weighting model, where Let S be the total number of samples in the sample set S, ∈ (0,1); and

[0037] The analyzer module is used to calculate the mean and confidence interval of the estimator of the precision index to be evaluated through Monte Carlo simulation; wherein the estimator of the precision index to be evaluated is obtained based on the estimators of other precision indices on which it depends, and the estimator of the precision index is based on a subsample set S obtained by resampling.* Calculated and obtained.

[0038] A third aspect of the present invention provides an electronic device including one or more processors and a memory, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-described fully interpretable sketch performance evaluation method based on statistical inference.

[0039] A fourth aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, is used to implement the above-described fully interpretable sketch performance evaluation method based on statistical inference.

[0040] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0041] (1) The present invention can perform performance evaluation of sketch based on more comprehensive accuracy indicators while ensuring low overhead.

[0042] (2) The present invention reconstructs the characteristic distribution of the running workload flow based on sample sizes of different scales through a statistical inference scheme, thereby achieving accurate estimation of the accuracy index. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the architecture of the fully interpretable sketch performance evaluation method based on statistical inference methods of the present invention;

[0044] Figure 2 This is a schematic diagram of the fully interpretable sketch performance evaluation device based on statistical inference methods of the present invention.

[0045] Figure 3 This is a schematic diagram of the structure of the electronic device of the present invention. Detailed Implementation

[0046] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application. It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application.

[0047] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0048] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "in response to determination," or "includes." Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process or method. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0049] The present invention will now be described in detail with reference to the accompanying drawings. Unless otherwise specified, the features of the following embodiments and implementations can be combined with each other.

[0050] This invention presents a comprehensive and interpretable sketch performance evaluation method based on statistical inference. It is a performance evaluation scheme for sketches, a summary data structure in the network measurement field. This method, based on a user-provided set of accuracy metrics for evaluating sketch performance, identifies the required flow-level features and performs flow-level sampling on the runtime workload. Using a Bootstrap method based on resampling principles, combined with Monte Carlo methods, it constructs a flow-level feature distribution through samples to estimate the accuracy metric values ​​and their confidence intervals. This invention addresses the evaluation requirement of simultaneously ensuring comprehensiveness and interpretability by enabling the analysis of multi-dimensional accuracy metrics and guaranteeing the reliability of intermediate processes and evaluation results.

[0051] See Figure 1 The Sketch performance evaluation method specifically includes the following steps:

[0052] (1) The set of precision indicators given by the user is parsed and processed to obtain the flow level features on which each precision indicator depends and other precision indicators involved in its calculation.

[0053] (1.1) User-defined set of precision metrics {μ1,μ2,…,μ} used to evaluate sketch performance n ,…,μ N}, where μ n Let N represent the nth precision index, and N represent the total number of precision indices.

[0054] (1.2) The precision index set from step (1.1) is parsed and processed, specifically as follows: for each precision index μ in the precision index set... n According to this precision index μ n The calculation formula is analyzed to obtain the accuracy index μ. n The flow-level characteristics on which it depends and other precision metrics involved in its calculation {μ n :({v i |1≤i≤I},{π(μ k )*μ k |1≤k≤N})}, where v i Let I represent the i-th flow level feature, and let π(μ) represent the total number of flow level features it depends on. k ) is an indicator function if and only if the precision index μ n The calculation relies on the precision index μ k It is 1 if it is true, otherwise it is 0.

[0055] It should be understood that the precision metrics used to evaluate sketch performance have corresponding calculation formulas. Based on the calculation formula of the precision metric, the flow level characteristics on which the precision metric depends and other precision metrics involved in its calculation can be determined.

[0056] (2) For each type of flow level feature v i Maintain a sampler to perform stream-level sampling on the runtime workload to obtain a sample set S corresponding to various types of stream-level features. The sampler consists of two parts: a full-stream sampler (Light Sampler) and a heavy-stream tracer (Heavy Tracer). The Light Sampler is responsible for tracing and unordering a specified number of stream key-value pairs (k). j ,v ij Heavy Tracer is used to track and orderly store a specified number of stream-level features v. i The stream key-value pairs (k) are sorted from largest to smallest. j ,v ij ), k j V represents the unique identifier of the j-th stream. ij v represents the flow level feature corresponding to the j-th flow. i The value of .

[0057] (2.1) The Light Sampler is allocated with two arrays Γ and Γ1 as the tracking part and the caching part respectively. Each cell of the arrays Γ and Γ1 stores a flow key-value pair, and their cell storage spaces are M and αM respectively, where M represents the maximum storage capacity of the key-value pairs specified by the user, and α ∈ (0, 1); an empty set for storing any number of keys is allocated. As the eviction part; the Heavy Tracer is allocated with an empty linked list L. Each node in the linked list L stores a flow key-value pair to ensure that the values of the linked list nodes are arranged in an orderly manner. The maximum node storage space of the linked list L is βM, where β ∈ (0, 1).

[0058] (2.2) When performing flow-level sampling on the runtime workload, data packets carrying flow key-value pairs (k j , v ij ) are received one by one. A probability value p is pseudo-randomly generated according to the system time, and the user sets a probability threshold Pr, where p ∈ (0, 1) and Pr ∈ (0, 1); and the following conditional judgments are executed:

[0059] (2.2.1) When Γ tracks k j , the value v j of the flow-level feature in the corresponding cell's flow key-value pair (k ij , v ij ) is updated; if Γ1 caches k j and p < Pr and the corresponding array cell of Γ that has not yet tracked k j has not been associated with other flow key-value pairs, then update the cached flow key-value pair (k j , v ij ) in Γ1 to the corresponding array cell of Γ, clear the cache cell of Γ1, and k j changes from the cached state to the tracked state.

[0060] (2.2.2) When then directly skip all subsequent conditional judgments.

[0061] (2.2.3) When neither Γ nor Γ1 has tracked or cached k j , and p < Pr, and the storage capacity of Γ has not reached the maximum value, update the flow key-value pair (k j , v ij ) to the corresponding storage cell of Γ.

[0062] (2.2.4) When neither Γ nor Γ1 has tracked or cached k j , and p ≥ Pr, and the storage capacity of Γ has reached the maximum value, cache the flow key-value pair (k j , v ij ) to the corresponding storage cell of Γ1.

[0063] (2.2.5) When neither Γ nor Γ1 is tracking or cached k j When p ≥ Pr, and the storage capacity of Γ and Γ1 has reached its maximum value, then k j Record in set For the set Any stream that is no longer tracked or cached is set to an eviction state.

[0064] It should be understood that the probability value p is used for both tracking and caching; that is, the probability threshold must be met before tracking / caching / eviction is performed. After confirming that the stream key-value pair has been tracked, the Light Sampler associates the key with the stored cell location using a hash algorithm, and the key is no longer replaced. The above judgment conditions need to be executed one by one in the order described in steps (2.2.1)-(2.2.5). Only if none of the preceding conditions are met will the last condition, setting the state to eviction, be executed.

[0065] (2.3) For those carrying stream key-value pairs (k j ,v ij The data message, when L does not contain k j The relevant nodes will then stream key-value pairs (k j ,v ij Update to L when the storage size of L reaches its maximum value and does not include k. j For the relevant nodes, replace the key of the node with the smallest value in L with k. j Then (k) j ,v ij Updated to L.

[0066] (2.4) Merge the stream key-value pairs stored in Light Sampler obtained in step (2.2) and the stream key-value pairs stored in Heavy Tracer obtained in step (2.3) to obtain the sample set S = {(k j ,v ij Let S = |j = 1, 2, ..., J, where J represents the total number of samples in the sample set S.

[0067] It should be noted that the evaluation device described in this invention and the sketch being evaluated are placed under the same runtime workload, and the above process is used to perform stream-level sampling on the runtime workload to obtain the sample set S = {(k j ,v ij For any sample key k j key value v ij Compared to real runtime workloads Consistent, S is A subset of.

[0068] (3) Use the sample set S obtained in step (2) to perform Bootstrap resampling, randomly selecting samples in each round. A subset is constructed from the subsamples. in The total number of samples in the sample set S, ∈ (0,1); according to the subsample set S * Number of samples Determine whether to introduce a Bayesian weighting model.

[0069] Furthermore, based on the subsample set S * Number of samples Determining whether to introduce a Bayesian weighting model specifically includes: judging the subsample set S * Total number of samples Does it exceed the user-defined threshold for the number of subsample sets, SamplesThreshold? If the number of subsample sets S... * Total number of samples If the sample size exceeds the user-defined threshold SamplesThreshold, then a predefined Dirichlet distribution from the Bayesian weighting model is introduced. As a weight, where Update the subsample set to Otherwise, the Bayesian weighting model is not introduced.

[0070] It should be understood that the Dirichlet distribution is a component of the Bayesian weighted model. This is because it effectively incorporates uncertainty into the model, generating different values.

[0071] (4) The mean and confidence interval of the estimators of the precision index to be evaluated are calculated by Monte Carlo simulation; wherein, the estimators of the precision index to be evaluated are obtained based on the estimators of other precision indices on which it depends, and the estimators of the precision indices are based on the subsample set S obtained by resampling. * Calculated and obtained.

[0072] Specifically, for the accuracy index μ n Calculate its estimate according to its calculation formula. Based on the subsample set S obtained through round t1 resampling * The accuracy index μ was calculated. n estimator set For those relying on other precision metrics {μ n The accuracy index μ to be evaluated is calculated for |n=1,2,...,u} hPerform t2 rounds of Monte Carlo simulations, meaning each round of Monte Carlo simulations uses the set of estimates for each dependent accuracy metric. The estimated value is randomly selected and substituted into the accuracy index μ to be evaluated. h The corresponding estimated value is obtained by performing calculations using the given formula. The t2 round Monte Carlo simulation ultimately yielded the accuracy index μ to be evaluated. h estimator set And calculate the accuracy index μ to be evaluated. h The mean and confidence interval of the estimator, where u is the precision index to be evaluated, are given. h The number of other accuracy metrics on which it is relied.

[0073] It should be understood that the present invention requires the measurement performance of the sketch (i.e., the estimation of a given accuracy index value). Since the accuracy index value is an estimate and may contain errors, a confidence interval needs to be given to indicate that the estimated value of the sketch's accuracy index is within a certain confidence interval to have a certain degree of credibility.

[0074] Furthermore, the accuracy index μ to be evaluated h The mean and confidence interval of the estimators are obtained through the following steps:

[0075] (4.1) Based on the accuracy index μ to be evaluated h estimator set Calculate the accuracy index μ to be evaluated h mean of the estimators and standard deviation

[0076] (4.2) If the user sets the confidence level θ, the corresponding significance level is 1-θ, where θ∈(0,1).

[0077] (4.3) Determine the accuracy index μ to be evaluated h The test checks whether the number of estimators t2 in the estimator set is greater than the sample set size threshold SamplesThreshold. If t2 > SamplesThreshold, then the critical value Z of the standard normal distribution Z is used. θ / 2 Determine the accuracy index to be evaluated, μ h confidence interval If t2 ≤ SamplesThreshold, then use the critical value of the t-distribution. Determine the accuracy index to be evaluated, μ h confidence interval Z θ / 2 It is the critical value of the standard normal distribution at confidence level θ. It is the critical value of the t distribution at confidence level θ and degrees of freedom n-1.

[0078] It should be understood that the critical value Z of the standard normal distribution Z is... θ / 2 It is well known that the critical value of the t-distribution is... Z can be obtained by looking up a table or using a publicly available software calculator. θ / 2 and The value of .

[0079] It is worth mentioning that this invention also provides a sketch performance evaluation device for implementing the fully interpretable sketch performance evaluation method based on statistical inference methods described in the above embodiments. This sketch performance evaluation device includes a precision index parsing module, a sampler module, a data processor module, and an analyzer module, such as... Figure 2 As shown.

[0080] In this embodiment, the precision index parsing module is used to parse the set of precision indices given by the user in order to obtain the flow level features on which each precision index depends and other precision indices involved in its calculation.

[0081] In this embodiment, the sampler module is used to maintain a sampler for each type of flow-level feature. The sampler performs flow-level sampling on the runtime workload to obtain the sample set S corresponding to each type of flow-level feature. The sampler includes a full-flow sampler and a reflow tracer. The full-flow sampler is responsible for tracking and storing a specified number of flow key-value pairs in an unordered manner. The reflow tracer is responsible for tracking and storing a specified number of flow key-value pairs whose values ​​are sorted from largest to smallest in an ordered manner.

[0082] In this embodiment, the data processor module is used to perform Bootstrap resampling using the sample set S obtained through the sampler module, randomly sampling in each round. A subset S is constructed from the subsamples. * According to the subsample set S * Number of samples Determine whether to introduce a Bayesian weighting model, where It is the total number of samples in the sample set S, ∈ (0,1).

[0083] In this embodiment, the analyzer module is used to calculate the mean and confidence interval of the estimated precision index to be evaluated through Monte Carlo simulation; wherein, the estimated precision index to be evaluated is obtained based on the estimated precision indices it depends on, and the estimated precision index is based on the subsample set S obtained by resampling. * Calculated and obtained.

[0084] In summary, in the evaluation device described in this invention, the sampler module effectively samples the flow-level characteristics of the workload and outputs them to the data processor module and the analyzer module for calculating accuracy indicators and their confidence intervals. The specific implementation process of the functions and roles of each unit in the above scheme is detailed in the corresponding steps of the above method, and will not be repeated here.

[0085] Corresponding to the aforementioned embodiment of the fully interpretable sketch performance evaluation method based on statistical inference, the present invention also provides an embodiment of an electronic device.

[0086] See Figure 3 An electronic device provided in this embodiment of the invention includes one or more processors and a memory, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the fully interpretable sketch performance evaluation method based on statistical inference method in the above embodiment.

[0087] The embodiments of the electronic device described in this invention can be applied to any device with data processing capabilities, such as a computer or other similar device. The electronic device embodiments can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logically defined electronic device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 3 The diagram shown is a hardware structure diagram of any device with data processing capabilities, including the electronic device described in this invention. Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0088] The specific implementation process of the functions and roles of each unit in the above electronic device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0089] For the electronic device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The electronic device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0090] This invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements the fully interpretable sketch performance evaluation method based on statistical inference methods described in the above embodiments.

[0091] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0092] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A comprehensive interpretable sketch performance evaluation method based on statistical inference method, characterized in that, The method comprises the following steps: (1) analyzing a set of precision indicators given by a user to obtain flow level features relied on by each precision indicator and other precision indicators involved in the calculation of the precision indicators; (2) maintaining a sampler for each type of flow level feature, and performing flow level sampling on a runtime workload through the sampler to obtain a sample set S corresponding to each type of flow level feature; wherein the sampler comprises a full flow sampler and a reflow tracker, the full flow sampler is configured to track and save a specified number of flow key-value pairs in disorder, and the reflow tracker is configured to track and save a specified number of flow level feature corresponding values in order in the top flow key-value pairs in descending order; (3) Bootstrap resampling is performed using the sample set S obtained in step (2), and a sub-sample set S* is constructed by randomly sampling sub-samples in each round, and the number of samples in the sub-sample set S* is determined whether to introduce a Bayesian weight model, wherein is the total number of samples in the sample set S, and ∈ ∈ (0, 1); (4) calculating the mean and confidence interval of the estimate of the to-be-evaluated precision indicator through Monte Carlo simulation; wherein the estimate of the to-be-evaluated precision indicator is calculated based on the estimates of other precision indicators relied on by the to-be-evaluated precision indicator, and the estimate of the precision indicator is calculated based on the sub-sample set S* obtained through resampling.

2. The comprehensive interpretable sketch performance evaluation method based on statistical inference method according to claim 1, characterized in that, The step (1) comprises the following sub-steps: (1.1) A user-defined set of precision metrics {μ1, μ2, …, μN} for evaluating the performance of a sketch, where μn denotes the nth precision metric, and N denotes the total number of precision metrics. n ,…,μ N}, wherein μ n n denotes the nth precision metric, and N denotes the total number of precision metrics. (1.2) The precision index set from step (1.1) is parsed and processed, specifically as follows: for each precision index μ in the precision index set... n According to this precision index μ n The calculation formula is analyzed to obtain the accuracy index μ. n The flow-level characteristics on which it depends and other precision metrics involved in its calculation {μ n :({v i |1≤i≤I},{π(μ k )*μ k |1≤k≤N})}, where v i Let I represent the i-th flow level feature, and let π(μ) represent the total number of flow level features it depends on. k ) is an indicator function if and only if the precision index μ n The calculation relies on the precision index μ k It is 1 if it is true, otherwise it is 0.

3. The comprehensive interpretable sketch performance evaluation method based on statistical inference method of claim 1, wherein, The stream key-value pair includes a unique identification k of the stream j and a value v of its corresponding stream level feature v i ij .​ 4. The comprehensive interpretable sketch performance evaluation method based on statistical inference method of claim 1, wherein, The step (2) comprises the following sub-steps: (2.1) The full-flow sampler is assigned two arrays Γ and Γ1 as the tracking part and the cache part respectively, each cell of the arrays Γ and Γ1 stores a flow key-value pair, and the cell storage space is M and αM respectively, where M represents the maximum storage amount of key-value pairs specified by the user, and α ∈ (0, 1); a set for storing any number of keys is assigned as the eviction part; The reflow tracker is allocated with an empty linked list L, each node in the linked list L stores a flow key-value pair, so as to ensure that the values of the linked list nodes are arranged in order, and the maximum node storage space of the linked list L is βM, wherein β ∈ (0, 1); (2.2) When sampling the runtime workload at the flow level, the data packets carrying the flow key-value pair (k j ,v ij ) are received one by one, a probability value p is generated pseudo-randomly according to the system time, the user sets a probability threshold Pr, where p∈(0,1), Pr∈(0,1); and the following conditional judgment is performed: (2.2.1) When Γ tracks k j , then update the value v j of the flow level feature in the corresponding cell flow key-value pair (k ij , v ij ); if Γ1 caches k j and p < Pr and Γ's corresponding array cell for k j has not yet been associated with another flow key-value pair, then update the cached flow key-value pair (k j , v ij ) in Γ1 to Γ's corresponding array cell, and clear the Γ1 cached cell, k j from the cached state to the tracked state; (2.2.2) when then skip all the following conditional tests directly; (2.2.3) When neither Γ nor Γ1 tracks or caches k j , and p < Pr, and the storage of Γ does not reach the maximum, update the flow key-value pair (k j ,v ij ) to the corresponding storage unit of Γ; (2.2.4) When neither Γ nor Γ1 tracks or caches k j , and p ≥ Pr, and the storage of Γ has reached the maximum value, cache the flow key-value pair (k j ,v ij ) to the corresponding storage unit of Γ1; (2.2.5) When neither Γ nor Γ1 tracks or caches k j , and p ≥ Pr, and the storage of both Γ and Γ1 has reached its maximum, k j is logged into the set For any flow logged into the set , which is no longer tracked or cached, is put into the eviction state; (2.3) For the data packet carrying the flow key-value pair (k j ,v ij ), if L does not contain the node related to k j , update the flow key-value pair (k j ,v ij ) to L, if the storage of L reaches the maximum value and does not contain the node related to k j , replace the key of the node with the minimum value in L with k j , and then update (k j ,v ij ) to L; (2.4) merging the stream key-value pairs stored by the full flow sampler obtained in step (2.2) and the stream key-value pairs stored by the re-flow tracer obtained in step (2.3) to obtain a sample set S = {(k j ,v ij )|j = 1, 2,..., J} corresponding to each type of flow level feature, where J represents the total number of samples in the sample set S.

5. The comprehensive interpretable sketch performance evaluation method based on statistical inference method of claim 1, wherein, In the step (3), the determining the sample number of the sub-sample set S * determining whether to introduce a Bayesian weight model, specifically comprising:​ determining whether the total number of samples of the sub-sample set S * is greater than a user-set sub-sample set number threshold SamplesThreshold, and if the total number of samples of the sub-sample set S * is greater than the user-set sub-sample set number threshold SamplesThreshold, introducing a Dirichlet distribution defined in advance in the Bayesian weight model as the weight, where and updating the sub-sample set to otherwise, not introducing the Bayesian weight model.​​ 6. The comprehensive interpretable sketch performance evaluation method based on statistical inference method of claim 1, wherein, The step (4) specifically comprises: For the precision indicator μ n , its estimator is calculated according to its calculation formula Based on the sub-sample set S * obtained by t1 rounds of resampling The estimator set of the precision indicator μ n is calculated For the precision indicator μ k to be evaluated which depends on other precision indicators {μ h |k=1,2,...,u}, t2 rounds of Monte Carlo simulation are performed, that is, in each round of Monte Carlo simulation, an estimator value is randomly selected from the estimator set of each dependent precision indicator and substituted into the calculation formula of the precision indicator μ h to be evaluated to perform calculation to obtain the corresponding estimator value The t2 rounds of Monte Carlo simulation finally obtain the estimator set of the precision indicator μ h to be evaluated And the mean value and confidence interval of the estimator of the precision indicator μ h to be evaluated are calculated, where u is the number of other precision indicators on which the precision indicator μ h to be evaluated depends.

7. The comprehensive interpretable sketch performance evaluation method based on statistical inference method according to claim 6, characterized in that, the precision indicator μ to be evaluated h The estimator mean and confidence interval of the precision indicator μ are computed by the following steps: (4.1) The estimator set of the accuracy indicator μ h to be evaluated is determined The estimator mean value of the accuracy indicator μ H to be evaluated is determined and the standard deviation (4.2) the user sets a confidence level θ, and the corresponding significance level is 1-θ, wherein θ ∈ (0, 1); (4.3) judge whether the number of estimators t2 in the estimator set of the to-be-evaluated precision indicator μ h is greater than the sub-sample set number threshold SamplesThreshold, if t2>SamplesThreshold, then the critical value Z θ / 2 of the standard normal distribution Z is used to determine the confidence interval of the to-be-evaluated precision indicator μ h if t2≤SamplesThreshold, then the critical value t of the t-distribution is used to determine the confidence interval of the to-be-evaluated precision indicator μ h where Z θ / 2 is the critical value of the standard normal distribution at the confidence level θ, and t is the critical value of the t-distribution at the confidence level θ and the degree of freedom n-1.​​ 8. A sketch performance evaluation device for implementing a statistically inference method based overall interpretable sketch performance evaluation method according to any one of claims 1 to 7, characterized by, The method comprises: The precision indicator analysis module is configured to analyze a set of precision indicators given by a user to obtain flow level features relied on by each precision indicator and other precision indicators involved in the calculation of the precision indicators; The sampler module is configured to maintain a sampler for each type of flow level feature, and perform flow level sampling on a runtime workload through the sampler to obtain a sample set S corresponding to each type of flow level feature; wherein the sampler comprises a full flow sampler and a reflow tracker, the full flow sampler is configured to track and save a specified number of flow key-value pairs in disorder, and the reflow tracker is configured to track and save a specified number of flow level feature corresponding values in order in the top flow key-value pairs in descending order; a data processor module for Bootstrap resampling using the sample set S obtained by the sampler module, each round randomly drawing a sub-sample to construct a sub-sample set S * , determining whether to introduce a Bayesian weight model according to the sample quantity of the sub-sample set S * , wherein is the total number of samples of the sample set S, ∈ ∈ (0, 1) ; and is the total number of samples of the sample set S, ∈ ∈ (0, 1) ; and an analyser module for calculating, by means of Monte Carlo simulation, an estimator mean value and a confidence interval of the accuracy indicator to be evaluated; wherein the estimator of the accuracy indicator to be evaluated is calculated on the basis of the estimators of the other accuracy indicators on which it depends, the estimators of the accuracy indicators being obtained on the basis of the sub-sample sets S obtained by resampling * are calculated.

9. An electronic device, comprising one or more processors and memory, wherein: The memory is coupled with the processor; wherein the memory is configured to store program data, and the processor is configured to execute the program data to implement the comprehensive interpretable sketch performance evaluation method based on the statistical inference method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, A program is stored thereon, and when the program is executed by a processor, the comprehensive interpretable sketch performance evaluation method based on the statistical inference method according to any one of claims 1-7 is implemented.

Images