Quantile estimation method and device for data stream

Abstract

The invention provides a quantile estimation method and device for a data stream. The method comprises the following steps: determining infinite positive and negative directions and generation number probability distribution according to a target quantile; before the to-be-processed data is inserted every time, generating a plurality of infinite values according to the probability distribution of the generation number and the positive and negative directions of the infinite; sequentially inserting the to-be-processed data and the plurality of infinite values into the storage table, and maintaining the storage space of the storage table by removing the maximum value and the minimum value in the data stored in the storage table; when a quantile query instruction is received, the median of the stored data in the target value storage table is returned, the originally distributed quantile is converted into the newly distributed median, the requirement of any quantile is covered by utilizing the high efficiency of median estimation, and the method can adapt to various distribution forms, and is particularly suitable for complex and dynamic change scenes in network delay.

Description

Technical Field

[0001] The present invention relates to the technical field of data statistics, and in particular to a quantile estimation method for data streams. Background Art

[0002] DDsketch and t-digest are two efficient data structures designed for quantile estimation, each with its own focus on technical measures and construction logic.

[0003] The core of DDsketch lies in its dynamic exponential bucketing mechanism. This divides the data range into multiple buckets based on exponential growth, creating narrower bucket intervals for low-value regions and wider intervals for high-value regions. This approach reduces storage requirements for high-value regions while ensuring high precision for extreme quantiles. Each bucket only records the number of data points within the interval. When inserting data, the corresponding bucket is quickly located based on the value and the count is updated. Quantiles are calculated using linear interpolation of the accumulated counts. This design can lead to reduced precision when estimating intermediate quantiles due to the wide bucket intervals.

[0004] t-digest uses a centroid clustering strategy to represent the data distribution by maintaining a set of dynamically merged centroids. When inserting data, new data points are initialized as independent centroids. Adjacent centroids are then merged based on a scale function (e.g., a sine function based on the quantile position). This allows for more centroids to be retained near the median in dense data areas (e.g., to improve accuracy). In sparse areas (e.g., at the ends of the distribution), a smaller number of centroids is merged to save memory. Quantiles are calculated using cumulative weighted interpolation estimates of centroids, which makes t-digest excellent at estimating central quantiles, but suffers from larger errors for extreme quantiles.

[0005] Both data structures rely on strong assumptions about the data distribution, for example, a long-tailed distribution or a center-dense distribution. Summary of the Invention

[0006] The present invention provides a data stream-oriented quantile estimation method to address the defect of the existing technology that the data structure depends on the strong assumption of data distribution, and to achieve adaptation to various distribution forms through calibration technology. It is particularly suitable for scenarios with complex and dynamic changes in network delay.

[0007] The present invention provides a data stream-oriented quantile estimation method, comprising the following steps.

[0008] Determine the positive and negative directions of infinity and the probability distribution of generated numbers based on the target quantile; Before inserting each data to be processed, multiple infinite values are generated according to the probability distribution of the generated number and the positive and negative directions of infinity; the data to be processed and the multiple infinite values are inserted into the storage table in sequence, and the storage space of the storage table is maintained by removing the maximum and minimum values in the data stored in the storage table; Among them, the number of infinite values follows the probability distribution of generating numbers; When receiving a quantile query instruction, the median of the data stored in the target value storage table is returned.

[0009] According to the present invention, a data stream quantile estimation method is provided, wherein the positive and negative directions of infinity and the probability distribution of generated numbers are determined according to the target quantile, including: When the target quantile is greater than 0.5, the direction of infinity is determined to be a positive number, and the probability distribution of the generated number is parameterized The geometric distribution of Represents the target percentile.

[0010] According to the present invention, a data stream quantile estimation method is provided, wherein the positive and negative directions of infinity and the probability distribution of generated numbers are determined according to the target quantile, and further comprising: When the target percentile is less than 0.5, the direction of infinity is determined to be a negative number, and the probability distribution of the generated number is parameterized The geometric distribution of .

[0011] According to the present invention, a data stream-oriented quantile estimation method is provided, wherein to-be-processed data and a plurality of infinite values are sequentially inserted into a storage table, and the storage space of the storage table is maintained by removing the maximum and minimum values in the data stored in the storage table, including: Inserting the data to be processed and multiple infinite values into the candidate storage table in sequence; when the remaining storage space of the candidate value storage table is insufficient, removing two intermediate value data from the stored data in the candidate value storage table each time, and sending the two intermediate value data to the target value storage table until the storage space of the candidate value storage table can accommodate the data to be processed and multiple infinite values; Insert the intermediate value data sent to the target value storage table into the target value storage table; When the remaining storage space of the target value storage table is insufficient, the maximum value and the minimum value in the data stored in the target value storage table are removed until the storage space of the target value storage table can accommodate the intermediate value data.

[0012] According to a data stream quantile estimation method provided by the present invention, wherein, after removing two intermediate value data from the data stored in the candidate value storage table each time and sending the two intermediate value data to the target value storage table, the method further includes: Remove all remaining data in the candidate value storage table storage data.

[0013] The present invention also provides a data stream quantile estimation method and device, comprising the following modules: An infinity value determination module is used to determine the positive and negative directions of infinity and the probability distribution of generated numbers according to the target quantile; The storage table data maintenance module is used to generate multiple infinite values according to the probability distribution of the generated number and the positive and negative directions of infinity before inserting the data to be processed each time; insert the data to be processed and the multiple infinite values into the storage table in sequence, and maintain the storage space of the storage table by removing the maximum and minimum values in the data stored in the storage table; Among them, the number of infinite values follows the probability distribution of generating numbers; The median return module is used to return the median of the data stored in the target value storage table when receiving a quantile query instruction.

[0014] According to a data stream quantile estimation method and device provided by the present invention, Infinity value determination module, including: The first determination submodule is used to determine that the direction of infinity is a positive number when the target percentile is greater than 0.5, and the probability distribution of the generated number is parameterized The geometric distribution of Indicates the target percentile.

[0015] According to a data stream quantile estimation device provided by the present invention, the infinity value determination module further includes: When the target percentile is less than 0.5, the direction of infinity is determined to be a negative number, and the probability distribution of the generated number is parameterized The geometric distribution of .

[0016] According to a data stream quantile estimation device provided by the present invention, the storage table data maintenance module includes: The candidate storage table maintenance submodule is used to sequentially insert the data to be processed and multiple infinite values into the candidate storage table; when the remaining storage space of the candidate value storage table is insufficient, remove two intermediate value data from the stored data in the candidate value storage table each time and send the two intermediate value data to the target value storage table until the storage space of the candidate value storage table can accommodate the data to be processed and multiple infinite values; The target value storage table maintenance submodule is used to insert the intermediate value data sent to the target value storage table into the target value storage table; when the remaining storage space of the target value storage table is insufficient, the maximum and minimum values in the target value storage table storage data are removed until the storage space of the target value storage table can accommodate the intermediate value data.

[0017] According to the data stream-oriented quantile estimation device provided by the present invention, the storage table data maintenance module is further used to remove all remaining data in the candidate value storage table storage data.

[0018] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the data stream-oriented quantile estimation method as described above is implemented.

[0019] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for estimating quantiles for data streams as described above is implemented.

[0020] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned data stream-oriented quantile estimation methods.

[0021] The present invention provides a data stream-oriented quantile estimation method, which determines the positive and negative directions of infinity and the probability distribution of generated numbers according to a target quantile; generates multiple infinity values according to the probability distribution of generated numbers and the positive and negative directions of infinity before each insertion of data to be processed; inserts the data to be processed and the multiple infinity values into a storage table in sequence, and maintains the storage space of the storage table by removing the maximum and minimum values in the data stored in the storage table; upon receiving a quantile query instruction, returns the median of the data stored in the target value storage table, converts the quantile of the original distribution into the median of the new distribution, utilizes the high efficiency of median estimation to cover the needs of any quantile, can adapt to various distribution forms, and is particularly suitable for scenarios with complex and dynamic changes in network delay. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 It is a flow chart of the data stream-oriented quantile estimation method provided by the present invention.

[0024] Figure 2a The original data distribution provided by the present invention Quantile diagram.

[0025] Figure 2bIt is a schematic diagram of the 0.5 quantile (median) in the new data distribution provided by the present invention.

[0026] Figure 3 These are three running examples of quantile estimation provided by the present invention.

[0027] Figure 4 It is a structural diagram of the data stream quantile estimation device provided by the present invention.

[0028] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0029] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0030] The following combination Figure 1-Figure 5 The present invention is described.

[0031] The present invention is implemented by introducing a MagnifierSketch data structure. The present invention provides a data stream-oriented quantile estimation method, that is, the purpose of the present invention is to estimate any quantile of data in a data stream.

[0032] The execution subject of the present invention is a measurement system, which includes an insertion phase and a query phase.

[0033] To facilitate understanding, several specific scenarios to which the present invention is applicable are introduced.

[0034] Scenario 1: Overall Service Quality Evaluation. Aggregate latency percentiles play an important role in assessing overall network service quality, particularly in large-scale systems where latency varies across flows. For example, service areas with high overall latency may indicate congestion, indicating the need for additional servers to improve the quality of service (QoS) for each customer. QoS describes and controls the quality of service provided by network data transmission. By estimating aggregate latency percentiles in real time, administrators can better understand the overall state of the network and more efficiently address performance bottlenecks.

[0035] Scenario 2: SLA compliance monitoring. Service Level Agreements (SLAs) define the performance guarantees provided by service providers, and tail latency is a key metric in these agreements. Service providers can monitor aggregate latency quantiles to ensure network compliance with SLA requirements, thereby providing high-quality service to users.

[0036] Scenario 3: Anomaly Detection. A sudden increase in per-flow latency may indicate an anomaly in the network, such as a potential cyberattack or offensive activity. For example, a Ωeb service experiencing a distributed denial of service (DDoS) attack may experience a sudden increase in response time. However, the aggregate latency quantile remains unchanged because millions of low-latency packets can mask this phenomenon. Therefore, per-flow latency quantile estimation can be used to detect anomalies.

[0037] Scenario 4: Per-Floω Resource Allocation. Finding an optimal resource allocation strategy can significantly reduce latency on a stream. Real-time measurement of per-stream latency quantiles provides valuable insights into individual client performance, enabling servers to make data-driven decisions about resource distribution across the network.

[0038] Figure 1 This is a flow chart of the data stream quantile estimation method provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 101: Determine the positive and negative directions of infinity and the probability distribution of generated numbers according to the target quantile.

[0039] In step 101 above, the target quantile refers to the quantile corresponding to any quantile of the data in the data stream that needs to be estimated by the present invention. The target quantile can be set according to user needs. For example, if the user needs to estimate the 0.9 quantile of the data in the data stream, the target quantile is 0.9. If the user needs to estimate the 0.2 quantile of the data in the data stream, the target quantile is 0.2.

[0040] For the convenience of description, the present invention also refers to the data distribution in the data stream as the original data distribution, and each data in the data stream as the to-be-processed data, which can also be referred to as delayed data.

[0041] The positive and negative directions of infinity refer to the positive direction of infinity and the negative direction of infinity. Infinity in the positive direction is a positive number, and infinity in the negative direction is a negative number.

[0042] This paper uses distribution calibration technology to convert quantile estimates under the original data distribution into estimates of the 0.5 quantile, or median, under the new data distribution. Leveraging the high efficiency of median estimates to cover any quantile, this approach overcomes the limitations of traditional methods like DDsketch and t-digest, which suffer from inaccuracies in the middle or at both ends.

[0043] In order to achieve calibration of data distribution, the embodiment of the present invention generates multiple infinite values during the insertion phase. The generated number probability distribution refers to the geometric distribution obeyed by the number of infinite values generated each time the data to be processed is inserted.

[0044] Specifically, we assume that the network delay data obeys a distribution denoted as F. In order to support arbitrary aggregate delay quantile estimation, a distribution calibration technique is proposed. The key idea of the distribution calibration technique is to construct a new distribution , so that the original data distribution of Quantiles are just new distributions By this method, any quantile is converted into an estimate of the 0.5 quantile, that is, for any quantile estimate, it is only necessary to require the 0.5 quantile estimate under the new distribution. Quantiles such as Figure 2a As shown, the new distribution The 0.5 quantile is Figure 2b Specifically, in order to calibrate the distribution, each time a delayed data is inserted, a probability method is applied to generate several positive infinities or negative infinities and insert them into the data structure together with the delayed data to be inserted each time, that is, the data to be processed.

[0045] Optionally, the above step 101 includes step A1: Step A1: When the target percentile is greater than 0.5, determine that the direction of infinity is a positive number, and the probability distribution of the generated number is parameterized The geometric distribution of Indicates the target percentile.

[0046] Optionally, the above step 101 further includes step A2: Step A2: When the target percentile is less than 0.5, determine that the direction of infinity is a negative number, and the probability distribution of the generated number is parameterized The geometric distribution of .

[0047] Step 102: Before inserting the data to be processed each time, generate multiple infinite values according to the probability distribution of the generated number and the positive and negative directions of infinity; insert the data to be processed and the multiple infinite values into the storage table in sequence, and maintain the storage space of the storage table by removing the maximum and minimum values in the data stored in the storage table; wherein the number of infinite values follows the probability distribution of the generated number.

[0048] In the above steps A1 and 102, when A random number is generated when , which obeys the parameter The geometric distribution of The positive infinity values and the data v to be processed are inserted into the storage table in sequence. Since a batch of positive infinity values are randomly inserted, from the expectation point of view, when ω>0.5, each time a data is inserted, 2ω-1 positive infinity values will be inserted at the same time. Therefore, each data in the new distribution after distribution calibration will have The probability of follows the original data distribution, and there is The probability is positive infinity.

[0049] The geometric distribution is a discrete probability distribution, typically expressed as X ~ GE(p), where p is a parameter. In this embodiment, a random number Z is generated according to the probability distribution of GE(1 / 2ω), i.e., Z ~ GE(p).

[0050] The value range of the general geometric distribution is 1 to positive infinity. The positive infinity value can ensure that the number of positive infinity values is from 0 to positive infinity. Mathematical proofs ensure that under this setting, the error can be limited to an acceptable range.

[0051] In the above steps A2 and 102, the situation when ω<0.5 is the same as The situation is similar. Similarly, a random number Z is generated, and the probability of random number Z being The geometric distribution of and Negative infinity is inserted into the storage table in sequence. Similarly, according to the expectation, under the new data distribution, each data will have The probability of follows the original data distribution, The probability is negative infinity.

[0052] Optionally, the above step 102 further includes steps B1 to B3: Step B1: insert the data to be processed and multiple infinite values into the candidate storage table in sequence; when the remaining storage space of the candidate value storage table is insufficient, remove two intermediate value data from the data stored in the candidate value storage table each time, and send the two intermediate value data to the target value storage table until the storage space of the candidate value storage table can accommodate the data to be processed and multiple infinite values.

[0053] Step B2: Insert the intermediate value data sent to the target value storage table into the target value storage table.

[0054] Step B3: When the remaining storage space of the target value storage table is insufficient, the maximum value and the minimum value in the data stored in the target value storage table are removed until the storage space of the target value storage table can accommodate the intermediate value data.

[0055] In the above steps B1 to B3, the MagnifierSketch data structure consists of two tables, which are respectively recorded as the candidate storage table (Candidate) and the target value storage table (Representative).

[0056] The design goal of Candidate is to retain multiple delayed data from the data stream, that is, the network delay data, and only the delayed data that may become the 0.5 quantile can be selected into the Representative. The present invention assumes that the table Candidate can retain r delayed data in the data structure. Specifically, when inserting the to-be-processed data and multiple infinite values into MagnifierSketch, for each to-be-processed data v in the network delay data, the present invention first inserts it into the Candidate. Then the present invention checks whether the Candidate is full after the insertion. If the number of data in the Candidate is less than r, the present invention only completes the insertion process and returns. Otherwise, the present invention selects two median delayed data in the Candidate and sends them to the Representative.

[0057] The design goal of the table Representative is to record the delay data close to the quantile that the present invention needs to estimate. The present invention assumes that the table Representative can retain s delay data. To store two delay samples selected from Candidate and Inserting into Representative, the present invention first checks whether Representative is full. If Representative still has space for storing delayed data, the present invention only needs to and Otherwise, the present invention traverses Representative and finds the minimum and maximum delay data from the s delay data in Representative and the delay data to be inserted. Assume that the minimum and maximum delay data are and . Two minimum and maximum delay data and will be evicted from the Representative table, and and The data is inserted into the Representative table. The data that remains in the Representative table is the data filtered by the Candidate table, which is closer to the median.

[0058] Optionally, after removing two intermediate value data from the data stored in the candidate value storage table each time in the above step B1 and sending the two intermediate value data to the target value storage table, step B11 is further included: B11: Remove all remaining data in the candidate value storage table.

[0059] The present invention clears all delayed data in Candidate to make room for other data in the network delayed data.

[0060] This method uses a two-level storage structure: the Candidate table temporarily caches data, and the Representative table screens key samples. During insertion, the data distribution is adjusted by generating positive / negative infinity probabilities, retaining only candidate values that are likely to become the target quantile, significantly reducing storage and computing overhead. This ensures both storage efficiency and accuracy.

[0061] Step 103: When a quantile query instruction is received, the median of the data stored in the target value storage table is returned.

[0062] During the query phase, we consider delayed samples in the Representative and return the median in the Representative as the 0.5 quantile of the data stream.

[0063] The following is further explained through specific examples.

[0064] This example assumes that the created tables Candidate and Representative each have only four storage locations. Figure 3The following figure shows three examples of running in latency quantile estimation. In the first example, to insert latency data v = 7, this example first generates a random number Z and sequentially inserts Z - 1 infinities and v into Candidate. The following mainly illustrates the insertion process for v = 7. Because Candidate is not full, the latency data is not inserted into Representative, completing the insertion process for v = 7. In the second example, to insert latency data v = 11, this example similarly generates a random number Z and sequentially inserts Z - 1 infinities and v into Candidate. The following mainly illustrates the insertion process for v = 11. This example inserts 11 into Candidate, which will be full after the insertion. Therefore, this example selects two median latency data from Candidate (here, 42 and 133). Because Representative still has space for these two latency data, 42 and 133 are inserted into Representative. In the final example, to insert a delay v = 35, this example again first generates a random number Z and sequentially inserts Z-1 infinities and v into Candidate. The following mainly illustrates the insertion process of v. This example inserts 35 into Candidate. After the insertion, Candidate is full, so this example selects the two median values (here, 56 and 35) and inserts them into Representative. Since Representative is full, this example selects the maximum and minimum delay samples from these six numbers (here, 18 and 109). 18 and 109 are evicted from the Representative table, and 56 and 35 take their places in the Representative table.

[0065] The embodiments of the present invention can achieve the following effects: 1. Accuracy of aggregated latency quantile estimation: On the CAIDA and Synthetic datasets, MagnifierSketch's average error (AE) is comparable to existing algorithms (e.g., GK, KLL, DDSketch), but significantly outperforms t-digest and RegSketch. For example, MagnifierSketch achieves a much lower AE than the latter two algorithms within a 10KB memory limit.

[0066] Throughput: Insert throughput reached approximately 20M items / s, twice that of t-digest and seven times that of GK, and only slightly lower than KLL and RegSketch. Throughput was highest at a quantile of 0.5. As the target quantile deviated from 0.5, throughput decreased due to the need to insert more virtual values (positive and negative infinity values), but still remained efficient.

[0067] 2. Single-stream delay quantile estimation Accuracy: On the CAIDA and Synthetic datasets, MagnifierSketch achieves an AE of less than 0.06 and 0.03, respectively, which is approximately half that of other algorithms. Its advantage is particularly evident in memory-constrained scenarios, for example, under 80 KB.

[0068] Throughput: Both insertion and query throughput are close to 10 Mops, significantly outperforming some baseline algorithms. For example, GK's insertion throughput is <5 Mops, and KLL's query throughput is <2 Mops.

[0069] The present invention provides a data stream-oriented quantile estimation method, which determines the positive and negative directions of infinity and the probability distribution of generated numbers according to the target quantile; before each insertion of the data to be processed, generates multiple infinite values according to the probability distribution of the generated numbers and the positive and negative directions of infinity; inserts the data to be processed and the multiple infinite values into the storage table in sequence, and maintains the storage space of the storage table by removing the maximum and minimum values in the data stored in the storage table; when receiving a quantile query instruction, returns the median of the data stored in the target value storage table. Compared with the limitations of the two traditional methods of DDsketch and t-digest, which are insufficient in accuracy in the middle or at both ends, the quantile of the original distribution is converted into the median of the new distribution, and the high efficiency of the median estimation is used to cover the needs of any quantile, thereby reducing the computational complexity. At the same time, the quantile estimation method provided by the present invention does not rely on strong assumptions about the data distribution, can adapt to a variety of distribution forms, and is particularly suitable for complex and dynamically changing scenarios in network delays. It solves the limitations of traditional methods of insufficient accuracy in the middle or at both ends.

[0070] The data stream oriented quantile estimation method and apparatus provided by the present invention is described below. The data stream oriented quantile estimation method and apparatus described below and the data stream oriented quantile estimation method described above can refer to each other.

[0071] like Figure 4 As shown, the present invention provides a data stream quantile estimation method and device, including the following modules: Infinity value determination module 401, used to determine the positive and negative directions of infinity and the probability distribution of generated numbers according to the target quantile; The storage table data maintenance module 402 is configured to generate a plurality of infinite values according to the probability distribution of the generated number and the positive and negative directions of the infinity before each insertion of the data to be processed; sequentially insert the data to be processed and the plurality of infinite values into the storage table, and maintain the storage space of the storage table by removing the maximum and minimum values in the data stored in the storage table; wherein the number of infinite values follows the probability distribution of the generated number; The median returning module 403 is configured to return the median of the data stored in the target value storage table upon receiving a quantile query instruction.

[0072] Optionally, the infinity value determination module 401 includes: The first determination submodule is used to determine that the direction of infinity is a positive number when the target percentile is greater than 0.5, and the probability distribution of the generated number is parameterized The geometric distribution of Indicates the target percentile.

[0073] Optionally, the infinity value determining module 401 further includes: When the target percentile is less than 0.5, the direction of infinity is determined to be a negative number, and the probability distribution of the generated number is parameterized The geometric distribution of .

[0074] Optionally, the storage table data maintenance module 402 includes: The candidate storage table maintenance submodule is used to sequentially insert the data to be processed and multiple infinite values into the candidate storage table; when the remaining storage space of the candidate value storage table is insufficient, remove two intermediate value data from the stored data in the candidate value storage table each time and send the two intermediate value data to the target value storage table until the storage space of the candidate value storage table can accommodate the data to be processed and multiple infinite values; The target value storage table maintenance submodule is used to insert the intermediate value data sent to the target value storage table into the target value storage table; when the remaining storage space of the target value storage table is insufficient, the maximum and minimum values in the target value storage table storage data are removed until the storage space of the target value storage table can accommodate the intermediate value data.

[0075] Optionally, the storage table data maintaining module 402 is further configured to remove all remaining data in the candidate value storage table storage data.

[0076] The present invention provides a data stream-oriented quantile estimation device, which determines the positive and negative directions of infinity and the probability distribution of generated numbers according to the target quantile; before each insertion of the data to be processed, generates multiple infinity values according to the probability distribution of the generated numbers and the positive and negative directions of infinity; inserts the data to be processed and the multiple infinity values into the storage table in sequence, and maintains the storage space of the storage table by removing the maximum and minimum values in the data stored in the storage table; and returns the median of the data stored in the target value storage table when receiving a quantile query instruction. Compared with the limitations of the two traditional methods of DDsketch and t-digest, which have insufficient accuracy in the middle or at both ends, the quantile of the original distribution is converted into the median of the new distribution, and the high efficiency of median estimation is used to cover the needs of any quantile without relying on strong assumptions about the data distribution. It can adapt to a variety of distribution forms and is particularly suitable for scenarios with complex and dynamic changes in network delays. It solves the limitations of traditional methods of insufficient accuracy in the middle or at both ends.

[0077] Figure 5 An example of a physical structure diagram of an electronic device is shown below. Figure 5 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call logic instructions in the memory 830 to execute the data stream-oriented quantile estimation method.

[0078] Furthermore, the logic instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0079] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the data stream-oriented quantile estimation method provided by the above methods.

[0080] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the data stream-oriented quantile estimation method provided by the above methods.

[0081] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0082] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A quantile estimation method for data streams, characterized in that: include: Determine the positive and negative directions of infinity and the probability distribution of generated numbers based on the target quantile; Before inserting the data to be processed each time, generating multiple infinite values according to the probability distribution of the generated number and the positive and negative directions of infinity; inserting the data to be processed and the multiple infinite values into the storage table in sequence, and maintaining the storage space of the storage table by removing the maximum and minimum values in the data stored in the storage table; wherein the number of said infinite values obeys said generator probability distribution; When a quantile query instruction is received, the median of the data stored in the target value storage table is returned.

2. The data stream quantile estimation method according to claim 1, characterized in that Determining the positive and negative directions of infinity and the probability distribution of generated numbers according to the target quantile includes: When the target percentile is greater than 0.5, the direction of infinity is determined to be a positive number, and the probability distribution of the generated number is parameterized by The geometric distribution of represents the target quantile.

3. The data stream quantile estimation method according to claim 2, characterized in that: The determining of the positive and negative directions of infinity and the probability distribution of the generated number according to the target quantile further includes: When the target percentile is less than 0.5, the direction of infinity is determined to be a negative number, and the probability distribution of the generated number is parameterized by The geometric distribution of .

4. The data stream oriented quantile estimation method according to claim 1, wherein the inserting the to-be-processed data and the plurality of infinite values into a storage table in sequence, and maintaining the storage space of the storage table by removing the maximum and minimum values in the data stored in the storage table, comprises: Inserting the data to be processed and the plurality of infinity values into a candidate storage table in sequence; When the remaining storage space of the candidate value storage table is insufficient, removing two intermediate value data from the data stored in the candidate value storage table each time, and sending the two intermediate value data to the target value storage table until the storage space of the candidate value storage table can accommodate the data to be processed and the plurality of infinite values; Insert the intermediate value data sent to the target value storage table into the target value storage table; When the remaining storage space of the target value storage table is insufficient, the maximum value and the minimum value in the data stored in the target value storage table are removed until the storage space of the target value storage table can accommodate the intermediate value data.

5. The data stream oriented quantile estimation method according to claim 4, wherein after removing two intermediate value data from the data stored in the candidate value storage table each time and sending the two intermediate value data to the target value storage table, the method further comprises: All remaining data in the candidate value storage table is removed.

6. A data stream quantile estimation method and device, characterized in that: include: An infinity value determination module is used to determine the positive and negative directions of infinity and the probability distribution of generated numbers according to the target quantile; a storage table data maintenance module, configured to generate a plurality of infinite values according to the probability distribution of the generated number and the positive and negative directions of infinity before inserting each data to be processed; insert the data to be processed and the plurality of infinite values into the storage table in sequence, and maintain the storage space of the storage table by removing the maximum and minimum values in the data stored in the storage table; wherein the number of said infinite values obeys said generator probability distribution; The median return module is used to return the median of the data stored in the target value storage table when receiving a quantile query instruction.

7. The data stream oriented quantile estimation device according to claim 6, characterized in that: The infinity value determination module includes: The first determination submodule is used to determine that the direction of infinity is a positive number when the target percentile is greater than 0.5, and the probability distribution of the generated number is parameterized by The geometric distribution of represents the target quantile.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the data stream-oriented quantile estimation method according to any one of claims 1 to 5 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data stream-oriented quantile estimation method according to any one of claims 1 to 5 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the data stream-oriented quantile estimation method according to any one of claims 1 to 5 is implemented.

Images