A distribution measurement method and device based on similarity dynamic compression, equipment and medium

Abstract

The application discloses a distribution measurement method and device based on similarity dynamic compression, equipment and medium, which can dynamically select row compression or column compression mode according to task characteristics. When data is inserted, a key-value pair is mapped to a specific row and column in a two-dimensional array through a hash function. When the counter of a certain row or column reaches a threshold, the compression operation of the row or column is triggered. When inquiring, the actual value is recovered by using the compression counter and the frequency approximation formula. The application realizes efficient and accurate network measurement in the per-key distribution scene, and is particularly suitable for a memory-limited environment. When processing large-scale data streams, the application can significantly reduce memory consumption and improve processing efficiency while maintaining high accuracy.

Description

Technical Field

[0001] This invention relates to the field of data traffic estimation, and in particular to a distributed measurement method, apparatus, device, and medium based on similarity dynamic compression. Background Technology

[0002] Data analytics is becoming crucial in numerous fields, particularly in cybersecurity, financial risk management, and IoT monitoring. Key-based distribution analysis tracks the distribution of items with the same key across histogram intervals, providing deeper insights compared to single-value metrics. This approach allows network administrators to identify traffic patterns and anomalies by analyzing the distribution of IP packet sizes, while financial analysts leverage transaction amount distributions to detect risky spending and predict defaults, thereby ensuring better risk control and regulatory compliance.

[0003] Sketch algorithms have become an important tool for flow estimation, achieving an optimal balance between accuracy, computational efficiency, and memory conservation. These probabilistic data structures enable efficient stream processing by allowing multiple keys to share a counter, while carefully controlling accuracy loss through collision management. By strategically trading minimal, controllable accuracy loss for significant memory reduction, Sketch algorithms provide an ideal real-time analytics solution for resource-constrained big data applications. Recent research has significantly expanded their applications, particularly in key-based distribution monitoring and other advanced detection tasks across various domains.

[0004] However, existing methods face key limitations in memory-constrained environments. While sampling techniques achieve resource efficiency, they introduce significant distortion to low-frequency key distributions. Histogram methods, though capable of comprehensive tracking, suffer from exponentially increasing memory requirements as the number of keys and intervals increases, limiting their scalability. Even HistSketch—despite its innovative approach to key-based distribution measurements—experiences significant performance degradation under strict memory constraints due to its inflexible architecture and computational complexity.

[0005] In many data stream processing scenarios, data appears in the form of key-value pairs. A per-key task involves processing the data associated with each key individually, such as the distribution of values ​​corresponding to each key. Current solutions for per-key tasks face significant challenges, especially under limited storage conditions. A major drawback is the significant increase in error rates when memory is constrained. Another limitation is the failure to effectively utilize the similarity of each key. For example, in network traffic monitoring, devices within the same subnet often perform similar data reporting tasks, resulting in highly similar packet length distributions. Existing methods fail to fully leverage this inherent similarity to optimize performance and accuracy. Summary of the Invention

[0006] To address the performance degradation issues of existing Sketch network measurement schemes in scenarios with complex sketch structures and unutilized data characteristics, which lead to decreased detection performance, this invention provides a distribution measurement method, apparatus, device, and medium based on similarity-based dynamic compression that can effectively improve memory resource utilization efficiency.

[0007] To achieve the above-mentioned technical objectives, the technical solution of the present invention is as follows:

[0008] A distribution measurement method based on similarity-based dynamic compression, when executing per-key tasks in a batch using the Sketch algorithm, if the per-key tasks in this batch are determined to have row similarity or column similarity, then performs the corresponding row compression or column compression through the following steps:

[0009] Step 1, Initialization: Set all base counters and compression counters in Sketch to zero; the base counters are set in the base buckets of the two-dimensional array that makes up Sketch, and the compression counters are set in each row and each column of the two-dimensional array of Sketch.

[0010] Step 2: Read the information of the single key-value pair to be inserted, and then determine the target insertion position of the key-value pair in the two-dimensional array based on the hash function;

[0011] Step 3: Based on the target insertion position and whether row compression or column compression is required, check if the corresponding compression counter is zero. If it is, insert the key-value pair into the base bucket at the target insertion position. Otherwise, calculate the probability of insertion based on the value of the compression counter and perform the insertion accordingly. After the insertion is performed, increment the corresponding base counter by 1 and proceed to step 4. If the insertion is not performed, jump to step 5.

[0012] Step 4: Check if the base counter in the inserted base bucket has reached the capacity of the base bucket. If so, divide each data in the entire row or column of the base bucket by the preset constant value c, round the quotient up and replace the original data. At the same time, increment the compression counter of the corresponding row or column by 1, and then proceed to step 5. Otherwise, proceed directly to step 5.

[0013] Step 5: Check if all key-value pair insertions for each key task in this batch have been processed. If so, end the method; otherwise, return to step 2 and repeat.

[0014] Furthermore, this method determines whether each key task in this batch has row similarity or column similarity through the following steps:

[0015] When different values ​​exhibit similar frequency distributions under the same key, each key task is considered to have row similarity;

[0016] When different keys exhibit similar distributions at the same value, each key task is considered to have column similarity.

[0017] Furthermore, in step 3 of this method, the probability is calculated based on the value of the compression counter as follows:

[0018] Calculate the probability using the following formula:

[0019] ;

[0020] Where c is a preset constant value, and b is the value recorded in the compression counter of the corresponding row or column.

[0021] Furthermore, when performing queries based on the Sketch algorithm, this method restores the compressed data to its original form through the following steps:

[0022] ;

[0023] Where 'a' represents the compressed data stored in the base bucket.

[0024] The present invention also provides a distribution measurement device based on similarity dynamic compression, comprising:

[0025] The compression method selection module is used to determine whether each key task in a batch has row similarity or column similarity when executing a batch of key tasks based on the Sketch algorithm, and then perform the corresponding row compression or column compression accordingly.

[0026] The initialization module is used to reset the count values ​​of all basic counters and compression counters in Sketch to zero. The basic counters are set in the basic buckets that make up the two-dimensional array of Sketch, and the count is incremented by 1 when a key-value pair is inserted. The compression counters are set in each row and each column of the two-dimensional array of Sketch, and the count is incremented by 1 when the corresponding row or column is compressed.

[0027] The key-value pair insertion position determination module is used to read the information of a single key-value pair to be inserted, and then determine the target insertion position of the key-value pair in the two-dimensional array based on a hash function;

[0028] The key-value pair insertion probability determination module checks whether the corresponding compression counter is zero based on the target insertion position and whether row compression or column compression is required. If it is, the key-value pair is inserted into the base bucket at the target insertion position. Otherwise, the insertion probability is calculated based on the value of the compression counter, and the insertion is performed according to the probability. After the insertion is performed, the compression module takes over. If the insertion is not performed, the termination module takes over.

[0029] The compression module checks whether the base counter in the inserted base bucket has reached the capacity of the base bucket. If so, it divides each data in the entire row or column of the base bucket by the preset constant value c, rounds the quotient up, replaces the original data, and then transfers the execution to the termination module; otherwise, it directly transfers the execution to the termination module.

[0030] The module ends and checks whether all key-value pair insertions for each key task in this batch have been processed. If so, the device operation ends; otherwise, the key-value pair insertion position determination module continues to execute in a loop.

[0031] Furthermore, the process by which the compression method selection module of this device determines whether each key task in this batch has row similarity or column similarity includes:

[0032] When different values ​​exhibit similar frequency distributions under the same key, each key task is considered to have row similarity;

[0033] When different keys exhibit similar distributions at the same value, each key task is considered to have column similarity.

[0034] Furthermore, the key-value pair insertion probability determination module of this device calculates the probability based on the value of the compression counter as follows:

[0035] Calculate the probability using the following formula:

[0036] ;

[0037] Where c is a preset constant value, and b is the value recorded in the compression counter of the corresponding row or column.

[0038] Furthermore, when this device performs a query based on the Sketch algorithm, it uses a data recovery module to restore the compressed data to its original state. This data recovery module performs the recovery based on the following formula:

[0039] ;

[0040] Where 'a' represents the compressed data stored in the base bucket.

[0041] In addition, the present invention also provides an electronic device, comprising:

[0042] One or more processors;

[0043] Storage device for storing one or more programs.

[0044] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0045] In addition, the present invention also provides a computer-readable medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0046] The technical advantages of this invention lie in its dynamic selection of row or column compression modes based on the similarity of key-value pairs, maintaining measurement accuracy while effectively reducing memory usage. It also employs a probabilistic insertion method, inserting key-value pairs with a certain probability based on the state of the compression counter, thereby reducing memory update operations, lowering memory access frequency and processing latency, and minimizing errors. Simultaneously, compression reduces memory waste. Furthermore, compression directly reduces memory usage by decreasing the values ​​stored in the base counter, and the compression counter tracks the degree of compression, ensuring accurate reconstruction of the original data during queries. Through these technical means, this invention enables real-time adjustment of the Sketch structure, improving memory utilization efficiency and increasing detection speed without compromising detection accuracy. This allows for efficient and accurate network measurement in per-key distribution scenarios, making it particularly suitable for memory-constrained environments. When processing large-scale data streams, this invention can significantly reduce memory consumption and improve processing efficiency while maintaining high accuracy. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating an embodiment of the present invention.

[0048] Figure 2 This is a comparison chart showing the performance of the embodiments of the present invention with other algorithms in terms of the accuracy of point queries. Among them, (a) is a comparison chart of interval ratios, and (b) is a comparison chart of the relative error of the average interval.

[0049] Figure 3 This is a comparison chart showing the performance of embodiments of the present invention with other algorithms in terms of the accuracy of histogram queries. (a) is a comparison chart of key ratios, and (b) is a comparison chart of the relative error of the average histogram.

[0050] Figure 4 This is a schematic diagram comparing the throughput of an embodiment of the present invention with that of the HistSketch algorithm.

[0051] Figure 5 This is a schematic diagram comparing the bandwidth of an embodiment of the present invention with that of the HistSketch algorithm. Detailed Implementation

[0052] This invention primarily achieves efficient data processing through similarity compression. In data processing scenarios, especially in areas such as network traffic monitoring, financial data analysis, and IoT systems, key-by-key distributed monitoring is crucial. However, existing methods struggle to simultaneously guarantee accuracy and low resource consumption in memory-constrained environments. The innovation of this embodiment lies in employing a dual-mode compression mechanism, dynamically switching between row and column compression based on key similarity patterns. This design leverages the similarity of keys or values ​​in key-value pairs to reduce storage overhead. For example, in network traffic monitoring, devices within the same subnet typically perform similar data reporting tasks, resulting in highly similar packet length distributions; multiple server IPs serving the same video stream service (despite different 5-tuples) often have packet lengths concentrated within a specific range due to transmitting similar content. This invention utilizes this similarity to improve storage efficiency.

[0053] This invention can dynamically select row or column compression mode based on task characteristics. During data insertion, a hash function maps key-value pairs to specific rows and columns in a two-dimensional array. When the counter for a row or column reaches a threshold, a compression operation is triggered. During querying, the actual value is reconstructed by approximating the value and frequency of the compression counter.

[0054] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0055] See Figure 1 This embodiment first provides a distribution measurement method based on similarity dynamic compression. When executing a batch of per-key tasks based on the Sketch algorithm, if it is determined that the per-key tasks in this batch have row similarity or column similarity, then the corresponding row compression or column compression is performed.

[0056] In determining row or column similarity, this embodiment considers each key task to have row similarity when different values ​​exhibit similar frequency distributions under the same key. For example, different ports of the same IP address may have similar traffic patterns, where the IP address can be considered as the key and the port as the value.

[0057] When different keys exhibit a similar distribution for the same value, each key task is considered to have column similarity. For example, multiple server IP addresses of the same video streaming service may have similar packet length distributions when transmitting the same content. Here, the IP address can be considered as the key, and the packet length can be considered as the value.

[0058] The specific steps of this embodiment include: Step 1, initialization: Reset the count values ​​of all basic counters and compression counters in Sketch to zero. The basic counters are set in the basic buckets of the two-dimensional array constituting Sketch, and the compression counters are set in each row and each column of the two-dimensional array of Sketch.

[0059] Step 2: Read the information of the single key-value pair to be inserted, and then determine the target insertion position of this key-value pair in the two-dimensional array based on the hash function. Specifically, this embodiment uses row hash function and column hash function respectively to determine the insertion position of the key-value pair in the row and column, thereby obtaining the target insertion position. Taking the key-value pair (k, v) as an example, the row hash function can be used to calculate the row insertion position i by i = h(k) mod r, and the column hash function can be used to calculate the column insertion position j by j = g(v) mod B, where r represents the number of rows in the two-dimensional array, and B represents the number of basic buckets in each row.

[0060] Step 3: Based on the target insertion position and whether row compression or column compression is required, check if the corresponding compression counter is zero. If it is, insert the key-value pair into the base bucket at the target insertion position. Otherwise, calculate the insertion probability based on the compression counter value and perform the insertion accordingly. For example, if the calculated insertion probability is 25%, then the probability of inserting is 25%, and the probability of not inserting is 75%. After insertion, the corresponding base counter is incremented by 1, and the process proceeds to Step 4. If insertion is not performed, the process jumps to Step 5. The probability is calculated using the following formula:

[0061] ;

[0062] Where 'c' is a preset constant value, and 'b' is the value recorded in the compression counter of the corresponding row or column. 'c' can be adjusted according to specific implementation needs. It should be noted that a smaller 'c' value may lead to more frequent scaling operations, while a larger 'c' value may reduce the scaling frequency but increase memory usage.

[0063] Step 4: Check if the base counter in the inserted base bucket has reached the capacity of the base bucket (i.e., greater than or equal to the capacity of the base bucket). If so, divide each data in the entire row or column of the base bucket by the preset constant value c, round the quotient up, and replace the original data. At the same time, increment the compression counter of the corresponding row or column by 1, and then proceed to Step 5. Otherwise, proceed directly to Step 5.

[0064] Step 5: Check if all key-value pair insertions for each key task in this batch have been processed. If so, end the method. Otherwise, return to step 2 and repeat the process.

[0065] After compression, when performing a query based on the Sketch algorithm, the compressed data is restored to its original form through the following steps:

[0066] ;

[0067] Where 'a' represents the compressed data stored in the base bucket.

[0068] The reason for performing probability calculations and inserting data according to probability in step 3 is that after compression in step 4, the count of data in the base bucket has decreased compared to the value before compression. If new data is directly inserted at this point, it would be equivalent to unreasonably adding an excessively large value to the compressed value, leading to overestimation during subsequent restoration. For example, suppose the original value is 6, and the preset constant value c is set to 2. During the first compression, 6 is divided by 2 and rounded down, resulting in a compressed value of 3. If we then insert 1 directly into the compressed value of 3, it would be equivalent to this 1 being calculated as 2 during subsequent restoration, causing overestimation. Therefore, step 3 uses probabilistic insertion to prevent the insertion of some key-value pairs. For example, the probability of insertion is calculated here, with c set to 2 and b set to 1, i.e., a probability of 1 / 2. This minimizes the possibility of overestimation and improves the accuracy of the method.

[0069] Meanwhile, this embodiment also provides a distribution measurement device based on similarity dynamic compression, including:

[0070] The compression method selection module is used to determine whether each key task in a batch has row similarity or column similarity when executing a batch of key tasks based on the Sketch algorithm, and thus perform the corresponding row compression or column compression. Specifically, key tasks are considered to have row similarity when different values ​​exhibit similar frequency distributions under the same key. Similarly, key tasks are considered to have column similarity when different keys exhibit similar distributions under the same value.

[0071] The initialization module is used to reset the count values ​​of all basic counters and compression counters in Sketch to zero. The basic counters are set in the basic buckets that make up the two-dimensional array of Sketch, and the count is incremented by 1 when a key-value pair is inserted. The compression counters are set in each row and each column of the two-dimensional array of Sketch, and the count is incremented by 1 when the corresponding row or column is compressed.

[0072] The key-value pair insertion position determination module is used to read the information of a single key-value pair to be inserted, and then determine the target insertion position of the key-value pair in the two-dimensional array based on a hash function.

[0073] The key-value pair insertion probability determination module checks whether the corresponding compression counter is zero, based on the target insertion position and whether row or column compression is required. If it is, the key-value pair is inserted into the base bucket at the target insertion position; otherwise, the probability is calculated based on the compression counter value, and insertion is performed accordingly. After insertion, the compression module takes over; if no insertion is performed, the termination module takes over. The probability calculation formula is as follows:

[0074] .

[0075] Where c is a preset constant value, and b is the value recorded in the compression counter of the corresponding row or column.

[0076] The compression module checks whether the base counter in the inserted base bucket has reached the capacity of the base bucket. If so, it divides all data in the entire row or column of the base bucket by a preset constant value c, rounds the quotient up, replaces the original data, and then transfers the execution to the termination module. Otherwise, it directly transfers the execution to the termination module.

[0077] The module that terminates checks whether all key-value pair insertions for each key task in this batch have been processed. If so, the device operation ends. Otherwise, the key-value pair insertion position determination module continues to execute in a loop.

[0078] When this device performs a query based on the Sketch algorithm, it uses a data recovery module to restore the compressed data to its original state. This data recovery module performs the recovery based on the following formula:

[0079] .

[0080] Where 'a' represents the compressed data stored in the base bucket.

[0081] According to embodiments of the present invention, the present invention also provides an electronic device and a computer-readable medium.

[0082] Electronic devices include:

[0083] One or more processors;

[0084] Storage device for storing one or more programs.

[0085] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0086] In practical use, users can interact with servers, which are also electronic devices, via a network to receive or send messages. Terminal devices are generally various electronic devices equipped with a display and used through a human-computer interface, including but not limited to smartphones, tablets, laptops, and desktop computers. Various specific application software can be installed on these terminal devices as needed, including but not limited to web browsers, instant messaging software, social media platforms, and shopping apps.

[0087] A server is a network server used to provide various services. The distribution measurement method based on similarity dynamic compression provided in this embodiment is generally executed by the server. In practical applications, under necessary conditions, the terminal device can also directly execute the distribution measurement method based on similarity dynamic compression. Correspondingly, the distribution measurement method device based on similarity dynamic compression can be set on the server. Similarly, under necessary conditions, the distribution measurement method based on similarity dynamic compression can also be set in the terminal device.

[0088] Similarly, the computer-readable medium of the present invention stores a computer program thereon, which, when executed by a processor, implements the distribution measurement method based on similarity dynamic compression according to embodiments of the present invention.

[0089] The method provided by this invention is compared with other existing solutions through experimental analysis below:

[0090] The experimental setup is as follows:

[0091] Platform: The experiment was conducted on a virtual machine equipped with a quad-core CPU (Intel(R) Core(TM) i5-8265U CPU @ 1.60GHz). Each core has three levels of cache: 128KB L1 data cache, 128KB L1 instruction cache, 1MB L2 cache, and 12MB L3 cache. The operating system used was Ubuntu 22.04.3.

[0092] Dataset: Anonymous real-world datasets collected using CAIDA in 2018. To facilitate key-value pair detection, the keys in this dataset are constructed using 13-byte complexes, including the source / destination IP address, port, and protocol, while the values ​​represent the length of the IP packet. To effectively capture data distribution characteristics and reduce memory usage, the packet length values ​​are divided into 16 equal-length intervals.

[0093] Experimental Methods: The method of this invention (hereinafter referred to as SC-sketch) will be compared with Count-Min Sketch (CM), Counting Bloom Filter (CBF) and HistSketch (HS). The comparison will focus on the interval ratio, average ARE, throughput and bandwidth of each scheme when using different memory spaces.

[0094] Figure 2 This demonstrates the performance of SC-Sketch in point query tasks. From Figure 1 (a) shows that the SC-Sketch interval ratio is close to 1 across all memory configurations, indicating that its estimated frequency is mostly within the error range of the actual frequency. For example, with 0.2MB of memory, the SC-Sketch interval ratio is close to 0.8, while CM and CBF are almost 0, and HS is about 0.3. Figure 1 (b) shows that the average relative error of SC-Sketch is significantly lower than that of other algorithms, especially under low memory conditions, its error is only about 1 / 5 of that of CM, CBF and HS, which reflects higher accuracy and stability.

[0095] Figure 3 The performance of SC-Sketch in histogram query tasks is described. From... Figure 2 (a) In terms of key ratio, SC-Sketch leads under all memory conditions, especially with 0.2MB of memory, where its key ratio is about 0.4, which is much higher than the close to 0 ratio of CM and CBF, and 0.1 of HS. Figure 2 (b) shows that the average histogram relative error of SC-Sketch is significantly lower than that of other algorithms, such as about 50 with 0.2MB of memory, while CM exceeds 300, and CBF and HS are about 150; when the memory is increased to 2MB, the error of SC-Sketch drops to about 20, while other algorithms are still above 50, highlighting its high-precision advantage in key value distribution estimation.

[0096] Figure 4 The throughput of SC-Sketch and HistSketch was compared. With 0.4MB of memory, SC-Sketch's throughput was approximately 12 Mips, higher than HS's 10 Mips. When memory increased to 4MB, SC-Sketch's throughput further improved to approximately 22 Mips, while HS's was approximately 18 Mips. This indicates that SC-Sketch, with its dynamic compression mechanism and probabilistic insertion technology, effectively reduces processing latency and improves data processing efficiency, especially with increased memory, where its speed advantage becomes more significant. SC-Sketch can better utilize resources when memory is limited, thus achieving higher throughput.

[0097] Figure 5 The bandwidth efficiency of SC-Sketch and HistSketch was compared. With 0.4MB of memory, SC-Sketch's byte ratio was approximately 0.25, lower than HistSketch's 0.3. When memory increased to 4MB, SC-Sketch's byte ratio further decreased to approximately 0.22, while HistSketch's was approximately 0.27. SC-Sketch better optimizes resource utilization under memory constraints, thus achieving higher bandwidth efficiency.

Claims

1. A distribution measurement method based on similarity dynamic compression, characterized in that, When executing a batch of per-key tasks based on the Sketch algorithm, if it is determined that the per-key tasks in this batch have row similarity or column similarity, then the corresponding row compression or column compression is performed through the following steps: Step 1, Initialization: Set all base counters and compression counters in Sketch to zero; the base counters are set in the base buckets of the two-dimensional array that makes up Sketch, and the compression counters are set in each row and each column of the two-dimensional array of Sketch. Step 2: Read the information of the single key-value pair to be inserted, and then determine the target insertion position of the key-value pair in the two-dimensional array based on the hash function; Step 3: Based on the target insertion position and whether row compression or column compression is required, check if the corresponding compression counter is zero. If it is, insert the key-value pair into the base bucket at the target insertion position. Otherwise, calculate the probability of insertion based on the value of the compression counter and perform the insertion accordingly. After the insertion is performed, increment the corresponding base counter by 1 and proceed to step 4. If the insertion is not performed, jump to step 5. Step 4: Check if the base counter in the inserted base bucket has reached the capacity of the base bucket. If so, divide each data in the entire row or column of the base bucket by the preset constant value c, round up the quotient and replace the original data. At the same time, increment the compression counter of the corresponding row or column by 1, and then proceed to step 5. Otherwise, proceed directly to step 5; Step 5: Check if all key-value pair insertions for each key task in this batch have been processed. If so, end the method; otherwise, return to step 2 and repeat. The following steps are used to determine whether each key in this batch has row similarity or column similarity: When different ports of the same IP address have similar traffic patterns, where the IP address is regarded as the key and the port is regarded as the value, each key task is considered to have row similarity; When multiple server IP addresses of the same video streaming service have similar packet length distributions when transmitting the same content, where the IP address is considered the key and the packet length is considered the value, then each key task is considered to have column similarity.

2. The method according to claim 1, characterized in that, In step 3, the probability is calculated based on the value of the compression counter: Calculate the probability using the following formula: 1 / c b ； Where c is a preset constant value, and b is the value recorded in the compression counter of the corresponding row or column.

3. The method according to claim 2, characterized in that, After compression is complete, when a query needs to be executed based on the Sketch algorithm, the compressed data is restored to its original state through the following steps: a×c b ; Where 'a' represents the compressed data stored in the base bucket.

4. A distribution measurement device based on similarity dynamic compression, characterized in that, include: The compression method selection module is used to determine whether each key task in a batch has row similarity or column similarity when executing a batch of key tasks based on the Sketch algorithm, and then perform the corresponding row compression or column compression accordingly. The initialization module is used to reset the count values ​​of all basic counters and compression counters in Sketch to zero. The basic counter is set in the basic bucket of the two-dimensional array that constitutes Sketch. The count is incremented by 1 when a key-value pair is inserted. The compression counter is set in each row and each column of the two-dimensional array of Sketch. The count is incremented by 1 when the corresponding row or column is compressed. The key-value pair insertion position determination module is used to read the information of a single key-value pair to be inserted, and then determine the target insertion position of the key-value pair in the two-dimensional array based on a hash function; The key-value pair insertion probability determination module is used to check whether the corresponding compression counter is zero based on the target insertion position and whether row compression or column compression needs to be performed. If it is, the key-value pair is inserted into the base bucket at the target insertion position. Otherwise, the insertion probability is calculated based on the value of the compression counter and the insertion is performed according to the probability. After the insertion is performed, the compression module will take over. If the insertion is not performed, the termination module will take over. The compression module checks whether the base counter in the inserted base bucket has reached the capacity of the base bucket. If so, it divides each data in the entire row or column of the base bucket by the preset constant value c, rounds the quotient up, replaces the original data, and then transfers the execution to the termination module; otherwise, it directly transfers the execution to the termination module. The module ends and checks whether all key-value pair insertions for each key task in this batch have been processed. If so, the device operation ends; otherwise, the key-value pair insertion position determination module continues to execute in a loop. The process by which the compression method selection module determines whether each key task in this batch has row similarity or column similarity includes: When different ports of the same IP address have similar traffic patterns, where the IP address is regarded as the key and the port is regarded as the value, each key task is considered to have row similarity; When multiple server IP addresses of the same video streaming service have similar packet length distributions when transmitting the same content, where the IP address is considered the key and the packet length is considered the value, then each key task is considered to have column similarity.

5. The apparatus according to claim 4, characterized in that, The key-value pair insertion probability determination module calculates the probability based on the value of the compression counter: Calculate the probability using the following formula: 1 / c b ； Where c is a preset constant value, and b is the value recorded in the compression counter of the corresponding row or column.

6. The apparatus according to claim 5, characterized in that, When performing a query based on the Sketch algorithm, a data recovery module is used to restore the compressed data to its original state. This data recovery module performs the recovery based on the following formula: a×c b ; Where 'a' represents the compressed data stored in the base bucket.

7. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-3.

8. A computer-readable medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-3.

Images