Distribution measurement method and device based on similarity dynamic compression, equipment and medium

Abstract

The invention discloses a distribution measurement method and device based on similarity dynamic compression, equipment and a medium. A row compression mode or a column compression mode can be dynamically selected according to task characteristics. During data insertion, key value pairs are mapped to specific rows and columns in a two-dimensional array through a hash function. And when the counter of a certain row or column reaches a threshold value, the compression operation of the row or column is triggered. During query, the actual value is recovered by using a compression counter and a frequency approximation formula. According to the invention, efficient and accurate network measurement is realized in a per-key distribution scene, and the method is particularly suitable for an environment with a limited memory. When large-scale data streams are processed, high precision can be kept, meanwhile, memory consumption can be remarkably reduced, and processing efficiency can be remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of data flow estimation, and in particular to a distribution measurement method, device, equipment and medium based on similarity dynamic compression. Background Art

[0002] Data analytics has become crucial in numerous fields, particularly in cybersecurity, financial risk management, and IoT monitoring. Key-based distribution analysis tracks the distribution of items with the same key across histogram bins, providing deeper insights than single-value metrics. This approach enables network administrators to identify traffic patterns and anomalies by analyzing IP packet size distribution. Financial analysts can use transaction amount distribution to detect risky spending behavior and predict defaults, thereby ensuring better risk control and regulatory compliance.

[0003] Sketch algorithms have become an essential tool for flow estimation, striking an optimal balance between accuracy, computational efficiency, and memory conservation. These probabilistic data structures enable efficient stream processing by allowing multiple keys to share counters, while carefully controlling accuracy loss through collision management. By strategically trading minimal, controllable accuracy loss for significant memory reduction, Sketch algorithms provide an ideal real-time analytics solution for resource-constrained big data applications. Recent research has significantly expanded their application, particularly in key-based distributed monitoring and other cross-domain advanced detection tasks.

[0004] However, existing methods face critical limitations in memory-constrained environments. Sampling techniques, while achieving resource efficiency, introduce significant distortions to the distribution of low-frequency keys. Histogram methods, while comprehensive, suffer from exponentially increasing memory requirements as the number of keys and intervals increases, limiting their scalability. Even HistSketch—despite its innovative approach to key-based distribution measurement—severely degrades in performance under strict memory constraints due to its inflexible architecture and computational complexity.

[0005] In many data stream processing scenarios, data appears in the form of key-value pairs. Per-key tasks involve processing the data associated with each key separately, such as the distribution of values corresponding to each key. Current solutions for per-key tasks face significant challenges, especially when storage conditions are limited. A major drawback is that the error rate increases significantly when memory is limited. Another limitation is the failure to effectively utilize the similarity of each key. For example, in network traffic monitoring, devices in the same subnet often perform similar data reporting tasks, resulting in highly similar message length distributions. Existing methods fail to fully utilize this inherent similarity to optimize performance and accuracy. Summary of the Invention

[0006] In order to solve the problem that existing Sketch network measurement solutions have complex sketch structures and fail to utilize data characteristics in per-key distribution scenarios, resulting in degraded detection performance, the present invention provides a distribution measurement method, device, equipment, and medium based on similarity dynamic compression that can effectively improve memory resource utilization efficiency.

[0007] In order to achieve the above technical objectives, the technical solution of the present invention is:

[0008] A distribution measurement method based on similarity dynamic compression. When executing a batch of per-key tasks based on the Sketch algorithm, if it is determined that the per-key tasks in this batch have row similarity or column similarity, the corresponding row compression or column compression is performed through the following steps:

[0009] Step 1: Initialize the Sketch by resetting the count values of all base counters and compression counters to zero. The base counters are set in the base buckets of the Sketch's two-dimensional array, and the compression counters are set on each row and column of the Sketch's two-dimensional array.

[0010] Step 2: Read the single key-value pair information to be inserted, and then determine the target insertion position of this key-value pair in the two-dimensional array based on the hash function;

[0011] Step 3: Based on the target insertion position and whether row or column compression is to be performed, check whether the corresponding compression counter is zero. If so, insert the key-value pair into the base bucket of the target insertion position. Otherwise, calculate the probability of insertion based on the value of the compression counter and perform the insertion based on the probability. After the insertion is performed, increment the corresponding base counter by 1 and proceed to step 4. If the insertion is not performed, jump to step 5.

[0012] Step 4: Check whether the basic counter in the inserted basic bucket reaches the capacity of the basic bucket. If so, divide all the data in the basic bucket of the entire row or column by the preset constant value c, round up the quotient and replace the original data. At the same time, increase the compression counter of the corresponding row or column by 1, and then go to step 5; otherwise, go directly to step 5;

[0013] Step 5: Check whether all key-value pair insertions for each key task in this batch have been processed. If so, end the method; otherwise, return to step 2 and execute the loop.

[0014] Furthermore, this method determines whether the per-key tasks in this batch have row similarity or column similarity through the following steps:

[0015] When different values show similar frequency distributions under the same key, the per-key tasks are considered to have row similarity;

[0016] When different keys exhibit similar distributions under the same value, the per-key tasks are said to have column similarity.

[0017] Furthermore, in step 3 of this method, the probability is calculated based on the value of the compression counter as:

[0018] The probability is calculated according to the following formula:

[0019] ;

[0020] Wherein, c is a preset constant value, and b is the value recorded in the compression counter of the corresponding row or column.

[0021] Furthermore, when executing a query based on the Sketch algorithm, this method restores the compressed data to the original data through the following steps:

[0022] ;

[0023] Where a represents the compressed data stored in the base bucket.

[0024] At the same time, the present invention also provides a distribution measurement device based on similarity dynamic compression, comprising:

[0025] The compression method selection module is used to determine whether the per-key tasks in a batch have row similarity or column similarity when executing them based on the Sketch algorithm, and then perform corresponding row compression or column compression;

[0026] An initialization module, used to reset the count values of all basic counters and compression counters in Sketch to zero; the basic counters are set in the basic buckets that constitute Sketch's two-dimensional array, and the count is increased by 1 when a key-value pair is inserted. The compression counters are set on each row and column of Sketch's two-dimensional array, and the count is increased by 1 when the corresponding row or column is compressed;

[0027] A key-value pair insertion position determination module is used to read the information of a single key-value pair to be inserted, and then determine the target insertion position of this key-value pair in the two-dimensional array based on a hash function;

[0028] The key-value pair insertion probability determination module checks whether the corresponding compression counter is zero based on the target insertion position and whether row or column compression needs to be performed. If so, the key-value pair is inserted into the base bucket of the target insertion position. Otherwise, the insertion probability is calculated based on the value of the compression counter and the insertion is performed based on the probability. After the insertion is performed, the execution is transferred to the compression module. If the insertion is not performed, the execution is transferred to the end module.

[0029] The compression module is used to check whether the basic counter in the inserted basic bucket has reached the capacity of the basic bucket. If so, it divides all the data in the basic bucket of the entire row or column by the preset constant value c, rounds up the quotient and replaces the original data, and then transfers the execution to the end module; otherwise, it directly transfers the execution to the end module;

[0030] The end module checks whether the key-value pair insertions of each key task in this batch have been processed. If so, the device operation ends; otherwise, the key-value pair insertion position determination module continues the loop execution.

[0031] Furthermore, the process of the compression mode selection module of the present device determining whether the per-key tasks of this batch have row similarity or column similarity includes:

[0032] When different values show similar frequency distributions under the same key, the per-key tasks are considered to have row similarity;

[0033] When different keys exhibit similar distributions under the same value, the per-key tasks are said to have column similarity.

[0034] Furthermore, the key-value pair insertion probability determination module of this device calculates the probability according to the value of the compression counter as follows:

[0035] The probability is calculated according to the following formula:

[0036] ;

[0037] Wherein, c is a preset constant value, and b is the value recorded in the compression counter of the corresponding row or column.

[0038] Furthermore, when executing a query based on the Sketch algorithm, the device uses a data recovery module to restore the compressed data to the original data. The data recovery module performs recovery based on the following formula:

[0039] ;

[0040] Where a represents the compressed data stored in the base bucket.

[0041] At the same time, the present invention also provides an electronic device, including:

[0042] one or more processors;

[0043] a storage device for storing one or more programs,

[0044] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0045] At the same time, the present invention also provides a computer-readable medium storing a computer program, which implements the aforementioned method when executed by a processor.

[0046] The technical effect of the present invention is that the present invention dynamically selects row compression or column compression mode according to the similarity of key-value pairs, maintaining measurement accuracy under the premise of effectively reducing memory usage. It also uses a probabilistic insertion method, that is, inserting key-value pairs with a certain probability according to the state of the compression counter, thereby reducing memory update operations, reducing memory access frequency and processing delay, and reducing errors. At the same time, memory waste is reduced through compression. And when compressing, the memory usage is directly reduced by reducing the value stored in the basic counter, and the compression counter is used to track the degree of compression to ensure that the original data can be accurately reconstructed during query. The present invention uses the above technical means to enable the Sketch structure to be adjusted in real time, improve memory utilization efficiency, and increase detection speed without affecting detection accuracy. As a result, the present invention realizes efficient and accurate network measurement in per-key distribution scenarios, which is particularly suitable for memory-constrained environments. When processing large-scale data streams, the present invention can significantly reduce memory consumption and improve processing efficiency while maintaining high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 Schematic diagram of a flow chart of an embodiment of the present invention.

[0048] Figure 2 The following are comparisons of the accuracy of point queries between the embodiments of the present invention and other algorithms: (a) is a comparison of interval ratios, and (b) is a comparison of average interval relative errors.

[0049] Figure 3 The following are comparisons of the accuracy of histogram queries between the embodiments of the present invention and other algorithms: (a) is a comparison of key ratios, and (b) is a comparison of average histogram relative errors.

[0050] Figure 4 Schematic diagram showing throughput comparison between an embodiment of the present invention and the HistSketch algorithm.

[0051] Figure 5 Schematic diagram of bandwidth comparison between an embodiment of the present invention and the HistSketch algorithm. DETAILED DESCRIPTION

[0052] The present invention primarily achieves efficient data processing through similarity compression. Key-by-key distributed monitoring is crucial in data processing scenarios, particularly in areas such as network traffic monitoring, financial data analysis, and IoT systems. However, existing methods struggle to simultaneously ensure accuracy and low resource consumption in memory-constrained environments. The innovation of this embodiment lies in the use of a dual-mode compression mechanism that dynamically switches between row and column compression based on the key similarity patterns. This design can leverage the similarity of keys or values in key-value pairs to reduce storage overhead. For example, in network traffic monitoring, devices in the same subnet typically perform similar data reporting tasks, resulting in highly similar message length distributions. Multiple server IP addresses serving the same video streaming service (despite their different quintuples) often have message lengths concentrated in a specific range because they transmit similar content. The present invention leverages this similarity to improve storage efficiency.

[0053] The present invention dynamically selects row or column compression mode based on task characteristics. During data insertion, a hash function maps key-value pairs to specific rows and columns in a two-dimensional array. When a row or column counter reaches a threshold, compression of the row or column is triggered. During queries, the actual value is recovered using the compressed counter value and frequency approximation.

[0054] The embodiments of the present invention are further described below with reference to the accompanying drawings.

[0055] See also Figure 1 This embodiment first provides a distribution measurement method based on similarity dynamic compression. When executing a batch of per-key tasks based on the Sketch algorithm, if it is determined that the per-key tasks of this batch have row similarity or column similarity, corresponding row compression or column compression is performed.

[0056] When determining row or column similarity, this embodiment considers per-key tasks to have row similarity when different values exhibit similar frequency distributions under the same key. For example, different ports of the same IP address may have similar traffic patterns. Here, the IP address can be considered the key, and the port can be considered the value.

[0057] Per-key tasks are considered to have column similarity when different keys exhibit similar distributions for the same values. For example, multiple server IP addresses for the same video streaming service may have similar packet length distributions when transmitting the same content. Here, the IP address can be considered the key, and the packet length can be considered the value.

[0058] The specific steps of this embodiment include: Step 1, performing initialization: resetting the count values of all basic counters and compression counters in Sketch to zero. The basic counters are set in the basic buckets constituting the two-dimensional array of Sketch, and the compression counters are set on each row and each column of the two-dimensional array of Sketch.

[0059] Step 2: Read the single key-value pair information to be inserted, and then determine the target insertion position of this key-value pair in the two-dimensional array based on the hash function. Specifically, this embodiment uses a row hash function and a column hash function to determine the insertion position of the key-value pair in the row and column, respectively, to obtain the target insertion position. Taking the key-value pair (k, v) as an example, the row hash function can use i=h(k)mod r to calculate the row insertion position i, and the column hash function can use j=g(v)mod B to calculate the column insertion position j, where r represents the number of rows in the two-dimensional array and B represents the number of base buckets for each row.

[0060] Step 3: Check whether the corresponding compression counter is zero based on the target insertion position and whether row or column compression needs to be performed. If so, insert the key-value pair into the base bucket of the target insertion position. Otherwise, calculate the probability of insertion based on the value of the compression counter and perform the insertion based on the probability. That is, if the calculated probability of insertion is 25%, then the probability of performing the insertion is 25% and the probability of not performing the insertion is 75%. After the insertion is performed, the corresponding base counter is incremented by 1 and the process goes to step 4. If the insertion is not performed, jump to step 5. The probability is calculated according to the following formula:

[0061] ;

[0062] Where c is a preset constant value, and b is the value recorded in the compression counter for the corresponding row or column. c can be adjusted based on specific implementation needs. It should be noted that a smaller c value may result in more frequent scaling operations, while a larger c value may reduce scaling frequency but increase memory usage.

[0063] Step 4: Check whether the basic counter in the inserted basic bucket reaches the capacity of the basic bucket, that is, it is greater than or equal to the capacity of the basic bucket. If so, divide all the data in the basic bucket of the entire row or column by the preset constant value c, round up the quotient and replace the original data. At the same time, increase the compression counter of the corresponding row or column by 1, and then proceed to step 5. Otherwise, proceed directly to step 5.

[0064] Step 5: Check whether all key-value pair insertions for this batch of key-value tasks have been processed. If so, the method ends. Otherwise, return to step 2 and execute the loop.

[0065] After compression is performed, when executing a query based on the Sketch algorithm, the following steps are used to restore the compressed data to the original data:

[0066] ;

[0067] Where a represents the compressed data stored in the base bucket.

[0068] The reason for performing a probability calculation and inserting data based on this probability in step 3 is that after compression in step 4, the counts in the base bucket have decreased compared to their pre-compression values. Directly inserting new data at this point would unreasonably increase the counts on top of the reduced counts, leading to overestimation during subsequent restoration. For example, suppose the original value is 6, and the constant c is set to 2. During the first compression, 6 is divided by 2 and rounded up, resulting in a compressed value of 3. If 1 were to be inserted directly over the compressed value 3, the 1 would be counted as 2 during subsequent restoration, resulting in an overestimation. Therefore, probabilistic insertion in step 3 prevents some key-value pairs from being inserted. For example, here, we calculate the probability of insertion, taking c to be 2 and b to be 1, resulting in a probability of 1 / 2. This minimizes the possibility of overestimation and improves the accuracy of this method.

[0069] At the same time, this embodiment also provides a distribution measurement device based on similarity dynamic compression, including:

[0070] The compression method selection module is used to determine whether a batch of per-key tasks has row or column similarity when executing them based on the Sketch algorithm, and then perform row or column compression accordingly. Per-key tasks are considered to have row similarity when different values under the same key exhibit similar frequency distributions. Per-key tasks are considered to have column similarity when different keys exhibit similar distributions under the same value.

[0071] Initialization module, used to reset the count values of all basic counters and compression counters in Sketch to zero. The basic counters are set in the basic buckets that make up Sketch's two-dimensional array, and the count increases by 1 when a key-value pair is inserted. The compression counters are set in each row and column of Sketch's two-dimensional array, and the count increases by 1 when the corresponding row or column is compressed.

[0072] The key-value pair insertion position determination module is used to read the single key-value pair information to be inserted, and then determine the target insertion position of this key-value pair in the two-dimensional array based on the hash function.

[0073] The key-value pair insertion probability determination module is used to check whether the corresponding compression counter is zero based on the target insertion position and whether row compression or column compression needs to be performed. If it is, the key-value pair is inserted into the base bucket of the target insertion position. Otherwise, the probability is calculated based on the value of the compression counter and the insertion is performed based on the probability. After the insertion is performed, the compression module will execute the operation. If the insertion is not performed, the end module will execute the operation. The probability calculation formula is:

[0074] .

[0075] Wherein, c is a preset constant value, and b is the value recorded in the compression counter of the corresponding row or column.

[0076] The compression module checks whether the basic counter in the inserted basic bucket has reached the capacity of the basic bucket. If so, it divides all the data in the basic bucket of the entire row or column by the preset constant value c, rounds up the quotient, and replaces the original data. The execution is then transferred to the end module. Otherwise, the execution is directly transferred to the end module.

[0077] The end module checks whether the key-value pair insertion of each key task in this batch has been processed. If so, the device operation is terminated. Otherwise, the key-value pair insertion position determination module continues the loop execution.

[0078] When the device performs a query based on the Sketch algorithm, the compressed data is restored to the original data through the data recovery module. The data recovery module performs the recovery based on the following formula:

[0079] .

[0080] Where a represents the compressed data stored in the base bucket.

[0081] According to an embodiment of the present invention, the present invention also provides an electronic device and a computer-readable medium.

[0082] Electronic equipment includes:

[0083] one or more processors;

[0084] a storage device for storing one or more programs,

[0085] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0086] In practice, users can use electronic devices acting as terminal devices to interact with servers, also serving as electronic devices, over a network to receive or send messages. Terminal devices are generally various electronic devices equipped with a display and operated through a human-computer interface, including but not limited to smartphones, tablets, laptops, and desktop computers. Various specific application software can be installed on terminal devices as needed, including but not limited to web browsers, instant messaging software, social networking platforms, and shopping apps.

[0087] The server is a network service end used to provide various services. The distribution measurement method based on similarity dynamic compression provided in this embodiment is generally executed by the server. In actual application, under the necessary conditions, the terminal device can also directly execute the distribution measurement method based on similarity dynamic compression. Accordingly, the distribution measurement method device based on similarity dynamic compression can be set on the server. Similarly, under the necessary conditions, the distribution measurement method based on similarity dynamic compression can also be set in the terminal device.

[0088] Similarly, the computer-readable medium of the present invention stores a computer program thereon, and when the computer program is executed by a processor, the distribution measurement method based on similarity dynamic compression according to an embodiment of the present invention is implemented.

[0089] The following is a comparative experimental analysis of the method provided by the present invention and other existing solutions:

[0090] The experimental setup is as follows:

[0091] Platform: Experiments were conducted on a virtual machine equipped with a quad-core Intel(R) Core(TM) i5-8265U CPU @ 1.60GHz. Each core has three levels of cache: 128KB of L1 data cache, 128KB of L1 instruction cache, 1MB of L2 cache, and 12MB of L3 cache. The operating system used was Ubuntu 22.04.3.

[0092] Dataset: An anonymized real-world dataset collected in 2018 using CAIDA. To facilitate key-value pair detection, the key is a 13-byte complex consisting of the source / destination IP address, port, and protocol, while the value represents the IP packet length. To effectively capture data distribution characteristics and reduce memory usage, the packet length value is divided into 16 equal-length intervals.

[0093] Experimental Method: Our method (hereafter referred to as SC-sketch) is compared with Count-Min Sketch (CM), Counting Bloom Filter (CBF), and HistSketch (HS). The comparison targets are the interval ratio, average area, throughput, and bandwidth of each scheme when using different memory spaces.

[0094] Figure 2 The performance of SC-Sketch in point query tasks is demonstrated. Figure 1 Looking at the interval ratios in (a), SC-Sketch is close to 1 for all memory configurations, indicating that its estimated frequencies are mostly within the error range of the true frequencies. For example, with 0.2MB of memory, SC-Sketch's interval ratio is close to 0.8, while CM and CBF are almost 0, and HS is approximately 0.3. Figure 1 The average relative error (b) shows that the error of SC-Sketch is significantly lower than that of other algorithms, especially under low memory conditions, where its error is only 1 / 5 of CM and about 1 / 3 of CBF and HS, reflecting higher accuracy and stability.

[0095] Figure 3 Depicts the performance of SC-Sketch on the histogram query task. Figure 2 From the key ratio of (a), SC-Sketch leads under all memory conditions, especially when the memory is 0.2MB, where its key ratio is about 0.4, which is much higher than the ratios of CM and CBF close to 0, and HS's 0.1. Figure 2 The average histogram relative error in (b) shows that the error of SC-Sketch is significantly lower than that of other algorithms. For example, with 0.2MB of memory, it is about 50, while CM exceeds 300, and CBF and HS are about 150. When the memory is increased to 2MB, the error of SC-Sketch drops to about 20, while the error of other algorithms is still above 50, highlighting its high-precision advantage in key-value distribution estimation.

[0096] Figure 4 The throughput of SC-Sketch and HistSketch was compared. With 0.4MB of memory, SC-Sketch achieved a throughput of approximately 12 Mips, exceeding HS's 10 Mips. When the memory was increased to 4MB, SC-Sketch's throughput further increased to approximately 22 Mips, compared to approximately 18 Mips for HS. This demonstrates that SC-Sketch, through its dynamic compression mechanism and probabilistic insertion technology, effectively reduces processing latency and improves data processing efficiency. Its processing speed advantage is particularly pronounced when memory is increased. SC-Sketch can better utilize resources when memory is limited, achieving higher throughput.

[0097] Figure 5 The bandwidth efficiency of SC-Sketch and HistSketch was compared. With 0.4MB of memory, SC-Sketch's byte ratio was approximately 0.25, lower than HS's 0.3. When the memory was increased to 4MB, SC-Sketch's byte ratio further decreased to approximately 0.22, while HS's byte ratio was approximately 0.27. SC-Sketch can better optimize resource utilization when memory is limited, achieving higher bandwidth efficiency.

Claims

1. A distribution measurement method based on similarity dynamic compression, characterized in that: When executing a batch of per-key tasks based on the Sketch algorithm, if it is determined that the per-key tasks in this batch have row similarity or column similarity, the corresponding row compression or column compression is performed through the following steps: Step 1: Initialize the Sketch by resetting the count values of all base counters and compression counters to zero. The base counters are set in the base buckets of the Sketch's two-dimensional array, and the compression counters are set on each row and column of the Sketch's two-dimensional array. Step 2: Read the single key-value pair information to be inserted, and then determine the target insertion position of this key-value pair in the two-dimensional array based on the hash function; Step 3: Based on the target insertion position and whether row or column compression is to be performed, check whether the corresponding compression counter is zero. If so, insert the key-value pair into the base bucket of the target insertion position. Otherwise, calculate the probability of insertion based on the value of the compression counter and perform the insertion based on the probability. After the insertion is performed, increment the corresponding base counter by 1 and proceed to step 4. If the insertion is not performed, jump to step 5. Step 4: Check whether the basic counter in the inserted basic bucket reaches the capacity of the basic bucket. If so, divide all the data in the basic bucket of the entire row or column by the preset constant value c, round up the quotient and replace the original data. At the same time, increase the compression counter of the corresponding row or column by 1, and then go to step 5. Otherwise go directly to step 5; Step 5: Check whether all key-value pair insertions for each key task in this batch have been processed. If so, end the method; otherwise, return to step 2 and execute the loop.

2. The method according to claim 1, characterized in that The following steps are used to determine whether the per-key tasks in this batch have row similarity or column similarity: When different values show similar frequency distributions under the same key, then the per-key tasks are considered to have row similarity; When different keys exhibit similar distributions under the same value, the per-key tasks are said to have column similarity.

3. The method according to claim 1, characterized in that In step 3, the probability is calculated based on the value of the compression counter: The probability is calculated according to the following formula: ； Wherein, c is a preset constant value, and b is the value recorded in the compression counter of the corresponding row or column.

4. The method according to claim 3, characterized in that After compression is complete, when a query needs to be executed based on the Sketch algorithm, the following steps are used to restore the compressed data to the original data: ； Where a represents the compressed data stored in the base bucket.

5. A distribution measurement device based on similarity dynamic compression, characterized in that: include: The compression method selection module is used to determine whether the per-key tasks in a batch have row similarity or column similarity when executing them based on the Sketch algorithm, and then perform corresponding row compression or column compression; Initialization module, used to reset the count values of all basic counters and compression counters in Sketch to zero; The basic counter is set in the basic bucket of the two-dimensional array constituting Sketch, and the count is increased by 1 when a key-value pair is inserted. The compression counter is set on each row and each column of the two-dimensional array of Sketch, and the count is increased by 1 when the corresponding row or column is compressed. A key-value pair insertion position determination module is used to read the information of a single key-value pair to be inserted, and then determine the target insertion position of this key-value pair in the two-dimensional array based on a hash function; The key-value pair insertion probability determination module is used to check whether the corresponding compression counter is zero based on the target insertion position and whether row compression or column compression needs to be performed. If so, the key-value pair is inserted into the base bucket of the target insertion position. Otherwise, the insertion probability is calculated based on the value of the compression counter and the insertion is performed based on the probability. After the insertion is performed, the execution is transferred to the compression module. If the insertion is not performed, the execution is transferred to the end module. The compression module is used to check whether the basic counter in the inserted basic bucket has reached the capacity of the basic bucket. If so, it divides all the data in the basic bucket of the entire row or column by the preset constant value c, rounds up the quotient and replaces the original data, and then transfers the execution to the end module; otherwise, it directly transfers the execution to the end module; The end module checks whether the key-value pair insertions of each key task in this batch have been processed. If so, the device operation ends; otherwise, the key-value pair insertion position determination module continues the loop execution.

6. The device according to claim 5, characterized in that The process of the compression mode selection module determining whether the per-key tasks of this batch have row similarity or column similarity includes: When different values show similar frequency distributions under the same key, then the per-key tasks are considered to have row similarity; When different keys exhibit similar distributions under the same value, the per-key tasks are said to have column similarity.

7. The device according to claim 5, characterized in that The key-value pair insertion probability determination module calculates the probability according to the value of the compression counter as follows: The probability is calculated according to the following formula: ； Wherein, c is a preset constant value, and b is the value recorded in the compression counter of the corresponding row or column.

8. The device according to claim 7, characterized in that When executing a query based on the Sketch algorithm, the compressed data is restored to the original data through the data recovery module. The data recovery module performs recovery based on the following formula: ； Where a represents the compressed data stored in the base bucket.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.

10. A computer-readable medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images