Statistical information cardinal number estimation method for distributed database

Abstract

The invention discloses a statistical information cardinal number estimation method for a distributed database. A distributed reservoir sampling algorithm and a cardinal number estimation algorithm are optimized. In the sampling process of the distributed reservoir, after each node performs independent sampling, a sample is sent to a gateway node serving as a data stream processing core. And the gateway node carries out summarizing processing on the samples and carries out further resampling, so that the data stream processing efficiency and the sampling precision are improved. The cardinality estimation adopts an optimized 4-bit cardinality estimation (QuadBit HLL +) algorithm, and memory use is dynamically optimized according to data characteristics through intelligent switching of sparse and dense modes and variable length coding and differential coding technologies. By adjusting the size of the register and adopting a method of combining a basic register and an offset register, the memory use is reduced. In addition, xxhash is adopted as a hash function of the algorithm, so that the efficiency and the accuracy of cardinality estimation are enhanced.

Description

Technical Field

[0001] The invention belongs to the technical field of databases, and in particular to an optimized method for estimating the cardinality of database statistical information. Background Art

[0002] In the current data-driven era, with the rapid development of technologies such as the Internet, the Internet of Things, and social media, the amount of data has exploded. According to IDC's forecast, by 2025, the amount of global data will reach 175ZB. In this context, efficient management and processing of data has become a hot topic in scientific research and industry. As the core technology for managing, processing, and analyzing large-scale data sets, the performance optimization of database management systems (DBMS) is critical.

[0003] In modern database management systems, statistical information mainly refers to data descriptions related to database tables, such as the number of rows in the table, the distribution of column values, the number of unique values ​​(cardinality), index usage, etc. These statistical information are used by the database query optimizer to evaluate the cost of different query strategies and select the most effective query execution plan. Accurate and efficient collection of statistical information is crucial to optimizing query execution and resource allocation.

[0004] Data sampling is a commonly used method to efficiently extract representative data subsets from large data sets to reduce processing time and resource consumption. In database systems, histograms and samples are typical products of sampling, which provide an overview of data distribution and help database management systems (DBMS) perform more accurate query plans and resource management. However, traditional data sampling methods such as simple random sampling are often difficult to adapt to rapidly changing data streams and large-scale distributed database environments. For example, although the reservoir sampling algorithm can perform equal probability sampling when the total data volume is unknown, it is often difficult to maintain the global representativeness of the data in a distributed environment because the data is usually distributed on different physical nodes.

[0005] Cardinality estimation refers to estimating the number of unique values ​​in a column or dataset in a database, which directly affects the efficiency and accuracy of database management systems in terms of execution plan selection, data storage, and index construction. HyperLogLog (HLL) is a cardinality estimation algorithm that is widely used in many commercial database systems and is favored for its high estimation accuracy and low memory requirements. Nevertheless, when dealing with large-scale datasets, HLL and its variants (such as HyperLogLog++) still have the problems of high memory usage and low accuracy. Especially when the data is sparse or non-uniformly distributed, the performance of these algorithms may drop significantly.

[0006] Therefore, the existing technology needs a new method that can adapt to dynamic data flows, support distributed environments, and provide high-precision cardinality estimation under low memory conditions. Such a method should be able to solve the challenges of data sampling and cardinality estimation in modern database systems, improve the efficiency and accuracy of statistical information collection, and thus optimize the overall performance and resource utilization of the database. Summary of the invention

[0007] In view of the deficiencies in the prior art, the present invention proposes a statistical information cardinality estimation method for a distributed database. The method of the present invention improves the efficiency and accuracy of statistical information collection and enhances the overall performance of the database.

[0008] In order to solve the above technical problem, the present invention proposes a statistical information cardinality estimation method for a distributed database, comprising the following steps:

[0009] S1, distributed water tank sampling, including initialization, local sampling and global sampling;

[0010] S2, 4-bit cardinality statistics algorithm, including mode definition and algorithm initialization, intelligent switching between sparse mode and dense mode, adjustment of the size of register M in dense mode, and calculation and return of the final cardinality estimation value according to the current mode;

[0011] The specific content of step S1 is as follows:

[0012] 1-1) Initialization: In the statistical information collection module of the distributed database system, a reservoir R of size K is allocated to each node i i , where K only represents the number of data samples stored locally on each node i;

[0013] 1-2) Local sampling: The data item x is obtained through the storage interface in the statistical information collection module. Each node i independently samples the read data and determines the reservoir R i The current size of the data item x is calculated as follows:

[0014] 1-2-1) If the water reservoir R i If the current size of is less than K, then directly add the data item x to the reservoir R i middle;

[0015] 1-2-2) If the water reservoir R i If the current size of the pool is K, a random integer j is generated from 1 to the total number of data items currently processed. If j ≤ K, the pool R i The j-th element in is replaced by the current data item x;

[0016] 1-3) Global sampling: the water reservoirs R1, R2, ..., R of all nodes n will be merged into a global reservoir R g Next, if the global reservoir R g If the size of > K, repeat 1-2-2) to the global reservoir R g Until the size of the global reservoir R = K, g Contains sample sets from the entire distributed database system;

[0017] The specific content of step S2 is as follows:

[0018] 2-1) Mode definition and algorithm initialization:

[0019] In the 4-bit cardinality statistics algorithm, there are dense mode and sparse mode, wherein: the dense mode algorithm is used to process the situation where the memory required for the cardinality estimation of the data set reaches the maximum memory threshold, at which time the algorithm allocates and maintains a complete register array; the sparse mode is used to process the situation where the memory required for the cardinality estimation of the data set is lower than the maximum memory threshold used by the dense mode, at which time the algorithm only records the register index actually touched and the corresponding count value;

[0020] In the 4-bit base statistical algorithm, xxhash is selected as the hash function of the algorithm;

[0021] In the 4-bit cardinality statistics algorithm, the algorithm parameters are set as follows:

[0022] sparse: whether to use sparse representation; p: precision in dense mode; p′: precision in sparse mode, p′>p; m: number of buckets; α m : used for deviation correction; tmpSet: temporary set; sparseList: array that can be stored in sparse representation, recorded as sparse list; M: register; b: basic register; capacity: maximum capacity of offset register;

[0023] 2-2) Intelligent switching between sparse mode and dense mode and adjustment of the size of register M in dense mode:

[0024] For each element ν in the sample set obtained in step S1, the following operations are performed until the operations on the element are completed:

[0025] 2-2-1) Read an element in the sample set;

[0026] 2-2-2) Use the hash function xxhash to calculate the hash value h(ν) of the current element;

[0027] If sparse == false, use dense mode and execute step 2-2-3);

[0028] If sparse == true, use sparse mode and execute step 2-2-4)

[0029] 2-2-3) Adjust the size of register M, thereby updating the corresponding maximum leading zeros in register M. The process is as follows:

[0030] First, initialize the base register b and all registers M n is 0, where the value range of n is [0,m);

[0031] From the hash value h(ν), extract the precision p bits in dense mode, calculate the index n of the register, and update the register corresponding to the index n; extract the remaining bits except the precision p bits and calculate the maximum leading zeros;

[0032] If the maximum leading zero zeros minus the base register b is greater than or equal to the maximum capacity of the offset register capacity, an offset overflow is detected; replace the capacity of the base register b with the minimum value among all registers M, and update the capacity of all registers M according to the current capacity of the base register b; execute steps 2-3);

[0033] If the largest leading zero zeros minus the base register b is less than the maximum capacity of the offset register, the largest leading zero zeros are added to register M. n Compare and take the larger value between the two to update M n ;

[0034] Read the next element in the sample set; return to step 2-2-2);

[0035] 2-2-4) compress the touched register index and the corresponding count value through variable length coding and differential coding and add them to the temporary set tmpSet;

[0036] If the size of the temporary set tmpSet exceeds the threshold 4m / 100, a merge operation is performed to sort and merge into the sparse list sparseList;

[0037] Determine whether the size of the sparse list sparseList exceeds the memory usage threshold of 4m bits. If the condition is met, switch from sparse mode to dense mode and convert the sparse list sparseList into a normal register array M; otherwise, read the next element in the sample set and return to step 2-2-2);

[0038] Repeat the above steps 2-2) to 2-4) until the operations for all elements in the sample set are completed;

[0039] 2-3) Calculate and return the final cardinality estimate based on the current mode, in one of the following situations:

[0040] First, the current mode is sparse mode, using linear counting method to calculate the cardinality estimate

[0041]

[0042] Where m is the total number of registers, V is the number of registers that do not record any elements during the counting process of register M;

[0043] Second, the current mode is dense mode, and the cardinality estimate is calculated using the following formula

[0044]

[0045] Where m is the total number of registers, α m It is a correction factor that depends on m to correct the deviation. The value range of m is 2 4 ~2 18 ; m = 2 4 When α m =0.673; m=2 5 When α m =0.679; m=2 6 When α m =0.709; m=2 7 ~2 18 , α m =0.7213 / (1+1.079 / m).

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] (1) In view of the lack of adaptability and accuracy of statistical information sampling methods in processing large-scale complex data, in the distributed reservoir sampling link, reservoir sampling is performed independently at each node, and then the samples are sent to the gateway node as the core of data flow processing. The gateway node aggregates the samples and performs further resampling, thereby improving the overall data flow processing efficiency and sampling accuracy.

[0048] (2) To address the high memory usage and low accuracy of traditional cardinality estimation algorithms, an optimized cardinality estimation algorithm, the QuadBit HLL+ algorithm, is used. Through intelligent switching between sparse mode and dense mode and variable length coding and differential coding technology, memory usage is dynamically optimized according to data characteristics. In the cardinality estimation algorithm, the size of registers is adjusted and a combination of base registers and offset registers is used to reduce memory usage. In addition, the hash function of the algorithm uses xxhash, which enhances the efficiency and accuracy of cardinality estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a timing diagram of a statistical information collection module involved in the method of the present invention;

[0050] Figure 2 It is a design diagram of a sampling and cardinality estimation module involved in the method of the present invention;

[0051] Figure 3 is a flow chart of the distributed water storage tank sampling algorithm in the present invention;

[0052] Figure 4 It is a flow chart of the 4-bit cardinality estimation algorithm in the present invention. DETAILED DESCRIPTION

[0053] The design concept of a statistical information cardinality estimation method for a distributed database proposed by the present invention is: by independently performing reservoir sampling at each node, and then sending the sample to the gateway node as the core of data flow processing. The gateway node performs sample aggregation processing and performs further resampling, thereby improving the processing efficiency of the overall data flow and the accuracy of sampling. The present invention adopts an optimized cardinality estimation algorithm (QuadBit HLL+ algorithm), and dynamically optimizes memory usage according to data characteristics through intelligent switching between sparse mode and dense mode and variable length coding and differential coding technology. By adjusting the size of the register, a method combining the basic register and the offset register is adopted to reduce memory usage. In addition, the hash function of the algorithm adopts xxhash, which enhances the efficiency and accuracy of cardinality estimation. In summary, the method of the present invention improves the collection efficiency and accuracy of statistical information in the database system through improved data sampling technology and cardinality estimation algorithm, improves the overall performance of the database, and thus optimizes the query performance and resource management of the database.

[0054] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, but the following embodiments are by no means intended to limit the present invention in any way.

[0055] The present invention proposes a statistical information cardinality estimation method for a distributed database, wherein the distributed database system includes a statistical information collection module, and the block diagram of the statistical information collection module is as follows Figure 1As shown, the basic components of the statistical information collection module include a connector in a distributed database, a distributed plan generator, an operator processor, a sample aggregator, a sampler, a distributed reservoir sampling component and a 4-bit cardinality estimation component.

[0056] like Figure 1 As shown, the connector is responsible for handling connections and requests from the client and starting the task of collecting statistical information. The distributed plan generator creates an execution plan, formulates a plan based on the collection requirements, and assigns tasks to different nodes. The operator processor performs specific statistical information collection operations. The collector performs actual data sampling work to obtain data samples and statistical information. The sample aggregator is responsible for collecting statistical data from each node and summarizing it. The distributed reservoir sampling component is used to collect and store data samples. The 4-bit cardinality estimation component uses an optimized 4-bit cardinality estimation algorithm for cardinality estimation.

[0057] like Figure 2 As shown, in the statistical information collection module, the sampler is the core of data sampling, responsible for directly interacting with the data source, sampling necessary data samples, and integrating the cardinality processing function.

[0058] mainLoop(): controls the continuous execution of data sampling.

[0059] SampleRow(): Calls the reservoir sampling algorithm.

[0060] InsertHash(): Implements the cardinality estimation algorithm.

[0061] The distributed water reservoir sampling component is used to temporarily store the data samples extracted from the sampler to support further analysis and processing.

[0062] GetSamples(): Provides an interface to access stored samples.

[0063] The 4-bit cardinality estimation component uses an optimized algorithm to estimate the cardinality of data.

[0064] InsertHash(): Encode the data into a hash value and process it.

[0065] Estimate(): Estimate the cardinality of the current data.

[0066] The sample aggregator aggregates data from various sampling nodes to obtain global statistical information.

[0067] mainLoop(): Main loop, controls the execution of data aggregation.

[0068] SampleRow(): Receives sample data from Sampler.

[0069] Estimate(): Calls the cardinality estimation algorithm to estimate the cardinality of the data.

[0070] WriteResults(): Outputs the final statistical information to support the query optimizer's decision making.

[0071] like Figure 1 and Figure 2 As shown, after the connector receives the statistical information collection request from the client, it calls the distributed plan generator to generate an execution plan and sends it to a specific computing node. Start the sample aggregator and sampler to perform data sampling. The sampler continuously extracts samples from the data source and transmits the data to the distributed reservoir sampling component for storage. At the same time, data is read from the data source and sent to the 4-bit cardinality estimation component for cardinality estimation to calculate the cardinality of the data. The sample aggregator collects the data returned by each node and summarizes it. The processed statistical information will be persistently stored. All components work together to ensure the efficiency and accuracy of the entire process, from receiving data collection requests to executing specific data sampling and cardinality estimation.

[0072] Figure 3 The following is an example of a data sampling system with five nodes. Figure 1 The statistics collection module shown and Figure 2 The distributed reservoir sampling component and the 4-bit cardinality estimation component are closely related, and together constitute the implementation details of the statistical information cardinality estimation method for distributed databases optimized by the present invention. Figure 3 Detailed description of each part shown in:

[0073] 1-1) Initialization, the data sampling system consists of five nodes (Node1 to Node5), each of which includes a data storage unit (data1 to data5). These data storage units are responsible for storing their own data sets, and each storage unit is connected to a sampler (Sampler). These samplers directly extract samples from the corresponding data storage unit. Figure 2 The functions and operations of the sampler components in the same way are consistent. For each node i (Node1 to Node5), a reservoir R of size K is assigned. i , where K only represents the number of data samples stored locally on each node i;

[0074] 1-2) Local sampling. The core function of each sampler is to execute the distributed reservoir sampling algorithm. The data item x is obtained through the data storage unit (data1 to data5) in the statistical information collection module. Each node i (Node1 to Node5) independently samples the read data and determines the reservoir R. i The current size of the data item x is calculated as follows:

[0075] 1-2-1) If the water reservoir R i If the current size of is less than K, then directly add the data item x to the reservoir R i middle;

[0076] 1-2-2) If the water reservoir R i If the current size of the pool is K, a random integer j is generated from 1 to the total number of data items currently processed. If j ≤ K, the pool R i The j-th element in is replaced by the current data item x;

[0077] 1-3) Global sampling: the samples output by the samplers of all nodes are collected into the central sample aggregator. This process is similar to Figure 1 The main function of the sample aggregator is to integrate all collected samples and perform necessary processing or analysis to support the final data decision and query optimization. The reservoirs R1, R2, ..., R5 of all nodes (Node1 to Node5) will be merged into a global reservoir R g Next, if the global reservoir R g If the size of > K, repeat 1-2-2) in 1-2) to the global reservoir R g Until the size of the global reservoir R = K, g Contains sample sets from the entire distributed database system;

[0078] After the sample aggregator completes processing, it outputs a response, which is a histogram generated based on the collected data to reflect the data distribution.

[0079] With respect to the above-mentioned statistical information collection module, the present invention optimizes the algorithms in the distributed reservoir sampling component and the 4-bit cardinality estimation component, and proposes a statistical information cardinality estimation method for distributed databases, such as Figure 4 As shown, the algorithm includes two parts: a distributed water tank sampling algorithm and a 4-bit cardinality statistical algorithm.

[0080] Part I: Distributed water tank sampling, including initialization, local sampling and global sampling; the specific contents are as follows:

[0081] Step 1-1) Initialization: In the statistical information collection module of the distributed database system, a reservoir R of size K is allocated to each node i. i , where K only represents the number of data samples stored locally on each node i;

[0082] Step 1-2) Local sampling: Obtain data item x through the storage interface in the statistical information collection module, and each node i independently samples the read data and determines the reservoir R i The current size of the data item x is calculated as follows:

[0083] 1-2-1) If the water reservoir R i If the current size of is less than K, then directly add the data item x to the reservoir R i middle;

[0084] 1-2-2) If the water reservoir R i If the current size of the pool is K, a random integer j is generated from 1 to the total number of data items currently processed. If j ≤ K, the pool R i The j-th element in is replaced by the current data item x;

[0085] Step 1-3) Global sampling: the water reservoirs R1, R2, ..., R of all nodes n will be merged into a global reservoir R g Next, if the global reservoir R g If the size of > K, repeat 1-2-2) in 1-2) to the global reservoir R g Until the size of the global reservoir R = K, g Contains samples from across the distributed database system set.

[0086] The second part, the 4-bit cardinality statistical algorithm is Figure 1 Statistics collection module and Figure 2 It is the core part of the 4-bit cardinality estimation component and is tightly integrated with these components to form a comprehensive statistical information processing framework.

[0087] A 4-bit cardinality statistics algorithm includes mode definition and algorithm initialization, intelligent switching between sparse mode and dense mode, adjustment of the size of register M in dense mode, and calculation and return of the final cardinality estimation value according to the current mode.

[0088] Step 2-1) Mode definition and algorithm initialization:

[0089] In the 4-bit cardinality statistics algorithm, sparse mode and dense mode are used, where: the sparse mode is used to handle the situation where the memory required for the cardinality estimation of the data set is lower than the maximum memory threshold used by the dense mode. At this time, the algorithm only records the register index actually touched and the corresponding count value; the dense mode is used to handle the situation where the memory required for the cardinality estimation of the data set reaches the maximum memory threshold. At this time, the algorithm allocates and maintains a complete register array;

[0090] In the 4-bit base statistical algorithm, xxhash is selected as the hash function of the algorithm.

[0091] In the 4-bit cardinality statistics algorithm, the algorithm parameters are set as follows:

[0092] sparse: whether to use sparse representation; p: precision in dense mode; p′: precision in sparse mode, p′>p; m: number of buckets; α m : used for deviation correction; tmpSet: temporary set; sparseList: array that can be stored in sparse representation, recorded as sparse list; M: register; b: basic register; capacity: maximum capacity of offset register.

[0093] Step 2-2) Intelligent switching between sparse mode and dense mode, where the dense mode requires adjustment of the size of register M. In this step, for each element ν in the sample set obtained in the first part, the following operations are performed until the operations on all elements in the sample set are completed.

[0094] 2-2-1) First, read an element in the sample set.

[0095] 2-2-2) Use hash function xxhash to calculate the hash value h(ν) of the current element; if sparse == false, use dense mode and execute step 2-2-3); if sparse == true, use sparse mode and execute step 2-2-4).

[0096] 2-2-3) The size of register M is adjusted, so as to update the corresponding maximum leading zeros in register M. The process is as follows:

[0097] First, initialize the base register b and all registers M n is 0, where the value range of n is [0,m);

[0098] From the hash value h(ν), extract the precision p bits in dense mode, calculate the index n of the register, and update the register corresponding to the index n; extract the remaining bits except the precision p bits and calculate the maximum leading zeros;

[0099] If the maximum leading zero zeros minus the base register b is greater than or equal to the maximum capacity of the offset register capacity, an offset overflow is detected; replace the capacity of the base register b with the minimum value among all registers M, and update the capacity of all registers M according to the current capacity of the base register b; execute steps 2-3);

[0100] If the largest leading zero zeros minus the base register b is less than the maximum capacity of the offset register, the largest leading zero zeros are added to register M. n Compare and take the larger value between the two to update M n ; Read the next element in the sample set; Return to step 2-2-2);

[0101] 2-2-4) Intelligent switching between sparse mode and dense mode:

[0102] Compress the touched register index and the corresponding count value through variable length coding and differential coding and add them to the temporary set tmpSet; if the size of the temporary set tmpSet exceeds the threshold 4m / 100, perform a merge operation, sort and merge into the sparse list sparseList; determine whether the size of the sparse list sparseList exceeds the memory usage threshold 4m bits. If the condition is met, switch from sparse mode to dense mode and convert the sparse list sparseList into a normal register array M; otherwise, read the next element in the sample set and return to step 2-2-2);

[0103] 2-3) Calculate and return the final cardinality estimate based on the current mode, in one of the following situations:

[0104] First, the current mode is sparse mode, using linear counting method to calculate the cardinality estimate

[0105]

[0106] Where m is the total number of registers, V is the number of registers that do not record any elements during the counting process of register M;

[0107] Second, the current mode is dense mode, and the cardinality estimate is calculated using the following formula

[0108]

[0109] Where m is the total number of registers, α m It is a correction factor that depends on m to correct the deviation. The value range of m is 2 4 ~2 18 ; m = 2 4When α m =0.673; m=2 5 When α m =0.679; m=2 6 When α m =0.709; m=2 7 ~2 18 , α m =0.7213 / (1+1.079 / m).

[0110] The 4-bit cardinality estimation algorithm proposed in the present invention is experimentally verified, and compared with the traditional cardinality estimation algorithm in terms of memory usage and accuracy, as shown in Tables 1 and 2, where the test data comes from the LiveJournal social network dataset.

[0111] Table 1 Comparison of memory usage of different cardinality estimation algorithms

[0112]

[0113]

[0114] Table 2 Comparison of accuracy of different cardinality estimation algorithms

[0115]

[0116] Experimental results show that in terms of memory usage, the 4-bit radix estimation algorithm reduces about 50% of the memory usage compared with the traditional HLL algorithm and the traditional HLL++ algorithm, and optimizes memory usage; in terms of accuracy, the 4-bit radix estimation algorithm is also improved compared with the traditional HLL algorithm and HLL++ algorithm, especially when the cardinality is small.

[0117] Although the present invention has been described above in conjunction with the accompanying drawings, the present invention is not limited to the above-mentioned specific embodiments, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can make many improvements and changes without departing from the purpose of the present invention, which are all within the protection of the present invention.

Claims

1. A statistical information cardinality estimation method for a distributed database, characterized in that: The following steps are involved: S1. Distributed water storage tank sampling, including: 1-1) Initialization: In the statistical information collection module of the distributed database system, a reservoir R of size K is allocated to each node i i , where K only represents the number of data samples stored locally on each node i; 1-2) Local sampling: The data item x is obtained through the storage interface in the statistical information collection module. Each node i independently samples the read data and determines the reservoir R i The current size of the data item x is calculated as follows: 1-2-1) If the water reservoir R i If the current size of is less than K, then directly add the data item x to the reservoir R i middle; 1-2-2) If the water reservoir R i If the current size of the pool is K, a random integer j is generated from 1 to the total number of data items currently processed. If j ≤ K, the pool R i The j-th element in is replaced by the current data item x; 1-3) Global sampling: the water reservoirs R1, R2, ..., R of all nodes n will be merged into a global reservoir R g Next, if the global reservoir R g If the size of > K, repeat steps 1-2-2) to the global reservoir R g Until the size of the global reservoir R = K, g Contains sample sets from the entire distributed database system; S2, 4-bit cardinality statistics algorithm, including: 2-1) Mode definition and algorithm initialization, including: In the 4-bit cardinality statistics algorithm, there are dense mode and sparse mode, where: The dense mode algorithm is used to handle the situation where the memory required for the cardinality estimation of the data set reaches the maximum memory threshold. In this case, the algorithm allocates and maintains a complete register array; The sparse mode is used to handle the situation where the memory required for the cardinality estimation of the data set is lower than the maximum memory threshold used by the dense mode. In this case, the algorithm only records the register index actually touched and the corresponding count value; In the 4-bit base statistical algorithm, xxhash is selected as the hash function of the algorithm; In the 4-bit cardinality statistics algorithm, the algorithm parameters are set as follows: sparse: whether to use sparse representation; p: precision in dense mode; p′: precision in sparse mode, p′>p; m: the number of buckets; α m : Used for deviation correction; tmpSet: temporary set; sparseList: an array that can be stored in a sparse representation, denoted as a sparse list; M: register; b: base register; capacity: maximum capacity of the offset register; 2-2) Intelligent switching between sparse mode and dense mode and adjustment of the size of register M in dense mode, For each element ν in the sample set obtained in step S1, the following operations are performed: 2-2-1) Read an element in the sample set; 2-2-2) Use the hash function xxhash to calculate the hash value h(ν) of the current element; If sparse == false, use dense mode and execute step 2-2-3); If sparse == true, use sparse mode and execute step 2-2-4) 2-2-3) Adjust the size of register M, thereby updating the corresponding maximum leading zeros in register M. The process is as follows: First, initialize the base register b and all registers M n is 0, where the value range of n is [0,m); From the hash value h(ν), extract the precision p bits in dense mode, calculate the index n of the register, and update the register corresponding to the index n; extract the remaining bits except the precision p bits and calculate the maximum leading zeros; If the maximum leading zero zeros minus the base register b is greater than or equal to the maximum capacity of the offset register capacity, an offset overflow is detected; replace the capacity of the base register b with the minimum value among all registers M, and update the capacity of all registers M according to the current capacity of the base register b; execute steps 2-3); If the largest leading zero zeros minus the base register b is less than the maximum capacity of the offset register, the largest leading zero zeros are added to register M. n Compare and take the larger value between the two to update M n ; Read the next element in the sample set; return to step 2-2-2); 2-2-4) compress the touched register index and the corresponding count value through variable length coding and differential coding and add them to the temporary set tmpSet; If the size of the temporary set tmpSet exceeds the threshold 4m / 100, a merge operation is performed to sort and merge into the sparse list sparseList; Determine whether the size of the sparse list sparseList exceeds the memory usage threshold of 4m bits. If the condition is met, switch from sparse mode to dense mode and convert the sparse list sparseList into a normal register array M; otherwise, read the next element in the sample set and return to step 2-2-2); Repeat the above steps 2-2-2) to 2-2-4) until the operations on all elements in the sample set are completed; 2-3) Calculate and return the final cardinality estimate based on the current mode, in one of the following situations: First, the current mode is sparse mode, using linear counting method to calculate the cardinality estimate Where m is the total number of registers, V is the number of registers that do not record any elements during the counting process of register M; Second, the current mode is dense mode, and the cardinality estimate is calculated using the following formula Where m is the total number of registers, α m It is a correction factor that depends on m to correct the deviation. The value range of m is 2 4 ~2 18 ; m = 2 4 When α m =0.673; m=2 5 When α m =0.679; m=2 6 When α m =0.709; m=2 7 ~2 18 , α m =0.7213 / (1+1.079 / m).

Images