A statistical information cardinality estimation method for distributed databases

Abstract

The application discloses a statistical information cardinality estimation method for a distributed database, and optimizes a distributed reservoir sampling algorithm and a cardinality estimation algorithm. In the distributed reservoir sampling process, each node independently samples, and then sends the sample to a gateway node as a data stream processing core. The gateway node performs sample aggregation processing and further resampling, thereby improving data stream processing efficiency and sampling accuracy. The cardinality estimation adopts an optimized QuadBit HLL+ algorithm, intelligently switches between sparse and dense modes, and uses variable-length coding and differential coding technology, so that memory use is dynamically optimized according to data characteristics. By adjusting the size of the register, the method combining the basic register and the offset register is used to reduce the memory use. In addition, the hash function of the algorithm adopts xxhash, thereby enhancing the efficiency and accuracy of the cardinality estimation.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of databases, and particularly relates to an optimized database statistical information cardinality estimation method. BACKGROUND

[0002] In the current data-driven era, with the rapid development of technologies such as the Internet, the Internet of Things and social media, the amount of data is showing explosive growth. According to the prediction of IDC, the global data volume will reach 175 ZB by 2025. Under such a background, the efficient management and processing of data have become the focus of attention in the scientific research and industrial circles. As the core technology for managing, processing and analyzing large-scale data sets, the performance optimization of a database management system (DBMS) is very crucial.

[0003] In a modern database management system, statistical information mainly refers to data descriptions related to a database table, such as the number of rows of the table, the distribution of column values, the number of unique values (cardinality), the usage of indexes and the like. These statistical information is used by a database query optimizer to evaluate the cost of different query strategies, so as to select the most effective query execution plan, and accurate and efficient collection of statistical information is crucial for optimizing query execution and resource allocation.

[0004] Data sampling is a commonly used method for efficiently extracting a representative subset of data from a large data set to reduce processing time and resource consumption. In a database system, histograms and samples are typical products of sampling, which provide an overview of the data distribution, helping a database management system (DBMS) to make more accurate query plans and resource management. However, traditional data sampling methods such as simple random sampling often fail to adapt to rapidly changing data streams and large-scale distributed database environments. For example, although the reservoir sampling algorithm can perform equal-probability sampling in the case of unknown total data volume, it is often difficult to maintain the global representativeness of data in a distributed environment, because data is usually distributed on different physical nodes.

[0005] Cardinality estimation refers to estimating the number of non-repeating values in a column or data set in a database, which directly affects the efficiency and accuracy of a database management system in aspects such as execution plan selection, data storage and index construction. HyperLogLog (HLL) is a cardinality estimation algorithm widely used in many commercial database systems, which is favored for its high estimation accuracy and low memory requirement. However, when dealing with large-scale data sets, HLL and its variants (such as HyperLogLog++) still have the problems of high memory occupation and low accuracy. Especially in the case of sparse or non-uniformly distributed data, the performance of these algorithms may decrease significantly.

[0006] Therefore, existing technologies require a new method that can adapt to dynamic data streams, support distributed environments, and provide high-precision cardinality estimation under low memory conditions. Such a method should address the challenges of data sampling and cardinality estimation in modern database systems, improve the efficiency and accuracy of statistical information collection, and thus optimize the overall performance and resource utilization of the database. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention proposes a statistical information cardinality estimation method for distributed databases. This method improves the efficiency and accuracy of statistical information collection and enhances the overall performance of the database.

[0008] To address the aforementioned technical problems, this invention proposes a method for estimating the cardinality of statistical information for distributed databases, comprising the following steps:

[0009] S1. Distributed water storage tank sampling, including initialization, local sampling and global sampling;

[0010] The S2, 4-bit radix statistics algorithm includes mode definition and algorithm initialization, intelligent switching between sparse and dense modes, adjustment of register M size in dense mode, and calculation and return of the final radix estimate based on the current mode.

[0011] The specific content of step S1 is as follows:

[0012] 1-1) Initialization: In the statistical information collection module of the distributed database system, a reservoir R of size K is allocated to each node i. i Where K represents only the number of data samples stored locally on each node i;

[0013] 1-2) Local sampling: Data item x is obtained through the storage interface in the statistical information collection module. Each node i independently samples the read data and determines the water storage tank R. i Given the current size, perform the following operation for each data item x:

[0014] 1-2-1) If the reservoir R i If the current size is less than K, then add the data item x directly to the reservoir R. i middle;

[0015] 1-2-2) If the reservoir R i If the current size is K, then generate a random integer j ranging from 1 to the total number of data items being processed. If j ≤ K, then the reservoir R will be... i Replace the j-th element in the data with the current data item x;

[0016] 1-3) Global Sampling: the reservoirs R1, R2,..., R n will be merged into one global reservoir R g Next, if the size of the global reservoir R g > K, then repeat 1-2-2) until the size of the global reservoir R g = K, which contains the sample set from the whole distributed database system. g

[0017] The specific content of the step S2 is as follows:

[0018] 2-1) Mode definition and algorithm initialization:

[0019] In the 4-bit radix counting algorithm, including dense mode and sparse mode, wherein: the dense mode algorithm is used to process the case where the memory required for data set cardinality estimation reaches the maximum memory threshold, at which time the algorithm allocates and maintains a complete register array; the sparse mode is used to process the case where the memory required for data set cardinality estimation is lower than the maximum memory threshold used by the dense mode, at which time the algorithm only records the actual touched register index and the corresponding count value;

[0020] In the 4-bit radix counting algorithm, xxhash is selected as the hash function of the algorithm;

[0021] In the 4-bit radix counting algorithm, the algorithm parameters are set as follows:

[0022] sparse: whether to use sparse representation; p: precision in dense mode; p': precision in sparse mode, p' > p; m: number of buckets; a m : used for bias correction; tmpSet: temporary set; sparseList: array that can be stored in sparse representation, denoted as sparse list; M: register; b: basic register; capacity: maximum capacity of offset register;

[0023] 2-2) Intelligent switching of sparse mode and dense mode and adjustment of register M size in dense mode:

[0024] For each element v in the sample set obtained in step S1, the following operations are performed until the operation of the element is completed:

[0025] 2-2-1) Read an element in the sample set;

[0026] 2-2-2) Calculate the hash value h(v) of the current element using the hash function xxhash;

[0027] ​If sparse==false, dense mode is adopted, and step 2-2-3) is executed;

[0028] If sparse==true, sparse mode is adopted, and step 2-2-4) is executed

[0029] 2-2-3) Adjustment of the size of the register M is performed to update the corresponding maximum leading zero zeros in the register M, and the process is as follows:

[0030] First, initialize the base register b and all registers M n to 0, where n ranges from 0 to m-1.

[0031] From the hash value h(v), extract the precision p bits in the dense mode, calculate the index n of the register, and update the register corresponding to the index n; extract the remaining bits other than the precision p bits, and calculate the maximum leading zero zeros;

[0032] If the maximum leading zero zeros minus the base register b is greater than or equal to the maximum capacity of the offset register capacity, an offset overflow is detected; replace the capacity of the base register b with the minimum value in all registers M, and update the capacity of all registers M according to the current capacity of the base register b; execute step 2-3);

[0033] If the maximum leading zero zeros minus the base register b is less than the maximum capacity of the offset register capacity, compare the maximum leading zero zeros with the register M n , and update M n with the larger value between them;

[0034] Read the next element in the sample set; return to step 2-2-2);

[0035] 2-2-4) The touched register index and the corresponding count value are compressed by variable-length encoding and differential encoding and added to the temporary set tmpSet;

[0036] If the size of the temporary set tmpSet exceeds the threshold 4m / 100, perform the merging operation, sort and merge into the sparse list sparseList;

[0037] Determine whether the size of the sparse list sparseList exceeds the memory usage threshold 4m bits. If the condition is met, switch from the sparse mode to the dense mode, and convert the sparse list sparseList to a normal register array M; otherwise, read the next element in the sample set and return to step 2-2-2);

[0038] Repeat the above steps 2-2) to step 2-4) until the operation of all elements in the sample set is completed;

[0039] 2-3) Calculate and return the final cardinality estimate according to the current mode, one of the following cases:

[0040] First, the current mode is sparse mode, and the cardinality estimate is calculated using linear counting method

[0041]

[0042] Where m is the total number of registers, V is the number of registers in register M that do not record any elements in the counting process;

[0043] Second, the current mode is dense mode, and the cardinality estimate is calculated using the following formula

[0044]

[0045] Where m is the total number of registers, and alpha m is a correction coefficient dependent on m to correct the bias, the value range of m is 2 4 ~ 2 18 ; when m = 2 4 , alpha m = 0.673; when m = 2 5 , alpha m = 0.679; when m = 2 6 , alpha m = 0.709; when m = 2 7 ~ 2 18 , alpha m = 0.7213 / (1+1.079 / m).

[0046] Compared with the prior art, the beneficial effects of the present application are:

[0047] (1) For the adaptability and accuracy problems of the statistical information sampling method in processing large-scale complex data, in the distributed reservoir sampling link, the reservoir sampling is independently executed at each node, and then the samples are sent to the gateway node as the core of data stream processing. The gateway node performs sample aggregation processing and further resampling, thereby improving the processing efficiency and sampling accuracy of the overall data stream.

[0048] (2) In order to solve the problems of high memory occupation and low accuracy of the traditional cardinality estimation algorithm, an optimized cardinality estimation algorithm, namely QuadBit HLL+ algorithm, is adopted, the intelligent switching of sparse mode and dense mode and the variable length coding and differential coding technology are used, and the memory usage is dynamically optimized according to the data characteristics. In the cardinality estimation algorithm, the size of the register is adjusted, the method of combining the basic register and the offset register is adopted, and the memory usage is reduced. In addition, the hash function of the algorithm adopts xxhash, and the efficiency and accuracy of the cardinality estimation are enhanced. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 is the timing diagram of the statistical information collection module involved in the method of the application;

[0050] Figure 2 is the design diagram of the sampling and cardinality estimation module involved in the method of the application;

[0051] Figure 3 is the flow chart of the distributed reservoir sampling algorithm in the application;

[0052] Figure 4 is the flow chart of the 4-bit cardinality estimation algorithm in the application. DETAILED DESCRIPTION

[0053] The design concept of the statistical information cardinality estimation method for the distributed database provided by the application is that: the reservoir sampling is independently performed at each node, and then the samples are sent to the gateway node as the core of the data stream processing. The gateway node performs the summary processing of the samples and performs further resampling, so as to improve the processing efficiency and the sampling accuracy of the overall data stream. The application adopts the optimized cardinality estimation algorithm (QuadBit HLL+ algorithm), the intelligent switching of sparse mode and dense mode and the variable length coding and differential coding technology are used, and the memory usage is dynamically optimized according to the data characteristics. The size of the register is adjusted, the method of combining the basic register and the offset register is adopted, and the memory usage is reduced. In addition, the hash function of the algorithm adopts xxhash, and the efficiency and accuracy of the cardinality estimation are enhanced. In summary, the method of the application improves the collection efficiency and accuracy of the statistical information in the database system by improving the data sampling technology and the cardinality estimation algorithm, improves the overall performance of the database, and optimizes the query performance and resource management of the database.

[0054] The application will be further described below in combination with the drawings and specific embodiments, but the following embodiments are absolutely not any limitation on the application.

[0055] The application provides a statistical information cardinality estimation method for a distributed database, wherein the distributed database system comprises a statistical information collection module, and a block diagram of the statistical information collection module is as shown in Figure 1As shown, the basic components of the statistics collection module include a connector, a distributed plan generator, an operator processor, a sample aggregator, a sampler, a distributed reservoir sampling component, and a 4-bit cardinality estimation component in a distributed database.

[0056] As shown, the connector is responsible for handling connections and requests from clients and initiating the tasks of statistics collection. The distributed plan generator creates execution plans, formulates plans according to collection requirements, and assigns tasks to different nodes. The operator processor then performs specific statistics collection operations. The sampler performs actual data sampling work to obtain data samples and statistical information. The sample aggregator is responsible for collecting statistical data from various nodes and performing summary processing. The distributed reservoir sampling component is used to collect and store data samples. The 4-bit cardinality estimation component uses an optimized 4-bit cardinality estimation algorithm for cardinality estimation. Figure 1

[0057] As shown, the sampler is the core of data sampling in the statistics collection module, responsible for directly interacting with data sources, sampling necessary data samples, and integrating cardinality processing functions. Figure 2

[0058] mainLoop(): controls the continuous execution of data sampling.

[0059] SampleRow(): calls the reservoir sampling algorithm.

[0060] InsertHash(): implements the cardinality estimation algorithm.

[0061] The distributed reservoir sampling component is used to temporarily store data samples extracted from the sampler, thereby supporting further analysis and processing.

[0062] GetSamples(): provides an interface to access stored samples.

[0063] The 4-bit cardinality estimation component uses an optimized algorithm for data cardinality estimation.

[0064] InsertHash(): encodes the hash value of the data and processes it.

[0065] Estimate(): estimates the cardinality of the current data.

[0066] The sample aggregator aggregates data from various sampling nodes to obtain global statistical information.

[0067] mainLoop(): main loop, controls the execution of data aggregation.

[0068] ​​SampleRow(): receives sample data from Sampler.

[0069] Estimate(): calls the cardinality estimation algorithm to estimate the data cardinality.

[0070] WriteResults(): outputs the final statistics to support the decision of the query optimizer.

[0071] As shown in Figure 1 and Figure 2 , the connector receives the client's statistics collection request, calls the distributed plan generator to generate an execution plan and issues it to the specific computing node. The sample aggregator and the sampler are started to perform data sampling. The sampler continuously extracts samples from the data source and transmits the data to the distributed reservoir sampling component storage. At the same time, the data is read from the data source and sent to the 4-bit cardinality estimation component for cardinality estimation to calculate the cardinality of the data. The sample aggregator collects the data returned by each node and performs aggregation. The processed statistics are stored persistently. All components work together to ensure the efficiency and accuracy of the entire process from receiving the data collection request to performing specific data sampling and cardinality estimation.

[0072] Figure 3 is an example of a data sampling system containing five nodes, and Figure 1 is a statistics collection module and Figure 2 the distributed reservoir sampling component and the 4-bit cardinality estimation component in Figure 3 are closely related and together constitute the implementation details of the optimized statistics cardinality estimation method for distributed databases of the present application. The following is a specific description of each part shown in

[0073] 1-1) Initialization, the data sampling system consists of five nodes (Node1 to Node5), each node includes a data storage unit (data1 to data5). These data storage units are responsible for storing their own data sets, and each storage unit is connected with a sampler (Sampler) below. These samplers directly extract samples from the corresponding data storage unit, which is consistent with the function and operation of the sampler component in Figure 2 . Assign a reservoir R i of size K to each node i (Node1 to Node5)

[0074] 1-2) Local sampling: The core function of each sampler is to execute the distributed reservoir sampling algorithm. It obtains data item x through the data storage units (data1 to data5) in the statistical information collection module. Each node i (Node1 to Node5) independently samples the read data and determines the reservoir R. i Given the current size, perform the following operation for each data item x:

[0075] 1-2-1) If the reservoir R i If the current size is less than K, then add the data item x directly to the reservoir R. i middle;

[0076] 1-2-2) If the reservoir R i If the current size is K, then generate a random integer j ranging from 1 to the total number of data items being processed. If j ≤ K, then the reservoir R will be... i Replace the j-th element in the data with the current data item x;

[0077] 1-3) Global sampling: Samples output from all nodes' samplers are aggregated into a central sample aggregator. This process is similar to... Figure 1 The description of the sample aggregator matches the given description. The main function of the sample aggregator is to synthesize all collected samples and perform necessary processing or analysis to support final data decisions and query optimization. The reservoirs R1, R2, ..., R5 of all nodes (Node1 to Node5) will be merged into a global reservoir R. g If the global water storage tank R g If the size is greater than K, then repeat step 1-2-2) in step 1-2) until the global reservoir R is reached. g The global reservoir R is defined as having a size equal to K. g It contains a sample set from the entire distributed database system;

[0078] After the sample aggregator finishes processing, it outputs a response, which is a histogram showing the distribution of the reaction data based on the collected data.

[0079] Regarding the aforementioned statistical information collection module, this invention optimizes the algorithms in the distributed reservoir sampling component and the 4-bit cardinality estimation component, proposing a statistical information cardinality estimation method for distributed databases, such as... Figure 4 As shown, it includes two parts: a distributed reservoir sampling algorithm and a 4-bit radix statistics algorithm.

[0080] Part 1: Distributed Reservoir Sampling, including initialization, local sampling, and global sampling; details are as follows:

[0081] Step 1-1) Initialization: In the statistical information collection module of the distributed database system, allocate a reservoir R of size K to each node i. i Where K represents only the number of data samples stored locally on each node i;

[0082] Step 1-2) Local Sampling: Data item x is obtained through the storage interface in the statistical information collection module. Each node i independently samples the read data and determines the water storage tank R. i Given the current size, perform the following operation for each data item x:

[0083] 1-2-1) If the reservoir R i If the current size is less than K, then add the data item x directly to the reservoir R. i middle;

[0084] 1-2-2) If the reservoir R i If the current size is K, then generate a random integer j ranging from 1 to the total number of data items being processed. If j ≤ K, then the reservoir R will be... i Replace the j-th element in the data with the current data item x;

[0085] Steps 1-3) Global sampling: Sampling the reservoirs R1, R2, ..., R of all nodes. n They will be merged into a global reservoir R g If the global water storage tank R g If the size is greater than K, then repeat step 1-2-2) in step 1-2) until the global reservoir R is reached. g The global reservoir R is defined as having a size equal to K. g It contains a sample set from the entire distributed database system.

[0086] Part Two, the 4-bit radix statistics algorithm is... Figure 1 Chinese statistical information collection module and Figure 2 The core of the 4-bit cardinality estimation component is tightly integrated with these components, together forming a comprehensive statistical information processing framework.

[0087] A 4-bit cardinality statistics algorithm, including mode definition and algorithm initialization, intelligent switching between sparse and dense modes, adjustment of register M size for dense mode, and calculation and return of the final cardinality estimate based on the current mode.

[0088] Step 2-1) Pattern definition and algorithm initialization:

[0089] In the 4-bit cardinality estimation algorithm, sparse mode and dense mode are adopted, wherein: the sparse mode is used to process the case where the memory required for data set cardinality estimation is lower than the maximum memory threshold used by the dense mode, at which time the algorithm only records the actual touched register index and the corresponding count value; the dense mode is used to process the case where the memory required for data set cardinality estimation reaches the maximum memory threshold, at which time the algorithm allocates and maintains a complete register array;

[0090] In the 4-bit cardinality estimation algorithm, xxhash is selected as the hash function of the algorithm.

[0091] In the 4-bit cardinality estimation algorithm, the algorithm parameters are set as follows:

[0092] sparse: whether to use sparse representation; p: precision in dense mode; p': precision in sparse mode, p'>p; m: number of buckets; alpha m : used for bias correction; tmpSet: temporary set; sparseList: array that can be stored in sparse representation, denoted as sparse list; M: register; b: base register; capacity: maximum capacity of offset register.

[0093] Step 2-2) Intelligent switching of sparse mode and dense mode, wherein the size of register M needs to be adjusted in dense mode. In this step, for each element v in the sample set obtained in the first part, the following operations are performed until all elements in the sample set are operated.

[0094] 2-2-1) First, read an element in the sample set.

[0095] 2-2-2) Calculate the hash value h(v) of the current element using the hash function xxhash; if sparse==false, use dense mode and perform step 2-2-3); if sparse==true, use sparse mode and perform step 2-2-4).

[0096] 2-2-3) Adjustment of the size of register M, thereby updating the corresponding maximum leading zero zeros in register M, the process being as follows:

[0097] First, initialize the base register b and all registers M n to 0, where n ranges from 0 to m-1;

[0098] From the hash value h(v), extract the precision p bits in dense mode to calculate the index n of the register, update the register corresponding to the index n; extract the remaining bits other than the precision p bits to calculate the maximum leading zero zeros;

[0099] If the maximum leading zeros zeros minus the base register b is greater than or equal to the offset register maximum capacity capacity, an offset overflow is detected; replace the capacity of the base register b with the minimum value in all registers M, and update the capacity of all registers M according to the current capacity of the base register b; perform step 2-3);

[0100] If the maximum leading zeros zeros minus the base register b is less than the offset register maximum capacity capacity, replace the maximum leading zeros zeros with the register M n and compare the maximum leading zeros zeros with the register M n ; read the next element in the sample set; return to step 2-2-2);

[0101] 2-2-4) Intelligent switching between sparse mode and dense mode:

[0102] The touched register index and the corresponding count value are compressed by variable length coding and differential coding and added to the temporary set tmpSet; if the size of the temporary set tmpSet exceeds the threshold 4m / 100, perform the merging operation, sort and merge to the sparse list sparseList; judge whether the size of the sparse list sparseList exceeds the memory usage threshold 4m bits, if the condition is met, switch from the sparse mode to the dense mode, and convert the sparse list sparseList to a normal register array M; otherwise, read the next element in the sample set, and return to step 2-2-2);

[0103] 2-3) Calculate and return the final cardinality estimate value according to the current mode, which has one of the following cases:

[0104] First, the current mode is sparse mode, and the linear counting method is used to calculate the cardinality estimate value

[0105]

[0106] Where m is the total number of registers, and V is the number of registers M that do not record any elements in the counting process.

[0107] Second, the current mode is dense mode, and the following formula is used to calculate the cardinality estimate value

[0108]

[0109] Where m is the total number of registers, and α m is a correction coefficient dependent on m to correct the bias, and the value range of m is 2 4 ~ 2 18 ; m = 2 4when a m = 0.673; m = 2 5 when a m = 0.679; m = 2 6 when a m = 0.709; m = 2 7 ~ 2 18 when a m = 0.7213 / (1+1.079 / m).

[0110] The 4-bit cardinality estimation algorithm proposed in the present application is experimentally verified, and compared with the traditional cardinality estimation algorithm in terms of memory occupation and accuracy, as shown in Table 1 and Table 2, wherein the test data comes from the LiveJournal social network dataset.

[0111] Table 1 Comparison of memory occupation of different cardinality estimation algorithms

[0112]

[0113]

[0114] Table 2 Comparison of accuracy of different cardinality estimation algorithms

[0115]

[0116] The experimental results show that, in terms of memory occupation, the 4-bit cardinality estimation algorithm reduces the memory occupation by about 50% compared with the traditional HLL algorithm and the traditional HLL++ algorithm, and optimizes the memory usage; in terms of accuracy, the 4-bit cardinality estimation algorithm also has an improvement compared with the traditional HLL algorithm and the HLL++ algorithm, especially in the case of small cardinality.

[0117] Although the present application has been described in conjunction with the accompanying drawings, the present application is not limited to the above-described specific embodiments, and the above-described specific embodiments are merely illustrative, rather than limiting, and many improvements and changes can be made by those of ordinary skill in the art under the inspiration of the present application without departing from the purpose of the present application, and these all belong to the protection of the present application.

Claims

1. A method for estimating the cardinality of statistical information for distributed databases, characterized in that, Includes the following steps: S1. Sampling of distributed water storage tanks, including: 1-1) Initialization: In the statistical information collection module of the distributed database system, a reservoir R of size K is allocated to each node i. i Where K represents only the number of data samples stored locally on each node i; 1-2) Local sampling: Data item x is obtained through the storage interface in the statistical information collection module. Each node i independently samples the read data and determines the water storage tank R. i Given the current size, perform the following operation for each data item x: 1-2-1) If the reservoir R i If the current size is less than K, then add the data item x directly to the reservoir R. i middle; 1-2-2) If the reservoir R i If the current size is K, then generate a random integer j ranging from 1 to the total number of data items being processed. If j ≤ K, then the reservoir R will be... i Replace the j-th element in the data with the current data item x; 1-3) Global sampling: Reservoirs R1, R2, ..., R of all nodes n They will be merged into a global reservoir R g If the global water storage tank R g If the size is greater than K, then repeat step 1-2-2) until the global reservoir R. g The global reservoir R is defined as having a size equal to K. g It contains a sample set from the entire distributed database system; S2, 4-bit radix statistics algorithm, including: 2-1) Pattern definition and algorithm initialization, including: The 4-bit radix statistics algorithm includes dense and sparse modes, where: The dense mode algorithm is used to handle the situation where the memory required for dataset cardinality estimation reaches the maximum memory threshold. In this case, the algorithm will allocate and maintain a complete register array. The sparse mode is used to handle cases where the memory required for dataset cardinality estimation is lower than the maximum memory threshold used by the dense mode. In this case, the algorithm only records the actual register indices touched and the corresponding count values. In the 4-bit radix statistics algorithm, xxhash is chosen as the hash function. In the 4-bit radix counting algorithm, the algorithm parameters are set as follows: sparse: Whether to use sparse representation; p: Precision in dense mode; p′: Precision in sparse mode, p′>p; m: the number of buckets; α m Used for deviation correction; tmpSet: Temporary collection; sparseList: An array that can be stored in a sparse representation, denoted as a sparse list; M: Register; b: Basic register; capacity: Maximum capacity of the offset register; 2-2) Intelligent switching between sparse and dense modes and adjustment of register M size in dense mode. For each element ν in the sample set obtained in step S1, perform the following operation: 2-2-1) Read an element from the sample set; 2-2-2) Calculate the hash value h(ν) of the current element using the hash function xxhash; If sparse == false, use dense mode and execute step 2-2-3); If sparse == true, adopt sparse mode and execute step 2-2-4). 2-2-3) Adjust the size of register M to update the corresponding maximum leading zeros in register M, as follows: First, initialize the base register b and all registers M. n The value of n is 0, where the range of n is [0, m); From the hash value h(ν), extract the precision p bits in the dense mode, calculate the register index n, and update the register corresponding to the index n; extract the remaining bits excluding the precision p bits, and calculate the maximum leading zeros zeros; If the maximum leading zeros minus the base register b is greater than or equal to the maximum capacity of the offset register, an offset overflow is detected; the capacity of the base register b is replaced with the minimum value among all registers M, and the capacity of all registers M is updated according to the current capacity of the base register b; proceed with steps 2-3); If the maximum leading zeros minus the base register b is less than the maximum capacity of the offset register, then the maximum leading zeros and register M are... n Compare the two values ​​and update M by taking the larger value. n ; Read the next element in the sample set; return to step 2-2-2); 2-2-4) Compress the touched register index and corresponding count value using variable-length encoding and differential encoding, and add them to the temporary set tmpSet; If the size of the temporary set tmpSet exceeds the threshold of 4m / 100, a merge operation is performed, sorting and merging the results into the sparse list sparseList; Determine if the size of the sparse list sparseList exceeds the memory usage threshold of 4m bits. If the condition is met, switch from sparse mode to dense mode and convert the sparse list sparseList into a normal register array M; otherwise, read the next element in the sample set and return to step 2-2-2). Repeat steps 2-2-2) to 2-2-4) above until all elements in the sample set have been processed. 2-3) Calculate and return the final cardinality estimate based on the current model, under one of the following circumstances: First, the current model is a sparse model, using linear counting to calculate the cardinality estimate. Where m is the total number of registers, and V is the number of registers M that did not record any elements during the counting process; Second, the current model is a dense model, and the cardinality estimate is calculated using the following formula. Where m is the total number of registers, α m It relies on a correction factor of m to correct for deviations, and the value of m ranges from 2. 4 ~2 18 m=2 4 At that time, α m =0.673; m=2 5 At that time, α m =0.679; m=2 6 At that time, α m =0.709; m=2 7 ~2 18 α m =0.7213 / (1+1.079 / m).

Images