A database frequency estimation method

Abstract

The present invention relates to a database frequency estimation method. The data structure of the database includes three two-dimensional arrays, namely H, M, and L, and two bitmaps. The number of rows of the three two-dimensional arrays is equal. The number of columns of the high-frequency two-dimensional array H is 1 / 2 of the number of columns of the medium-frequency two-dimensional array M, and the number of columns of the medium-frequency two-dimensional array M is 1 / 2 of the number of columns of the low-frequency two-dimensional array L. The number of rows and columns of the low-frequency bitmap F and the medium-frequency bitmap S are the same as those of the low-frequency two-dimensional array L and the medium-frequency two-dimensional array M, respectively. Compared with the traditional method, the L layer has more counters and smaller memory space for the counters, which can reduce the waste of space when the counters record low-frequency data items, achieving the purpose of efficient memory utilization. For those medium- and high-frequency data items, after falling into the L layer, they will overflow to the M layer and the H layer again, still ensuring that while reducing hash conflicts, the memory space can be fully utilized.

Description

Technical Field

[0001] The present invention belongs to the technical field of databases, and particularly relates to a new database frequency estimation method, which provides frequency statistical information of data stored in the database, thereby making the query efficiency of the database higher. Background Art

[0002] Count-Min Sketch is a commonly used frequency estimation method in databases. For example, TiDB uses this method for collecting statistical information and performing frequency estimation during database point queries. The data structure of Count-Min Sketch is a two-dimensional array with width w and depth d, having d pairwise independent hash functions h1...h d , where the number of rows in the two-dimensional array represents the number of hash functions, and the number of columns represents the magnitude of the estimate. When inserting a data item e, d different hash values are calculated through these hash functions, and then the original value c at the corresponding position is incremented by 1. Each small square represents a counter (see Figure 1 ).

[0003] The disadvantage of Count-Min Sketch is that in order to count data items with higher frequencies, the counters must allocate space based on the highest frequency and open up counters of the same size. However, in actual applications, there will be a great waste of space for those counters with fewer counts. Moreover, since these spaces are allocated from memory and held for a long time during the operation of the database, this disadvantage will have a great impact on the performance of the database.

[0004] Currently, the method mainly used in database frequency estimation is Count-Min Sketch, and the main problems of this method are as follows:

[0005] The accuracy decreases in the case of a large amount of data due to hash collisions;

[0006] The memory usage efficiency is low. Because the counter sizes need to evenly allocate memory based on the highest frequency, the counters with lower counting frequencies will cause a great waste of space. If these wasted spaces can be effectively utilized, the operating pressure of the database will be improved. Summary of the Invention

[0007] Aiming at the deficiencies of the prior art, the technical problem to be solved by the present invention is to provide a new database frequency estimation method. This estimation method is based on the basic idea of Count-Min Sketch and adopts a hierarchical method to ensure that while reducing hash collisions, the memory space is fully utilized, effectively improving the operating pressure of the database.

[0008] The technical solution adopted by the present invention to solve the above technical problem is:

[0009] In a first aspect, the present invention provides a database frequency estimation method, and the estimation method includes the following:

[0010] The data structure of the database includes three two-dimensional arrays, namely a low-frequency two-dimensional array L, a medium-frequency two-dimensional array M, and a high-frequency two-dimensional array H. The number of rows of the three two-dimensional arrays is equal. The number of columns of the high-frequency two-dimensional array H is 1 / 2 of the number of columns of the medium-frequency two-dimensional array M, and the number of columns of the medium-frequency two-dimensional array M is 1 / 2 of the number of columns of the low-frequency two-dimensional array L. The data structure further includes two bitmaps, namely a low-frequency bitmap F and a medium-frequency bitmap S. The number of rows and columns of the low-frequency bitmap F and the medium-frequency bitmap S are respectively the same as the number of rows and columns of the low-frequency two-dimensional array L and the medium-frequency two-dimensional array M.

[0011] The structure of the two-dimensional array is composed of storage units of a fixed size, and these storage units are called counters. The position of the counter in the two-dimensional array is located according to the row number and column number of the two-dimensional array, and the counter is used to record the frequency of the data. The maximum thresholds of the respective counters of the three two-dimensional arrays, namely the low-frequency two-dimensional array L, the medium-frequency two-dimensional array M, and the high-frequency two-dimensional array H, can be the same or different, and the maximum thresholds of all counters in the same layer are the same.

[0012] In the structure of the bitmap, the size of each storage unit is 1 bit, and these storage units are called bits. The values in the bitmap can only be 0 and 1. The positions of these bits in the bitmap are determined by the row number and column number of the bitmap. The low-frequency bitmap F and the medium-frequency bitmap S are respectively used to record the overflow situations of the counters in L and M.

[0013] For each counter L[i][j] in L, the corresponding bit F[i][j] can be found in the low-frequency bitmap F using the same row number i and column number j. If the value stored in bit F[i][j] is 0, it means the counter has not overflowed, indicating that there is no carry at the position corresponding to 0 in L, and the value corresponding to the current position in L is taken as the frequency count. If the value stored in bit F[i][j] is 1, it means the counter has overflowed and a carry has occurred, and then a hash calculation is performed on M. If the value stored in bit S[i][j] is 0, it means there is no carry at the position corresponding to 0 in M, and then the frequency count at this time is M[i][D M，i *MAX L +L[i][D L，i . If the value stored in bit S[i][j] is 1, it means there is a carry at the position corresponding to 1 in M, and then a hash calculation is performed on H, and then the frequency count at this time is (H[i][D H，i *MAX M +M[i][D M，i )*MAX L +L[i][D L，i . Thus, the frequency count estimation is completed.

[0014] The number of rows of the three - layer two - dimensional arrays L, M, and H is 5. The number of columns of H is 1 / 2 of the number of columns of M, the number of columns of M is 1 / 2 of the number of columns of L, and the number of columns of L is defaulted to 2048; there are 5 hash functions. The size of the counters in the L layer is set to lbyte, the size of the counters in the M layer is set to 2byte, and the size of the counters in the H layer is set to 2byte. The total occupied space of all counters is 25K.

[0015] The estimation method includes the processes of inserting data, querying data, and deleting data. The maximum thresholds of the counters in the high - frequency, medium - frequency, and low - frequency two - dimensional arrays are set to MAX H 、MAX M and MAX L respectively. The specific steps are as follows:

[0016] Step S1: Insert the data e into the two - dimensional array. Before inserting the first data, initialize all the two - dimensional arrays and bitmaps, that is, the values of each counter and bit are initialized to 0;

[0017] Step S1 - 1: Set i = 0, start calculating from the low - frequency two - dimensional array L. Use the hash function to calculate the hash value of the data e, and this hash value is the column number D L，i in the low - frequency two - dimensional array L to which the data e is mapped, D L，0 = Hash0(e). Locate the counter L[0][D L，0 and the bit F[0][D L，0 ;

[0018] Step S1 - 2: Judge whether the value stored on the counter L[0][D L，0 +1 is greater than MAX L . If it is not greater, then store the value + l on the counter L[0][D L，0 , and then jump to step S1 - 6 to repeat the hash calculation of other rows; if the value stored on the counter L[0][D L，0 +1 is greater than MAX L , it means that the counter overflows. Then modify the values stored on the counter L[0][D L，0 and the bit F[0][D L，0 to 1, and continue to execute step S1 - 3 to judge whether the medium - frequency two - dimensional array M overflows;

[0019] Step S1 - 3: Divide D L，i by 2 and round down to obtain the position in the medium - frequency two - dimensional array M to which the hash value of the data e calculated by the hash function is mapped. Starting from i = 0, the mapped column number is D M，0 . Then the positions of the data e in the medium - frequency two - dimensional array M and the medium - frequency bitmap S are: Locate the counter M[0][DM，0 and bit S[0][D M，0 ;

[0020] Step S1-4: Determine whether the value stored in counter M[0][D M，0 plus 1 is greater than MAX M , if not, then add 1 to the value stored in counter M[0][D M，0 , and then jump to step S1-6 to repeat the hash calculation of other lines; if the value stored in counter M[0][D M，0 plus 1 is greater than MAX M , it means the counter overflows, then modify the values stored in both counter M[0][D M，0 and bit S[0][D M，0 to 1, continue to execute step S1-5 to determine whether the high-frequency two-dimensional array H overflows;

[0021] Step S1-5: Divide D M，i by 2 and round down to obtain the position in the high-frequency two-dimensional array H where the hash value of data e calculated using the hash function is mapped. Starting from i = 0, the mapped column number D H，0 = [D M，0 * (1 / 2)」, locate counter H[0][D H，0 , and then determine whether the value stored in counter H[0][D H，0 plus 1 is greater than MAX H , if not, then increment the value stored in counter H[0][D H，0 ; if the value stored in counter H[0][D H，0 plus 1 is greater than MAX H , then throw an exception and need to increase the MAX H value of the counter, and jump to step Sl-7;

[0022] Step Sl-6: Set i = 1, 2, 3, 4 respectively, and repeat steps S1-1, S1-2, S1-3, S1-4, S1-5;

[0023] Step S1-7: Insert data e ends;

[0024] Step S2: Query the frequency of data e appearing in the two-dimensional array to complete the frequency estimation of data e

[0025] Step S2-1: Start querying from the low-frequency two-dimensional array L, calculate the hash value Hash i (e), i ∈ {0, 1, 2, 3, 4}, and this hash value is the column number D L，i where data e is mapped to the low-frequency two-dimensional array L, then locate counter L[i][DL，i AND bit F[i][D L，i ;

[0026] Step S2-2, determine whether the values stored in the bit F[i][D L，i obtained by positioning in Step S2-1 are all 1. If they are not all 1, take out the counter values of all non-1 corresponding low-frequency two-dimensional arrays L on the low-frequency bitmap F; then compare and return the minimum value, which is the frequency of occurrence of e;

[0027] Step S2-3, if the values stored in the bit F[i][D L，i obtained by positioning in Step S2-1 are all 1, it means that all low-frequency two-dimensional arrays L have overflowed. Then query in the medium-frequency two-dimensional array M, and divide the hash value Hash i (e) of data e by 2 and take the integer part downwards to obtain the column number D in the medium-frequency two-dimensional array M to which data e is mapped M，i , D M，i = [D L，i * (1 / 2)」, locate the counter M[i][D M，i and bit S[i][D M，i ;

[0028] Step S2-4, determine whether the values stored in the bit S[i][D M，i obtained by positioning in Step S2-3 are all 1. If they are not all 1, take out the counter values of all non-1 corresponding medium-frequency two-dimensional arrays M on the medium-frequency bitmap S, multiply by the maximum threshold MAX of the counters in the low-frequency two-dimensional array L and then add the counter values stored in the low-frequency two-dimensional array corresponding to the same hash function mapping, that is, M[i][D M，i * MAX L + L[i][D L，i ; Finally, compare and return the minimum value, which is the frequency of occurrence of data e;

[0029] Step S2-5, if the values stored in the bit S[i][D M，i obtained by positioning in Step S2-3 are all 1, it means that all medium-frequency two-dimensional arrays M have overflowed. Then query in the high-frequency two-dimensional array H, divide the column number D in the medium-frequency two-dimensional array M to which data e is mapped M，i by 2 and take the integer part downwards to obtain the column number D in the high-frequency two-dimensional array H to which data e is mapped H，i , that is locate the counter H[i][D H，i , then compare and take out the minimum value, and calculate with the minimum H[i][D H，i . Finally, the frequency of occurrence of e is obtained as (H[i][D H，i * MAXM +M[i][D M，i )*MAX L +L[i][D L，i ;

[0030] Step S3: Delete the insertion record of a data e from the two-dimensional array

[0031] Step S3-1: Start deleting from the low-frequency two-dimensional array L. Set i = 0, calculate the hash value Hash0(e) of the data e, and this hash value is the column number D that maps the data e to the low-frequency two-dimensional array L L，0 , locate the counter L[0][D L，0 and the bit F[0][D L，0 ;

[0032] Step S3-2: If the value stored in the counter L[0][D L，0 is greater than 0, then subtract 1 from the value stored in the counter L[0][D L，0 , and then jump to S3-6; if the value stored in the counter L[0][D L，0 is equal to 0, check the bit F[0][D L，0 , if the bit F[0][D L，0 is equal to 0, then throw a deletion exception, if the bit F[0][D L，0 is equal to 1, then modify the value stored in the counter L[0][D L，0 to MAX L -1, and continue to execute Step S3-3 to delete from the medium-frequency two-dimensional array M;

[0033] Step S3-3: Divide D L，i by 2 and round down to obtain the position in the medium-frequency two-dimensional array mapped by the hash value of the data e calculated using the hash function. Starting from i = 0, the mapped column number is D M，0 = [D L，0 *(1 / 2)」, locate the counter M[0][D M，0 and the bit S[0][D M，0 ;

[0034] Step S3-4: If the value stored in the counter M[0][D M，0 is greater than 0, then subtract 1 from the value stored in the counter M[0][D M，0 , after the -1 operation, if the value stored in M[0][D M，0 is not 0, then jump to S3-6; after the -1 operation, if the value stored in M[0][D M，0 is 0, then judge whether S[0][D M，0 is 0, if S[0][D M，0If it is 0, then set F[0][D] in the low-frequency bitmap F to 0, and then jump to S3-6; if S[0][D] is not 0, then modify the value stored in the counter M[0][D] to MAX L，0 -1, and then divide D by 2 and round down to obtain the position in the high-frequency two-dimensional array H where the hash value of the data e calculated using the hash function is mapped. Starting from i = 0, the number of mapped columns is M，0 If it is not 0, then set the value stored in the counter M[0][D] to MAX M，0 -1, and then divide D by 2 and round down to obtain the position in the high-frequency two-dimensional array H where the hash value of the data e calculated using the hash function is mapped. Starting from i = 0, the number of mapped columns is M -1, and then divide D by 2 and round down to obtain the position in the high-frequency two-dimensional array H where the hash value of the data e calculated using the hash function is mapped. Starting from i = 0, the number of mapped columns is M，i -1, and then divide D by 2 and round down to obtain the position in the high-frequency two-dimensional array H where the hash value of the data e calculated using the hash function is mapped. Starting from i = 0, the number of mapped columns is Locate the counter H[0][D H，0 , subtract 1 from the value stored in the counter H[0][D H，0 . After the -1 operation, if the value stored in H[0][D H，0 is not 0, then jump to S3-6; after the -1 operation, if the value stored in H[0][D H，0 is 0, then set S[0][D] in the medium-frequency bitmap S to 0 and then jump to S3-6; M，0 is 0, then set F[0][D] in the low-frequency bitmap F to 0 and then jump to S3-6; if S[0][D

[0035] Step S3-5: If the value of the counter M[0][D M，0 is equal to 0, check the bit S[0][D M，0 . If the bit S[0][D M，0 is equal to 0, then set F[0][D] in the low-frequency bitmap F to 0 and then jump to S3-6; if S[0][D L，0 is equal to 1, then modify the value stored in the counter M[0][D M，0 to MAX M，0 -1 and continue to execute step S3-5; M -1 and continue to execute step S3-5;

[0036] Step S3-5: Divide D by 2 and round down to obtain the position in the high-frequency two-dimensional array H where the hash value of the data e calculated using the hash function is mapped. Starting from i = 0, the number of mapped columns is M，i -1, and then divide D by 2 and round down to obtain the position in the high-frequency two-dimensional array H where the hash value of the data e calculated using the hash function is mapped. Starting from i = 0, the number of mapped columns is Locate the counter H[0][D H，0 , subtract 1 from the value stored in the counter H[0][D H，0 . After the -1 operation, if the value stored in H[0][D H，0 is not 0, then jump to S3-6; after the -1 operation, if the value stored in H[0][D H，0 is 0, then set S[0][D] in the medium-frequency bitmap S to 0 and then jump to S3-6; M，0 is 0, then set S[0][D] in the medium-frequency bitmap S to 0 and then jump to S3-6;

[0037] Step S3-6: Set i = 1, 2, 3, 4 respectively, and repeat the execution of steps S3-1, S3-2, S3-3, S3-4, S3-5;

[0038] Step S3-7, deletion completed.

[0039] The estimation method can estimate the access frequency of data in the database, realize the separation of hot and cold data. Set the hot data threshold top-k, and define the top k elements with the largest access frequencies as hot data and move them to memory, achieving the separation of hot and cold data, alleviating the pressure on the database, and improving the query efficiency.

[0040] The estimation method is an improvement of the Count-Min Sketch data structure and can be applied to natural language processing, data stream statistics, calculation of point mutual information, sparse approximation of compressive sensing, detection of network abnormal flows, and processing of distributed data sets, which can improve the operating pressure of the database and the utilization rate of memory space.

[0041] In a second aspect, the present invention provides a database frequency estimation method in data stream statistics. The steps of the estimation method are specifically as follows:

[0042] Step 1: Find the data stream to be processed;

[0043] Step 2: Initialize the data structure described in claim 1, set the values of all counters in the L layer, M layer, and H layer of the data structure to 0, and at the same time set the values of the low-frequency bitmap F and the medium-frequency bitmap S to 0, and insert the data in the data stream into the data structure in sequence;

[0044] When inserting the first piece of data e, set i = 0, start calculating from the low-frequency two-dimensional array L, calculate the hash value of the data e using the hash function, and this hash value is the column number D in which the data e is mapped to the low-frequency two-dimensional array L L，i , D L，0 = Hash0(e), locate the counter L[0][D L，0 and the bit F[0][D L，0 ;

[0045] Step 4: Determine whether the value stored on the counter L[0][D L，0 +1 is greater than MAX L . If it is not greater, then add 1 to the value stored on the counter L[0][D L，0 , and then jump to step 8 to repeat the hash calculation of other lines; if the value stored on the counter L[0][D L，0 +1 is greater than MAX L , it means that the counter overflows, then modify the values stored on both the counter L[0][D L，0 and the bit F[0][D L，0 to 1, and continue to execute step 5 to determine whether the medium-frequency two-dimensional array overflows;

[0046] Step 5: Divide D L，i by 2 and round down to obtain the position in the intermediate-frequency two-dimensional array mapped by calculating the hash value of data e using the hash function. Starting from i = 0, the number of mapped columns is D M，0 , then the positions of data e in the intermediate-frequency two-dimensional array M and the intermediate-frequency bitmap S are: locate the counter M[0][D M，0 and bit S[0][D M，0 ;

[0047] Step 6: Determine whether the value stored on the counter M[0][D M，0 plus 1 is greater than MAX M . If it is not greater, then store the value stored on the counter M[0][D M，0 plus 1, and then jump to Step 8 to repeat the hash calculation for other rows; if the value stored on the counter M[0][D M，0 plus 1 is greater than MAX M , it means the counter overflows. Then modify the values stored on both the counter M[0][D M，0 and the bit S[0][D M，0 to 1, continue to execute Step 7, and determine whether the high-frequency two-dimensional array H overflows;

[0048] Step 7: Divide D M，i by 2 and round down to obtain the position in the high-frequency two-dimensional array H mapped by calculating the hash value of data e using the hash function. Starting from i = 0, the number of mapped columns D H，0 = [D M，0 *(1 / 2)」, locate the counter H[0][D H，0 , and then determine whether the value stored on the counter H[0][D H，0 plus 1 is greater than MAX H . If it is not greater, then increment the value stored on the counter H[0][D H，0 ; if the value stored on the counter H[0][D H，0 plus 1 is greater than MAX H , then throw an exception, and it is necessary to increase the value of MAX of the counter H , and jump to Step 9;

[0049] Step 8: Set i = 1, 2, 3, 4 respectively, and repeat Steps 3, 4, 5, 6, 7;

[0050] Step 9: Finish inserting data e;

[0051] Step 10: Subsequently, whenever there is data to be inserted, repeat Steps 3 to 9 until all the data in the data stream are inserted into the data structure;

[0052] Step 11: After all the data in the data stream have been inserted, start calculating the frequencies of the data occurrences in the data stream;

[0053] Step 12: First, query the first data e. Start querying from the low-frequency two-dimensional array L, and calculate the hash value Hash i (e), where i ∈ {0, 1, 2, 3, 4}. This hash value is the column number D in the low-frequency two-dimensional array L to which the data e is mapped L，i , and then locate the counter L[i][D L，i and the bit F[i][D L，i ;

[0054] Step 13: Determine whether the values stored in the bit F[i][D L，i located in Step 12 are all 1. If they are not all 1, take out the values of the counters in the low-frequency two-dimensional array L corresponding to all the non-1s on the low-frequency bitmap F; then compare and return the minimum value, which is the frequency of e's occurrence;

[0055] Step 14: If the values stored in the bit F[i][D L，i located in Step 12 are all 1, it means that the entire low-frequency two-dimensional array L has overflowed. Then query in the medium-frequency two-dimensional array M. Divide the hash value Hash i (e) of the data e by 2 and round down to obtain the column number D in the medium-frequency two-dimensional array M to which the data e is mapped M，i , locate the counter M[i][D M，i and the bit S[i][D M，i ;

[0056] Step 15: Determine whether the values stored in the bit S[i][D M，i located in Step 14 are all 1. If they are not all 1, take out the values of the counters in the medium-frequency two-dimensional array M corresponding to all the non-1s on the medium-frequency bitmap S, multiply by the maximum threshold MAX L of the counters in the low-frequency two-dimensional array, and then add the values stored in the counters in the low-frequency two-dimensional array corresponding to the same hash function mapping, that is, M[i][D M，i *MAX L +L[i][D L，i ; Finally, compare and return the minimum value, which is the frequency of e's occurrence;

[0057] Step 16: If the values stored in the bit S[i][D M，i located in Step 14 are all 1, it means that the entire medium-frequency two-dimensional array M has overflowed. Then query in the high-frequency two-dimensional array H. Map the column number D in the medium-frequency two-dimensional array M to which the data e is mapped M，iDivide by 2 and round down to obtain the number of columns D of the data e mapped to the high-frequency two-dimensional array H H，i , that is Locate the counter H[i][D H，i , then compare and extract the minimum value, and use the minimum H[i][D H，i for calculation. Finally, the frequency of e appearing is (H[i][D H，i *MAX M +M[i][D M，i )*MAX L +L[i][D L，i ;

[0058] Step 17: Obtain the frequency of the first data e;

[0059] Step 18: Repeat the subsequent data according to Steps 12 to 16, and finally obtain the frequencies of all data in the data stream.

[0060] In a third aspect, the present invention provides a computer-readable storage medium storing computer instructions, which when executed by one or more processors, cause the one or more processors to execute the steps in the estimation method.

[0061] In a fourth aspect, the present invention provides a data processing system. The data structure of the database in the data processing system includes three two-dimensional arrays: a low-frequency two-dimensional array L, a medium-frequency two-dimensional array M, and a high-frequency two-dimensional array H. The number of rows of the three two-dimensional arrays is equal. The number of columns of the high-frequency two-dimensional array H is 1 / 2 of the number of columns of the medium-frequency two-dimensional array M, and the number of columns of the medium-frequency two-dimensional array M is 1 / 2 of the number of columns of the low-frequency two-dimensional array L; the data structure further includes two bitmaps, namely a low-frequency bitmap F and a medium-frequency bitmap S. The number of rows and columns of the low-frequency bitmap F and the medium-frequency bitmap S are the same as the number of rows and columns of the low-frequency two-dimensional array L and the medium-frequency two-dimensional array M respectively;

[0062] The structure of the two-dimensional array is composed of storage units of a fixed size. These storage units are called counters. The position of the counter in the two-dimensional array is located according to the row number and column number of the two-dimensional array, and the counter is used to record the frequency of the data; the maximum thresholds of all counters in the same layer are the same;

[0063] In the structure of the bitmap, the size of each storage unit is 1 bit. These storage units are called bits. The values in the bitmap can only be 0 and 1. The positions of these bits in the bitmap are determined by the row number and column number of the bitmap. The low-frequency bitmap F and the medium-frequency bitmap S are respectively used to record the overflow conditions of the counters in L and M.

[0064] Each counter L[i][j] in L can find the corresponding bit F[i][j] in the low-frequency bitmap F using the same row number i and column number j. If the value stored in bit F[i][j] is 0, it means the counter has not overflowed, indicating that there is no carry at the corresponding position in L. The value corresponding to the current position in L is taken as the frequency count. If the value stored in bit F[i][j] is 1, it means the counter has overflowed and there has been a carry, and then a hash calculation is performed on M. If the value stored in bit S[i][j] is 0, it means there is no carry at the corresponding position in M, and the frequency count at this time is M[i][D M，i *MAX L +L[i][D L，i ; if the value stored in bit S[i][j] is 1, it means there has been a carry at the corresponding position in M, and then a hash calculation is performed on H. The frequency count at this time is (H[i][D H，i *MAX M +M[i][D M，i )*MAX L +L[i][D L，i .

[0065] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0066] The traditional Count-Min Sketch method uses a single-layer structure (a two-dimensional array), and the capacity of the counters is evenly distributed according to the maximum count value. In the actual production environment, there will be many low-frequency data items, and the counters in the Count-Min Sketch structure for recording these low-frequency data will waste a large amount of memory space. The new method proposed in the article adopts a hierarchical design idea, a more memory-efficient frequency estimation method, which is divided into three layers of structures: L, M, and H (the L layer has the most counters, the M layer has the second most, and the H layer has the least). Data items with lower frequencies fall into the L layer and overflow to the M and H layers in turn as the data frequency increases. Compared with the traditional method, the L layer has more counters and smaller memory space for the counters (smaller counting range). More counters can reduce the probability of hash collisions, and smaller memory space can reduce the waste of space when the counters record low-frequency data items, achieving the purpose of high memory efficiency. For those medium- and high-frequency data items, after falling into the L layer, they will overflow to the M and H layers again. Since the number of data items with higher frequencies is smaller in the actual production environment, although the number of counters in the M and H layers decreases layer by layer, it can still ensure that while reducing hash collisions, the memory space is fully utilized.

[0067] The main reason for such emphasis on memory efficiency is that in a database with many tables and many columns in each table, there will be a large number of frequency estimation statistics. These statistics need to be resident in memory for quick response, consuming a lot of computer memory. Therefore, efficient use of memory becomes extremely important. The invention in this article solves this problem to a certain extent.

[0068] In summary, the method of the present invention is more efficient in memory usage and more rapid in point query efficiency under certain specific circumstances (for example: in a hash-mapped counter, if only one mapped counter does not overflow, the value is directly returned as the estimated minimum value without comparing the sizes of each estimated value).

[0069] The present invention can overflow from layer L to layer M and layer H successively as the data frequency increases. According to the hierarchical structure idea, the space occupied by each counter in the three groups of H, M, and L can be set respectively, that is, MAX L 、MAX M 、MAX H can be set respectively. The calculation process is only related to the sizes of MAX L 、MAX M 、MAX H , which is more flexible, saves more memory space, and improves the running efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is a schematic diagram of the traditional Count-Min Sketch data structure.

[0071] Figure 2 It is a schematic diagram of the new Count-Min Sketch data structure proposed by the present invention.

[0072] Figure 3 It is a flowchart of inserting data in the present invention.

[0073] Figure 4 It is a flowchart of querying the frequency of data in the present invention.

[0074] Figure 5 It is a flowchart of deleting data in the present invention.

[0075] Figure 6 It is a result diagram stored in the data structure after inserting the first six pieces of data in Embodiment 1.

[0076] Figure 7 It is a result diagram stored in the data structure after inserting all the data in Table 1 in Embodiment 1.

[0077] Figure 8 It is a performance comparison diagram between the data structure of the present invention and the traditional CMS structure. Detailed implementation mode

[0078] The present invention will be further explained below in conjunction with embodiments and the accompanying drawings, but this is not intended to limit the protection scope of the present application.

[0079]

[0080]

[0081] A database frequency estimation method of the present invention, the estimation method includes the following content:

[0082] The data structure of the database is as Figure 2 shown, including three two-dimensional arrays: a low-frequency two-dimensional array L, a medium-frequency two-dimensional array M, and a high-frequency two-dimensional array H. The number of rows of the three two-dimensional arrays is equal. The number of columns of the high-frequency two-dimensional array H is 1 / 2 of the number of columns of the medium-frequency two-dimensional array M, and the number of columns of the medium-frequency two-dimensional array M is 1 / 2 of the number of columns of the low-frequency two-dimensional array L; the data structure also includes two bitmaps, namely a low-frequency bitmap F and a medium-frequency bitmap S. The number of rows and columns of the low-frequency bitmap F and the medium-frequency bitmap S are the same as the number of rows and columns of the low-frequency two-dimensional array L and the medium-frequency two-dimensional array M respectively;

[0083] The structure of the two-dimensional array is composed of storage units of a fixed size. These storage units are called counters. According to the row number and column number of the two-dimensional array, the position of the counter in the two-dimensional array can be located. Therefore, the function of the two-dimensional array is to use the counter to record the frequency of data; the maximum thresholds of the respective counters of the three two-dimensional arrays, namely the low-frequency two-dimensional array L, the medium-frequency two-dimensional array M, and the high-frequency two-dimensional array H, can be the same or different, and the maximum thresholds of all counters in the same layer are the same. The maximum threshold of the counter is the storage capacity of the counter.

[0084] The bitmap structure is similar to the two-dimensional array, but the size of each storage unit is 1 bit. These storage units are called bits. The values in the bitmap can only be 0 and 1. Similarly, the positions of these bits in the bitmap can be determined by the row number and column number of the bitmap. The functions of the low-frequency bitmap F and the medium-frequency bitmap S are to record the overflow situations of the counters in L and M respectively.

[0085] In this embodiment, the number of rows of the three-layer two-dimensional arrays (L, M, H) is 5. The number of columns of H is 1 / 2 of the number of columns of M, the number of columns of M is 1 / 2 of the number of columns of L, and the number of columns of L is defaulted to 2048. Each small box represents a counter, and a storage capacity is set for each counter. The maximum value of the occupied amount is set by oneself (the number of rows and columns of L can be dynamically increased or decreased according to the specific usage scenario). The number of rows and columns of the bitmap F is the same as that of L, and the number of rows and columns of S is the same as that of M. Thus, for each counter L[i][j] in L, using the same row number i and column number j, the corresponding bit F[i][j] can be found in F. The function of the bit F[i][j] is to record the overflow situation of the counter L[i][j]. If the value stored on the bit F[i][j] is 0, it means the counter has not overflowed, indicating that there is no carry at the corresponding position on L, and the value corresponding to the current position can be taken out; if the value stored on the bit F[i][j] is 1, it means the counter has overflowed and a carry has occurred, and then a hash calculation is performed on M. The correspondence between the counters in M and the bits in S is also the same.

[0086] If the value stored on the bit S[i][j] is 0, it means there is no carry at the corresponding position on M, and the frequency at this time is M[i][D M，i *MAX L +L[i][D L，i ; if the value stored on the bit S[i][j] is 1, it means there is a carry at the corresponding position on M, and then a hash calculation is performed on H, and the frequency at this time is (H[i][D H，i *MAX M +M[i][D M，i )*MAX L +L[i][D L，i . Only when there are carries in both L and M, that is, when the values stored in F[i][j] and S[i][j] are both 1, the three-layer structure is fully used to complete the frequency estimation.

[0087] The specific steps of the estimation method are as follows:

[0088] Step S1: Before inserting the first data, initialize all two-dimensional arrays and bitmaps, that is, initialize the values of each counter and bit to 0; insert the data e into the two-dimensional array; Steps S1-1 to S1-7 are the complete steps for inserting a data. As Figure 3 shown.

[0089] Step S1-1: Set i = 0, start calculating from the low-frequency two-dimensional array L, and use the hash function to calculate the hash value of the data e. This hash value is the column number D L，i , D L，0= Hash0(e), locate the counter L[0][D L，0 and bit F[0][D L，0 ;

[0090] Step S1-2, determine whether the value stored on the counter L[0][D L，0 +1 is greater than MAX L , if not, store the value +1 on the counter L[0][D L，0 , then jump to step S1-6 to repeat the hash calculation of other lines; if the value stored on the counter L[0][D L，0 +1 is greater than MAX L , it means the counter overflows, then modify the values stored on both the counter L[0][D L，0 and bit F[0][D L，0 to 1, and continue to execute step S1-3 to determine whether the intermediate frequency two-dimensional array overflows;

[0091] Step S1-3, divide D L，i by 2 and round down to obtain the position in the intermediate frequency two-dimensional array where the hash value of data e calculated by the hash function is mapped. Starting from i = 0, the number of mapped columns is D M，0 , then the positions of data e in the intermediate frequency two-dimensional array M and the intermediate frequency bitmap S are: locate the counter M[0][D M，0 and bit S[0][D M，0 ;

[0092] Step S1-4, determine whether the value stored on the counter M[0][D M，0 +1 is greater than MAX M , if not, store the value +1 on the counter M[0][D M，0 , then jump to step S1-6 to repeat the hash calculation of other lines; if the value stored on the counter M[0][D M，0 +1 is greater than MAX M , it means the counter overflows, then modify the values stored on both the counter M[0][D M，0 and bit S[0][D M，0 to 1, and continue to execute step S1-5 to determine whether the high-frequency two-dimensional array H overflows;

[0093] Step S1-5, divide D M，i by 2 and round down to obtain the position in the high-frequency two-dimensional array H where the hash value of data e calculated by the hash function is mapped. Starting from i = 0, the number of mapped columns D H，0 =[D M，0 *(1 / 2)」, locate the counter H[0][D H，0, then determine the value stored in counter H[0][D H，0 plus 1, and check if it is greater than MAX H , if not, increment the value stored in counter H[0][D H，0 ; if the value stored in counter H[0][D H，0 plus 1 is greater than MAX H , then throw an exception and adjust the MAX value of the counter H , and jump to step S1-7;

[0094] Step S1-6: Set i = 1, 2, 3, 4 respectively, and repeat steps S1-1, S1-2, S1-3, S1-4, S1-5;

[0095] Step S1-7: Insert data e and end;

[0096] Step S2: Query the frequency of data e from the two-dimensional array to complete the frequency estimation of data e; as Figure 4 shown.

[0097] Step S2-1: Start querying from the low-frequency two-dimensional array L, calculate the hash value Hash i (e), i ∈ {0, 1, 2, 3, 4}, and this hash value is the column number D that maps data e to the low-frequency two-dimensional array L L，i , then locate the counter L[i][D L，i and bit F[i][D L，i ;

[0098] Step S2-2: Determine whether the values stored in the bit F[i][D L，i located in step S2-1 are all 1. If not, take out the values of the counters of the low-frequency two-dimensional array L corresponding to all non-1 bits on the bitmap F; then compare and return the minimum value, which is the frequency of e;

[0099] Step S2-3: If the values stored in the bit F[i][D L，i located in step S2-1 are all 1, it means that the low-frequency two-dimensional array L is all overflowed. Then query in the medium-frequency two-dimensional array M, and divide the hash value Hash i (e) of data e by 2 and round down to obtain the column number D that maps data e to the medium-frequency two-dimensional array M M，i , locate the counter M[i][D M，i and bit S[i][D M，i ;

[0100] Step S2-4: Determine the bit S[i][DM，i Whether all the values stored on it are 1. If not, take out the values of the counters of the intermediate-frequency two-dimensional array M corresponding to all non-1 on the bitmap S, and multiply them by the maximum threshold MAX of the counters in the low-frequency two-dimensional array L Then add the value stored in the counter of the corresponding low-frequency two-dimensional array mapped by the same hash function, that is, M[i][D M，i *MAX L +L[i][D L，i ; Finally, compare and return the minimum value, which is the occurrence frequency of e;

[0101] Step S2-5: If the value stored on the bit S[i][D M，i is all 1, it means that the entire intermediate-frequency two-dimensional array M has overflowed. Then query in the high-frequency two-dimensional array H, and map the data e to the column number D in the intermediate-frequency two-dimensional array M M，i Divide by 2 and round down to get the column number D where the data e is mapped to the high-frequency two-dimensional array H, that is H，i , that is Locate the counter H[i][D H，i , then compare and take out the minimum value, and use the smallest H[i][D H，i for calculation. Finally, the obtained occurrence frequency of e is (H[i][D H，i *MAX M +M[i][D M，i )*MAX L +L[i][D L，i ;

[0102] Step S3: Delete the insertion record of a data e from the two-dimensional array; as Figure 5 shown.

[0103] Step S3-1: Start deleting from the low-frequency two-dimensional array L. Set i = 0, calculate the hash value Hash0(e) of the data e, and this hash value is the column number D where the data e is mapped to the low-frequency two-dimensional array L L，0 , locate the counter L[0][D L，0 and the bit F[0][D L，0 ;

[0104] Step S3-2: If the value stored in the counter L[0][D L，0 is greater than 0, then subtract 1 from the value stored in the counter L[0][D L，0 , and then jump to S3-6; if the value stored in the counter L[0][D L，0 is equal to 0, check the bit F[0][D L，0 , if the bit F[0][D L，0If it is equal to 0, a deletion exception is thrown. If bit F[0][D L，0 is equal to 1, the value stored in counter L[0][D L，0 is modified to MAX L -1, and step S3-3 is continued to delete from the intermediate-frequency two-dimensional array M;

[0105] Step S3-3: Divide D L，i by 2 and round down to obtain the position in the intermediate-frequency two-dimensional array mapped by the hash value of data e calculated using the hash function. Starting from i = 0, the number of mapped columns is D M，0 = [D L，0 * (1 / 2)」, locate counter M[0][D M，0 and bit S[0][D M，0 ;

[0106] Step S3-4: If the value stored in counter M[0][D M，0 is greater than 0, then subtract 1 from the value stored in counter M[0][D M，0 . After the -1 operation, if the value stored in M[0][D M，0 is not 0, then jump to S3-6; after the -1 operation, if the value stored in M[0][D M，0 is 0, then determine whether S[0][D M，0 is 0. If S[0][D M，0 is 0, then set F[0][D L，0 in bitmap F to 0, and then jump to S3-6; if S[0][D M，0 is not 0, then modify the value stored in counter M[0][D M，0 to MAX M -1, and then divide D M，i by 2 and round down to obtain the position in the high-frequency two-dimensional array H mapped by the hash value of data e calculated using the hash function. Starting from i = 0, the number of mapped columns is Locate counter H[0][D H，0 , subtract 1 from the value stored in counter H[0][D H，0 . After the -1 operation, if the value stored in H[0][D H，0 is not 0, then jump to S3-6; after the -1 operation, if the value stored in H[0][D H，0 is 0, then set S[0][D M，0 in bitmap S to 0 and then jump to S3-6;

[0107] Step S3-5: If the value of counter M[0][D M，0 is equal to 0, check bit S[0][D M，0 . If bit S[0][DM，0 If it is equal to 0, then set F[0][D] in the bitmap F to 0 and then jump to S3-6; if S[0][D L，0 is equal to 1, then modify the value stored in the counter M[0][D M，0 to MAX M，0 -1, and continue to execute step S3-5; M -1, continue to execute step S3-5;

[0108] Step S3-5: Divide D M，i by 2 and round down to obtain the position in the high-frequency two-dimensional array H where the hash value of the data e calculated by the hash function is mapped. Starting from i = 0, the number of mapped columns is locate the counter H[0][D H，0 , and subtract 1 from the value stored in the counter H[0][D H，0 . After the -1 operation, if the value stored in H[0][D H，0 is not 0, then jump to S3-6. After the -1 operation, if the value stored in H[0][D H，0 is 0, then set S[0][D] in the bitmap S M，0 to 0, and then jump to S3-6;

[0109] Step S3-6: Set i = 1, 2, 3, 4 respectively, and repeat steps S3-1, S3-2, S3-3, S3-4, S3-5;

[0110] Step S3-7: Delete and end;

[0111] Embodiment 1

[0112] Suppose there is a table in the database (the table structure is shown in Table 1) that records the movie-on-demand records of a certain evening movie-on-demand platform within a week. Use the method of this application to estimate the frequency of the Movie_code column in the table. In this example, the number of rows of the three two-dimensional arrays L, M, and H is 5, the number of columns of L is 20, the number of columns of M is 10, and the number of columns of H is 5. Suppose the hash functions used are: Hash0(e) = (e + 1) % 20; Hash1(e) = (e + 4) % 20; Hash2(e) = (e + 7) % 20; Hash3(e) = (e + 9) % 20; and Hash4(e) = (e + 10) % 20 (explanation: % is the modulo operator, such as 5 % 4 = 1). Here, set MAX H 、MAX M and MAX L to 2 respectively.

[0113] Table 1 Movie-on-demand record table

[0114] Movie Code Movie Name Release Date 155934 The Shawshank Redemption 2020-3-29 278955 Forrest Gump 2020-3-30 466008 Jaws 2020-3-31 155934 The Shawshank Redemption 2020-4-1 962377 The Crazy Alien 2020-4-2 278955 Forrest Gump 2020-4-3 155934 The Shawshank Redemption 2020-4-4

[0115] Insert the first piece of data 155934 in Table 1:

[0116] Step S1-1: Insert the first piece of data 155934, set i = 0, and calculate the hash value D of data 155934 L，0 =(155934 + 1) % 20 = 15, locate the counter L[0]

[15] and bit F[0]

[15] ;

[0117] Step S1-2: The value stored on counter L[0]

[15] is 0, 0 + 1 is less than MAX L , modify the value stored on the counter to 1;

[0118] Note: The insertion position needs to be found for each row. Since i starts from 0, i must take values 1, 2, 3, 4 in sequence to complete the insertion of one piece of data, that is, use Hash0(e), Hash1(e), Hash2(e), Hash3(e), Hash4(e), and each row uses a hash function to hash out a position.

[0119] Step S1-6: Set i = 1, 2, 3, 4 respectively. Repeat steps S1-1, S1-2, S1-3, S1-4, S1-5;

[0120] Step S1-7: The insertion of the first piece of data 155934 ends;

[0121] The insertion method of the remaining data is the same as the above steps. After the 6th piece of data 278955 in Table 1 is inserted, the obtained result is as Figure 6 shown. At this time, there is no 1 in bitmap F and bitmap S, indicating that there is no carry at present. However, when inserting the 7th piece of data 155934 (the last piece of data) in Table 1, Hash0(155934) = 15, Hash1(155934) = 18, Hash2(155934) = 1, Hash3(155934) = 3, Hash4(155934) = 4. After locating L[0]

[15] , L[1]

[18] , L[2][1], L[3][3], L[4][4], it is found that the values of these positions are all 2; because 2 + 1 = 3 is greater than MAX L (MAX L= 2), at this time, the counter overflows and a carry is required. Modify the values stored in L[0]

[15] , L[1]

[18] , L[2][1], L[3][3], L[4][4] and the values stored in F[0]

[15] , F[1]

[18] , F[2][1], F[3][3], F[4][4] to 1. In addition, the values stored in M[0][7], M[1][9], M[2][0], M[3][1], M[4][2] need to be incremented by 1, and after the increment, they are all not greater than MAX M , then increment the value stored in the corresponding counter in M by 1, and the final result is as Figure 7 shown.

[0122] The result after all the data in Table 1 are inserted is as Figure 7 shown.

[0123] Query data 466008:

[0124] Step S2-1: Query data 466008. First, start from i = 0, i ∈ {0, 1, 2, 3, 4}, and calculate the hash value D of data 466008 L，0 = (466008 + 1) % 20 = 9, locate the counter L[0][9] and bit F[0][9]; D L.1 = (466008 + 4) % 20 = 12, locate the counter L[1]

[12] and bit F[1]

[12] ; D L，2 = (466008 + 7) % 20 = 15, locate the counter L[2]

[15] and bit F[2]

[15] ; D L，3 = (466008 + 9) % 20 = 17, locate the counter L[3]

[17] and bit F[3]

[17] ; D L，4 = (466008 + 10) % 20 = 18, locate the counter L[4]

[18] and bit F[4]

[18] ;

[0125] Step S2-2: Determine whether the values stored in F[0][9], F[1]

[12] , F[2]

[15] , F[3]

[17] , F[4]

[18] are all 1; after checking, it is found that the values stored in these positions are all 0, indicating that there is no overflow in L;

[0126] Step S2-3: Take out the values stored in L[0][9], L[1]

[12] , L[2]

[15] , L[3]

[17] , L[4]

[18] , from Figure 7 It can be seen that the values stored in this embodiment are all 1, and then compare the sizes. The minimum value 1 is the frequency of the occurrence of data 466008, and the frequency estimate value of 466008 is 1;

[0127] Delete data 466008:

[0128] Step S3-1: Set i = 0, calculate the hash value D of data 466008 L，0 =(466008 + 1)%20 = 9, locate the counter L[0][9] and bit F[0][9];

[0129] Step S3-2: As can be seen from the above figure, the value of counter L[0][9] is greater than 0 at this time. Subtract 1 from the value of counter L[0][D L，0 ;

[0130] Step S3-6: Set i = 1, 2, 3, 4 respectively, and repeat steps S3-1, S3-2, S3-3, S3-4, S3-5;

[0131] Step S3-7: Deletion ends;

[0132] Specific practical application: The query operation is mainly divided into in-memory search and out-of-memory search. When the target data does not exist in memory, it is necessary to search in the out-of-memory. There is a large gap in the response speed between the two. The search speed in memory is much higher than that in the out-of-memory. Therefore, try to move the data with high access frequency to memory to improve the memory hit rate and thus make the query efficiency higher. All movie-on-demand situations can be obtained from Table 1, the movie-on-demand record table. Through the estimation method of the present invention, the on-demand frequency of each movie can be estimated, and the data of several movies with the highest on-demand frequency are moved to memory. Here, "The Shawshank Redemption" can be moved to memory. By estimating the access frequency of the data in the database, the separation of hot and cold data can be realized. Define some data with the most frequent access (related to the access frequency, and the hot data threshold top-k can be set according to the actual situation according to the access frequency, and the first k elements with the highest access frequency are put into memory) as hot data and move them to memory, which can effectively relieve the pressure on the database and improve the query efficiency, and realize the separation of hot and cold data.

[0133] Note: top-k means to find the first k largest or smallest elements in the data set.

[0134] The present invention is an improvement on Count-Min Sketch (abbreviated as CMS), so it is applicable to all usage scenarios of Count-Min Sketch, including natural language processing, data stream statistics, calculation of point mutual information, sparse approximation of compressive sensing, detection of network abnormal flows, processing of distributed data sets, etc., and can improve the operating pressure of the database and improve the utilization rate of memory space.

[0135] From Figure 8It can be seen that the number of columns of the traditional CMS is set to 2048 columns, 5 hash functions, the size of each counter is 4 bytes, and the maximum threshold of each counter is 2 32 , and the total occupied space of all counters is 40K. The estimation method of the present invention is abbreviated as My-CMS, and the size of the counters in the L layer, M layer, and H layer can be freely adjusted. When the number of columns in the L layer is also 2048 columns, by reasonably setting the size of the counters in each layer, the occupied space can be reduced. In this embodiment, the size of the counters in the L layer is set to 1 byte, the size of the counters in the M layer is set to 2 bytes, and the size of the counters in the H layer is set to 2 bytes. The maximum representable range of the estimation method of the present invention is 2 32 +2 24 +2 8 , and the total occupied space of all counters is 25K. Compared with the traditional CMS, the present invention is still superior in processing data volume when the used space is less than 37.5%. When the same memory is used, the number of columns in the L layer can be appropriately increased, which can reduce the mutual conflict between hash functions and make the accuracy of the present invention higher.

[0136] Embodiment 2

[0137] In this embodiment, the database frequency estimation method in the data stream statistics is applied in the data stream statistics to estimate the frequency of data in the data stream. The specific steps are as follows:

[0138] Step 1: Find the data stream to be processed;

[0139] Step 2: Initialize the data structure of the present application, set the values of all counters in the L layer, M layer, and H layer of the data structure to 0, and at the same time set the values of bitmap F and bitmap S to 0, and insert the data in the data stream into the data structure in turn;

[0140] Step 3: When inserting the first data e, set i = 0, start calculating from the low-frequency two-dimensional array L, and use the hash function to calculate the hash value of the data e. This hash value is the column number D where the data e is mapped to the low-frequency two-dimensional array L L，i , D L，0 =Hash0(e), locate the counter L[0][D L，0 and bit F[0][D L，0 ;

[0141] Step 4: Determine whether the value stored on the counter L[0][D L，0 +l is greater than MAX L , if not, add 1 to the value stored on the counter L[0][D L，0 , and then jump to step 8 to repeat the hash calculation of other rows; if the counter L[0][DL，0 The value stored in [] + l is greater than MAX L , indicating that the counter has overflowed. Then set the values stored in counter L[0][D L，0 and bit F[0][D L，0 to 1, and continue to execute step 5 to determine whether the intermediate frequency two-dimensional array has overflowed;

[0142] Step 5: Divide D L，i by 2 and round down to obtain the position in the intermediate frequency two-dimensional array where the hash value of data e is mapped using the hash function. Starting from i = 0, the number of mapped columns is D M，0 , then the positions of data e in the intermediate frequency two-dimensional array M and the intermediate frequency bitmap S are: Locate the counter M[0][D M，0 and bit S[0][D M，0 ;

[0143] Step 6: Determine whether the value stored in counter M[0][D M，0 + 1 is greater than MAX M . If it is not greater, then increment the value stored in counter M[0][D M，0 , and then jump to step 8 to repeat the hash calculation for other rows; If the value stored in counter M[0][D M，0 + 1 is greater than MAX M , indicating that the counter has overflowed. Then set the values stored in counter M[0][D M，0 and bit S[0][D M，0 to 1, and continue to execute step 7 to determine whether the high-frequency two-dimensional array H has overflowed;

[0144] Step 7: Divide D M，i by 2 and round down to obtain the position in the high-frequency two-dimensional array H where the hash value of data e is mapped using the hash function. Starting from i = 0, the number of mapped columns Locate the counter H[0][D H，0 , and then determine whether the value stored in counter H[0][D H，0 + 1 is greater than MAX H . If it is not greater, then increment the value stored in counter H[0][D H，0 ; If the value stored in counter H[0][D H，0 + 1 is greater than MAX H , then throw an exception, and it is necessary to increase the MAX H value of the counter and jump to step 9;

[0145] Step 8: Set i = 1, 2, 3, 4 respectively, and repeat steps 3, 4, 5, 6, 7;

[0146] Step 9, the data e insertion ends;

[0147] Step 10, subsequently, every time data is inserted, repeat Steps 3 to 9 until all data in the data stream is inserted into the data structure;

[0148] Step 11, after all data is inserted, start calculating the frequency of data occurrences in the data stream;

[0149] Step 12, first query the first data e, start querying from the low-frequency two-dimensional array L, and calculate the hash value Hash i (e), i ∈ {0, 1, 2, 3, 4}, and this hash value is the column number D in the low-frequency two-dimensional array L to which the data e is mapped L，i , then locate the counter L[i][D L，i and the bit F[i][D L，i ;

[0150] Step 13, determine whether the values stored on the bit F[i][D L，i located in Step 12 are all 1. If they are not all 1, take out the values of the counters in the low-frequency two-dimensional array L corresponding to all non-1s on the bitmap F; then compare and return the minimum value, which is the frequency of e;

[0151] Step 14, if the values stored on the bit F[i][D L，i located in Step 12 are all 1, it means that the entire low-frequency two-dimensional array L is overflowed, then query in the medium-frequency two-dimensional array M, and divide the hash value Hash i (e) of the data e by 2 and take the floor to obtain the column number D in the medium-frequency two-dimensional array M to which the data e is mapped M，i , locate the counter M[i][D M，i and the bit S[i][D M，i ;

[0152] Step 15, determine whether the values stored on the bit S[i][D M，i located in Step 14 are all 1. If they are not all 1, take out the values of the counters in the medium-frequency two-dimensional array M corresponding to all non-1s on the bitmap S, multiply by the maximum threshold MAX L of the counters in the low-frequency two-dimensional array and then add the values stored in the counters in the low-frequency two-dimensional array corresponding to the same hash function mapping, that is, M[i][D M，i *MAX L +L[i][D L，i ; finally, compare and return the minimum value, which is the frequency of e;

[0153] Step 16: If the values stored at bit S[i][D M，i are all 1, it indicates that the entire intermediate-frequency two-dimensional array M has overflowed. Then, query in the high-frequency two-dimensional array H, and map the data e to the column number D in the intermediate-frequency two-dimensional array M M，i Divide the column number D by 2 and round down to obtain the column number D where the data e is mapped in the high-frequency two-dimensional array H H，i , that is Locate the counter H[i][D H，i , then compare and extract the minimum value, and use the minimum H[i][D H，i for calculation. Finally, the frequency of the occurrence of e is obtained as (H[i][D H，i *MAx M +M[i][D M，i )*MAX L +L[i][D L，i ;

[0154] Step 17: Obtain the frequency of the first data e;

[0155] Step 18: Repeat the subsequent data according to Steps 12 to 16 until the frequencies of all data in the data stream are finally obtained.

[0156] The parts not described in this invention are applicable to the prior art.

Claims

1. A database frequency estimation method, characterized in that, The estimation method includes the following: The data structure of the database includes three two-dimensional arrays, namely the low-frequency two-dimensional array L, the medium-frequency two-dimensional array M, and the high-frequency two-dimensional array H. The number of rows of the three two-dimensional arrays is equal. The number of columns of the high-frequency two-dimensional array H is 1 / 2 of the number of columns of the medium-frequency two-dimensional array M, and the number of columns of the medium-frequency two-dimensional array M is 1 / 2 of the number of columns of the low-frequency two-dimensional array L. The data structure also includes two bitmaps, namely the low-frequency bitmap F and the medium-frequency bitmap S. The number of rows and columns of the low-frequency bitmap F and the medium-frequency bitmap S is the same as the number of rows and columns of the low-frequency two-dimensional array L and the medium-frequency two-dimensional array M respectively. The structure of the two-dimensional array is composed of storage units of a fixed size, which are called counters. The position of the counter in the two-dimensional array is located according to the row number and column number of the two-dimensional array, and the counter is used to record the frequency of the data. The maximum thresholds of the respective counters of the three two-dimensional arrays, namely the low-frequency two-dimensional array L, the medium-frequency two-dimensional array M, and the high-frequency two-dimensional array H, can be the same or different, and the maximum thresholds of all counters in the same layer are the same. In the structure of the bitmap, the size of each storage unit is 1 bit, and these storage units are called bits. The values in the bitmap can only be 0 and 1. The positions of these bits in the bitmap are determined by the row number and column number of the bitmap. The low-frequency bitmap F and the medium-frequency bitmap S are respectively used to record the overflow conditions of the counters in L and M. Each counter L[i][j] in L can find the corresponding bit F[i][j] in the low-frequency bitmap F using the same row number i and column number j. If the value stored in bit F[i][j] is 0, it means the counter has not overflowed, indicating that there is no carry at the corresponding position in L. The value corresponding to the current position in L is taken as the frequency. If the value stored in bit F[i][j] is 1, it means the counter has overflowed and a carry has occurred, and then a hash calculation is performed on M. If the value stored in bit S[i][j] is 0, it means there is no carry at the corresponding position in M, and the frequency at this time is M[i][D M,i *MAX L +L[i][D L，i ; If the value stored in bit S[i][j] is 1, it means there is a carry at the corresponding position in M, and then a hash calculation is performed on H. The frequency at this time is (H[i][D H,i *MAX M +M[i][D M,i )*MAXL+L[i][D L，i , and the frequency estimation is completed here.

2. The database frequency estimation method according to claim 1, wherein The number of rows of the three two-dimensional arrays L, M, and H is 5. The number of columns of H is 1 / 2 of the number of columns of M, and the number of columns of M is 1 / 2 of the number of columns of L. The number of columns of L is default 2048. There are 5 hash functions. The size of the counters in layer L is set to 1 byte, the size of the counters in layer M is set to 2 bytes, and the size of the counters in layer H is set to 2 bytes. The total occupied space size of all counters is 25K.

3. The database frequency estimation method according to claim 1, wherein The estimation method includes processes of inserting data, querying data, and deleting data, and sets the maximum thresholds of the counters in the high-frequency, medium-frequency, and low-frequency two-dimensional arrays to MAX H , MAX M , and MAX L , respectively. The specific steps are as follows: Step S1: Insert the data e into the two-dimensional array. Before inserting the first data, initialize all the two-dimensional arrays and bitmaps, that is, the values of each counter and bit are initialized to 0. Step S1-1: Set i = 0, start calculating from the low-frequency two-dimensional array L, use the hash function to calculate the hash value of the data e, and this hash value is the column number D in the low-frequency two-dimensional array L where the data e is mapped L，i , D L，0 = Hash0(e), locate the counter L[0][D L，0 and the bit F[0][D L，0 ; Step S1-2: Determine whether the value stored in counter L[0][D L，0 plus 1 is greater than MAX L . If it is not greater, increment the value stored in counter L[0][D L,0 by 1, then jump to step S1-6 to repeat the hash calculation for other lines; if the value stored in counter L[0][D L,0 plus 1 is greater than MAX L , it means the counter overflows, then modify the values stored in both counter L[0][D L,0 and bit F[0][D L，0 to 1, and continue to execute step S1-3 to determine whether the intermediate frequency two-dimensional array M overflows; Step S1-3: Divide D L,i by 2 and round down to obtain the position in the intermediate-frequency two-dimensional array M where the hash value of data e calculated using the hash function is mapped. Starting from i = 0, the number of columns mapped is D M，0 , then the positions of data e in the intermediate-frequency two-dimensional array M and the intermediate-frequency bitmap S are: Locate the counter M[0][D M，0 and bit S[0][D M,0 ; Step S1-4: Determine whether the value stored in counter M[0][D M，0 plus 1 is greater than MAX M . If it is not greater, increment the value stored in counter M[0][D M，0 by 1, then jump to step S1-6 to repeat the hash calculation for other lines; if the value stored in counter M[0][D M，0 plus 1 is greater than MAX M , it means the counter overflows. Then modify the values stored in both counter M[0][D M，0 and bit S[0][D M,0 to 1, and continue to execute step S1-5 to determine whether the high-frequency two-dimensional array H overflows; Step S1-5: Divide D M，i by 2 and round down to obtain the position in the high-frequency two-dimensional array H where the hash value of data e calculated using the hash function is mapped. Starting from i = 0, the number of columns of the mapping locate the counter H[0][D H,0 , then determine whether the value stored on the counter H[0][D H,0 +1 is greater than MAX H . If it is not greater, increment the value stored on the counter H[0][D H,0 by 1; if the value stored on the counter H[0][D H,0 +1 is greater than MAX H , then throw an exception and need to increase the value of MAX of the counter, and jump to step Sl-7; H ​ Step S1-6: Set i = 1, 2, 3, 4 respectively, and repeat steps Sl-1, Sl-2, Sl-3, S1-4, S1-5. Step Sl-7: The insertion of the data e ends. Step S2: Query the frequency of the data e appearing in the two-dimensional array to complete the frequency estimation of the data e. Step S2-1: Start querying from the low-frequency two-dimensional array L, and calculate the hash value Hash i (e), where i ∈ {0, 1, 2, 3, 4}. This hash value is the number of columns D in the low-frequency two-dimensional array L to which the data e is mapped L，i , then locate the counter L[i][D L，i and the bit F[i][D L，i ; Step S2-2: Determine whether the values stored at the position F[i][D obtained by positioning in Step S2-1 L，i are all 1. If they are not all 1, take out the counter values of all the low-frequency two-dimensional arrays L corresponding to the non-1s on the low-frequency bitmap F; then compare them and return the minimum value, which is the frequency of occurrence of e; Step S2-3. If the values stored at the positions F[i][D L，i obtained by positioning in Step S2-1 are all 1, it indicates that the entire low-frequency two-dimensional array L has overflowed. Then, query in the medium-frequency two-dimensional array M, and divide the hash value Hash i (e) of data e by 2 and take the floor to obtain the column number D M,i to which data e is mapped in the medium-frequency two-dimensional array M , and locate the counter M[i][D M,i and the bit S[i][D M，i ; Step S2-4: Determine whether the values stored at the position S[i][D obtained by the positioning in step S2-3 M，i are all 1. If they are not all 1, take out the values of the counters of all the intermediate-frequency two-dimensional arrays M corresponding to the non-1 positions on the intermediate-frequency bitmap S, multiply them by the maximum threshold value MAX of the counters in the low-frequency two-dimensional array L and then add the values stored in the counters of the low-frequency two-dimensional arrays corresponding to the same hash function mapping, that is, M[i][D M，i *MAX L +L[i][D L，i ; Finally, compare and return the minimum value, which is the occurrence frequency of the data e Step S2-5. If the values stored at the positions S[i][D M,i are all 1, it indicates that the intermediate-frequency two-dimensional array M is completely overflowed. Then, query in the high-frequency two-dimensional array H, and map the data e to the column number D in the intermediate-frequency two-dimensional array M M,i . Divide the column number D by 2 and round down to obtain the column number D where the data e is mapped to the high-frequency two-dimensional array H H，i , that is . Locate the counter H[i][D H，i , then compare and take the minimum value. Use the minimum H[i][D H，i for calculation. Finally, the frequency of e occurrence is obtained as (H[i][D H，i *MAX M +M[i][D M，i )*MAX L +L[i][D L，i ; Step S3: Delete the insertion record of a data e from the two-dimensional array. Step S3-1: Start deleting from the low-frequency two-dimensional array L. Set i = 0, calculate the hash value Hash0(e) of the data e, and this hash value is the column number D in the low-frequency two-dimensional array L where the data e is mapped L，0 , and locate the counter L[0][D L，0 and the bit F[0][D L，0 ; Step S3-2. If the value stored in counter L[0][D L,0 is greater than 0, decrement the value stored in counter L[0][D L,0 by 1, and then jump to S3-6. If the value stored in counter L[0][D L,0 is equal to 0, check bit F[0][D L，0 . If bit F[0][D L，0 is equal to 0, throw a deletion exception. If bit F[0][D L，0 is equal to 1, modify the value stored in counter L[0][D L,0 to MAX L - 1, and continue to execute Step S3-3 to delete from the intermediate frequency two-dimensional array M. Step S3-3: Divide D L,i by 2 and round down to obtain the position in the intermediate frequency two-dimensional array where the hash value of data e calculated using the hash function is mapped. Starting from i = 0, the number of mapped columns is Locate the counter M[0][D M,0 and bit S[0][D M,0 ; Step S3-4. If the value stored in counter M[0][D M,0 is greater than 0, then decrement the value stored in counter M[0][D M,0 by 1. After the -1 operation, if the value stored in M[0][D M,0 is not 0, then jump to S3-6. After the -1 operation, if the value stored in M[0][D M,0 is 0, then determine whether S[0][D M,0 is 0. If S[0][D M,0 is 0, then set F[0][D L，0 in the low-frequency bitmap F to 0, and then jump to S3-6. If S[0][D M,0 is not 0, then modify the value stored in counter M[0][D M,0 to MAX M - 1, and then divide D M,i by 2 and round down to obtain the position in the high-frequency two-dimensional array H where the hash value of data e calculated using the hash function is mapped. Starting from i = 0, the number of mapped columns is Locate counter H[0][D H,0 , and decrement the value stored in counter H[0][D H,0 by 1. After the -1 operation, if the value stored in H[0][D H,0 is not 0, then jump to S3-6. After the -1 operation, if the value stored in H[0][D H,0 is 0, then set S[0][D M，0 in the medium-frequency bitmap S to 0 and then jump to S3-6; Step S3-5: If the value of counter M[0][D M,0 is equal to 0, check bit S[0][D M，0 . If bit S[0][D M，0 is equal to 0, set F[0][D L，0 in the low-frequency bitmap F to 0 and then jump to S3-6; if S[0][D M,0 is equal to 1, modify the stored value of counter M[0][D M,0 to MAX M - 1, and continue to execute step S3-5; Step S3-5: Divide D M,i by 2 and round down to obtain the position in the high-frequency two-dimensional array H where the hash value of data e calculated using the hash function is mapped. Starting from i = 0, the number of mapped columns is Locate the counter H[0][D H,0 , subtract 1 from the value stored in the counter H[0][D H,0 . If the value stored in H[0][D H,0 after the -1 operation is not 0, then jump to S3-6; if the value stored in H[0][D H,0 after the -1 operation is 0, then set S[0][D M，0 in the medium-frequency bitmap S to 0, and then jump to S3-6; Step S3-6: Set i = 1, 2, 3, 4 respectively, and repeat steps S3-1, S3-2, S3-3, S3-4, S3-5. Step S3-7: The deletion ends.

4. The database frequency estimation method according to claim 1, characterized in that The estimation method can estimate the access frequency of the data in the database, realize the separation of hot and cold data. Set the hot data threshold top-k, and define the top k elements with the largest access frequencies as hot data and move them to the memory to realize the separation of hot and cold data, relieve the pressure on the database, and improve the query efficiency.

5. The database frequency estimation method according to claim 1, characterized in that The estimation method is an improvement of the Count-Min Sketch data structure and can be applied to natural language processing, data stream statistics, calculation of point mutual information, sparse approximation of compressive sensing, detection of network abnormal flows, and processing of distributed datasets, which can improve the operating pressure of the database and increase the utilization rate of memory space.

6. A method for estimating database frequency in data stream statistics, and the steps of the estimation method are specifically as follows: Step 1: Locate the data stream to be processed. Step 2: Initialize the data structure described in claim 1, set the values of all counters in the L layer, M layer, and H layer of the data structure to 0, and at the same time set the values of the low-frequency bitmap F and the medium-frequency bitmap S to 0, and insert the data in the data stream into the data structure in sequence. Step 3: When inserting the first piece of data e, set i = 0, start calculating from the low-frequency two-dimensional array L, and use the hash function to calculate the hash value of the data e. This hash value is the column number D in the low-frequency two-dimensional array L to which the data e is mapped L，i , D L，0 = Hash0(e), and locate the counter L[0][D L，0 and the bit F[0][D L，0 ; Step 4: Determine whether the value stored in counter L[0][D L，0 plus 1 is greater than MAX L . If it is not greater, increment the value stored in counter L[0][D L，0 by 1, then jump to Step 8 and repeat the hash calculation for other lines; if the value stored in counter L[0][D L，0 plus 1 is greater than MAX L , it indicates that the counter has overflowed. Then modify the values stored in both counter L[0][D L，0 and bit F[0][D L，0 to 1, and continue to execute Step 5 to determine whether the intermediate frequency two-dimensional array has overflowed; Step 5. Divide D L，i by 2 and round down to obtain the position in the intermediate frequency two-dimensional array mapped by the hash value of data e calculated using the hash function. Starting from i = 0, the number of mapped columns is D M,0 , then the positions of data e in the intermediate frequency two-dimensional array M and the intermediate frequency bitmap S are: locate the counter M[0][D M,0 and bit S[0][D M,0 ; Step 6. Determine whether the value stored in counter M[0][D M,0 +1 is greater than MAX M . If it is not greater, increment the value stored in counter M[0][D M,0 by 1, then jump to Step 8 to repeat the hash calculation for other rows; if the value stored in counter M[0][D M,0 +1 is greater than MAX M , it indicates that the counter has overflowed. Then modify the values stored in both counter M[0][D M,0 and bit S[0][D M,0 to 1, and continue to execute Step 7 to determine whether the high-frequency two-dimensional array H has overflowed; Step 7. Divide D M,i by 2 and round down to obtain the position in the high-frequency two-dimensional array H where the hash value of data e calculated using the hash function is mapped. Starting from i = 0, the number of columns D H,O = [D M,0 *(1 / 2)」, locate the counter H[0][D H,0 , then determine whether the value stored on the counter H[0][D H,0 +1 is greater than MAX H . If it is not greater, increment the value stored on the counter H[0][D H,0 by 1; if the value stored on the counter H[0][D H,0 +1 is greater than MAX H , then throw an exception, and it is necessary to increase the value of MAX of the counter and jump to Step 9; H ​ Step 8: Respectively set i = 1, 2, 3, 4, and repeat steps 3, 4, 5, 6, and 7. Step 9: The insertion of data e ends. Step 10: Subsequently, whenever data is inserted, repeat steps 3 to 9 until all data in the data stream is inserted into the data structure. Step 11: After all data in the data stream is inserted, start calculating the frequency of the data appearing in the data stream. Step 12: First, query the first data e. Start querying from the low-frequency two-dimensional array L, and calculate the hash value Hash i (e), where i ∈ {0, 1, 2, 3, 4}. This hash value is the number of columns D in the low-frequency two-dimensional array L to which the data e is mapped L，i , then locate the counter L[i][D L，i and the bit F[i][D L，i ; Step 13. Determine whether the values stored at bit F[i][D L，i obtained by positioning in Step 12 are all 1. If they are not all 1, take out the counter values of the low-frequency two-dimensional arrays L corresponding to all non-1 values on the low-frequency bitmap F; then return the minimum value after comparison, which is the frequency of occurrence of e; Step 14: If the values stored at bit F[i][D L，i are all 1, it indicates that the entire low-frequency two-dimensional array L has overflowed. Then, query the medium-frequency two-dimensional array M, and divide the hash value Hash i (e) of data e by 2 and round down to obtain the column number D M，i to which data e is mapped in the medium-frequency two-dimensional array M , and locate the counter M[i][D M，i and bit S[i][D M，i ; Step 15: Determine whether the values stored at the position S[i][D M，i are all 1. If they are not all 1, take out the counter values of the intermediate-frequency two-dimensional array M corresponding to all non-1s on the intermediate-frequency bitmap S, and multiply them by the maximum threshold MAX of the counter in the low-frequency two-dimensional array L Then add the counter values stored in the low-frequency two-dimensional array corresponding to the same hash function mapping, that is, M[i][D M，i *MAX L +L[i][D L，i ; Finally, compare and return the minimum value, which is the occurrence frequency of e; Step 16: If the values stored at the bit S[i][D M，i are all 1, it indicates that the entire intermediate-frequency two-dimensional array M has overflowed. Then, query in the high-frequency two-dimensional array H, and map the data e to the column number D of the intermediate-frequency two-dimensional array M M，i Divide the column number D by 2 and round down to obtain the column number D where the data e is mapped to the high-frequency two-dimensional array H H，i , that is Locate the counter H[i][D H，i , and then compare to take the minimum value. Use the minimum H[i][D H，i for calculation. Finally, the frequency of e appearing is (H[i][D H，i *MAX M +M[i][D M，i )*MAX L +L[i][D L，i ; Step 17: Obtain the frequency of the first data e. Step 18: The subsequent data is repeated according to steps 12 to 16, and finally the frequencies of all data in the data stream are obtained.

7. A computer-readable storage medium storing computer instructions, characterized in that, When the computer instructions are executed by one or more processors, the one or more processors are caused to execute the steps in the estimation method described in any one of claims 1-6.

8. A data processing system, characterized in that, The data structure of the database in the data processing system includes three two-dimensional arrays, namely, the low-frequency two-dimensional array L, the medium-frequency two-dimensional array M, and the high-frequency two-dimensional array H. The number of rows of the three two-dimensional arrays is equal. The number of columns of the high-frequency two-dimensional array H is 1 / 2 of the number of columns of the medium-frequency two-dimensional array M, and the number of columns of the medium-frequency two-dimensional array M is 1 / 2 of the number of columns of the low-frequency two-dimensional array L; the data structure also includes two bitmaps, namely, the low-frequency bitmap F and the medium-frequency bitmap S. The number of rows and columns of the low-frequency bitmap F and the medium-frequency bitmap S are the same as the number of rows and columns of the low-frequency two-dimensional array L and the medium-frequency two-dimensional array M respectively. The structure of the two-dimensional array is composed of fixed-size storage units, and these storage units are called counters. The position of the counter in the two-dimensional array is located according to the row number and column number of the two-dimensional array, and the counter is used to record the frequency of the data; the maximum thresholds of all counters in the same layer are the same. In the structure of the bitmap, the size of each storage unit is 1 bit, and these storage units are called bits. The values in the bitmap can only be 0 and 1. The positions of these bits in the bitmap are determined by the row number and column number of the bitmap. The low-frequency bitmap F and the medium-frequency bitmap S are respectively used to record the overflow conditions of the counters in L and M.

9. The data processing system according to claim 8, wherein Each counter L[i][j] in L can find the corresponding bit F[i][j] in the low-frequency bitmap F using the same row number i and column number j. If the value stored in bit F[i][j] is 0, it means the counter has not overflowed, indicating that there is no carry at the corresponding position in L. The value corresponding to the current position in L is taken as the frequency. If the value stored in bit F[i][j] is 1, it means the counter has overflowed and there has been a carry. Then, a hash calculation is performed on M. If the value stored in bit S[i][j] is 0, it means there is no carry at the corresponding position in M. At this time, the frequency is M[i][D M，i *MAX L +L[i][D L，i ; If the value stored in bit S[i][j] is 1, it means there has been a carry at the corresponding position in M. Then, a hash calculation is performed on H. At this time, the frequency is (H[i][D H，i *MAX M +M[i][D M，i )*MAX L +L[i][D L，i .

Images