Method, device and storage medium for implementing improved bloom filter
By using a hash function and a mutually primitive array in the Bloom filter, the problem of false positives caused by a single hash function is solved, and efficient data filtering on devices with limited computing power is achieved.
Patent Information
- Application Number
- CN202311248007.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-26
- Publication Date
- 2025-04-04
AI Technical Summary
Existing Bloom filters are prone to false positives when using only a single hash function, which leads to inconvenient use on embedded devices with limited computing power.
The improved Bloom filter bit array is constructed by using a hash function and a coprime array composed of a coprime array. The hash function and coprime array are used to hash and modulo operations on the filtered sample data, generate the remainder for updating the bit array, and update the bit array through bit and operation.
When only one hash function is used, the false positive rate in filtering is reduced, the accuracy of filtering is improved, and the computing power and memory consumption of hash calculations are saved, making it suitable for embedded devices with limited computing power.
Smart Images
Figure CN119759894A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computers, and in particular to a method, a device and a storage medium for implementing an improved Bloom filter. Background Art
[0002] Bloom filters are used to quickly detect whether a specific element exists in a data set. Existing Bloom filters consist of a bit array and multiple independent hash functions. The bit arrays are initialized to 0, and the input data is calculated by each hash function to generate a hash value. The sum of the bit lengths of all hash values is the total length of the bit data. The hash values obtained by each hash function have a fixed corresponding position in the bit array. The bit AND operation is used to merge the hash bits into the corresponding bits of the bit array. When an element is to be queried, it is also calculated by the hash function to generate a hash value, and then the bit value of the corresponding bit of the hash value in the bit data is checked. If any hash value bit is 1 but 0 in the corresponding bit array, it indicates that the element is definitely not in the set; otherwise, it is considered that the element may be in the set.
[0003] The existing Bloom filter must use multiple independent hash functions because: if only one hash function is used, hash collision may occur, which may cause a non-existent element to have all corresponding bits 1, which is the so-called "false positive". Figure 1 This situation is described.
[0004] like Figure 1, suppose there are two data, M1 and M2, and the HASH value obtained by the same HASH operation is MH1, whose binary representation is 1 0 1 0 0 0 1 0 0 0 0; and MH2, whose binary representation is 0 0 0 0 1 1 1 0 0 1 0 0. The bit length is the bit length of the maximum hash value obtained by the hash algorithm, and each bit is initially 0. After M1 is merged into the bit array using bitwise AND, the cache value is 1 0 1 0 00 1 0 0 0 0 0; after M2 is merged into the bit array using bitwise AND, the bit array value is 1 0 1 0 1 11 0 0 1 0 0; suppose when judging whether a number M3 that does not exist in the set exists, its hash value is calculated as MH3. Assuming that the value of MH3 is 1 0 0 0 0 0 0 0 0 0 0, the bit values of the corresponding bits of the hash value in the bit data are all 1, and it is judged that the number M3 that does not exist in the set exists, which is a false positive. Therefore, using only a single hash may result in misjudgment. In order to reduce false positives and enable it to be used in situations where low error rates are tolerated, it is necessary to increase the number of independent hash functions and the length of the bit array. Since the calculation of hash functions consumes more system computing power, the more different hash functions are added, the greater the pressure on the system's computing power resources. Therefore, traditional Bloom filters are not suitable for use on embedded devices with limited computing power. Summary of the invention
[0005] The present invention provides a method, device and storage medium for realizing an improved Bloom filter, so as to have a more accurate filtering effect while using only one hash function, save computing power, and can be applied to embedded devices with limited computing power.
[0006] In order to achieve the above object, on the one hand, a method for implementing an improved Bloom filter is provided, using a computing device, comprising: S1, a step of creating a Bloom filter bit array, wherein the step of creating the Bloom filter bit array comprises:
[0007] S11, creating a coprime array M and a hash function H, wherein the coprime array M includes I elements m i , where 1≤i≤I, the maximum value of the hash value generated by the hash function H includes n bits;
[0008] S12, creating a bit array of length n*I, in which each n elements correspond to an element of the coprime array M, and the initial value of each element in the bit array is 0;
[0009] S13, for each filtered sample data x in the filtered sample set X, j The following processing is performed, j=1 to N, N is the number of sample filtering data included in the X:
[0010] The hash function H is used to perform hash calculation to obtain a hash value H(x j ), using each element m in the coprime array M i The hash value H(x j ) performs a modular operation to obtain the same i The corresponding remainder y i , 1≤i≤I, the remainder y i comprises n bits; and
[0011] Each of the remainders y i With the bit array and element m i Perform bitwise AND operation on the corresponding n elements to obtain a bit array incorporating the remainder;
[0012] S14, for each filtered sample data x j After the processing of step S13 is performed, the Bloom filter bit array is obtained.
[0013] Preferably, in the method, in step S12, in the n*I bit array, a group of n elements corresponding to an element in the coprime array M are continuous in position.
[0014] Preferably, the method further comprises: S2, a step of using the created Bloom filter bit array to perform data filtering, wherein the step of using the created Bloom filter bit array to perform data filtering comprises:
[0015] S21, performing hash calculation on the filtered data k using the hash function H to obtain a hash value H(k);
[0016] S22, using each element m in the coprime array M i Perform a modulus operation on the hash value H(k) to obtain the value corresponding to m i The corresponding remainder y i , 1≤i≤I, the remainder y i It includes n bits;
[0017] S23, the remainder y i Each bit of is bit-matched with the corresponding n elements in the Bloom filter bit array; if the remainder y exists i If the bit of is 1 and the corresponding bit in the Bloom filter bit array is 0, it is determined that the data k is not in the filtered sample set X.
[0018] Preferably, in the method, step S23 further comprises:
[0019] After all the remainders y iAfter the bits of are bit-matched with the corresponding n elements in the Bloom filter bit array, if for all the remainders y i If the bits in the Bloom filter bit array are all 1, then it is determined that the data k is in the filtered sample set X.
[0020] Preferably, in the method, in step S23, it is first determined whether there is a remainder y i The bit of is 1 and the corresponding bit in the Bloom filter bit array is 0; if so, it is determined that the data k is definitely not in the sample data set; otherwise, continue to judge for all remainders y i For the bits that are 1 in the Bloom filter, are the corresponding bits in the Bloom filter bit array all 1?
[0021] On the other hand, a computing device is provided, including a memory and a processor, wherein the memory stores at least one program, and the at least one program is executed by the processor to implement any of the methods described above.
[0022] On the other hand, a computer-readable storage medium is provided, wherein at least one program is stored in the storage medium, and the at least one program is executed by a processor to implement any of the methods described above.
[0023] The above technical solution has the following technical effects:
[0024] The technical solution of the embodiment of the present invention uses a hash function and a coprime array composed of coprime numbers to construct an improved Bloom filter bit array, and the length of the Bloom filter bit array is determined by the number of bits contained in the maximum hash value generated by the hash function and the number of elements contained in the coprime array; the hash function is used to perform hash calculation on the filtered sample data to generate a hash value, and the hash value is modulo operation is performed on each element in the coprime array, and the bits contained in the remainder obtained after the modulo operation are used to perform a bitwise AND operation with the initialized Bloom filter bit array to obtain a value that can be used to judge whether a data belongs to the set composed of the filtered sample data. Using the solution of the embodiment of the present invention, the false positive rate in filtering can be reduced when only one hash function is used, and the accuracy of filtering is improved. Since only one hash function is used, the cost is only to increase the length of the bit array, so that a larger computing power can be obtained with a smaller memory cost, the hash calculation computing power is saved, the memory consumption is reduced, and the computing power and memory consumption caused by using multiple hash functions are avoided. It can be used on embedded devices with limited computing power resources. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 A schematic diagram of the principle of filtering by a Bloom filter implemented by the prior art of the present invention;
[0026] Figure 2 A schematic flow chart of a method for implementing an improved Bloom filter according to an embodiment of the present invention;
[0027] Figure 3 A schematic diagram of the structure of a computing device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0028] To further illustrate the various embodiments, the present invention provides drawings. These drawings are part of the disclosure of the present invention, which are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, a person of ordinary skill in the art should be able to understand other possible implementations and advantages of the present invention. The components in the figures are not drawn to scale, and similar component symbols are generally used to represent similar components.
[0029] The present invention will now be further described with reference to the accompanying drawings and specific implementation methods.
[0030] Embodiment 1:
[0031] Figure 2 The flowchart of the method for implementing the improved Bloom filter according to one embodiment of the present invention is shown in FIG. The method for implementing the improved Bloom filter according to this embodiment can be applied to a computing device such as an embedded device. The method according to this embodiment includes: S1, a step of creating a Bloom filter bit array. Figure 2 , creating a Bloom filter bit array includes the following steps:
[0032] S11, create a coprime array M and a hash function H, the coprime array M includes I elements m i , where 1≤i≤I, the maximum value of the hash value generated by the hash function H includes n bits;
[0033] Among them, the coprime array M includes I elements m i are mutually prime numbers, that is, the greatest common divisor of these I elements is 1; i is the sequence number of the element in the array;
[0034] S12, creating a bit array of length n*I, in which each n elements correspond to an element of the coprime array M, and the initial value of each element in the bit array is 0, that is, it is initialized to 0;
[0035] Preferably, in step S12, in the n*I bit array, a group of n elements corresponding to an element in the mutually prime array M are continuous in position; an element in the mutually prime array M corresponds to a region consisting of n connected elements in the n*I bit array, each element corresponds to such a region, and different elements correspond to different regions;
[0036] S13, for each filtered sample data x in the filtered sample set X, j The following processing is performed, j=1 to N, N is the number of sample filtering data included in the X:
[0037] Use hash function H to calculate the hash value H(x j ), using each element m in the coprime array M i For the hash value H(x j ) to perform a modulo operation, and obtain the same value as m i The corresponding remainder y i , 1≤i≤I, remainder y i comprises n bits; and
[0038] Each remainder y i And element m in the bit array i The corresponding n elements are bitwise ANDed to obtain a bit array incorporating the remainder; that is, each remainder y i The n bits contained are respectively associated with element m in the above initialized bit array i Perform bitwise AND operation on the corresponding n elements;
[0039] The filtered sample set X can include multiple filtered sample data x j , x j Refers to different data in X, j is x j The number of the filtered sample data in X; the above H(x j ) contains n bits; the above modular operation is the mod operation, and the remainder y is obtained by modular operation i =H(x j )mod m i ;
[0040] S14, for each filtered sample data x j After the processing of step S13 is performed, a Bloom filter bit array is obtained.
[0041] Filter the sample data x using each of the above filtered sample sets X j After the above processing, the final bit array incorporating the remainder is the created Bloom filter bit array; the created Bloom filter bit array can be used to determine whether a data to be retrieved, queried or filtered is included in the above filter sample set X.
[0042] The method for implementing the improved Bloom filter in the embodiment of the present invention further includes: S2, a step of using the created Bloom filter bit array to perform data filtering. The step S2 includes:
[0043] S21, using the hash function H to perform hash calculation on the filtered data k to obtain a hash value H(k);
[0044] The hash function used here is the same as the hash function used in the stage of creating the Bloom filter bit array; the hash value H(k) consists of n bits;
[0045] S22, use each element m in the coprime array M i Perform modulus operations on the hash value H(k) to obtain the value that is equal to m i The corresponding remainder y i , 1≤i≤I, remainder y i It includes n bits;
[0046] The coprime array used here is the same as the coprime array used in the stage of creating the Bloom filter bit array; i is the sequence number of the element in the array;
[0047] S23, the remainder y i Each bit of is matched with the corresponding n elements in the Bloom filter bit array; if there is a remainder y i If the bit of is 1 and the corresponding bit in the Bloom filter bit array is 0, it is determined that the data k is not in the filtered sample set X; further, the filtering result of "not in the set" can be output.
[0048] In a specific implementation, step S23 further includes: after all the remainders y i After the bits of are matched with the corresponding n elements in the Bloom filter bit array, if for all remainders y i If the corresponding bits in the Bloom filter bit array are all 1, it is determined that the data k is in the filtered sample set X. Further, the filtering result of "in the set" can be output; then, the filtering judgment ends.
[0049] Preferably, first determine whether there is a remainder y i The bit of is 1 and the corresponding bit in the Bloom filter bit array is 0; if so, it is determined that the data k is definitely not in the sample data set; otherwise, continue to judge for all remainders y i For the bits that are 1 in the Bloom filter bit array, are the corresponding bits in the Bloom filter bit array all 1? i If the bits in the Bloom filter bit array are all 1, then it is determined that the data k must be in the sample data set.
[0050] Embodiment 2:
[0051] Figure 3 FIG. 1 is a schematic diagram of the structure of a computing device according to an embodiment of the present invention. Figure 3As shown, the computing device includes a processor 301, a memory 302, a bus 303, and a computer program stored in the memory 302 and executable on the processor 301. The processor 301 includes one or more processing cores. The memory 302 is connected to the processor 301 via the bus 303. The memory 302 is used to store program instructions. When the processor executes the computer program, the steps in the above method embodiment of the first embodiment of the present invention are implemented. In a specific implementation, the computing device is an embedded device.
[0052] Further, as an executable solution, the computing device may be a computer unit, which may be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The computer unit may include, but is not limited to, a processor and a memory. Those skilled in the art may understand that the composition structure of the above-mentioned computer unit is only an example of a computer unit and does not constitute a limitation on the computer unit, and may include more or less components than the above-mentioned, or a combination of certain components, or different components. For example, the computer unit may also include input and output devices, network access devices, buses, etc., which are not limited in the embodiments of the present invention.
[0053] Further, as an executable solution, the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the computer unit, and various interfaces and lines are used to connect the various parts of the entire computer unit.
[0054] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the computer unit by running or executing the computer program and / or module stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required for a function; the data storage area can store data created according to the use of the mobile phone, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0055] Embodiment 4:
[0056] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method embodiment of the present invention are implemented.
[0057] If the modules included in the computer program are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory) and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.
[0058] Although the present invention has been specifically shown and described in conjunction with the preferred embodiments, it should be understood by those skilled in the art that various changes may be made to the present invention in form and details without departing from the spirit and scope of the present invention as defined by the appended claims, all of which are within the scope of protection of the present invention.
[0059] Although the present invention has been specifically shown and described in conjunction with the preferred embodiments, it should be understood by those skilled in the art that various changes may be made to the present invention in form and details without departing from the spirit and scope of the present invention as defined by the appended claims, all of which are within the scope of protection of the present invention.
Claims
1. A method for implementing an improved Bloom filter, using a computing device, characterized in that: The method comprises: S1, a step of creating a Bloom filter bit array, wherein the step of creating the Bloom filter bit array comprises: S11, creating a coprime array M and a hash function H, wherein the coprime array M includes I elements m i , where 1≤i≤I, the maximum value of the hash value generated by the hash function H includes n bits; S12, creating a bit array of length n*I, in which each n elements correspond to an element of the coprime array M, and the initial value of each element in the bit array is 0; S13, for each filtered sample data x in the filtered sample set X, j The following processing is performed, j=1 to N, N is the number of sample filtering data included in the X: The hash function H is used to perform hash calculation to obtain a hash value H(x j ), using each element m in the coprime array M i The hash value H(x j ) performs a modular operation to obtain the same i The corresponding remainder y i , 1≤i≤I, the remainder y i comprises n bits; and Each of the remainders y i With the bit array and element m i Perform bitwise AND operation on the corresponding n elements to obtain a bit array incorporating the remainder; S14, for each filtered sample data x j After the processing of step S13 is performed, the Bloom filter bit array is obtained.
2. The method according to claim 1, characterized in that In the step S12, in the n*I bit array, a group of n elements corresponding to an element in the coprime array M are continuous in position.
3. The method according to claim 1, characterized in that Also includes: S2, a step of using the created Bloom filter bit array to perform data filtering, wherein the step of using the created Bloom filter bit array to perform data filtering includes: S21, performing hash calculation on the filtered data k using the hash function H to obtain a hash value H(k); S22, using each element m in the coprime array M i Perform a modulus operation on the hash value H(k) to obtain the value corresponding to m i The corresponding remainder y i , 1≤i≤I, the remainder y i It includes n bits; S23, the remainder y i Each bit of is bit-matched with the corresponding n elements in the Bloom filter bit array; if the remainder y exists i If the bit of is 1 and the corresponding bit in the Bloom filter bit array is 0, it is determined that the data k is not in the filtered sample set X.
4. The method according to claim 3, characterized in that The step S23 further comprises: After all the remainders y i After the bits of are bit-matched with the corresponding n elements in the Bloom filter bit array, if for all the remainders y i If the bits in the Bloom filter bit array are all 1, then it is determined that the data k is in the filtered sample set X.
5. The method according to claim 4, characterized in that In step S23, first determine whether there is a remainder y i The bit of is 1 and the corresponding bit in the Bloom filter bit array is 0; if so, it is determined that the data k is definitely not in the sample data set; otherwise, continue to judge for all remainders y i For the bits that are 1 in the Bloom filter, are the corresponding bits in the Bloom filter bit array all 1? 6. A computing device, characterized in that The method comprises a memory and a processor, wherein the memory stores at least one program, and the at least one program is executed by the processor to implement the method according to any one of claims 1 to 5.
7. A computer-readable storage medium, characterized in that: The storage medium stores at least one program, and the at least one program is executed by a processor to implement the method according to any one of claims 1 to 5.