Cloud storage data fusion method based on double hash fuzzy bloom filter
The two Bloom filters are compressed into one through dual hash fuzzy Bloom filter technology, which solves the problem of increased query complexity in IIoT data processing, improves storage efficiency and data residence time, and optimizes storage space utilization.
Patent Information
- Application Number
- CN202011319722.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-11-23
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2040-11-23
AI Technical Summary
In the existing IIoT data processing methods, the query complexity increases with the amount of input data, which seriously affects the space utilization of the memory and leads to insufficiency of storage.
Using the cloud storage data fusion method of double hash fuzzy Bloom filter, the two Bloom filters are compressed into one, and multiple hash functions are generated through double hash technology to reduce the hash operation time, and the fuzzy cross operation is used and The double hash function optimizes the data insertion and query process.
It significantly improves storage efficiency, reduces computing costs, extends the dwell time of data in memory, and optimizes storage space utilization without losing accuracy.
Smart Images

Figure QLYQS_1 
Figure QLYQS_2 
Figure QLYQS_6
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data fusion, and in particular to a cloud storage data fusion method based on double hash fuzzy Bloom filter. Background Art
[0002] There are many methods for processing IIoT (Industrial Internet of Things) data, such as: key value storage method based on Bloom filter; efficient representation of membership of dynamic Bloom filter array cloud storage system; adjustable Bloom filter batch data insertion method and SDN-based big data management method. The common disadvantage of these methods is that the query complexity increases with the amount of input data, which seriously affects the space utilization of the memory. Summary of the invention
[0003] In view of the above-mentioned technical deficiencies, the purpose of the present invention is to provide a cloud storage data fusion method based on double hash fuzzy Bloom filter, which uses double hash technology to compress two Bloom filters into one filter while ensuring the robustness of failed data. Two hash functions generate k hash functions, which greatly reduces the time of hash operations, more effectively utilizes storage capacity, and improves the storage efficiency of big data across multiple regions.
[0004] In order to solve the above technical problems, the present invention adopts the following technical solutions:
[0005] The cloud storage data fusion method based on double hash fuzzy Bloom filter is characterized by the following specific steps:
[0006] 1) Setting up the Bloom filter: The Bloom filter consists of an array of “m” bits, initially set to all zeros, and a set of k hash functions;
[0007] 2) Define the mathematical model of the problem description: Given a data stream with n elements (D s ), namely D s ={x1,x2,…,x n}, the mathematical model is:
[0008] 3) Fuzzy crossover operation: merge a x ∈BF i [] and b y ∈BF j [], where x=y; these two elements have the same index in both parts, overlap each other and are stored as a single fuzzy value in the upper half; in this process, the index bits are used for data compression; the two fused Bloom filters, BF i [] and BF j[], is called the first cross or first compressed form; it is represented by the symbol CR i,j Representation, and block bits and fingerprint bits are needed to represent the elements stored in it using fuzzy symbols;
[0009] The fuzzy crossover operation is represented by the following model:
[0010]
[0011] Among them, NFO means no fuzzy operation is performed;
[0012] 4) Data insertion in fuzzy cross-Bloom filter: The fuzzy cross-Bloom filter is represented by BF[], which consists of m elements, one of which is a fingerprint. Each block bit used for compression operation In association, the input data from the set S is added to the Bloom filter using a double hashing method, which significantly reduces the computation time. In the proposed Bloom filter variant, a double hashing function is used to generate k hash functions (i.e. ), the mathematical formula of the hash function is as follows:
[0013] g i (x)={h1(x)+i×h2(x)}mod m p
[0014] Among them, m p is the value of the hash function between the maximum limit range (1:m) and the closest prime number relative to the size of BF(m), m p The selection is made by generating the best hash value. First, the array of size m is divided into two Bloom filters of the same size:
[0015]
[0016] The element is added to the i-th Bloom filter, when BF i The filling capacity of [] exceeds the threshold filling ratio (F thres ), insert from BF i+1 [] Starting at the first level of insertion, only the block bit is set to 1 according to the following hash value:
[0017]
[0018] Once you reach BF i+1 [] Filter threshold, fuzzy crossover operation is applied to two filters (BF i [] and BF i+1[]) to store more data in the existing Bloom filter. To make the fuzzy crossover operation effective, m and k should be multiples of 2.
[0019] 5) Data query in fuzzy cross Bloom filter: In fuzzy cross Bloom filter, the query process always starts from the active time slot A. If the element is found in the Ath time slot, the query process returns TRUE; otherwise, the scan will continue until A=1 and the search starts. The hash query method is as follows:
[0020]
[0021] In the above formula, the hash index uses h i Indicates that in BF i In, if Then i is considered a member of the set S in the BF[a] slot if the ath position is denoted by CR i,i+1 , and then use the hResult() function, which checks the The number of α, β, and γ that appear at each position, and their corresponding values are stored in C α ,C β ,C γ Next, calculate BF respectively. i [] and BF i+1 The two membership functions of [] are as follows:
[0022]
[0023] Here are the conclusions drawn from the hResult() function defined above:
[0024]
[0025] The time complexity of searching for an item (y∈Q) in CR is O(k) if CR i,i+1 Represents the 2n elements of time slots BF[i] and BF[i+1].
[0026] Preferably, the mathematical model in step 2 represents: ① the computational cost associated with hashing (C c ) minimized; ② query complexity when processing dynamic data sets (Q c ) optimization; ③ The memory used to store data is optimized in a way that can accommodate the maximum number of elements (E a );④False positive (f p ), the important performance parameters of the Bloom filter do not exceed the predetermined limits.
[0027] The beneficial effects of the present invention are: 1. Fuzzy crossover operation is used to merge and compress two Bloom filters, so that hash data can be shared and accommodated in two Bloom filters, reducing the demand for massive data storage; 2. Double hashing is used to calculate multiple hash functions to reduce computing costs, which has little impact on failed data of industrial Internet of Things networks, data decays slowly, and allows streaming data to reside in memory for a considerable period of time; 3. Storage space is efficiently and optimally utilized without losing accuracy. DETAILED DESCRIPTION
[0028] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0029] The cloud storage data fusion method based on double hash fuzzy Bloom filter has the following specific steps:
[0030] 1) Setting up the Bloom filter: The Bloom filter consists of an array of “m” bits, initially set to all zeros, and a set of k hash functions;
[0031] 2) Define the mathematical model of the problem description: Given a data stream with n elements (D s ), namely D s ={x1,x2,…,x n}, the main requirement is to improve the performance of the existing Bloom filter in terms of memory and search complexity. The mathematical model is:
[0032] The mathematical model represents the following problem:
[0033] ① Streaming data is available in a very short time, so it must be processed once and kept in memory long enough for query;
[0034] ②Computational costs associated with hashing (C c ) should be minimized;
[0035] ③Query complexity when processing dynamic data sets (Q c ) should be optimized;
[0036] ④ The memory used to store data should be optimized in a way that the maximum number of elements can be accommodated (E a ).
[0037] ⑤ False positive (f p ), the important performance parameters of the Bloom filter should not exceed the predetermined limit;
[0038] 3) Fuzzy crossover operation: merge a x ∈BF i [] and b y ∈BF j [], where x=y; these two elements have the same index in both parts, overlap each other and are stored as a single fuzzy value in the upper half; in this process, the index bits are used for data compression; the two fused Bloom filters, BF i [] and BF j [], is called the first cross or first compressed form; it is represented by the symbol CR i,j Representation, and block bits and fingerprint bits are needed to represent the elements stored in it using fuzzy symbols;
[0039] The fuzzy crossover operation is represented by the following model:
[0040]
[0041] Among them, NFO means no fuzzy operation is performed;
[0042] when Stored in the initial m / 2 space, once the available space threshold is reached, the space m / 2 is exhausted, and then the next crossover process is applied to move the data stored in the m / 2 space to the m / 4 space, and generate some new space again to store more incoming data in the Bloom filter of the same size m. Due to the use of fuzzy crossover, the above storage method has a unique signature, which retains the two partial data of the Bloom filter so that it can be queried over a longer period of time;
[0043] 4) Data insertion in fuzzy cross-Bloom filter: The fuzzy cross-Bloom filter is represented by BF[], which consists of m elements, one of which is a fingerprint. Each block bit used for compression operation In association, the input data from the set S is added to the Bloom filter using a double hashing method, which significantly reduces the computation time. In the proposed Bloom filter variant, a double hashing function is used to generate k hash functions (i.e. ), the mathematical formula of the hash function is as follows:
[0044] g i (x)={h1(x)+i×h2(x)}mod m p
[0045] Among them, m p is the value of the hash function between the maximum limit range (1:m) and the closest prime number relative to the size of BF(m), m pThe selection is made by generating the best hash value. First, the array of size m is divided into two Bloom filters of the same size:
[0046]
[0047] Elements are added to the i-th Bloom filter when The filling capacity exceeds the threshold filling ratio (F thres ), insert from BF i+1 [] Starting at the first level of insertion, only the block bit is set to 1 according to the following hash value:
[0048]
[0049] Once you reach BF i+1 [] Filter threshold, fuzzy crossover operation is applied to two filters (BF i [] and BF i+1 []) to store more data in the existing Bloom filter. To make the fuzzy crossover operation effective, m and k should be multiples of 2.
[0050] 5) Data query in fuzzy cross Bloom filter: In fuzzy cross Bloom filter, the query process always starts from the active time slot A. If the element is found in the Ath time slot, the query process returns TRUE; otherwise, the scan will continue until A=1 and the search starts. The hash query method is as follows:
[0051]
[0052] In the above formula, the hash index uses h i Indicates that in BF i In, if Then y i is considered a member of the set S in the BF[a] slot if the ath position is denoted by CR i,i+1 , and then use the hResult() function, which checks the The number of α, β, and γ that appear at each position, and their corresponding values are stored in C α ,C β ,C γ Next, calculate BF respectively. i [] and BF i+1 The two membership functions of [] are as follows:
[0053]
[0054] Here are the conclusions drawn from the hResult() function defined above:
[0055]
[0056] The time complexity of searching for an item (y∈Q) in CR is O(k) if CR i,i+1 Represents the 2n elements of time slots BF[i] and BF[i+1].
[0057] The performance test of the cloud storage data fusion algorithm involved in the present invention is carried out. The specific content is: the PBC0.5.15 test library is selected for simulation test, which can realize the design of batch audit model under the condition of file failure. At the same time, three cloud storage algorithms in relevant literature are selected for comparative experiments. The development language selected for the test system is C language. The test system platform software selects Linux 3.8.0-29, the processor configuration is CPU Intel (R) E5605 @ 2.55GHz, the system memory size is 32GB, and the system hard disk is a 1TB Seagate mechanical hard disk.
[0058] The data block size in the cloud storage process is set to |id|=50b, the test file size in the cloud storage process is 1GB, the maximum proportion of file damage in the simulation test process is set to 1%, and 500 groups of all data blocks are selected as simulation objects for data auditing. Experimental comparison indicators First, the communication data overhead in the cloud storage process is selected for experimental comparison. To ensure the stability of the results obtained in the test process, each group of experiments is run 30 times separately to obtain the average of the experimental results for comparative testing.
[0059] The design of the present invention uses fuzzy crossover operation to merge and compress two Bloom filters, so as to realize the shared storage of hash data in the two Bloom filters and reduce the demand for massive data storage. It uses double hashing to calculate multiple hash functions to reduce the computing cost, has little impact on the invalid data of the industrial Internet of Things network, and the data decays slowly, and allows the streaming data to reside in the memory for a considerable time. It efficiently optimizes the use of storage space without losing precision.
[0060] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A cloud storage data fusion method based on double hash fuzzy Bloom filter, characterized in that: The specific steps are as follows: 1) Setting up the Bloom filter: The Bloom filter consists of an array of "m" bits, initially set to all 0s, and a set of k hash functions; 2) Define the mathematical model of the problem description: Given a data stream D with n elements s , that is, D s ={x1,x2,…,x n }, the mathematical model is: 3) Fuzzy crossover operation: merge a x ∈BF i [] and b y ∈BF j [], where x=y; these two elements have the same index in both parts, overlap each other and are stored as a single fuzzy value in the upper half; in this process, the index bits are used for data compression; the two fused Bloom filters, BF i [] and BF j [], known as the first cross or first compressed form; It is represented by the symbol CR i,j Representation, and block bits and fingerprint bits are needed to represent the elements stored in it using fuzzy symbols; The fuzzy crossover operation is represented by the following model: Among them, NFO means do not perform fuzzy operation; the lowercase letter l represents the lth element of the array; 4) Data insertion in fuzzy cross-Bloom filter: The fuzzy cross-Bloom filter is represented by BF[], which consists of m elements, one of which is a fingerprint. Each block bit used for compression operation In association, the input data from the set S is added to the Bloom filter using the double hashing method. In the proposed Bloom filter variant, the double hashing function is used to generate k hash functions. The mathematical formula of the hash function is as follows: g i (x)={h1(x)+i×h2(x)}mod m p The value range of i is: m p is the value of the hash function between m and the nearest prime number with respect to the maximum limit range of BF(m) size 1: p The selection is made by generating the best hash value, dividing the array of size m into two Bloom filters of the same size: The element is added to the i-th Bloom filter, when BF i The filling capacity of [] exceeds the threshold filling ratio F thres When inserting from BF i+1 [] Starting at the first level of insertion, only the block bit is set to 1 according to the following hash value: Once you reach BF i+1 [] filter threshold, fuzzy crossover operation ⊙ is applied between the two filters BF i [] and BF i+1 [] in order to save more data storage space in the existing Bloom filter. To make the fuzzy crossover operation effective, m and k are multiples of 2; 5) Data query in fuzzy cross Bloom filter: In fuzzy cross Bloom filter, the query process always starts from the active time slot A. If the element is found in the Ath time slot, the query process returns TRUE; otherwise, the scan will continue until A=1 and the search starts. The hash query method is as follows: In the above formula, the hash index uses h i Indicates that in BF i BF i Filter i and hash index h i When the match is 1, Then y i is considered to be a member of the set S in the BF[a] time slot, and the ath position is denoted as CR i,i+1 , and then use the hResult() function, which checks the The number of α, β, and γ that appear at each position, and their corresponding values are stored in C α ,C β ,C γ Next, calculate BF respectively. i [] and BF i+1 The two membership functions of [] are as follows: Here are the conclusions drawn from the hResult() function defined above: The time complexity of searching for an item in CR is O(k), where the query is denoted as (y∈Q) and CR i,i+1 2n elements representing time slots BF[i] and BF[i+1]; The mathematical model in step 2 represents: ① Minimize the computational cost associated with hashing min(C c );②Optimize query complexity when processing dynamic data sets min(Q c ); ③ The memory used to store data is optimized in a way that the maximum number of elements can be accommodated max(E a );④ False positive min(f p ), the important performance parameters of the Bloom filter do not exceed the predetermined limits.
Citation Information
Patent Citations
Method for utilizing content-centric network nodes to process data based on tree and hash table
CN103595637A
Data processing method and device, electronic equipment and storage medium
CN109766479A
Data processing method and data processing device
CN111858651A
A method for generating a randomized data structure for representing sets, based on bloom filters
IL211163A
