Universal parallel mass data processing method

Abstract

The invention relates to the technical field of data processing, and provides a universal parallel mass data processing method, which comprises the following steps of: S1, constructing a bitmap of a Bloom filter; s2, calculating the number of first hash functions of the Bloom filter; s3, configuring a second hash function of the HLL + + algorithm; s4, querying a data processing state based on the data identifier; s5, processing the data, and registering a data identifier to a bitmap of the Bloom filter and a bucket of an HLL + + algorithm; and S6, executing the step S4 until all data are processed. According to the method, parallel processing nodes can be simply added to improve the processing capacity of the system, meanwhile, the probability that the system processes repeated data is greatly reduced by means of the data processing state service, the data processing efficiency is further improved, and meanwhile, the data processing accuracy is improved through a configured algorithm. Compared with a traditional traversal data processing scheme, the method has the advantages of being good in expansibility, low in hardware requirement, high in resource utilization rate and the like.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and particularly to a general parallel massive data processing method. Background Art

[0002] The parallel processing of massive data is a relatively common task in the field of information technology. While processing data in parallel on multiple nodes, the status of data processing needs to be recorded and shared among nodes. For massive data processing scenarios, whether it is the upstream data source or the downstream data result storage system, a distributed cluster architecture is adopted, and a real-time deduplication function similar to an index cannot be provided. To register the data processing status, a database or a hash table needs to be used, and these solutions have low performance or consume a large amount of memory resources, increasing the cost of data processing. Summary of the Invention

[0003] To help solve the above technical problems, this application provides a general parallel massive data processing method, adopting the following technical solutions: A general parallel massive data processing method, wherein the general parallel massive data processing method adopts a general parallel massive data processing system, and the system includes: A data source, a status sharing service module, and multiple parallel data processing nodes. The data processing nodes obtain data through the data source, query the data processing status through the status sharing service module before processing the data, and update the data processing status through the status sharing service module after processing the data. The general parallel massive data processing method includes: Step S1: Calculate the number of bits of the bitmap, and construct the bitmap of the Bloom filter according to the number of bits of the bitmap; Step S2: Calculate the number of the first hash functions of the Bloom filter, and map each data element to k different positions in the bit array through k hash functions, where k is a positive integer; Step S3: Configure the second hash function of the HLL++ algorithm, and map the data element to the bucket of the HLL++ algorithm through the second hash function; Step S4: Query the data processing status based on the data identifier, including: Step S41: Obtain the bitmap index number, which is used to indicate the position of the data identifier in the bit array. Check the bit corresponding to the calculated index number in the bitmap of the Bloom filter. If all the bits corresponding to the index numbers are 0, it is considered that the data has not been processed, and execute Step S5; Step S42: Obtain the bucket value, calculate the number of leading zeros of the extracted bucket value according to the bucket value. If the number of leading zeros is less than the value stored in the bucket array of the HLL++ algorithm, execute Step S6; Step S5: Process the data, and register the data identifier into the bitmap of the Bloom filter and the bucket of the HLL++ algorithm, including: Step S51: Process the data, obtain the bitmap index number, which is used to indicate the position of the data identifier in the bit array, and set all the bits corresponding to the calculated index number in the bitmap of the Bloom filter to 1; Step S52: Process the data, obtain the bucket value, calculate the number of leading zeros of the extracted bucket value according to the bucket value, update the bucket index to j = hash value mod number of buckets, and if the currently calculated number of leading zeros is greater than the value stored in the bucket array, update the value of the bucket array to the currently calculated number of leading zeros; Step S6: Execute Step S4 until all data has been processed.

[0004] Preferably, the data identifier includes a universally unique identifier and a hash digest obtained after being processed by a hash function.

[0005] Preferably, Step S1 includes: constructing the bitmap of the Bloom filter in the following manner: , where n is the estimated data volume, p is the expected false positive rate, and m is the number of bits in the bitmap.

[0006] Preferably, Step S2 includes: calculating the number of the first hash functions of the Bloom filter in the following manner: , where m is the number of bits in the bitmap, p is the expected false positive rate, n is the estimated data volume, and k is the number of the first hash functions.

[0007] Preferably, Step S3 includes: configuring the error range, bucket index length, number of buckets, and memory occupancy of the HLL++ algorithm.

[0008] Preferably, Step S41 and Step S51 include: obtaining the bitmap index number in the following manner: Obtain the data identifier of the data to be processed, calculate the first hash value according to the data identifier through the first hash function, and take the modulo of the first hash value with the length of the bitmap of the Bloom filter to obtain the bitmap index number.

[0009] Preferably, Step S42 and Step S52 include: obtaining the bucket value in the following manner: Calculate the second hash value according to the data identifier through the second hash function, and extract the value of a specific number of bits in the second hash value as the bucket value, where the specific number of bits is the difference between the second hash value and the bucket index length.

[0010] In summary, the present application can simply add parallel processing nodes to improve the system processing capacity. At the same time, relying on the data processing status service, the probability of the system processing duplicate data is greatly reduced, further improving the efficiency of data processing. Meanwhile, the accuracy of data processing is improved through the configured algorithm. Compared with the traditional cross-through data processing solution, the present application has the characteristics of good scalability, low hardware requirements, and high resource utilization rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 FIG. is a schematic block diagram of an embodiment of a general parallel massive data processing system of the present application; Figure 2 FIG. is a schematic flowchart of an embodiment of a general parallel massive data processing method of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0012] The present application will be further described below with reference to the accompanying drawings. The structure and principle of the present application are very clear to those skilled in the art. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0013] Figure 1 FIG. is a schematic block diagram of an embodiment of a general parallel massive data processing system of the present application; Figure 2 FIG. is a schematic flowchart of an embodiment of a general parallel massive data processing method of the present application.

[0014] Combined with Figure 1 and Figure 2 It can be understood that the general parallel massive data processing method adopts a general parallel massive data processing system. The system includes a data source, a status sharing service module, and multiple parallel data processing nodes. The data processing nodes obtain data through the data source, query the data processing status through the status sharing service module before processing the data, and update the data processing status through the status sharing service module after processing the data.

[0015] The bitmap data involved in the following Bloom filter and HLL++ algorithm can be stored in a unified shared status service. For example, systems such as Redis can provide data sharing and change for multiple nodes.

[0016] The general parallel massive data processing method of the present application includes: Step S1: Calculate the number of bits of the bitmap, and construct the bitmap of the Bloom filter according to the number of bits of the bitmap. Specifically, the bitmap of the Bloom filter is constructed in the following manner: , where n is the estimated data volume, p is the expected false positive rate, and m is the number of bits of the bitmap. For example, it is expected that the data processing quantity is 10 10If the acceptable false positive rate is 1%, then the size m of the bitmap is approximately 9.59×10 10 bits, which occupies approximately 11.98 GB of memory.

[0017] Step S2: Calculate the number of the first hash functions of the Bloom filter. Map each data element to k different positions in the bit array through k hash functions, where k is a positive integer. Calculate the number of the first hash functions of the Bloom filter in the following way: , where m is the number of bits of the bitmap, p is the expected false positive rate, n is the estimated data volume, and k is the number of the first hash functions. In step S2, determine the number of functions of the Bloom filter according to the formula. The hash function can be a function with fast calculation speed, uniform distribution, and certain anti-collision property, such as algorithms like MurmurHash and CityHash. For example, if the size m of the bitmap is approximately 9.59×10 10 bits and the expected number of data to be processed is 10 10 items, then the number of functions of the Bloom filter is 7.

[0018] Step S3: Configure the second hash function of the HLL++ algorithm, and at the same time configure the error range, bucket index length, number of buckets, and memory occupancy of the HLL++ algorithm. Map the data element to the bucket of the HLL++ algorithm through the second hash function. In step S3, control the error of the HLL++ algorithm to be about 0.57%, select the bucket index length hp = 15, that is, hm = 2 15 = 32768 as the number of buckets, and use the FNV function to construct the hash mapping, which occupies approximately 24 KB of memory.

[0019] Step S4: Query the data processing status based on the data identifier, including: Step S41: Obtain the data identifier of the data to be processed. Calculate the first hash value according to the data identifier through the first hash function. Take the modulus of the first hash value with the length of the bitmap of the Bloom filter to obtain the bitmap index number. The bitmap index number is used to indicate the position of the data identifier in the bit array. Check the bit corresponding to the calculated index number in the bitmap of the Bloom filter. If all the bits corresponding to the index numbers are 0, it is considered that the data has not been processed, and execute step S5.

[0020] Step S42: Calculate a second hash value based on the data identifier using the second hash function. Extract a value with a specific number of bits from the second hash value as the bucket value. The specific number of bits is the difference between the second hash value and the bucket index length. Calculate the number of leading zeros of the extracted bucket value based on the bucket value. If the number of leading zeros is less than the value stored in the bucket array of the HLL++ algorithm, execute step S6. In step S42, perform the hash function specified in step S2 on the UUID or hash digest of the data to obtain a binary result hr. Extract the last hr - hp bits of the result as the bucket value w. Calculate the number of leading zeros lead(w) of the bucket value w. The number of leading zeros refers to the number of consecutive zeros starting from the highest bit.

[0021] Perform the hash function specified in step S2 on the UUID or hash digest of the data to obtain a binary result hr. Extract the last hr - hp bits of the result as the bucket value w. Calculate the number of leading zeros lead(w) of the bucket value w. The number of leading zeros refers to the number of consecutive zeros starting from the highest bit. Compare lead(w) with the value of bucket[j]. If lead(w) > bucket[j], it indicates that the data may not exist, and then jump to step S5 to perform data processing. Otherwise, jump to step S6 to skip the current data processing.

[0022] Step S5: Process the data and register the data identifier into the bitmap of the Bloom filter and the bucket of the HLL++ algorithm, including: Step S51: Process the data. Obtain the data identifier of the data to be processed. Calculate a first hash value based on the data identifier using the first hash function. Take the modulus of the first hash value with the length of the bitmap of the Bloom filter to obtain the bitmap index number. The bitmap index number is used to indicate the position of the data identifier in the bit array. Set all the bits corresponding to the calculated index number in the bitmap of the Bloom filter to 1.

[0023] Step S52: Process the data. Calculate a second hash value based on the data identifier using the second hash function. Extract a value with a specific number of bits from the second hash value as the bucket value. The specific number of bits is the difference between the second hash value and the bucket index length. Calculate the number of leading zeros of the extracted bucket value based on the bucket value. Update the bucket index to j = hash value mod the number of buckets. If the currently calculated number of leading zeros is greater than the value stored in the bucket array, update the value of the bucket array to the currently calculated number of leading zeros. The number of leading zeros refers to the number of consecutive zeros starting from the highest bit. It is necessary to update the bucket index to j = hr mod hm, and update the j-th bucket to max(bucket[j], lead(w)), that is, bucket[j] = max(bucket[j], lead(w)). Complete the update of the HLL++ bucket record.

[0024] It should be noted here that the "mod" operation is a mathematical operation method, specifically the modulo operation (also known as the modulus operation or the remainder operation). The modulo operation means dividing a number (dividend) by another number (divisor) and then taking the remainder as the result. The bucket value w is extracted from the hash result of the data and is used to determine the bucket position of the data in the HLL++ algorithm (by calculating the number of leading zeros). The value of bucket[j] is the value stored in the bucket array of the HLL++ algorithm, representing the maximum value (or related value) of the number of leading zeros stored in that bucket. The bucket value w is calculated dynamically, while the value of bucket[j] is updated as needed during the operation of the algorithm. In the HLL++ algorithm, the bucket value w and the value of bucket[j] jointly act on the process of estimating the cardinality of the data. By comparing and updating the values in the bucket, the algorithm can gradually approach the true cardinality of the data set.

[0025] Step S6: Execute Step S4 until all data has been processed.

[0026] It should be noted here that the data identifier includes the Universally Unique Identifier (UUID) and the hash digest obtained after being processed by a hash function.

[0027] In summary, in this application, it is expected that the number of data to be processed is 10 10 pieces, and the acceptable false positive rate is 1%. Then the size m of the bitmap is approximately 9.59×10 10 bits, occupying approximately 11.98 GB of memory.

[0028] Compared with using a traditional hash table to record the UUID of the data (calculated as 16 bytes), it occupies approximately 400 GB of memory space. The bitmap length of the Bloom filter used in the present invention is 9.59×10 10 bits, the error rate can be controlled within about 1%, and it occupies approximately 11.98 GB of space. By jointly using the HLL++ algorithm, 32768 buckets are used, and each bucket uses 6 bits of memory. The error rate is approximately 0.81%, and the memory used is 24 KB. According to the error calculation, the combined false positive rate of the overall system is 0.0057%, that is, when processing 10 10 records, the amount of data that may be processed repeatedly is 5.7×10 5 pieces, and the memory usage is reduced to 3%.

Claims

1. A general parallel massive data processing method, characterized in that, The described general parallel massive data processing method uses a general parallel massive data processing system, which includes: A data source, a status sharing service module, and multiple parallel data processing nodes. The data processing nodes obtain data through the data source, query the data processing status through the status sharing service module before processing the data, and update the data processing status through the status sharing service module after processing the data. The general parallel massive data processing method includes: Step S1: Calculate the number of bits of the bitmap and construct the bitmap of the Bloom filter according to the number of bits of the bitmap. Step S2: Calculate the number of the first hash functions of the Bloom filter, and map each data element to k different positions in the bit array through k hash functions, where k is a positive integer. Step S3: Configure the second hash function of the HLL++ algorithm, and map the data element to the bucket of the HLL++ algorithm through the second hash function. Step S4: Query the data processing status based on the data identifier, including: Step S41: Obtain the bitmap index number, check the bit corresponding to the calculated index number in the bitmap of the Bloom filter. If all the bits corresponding to the index numbers are 0, it is considered that the data has not been processed, and execute Step S5. Step S42: Obtain the bucket value, calculate the number of leading zeros of the extracted bucket value according to the bucket value. If the number of leading zeros is less than the value stored in the bucket array of the HLL++ algorithm, execute Step S6. Step S5: Process the data, and register the data identifier to the bitmap of the Bloom filter and the bucket of the HLL++ algorithm, including: Step S51: Process the data, obtain the bitmap index number, and set all the bits corresponding to the calculated index number in the bitmap of the Bloom filter to 1. Step S52: Process the data, obtain the bucket value, calculate the number of leading zeros of the extracted bucket value according to the bucket value, update the bucket index to j = hash value mod the number of buckets. If the currently calculated number of leading zeros is greater than the value stored in the bucket array, update the value of the bucket array to the currently calculated number of leading zeros. Step S6: Execute Step S4 until all data is processed.

2. The general parallel massive data processing method according to claim 1, characterized in that, The data identifier includes a universally unique identifier and a hash digest obtained after being processed by a hash function.

3. The general parallel mass data processing method according to claim 1, characterized in that The Step S1 includes: Constructing the bitmap of the Bloom filter in the following way: , where n is the estimated data volume, p is the expected misjudgment rate, and m is the number of bits of the bitmap.

4. The general parallel massive data processing method according to claim 1, characterized in that, The Step S2 includes: Calculating the number of the first hash functions of the Bloom filter in the following way: , where m is the number of bits of the bitmap, p is the expected false positive rate, n is the estimated data volume, and k is the number of the first hash functions.

5. The general parallel mass data processing method according to claim 1, wherein The Step S3 includes: Configuring the error range, bucket index length, number of buckets, and memory occupancy of the HLL++ algorithm.

6. The general parallel mass data processing method according to claim 1, characterized in that, The Step S41 and the Step S51 include: Obtaining the bitmap index number in the following way: Obtain the data identifier of the data to be processed, calculate the first hash value according to the data identifier through the first hash function, and take the modulo of the first hash value by the length of the bitmap of the Bloom filter to obtain the bitmap index number.

7. The general parallel massive data processing method according to claim 1, characterized in that, The Step S42 and the Step S52 include: Obtaining the bucket value in the following way: Calculate the second hash value according to the data identifier through the second hash function, and extract the value of a specific number of bits in the second hash value as the bucket value. The specific number of bits is the difference between the second hash value and the bucket index length.

Images