Bitmap optimization algorithm capable of improving data de-duplication and noise reduction performance

By using a high-low bit splitting scheme of arrays and bitmap containers in the data deduplication process, the problems of data skew and space waste in the data deduplication process are solved, and efficient data deduplication and noise reduction effects are achieved.

CN119003508APending Publication Date: 2024-11-22湖南赢科云行科技有限公司 +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411093786.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

Existing data deduplication technologies perform poorly under the pressure of big data, are prone to data skew during the shuffle stage, and have large data footprints and low data addressing efficiency.

Method used

An array container is used to store the first 4096 integers. When there are more than 4096 integers, a bitmap container is used. Combined with a high-low bit splitting scheme, the array container is used to store sparse data and the bitmap container is used to store dense data. Two levels of indexes are set to improve query efficiency.

Benefits of technology

It effectively avoids data skew during the Shuffle stage, improves data deduplication efficiency, saves memory space, reduces data addressing time, and improves data query efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003508A_ABST
    Figure CN119003508A_ABST
Patent Text Reader

Abstract

The invention relates to data de-duplication and noise reduction, in particular to an optimized bitmap algorithm capable of improving data de-duplication and noise reduction performance, which comprises the following steps of: storing first 4096 integers by using an array container, and storing the first 4096 integers by using a bitmap container when the first 4096 integers exceed the first 4096 integers; for a scene which is continuous for a long time and large in data volume, in order to avoid long-time waiting of downstream services, data of the same dimension is stored in the same partition in sequence, and two levels of indexes are set, so that data addressing time is shortened, data query efficiency is improved, and then data deduplication efficiency is improved; for the problem of data skew occurring in the data partitioning process, a Bitmap algorithm is introduced for data deduplication, data skew occurring in the Shuffle stage is avoided, and the data deduplication efficiency is improved; according to the technical scheme provided by the invention, the defects of data inclination, relatively large space occupied by data and relatively low data addressing efficiency in the Shuffle stage in the prior art can be effectively overcome.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to data deduplication and noise reduction, and in particular to an optimized bitmap algorithm capable of improving data deduplication and noise reduction performance. Background Art

[0002] The overall architecture of existing data flow is as follows Figure 4 As shown in the figure, when the Collector collects a piece of data, it will be sent to Kafka. The Flink program will listen to the sent data, deduplicate the data through the optimized BitMap, perform corresponding business logic processing on the deduplicated data, and send it to Kafka after processing. The Java program will then perform rule-based noise reduction and put it into the warehouse after processing.

[0003] For the above data deduplication process, there are several existing technical solutions:

[0004] Solution 1: Each time a piece of data is received, it is stored as a basis on disk or in cache, and the basis table is queried according to the dimension. If duplicates exist, the data is deduplicated. When N pieces of data are stored, N queries are required.

[0005] Problem: Every piece of data received needs to be compared. Under the pressure of PB-level data, the processing performance is low. In severe cases, the program may crash.

[0006] Solution 2: Use the most popular big data computing engine to deduplicate.

[0007] Problem: Group and deduplicate data according to the required dimensions. Ideally, we hope that the number of duplicated data in each group is equal, but in reality, for example, there are 10 data items with ID 1, while there are tens of millions of data items with ID 2. When the big data computing engine is executed, there will be a MapReduce process, and there will be a Shuffle between "Map" and "Reduce" (where performance problems are most likely to occur). Unfortunately, Solution 2 happens to cause data skew in the Shuffle stage. To solve this problem, Solution 3 is introduced below.

[0008] Solution 3: Deduplication using the Bitmap algorithm.

[0009] Problem: The Bitmap algorithm has a very obvious problem. Whether there is only one element or 4 billion elements, they all need to occupy the same space. In actual scenarios, such as product access, not every product will have so many users visiting it. Some popular products may have hundreds of millions of visits, but some less popular products may only have a few users browsing them. If the Bitmap algorithm is used for both, then the space they occupy will be the same, which is obviously unacceptable. Summary of the invention

[0010] 1. Technical issues to be resolved

[0011] In view of the above-mentioned shortcomings of the prior art, the present invention provides an optimized bitmap algorithm that can improve the performance of data deduplication and noise reduction, which can effectively overcome the defects of the prior art such as data skew that is prone to occur in the Shuffle stage, large data space occupation, and low data addressing efficiency.

[0012] (II) Technical solution

[0013] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0014] An optimized bitmap algorithm that can improve data deduplication and noise reduction performance uses an array container ArrayContainer to store the first 4096 integers. When the integers exceed 4096, a bitmap container Bitmap Container is used to store them.

[0015] Preferably, the storing the first 4096 integers using an array container Array Container includes:

[0016] Divide the 32-bit unsigned integer into high 16 bits and low 16 bits, find the corresponding array container Array Container in the container array ContainerArray through the high 16 bits, and store the low 16 bits in the corresponding array container Array Container;

[0017] If the corresponding array container ArrayContainer cannot be found in the container array ContainerArray, a new array container ArrayContainer is created.

[0018] Preferably, storing the lower 16 bits into a corresponding array container Array Container includes:

[0019] The lower 16 bits are converted into a decimal number. If the decimal number is not stored in the corresponding array container Array Container, the decimal number is stored in the array container Array Container.

[0020] Preferably, the array container Array Container is used to store sparse data, the array is always ordered, and binary search is convenient. The initial container capacity is 4, and the maximum container capacity is 4096.

[0021] Preferably, when the integers exceed 4096, a bitmap container is used for storage, including:

[0022] Divide the 32-bit unsigned integer into high 16 bits and low 16 bits, find the corresponding bitmap container Bitmap Container in the container array ContainerArray through the high 16 bits, and store the low 16 bits in the corresponding bitmap container Bitmap Container;

[0023] If the corresponding bitmap container BitmapContainer cannot be found in the container array ContainerArray, a new bitmap container Bitmap Container is created.

[0024] Preferably, storing the lower 16 bits into a corresponding bitmap container includes:

[0025] Convert the lower 16 bits to a decimal number. If the decimal number is not stored in the corresponding bitmap container, set the position represented by the decimal number in the bitmap container to 1.

[0026] Preferably, the bitmap container Bitmap Container is used to store dense data, can be addressed according to a subscript, and the container capacity is fixed at 1024.

[0027] Preferably, the upper 16 bits are shared valid bits, which are used to index which storage container to search for the corresponding value, and serve as a primary index in the database;

[0028] The lower 16 bits are used for indexing to effectively obtain the corresponding value in the storage container as a secondary index of the database.

[0029] (III) Beneficial effects

[0030] Compared with the prior art, the optimized bitmap algorithm provided by the present invention can improve the performance of data deduplication and noise reduction, and has the following beneficial effects:

[0031] 1) For scenarios with long duration and large data volume, in order to avoid long waiting time for downstream businesses, data of the same dimension is stored in the same partition in sequence, and two-level indexes are set to reduce data addressing time, improve data query efficiency, and thus improve data deduplication efficiency;

[0032] 2) To address the data skew problem that occurs during data partitioning, the Bitmap algorithm is introduced to perform data deduplication, avoiding data skew in the Shuffle phase and improving data deduplication efficiency;

[0033] 3) The data is stored by classifying it based on the amount of data and dividing it into high and low bits, which effectively optimizes the existing Bitmap algorithm, reasonably allocates system resources, and avoids wasting memory space;

[0034] 4) In terms of the time complexity of adding, deleting, modifying and searching, the Bitmap Container only involves bit operations and can be directly addressed according to the subscript, so its time complexity is obviously O(1), while the Array Container needs to use binary search to locate data in the ordered array, so its time complexity is O(logN);

[0035] 5) In terms of space complexity, the space occupied by the bitmap container is constant at 8kB, and the space occupied by the array container is related to the cardinality. Using this storage structure greatly saves memory space. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0037] Figure 1 It is a schematic diagram of the process of the present invention;

[0038] Figure 2 It is a schematic diagram of using an array container Array Container and a bitmap container Bitmap Container to store data in the present invention;

[0039] Figure 3 This is a schematic diagram of the memory size occupied by the array container Array Container and the bitmap container Bitmap Container;

[0040] Figure 4 This is the overall architecture diagram of the existing data flow. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] An optimized bitmap algorithm that can improve data deduplication and noise reduction performance, such as Figure 1 As shown, the first 4096 integers are stored in an array container Array Container, and when the integers exceed 4096, a bitmap container BitmapContainer is used for storage.

[0043] ① Use the array container Array Container to store the first 4096 integers, including:

[0044] Divide the 32-bit unsigned integer into high 16 bits and low 16 bits, find the corresponding array container Array Container in the container array ContainerArray through the high 16 bits, and store the low 16 bits in the corresponding array container Array Container;

[0045] If the corresponding array container ArrayContainer cannot be found in the container array ContainerArray, a new array container ArrayContainer is created.

[0046] Specifically, the lower 16 bits are stored in the corresponding array container Array Container, including:

[0047] The lower 16 bits are converted into a decimal number. If the decimal number is not stored in the corresponding array container Array Container, the decimal number is stored in the array container Array Container.

[0048] In the technical solution of the present application, the array container Array Container is used to store sparse data. The array is always ordered, which is convenient for binary search. The initial container capacity is 4 and the maximum container capacity is 4096.

[0049] ② When the number exceeds 4096 integers, use the bitmap container to store them, including:

[0050] Divide the 32-bit unsigned integer into high 16 bits and low 16 bits, find the corresponding bitmap container Bitmap Container in the container array ContainerArray through the high 16 bits, and store the low 16 bits in the corresponding bitmap container Bitmap Container;

[0051] If the corresponding bitmap container BitmapContainer cannot be found in the container array ContainerArray, a new bitmap container Bitmap Container is created.

[0052] Specifically, the lower 16 bits are stored in the corresponding bitmap container Bitmap Container, including:

[0053] Convert the lower 16 bits to a decimal number. If the decimal number is not stored in the corresponding bitmap container, set the position represented by the decimal number in the bitmap container to 1.

[0054] In the technical solution of the present application, the bitmap container Bitmap Container is used to store dense data, which can be addressed according to the subscript, and the container capacity is fixed at 1024.

[0055] The technical solution of the present application also includes a Run container, which is used to store continuous data and is mostly used for range operations. The specific selection and use depends on the actual business scenario.

[0056] In the technical solution of the present application, the upper 16 bits are shared valid bits, which are used to index which storage container to find the corresponding value, as a primary index in the database;

[0057] The lower 16 bits are used for indexing to effectively obtain the corresponding value in the storage container, serving as a secondary index of the database.

[0058] In order to better illustrate the technical content of the high-low bit segmentation scheme in this application, Figure 2 Given an example:

[0059] Suppose there are two decimal integers: 821697800 and 191037. Convert the two decimal integers into hexadecimal respectively: 30FA1D08 and 0002EA3D, and extract the corresponding values ​​according to the high and low bits.

[0060] For the hexadecimal number 30FA1D08, take out the high bit: 30FA, and take out the low bit: 1D08. Use the high bit 30FA to find the corresponding bitmap container in the container array ContainerArray. If it does not exist, create a new bitmap container Bitmap Container; then convert the low bit 1D08 to the decimal number 7432. If the decimal number 7432 is not stored in the corresponding bitmap container Bitmap Container (judged by checking whether the position represented by the decimal number 7432 is set to 1), then set the position represented by the decimal number 7432 in the bitmap container Bitmap Container to 1.

[0061] For the hexadecimal number 0002EA3D, take out the high bit: 0002, and the low bit: EA3D. Use the high bit 0002 to find the corresponding array container Array Container in the container array ContainerArray. If it does not exist, create a new array container Array Container. Then convert the low bit EA3D to the decimal number 59965. If the corresponding array container Array Container does not store the decimal number 59965, store the decimal number 59965 in the array container ArrayContainer.

[0062] Figure 3 This is a schematic diagram of the memory size occupied by the array container and the bitmap container. It can be seen that when the amount of data is less than 4096, the array container saves more memory space; and when the amount of data is greater than 4096, the bitmap container saves more memory space. Therefore, the array container stores sparse data, and the bitmap container stores dense data, which can minimize the waste of memory space.

[0063] The following is an explanation of the relevant professional terms that appear in this application:

[0064] 1) The BitMap algorithm uses a bit to mark the value corresponding to an element, and the key is the element. Since data is stored in bits, storage space can be greatly saved. On a 32-bit machine, an integer, such as int a, occupies 32 bits in memory, and the corresponding 32 bits can be used to represent decimal 0 to 31. The BitMap algorithm uses this idea to process the sorting and query of large amounts of data.

[0065] 2) HDFS: Hadoop Distributed File System (HDFS) is a distributed file system designed to run on general-purpose hardware. It has many similarities with existing distributed file systems, but at the same time, the differences between it and existing distributed file systems are also very obvious. HDFS is a highly fault-tolerant system suitable for deployment on cheap machines. HDFS can provide high-throughput data access and is very suitable for applications on large-scale data sets. HDFS relaxes some POSIX constraints to achieve the purpose of streaming file system data.

[0066] HDFS is highly fault-tolerant and designed to be deployed on low-cost hardware. It can provide high throughput to access application data and is suitable for applications with very large data sets. HDFS relaxes the requirements of POSIX, so that data in the file system can be accessed in a stream-based manner.

[0067] 3) MapReduce is a programming model for parallel computing of large-scale data sets (greater than 1TB). The concepts of "Map" and "Reduce" are its main ideas, both of which are borrowed from functional programming languages, and features borrowed from vector programming languages. It greatly facilitates programmers to run their programs on distributed systems without knowing distributed parallel programming. The current software implementation specifies a Map function to map a set of key-value pairs into a new set of key-value pairs, and specifies a concurrent Reduce function to ensure that each of the mapped key-value pairs shares the same key group.

[0068] 4) Shuffle: In a distributed computing framework, data locality is a very important consideration, that is, the calculation needs to be distributed to the location of the data, so as to reduce the movement of data and improve the operation efficiency. The input data of MapReduce is usually files in HDFS, so data locality requires that the Map task be scheduled to the node where the input file is stored as much as possible. However, some computing logic cannot simply obtain local data, such as the Reduce logic.

[0069] For Reduce, the input of the processing function is all values ​​with the same key, but the data sets of these values ​​(i.e., the output of Map) are located on different nodes, so the output of Map needs to be reorganized so that the same key enters the same Reducer. Shuffle moves a large amount of data, which consumes a lot of computing, memory, network, and disk. Therefore, Shuffle should be performed only where it is really needed, otherwise it should be avoided as much as possible.

[0070] 5) Time complexity: It is a parameter used to help developers estimate the running time of a program. The number of operation units of an algorithm is usually estimated to represent the running time of the program. Here, it is assumed that the time consumed by each unit of the CPU is the same. Assuming that the data scale of the algorithm is n, the number of operation units is represented by the function f(n). As the data scale n increases, the growth rate of the algorithm execution time is the same as the growth rate of f(n). This is called the asymptotic time complexity of the algorithm, or time complexity for short, denoted as O(f(n)).

[0071] The seven most common examples, in descending order of efficiency, are as follows:

[0072] O(1) — constant complexity;

[0073] O(logn) — logarithmic complexity;

[0074] O(n) — linear complexity;

[0075] O(nlogn) — logarithmic linear complexity;

[0076] O(n k )—polynomial complexity;

[0077] O(k n )—exponential complexity;

[0078] O(n!) — factorial complexity.

[0079] 6) Space complexity: It is a measure of the amount of storage space temporarily occupied by an algorithm during its operation, denoted as S(n)=O(f(n)). For example, the time complexity of direct insertion sort is O(n^2) and the space complexity is O(1), while a general recursive algorithm has a space complexity of O(n) because each recursion needs to store the return information.

[0080] The quality of an algorithm is mainly measured from two aspects: the execution time of the algorithm and the storage space required. For an algorithm, its time complexity and space complexity often affect each other. When pursuing a better time complexity, the space complexity may deteriorate, which may lead to more storage space; conversely, when pursuing a better space complexity, the time complexity may deteriorate, which may lead to a longer program running time. In addition, all the performance of the algorithm has more or less mutual influence. Therefore, when designing an algorithm (especially a large algorithm), it is necessary to comprehensively consider various performance factors of the algorithm, such as the frequency of use of the algorithm, the amount of data processed by the algorithm, the characteristics of the algorithm description language, the machine system environment in which the algorithm runs, and other factors, in order to design a better algorithm. The time complexity and space complexity of an algorithm are collectively referred to as the complexity of the algorithm.

[0081] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An optimized bitmap algorithm capable of improving data deduplication and noise reduction performance, characterized in that: The first 4096 integers are stored in an array container ArrayContainer. When the integers exceed 4096, a bitmap container Bitmap Container is used for storage.

2. The optimized bitmap algorithm capable of improving data deduplication and noise reduction performance according to claim 1, characterized in that: The use of the array container Array Container to store the first 4096 integers includes: Divide the 32-bit unsigned integer into high 16 bits and low 16 bits, find the corresponding array container Array Container in the container array ContainerArray through the high 16 bits, and store the low 16 bits in the corresponding array container ArrayContainer; If the corresponding array container Array Container cannot be found in the container array ContainerArray, a new array container Array Container is created.

3. The optimized bitmap algorithm capable of improving data deduplication and noise reduction performance according to claim 2, characterized in that: The storing of the lower 16 bits into the corresponding array container Array Container includes: The lower 16 bits are converted into a decimal number. If the decimal number is not stored in the corresponding array container Array Container, the decimal number is stored in the array container Array Container.

4. The optimized bitmap algorithm capable of improving data deduplication and noise reduction performance according to claim 2, characterized in that: The array container Array Container is used to store sparse data. The array is always ordered, which is convenient for binary search. The initial container capacity is 4 and the maximum container capacity is 4096.

5. The optimized bitmap algorithm capable of improving data deduplication and noise reduction performance according to claim 1, characterized in that: When the integer exceeds 4096, a bitmap container is used for storage, including: Divide the 32-bit unsigned integer into high 16 bits and low 16 bits, find the corresponding bitmap container Bitmap Container in the container array ContainerArray through the high 16 bits, and store the low 16 bits in the corresponding bitmap container BitmapContainer; If the corresponding bitmap container BitmapContainer cannot be found in the container array ContainerArray, a new bitmap container Bitmap Container is created.

6. The optimized bitmap algorithm capable of improving data deduplication and noise reduction performance according to claim 5, characterized in that: The storing of the lower 16 bits into the corresponding bitmap container includes: Convert the lower 16 bits to a decimal number. If the decimal number is not stored in the corresponding bitmap container, set the position represented by the decimal number in the bitmap container to 1.

7. The optimized bitmap algorithm capable of improving data deduplication and noise reduction performance according to claim 5, characterized in that: The bitmap container Bitmap Container is used to store dense data and can be addressed according to the subscript. The container capacity is fixed at 1024.

8. The optimized bitmap algorithm capable of improving data deduplication and noise reduction performance according to claim 2 or 5, characterized in that: The upper 16 bits are shared valid bits, which are used to index which storage container to search for the corresponding value, and serve as a primary index in the database; The lower 16 bits are used for indexing to effectively obtain the corresponding value in the storage container as a secondary index of the database.