A method and system for data deduplication optimization

Abstract

This invention discloses a method and system for deduplicating accumulated data, relating to the field of data management technology. The method includes the following steps: S1, data preprocessing; S2, rolling hash calculation; S3, bit operations and finite state representation; S4, deduplication decision and state update; S5, performance monitoring and adjustment. The system includes a data preprocessing module, a hash calculation module, a bit vector management module, a deduplication decision module, a performance monitoring module, and an optimization and adjustment module. Before writing data to disk, this method and system perform a hash calculation on the data and compares the hash value with the data on the disk. If the data already exists, a reference to that data is directly returned; otherwise, the data is written to disk and the index is updated. This not only effectively removes duplicate data and improves data storage efficiency but also does not occupy memory space, while ensuring the real-time performance and accuracy of data processing.

Description

Technical Field

[0001] This invention relates to the field of data management technology, specifically to a method and system for deduplicating and optimizing accumulated data. Background Technology

[0002] Accumulated data, as the name suggests, refers to the total amount of data accumulated and stored in a database or data system over a certain period of time. This data may include historical records, market analysis, user behavior, and enterprise operational data, containing a wealth of information and value. Accumulated data is a crucial foundation for data analysis, mining, and decision support. However, accumulated data often contains a large amount of duplicate data. This duplicate data increases data processing time and costs, consumes storage space, leads to massive datasets, reduces data processing efficiency, and can also affect the accuracy of analytical results. Therefore, deduplication optimization of accumulated data is essential.

[0003] Deduplication optimization, also known as duplicate removal or data deduplication, is an optimization process aimed at addressing the problem of duplicate data in accumulated data. It is a data management technique designed to improve data quality, reduce storage space requirements, and optimize data performance by eliminating redundant and duplicate data. Existing deduplication optimization methods typically compare data entries or records in a dataset and identify duplicates based on specific criteria (such as matching values ​​in key fields), then delete or merge these duplicates. Furthermore, more advanced techniques, such as block-based deduplication, file-level deduplication, or memory-cached deduplication, can be employed to improve the efficiency and accuracy of deduplication.

[0004] However, in practical use, existing deduplication algorithms generally have high computational complexity when processing large-scale datasets. This not only leads to a significant increase in processing time, but also makes it easy for data streams that need to be processed in real time to be delayed or lost due to the inability to respond in time. At the same time, a large amount of memory is required to store intermediate results or perform complex calculations. Memory consumption is a significant problem, and the algorithm may fail or its performance may degrade due to insufficient memory during execution. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for deduplicating and optimizing accumulated data, so as to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] On the one hand, a method for optimizing the deduplication of accumulated data is provided, including the following steps:

[0008] S1. Data preprocessing: Preprocess the input data to ensure a consistent data format;

[0009] S2, Rolling Hash Calculation: Select a hash function to map the preprocessed data to a fixed-size hash value, and implement the rolling hash algorithm to process the input data character by character or block by block, and update the current hash value in real time;

[0010] S3. Bit operations and finite state representation: Design a fixed-size bit vector (or bitmap) to record the hash values ​​that have been seen, and map the calculated hash values ​​to one or more bits of the bit vector. Use bit operations (such as setting bits, checking bits, etc.) to update and query the bit vector.

[0011] S4. Deduplication Decision and State Update: Make a deduplication decision based on the state of the bit vector and update the state of the bit vector to reflect the latest data reception status.

[0012] S5. Performance Monitoring and Adjustment: Monitor the performance of the deduplication algorithm, including hash collision rate, data processing speed, etc., and adjust it as needed.

[0013] Furthermore, in step S1, the specific preprocessing operations are as follows: use string processing functions to remove irrelevant characters such as spaces and newlines from the data, and convert the data into a unified encoding format (such as UTF-8) to ensure the consistency of the input data format. For data containing variable-length fields, these fields are truncated or normalized to reduce the diversity of the data.

[0014] Furthermore, in step S2, the hash function is selected as either FNV hash or MurmurHash function, both of which have low computational complexity and good distribution characteristics;

[0015] The FNV hash function calculates the hash value by performing bit operations on each byte of the input data with the current hash value. Specifically, after initializing a hash value (usually a specific prime number), for each byte of the input data, the current hash value is multiplied by a constant called the "FNV prime number" (this prime number is part of the algorithm design to ensure the distribution of hash values), and then the product is XORed with the current byte. The final hash value is then used as the output.

[0016] The MurmurHash function uses a series of complex bitwise and rotation operations to calculate the hash value. Specifically, after initializing a hash value (usually a specific seed value), for each block of input data (usually 4 bytes), the current hash value is XORed with the least significant byte of the data block. A series of bitwise operations are then performed, including multiplication, bit rotation, and addition. Depending on the length of the data block, additional processing steps may be required. If the length of the input data is not an integer multiple of the block length, the remaining data bytes are processed. Finally, a final mixing operation is performed to ensure the distribution of the hash value, and the final hash value is output.

[0017] Furthermore, in step S2, the rolling hash algorithm uses the hash value of the previous data block (or character) and the newly added data block (or character) to calculate the current hash value of the entire data stream. This eliminates the need to recalculate the hash of the entire data stream, thus avoiding the need to store the entire dataset to calculate the hash value. This method only requires maintaining a current hash state, significantly reducing memory usage. The specific process is as follows:

[0018] Initialization: Before processing begins, the hash value of the initial data block is calculated as a baseline;

[0019] Character-by-character or block-by-block processing: As the data stream continues to enter, the hash value is updated character-by-character or block-by-block, that is, the influence of old characters or data blocks is removed (by multiplying by the inverse of the "rolling factor" of a hash function and subtracting the contribution of old characters or data blocks), and the influence of new characters or data blocks is added (by multiplying by the "rolling factor" of a hash function and adding the hash contribution of new characters or data blocks).

[0020] Real-time updates: As data streams are input, the current hash value is continuously updated in real time so that the hash representation of the entire data stream can be quickly obtained at any time.

[0021] Furthermore, in step S3, the hash value is mapped to the index range of the bit vector by performing a modulo operation (modulo operation) or a bit mask operation on the hash value, and a bit segmentation strategy is adopted to avoid hash collisions (i.e., different data may generate the same hash value). The bit segmentation divides the hash value into multiple parts and maps each part to a different position in the bit vector.

[0022] Furthermore, in step S3, the specific operations of the bit operation are as follows: when new data is received, its hash value is calculated, it is mapped onto a bit vector, and the corresponding bit is checked to see if it has been set.

[0023] Furthermore, in step S4, if the corresponding bit in the bit vector has been set, the data is determined to be duplicated, and a corresponding deduplication operation (such as discarding the data) is performed; if the bit is empty, the data is accepted, and its corresponding bit is set to 1.

[0024] Furthermore, step S5 specifically includes the following sub-steps:

[0025] S51. Collision Rate Monitoring: By adding a counter during the data deduplication process, the number of hash collisions and the total number of hash calculations are recorded, and the collision rate of the bit vector is calculated, which is the probability that the same bit in the bit vector is set multiple times due to different data generating the same hash value. If the collision rate is found to be too high (exceeding the set threshold), the hash function is changed (selecting a hash function with a lower collision rate) or the size of the bit vector is increased (reducing the possibility of collision by increasing the number of bits in the bit vector).

[0026] S52. Memory usage optimization: Track memory usage through monitoring tools or custom memory usage counters, and dynamically adjust the size of bit vectors to optimize memory usage based on actual application scenarios and changes in data volume.

[0027] S53. Algorithm parameter adjustment: Based on performance monitoring results, adjust the parameters of the hash function, the size of the bit vector, etc., to optimize the deduplication effect and memory usage.

[0028] On the other hand, a system for optimizing deduplication of accumulated data is provided, applied to the aforementioned method for optimizing deduplication of accumulated data, the system comprising:

[0029] Data preprocessing module: responsible for receiving input data, removing irrelevant characters from the data, and converting the data into a unified encoding format;

[0030] Hash calculation module: responsible for implementing the rolling hash algorithm, calculating and updating the hash value of data;

[0031] Bit vector management module: responsible for managing bit vectors (or bitmaps), recording hash values, and performing bit operations to update and query the status;

[0032] Deduplication decision module: Responsible for making a decision on whether to deduplicat based on the state of the bit vector;

[0033] Performance monitoring module: Responsible for monitoring the performance of the entire system, including hash collision rate, data processing speed, etc.

[0034] Optimization and Adjustment Module: Responsible for adjusting the parameters of the hash function and the size of the bit vector based on monitoring results.

[0035] Furthermore, the system also includes:

[0036] The processor is used to execute computer instructions, causing the system to perform: a data preprocessing module, a hash calculation module, a bit vector management module, a deduplication decision module, a performance monitoring module, and an optimization and adjustment module;

[0037] A memory for storing computer instructions, wherein the memory stores a computer program that, when executed by a processor, implements a method for deduplicating accumulated data.

[0038] This invention provides a method and system for deduplicating and optimizing accumulated data, which has the following beneficial effects:

[0039] This invention utilizes the time difference between disk I / O operations and data write operations to achieve real-time deduplication and efficient storage of data. Specifically, before writing data to disk, a hash calculation is performed on the data, and the hash value is compared with the data on the disk. If the data already exists, a reference to that data is returned directly; otherwise, the data is written to disk and the index is updated. In this way, this invention not only effectively removes duplicate data and improves data storage efficiency, but also does not occupy memory space, effectively reducing system resource consumption, while ensuring the real-time performance and accuracy of data processing. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the steps of a method for optimizing deduplication of accumulated data according to the present invention;

[0041] Figure 2 This is a logical block diagram of an accumulated data deduplication optimization system according to the present invention. Detailed Implementation

[0042] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and should not be construed as limiting the scope of the invention.

[0043] like Figure 1 As shown, a method for optimizing deduplication of accumulated data includes the following steps:

[0044] S1. Data Preprocessing: Use string processing functions to remove irrelevant characters such as spaces and newlines from the data, and convert the data to a uniform encoding format (such as UTF-8) to ensure the consistency of the input data format. For data containing variable-length fields, truncate or normalize these fields to reduce data diversity.

[0045] S2. Rolling Hash Calculation: Select a hash function to map the preprocessed data to a fixed-size hash value and implement the rolling hash algorithm to process the input data character by character or block by block and update the current hash value in real time.

[0046] In this embodiment, the hash function chosen is either FNV hash or MurmurHash function, both of which have low computational complexity and good distribution characteristics.

[0047] The FNV hash function calculates a hash value by performing bitwise operations on each byte of the input data with the current hash value. Specifically, after initializing a hash value (a specific prime number), for each byte of the input data, the current hash value is multiplied by a constant called the "FNV prime number" (this prime number is part of the algorithm design to ensure the distribution of hash values), and then the product is XORed with the current byte. The final hash value is then used as the output.

[0048] The MurmurHash function uses a series of complex bitwise and rotation operations to calculate the hash value. Specifically, after initializing a hash value (a specific seed value), for each block of input data (usually 4 bytes), the current hash value is XORed with the least significant byte of the data block. A series of bitwise operations are then performed, including multiplication, bit rotation, and addition. Depending on the length of the data block, additional processing steps may be required. If the length of the input data is not an integer multiple of the block length, the remaining data bytes are processed. Finally, a final mixing operation is performed to ensure the distribution of the hash value, and the final hash value is output.

[0049] In this embodiment, the rolling hash algorithm uses the hash value of the previous data block (or character) and the newly added data block (or character) to calculate the current hash value of the entire data stream. This eliminates the need to recalculate the hash of the entire data stream, thus avoiding the need to store the entire dataset to calculate the hash value. This method only requires maintaining a current hash state, significantly reducing memory usage. The specific process is as follows:

[0050] Initialization: Before processing begins, the hash value of the initial data block is calculated as a baseline;

[0051] Character-by-character or block-by-block processing: As the data stream continues to enter, the hash value is updated character-by-character or block-by-block, that is, the influence of old characters or data blocks is removed (by multiplying by the inverse of the "rolling factor" of a hash function and subtracting the contribution of old characters or data blocks), and the influence of new characters or data blocks is added (by multiplying by the "rolling factor" of a hash function and adding the hash contribution of new characters or data blocks).

[0052] Real-time updates: As data streams are input, the current hash value is continuously updated in real time so that the hash representation of the entire data stream can be quickly obtained at any time.

[0053] S3. Bit Operations and Finite State Representation: Design a fixed-size bit vector (or bitmap) to record the hash values ​​that have been seen. Map the calculated hash values ​​to one or more bits of the bit vector. Specifically, map the hash values ​​to the index range of the bit vector by performing modulo operations (modulo operation) or bitmasking. Use a bit segmentation strategy to divide the hash values ​​into multiple parts and map each part to a different position in the bit vector to avoid hash collisions (i.e., different data may produce the same hash value). Use bit operations (such as setting bits, checking bits, etc.) to update and query the bit vector. That is, when new data is received, calculate its hash value, map it to the bit vector, and check whether the corresponding bits have been set.

[0054] S4. Deduplication Decision and State Update: Make a deduplication decision based on the state of the bit vector and update the state of the bit vector to reflect the latest data reception status. In this step, if the corresponding bit in the bit vector has been set, the data is determined to be duplicated, and the corresponding deduplication operation (such as discarding the data) is performed; if the bit is empty, the data is accepted, and its corresponding bit is set to 1.

[0055] S5. Performance Monitoring and Adjustment: Monitor the performance of the deduplication algorithm, including hash collision rate, data processing speed, etc., and adjust it as needed.

[0056] S51. Collision Rate Monitoring: By adding a counter during the data deduplication process, the number of hash collisions and the total number of hash calculations are recorded, and the collision rate of the bit vector is calculated, which is the probability that the same bit in the bit vector is set multiple times due to different data generating the same hash value. If the collision rate is found to be too high (exceeding the set threshold), the hash function is changed (selecting a hash function with a lower collision rate) or the size of the bit vector is increased (reducing the possibility of collision by increasing the number of bits in the bit vector).

[0057] S52. Memory usage optimization: Track memory usage through monitoring tools or custom memory usage counters, and dynamically adjust the size of bit vectors to optimize memory usage based on actual application scenarios and changes in data volume.

[0058] S53. Algorithm parameter adjustment: Based on performance monitoring results, adjust the parameters of the hash function, the size of the bit vector, etc., to optimize the deduplication effect and memory usage.

[0059] like Figure 2 As shown, a backlog data deduplication optimization system is applied to the backlog data deduplication optimization method described above. The system includes:

[0060] Data preprocessing module: responsible for receiving input data, removing irrelevant characters from the data, and converting the data into a unified encoding format;

[0061] Hash calculation module: responsible for implementing the rolling hash algorithm, calculating and updating the hash value of data;

[0062] Bit vector management module: responsible for managing bit vectors (or bitmaps), recording hash values, and performing bit operations to update and query the status;

[0063] Deduplication decision module: Responsible for making a decision on whether to deduplicat based on the state of the bit vector;

[0064] Performance monitoring module: Responsible for monitoring the performance of the entire system, including hash collision rate, data processing speed, etc.

[0065] Optimization and adjustment module: Responsible for adjusting the parameters of the hash function and the size of the bit vector based on monitoring results;

[0066] The processor executes computer instructions, enabling the system to perform: a data preprocessing module, a hash calculation module, a bit vector management module, a deduplication decision module, a performance monitoring module, and an optimization and adjustment module.

[0067] Memory is used to store computer instructions. The computer program stored in the memory implements a method for deduplicating accumulated data when the computer program is executed by the processor.

[0068] The embodiments of the present invention are given for illustrative and descriptive purposes only, and are not intended to be exhaustive or to limit the invention to the forms disclosed. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of the invention, and to enable those skilled in the art to understand the invention and to design various embodiments with various modifications suitable for a particular purpose.

Claims

1. A method for optimizing the deduplication of accumulated data, characterized in that, Includes the following steps: S1. Data preprocessing: Preprocess the input data to ensure a consistent data format; S2, Rolling Hash Calculation: Select a hash function to map the preprocessed data to a fixed-size hash value, and implement the rolling hash algorithm to process the input data character by character or block by block, and update the current hash value in real time; S3. Bit Operations and Finite State Representation: Design a fixed-size bit vector to record the hash values ​​that have been seen, and map the calculated hash values ​​to one or more bits of the bit vector. Use bit operations to update and query the bit vector. S4. Deduplication Decision and State Update: Make a deduplication decision based on the state of the bit vector and update the state of the bit vector to reflect the latest data reception status. S5. Performance Monitoring and Adjustment: Monitor the performance of the deduplication algorithm, including hash collision rate and data processing speed, and adjust it as needed; In step S2, the rolling hash algorithm uses the hash value of the previous data block and the newly added data block to calculate the current hash value of the entire data stream, without recalculating the hash of the entire data stream. The specific process is as follows: Initialization: Before processing begins, the hash value of the initial data block is calculated as a baseline; Character-by-character or block-by-block processing: As the data stream continues to enter, the hash value is updated character-by-character or block-by-block, that is, the effects of old characters or data blocks are removed, and the effects of new characters or data blocks are added. Real-time updates: The current hash value is continuously updated in real time as data streams are input; Step S5 specifically includes the following sub-steps: S51. Collision Rate Monitoring: By recording the number of hash collisions and the total number of hash calculations, the collision rate of the bit vector is calculated, which is the probability that different data will produce the same hash value, resulting in the same bit being set multiple times in the bit vector. If the collision rate is found to be too high, the hash function is changed or the size of the bit vector is increased. S52. Memory usage optimization: Track memory usage through monitoring tools or memory usage counters, and dynamically adjust the size of bit vectors to optimize memory usage based on actual application scenarios and changes in data volume. S53. Algorithm parameter adjustment: Based on performance monitoring results, adjust the parameters of the hash function and the size of the bit vector to optimize deduplication effect and memory usage.

2. The method for optimizing deduplication of accumulated data according to claim 1, characterized in that, In step S1, the specific preprocessing operations are as follows: use string processing functions to remove irrelevant characters from the data and convert the data into a unified encoding format. For data containing variable-length fields, perform truncation or normalization processing to reduce data diversity.

3. The method for optimizing deduplication of accumulated data according to claim 1, characterized in that, In step S2, the hash function is selected as either the FNV hash function or the MurmurHash function; The FNV hash function calculates the hash value by performing bit operations on each byte of the input data with the current hash value. Specifically, after initializing a hash value, for each byte of the input data, the current hash value is multiplied by a constant called the "FNV prime number", and then the product is XORed with the current byte. The final hash value is then used as the output. The MurmurHash function uses a series of bit operations and rotation operations to calculate the hash value. Specifically, after initializing a hash value, for each block of input data, the current hash value is XORed with the least significant byte of the data block, and a series of bit operations are performed, including multiplication, bit rotation, and addition. If the length of the input data is not an integer multiple of the block, the remaining data bytes are processed, and then the final mixing operation is performed, with the final hash value as the output.

4. The method for optimizing deduplication of accumulated data according to claim 1, characterized in that, In step S3, the hash value is mapped to the index range of the bit vector by performing a modulo operation or bitmasking operation on the hash value, and a bit segmentation strategy is adopted to avoid hash collisions. The bit segmentation divides the hash value into multiple parts and maps each part to a different position in the bit vector.

5. The method for optimizing deduplication of accumulated data according to claim 1, characterized in that, In step S3, the specific operations of bit operations are as follows: when new data is received, its hash value is calculated, it is mapped onto a bit vector, and the corresponding bit is checked to see if it has been set.

6. The method for optimizing deduplication of accumulated data according to claim 1, characterized in that, In step S4, if the corresponding bit in the bit vector has been set, the data is determined to be duplicated, and a corresponding deduplication operation is performed; if the bit is empty, the data is accepted, and its corresponding bit is set to 1.

7. A system for optimizing deduplication of accumulated data, applied to the method for optimizing deduplication of accumulated data as described in any one of claims 1-6, characterized in that, The system includes: Data preprocessing module: responsible for receiving input data, removing irrelevant characters from the data, and converting the data into a unified encoding format; Hash calculation module: responsible for implementing the rolling hash algorithm, calculating and updating the hash value of data; Bit vector management module: responsible for managing bit vectors, recording hash values, and performing bit operations to update and query the status; Deduplication decision module: Responsible for making a decision on whether to deduplicat based on the state of the bit vector; Performance monitoring module: responsible for monitoring the performance of the entire system, including hash collision rate and data processing speed; Optimization and Adjustment Module: Responsible for adjusting the parameters of the hash function and the size of the bit vector based on monitoring results.

8. The accumulated data deduplication optimization system according to claim 7, characterized in that, The system also includes: The processor is used to execute computer instructions, causing the system to perform: a data preprocessing module, a hash calculation module, a bit vector management module, a deduplication decision module, a performance monitoring module, and an optimization and adjustment module; A memory for storing computer instructions, wherein the memory stores computer programs.

Images