Data reduction method and device, equipment, storage medium and computer program product
By performing similar aggregation of data blocks and deduplicating variable-length eigenvalues, the problem of inefficiency of traditional merge compression methods in processing massive data is solved, and efficient data compression is achieved.
Patent Information
- Application Number
- CN202411407814.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-09
- Publication Date
- 2025-01-24
AI Technical Summary
Traditional merging and compression methods are less efficient when processing massive data, mainly because each data block needs to be compared one by one to calculate the similarity, resulting in excessive consumption of performance resources.
By dividing the data to be processed into multiple data blocks, and splitting these data blocks into small blocks of different lengths, calculating similar feature values of the small blocks for similar aggregation, using sliding windows to calculate variable length feature values and deduplication, Finally, the deduplicated data blocks are merged and compressed.
Significantly reduces the amount of data required to be compressed, improves data processing efficiency, and achieves efficient data compression.
Smart Images
Figure CN119357143A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a data reduction method, apparatus, computer equipment, storage medium and computer program product. Background Art
[0002] As storage technology continues to develop, the amount of user data is also growing, and the storage space required to store large amounts of data has become an expensive and challenging task. Data reduction technology can reduce the size of data, save storage resources such as hard disks and cloud storage, and reduce storage costs.
[0003] In traditional technology, merge compression is a relatively common compression method. Specifically, merge compression identifies and eliminates redundant information in the data, merges duplicate data into a single instance, and reduces data storage and transmission costs. Merge compression has many advantages, but the compression effect of merge compression is closely related to the composition of the merged blocks. Only when data blocks with high similarity are combined for merge compression, the compression rate of merge compression can be higher.
[0004] In traditional technology, the similarity between data blocks usually requires calculating the similarity feature values of each data block, and then comparing the feature values of these data blocks one by one to obtain a merged and compressed block with high similarity. With the continuous development of storage technology, storage capacity has been greatly improved. Comparing massive data one by one consumes a lot of performance resources, which makes the traditional merge compression method less efficient. Summary of the invention
[0005] Based on this, it is necessary to provide an efficient data reduction method, apparatus, computer device, storage medium and computer program product to address the above technical problems.
[0006] In a first aspect, the present application provides a data reduction method. The method comprises:
[0007] Acquire data to be processed, and divide the data to be processed into multiple data blocks;
[0008] Split multiple data blocks into smaller blocks of different lengths;
[0009] Calculating similar feature values of small blocks between the multiple data blocks during the splitting process, and similarly aggregating the multiple data blocks based on the similar feature values;
[0010] Using a sliding window to calculate variable-length feature values of similar aggregated data blocks, and deduplicating similar aggregated data blocks based on the variable-length feature values;
[0011] Merge and compress similar aggregated data blocks after deduplication.
[0012] The above data reduction method first obtains the data to be processed and divides the data to be processed into multiple data blocks; then splits the multiple data blocks into small blocks of different lengths; calculates the similarity feature values of the small blocks between the multiple data blocks during the splitting process, and similarly aggregates the multiple data blocks based on the similar feature values; then uses a sliding window to calculate the variable-length feature values of the similar aggregated data blocks, and deduplicates the similar aggregated data blocks based on the variable-length feature values; finally, merges and compresses the deduplicated similar aggregated data blocks. In the whole process, similar aggregation and deduplication based on variable-length feature values for the data to be processed can significantly reduce the amount of data that needs to be compressed, and ultimately achieve efficient data compression.
[0013] In one embodiment, splitting the multiple data blocks into small blocks of different lengths includes:
[0014] A greedy algorithm is used to split multiple data blocks into small blocks of different lengths.
[0015] In this embodiment, a greedy algorithm is used to split multiple data blocks. Since the greedy algorithm can take the best or optimal (i.e., most favorable) option in the current state in each selection step, each data block is reasonably split into small blocks of appropriate lengths.
[0016] In one embodiment, the calculating similarity feature values of small blocks among the multiple data blocks during the splitting process, and similarly aggregating the multiple data blocks based on the similarity feature values includes:
[0017] Calculating similar feature values of small blocks between the multiple data blocks in the splitting process;
[0018] Based on the similar eigenvalues, generating a similarity relationship matrix between different data blocks;
[0019] The multiple data blocks are similarly aggregated according to the similarity relationship matrix.
[0020] In this embodiment, the similar eigenvalues of small blocks between multiple data blocks are first calculated, and then a similarity relationship matrix is generated based on the similar eigenvalues. Finally, similarity aggregation is performed based on the similarity relationship matrix. This can significantly reduce the amount of data required to be processed during the eigenvalue comparison process, thereby improving data processing efficiency.
[0021] In one embodiment, generating a similarity relationship matrix between different data blocks based on the similar eigenvalues includes:
[0022] Obtain similar feature values of an original small block and a comparison small block, wherein the original small block and the comparison small block are small blocks in a first data block and a second data block respectively; the first data block and the second data block are any two different data blocks in the multiple data blocks;
[0023] Arranging similar feature values of the original small block and the comparison small block into a feature matrix;
[0024] Based on the feature matrix, a dimensionality reduction index algorithm is used to calculate a similarity relationship matrix between the first data block and the second data block.
[0025] In this embodiment, the similar feature values of the original small block and the compared small block are arranged into a feature matrix to compress the amount of data required for comparison processing during comparison, and a dimensionality reduction index algorithm is used to calculate the similarity relationship matrix, which significantly reduces the amount of data processing in the entire similarity comparison process and improves data processing efficiency.
[0026] In one embodiment, the step of calculating variable-length feature values of similar aggregated data blocks by using a sliding window, and deduplicating similar aggregated data blocks based on the variable-length feature values includes:
[0027] Obtaining a preset sliding window size, and slidingly intercepting data from similar aggregated data blocks according to the preset sliding window size;
[0028] Respectively calculating first variable-length feature values corresponding to sliding intercepted data of different similar aggregated data blocks;
[0029] Comparing the first variable-length feature values corresponding to different similar aggregated data blocks to obtain a first variable-length feature value comparison result;
[0030] Deduplication of similar aggregated data blocks is performed based on the comparison result of the first variable-length feature value.
[0031] In this embodiment, data is intercepted based on a preset sliding window, and then the first variable-length feature value of the intercepted data is calculated, and then the same data blocks are identified based on the first variable-length feature value to deduplicate similar aggregated data blocks. Since variable-length feature value calculation and comparison are only performed on the intercepted data, the amount of data required for comparison can be significantly reduced, thereby improving data processing efficiency.
[0032] In one embodiment, deduplicating similar aggregated data blocks according to the comparison result of the first variable-length feature value includes:
[0033] According to the comparison result of the first variable-length feature value, searching for similar aggregated data block pairs having the same first variable-length feature value to obtain candidate similar aggregated data block pairs;
[0034] Sliding and intercepting data again for the candidate similar aggregated data block pair based on the preset sliding window size;
[0035] Calculate the second variable-length eigenvalue corresponding to the two sliding intercepted data;
[0036] Searching for candidate similar aggregated data block pairs having the same second variable-length feature value to obtain target similar aggregated data block pairs;
[0037] Deduplication is performed on the target similar aggregated data block pairs.
[0038] In this embodiment, candidate similar aggregated data block pairs are first determined based on the first variable-length eigenvalue, and then a second sliding interception and second variable-length eigenvalue calculation and comparison are performed on this basis. The accuracy of identifying identical data blocks is significantly improved through two variable-length eigenvalue comparisons, avoiding false deduplication. On the basis of improving data processing efficiency, the accuracy of deduplication can also be guaranteed.
[0039] In one embodiment, deduplicating the target similar aggregated data block pair comprises:
[0040] Comparing two data blocks in the target similar aggregated data block pair byte by byte;
[0041] If the byte-by-byte comparison result indicates that the two data blocks are completely identical, deduplication is performed on the target similar aggregated data block pair.
[0042] In this embodiment, a byte-by-byte comparison is performed on the basis of the above two variable-length feature value comparisons. On the one hand, it avoids large-scale byte-by-byte comparison and reduces the amount of data processing; on the other hand, it further improves the recognition accuracy of the same data blocks and the accuracy of deduplication.
[0043] In one embodiment, the data reduction method further includes:
[0044] If no similar aggregated data block having the first variable-length feature value is found according to the comparison result of the first variable-length feature value, it is determined that no completely identical similar aggregated data block exists.
[0045] In this embodiment, when it is found that no similar aggregated data block with the first variable-length characteristic value is searched, it is determined that there is no completely identical similar aggregated data block in the current similar aggregated data blocks, and subsequent processing is directly performed to ensure that the entire merging and compression process proceeds normally.
[0046] In one embodiment, after merging and compressing the similar aggregated data blocks after deduplication, the method further includes:
[0047] The combined compressed data blocks and the variable-length feature values are stored in association.
[0048] In this embodiment, the merged and compressed data blocks and the variable-length feature values are stored in association, so that when the data blocks are subsequently deduplicated, it is also possible to identify whether there is a data block identical to the current data block in the stored data based on the variable-length feature value. If so, the current data block is directly discarded to reduce unnecessary data storage.
[0049] In a second aspect, the present application also provides a data reduction device. The device comprises:
[0050] A data partitioning module, used for acquiring data to be processed and partitioning the data to be processed into a plurality of data blocks;
[0051] A data splitting module is used to split multiple data blocks into small blocks of different lengths;
[0052] A similarity aggregation module, used to calculate similarity feature values of small blocks between the multiple data blocks in the splitting process, and similarly aggregate the multiple data blocks based on the similarity feature values;
[0053] A deduplication module, used to calculate variable-length feature values of similar aggregated data blocks using a sliding window, and to deduplicate similar aggregated data blocks based on the variable-length feature values;
[0054] The compression module is used to compress similar aggregated data blocks after deduplication.
[0055] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:
[0056] Acquire data to be processed, and divide the data to be processed into multiple data blocks;
[0057] Split multiple data blocks into smaller blocks of different lengths;
[0058] Calculating similar feature values of small blocks between the multiple data blocks during the splitting process, and similarly aggregating the multiple data blocks based on the similar feature values;
[0059] Using a sliding window to calculate variable-length feature values of similar aggregated data blocks, and deduplicating similar aggregated data blocks based on the variable-length feature values;
[0060] Merge and compress similar aggregated data blocks after deduplication.
[0061] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:
[0062] Acquire data to be processed, and divide the data to be processed into multiple data blocks;
[0063] Split multiple data blocks into smaller blocks of different lengths;
[0064] Calculating similar feature values of small blocks between the multiple data blocks during the splitting process, and similarly aggregating the multiple data blocks based on the similar feature values;
[0065] Using a sliding window to calculate variable-length feature values of similar aggregated data blocks, and deduplicating similar aggregated data blocks based on the variable-length feature values;
[0066] Merge and compress similar aggregated data blocks after deduplication.
[0067] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:
[0068] Acquire data to be processed, and divide the data to be processed into multiple data blocks;
[0069] Split multiple data blocks into smaller blocks of different lengths;
[0070] Calculating similar feature values of small blocks between the multiple data blocks during the splitting process, and similarly aggregating the multiple data blocks based on the similar feature values;
[0071] Using a sliding window to calculate variable-length feature values of similar aggregated data blocks, and deduplicating similar aggregated data blocks based on the variable-length feature values;
[0072] Merge and compress similar aggregated data blocks after deduplication.
[0073] The above-mentioned data reduction device, computer equipment, storage medium and computer program product obtain the data to be processed, divide the data to be processed into multiple data blocks; split the multiple data blocks into small blocks of different lengths; calculate the similarity feature values of the small blocks between the multiple data blocks during the splitting process, and similarly aggregate the multiple data blocks based on the similar feature values; use a sliding window to calculate the variable-length feature values of the similar aggregated data blocks, and deduplicate the similar aggregated data blocks based on the variable-length feature values; merge and compress the deduplicated similar aggregated data blocks. In the whole process, similar aggregation and deduplication based on variable-length feature values for the data to be processed can significantly reduce the amount of data required to be compressed, and ultimately achieve efficient data compression. BRIEF DESCRIPTION OF THE DRAWINGS
[0074] FIG1 is a diagram of an application environment of a data reduction method in one embodiment;
[0075] FIG2 is a schematic diagram of a flow chart of a data reduction method in one embodiment;
[0076] FIG3 is a schematic diagram of a similar feature calculation process between data blocks;
[0077] FIG4 is a schematic flow chart of a data reduction method in another embodiment;
[0078] FIG5 is a schematic diagram of the principle of similar feature dimensionality reduction indexing;
[0079] FIG6 is a schematic diagram of a sub-process of S400 in one embodiment;
[0080] FIG7 is a schematic diagram of sliding window calculation of variable length eigenvalues;
[0081] FIG8 is a schematic diagram showing the principle of applying a data reduction method to a system in an application example;
[0082] FIG9 is a flow chart of a data reduction method in an application example;
[0083] FIG10 is a structural block diagram of a data reduction device in one embodiment;
[0084] FIG. 11 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0085] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0086] The data reduction method provided in the embodiment of the present application can be applied in the application environment shown in FIG. 1. The terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 sends a data reduction request to the server 104, and the server 104 responds to the request, extracts the data to be processed from the request, and divides the data to be processed into multiple data blocks; splits the multiple data blocks into small blocks of different lengths; calculates the similar feature values of the small blocks between the multiple data blocks during the splitting process, and similarly aggregates the multiple data blocks based on the similar feature values; uses a sliding window to calculate the variable-length feature values of the similar aggregated data blocks, and deduplicates the similar aggregated data blocks based on the variable-length feature values; and merges and compresses the deduplicated similar aggregated data blocks. The terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable device may be a smart watch, a smart bracelet, a head-mounted device, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.
[0087] In one embodiment, as shown in FIG. 2 , a data reduction method is provided, which is described by taking the method applied to the server 104 in FIG. 1 as an example, and includes the following steps:
[0088] S100: Obtain data to be processed, and divide the data to be processed into multiple data blocks.
[0089] The data to be processed refers to the data that needs to be reduced this time, which can be a batch of data extracted from a database. After obtaining the data to be processed, the data to be processed is divided into a plurality of data blocks. Specifically, the division here can be to divide the data to be processed according to the same size, for example, the data to be processed can be divided into a plurality of data blocks of the same size according to the size of 8k, 16k, etc.
[0090] S200: Split multiple data blocks into small blocks of different lengths.
[0091] For the divided data blocks, they are split into small blocks of different lengths. Here, each data block is split separately, and each data block is split into small blocks of different lengths. Specifically, a sliding splitting method can be used in the splitting process to align the position of each small block in the data block, so as to facilitate the subsequent similarity aggregation based on characteristic values. Further, the small blocks after the splitting of a single data block can be grouped together. Assume that the data to be processed is divided into {A1, A2, ..., AN} data blocks in S100, and here A1, A2, ..., AN are split into small blocks of different lengths, for example, data block A1 is split into a11, a12, ..., a1m small blocks of different lengths, and data block A2 can be split in the same way, which will not be repeated here.
[0092] S300: Calculate similar feature values of small blocks between multiple data blocks during the splitting process, and perform similar aggregation on the multiple data blocks based on the similar feature values.
[0093] In the process of splitting multiple data blocks into multiple small blocks, similar feature values of small blocks between different data blocks are calculated respectively. Specifically, a first data block and a second data block can be randomly selected from multiple data blocks, and the first data block is used as the original data and the second data block is used as the comparison data. While splitting the first data block and the second data block into different small blocks, similar feature values between the split small blocks are calculated. Based on the similar feature values between these small blocks, the similarity between the first data block and the second data block can be obtained. By adopting the above-mentioned similar processing methods for different first data blocks and second data blocks respectively, similar feature values of small blocks between multiple data blocks can be obtained. These similar feature values characterize the similarity between different data blocks, and similar aggregation is performed on multiple data blocks based on the similarity between different data blocks.
[0094] Furthermore, in the splitting process of S200, a greedy algorithm may be used to split the multiple data blocks into small blocks of different lengths. Specifically, a greedy algorithm is an algorithm that takes the best or optimal (i.e., most favorable) choice in the current state in each step of selection, thereby hoping to lead to a globally best or optimal result.
[0095] In the traditional technology, directly calculating the similarity between two data blocks requires a lot of computing resources. Taking Jaccard similarity calculation as an example, the similarity between two data blocks is the ratio of the intersection of the elements of the two data to the union of the elements of the two data. If the two data blocks are completely similar, the similarity is 1, otherwise it is 0. If you want to get the intersection and union of two data, you need to compare each element of each two data one by one, which will consume a lot of resources. In view of the situation, in this application, the processing methods of S200 and S300 are used to calculate the similarity. Further, here, the greedy algorithm can be used to divide the data into multiple small blocks of different lengths, and then a similar feature value is calculated for each small block of the corresponding position of the sliding split in two different data blocks, and the similarity of the data is determined by comparing the similar feature values. The specific process of calculating the similar feature value between the two data blocks is shown in Figure 3. The dividing line of the original data (first data block) and the comparison data (second data block) will slide at the same time, and the similar feature values of the original data and the comparison data will be calculated simultaneously during the sliding process. When the similarity between two similar feature values meets the set threshold, the dividing line stops sliding. When the two pieces of data are completely different, the dividing line will stop sliding when the length reaches a certain upper limit. The similar feature values are accumulated and calculated, and multiple data blocks are similarly aggregated based on the similar feature values.
[0096] S400: Calculate variable-length feature values of similar aggregated data blocks using a sliding window, and deduplicate similar aggregated data blocks based on the variable-length feature values.
[0097] For similar aggregated data blocks, a sliding window method is used to calculate their variable-length feature values, and then the data blocks that are exactly the same in the similar aggregated data blocks are searched based on the variable-length feature values, and these identical data blocks are deduplicated. For example, assuming that the similar aggregated data blocks obtained after S300 processing are {A1, A3, A5}, a sliding window method is further used here to calculate the variable-length feature values of A1, A3 and A5, and to determine whether the variable-length feature values of A1, A3 and A5 are the same. If it is found that the variable-length feature values of A1 and A5 are the same, it indicates that A1 and A5 may be complete data blocks. At this time, the A1 and A5 data blocks are further analyzed. If it is determined that the two are indeed the same data blocks, A1 and A5 are deduplicated, that is, only A1 is retained or only A5 is retained for subsequent merging and compression.
[0098] Specifically, the sliding window calculation here refers to intercepting data from similar aggregated data blocks according to a preset window size, and then calculating the variable-length feature values of the intercepted part of the data. If the variable-length feature values are the same, it indicates that the two similar aggregated data blocks are very likely to be the same data blocks. At this time, further analysis can be performed to deduplicate.
[0099] S500: Merge and compress similar aggregated data blocks after deduplication.
[0100] The similar aggregated data blocks after deduplication have aggregated similar data together, and performed deduplication operations to remove duplicate data. The data is then merged and compressed, which can significantly reduce the amount of data processing for merging and compression, thereby improving the efficiency of merging and compression.
[0101] The above data reduction method obtains the data to be processed, divides the data to be processed into multiple data blocks; splits the multiple data blocks into small blocks of different lengths; calculates the similarity feature values of the small blocks between the multiple data blocks during the splitting process, and similarly aggregates the multiple data blocks based on the similar feature values; uses a sliding window to calculate the variable-length feature values of the similar aggregated data blocks, and deduplicates the similar aggregated data blocks based on the variable-length feature values; and merges and compresses the deduplicated similar aggregated data blocks. In the whole process, similar aggregation and deduplication based on variable-length feature values for the data to be processed can significantly reduce the amount of data required to be compressed, and ultimately achieve efficient data compression.
[0102] As shown in FIG. 4 , in one embodiment, S300 includes:
[0103] S320: Calculate similar feature values of small blocks between multiple data blocks during the splitting process.
[0104] Based on reference Figure 3, the data block is continuously split into small blocks of different sizes during the sliding splitting process. For the first data block (original data) and the second data block (comparison data) in the multiple data blocks, the two data segmentation lines slide simultaneously to generate corresponding small blocks, and calculate the similarity feature values between the two small blocks. For example, the small block divided from the original data at time point 1 in Figure 3 is X1-1; the small block divided from the comparison data at time point 1 is X2-1, and the similarity feature value between X1-1 and X2-1 is calculated to be H1-1; correspondingly, the similarity feature value between X2-1 and X1-1 is recorded as H2-1; enter the next time point (time point 2), and the small block divided from time point 2 is processed in a similar manner as described above, and finally the similarity feature values of the small blocks between the multiple data blocks are obtained.
[0105] It should be noted that in the above sliding process, the two data segmentation lines slide simultaneously until X1-i and X2-i have similar data or the segmentation length exceeds the upper limit. That is, in the above sliding process, when the similarity of the two similar feature values meets the set threshold, the segmentation line stops sliding. When the two pieces of data are completely different, the segmentation line will also stop when the sliding length reaches a certain upper limit, and finally the similar feature values of each original data and the comparison data can be obtained.
[0106] S340: Generate a similarity relationship matrix between different data blocks based on similar eigenvalues.
[0107] Based on the similar eigenvalues obtained in S320, the dimensionality reduction is performed for different data blocks in a matrix manner to generate a similarity relationship matrix between different data blocks. Specifically, if there is a large amount of original data to be compared, the cost of comparing the n similar eigenvalues in each data block with the n similar eigenvalues in another data block is still very huge, so these features will also be processed by dimensionality reduction indexing to generate a similarity relationship matrix between different data blocks.
[0108] In one embodiment, generating a similarity relationship matrix between different data blocks based on similar eigenvalues includes: obtaining similarity eigenvalues of an original small block and a comparison small block, wherein the original small block and the comparison small block are small blocks in a first data block and a second data block respectively; the first data block and the second data block are any two different data blocks in a plurality of data blocks; arranging similarity eigenvalues of the original small block and the comparison small block into a feature matrix; and calculating a similarity relationship matrix between the first data block and the second data block using a dimensionality reduction indexing algorithm based on the feature matrix.
[0109] As mentioned above, the first data block and the second data block here do not specifically refer to any data block among the multiple data blocks, but the first data block is used as the original data and the second data block is used as the comparison data for processing. In the process of generating the similarity relationship matrix, the similar feature values of the original small block and the comparison small block are arranged into a feature matrix, and then the dimensionality reduction index algorithm is used to calculate the similarity relationship matrix between the first data block and the second data block. Specifically, the dimensionality reduction index algorithm is an important technology in the field of machine learning and data mining. It aims to improve data processing efficiency by reducing the dimension of the data set while retaining as much useful information as possible in the data.
[0110] Furthermore, as shown in FIG5 , in actual applications, assuming that there are currently m data blocks, and n similar eigenvalues in each data block must be compared with n similar eigenvalues in another block, the similar eigenvalues of the original data will be calculated first, and then these eigenvalues will be arranged into an m×n matrix, and then the similarity relationship matrix (MH1, MH2, …, MHm) will be calculated using the dimensionality reduction index algorithm. By processing the data of the MHm matrix, the similarity relationship of the original data can be obtained, and a batch of data blocks with high similarity can be screened from a large number of data blocks.
[0111] S360: Aggregate multiple data blocks by similarity according to the similarity relationship matrix.
[0112] The similarity between multiple data blocks has been accurately represented in the similarity relationship matrix, and the data has been processed with dimensionality reduction, which significantly reduces the amount of data processing. It can achieve efficient and accurate similarity aggregation of multiple data blocks to obtain similar aggregated data blocks.
[0113] As shown in FIG. 6 , in one embodiment, S400 includes:
[0114] S420: Obtain a preset sliding window size, and slide and intercept data from similar aggregated data blocks according to the preset sliding window size.
[0115] The preset sliding window size is a pre-set window size, which can be specifically 1 / 4 or 1 / 5, etc. That is, 1 / 4 or 1 / 5 of the entire similar aggregated data block is selected here to intercept data. For example, if the similar aggregated data block is {A1, A3, A5}, then a sliding window of 1 / 4 or 1 / 5 is used to intercept the data in A1, A3, and A5 respectively.
[0116] S440: Calculate first variable-length feature values corresponding to sliding intercepted data of different similar aggregated data blocks respectively.
[0117] For each similar aggregated data block, the first variable-length feature value corresponding to each intercepted data is calculated respectively. Specifically, the variable-length feature value can be understood as a feature identifier or measurement value with a variable length extracted from the data block.
[0118] S460: Compare the first variable-length feature values corresponding to different similar aggregated data blocks to obtain a first variable-length feature value comparison result.
[0119] After calculating the first variable-length feature value FP1 corresponding to different similar aggregated data blocks, the FP1s of the different similar aggregated data blocks are compared to obtain the first variable-length feature value comparison result. For example, the first variable-length feature value FP1 corresponding to the similar aggregated data blocks {A1, A3, A5} is calculated respectively, and then the FP1s of A1 and A3, A3 and A5, and A1 and A5 are compared to obtain the first variable-length feature value comparison result.
[0120] S480: De-duplicate similar aggregated data blocks according to the comparison result of the first variable-length feature value.
[0121] The comparison result of the first variable-length eigenvalue represents the degree of similarity of the data intercepted by the compared similar aggregated data blocks. If the first variable-length eigenvalues of two similar aggregated data blocks are the same, it indicates that the two similar aggregated data blocks are most likely the same. At this time, the two similar aggregated data blocks can be deduplicated to significantly reduce the amount of data.
[0122] Furthermore, in some scenarios, the first variable-length feature value of two similar aggregated data blocks may be the same, but in fact, they are not the same data blocks, but the variable-length feature values corresponding to the data intercepted in the current time window are the same, so in this case, there may be "false deduplication". In response to this situation, other methods of judgment can be further performed based on the first variable-length feature value to identify completely identical similar aggregated data blocks for deduplication.
[0123] In one embodiment, deduplication of similar aggregated data blocks according to the comparison result of the first variable-length feature value includes:
[0124] Step 1: Based on the comparison result of the first variable-length feature value, search for similar aggregated data block pairs with the same first variable-length feature value to obtain candidate similar aggregated data block pairs.
[0125] According to the comparison result of the first variable-length feature value, a pair of similar aggregated data blocks with the same first variable-length feature value is searched from multiple similar aggregated data blocks. Specifically, the similar aggregated data block pair here generally includes two similar aggregated data blocks, and the first variable-length feature values of the two similar aggregated data blocks are the same. For example, the similar aggregated data blocks A1 and A5 mentioned above have the same FP1, and the two similar aggregated data blocks are paired to form a candidate similar aggregated data block pair of A1-A5. If a pair of similar aggregated data blocks with the same first variable-length feature value is not found, it means that there are no identical data blocks in the similar aggregated data blocks, and no data blocks need to be deduplicated, and they can be directly merged and compressed.
[0126] Step 2: Slide and intercept data again for candidate similar aggregated data block pairs based on a preset sliding window size.
[0127] For the candidate similar aggregated data block pair, data is intercepted again based on the preset sliding window size. That is, here, sliding window data interception is performed for the two similar aggregated data blocks included in the candidate similar aggregated data block pair. For example, 1 / 5 of the data in the A1, A3, and A5 data blocks are intercepted for the first time, and then the first variable-length feature values corresponding to the intercepted data of these three data blocks are calculated. Based on the first variable-length feature values, it is identified that the first variable-length feature values of A1 and A5 are the same. At this time, based on the first interception, the window is slid again to continue intercepting new 1 / 5 of the data in A1 and A5.
[0128] Step 3: Calculate the second variable-length eigenvalue corresponding to the two sliding intercepted data.
[0129] The data intercepted by two slides are accumulated, and the second variable length feature value corresponding to the two slide intercepted data is calculated. That is, in the embodiment, based on the first interception of 1 / 5 data of A1 and A5, the data of A1 / 5 is intercepted again, and a total of 2 / 5 data of A1 and A5 are intercepted to calculate the corresponding second variable length feature value FP2.
[0130] Step 4: Search for candidate similar aggregated data block pairs with the same second variable-length feature value to obtain target similar aggregated data block pairs.
[0131] Continue searching for candidate similar aggregated data block pairs with the same second variable-length eigenvalue, that is, here, find completely identical similar aggregated data block pairs based on the side length eigenvalues of the data intercepted by the two sliding windows, and then deduplicate the similar aggregated data block pairs.
[0132] Step 5: De-duplicate the target similar aggregated data block pairs.
[0133] The target similar aggregated data block pair obtained in step 4 can be directly deduplicated. Furthermore, the target similar aggregated data block pair can be further compared byte by byte to more fully and accurately determine whether the two data blocks are completely identical data blocks. If so, the target similar aggregated data block pair is directly deduplicated; if not, no deduplication is performed.
[0134] In order to explain in detail the process of deduplication processing of similar aggregated data blocks in the above embodiment, a specific application example will be used for detailed description.
[0135] This is shown in Figure 7. First, select a part of the data to calculate the variable-length eigenvalue, for example, select 1 / 4 or 1 / 5 of the entire block of data to calculate a variable-length eigenvalue FP1, and then use FP1 to compare with the variable-length eigenvalues of the same length of other data. If there is no completely identical variable-length eigenvalue, it means that there must be no identical data in the database. If the variable-length eigenvalues of other data are consistent with the variable-length eigenvalues of this data, a sliding calculation is performed on part of the data involved in the eigenvalue calculation, and a new variable-length eigenvalue FP2 is used to compare with the corresponding length variable-length eigenvalue of the same data block compared last time. When the variable-length eigenvalues calculated by selecting all the data content are still consistent, the probability that the two data blocks are completely consistent will be very high, and then the two copies of the data are compared byte by byte to confirm whether the two data are exactly the same.
[0136] In one embodiment, after merging and compressing the deduplicated similar aggregated data blocks, the method further includes:
[0137] The combined compressed data blocks and variable-length feature values are stored in an associated manner.
[0138] In this embodiment, the merged and compressed data blocks and the corresponding variable-length feature values are stored in association in the database, so that in subsequent processing, the variable-length feature values of the data blocks currently being merged and compressed can be compared based on the variable-length feature values of the data blocks already stored in the database to determine whether there are data blocks already stored in the database in the data blocks currently being merged and compressed. If so, the data blocks can be directly deduplicated. That is, in actual applications, after the merged and compressed data blocks and the variable-length feature values are stored in association, when deduplication processing is performed, in addition to removing duplicate data blocks in similar aggregated data blocks, data blocks identical to those already stored in the database can also be removed.
[0139] In practical applications, the data reduction method of the present application can be applied to a data processing system as shown in FIG8 , in which a data processing module, a merging and compression module and a database module are included. When the entire data processing system is running, the data processing module takes out different types of fixed-length data blocks from the storage database; calculates similar feature values of these taken-out data blocks through dimensionality reduction operation; determines whether there are similar data blocks, and if not, returns incompressible data to the database module for re-storage; if so, uses a sliding window to calculate variable-length feature values, finds identical data, reads existing variable-length feature values and data, removes identical data blocks, and determines the composition of the merged and compressed blocks. The merged and compressed module uses a merged and compressed algorithm to merge and compress the merged blocks, and then stores the merged and compressed results and new feature values in the database module.
[0140] The overall processing flow of the data reduction method of this application is shown in Figure 9, which specifically includes the following steps:
[0141] 1. Take out a batch of data from the storage device, each data has the same size;
[0142] 2. Calculate similar eigenvalues of the data and perform dimensionality reduction on the similar eigenvalues;
[0143] 3. Use sliding windows to calculate variable-length eigenvalues and remove duplicate data;
[0144] 4. Select data with similar eigenvalues and determine the unit block composition of the merged compressed block;
[0145] 5. Compress the merged compressed block using the merge compression algorithm;
[0146] 6. Add the shortened data and variable length feature values to the storage device.
[0147] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0148] Based on the same inventive concept, the embodiment of the present application also provides a device for implementing the data reduction method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data reduction device embodiments provided below can refer to the limitations on the data reduction method above, and will not be repeated here.
[0149] In one embodiment, as shown in FIG10 , a data reduction device is provided, including:
[0150] The data partitioning module 100 is used to obtain the data to be processed and partition the data to be processed into a plurality of data blocks;
[0151] The data splitting module 200 is used to split the multiple data blocks into small blocks of different lengths;
[0152] A similarity aggregation module 300 is used to calculate similarity feature values of small blocks between multiple data blocks during the splitting process, and to perform similarity aggregation on multiple data blocks based on the similarity feature values;
[0153] A deduplication module 400, configured to calculate variable-length feature values of similar aggregated data blocks using a sliding window, and to dedupe similar aggregated data blocks based on the variable-length feature values;
[0154] The compression module 500 is used to compress the similar aggregated data blocks after deduplication.
[0155] In one embodiment, the data splitting module 200 is further configured to split the multiple data blocks into small blocks of different lengths using a greedy algorithm.
[0156] In one embodiment, the similarity aggregation module 300 is also used to calculate similarity feature values of small blocks between multiple data blocks during the splitting process; based on the similarity feature values, generate a similarity relationship matrix between different data blocks; and similarly aggregate multiple data blocks according to the similarity relationship matrix.
[0157] In one embodiment, the similarity aggregation module 300 is also used to obtain similar feature values of the original small block and the comparison small block, where the original small block and the comparison small block are small blocks in the first data block and the second data block respectively; the first data block and the second data block are any two different data blocks in multiple data blocks; the similar feature values of the original small block and the comparison small block are arranged into a feature matrix; based on the feature matrix, a dimensionality reduction index algorithm is used to calculate a similarity relationship matrix between the first data block and the second data block.
[0158] In one of the embodiments, the deduplication module 400 is also used to obtain a preset sliding window size, and slide and intercept data from similar aggregated data blocks according to the preset sliding window size; respectively calculate the first variable-length feature values corresponding to the sliding intercepted data of different similar aggregated data blocks; compare the first variable-length feature values corresponding to different similar aggregated data blocks to obtain the first variable-length feature value comparison result; and deduplicate the similar aggregated data blocks according to the first variable-length feature value comparison result.
[0159] In one of the embodiments, the deduplication module 400 is also used to search for similar aggregated data block pairs having the same first variable-length feature value based on the comparison result of the first variable-length feature value to obtain candidate similar aggregated data block pairs; slide-cut data for the candidate similar aggregated data block pairs again based on a preset sliding window size; calculate the second variable-length feature value corresponding to the two sliding cut data; search for candidate similar aggregated data block pairs having the same second variable-length feature value to obtain target similar aggregated data block pairs; and deduplicate the target similar aggregated data block pairs.
[0160] In one of the embodiments, the deduplication module 400 is further used to perform a byte-by-byte comparison on two data blocks in the target similar aggregated data block pair; if the byte-by-byte comparison result indicates that the two data blocks are completely identical, the target similar aggregated data block pair is deduplicated.
[0161] In one embodiment, the deduplication module 400 is further configured to determine that there are no completely identical similar aggregated data blocks if no similar aggregated data block having the first variable-length feature value is found according to the comparison result of the first variable-length feature value.
[0162] In one embodiment, the compression module 500 is further used to store the combined compressed data blocks and the variable-length feature values in an associated manner.
[0163] Each module in the above data reduction device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.
[0164] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as shown in FIG11. The computer device includes a processor, a memory, and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store preset data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a data reduction method is implemented.
[0165] Those skilled in the art will understand that the structure shown in FIG. 11 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.
[0166] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above data reduction method when executing the computer program.
[0167] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned data reduction method is implemented.
[0168] In one embodiment, a computer program product is provided, comprising a computer program, which implements the above data reduction method when executed by a processor.
[0169] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.
[0170] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0171] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A data reduction method, characterized in that: The method comprises: Acquire data to be processed, and divide the data to be processed into multiple data blocks; Split multiple data blocks into smaller blocks of different lengths; Calculating similar feature values of small blocks between the multiple data blocks during the splitting process, and similarly aggregating the multiple data blocks based on the similar feature values; Using a sliding window to calculate variable-length feature values of similar aggregated data blocks, and deduplicating similar aggregated data blocks based on the variable-length feature values; Merge and compress similar aggregated data blocks after deduplication.
2. The method according to claim 1, characterized in that Splitting the multiple data blocks into small blocks of different lengths comprises: A greedy algorithm is used to split multiple data blocks into small blocks of different lengths.
3. The method according to claim 1, characterized in that The calculating of similar feature values of small blocks between the multiple data blocks in the splitting process, and similarly aggregating the multiple data blocks based on the similar feature values includes: Calculating similar feature values of small blocks between the multiple data blocks in the splitting process; Based on the similar eigenvalues, generating a similarity relationship matrix between different data blocks; The multiple data blocks are similarly aggregated according to the similarity relationship matrix.
4. The method according to claim 3, characterized in that The generating a similarity relationship matrix between different data blocks based on the similarity eigenvalues comprises: Obtain similar feature values of an original small block and a comparison small block, wherein the original small block and the comparison small block are small blocks in a first data block and a second data block respectively; the first data block and the second data block are any two different data blocks in the multiple data blocks; Arranging similar feature values of the original small block and the comparison small block into a feature matrix; Based on the feature matrix, a dimensionality reduction index algorithm is used to calculate a similarity relationship matrix between the first data block and the second data block.
5. The method according to claim 1, characterized in that The method of calculating the variable-length feature value of the similar aggregated data block by using a sliding window, and deduplicating the similar aggregated data block based on the variable-length feature value includes: Obtaining a preset sliding window size, and slidingly intercepting data from similar aggregated data blocks according to the preset sliding window size; Respectively calculating first variable-length feature values corresponding to sliding intercepted data of different similar aggregated data blocks; Comparing the first variable-length feature values corresponding to different similar aggregated data blocks to obtain a first variable-length feature value comparison result; Deduplication of similar aggregated data blocks is performed based on the comparison result of the first variable-length feature value.
6. The method according to claim 5, characterized in that The deduplication of similar aggregated data blocks according to the comparison result of the first variable-length feature value comprises: According to the comparison result of the first variable-length feature value, searching for similar aggregated data block pairs having the same first variable-length feature value to obtain candidate similar aggregated data block pairs; Sliding and intercepting data again for the candidate similar aggregated data block pair based on the preset sliding window size; Calculate the second variable-length eigenvalue corresponding to the two sliding intercepted data; Searching for candidate similar aggregated data block pairs having the same second variable-length feature value to obtain target similar aggregated data block pairs; Deduplication is performed on the target similar aggregated data block pairs.
7. The method according to claim 6, characterized in that The deduplication of the target similar aggregated data block pair comprises: Comparing two data blocks in the target similar aggregated data block pair byte by byte; If the byte-by-byte comparison result indicates that the two data blocks are completely identical, deduplication is performed on the target similar aggregated data block pair.
8. The method according to claim 6, characterized in that The method further comprises: If no similar aggregated data block having the first variable-length feature value is found according to the comparison result of the first variable-length feature value, it is determined that no completely identical similar aggregated data block exists.
9. The method according to claim 1, characterized in that: After merging and compressing the similar aggregated data blocks after deduplication, the method further includes: The combined compressed data blocks and the variable-length feature values are stored in association.
10. A data reduction device, characterized in that: The device comprises: A data partitioning module, used for acquiring data to be processed and partitioning the data to be processed into a plurality of data blocks; A data splitting module is used to split multiple data blocks into small blocks of different lengths; A similarity aggregation module, used to calculate similarity feature values of small blocks between the multiple data blocks in the splitting process, and similarly aggregate the multiple data blocks based on the similarity feature values; A deduplication module, used to calculate variable-length feature values of similar aggregated data blocks using a sliding window, and to deduplicate similar aggregated data blocks based on the variable-length feature values; The compression module is used to compress similar aggregated data blocks after deduplication.
11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.
12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.
13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.