An adaptive feature vector aggregation method and device based on similar deduplication history

Abstract

The present application relates to the technical field of data management, and provides a self-adaptive feature vector aggregation method and device based on similar re-deletion history, comprising: dividing data blocks into fixed-size minimum windows, generating an initial feature vector based on the minimum windows; maintaining a bitmap for a data block as a reference block, each bit of the bitmap corresponding to a minimum window, and updating the bitmap using window difference information generated in a similarity comparison process; when the number of similarity comparisons for the same reference block reaches a preset condition, determining an aggregation granularity according to a window region continuously marked as stable in the bitmap; aggregating the corresponding minimum windows into larger windows according to the determined aggregation granularity, regenerating the feature vector of the reference block, and updating the feature vector stored in the memory index. The present application can realize progressive self-adaptive optimization of the feature vector, significantly reduce memory occupation, and improve query efficiency under the premise of ensuring detection accuracy.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and in particular to an adaptive feature vector aggregation method and device based on similar deduplication history. Background Technology

[0002] With the rapid development of big data and artificial intelligence technologies, unstructured data is experiencing explosive growth, putting increasing pressure on storage system capacity. To address this challenge, modern storage systems widely employ data reduction technologies, primarily including data compression, deduplication, and similarity compression.

[0003] Data compression techniques, such as LZ4 and Zstandard, encode repeating patterns within a single data block, effectively reducing data size. However, they cannot identify duplicate data across blocks, and their compression ratio is limited by the redundancy within a single block. Deduplication techniques identify identical data blocks and store only one copy, proving highly effective for completely duplicated data. However, this requires the data blocks to be completely identical; if even a single byte differs, a new block must be created. In practical applications, large amounts of data are highly similar rather than identical, such as virtual machine images, log files, and database backups. Traditional deduplication techniques cannot effectively handle this type of data.

[0004] Similarity compression technology is an advanced data reduction scheme that has emerged in recent years. It can identify similar but not identical data blocks and store only the differences. This type of scheme uses a special hash function to map similar data to similar hash values, thereby achieving similarity detection. In the similarity detection process, data blocks are typically divided into fixed-size windows, and the hash value of each window is calculated to form a feature vector. This feature vector is then stored in an in-memory index for fast lookup of similar data blocks. The length of the feature vector directly affects the index storage overhead and query efficiency.

[0005] However, existing similarity compression techniques have the following drawbacks and shortcomings:

[0006] First, there's the issue of fixed window granularity. Fixed-size windows cannot adapt to the unevenness within the data. Using a smaller window, such as 32 bytes, results in higher detection accuracy, but the feature vectors are too long; for example, an 8KB data block requires 256 feature values, leading to high memory overhead. Using a larger window, such as 256 bytes, results in shorter feature vectors, but it may miss similarities within a small range, causing false negatives.

[0007] Second, the subjectivity of aggregation decisions. In existing solutions, window aggregation decisions are often based on heuristic rules or statistical predictions, lacking verification with actual data. Predictions may be incorrect, leading to inappropriate aggregation and affecting the similarity detection performance.

[0008] Third, the limitations of static aggregation. Once an aggregation strategy is determined, it becomes fixed and cannot be continuously optimized as more data is processed. Data characteristics may change over time, and static aggregation cannot adapt to these changes.

[0009] Fourth, there is a lack of feedback mechanisms. Existing solutions do not utilize historical results of similarity deduplication to optimize feature representations. Matching information generated from each similarity deduplication is discarded and not used to improve future detections.

[0010] Fifth, feature vectors are separated from validation. Feature vectors are used for fast filtering, but validation after filtering requires an additional mechanism. The design of feature vectors does not form a closed-loop optimization with the validation mechanism.

[0011] Chinese patent CN117369731A discloses a data reduction processing method. This scheme attempts to improve the deduplication value by piecing together data blocks and using a similar fingerprint database. However, its redundant processing flow, static aggregation strategy, and lack of historical feedback mechanism limit its performance and adaptability in actual storage systems.

[0012] Therefore, how to provide a method that can use the actual comparison results generated during the similarity deduplication process to guide feature vector optimization, so that the feature vector becomes more compact adaptively as the system runs, thereby reducing memory usage and improving query efficiency while ensuring detection accuracy, has become an urgent technical problem to be solved. Summary of the Invention

[0013] In view of this, in order to overcome the shortcomings of the prior art, the present invention aims to provide an adaptive feature vector aggregation method and device based on similar deduplication history.

[0014] According to a first aspect of the present invention, an adaptive feature vector aggregation method based on similar deduplication history is provided, the method comprising:

[0015] Step S1: Divide the data block into a minimum window of fixed size, and generate an initial feature vector based on the minimum window;

[0016] Step S2: Maintain a bitmap for the data block that serves as the reference block. Each bit of the bitmap corresponds to a minimum window, and update the bitmap using the window difference information generated during the similarity comparison process.

[0017] Step S3: When the number of similarity comparisons for the same reference block reaches the preset condition, the aggregation granularity is determined based on the continuously marked stable window regions in the bitmap;

[0018] Step S4: Aggregate the corresponding minimum window into a larger window according to the determined aggregation granularity, regenerate the feature vector of the reference block, and update the feature vector stored in the memory index.

[0019] Optionally, the adaptive feature vector aggregation method based on similar deduplication history of the present invention includes step S1 as follows:

[0020] The data block is divided into several consecutive minimum windows according to the preset minimum window size, and each minimum window has a fixed byte length;

[0021] Calculate the feature value for each minimum window, arrange the feature values ​​of all minimum windows in window order to form an initial feature vector and store it in the memory index.

[0022] Optionally, in the adaptive feature vector aggregation method based on similarity deduplication history of the present invention, in step S2, the bitmap is initially set to all 1s, and the bitmap is updated using the window difference information generated during the similarity comparison process in the following manner:

[0023] When a new data block is compared with a reference block, the original data content of the two blocks is compared window by window.

[0024] For windows with different content, set the corresponding bit in the bitmap to 0;

[0025] For windows with identical content, keep the original values ​​of the corresponding bits unchanged.

[0026] Optionally, in the adaptive feature vector aggregation method based on similar deduplication history of the present invention, in step S3, the preset condition is that the number of similar data for the same reference block reaches a preset threshold. This threshold is preset according to the system scale, memory resources or performance requirements. When the cumulative number of similar data reaches the threshold, the aggregation analysis of the reference block is triggered.

[0027] Optionally, in the adaptive feature vector aggregation method based on similar deduplication history of the present invention, in step S3, the aggregation granularity is determined according to the continuous window regions marked as stable in the bitmap as follows: scan the window intervals in the bitmap that are continuously marked as 1, take each continuous interval as an aggregateable region, count the number of continuous windows in the interval, and use the number as the basic aggregation granularity of the region. The final value of the aggregation granularity is in the form of a power of 2, and the maximum power of 2 value not exceeding the length of the continuous length of the aggregateable region is selected as the actual aggregation granularity.

[0028] Optionally, in the adaptive feature vector aggregation method based on similar deduplication history of the present invention, in step S4, the corresponding minimum window is aggregated into a larger window according to the determined aggregation granularity by adopting a unified window aggregation mode: traversing all aggregateable regions, obtaining the smallest aggregation granularity among them, using the smallest aggregation granularity as the unified window size of the entire data block, re-dividing the entire data block according to the unified window size, and generating a fixed-length feature vector.

[0029] Optionally, in the adaptive feature vector aggregation method based on similar deduplication history of the present invention, in step S4, the self-describing window aggregation mode is used to aggregate the corresponding minimum window into a larger window according to the determined aggregation granularity in the following manner: for a window interval with consecutive 1s in the bitmap, multiple consecutive minimum windows in the interval are aggregated into a larger window according to the determined aggregation granularity, and the size of the aggregated window is equal to the aggregation granularity multiplied by the size of the minimum window; for a window position with 0s in the bitmap, the minimum window remains unchanged; a variable-length feature vector is generated, wherein each feature element carries the corresponding window granularity information.

[0030] Optionally, in the adaptive feature vector aggregation method based on similar deduplication history of the present invention, in step S4, when the length of the interval with consecutive 1s in the bitmap is inconsistent with the selected aggregation granularity, a segmented aggregation method is adopted: the aggregateable region is divided into multiple complete segments with a length equal to the aggregation granularity for aggregation, and the remaining windows with less than one aggregation granularity keep the minimum window unchanged, or are merged with adjacent regions.

[0031] Optionally, in the adaptive feature vector aggregation method based on similar deduplication history of the present invention, in step S4, after regenerating the feature vector of the reference block and updating the feature vector stored in the memory index, subsequent similarity searches for the reference block are all performed using the updated feature vector. When the preset conditions are met again, the aggregation analysis and feature vector update process are repeated.

[0032] According to a second aspect of the present invention, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the first aspect of the present invention.

[0033] This invention relates to an adaptive feature vector aggregation method and device based on similar deduplication history, which has the following beneficial technical effects:

[0034] I. Implementing Incremental Adaptive Optimization of Feature Vectors

[0035] Similarity detection and difference storage are performed at a fixed minimum window granularity. A bitmap is maintained for each baseline block, and the bitmap is updated using window difference information naturally generated during similarity comparison. As the system runs longer and similar data accumulates, the feature vectors are gradually optimized and continuously compacted, achieving adaptive evolution of feature representation.

[0036] II. Significantly reduce memory usage

[0037] The initial feature vectors are generated with a minimum window granularity and are relatively long. After multiple aggregation optimizations, the feature vector length is significantly reduced.

[0038] III. Maintaining the same similarity detection accuracy

[0039] The aggregation decision in this invention is based on a true history of similarity comparisons, rather than statistical predictions or heuristic rules. Regions in the bitmap that are 1 have high stability and will not affect the accuracy of similarity detection after aggregation. Regions in the bitmap that are 0 ensure that the ability to detect local changes is not affected. Detection accuracy is fully preserved while reducing memory usage.

[0040] IV. Improve query efficiency

[0041] Shortening the feature vector length reduces the time required for comparison operations in the memory index, thus lowering the latency of similarity lookup. Furthermore, in self-describing window aggregation mode, each feature element carries window granularity information, further optimizing similarity calculation efficiency.

[0042] V. Zero additional expenses

[0043] The bitmap is generated directly from the similarity comparison process, requiring no additional data reading or computation. During similarity comparison, the original data content is compared window by window, and the differences are synchronously used to update the bitmap, without introducing additional storage access or computational overhead. Aggregate analysis and feature vector reconstruction are executed asynchronously in the background, without blocking frontend write responses.

[0044] VI. Reliable decision-making with no risk of data errors.

[0045] Aggregation decisions are made based on actual comparison results. Regions marked as 1 in the bitmap have been verified by multiple sets of similar data, ensuring the reliability of the decision-making basis and eliminating the risk of misjudgment. Even if some window regions do not temporarily meet the aggregation conditions, the smallest window remains unchanged, preventing data errors or system anomalies.

[0046] VII. Flexible adaptation to different application scenarios

[0047] The unified window aggregation mode is suitable for scenarios with relatively simple data patterns and extreme sensitivity to memory usage, and can compress the feature vector length to the greatest extent. The self-describing window aggregation mode is suitable for scenarios with complex data patterns and the need to balance detection accuracy, and can minimize memory usage while maintaining accuracy.

[0048] 8. Continuous optimization, performance improves over time.

[0049] The optimization mechanism of this invention is deeply integrated with the system operation. As similar data accumulates, the feature vector is continuously optimized. The system performance will not degrade due to the increase of running time, but will instead show a benign development trend of gradual performance improvement. Attached Figure Description

[0050] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart illustrating the steps of an adaptive feature vector aggregation method based on similar deduplication history according to an exemplary embodiment 1 of the present invention.

[0052] Figure 2 This is a schematic diagram illustrating the bitmap initialization and update process according to the adaptive feature vector aggregation method based on similar deduplication history in Exemplary Embodiment 1 of the present invention.

[0053] Figure 3 This is a flowchart illustrating the direct data comparison process of the adaptive feature vector aggregation method based on similar deduplication history according to Exemplary Embodiment 1 of the present invention.

[0054] Figure 4 This is a flowchart illustrating the threshold triggering process of the adaptive feature vector aggregation method based on similar deduplication history according to Exemplary Embodiment 1 of the present invention.

[0055] Figure 5 This is a flowchart illustrating the process of determining the aggregation granularity of the adaptive feature vector aggregation method based on similar deduplication history according to Exemplary Embodiment 1 of the present invention.

[0056] Figure 6 This is a flowchart illustrating the unified window aggregation mode of the adaptive feature vector aggregation method based on similar deduplication history according to Exemplary Embodiment 1 of the present invention.

[0057] Figure 7 This is a flowchart illustrating the self-describing window aggregation mode of the adaptive feature vector aggregation method based on similar deduplication history according to Exemplary Embodiment 1 of the present invention.

[0058] Figure 8 This is a flowchart illustrating the segmented aggregation process of the adaptive feature vector aggregation method based on similar deduplication history according to Exemplary Embodiment 1 of the present invention.

[0059] Figure 9 This is a flowchart illustrating the progressive continuous optimization process of the adaptive feature vector aggregation method based on similar deduplication history according to Exemplary Embodiment 1 of the present invention.

[0060] Figure 10 This is a schematic diagram of the asynchronous embedding execution process of the adaptive feature vector aggregation method based on similar deduplication history according to an exemplary embodiment 1 of the present invention;

[0061] Figure 11 This is a schematic diagram of the structure of the device provided by the present invention. Detailed Implementation

[0062] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0063] It should be noted that, in the absence of conflict, the following embodiments and features can be combined with each other; and, based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0064] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0065] Example 1

[0066] Exemplary embodiment 1 of the present invention provides an adaptive feature vector aggregation method based on similar deduplication history. Figure 1 The flowchart below illustrates the steps of an adaptive feature vector aggregation method based on similar deduplication history according to Exemplary Embodiment 1 of the present invention. Figure 1 As shown, the method of this embodiment is implemented according to the following steps:

[0067] Step S1: Divide the data block into a minimum window of fixed size, and generate an initial feature vector based on the minimum window.

[0068] In this embodiment, the data block is divided into several consecutive minimum windows according to a preset minimum window size, and each minimum window has a fixed byte length; the feature value of each minimum window is calculated, and the feature values ​​of all minimum windows are arranged in window order to form an initial feature vector and stored in the memory index.

[0069] Step S2: Maintain a bitmap for the data block that serves as the reference block. Each bit in the bitmap corresponds to a minimum window, and update the bitmap using the window difference information generated during the similarity comparison process.

[0070] Figure 2This is a schematic diagram illustrating the bitmap initialization and update process according to the adaptive feature vector aggregation method based on similar deduplication history in Exemplary Embodiment 1 of the present invention. Figure 2 As shown, in this embodiment, the bitmap is initially set to all 1s, indicating that all windows are in a stable state. The bitmap is updated using the window difference information generated during the similarity comparison process in the following manner: when a new data block is compared with a reference block, the original data content of the two is compared window by window; for windows with different content, the corresponding bit in the bitmap is set to 0; for windows with the same content, the original value of the corresponding bit remains unchanged.

[0071] Figure 3 This is a schematic diagram illustrating the process of direct data comparison using the adaptive feature vector aggregation method based on similar deduplication history according to Exemplary Embodiment 1 of the present invention. Figure 3 As shown, in this embodiment, the similarity comparison directly uses the original data of the data block itself to perform window-by-window content comparison, without relying on the pre-calculation of the window hash value. The window difference information is obtained synchronously during the similarity comparison process without additional reading or calculation overhead.

[0072] Step S3: When the number of similar comparisons for the same reference block reaches the preset condition, the aggregation granularity is determined based on the window regions continuously marked as stable in the bitmap.

[0073] Figure 4 This is a schematic diagram illustrating the threshold-triggered process of the adaptive feature vector aggregation method based on similar deduplication history according to Exemplary Embodiment 1 of the present invention. Figure 4 As shown, the preset condition in this embodiment is that the number of similar data for the same benchmark block reaches a preset threshold. This threshold is preset according to the system size, memory resources or performance requirements. When the cumulative number of similar data reaches this threshold, the aggregation analysis of the benchmark block is triggered.

[0074] Figure 5 This is a schematic diagram illustrating the process of determining the aggregation granularity of the adaptive feature vector aggregation method based on similar deduplication history according to Exemplary Embodiment 1 of the present invention. Figure 5 As shown, in this embodiment, the aggregation granularity is determined based on the continuous window regions marked as stable in the bitmap in the following manner: scan the window intervals in the bitmap that are continuously marked as 1, take each continuous interval as an aggregateable region, count the number of continuous windows in the interval, and use the number as the basic aggregation granularity of the region. The final value of the aggregation granularity is in the form of a power of 2. The maximum power of 2 value not exceeding the length of the continuous length of the aggregateable region is selected as the actual aggregation granularity.

[0075] Step S4: Aggregate the corresponding minimum window into a larger window according to the determined aggregation granularity, regenerate the feature vector of the reference block, and update the feature vector stored in the memory index.

[0076] Figure 6 This is a flowchart illustrating the unified window aggregation mode of the adaptive feature vector aggregation method based on similar deduplication history according to Exemplary Embodiment 1 of the present invention. Figure 6 As shown, in this embodiment, the unified window aggregation mode is used to aggregate the corresponding smallest window into a larger window according to the determined aggregation granularity in the following way: traverse all aggregateable regions, obtain the smallest aggregation granularity among them, use the smallest aggregation granularity as the unified window size of the entire data block, redivide the entire data block according to the unified window size, and generate a fixed-length feature vector.

[0077] Figure 7 This is a flowchart illustrating the self-describing window aggregation mode of the adaptive feature vector aggregation method based on similar deduplication history according to Exemplary Embodiment 1 of the present invention, as follows: Figure 7 As shown, in this embodiment, a self-describing window aggregation mode can also be used to aggregate the corresponding minimum window into a larger window according to the determined aggregation granularity in the following way: For a window interval with consecutive 1s in the bitmap, multiple consecutive minimum windows in the interval are aggregated into a larger window according to the determined aggregation granularity, and the size of the aggregated window is equal to the aggregation granularity multiplied by the minimum window size; For a window position with 0s in the bitmap, the minimum window remains unchanged; A variable-length feature vector is generated, where each feature element carries the corresponding window granularity information.

[0078] Figure 8 This is a schematic diagram illustrating the process of segmented aggregation using the adaptive feature vector aggregation method based on similar deduplication history according to Exemplary Embodiment 1 of the present invention; as follows: Figure 8 As shown, in the self-describing window aggregation mode, when the length of the interval with consecutive 1s in the bitmap is inconsistent with the selected aggregation granularity, a segmented aggregation method is adopted: the aggregateable area is divided into multiple complete segments with a length equal to the aggregation granularity for aggregation, and the remaining windows with less than one aggregation granularity keep the minimum window unchanged, or are merged with adjacent areas.

[0079] Figure 9 This is a flowchart illustrating the progressive continuous optimization process of the adaptive feature vector aggregation method based on similar deduplication history according to Exemplary Embodiment 1 of the present invention, as follows: Figure 9 As shown, in this embodiment, after regenerating the feature vector of the benchmark block and updating the feature vector stored in the memory index, subsequent similarity searches for the benchmark block are all performed using the updated feature vector. When the preset conditions are met again, the aggregation analysis and feature vector update process are repeated.

[0080] Figure 10This is a schematic diagram of the asynchronous embedding execution process of the adaptive feature vector aggregation method based on similar deduplication history according to Exemplary Embodiment 1 of the present invention, as shown below. Figure 10 As shown, the method in this embodiment is embedded in the write process of the similar deduplication system. The maintenance and updating of the bitmap are completed synchronously with the difference storage in the similarity comparison process. The aggregation analysis and feature vector update are executed asynchronously in the background without blocking the front-end write response.

[0081] Example 2

[0082] Exemplary embodiment 2 of the present invention provides an adaptive feature vector aggregation method based on similarity deduplication history. In this embodiment, the data block is divided into the smallest window for similarity detection and difference storage. On this basis, a bitmap is maintained for each reference block. By recording the difference history between similar data and the reference block, the aggregation of the smallest window is guided, so that the feature vector is continuously optimized as the similarity history accumulates.

[0083] Step 1: Minimum Window Partition and Initial Feature Vector Generation

[0084] The data block is divided into fixed-size minimum windows. Taking an 8KB data block as an example, if the minimum window is 32 bytes, then it is divided into 256 windows. An initial feature vector is generated based on these minimum windows and stored in a memory index for similarity lookup.

[0085] Step 2: Maintain a bitmap for each reference block.

[0086] For each data block used as the reference block, a Bitmap is maintained. The length of the Bitmap is equal to the minimum number of windows in that data block, with each bit corresponding to one minimum window. The Bitmap is initially set to all 1s. This initial state means that when the reference block is compared with itself, all windows are the same.

[0087] Step 3: Similarity Comparison and Bitmap Update

[0088] When a new data block is compared to a baseline block, the contents of the two data blocks are compared window by window. The comparison uses the raw data of the data blocks themselves, without the need to pre-calculate hash values.

[0089] For each window: if the content of the window is the same as the base block, the value at that position in the Bitmap remains unchanged. If the content of the window is different from the base block, the value at that position in the Bitmap is set to 0.

[0090] In this embodiment, a position of 1 in the Bitmap indicates that "so far, this window is identical to the baseline block among all similar data being compared." Once any similar data is found to differ within this window, it means that this window is not absolutely stable, and therefore it is marked as 0.

[0091] Step 4: Trigger aggregation analysis after the threshold is reached.

[0092] Aggregation analysis is triggered when the number of similar data points compared to a baseline block reaches a preset threshold. The threshold can be set according to system size and requirements, for example, to 5 similar data points. The aggregation analysis targets consecutive regions of 1 in the Bitmap. These regions represent areas where, among all processed similar data, these consecutive windows remain identical to the baseline block, exhibiting high stability and allowing for safe aggregation. Positions of 0 in the Bitmap indicate past differences, requiring smaller windows to ensure detection accuracy.

[0093] Step 5: Generate new feature vectors using aggregation patterns.

[0094] Based on the aggregation analysis results, the feature vector can be regenerated using one of the following two modes:

[0095] Mode 1: Unified Window Aggregation

[0096] If a feature vector with a uniform window size is chosen, the smallest aggregation length among all aggregable regions is used as the uniform window size. For example, if a Bitmap has a region with 16 consecutive windows of 1, but another region with only 2 consecutive windows of 1, then the entire feature vector is regenerated using the aggregation length of 2 windows (i.e., 64 bytes). This mode is suitable for situations where the pattern within a data block is relatively simple.

[0097] Mode 2: Self-describing window aggregation

[0098] If a self-describing window mode is used, regions of consecutive 1s in the Bitmap are represented by large windows, while regions of 0s are represented by small windows. Specifically:

[0099] For regions in a Bitmap that are consecutively 1, these consecutive windows are aggregated into a larger window, the size of which is equal to the consecutive length multiplied by the smallest window size.

[0100] For positions in the Bitmap that are 0, each window is reserved separately to maintain the minimum window size.

[0101] The feature vectors generated in this way are variable-length, with each element carrying its own granular information, which can more accurately reflect the actual characteristics of the data.

[0102] Step 6: Update memory index

[0103] Replace the original feature vector in the memory index with the newly generated feature vector. Subsequent similarity searches will use the updated feature vector.

[0104] Example 3

[0105] Exemplary Example 3 of the present invention provides an adaptive feature vector aggregation method based on similar deduplication history. The method of this embodiment will be further described in detail below in a specific scenario.

[0106] In this scenario, taking an 8KB data block as an example, the smallest window is 32 bytes, and there are a total of 256 windows.

[0107] Initial state: Bitmap is all 1s, totaling 256 bits.

[0108] After comparing the first similar data B1 with the baseline block, it was found that the contents of windows 2, 5, and 10 were different. The Bitmap was updated as follows: the positions of windows 2, 5, and 10 became 0, while the remaining positions remained 1.

[0109] After comparing the second similar data B2 with the baseline block, it was found that the contents of windows 1, 3, and 8 were different. The Bitmap was updated as follows: the positions of windows 1, 3, and 8 were changed to 0, while the rest remained in their original state.

[0110] After comparing the third similar data B3 with the baseline block, it was found that the contents of windows 0 and 4 were different. The Bitmap was updated so that the positions of windows 0 and 4 became 0.

[0111] After accumulating the 5th similar data point, the Bitmap may show several consecutive regions of 1. For example, 16 consecutive positions in window 20-35 are 1, 4 consecutive positions in window 100-103 are 1, and the rest are 0.

[0112] This triggers aggregation analysis. If the self-describing window mode is used, then:

[0113] The 16 consecutive windows, 20-35, are aggregated into a single large window of 512 bytes.

[0114] The four consecutive windows, 100-103, are aggregated into a medium-sized window of 128 bytes.

[0115] Other locations should maintain a 32-byte small window.

[0116] The length of the regenerated feature vector was reduced from 256 elements to approximately 200 elements, resulting in a corresponding reduction in memory usage. As more similar data is added, the region of consecutive 1s in the Bitmap may further expand, requiring continued optimization of the feature vector.

[0117] Example 4

[0118] Exemplary Example 4 of the present invention provides an adaptive feature vector aggregation method based on similar deduplication history. The method of this embodiment will be further described in detail below in a specific scenario.

[0119] In this scenario, when a user writes a new data block X, the system needs to determine whether there is a similar data block to X in order to perform similarity compression.

[0120] The environment is set as follows:

[0121] Data block size: 8KB.

[0122] Minimum window size: 32 bytes, total of 256 windows.

[0123] Memory index: Stores the feature vectors of all data blocks.

[0124] Bitmap threshold: set to 5, meaning that aggregation analysis is triggered after accumulating 5 similar data points.

[0125] In this scenario, the method of the present invention is implemented according to the following steps:

[0126] Step 1: Receive write request

[0127] A write request for data block X has been received. Data block X is first written to the front-end high-speed pool, and a write success confirmation is immediately returned to ensure that write performance is not affected.

[0128] Step 2: Generate feature vectors

[0129] Divide the data block X into 256 32-byte windows, and calculate the feature value window by window (using simple hashing or directly using data features) to generate an initial feature vector FV_X. At this point, FV_X has a length of 256 elements.

[0130] Use FV_X to find candidate blocks in the memory index whose similarity exceeds a threshold (e.g., 70%).

[0131] Step 3: Find similar blocks

[0132] Suppose there exists a baseline block A in the memory index, whose feature vector FV_A has a similarity of 85% with FV_X, exceeding the threshold. Baseline block A is then selected as a candidate block.

[0133] Step 4: Precise Comparison and Bitmap Update

[0134] Read the raw data of the reference block A (or retrieve it from the cache) and perform a window-by-window exact comparison with the data block X, as shown in Table 1.

[0135] Table 1

[0136]

[0137] Based on the comparison results in Table 1, perform the following two operations:

[0138] Operation 1: Differential Storage

[0139] The same windows (250 in total) reference data from base block A;

[0140] Different windows (6 in total: windows 102, 103, and 151) are stored separately as difference data;

[0141] Record a reference relationship: data block X is based on base block A, and the difference window is 102, 103, 151.

[0142] Operation 2: Update the Bitmap of base block A

[0143] Base block A maintains a Bitmap that records the difference history of all similar data compared to it. After the initial state, four similar data have been compared with A, and the Bitmap is currently as follows:

[0144] Positions 0-100: 11111... (consecutive positions are 1)

[0145] Position 101: 1

[0146] Position 102: 0 (Previous data differs here)

[0147] Position 103: 1 (Previously there was no data that differed here)

[0148] Positions 104-150: 11111... (consecutive 1s)

[0149] Position 151: 1 (No data previously available, but different here)

[0150] Positions 152-255: 11111... (consecutive 1s)

[0151] Update the bitmap using the comparison result of data block X:

[0152] Window 102: It was already 0 before, so it remains unchanged;

[0153] Window 103: Previously it was 1, but this comparison found a difference, so it was set to 0;

[0154] Window 151: It was previously 1, but this comparison found a difference, so it was set to 0;

[0155] Other windows: Remain unchanged from their previous state.

[0156] The updated bitmap is as follows:

[0157] Positions 0-100: 11111... (consecutive positions are 1)

[0158] Position 101: 1

[0159] Position 102: 0

[0160] Position 103: 0 (Newly set to 0)

[0161] Positions 104-150: 11111... (consecutive 1s)

[0162] Position 151: 0 (Newly set to 0)

[0163] Positions 152-255: 11111... (consecutive 1s)

[0164] Step 5: Check aggregation trigger conditions

[0165] The similarity data count for baseline block A increased from 4 to 5, reaching the preset threshold. This triggers an aggregation analysis of baseline block A.

[0166] Step 6: Aggregate Analysis

[0167] Analyze the current Bitmap of base block A:

[0168] Regions with consecutive values ​​of 1:

[0169] Region 1: Positions 0-100, length 101 windows (not powers of 2, but the largest power of 2 needs to be taken).

[0170] Region 2: Location 101, Length 1 (cannot be aggregated)

[0171] Region 3: Position 104-150, length 47 windows (not a power of 2)

[0172] Region 4: Positions 152-255, length 104 windows (not a power of 2)

[0173] Positions where 0 is:

[0174] Positions 102, 103, and 151 (differences have occurred)

[0175] Since the aggregation length must be a power of 2 (1, 2, 4, 8, 16, 32, 64, 128, 256), we need to extract the largest power of 2 length from the region of consecutive 1s:

[0176] Region 1 (positions 0-100): The largest power of 2 is 64 (positions 0-63). These 64 windows are the same in all 5 similar data, and are aggregated into a window of 64 × 32 = 2048 bytes.

[0177] The remaining part of region 1 (positions 64-100): Although it is also a continuous string of 1s, the length 37 is not a power of 2 and may be covered by subsequent data, so it will remain a small window for now;

[0178] Region 3 (positions 104-150): The largest power of 2 is 32 (positions 104-135), and these 32 windows can be aggregated into a 1024-byte window;

[0179] The remaining part of region 3 (positions 136-150): length 15, not a power of 2, maintain a small window;

[0180] Region 4 (positions 152-255): The largest powers of 2 are 64 (positions 152-215) and 32 (positions 216-247), which are aggregated in segments.

[0181] Step 7: Regenerate feature vectors

[0182] Based on the aggregation analysis results, the feature vector of the baseline block A is regenerated using a self-describing window mode:

[0183] Positions 0-63: Aggregated into a single 2048-byte window (granularity g=6)

[0184] Positions 64-100: Maintain 37 windows of 32 bytes each.

[0185] Position 101: Maintain a 32-byte window

[0186] Positions 102-103: Maintain two 32-byte windows (difference regions)

[0187] Positions 104-135: Aggregated into a single 1024-byte window (granularity g=5)

[0188] Positions 136-150: Maintain 15 windows of 32 bytes each

[0189] Position 151: Maintain a 32-byte window (difference region)

[0190] Positions 152-215: Aggregate into a single 2048-byte window (granularity g=6)

[0191] Positions 216-247: Aggregate into a single 1024-byte window (granularity g=5)

[0192] Positions 248-255: Maintain 8 32-byte windows

[0193] The number of elements in the new feature vector is reduced from the original 256 to approximately 37 + 1 + 2 + 15 + 1 + 8 + 3 aggregation windows = approximately 67 elements, resulting in a memory reduction of approximately 74%.

[0194] Step 8: Update memory index

[0195] The newly generated feature vector replaces the original feature vector of the reference block A in the memory index. All subsequent similarity searches for the reference block A will use this more compact new feature vector.

[0196] In this scenario, when data block X needs to be read, the reference relationship of data block X is found based on the stored metadata: based on the base block A, the difference windows are 102, 103, and 151; the complete data of the base block A is read; for windows 102, 103, and 151, the stored difference data is used to replace them; other windows directly use the data of the base block A; and the complete data block X is assembled and returned.

[0197] It should be noted that this scenario uses a self-describing window mode for aggregation. If it is found that in the Bitmap of the base block A, the minimum aggregation length of all consecutive regions with a value of 1 is 32, and most regions can be aggregated using 32, a uniform window mode can also be selected.

[0198] By uniformly using 32 windows (1024 bytes) as the new window size, the entire data block is re-divided into 8 windows of 1024 bytes each, generating a feature vector with only 8 elements, reducing memory usage by 97%. This approach is suitable for scenarios that are extremely sensitive to memory.

[0199] The adaptive feature vector aggregation method based on similar deduplication history in this invention has the following beneficial technical effects:

[0200] I. Implementing Incremental Adaptive Optimization of Feature Vectors

[0201] Similarity detection and difference storage are performed at a fixed minimum window granularity. A bitmap is maintained for each baseline block, and the bitmap is updated using window difference information naturally generated during similarity comparison. When a certain amount of similar data accumulates, highly stable window regions are identified by analyzing regions with consecutive 1s in the bitmap. These regions are then aggregated into larger windows, and feature vectors are regenerated. As the system runs longer and similar data continues to accumulate, the feature vectors are gradually optimized and continuously compacted, achieving adaptive evolution of feature representation.

[0202] II. Significantly reduce memory usage

[0203] The initial feature vector is generated with a minimum window granularity and is relatively long. After multiple aggregation optimizations, the feature vector length is significantly reduced. Taking an 8KB data block and a 32-byte minimum window as an example, the initial feature vector contains 256 elements, which can be reduced to approximately 150 to 200 elements after aggregation optimization. For a large-scale storage system with 100 million data blocks, this can save more than 10GB of memory, significantly reducing the storage overhead of memory indexes.

[0204] III. Maintaining the same similarity detection accuracy

[0205] The aggregation decision in this invention is based on real-world similarity comparison history, rather than statistical predictions or heuristic rules. Regions marked as 1 in the bitmap have been verified using multiple similar data sets, proving that their content is identical to the baseline block across all compared data, demonstrating high stability and ensuring that aggregation does not affect the accuracy of similarity detection. Regions marked as 0 in the bitmap indicate that the window has at least one difference, requiring the maintenance of the smallest window granularity to ensure that the detection capability of local changes is not affected. Therefore, while reducing memory usage, detection accuracy is fully preserved.

[0206] IV. Improve query efficiency

[0207] Shortening the feature vector length reduces the time required for comparison operations in the memory index, thus lowering the latency of similarity lookup. Furthermore, in self-describing window aggregation mode, each feature element carries window granularity information, further optimizing similarity calculation efficiency.

[0208] V. Zero additional expenses

[0209] The bitmap is generated directly from the similarity comparison process, requiring no additional data reading or computation. During similarity comparison, the original data content is compared window by window, and the differences are synchronously used to update the bitmap, without introducing additional storage access or computational overhead. Aggregate analysis and feature vector reconstruction are executed asynchronously in the background, without blocking frontend write responses.

[0210] VI. Reliable decision-making with no risk of data errors.

[0211] Existing aggregation decisions based on statistical predictions carry the risk of inaccurate predictions, potentially leading to improper aggregation and affecting detection results. This invention makes aggregation decisions based on actual comparison results. Regions marked as 1 in the bitmap have been verified by multiple sets of similar data, ensuring the decision-making basis is authentic and reliable, eliminating the risk of misjudgment. Even if some window regions do not temporarily meet the aggregation conditions, the smallest window remains unchanged, preventing data errors or system anomalies.

[0212] VII. Flexible adaptation to different application scenarios

[0213] Two aggregation modes are provided: the unified window aggregation mode is suitable for scenarios with relatively simple data patterns and extreme sensitivity to memory usage, and can compress feature vector length to the greatest extent; the self-describing window aggregation mode is suitable for scenarios with complex data patterns and the need to balance detection accuracy, minimizing memory usage while maintaining accuracy. Users can flexibly choose according to data type and system requirements.

[0214] 8. Continuous optimization, performance improves over time.

[0215] The optimization mechanism of this invention is deeply integrated with the system operation. As similar data accumulates, the feature vector is continuously optimized. The system performance will not degrade due to the increase of running time, but will instead show a benign development trend of gradual performance improvement.

[0216] like Figure 11 As shown, the present invention also provides a device including a processor 310, a communication interface 320, a memory 330 for storing a processor-executable computer program, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 implements the aforementioned adaptive feature vector aggregation method based on similarity deduplication history by running the executable computer program.

[0217] The computer program in memory 330, when implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0218] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected based on actual needs to achieve the purpose of this embodiment. Those skilled in the art can understand and implement this without any creative effort.

[0219] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0220] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An adaptive feature vector aggregation method based on similar deduplication history, characterized in that, The method includes: Step S1: Divide the data block into a minimum window of fixed size, and generate an initial feature vector based on the minimum window; Step S2: Maintain a bitmap for the data block that serves as the reference block. Each bit of the bitmap corresponds to a minimum window, and update the bitmap using the window difference information generated during the similarity comparison process. Step S3: When the number of similarity comparisons for the same reference block reaches the preset condition, the aggregation granularity is determined based on the continuously marked stable window regions in the bitmap; Step S4: Aggregate the corresponding minimum window into a larger window according to the determined aggregation granularity, regenerate the feature vector of the reference block, and update the feature vector stored in the memory index.

2. The adaptive feature vector aggregation method based on similar deduplication history according to claim 1, characterized in that, Step S1 includes: The data block is divided into several consecutive minimum windows according to the preset minimum window size, and each minimum window has a fixed byte length; Calculate the feature value for each minimum window, arrange the feature values ​​of all minimum windows in window order to form an initial feature vector and store it in the memory index.

3. The adaptive feature vector aggregation method based on similar deduplication history according to claim 1, characterized in that, In step S2, the bitmap is initially set to all 1s, and then updated using the window difference information generated during the similarity comparison process: When a new data block is compared with a reference block, the original data content of the two blocks is compared window by window. For windows with different content, set the corresponding bit in the bitmap to 0; For windows with identical content, keep the original values ​​of the corresponding bits unchanged.

4. The adaptive feature vector aggregation method based on similar deduplication history according to claim 1, characterized in that, In step S3, the preset condition is that the number of similar data for the same benchmark block reaches a preset threshold. This threshold is preset according to the system size, memory resources or performance requirements. When the cumulative number of similar data reaches this threshold, the aggregation analysis of the benchmark block is triggered.

5. The adaptive feature vector aggregation method based on similar deduplication history according to claim 1, characterized in that, In step S3, the aggregation granularity is determined based on the continuous window regions marked as stable in the bitmap as follows: scan the window intervals in the bitmap that are continuously marked as 1, take each continuous interval as an aggregateable region, count the number of continuous windows in the interval, and use the number as the basic aggregation granularity of the region. The final value of the aggregation granularity is in the form of a power of 2. The maximum power of 2 value not exceeding the length of the continuous length of the aggregateable region is selected as the actual aggregation granularity.

6. The adaptive feature vector aggregation method based on similar deduplication history according to claim 1, characterized in that, In step S4, the corresponding smallest window is aggregated into a larger window according to the determined aggregation granularity using the unified window aggregation mode as follows: traverse all aggregateable regions, obtain the smallest aggregation granularity, use the smallest aggregation granularity as the unified window size of the entire data block, redivide the entire data block according to the unified window size, and generate a fixed-length feature vector.

7. The adaptive feature vector aggregation method based on similar deduplication history according to claim 1, characterized in that, In step S4, the self-describing window aggregation mode is used to aggregate the corresponding minimum windows into larger windows according to the determined aggregation granularity in the following manner: For a window interval with consecutive 1s in the bitmap, multiple consecutive minimum windows in the interval are aggregated into a larger window according to the determined aggregation granularity, and the size of the aggregated window is equal to the aggregation granularity multiplied by the minimum window size; For a window position with 0s in the bitmap, the minimum window remains unchanged; A variable-length feature vector is generated, where each feature element carries the corresponding window granularity information.

8. The adaptive feature vector aggregation method based on similar deduplication history according to claim 7, characterized in that, In step S4, in the self-describing window aggregation mode, when the length of the interval with consecutive 1s in the bitmap is inconsistent with the selected aggregation granularity, a segmented aggregation method is adopted: the aggregateable area is divided into multiple complete segments with a length equal to the aggregation granularity for aggregation, and the remaining windows with less than one aggregation granularity keep the minimum window unchanged, or are merged with adjacent areas.

9. The adaptive feature vector aggregation method based on similar deduplication history according to claim 1, characterized in that, In step S4, the feature vector of the benchmark block is regenerated and the feature vector stored in the memory index is updated. Subsequent similarity searches for the benchmark block are performed using the updated feature vector. When the preset conditions are met again, the aggregation analysis and feature vector update process are repeated.

10. A computer device, characterized in that, The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method according to any one of claims 1-9.

Images