Hybrid granularity mapping system based on data compression

Abstract

The invention relates to the technical field of hybrid granularity mapping, in particular to a hybrid granularity mapping system based on data compression. Comprising a data preprocessing module, a compressed sensing block division module, a dynamic compression execution module, a mixed granularity mapping module, an entropy balance metadata module, a dynamic wear distribution module, a compressed sensing recovery triggering module and a self-adaptive strategy optimization module. The data preprocessing module is used for data stream fragmentation and hot and cold data identification; and the compressed sensing block division module is used for entropy-driven block division and compressed residual prediction. Through an entropy-driven dynamic blocking mechanism and a compressed sensing recovery strategy, data features are quantized through information entropy, a compressed residual space is predicted through LSTM, block granularity and a GC strategy are optimized in real time in combination with a genetic algorithm, optimal matching of compression efficiency and block allocation is achieved, compared with a traditional scheme, write amplification is reduced, effective storage density is improved, and the compression efficiency is improved. Meanwhile, the service life of equipment is prolonged through wear entropy balance, and the heterogeneous storage load adaptability is higher.

Description

Technical Field

[0001] The present invention relates to the technical field of hybrid granularity mapping, and specifically to a hybrid granularity mapping system based on data compression. Background Art

[0002] Data compression is a method of reducing the physical occupancy of data through coding techniques, which can be divided into two categories: lossy and lossless. Its core lies in eliminating redundant information to improve storage efficiency. Hybrid granularity refers to managing data blocks of different sizes simultaneously in a storage system, such as 4KB basic blocks and 16KB super blocks. By dynamically selecting the block size to adapt to data characteristics, both access efficiency and space utilization are taken into account. There is a deep coupling relationship between hybrid granularity and data compression. The change in the volume of compressed data requires dynamic adaptation of block granularity to reduce internal fragmentation, and reasonable granularity division can improve the locality of the compression algorithm. The two work together to maximize storage benefits.

[0003] Generally, traditional hybrid granularity mapping mostly uses fixed rules to switch block sizes, without considering the changes in data characteristics after compression, resulting in ineffective utilization of compressed residual space. Moreover, static strategies are difficult to adapt to load fluctuations, easily leading to increased write amplification and uneven wear.

[0004] Based on this, the present invention provides a hybrid granularity mapping system based on data compression to solve the above-mentioned technical problems. Summary of the Invention

[0005] The purpose of the present invention is to provide a hybrid granularity mapping system based on data compression to solve the problems mentioned in the background art.

[0006] To achieve the above purpose, the present invention provides the following technical solutions:

[0007] The present invention proposes a hybrid granularity mapping system based on data compression, including a data preprocessing module, a compressive sensing block partitioning module, a dynamic compression execution module, a hybrid granularity mapping module, an entropy equalization metadata module, a dynamic wear distribution module, a compressive sensing recovery trigger module, and an adaptive strategy optimization module;

[0008] The data preprocessing module is used for data stream sharding and hot / cold data identification;

[0009] The compressive sensing block partitioning module is used for entropy-driven block partitioning and prediction of compression residues;

[0010] The dynamic compression execution module is used for multi-mode compression and adaptation of compression parameters;

[0011] The hybrid granularity mapping module is used for block granularity decision-making and dynamic reorganization of the mapping table;

[0012] The entropy equalization metadata module is used for metadata compressive sensing and metadata distributed caching;

[0013] The dynamic wear leveling module is used for compression space recycling and wear entropy equalization;

[0014] The compressive sensing recycling trigger module is used for fragment entropy value monitoring and cross-granularity recycling coordination;

[0015] The adaptive strategy optimization module is used for feedback compression efficiency and loading gene strategy.

[0016] Preferably, the data preprocessing module further includes a data stream sharding unit and a hot and cold data identification unit;

[0017] The data stream sharding unit dynamically shards continuously written data through a sliding window algorithm, which is used to identify the boundary of compressible data segments. The dynamic window segmentation function is shown in Equation (1):

[0018]

[0019] In the formula, W t is the window size at time t, S compress (d i ) is the compressibility score of data block d i . When the average compressibility within the window exceeds the threshold τ c , the sharding boundary is determined;

[0020] The hot and cold data identification unit marks the data access frequency based on the time decay model and the LRU algorithm, and provides hot and cold labels for subsequent compression and mapping. The time decay access weight is shown in Equation (2):

[0021]

[0022] In the formula, λ is the decay coefficient, γ is the static access weight, LRU_rank represents the least recently used sorting value, and the weight value higher than the threshold θ hot is determined as hot data.

[0023] Preferably, the compressive sensing block partitioning module further includes an entropy value-driven block unit and a compression residue prediction unit;

[0024] The entropy value-driven block unit calculates the information density of the data segment through Shannon entropy, and dynamically selects the 4KB / 16KB mixed granularity block threshold. The Shannon entropy-granularity mapping is shown in Equation (3):

[0025]

[0026] In the formula, H(d) is the information entropy of data segment d, η high is the high entropy threshold;

[0027] The compression residue prediction unit uses an LSTM model to predict the proportion of the remaining space after different compression algorithms, guiding the block granularity selection. The remaining space prediction is shown in Equation (4):

[0028]

[0029] In the formula, the input includes the current entropy value H(d t ), the historical compression ratio C t-1 , the fragment change amount ΔS t-k , and the output predicts the proportion of the remaining space to guide the block granularity selection.

[0030] Preferably, the dynamic compression execution module further includes a multi-mode compression unit and a compression parameter adaptation unit;

[0031] The multi-mode compression unit realizes hierarchical compression through the LZ4 / Zstandard algorithm library, and adopts a low-latency compression mode for hot data. The hot data compression priority function is shown in Equation (5):

[0032]

[0033] Hot data adopts a low-latency compression mode, and cold data adopts a high-compression ratio algorithm;

[0034] The compression parameter adaptation unit dynamically adjusts the dictionary size and compression level based on a genetic algorithm to balance the compression ratio and computational overhead. The compression benefit optimization function is shown in Equation (6):

[0035]

[0036] In the formula, the dictionary size dict and the compression level level are optimized based on a genetic algorithm, and WA pred is the write amplification prediction value.

[0037] Preferably, the hybrid granularity mapping module further includes a block granularity decision unit and a mapping table dynamic reorganization unit;

[0038] The block granularity decision unit selects the mapping method of the basic block of 4KB or the super block of 16KB through a random forest classifier. The random forest classification decision is shown in Equation (7):

[0039]

[0040] In the formula, the input features are the entropy value, the predicted residue, and the 4KB / 16KB granularity decision of the hot weight output;

[0041] The mapping table dynamic reorganization unit adopts a hybrid structure of B+ tree and hash table, and optimizes the mapping table access path through a locality-aware algorithm. The locality weight calculation is shown in Equation (8):

[0042]

[0043] Wherein, N access is the access times of block b, Dist represents the logical distance from the current written block, and high-weight blocks promote the storage level in the B+ tree.

[0044] Preferably, the entropy balance metadata module further includes a compressive sensing metadata unit and a metadata distributed cache unit;

[0045] The compressive sensing metadata unit records the compressive block feature values through a Bloom filter to achieve fast verification of the compression state. The feature encoding of the Bloom filter is shown in Equation (9):

[0046]

[0047] The metadata distributed cache unit uses the consistent hashing algorithm to dynamically allocate metadata storage nodes, reducing the access pressure on a single node. The consistent hashing position mapping is shown in the expression of Equation (10):

[0048]

[0049] Wherein, K meta maps the metadata key to the nearest node on the virtual ring.

[0050] Preferably, the dynamic wear leveling module further includes a compression space recovery unit and a wear entropy balance unit;

[0051] The compression space recovery unit manages the blank areas released by compression through the reserved space weight algorithm, constructs a dynamic virtual block pool, and the weight allocation of the virtual block pool is shown in Equation (11):

[0052]

[0053] Dynamically adjusts according to the proportion of the compressed and released space and the write amplification cycle;

[0054] The wear entropy balance unit calculates the dispersion degree of the block wear distribution based on the information entropy theory, and drives the wear value to approach the equilibrium state. The calculation of the wear distribution dispersion degree is shown in Equation (12):

[0055]

[0056] Wherein, w i is the wear value (erase times) of the i-th block, is the total wear value, and N is the total number of storage blocks.

[0057] Preferably, the compressive sensing recovery trigger module further includes a fragment entropy value monitoring unit and a cross-granularity recovery coordination unit;

[0058] The fragment entropy value monitoring unit calculates the fragmentation entropy value threshold by statistically analyzing the internal fragment distribution of compression blocks through a sliding window;

[0059] The cross-granularity recycling coordination unit uses a dual-queue structure to manage the recycling priorities of basic blocks and super blocks respectively.

[0060] Preferably, the adaptive strategy optimization module further includes a compression efficiency feedback unit and a genetic strategy loading unit;

[0061] The compression efficiency feedback unit models the compression gain curve through a Markov decision process and dynamically adjusts the compression trigger condition. The Markov gain model is shown in Equation (15):

[0062]

[0063] In the formula, the state S includes the compression ratio, CPU load, and fragment entropy value, and the action a is the compression strategy adjustment option;

[0064] The genetic strategy loading unit converts the GC strategy parameters generated by the genetic algorithm into drive instructions and injects them into the block allocator and recycler.

[0065] Compared with the prior art, the beneficial effects of the present invention are:

[0066] Through the entropy-driven dynamic block mechanism and the compression-aware recycling strategy, the present invention quantifies data features through information entropy, predicts the compression residual space through LSTM, and combines the genetic algorithm to optimize the block granularity and GC strategy in real time, achieving the optimal matching of compression efficiency and block allocation. Compared with traditional solutions, it reduces write amplification, improves the effective storage density, and at the same time extends the device life through wear entropy balance, has higher adaptability to heterogeneous storage loads, and solves the problems of traditional methods in terms of space utilization and performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 Shows the topology diagram of the hybrid-granularity mapping system based on data compression of the present invention;

[0068] Figure 2 Shows the flowchart of the hybrid-granularity mapping method based on data compression of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0069] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0070] Example 1, please refer to Figure 1 , the present invention proposes a hybrid granularity mapping system based on data compression, including a data preprocessing module, a compressive sensing block partitioning module, a dynamic compression execution module, a hybrid granularity mapping module, an entropy equalization metadata module, a dynamic wear allocation module, a compressive sensing recycling trigger module, and an adaptive strategy optimization module;

[0071] The data preprocessing module is used for data stream sharding and hot and cold data identification. The compressive sensing block partitioning module is used for entropy-driven chunking and compression residue prediction. The dynamic compression execution module is used for multi-mode compression and compression parameter adaptation. The hybrid granularity mapping module is used for block granularity decision-making and dynamic reorganization of the mapping table. The entropy equalization metadata module is used for metadata compressive sensing and metadata distributed caching. The dynamic wear allocation module is used for compression space recycling and wear entropy equalization. The compressive sensing recycling trigger module is used for fragment entropy value monitoring and cross-granularity recycling coordination. The adaptive strategy optimization module is used for feedback on compression efficiency and loading genetic strategies.

[0072] In this embodiment, it should also be noted that the data preprocessing module further includes a data stream sharding unit and a hot and cold data identification unit;

[0073] Furthermore, the data stream sharding unit dynamically shards continuously written data through a sliding window algorithm to identify the boundaries of compressible data segments. The dynamic window segmentation function is shown in Equation (1):

[0074]

[0075] In the formula, W t is the window size at time t, and S compress (d i ) is the compressibility score of data block d i . When the average compressibility within the window exceeds the threshold τ c , the sharding boundary is determined;

[0076] Furthermore, the hot and cold data identification unit marks the data access frequency based on the time decay model and the LRU algorithm to provide hot and cold labels for subsequent compression and mapping. The time decay access weight is shown in Equation (2):

[0077]

[0078] In the formula, λ is the decay coefficient, γ is the static access weight, LRU_rank represents the least recently used sorting value, and when the weight value is higher than the threshold θ hot , it is determined as hot data.

[0079] In this embodiment, it should also be noted that the compressive sensing block partitioning module further includes an entropy-driven chunking unit and a compression residue prediction unit;

[0080] Furthermore, the entropy-driven chunking unit calculates the information density of the data segment through Shannon entropy, and dynamically selects the 4KB / 16KB hybrid granularity chunking threshold. The Shannon entropy - granularity mapping is shown in Equation (3):

[0081]

[0082] In the formula, H(d) is the information entropy of the data segment d, and η high is the high entropy threshold;

[0083] Furthermore, the compression residue prediction unit uses the LSTM model to predict the proportion of the remaining space after different compression algorithms, and guides the selection of block granularity. The remaining space prediction is shown in Equation (4):

[0084]

[0085] In the formula, the input includes the current entropy value H(d t ), the historical compression rate C t-1 , the fragment change amount ΔS t-k , and the output predicts the proportion of the remaining space to guide the selection of block granularity.

[0086] In this embodiment, it should also be noted that the dynamic compression execution module further includes a multi - mode compression unit and a compression parameter adaptation unit;

[0087] Furthermore, the multi - mode compression unit realizes hierarchical compression through the LZ4 / Zstandard algorithm library, and adopts a low - latency compression mode for hot data. The hot data compression priority function is shown in Equation (5):

[0088]

[0089] Hot data adopts a low - latency compression mode, and cold data adopts a high - compression - rate algorithm;

[0090] Furthermore, the compression parameter adaptation unit dynamically adjusts the dictionary size and compression level based on the genetic algorithm to balance the compression rate and computational overhead. The compression gain optimization function is shown in Equation (6):

[0091]

[0092] In the formula, the dictionary size dict and the compression level level are optimized based on the genetic algorithm, and WA pred is the write amplification prediction value.

[0093] In this embodiment, it should also be noted that the hybrid granularity mapping module further includes a block granularity decision unit and a mapping table dynamic reorganization unit;

[0094] Furthermore, the block granularity decision unit selects the 4KB basic block or 16KB super block mapping method through a random forest classifier. The random forest classification decision is shown in Equation (7):

[0095]

[0096] In the formula, the input features are entropy value, prediction residue, and thermal weight output for 4KB / 16KB granularity decision;

[0097] Furthermore, the mapping table dynamic reorganization unit adopts a hybrid structure of B+ tree and hash table, and optimizes the mapping table access path through a locality-aware algorithm. The locality weight calculation is shown in Equation (8):

[0098]

[0099] In the formula, N access is the access count of block b, Dist represents the logical distance from the current written block, and high-weight blocks improve the storage level in the B+ tree.

[0100] In this embodiment, it should also be noted that the entropy balance metadata module further includes a compressive sensing metadata unit and a metadata distributed cache unit;

[0101] Furthermore, the compressive sensing metadata unit records the compressed block feature values through a Bloom filter to achieve fast verification of the compression state. The Bloom filter feature encoding is shown in Equation (9):

[0102]

[0103] Furthermore, the metadata distributed cache unit uses a consistent hashing algorithm to dynamically allocate metadata storage nodes, reducing the access pressure on a single node. The consistent hashing location mapping is shown in Equation (10) expression:

[0104]

[0105] In the formula, K meta maps the metadata key to the nearest node on the virtual ring.

[0106] In this embodiment, it should also be noted that the dynamic wear leveling module further includes a compression space recovery unit and a wear entropy balance unit;

[0107] Furthermore, the compression space recovery unit manages the blank areas released by compression through a reserved space weight algorithm, constructs a dynamic virtual block pool, and the virtual block pool weight allocation is shown in Equation (11):

[0108]

[0109] Dynamically adjust according to the proportion of the compressed released space and the write amplification cycle;

[0110] Further, the wear entropy balancing unit calculates the dispersion of the block wear distribution based on the information entropy theory, driving the wear value to approach the equilibrium state. The calculation of the wear distribution dispersion is shown in Equation (12):

[0111]

[0112] In the formula, w i is the wear value (erase count) of the i-th block, is the total wear value, and N is the total number of storage blocks.

[0113] In this embodiment, it should also be noted that the compressive sensing recycling trigger module further includes a fragment entropy value monitoring unit and a cross-granularity recycling coordination unit;

[0114] Further, the fragment entropy value monitoring unit calculates the fragmentation entropy value threshold by statistically analyzing the internal fragment distribution of the compressed block through a sliding window;

[0115] Further, the cross-granularity recycling coordination unit adopts a dual-queue structure to manage the recycling priorities of the base blocks and the super blocks respectively.

[0116] In this embodiment, it should also be noted that the adaptive strategy optimization module further includes a compression efficiency feedback unit and a genetic strategy loading unit;

[0117] Further, the compression efficiency feedback unit models the compression benefit curve through a Markov decision process and dynamically adjusts the compression trigger condition. The Markov benefit model is shown in Equation (15):

[0118]

[0119] In the formula, the state S includes the compression ratio, CPU load, and fragment entropy value, and the action a is the compression strategy adjustment option;

[0120] Further, the genetic strategy loading unit converts the GC strategy parameters generated by the genetic algorithm into driving instructions and injects them into the block allocator and recycler.

[0121] Embodiment 2. Please refer to Figure 2 , based on the above hybrid granularity mapping system based on data compression. In practical applications, specifically, the hybrid granularity mapping method based on data compression includes the following steps:

[0122] S1: Data writing and preprocessing:

[0123] Dynamic sharding: The data stream sharding unit divides the continuously written data stream in real time through a sliding window algorithm (Equation 1). Specifically, the window size W t= 8. When the average compressibility score of 8 consecutive 4KB data blocks is ≥ 0.7, they are merged into a 64KB shard, and the segmented data is output and marked as an independently compressible unit;

[0124] Hot and cold data marking: The hot and cold data recognition unit calculates the access weight w by combining the time decay model and the LRU algorithm hot , for the data that has been accessed frequently recently (such as accessed 3 times within 10 minutes), the weight w hot > θ hot , is marked as hot data, and the data segment set with hot and cold labels is output;

[0125] S2: Compressive sensing block division:

[0126] Entropy-driven chunking: The entropy-driven chunking unit calculates the information entropy H(d) of the data segment. If H(d) ≥ 2.4 (high entropy threshold), 4KB granularity chunking is selected; otherwise, 16KB super-chunks are selected, and a set of candidate chunks with mixed granularity (4KB / 16KB) is output;

[0127] Residual space prediction: The compression residue prediction unit uses the LSTM model to predict the proportion of the residual space after compression The input features include the current entropy value, historical compression ratio, and fragment change trend. If the prediction forces degradation to a smaller granularity chunk, and the final chunk granularity decision and compression parameter suggestions are output;

[0128] S3: Dynamic compression execution:

[0129] Hierarchical compression: The multi-mode compression unit selects the algorithm according to the hot and cold labels. For hot data, LZ4 fast compression is used (delay < 50μs), and for cold data, the Zstandard high compression rate mode is used;

[0130] The compressed data blocks, their compression rates, and timestamp metadata are output;

[0131] Parameter dynamic optimization: The compression parameter adaptation unit optimizes the dictionary size and compression level through the genetic algorithm:

[0132] The parameters are adjusted once every 1 hour to balance the compression rate and CPU utilization, adjusted from 64KB to 128KB, and the updated compression parameter configuration table is output;

[0133] S4: Mixed granularity mapping:

[0134] Granularity decision: The block granularity decision unit selects the mapping method through a random forest classifier:

[0135] Input features: Entropy value H(d) = 2.1, predicted residue Hot weight w hot= 0.75, output a 4KB basic block mapping (classification probability > 60%), and output the physical address mapping relationship table;

[0136] Mapping table optimization: The mapping table dynamic reorganization unit calculates the locality weight w loc , and reconstructs the B+ tree level:

[0137] Frequently accessed blocks (such as metadata blocks) are promoted to the top layer of the B+ tree to reduce the query depth, and output the optimized mixed granularity mapping table;

[0138] S5: Metadata management for deskewing:

[0139] Metadata compression verification: The compressed sensing metadata unit quickly verifies the integrity of the compressed block through a Bloom filter:

[0140] Use SHA-256 hash mixed with salt value to generate a feature code, with a false positive rate < 0.1%, and output the compression status verification result (valid / invalid);

[0141] Distributed cache scheduling: The metadata distributed cache unit uses consistent hashing to allocate metadata storage nodes:

[0142] Metadata key K meta Hash to the virtual ring, select the nearest node in the clockwise direction, and output the metadata access path table;

[0143] S6: Compressed sensing recovery trigger:

[0144] Fragmentation entropy monitoring: The fragmentation entropy monitoring unit calculates the fragmentation distribution entropy H frag :

[0145] When H frag > 2.5 (such as more than 10 fragment sizes appear), trigger recovery, output the fragmentation warning signal and the target block list, Cross-granularity coordination: The cross-granularity recovery coordination unit performs a dual-queue priority decision:

[0146] When the fragment ratio of the super block queue ≥ 70%, it is processed preferentially, otherwise the basic block queue is processed. After processing the super block queue continuously 3 times, it is forced to switch to the basic block queue, and output the recovery task scheduling instruction;

[0147] S7: Dynamic wear leveling and recovery:

[0148] Virtual block pool management: The compression space recovery unit calculates the virtual block pool weight w virtual :

[0149] Dynamically allocate reserved areas according to the ratio of the compressed release space, and output the virtual block address mapping table;

[0150] Wear leveling: The wear entropy leveling unit calculates the wear distribution entropy H wear:

[0151] When H wear <2.3, start the wear leveling algorithm to migrate data of highly worn blocks and output the wear leveling operation log;

[0152] S8: Adaptive policy optimization:

[0153] Compression efficiency feedback The compression efficiency feedback unit models the long-term benefit through a Markov decision process:

[0154] The state s includes the current compression ratio, CPU load, fragmentation value (2.1), and the action a selects to adjust the compression trigger threshold. For example, reduce it from 70% to 65%, and output the policy adjustment suggestion;

[0155] Genetic policy loading: The genetic policy loading unit injects the output parameters of the genetic algorithm into the system, converts the genetic encoding into physical parameters through the parameter space mapping function, and outputs the updated GC policy configuration file.

[0156] Through the above steps S1 to S8, the present invention uses an entropy-driven dynamic block mechanism and a compressive sensing recycling strategy. It quantifies data features through information entropy, predicts the compressed residual space by LSTM, combines a genetic algorithm to optimize the block granularity and GC policy in real time, realizes the optimal matching of compression efficiency and block allocation, reduces the write amplification and improves the effective storage density compared with the traditional scheme. At the same time, it extends the device life through wear entropy leveling, has higher adaptability to heterogeneous storage loads, and solves the problems of the traditional method between space utilization and performance.

[0157] In the description of this specification, the descriptions referring to the terms "one embodiment", "example", "specific example", etc. mean that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0158] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made according to the content of this specification. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the present invention, so that those skilled in the art in the relevant technical field can understand and utilize the present invention well. The present invention is only limited by the claims and their full scope and equivalents.

Claims

1. A hybrid granularity mapping system based on data compression, characterized in that, It includes a data preprocessing module, a compressive sensing block division module, a dynamic compression execution module, a hybrid granularity mapping module, an entropy equilibrium metadata module, a dynamic wear allocation module, a compressive sensing recycling trigger module, and an adaptive strategy optimization module; The data preprocessing module is used for data stream sharding and hot and cold data identification; The compressive sensing block division module is used for entropy-driven block division and compression residue prediction; The dynamic compression execution module is used for multi-mode compression and compression parameter adaptation; The hybrid granularity mapping module is used for block granularity decision-making and dynamic reorganization of the mapping table; The entropy equilibrium metadata module is used for metadata compressive sensing and metadata distributed caching; The dynamic wear allocation module is used for compression space recycling and wear entropy equilibrium; The compressive sensing recycling trigger module is used for fragment entropy value monitoring and cross-granularity recycling coordination; The adaptive strategy optimization module is used for feedback on compression efficiency and loading genetic strategies.

2. The hybrid granularity mapping system based on data compression according to claim 1, characterized in that, The data preprocessing module further includes a data stream sharding unit and a hot and cold data identification unit; The data stream sharding unit dynamically shards continuously written data through a sliding window algorithm to identify the boundaries of compressible data segments. The dynamic window segmentation function is shown in Equation (1): Where, W t is the window size at time t, S compress (d i ) is the compressibility score of data block d i . When the average compressibility within the window exceeds the threshold τ c , the shard boundary is determined; The hot and cold data identification unit marks the data access frequency based on the time decay model and the LRU algorithm to provide hot and cold labels for subsequent compression and mapping. The time decay access weight is shown in Equation (2): Where λ is the attenuation coefficient, γ is the static access weight, LRU_rank represents the least recently used sorting value, and the weight value is higher than the threshold θ hot is determined as hot data.

3. The hybrid granularity mapping system based on data compression according to claim 2, wherein, The compressive sensing block division module further includes an entropy-driven block division unit and a compression residue prediction unit; The entropy-driven block division unit calculates the information density of the data segment through Shannon entropy and dynamically selects the 4KB / 16KB hybrid granularity block division threshold. The Shannon entropy - granularity mapping is shown in Equation (3): where \(H(d)\) is the information entropy of data segment \(d\), and \(\eta\) high is the high entropy threshold; The compression residue prediction unit uses the LSTM model to predict the proportion of the remaining space after different compression algorithms to guide the selection of block granularity. The remaining space prediction is shown in Equation (4): Wherein, the input includes the current entropy value H(d t ), the historical compression rate C t-1 , the fragment change amount ΔS t-k , and the output predicted residual space occupancy ratio to guide the selection of the guide block granularity.

4. A hybrid granularity mapping system based on data compression according to claim 3, characterized in that, The dynamic compression execution module further includes a multi-mode compression unit and a compression parameter adaptation unit; The multi-mode compression unit realizes hierarchical compression through the LZ4 / Zstandard algorithm library, and adopts a low-latency compression mode for hot data. The hot data compression priority function is shown in Equation (5): Hot data adopts a low-latency compression mode, and cold data adopts a high compression ratio algorithm; The compression parameter adaptation unit dynamically adjusts the dictionary size and compression level based on the genetic algorithm to balance the compression ratio and computational overhead. The compression benefit optimization function is shown in Equation (6): Wherein, based on the genetic algorithm, the dictionary size dict and the compression level level are optimized, and WA pred is the write amplification prediction value.

5. A hybrid granularity mapping system based on data compression according to claim 4, wherein The hybrid granularity mapping module further includes a block granularity decision-making unit and a mapping table dynamic reorganization unit; The block granularity decision-making unit selects the mapping method of the basic block of 4KB or the super block of 16KB through a random forest classifier. The random forest classification decision is shown in Equation (7): In the formula, the input features are entropy value, predicted residue, and hot weight, and the 4KB / 16KB granularity decision is output; The mapping table dynamic reorganization unit adopts a hybrid structure of B+ tree and hash table, and optimizes the mapping table access path through a locality-aware algorithm. The locality weight calculation is shown in Equation (8): Where N access is the access count of block b, Dist represents the logical distance from the currently written block, and high-weight blocks promote the storage level in the B+ tree.

6. The hybrid granularity mapping system based on data compression according to claim 5, wherein, The entropy equilibrium metadata module further includes a compressive sensing metadata unit and a metadata distributed caching unit; The compression sensing metadata unit records the compressed block feature values through a Bloom filter to achieve fast verification of the compression state. The feature encoding of the Bloom filter is shown in Equation (9): The metadata distributed cache unit uses the consistent hashing algorithm to dynamically allocate metadata storage nodes, reducing the access pressure on a single node. The consistent hashing position mapping is shown in the expression of Equation (10): where K meta The metadata key is mapped to the nearest neighbor node of the virtual ring.

7. A hybrid granularity mapping system based on data compression according to claim 6, characterized in that The dynamic wear leveling module further includes a compression space recovery unit and a wear entropy balancing unit; The compression space recovery unit manages the blank areas released by compression through the reserved space weight algorithm, constructs a dynamic virtual block pool, and the weight allocation of the virtual block pool is shown in Equation (11): Dynamically adjust according to the proportion of the compressed and released space and the write amplification cycle; The wear entropy balancing unit calculates the dispersion of the block wear distribution based on the information entropy theory, driving the wear values to approach the equilibrium state. The calculation of the wear distribution dispersion is shown in Equation (12): where w i is the wear value of the i-th block, is the total wear value, and N is the total number of storage blocks.

8. A hybrid granularity mapping system based on data compression according to claim 7, characterized in that The compression sensing recovery trigger module further includes a fragmentation entropy value monitoring unit and a cross-granularity recovery coordination unit; The fragmentation entropy value monitoring unit calculates the fragmentation entropy value threshold by statistically analyzing the internal fragmentation distribution of the compressed blocks through a sliding window; The cross-granularity recovery coordination unit uses a dual-queue structure to manage the recovery priorities of the base blocks and super blocks respectively.

9. A hybrid granularity mapping system based on data compression according to claim 8, wherein The adaptive strategy optimization module further includes a compression efficiency feedback unit and a genetic strategy loading unit; The compression efficiency feedback unit models the compression revenue curve through a Markov decision process and dynamically adjusts the compression trigger condition. The Markov revenue model is shown in Equation (15): In the formula, the state S includes the compression ratio, CPU load, and fragmentation entropy value, and the action a is the compression strategy adjustment option; The genetic strategy loading unit is used to inject driving instructions into the block allocator and recycler.

Images