Enhanced eMMC / SSD data integrity protection framework

Abstract

The invention discloses an enhanced eMMC / SSD data integrity protection framework, and the framework comprises a multi-level adaptive ECC architecture module which is used for carrying out the at least one operation of a dynamic protection operation, an adaptive error correction operation, an increment verification operation and a variable code rate ECC operation through employing a plurality of ECC algorithms; the AI auxiliary error management module is used for carrying out real-time error feature analysis and error propagation prediction on the storage equipment; the distributed verification mechanism module is used for carrying out cross-device verification, constructing multiple protection layers and carrying out verification performance optimization; a write acceleration module based on a dual-interface SSD performs dynamic I / O redirection, device end LSM write cache, multi-level cache and host device collaborative consistency by adopting a mixed interface SSD architecture; the software instruction recombination and hardware part reconfiguration module is used for carrying out software instruction recombination and hardware part reconfiguration when a fault occurs; and the multi-tenant support module is used for realizing multi-tenant isolation. Through the mode, the data reliability can be improved, and the system performance can be optimized.

Description

Technical Field

[0001] The present application relates to the field of eMMC / SSD technology, and in particular to an enhanced eMMC / SSD data integrity protection framework. Background Art

[0002] Currently, eMMC / SSD storage devices face multiple challenges in data integrity. Traditional error correction code (ECC) technology has limitations in dealing with complex error patterns, especially in the following aspects: Complex error modes: Traditional ECC technology has difficulty effectively detecting and recovering complex error modes caused by multiple factors, such as process differences, temperature changes, radiation, electromagnetic interference, etc. These error modes include random errors, burst errors, and permanent errors caused by hardware aging or damage.

[0003] Performance overhead: Performing data integrity checks (such as ECC checks and data comparisons) introduces additional performance overhead, reducing the overall efficiency of the storage system. Frequent check operations consume CPU resources and increase the I / O latency of storage devices.

[0004] Error recovery capability: The error recovery capability of existing technologies is limited, making it difficult to ensure data reliability under various abnormal circumstances. For example, traditional ECC may not be able to recover errors that exceed its error correction capability.

[0005] Silent data corruption: Silent data corruption is difficult to prevent and detect, i.e., errors that occur when data is not being used. Such errors are usually difficult to detect and may result in data loss or corruption.

[0006] New storage requirements: New storage devices (such as high-density eMMC / SSD, STT-MRAM, etc.) have higher requirements for data protection and performance. For example, STT-MRAM is greatly affected by temperature changes and requires adaptive error correction strategies.

[0007] Hardware failure: Space radiation, malicious attacks and other factors may cause hardware failure, which requires a mechanism that can quickly detect, recover and ensure continuous operation of the system. Traditional fault-tolerance methods, such as triple modular redundancy (TMR), can handle hardware failures, but they will bring huge overhead.

[0008] Write delay: Key-value storage systems based on log-structured merge trees (LSM-trees) will generate write delays when performing compaction operations (reading data from the hard disk, then reorganizing the data structure and index, and then writing it back to the hard disk, which can double the reading performance), affecting system performance. Traditional I / O scheduling methods or multi-threaded compaction can alleviate this problem, but they cannot eliminate it. Summary of the invention

[0009] The enhanced eMMC / SSD data integrity protection framework provided in this application can improve data reliability and optimize system performance.

[0010] In the first aspect, the present application provides an enhanced eMMC / SSD data integrity protection framework, including: a multi-level adaptive ECC architecture module, which is used to use multiple ECC algorithms to perform at least one of dynamic protection operations, adaptive error correction operations, incremental verification operations, and variable bit rate ECC operations; an AI-assisted error management module, which is used to perform real-time error feature analysis, error propagation prediction, and intelligent recovery strategy selection on storage devices, and use a time-to-digital converter sensor to monitor the voltage fluctuations and signal delay changes of the chip in real time; a distributed verification mechanism module, which is used to perform cross-device verification, build multiple protection layers, optimize verification performance, and use incremental synchronization technology for data recovery; a dual-interface SSD-based write acceleration module, which uses a hybrid interface SSD architecture for dynamic I / O redirection, device-side LSM write cache, multi-level cache, host device collaborative consistency, and load-based rollback adjustment; a software instruction reorganization and hardware partial reconfiguration module, which is used to perform software instruction reorganization and hardware partial reconfiguration in the event of a failure; a multi-tenant support module, which is used to achieve multi-tenant isolation.

[0011] Among them, the multi-level adaptive ECC architecture module includes: a dynamic protection unit, which is used to dynamically adjust the ECC protection strength according to the importance of the data; an adaptive error correction unit, which is used to automatically select the most appropriate error correction strategy according to the real-time error situation and channel conditions; an incremental check unit, which is used to check only the modified data blocks; a variable code rate ECC unit, which is used to dynamically adjust the ECC code rate according to the real-time channel conditions and data importance.

[0012] Among them, the AI-assisted error management module includes: an error feature analysis unit, which is used to perform real-time analysis on errors generated by storage devices and extract features; an error propagation prediction unit, which is used to predict potential error propagation paths using AI models; an intelligent recovery strategy unit, which is used to intelligently select the best recovery strategy based on the error type and prediction results; a time-to-digital converter sensor unit, which uses a TDC sensor to monitor the voltage fluctuations and signal delay changes of the chip in real time.

[0013] Among them, the distributed verification mechanism module includes: a cross-device verification unit, which is used to perform data verification between multiple storage devices; a multiple protection layer unit, which is used to establish a multi-layer data protection mechanism; a verification performance optimization unit, which is used to use parallel verification algorithms and hardware accelerators for performance optimization; a fast recovery unit, which is used to use incremental synchronization technology to synchronize only the data that has changed.

[0014] Among them, the write acceleration module based on dual-interface SSD includes: a dynamic I / O redirection unit, which is used to utilize the characteristics of dual-interface SSD to redirect the write request to the KV interface on the device side as a temporary write buffer when a write pause occurs in the host-side LSM-KVS; a device-side LSM write cache unit, which is used to build a write buffer using the LSM tree structure on the device side, and implement a range scanning mechanism through an iterator to synchronize the cached data back to the host-side LSM; a multi-level cache unit, which is used to adopt a multi-level cache strategy in the device-side LSM write cache; a host-device collaborative consistency unit, which is used to ensure the consistency of host-side and device-side data through a range scanning rollback mechanism; and a rollback unit, which is used to dynamically adjust the rollback strategy according to the load type.

[0015] Among them, the host-side LSM-KVS adopts two-layer perfect hash tables, perfect hash technology, intelligent partitioning technology, fragment file structure and data sorting technology.

[0016] The first layer of the two-layer perfect hash table is configured with a fragment file, and the second layer is configured with a local hash table; each fragment file is used to store a portion of key-value pair data, and each fragment file contains a local hash table.

[0017] Among them, the perfect hashing technology divides the hash bucket into dense buckets and sparse buckets. When constructing a new hash bucket, it reuses the existing hash parameters and uses SIMD instructions (parallel calculation of the key hash value and index position.

[0018] Among them, the software instruction reorganization and hardware partial reconfiguration module includes: a software instruction reorganization unit, which is used to replace the affected instructions with other available ALU instructions or instruction sequences through software instruction reorganization when an arithmetic logic unit component failure is detected; a hardware partial reconfiguration unit, which is used to replace the damaged processor component with an undamaged area on the chip through dynamic partial reconfiguration when the hardware fails.

[0019] Among them, the multi-tenant support module includes: a namespace isolation unit, which is used to use the namespace technology of the NVMe standard to implement multi-tenant isolation on the block interface and the key-value interface respectively.

[0020] The beneficial effects of the present application are as follows: Different from the prior art, the enhanced eMMC / SSD data integrity protection framework provided by the present application includes: a multi-level adaptive ECC architecture module, which is used to use multiple ECC algorithms to perform at least one of dynamic protection operations, adaptive error correction operations, incremental verification operations and variable bit rate ECC operations; an AI-assisted error management module, which is used to perform real-time error feature analysis, error propagation prediction, and intelligent recovery strategy selection on storage devices, and use a time-to-digital converter sensor to monitor the voltage fluctuation and signal delay changes of the chip in real time; a distributed verification mechanism module, which is used to perform cross-device verification, build multiple protection layers, optimize verification performance, and use incremental synchronization technology for data recovery; a write acceleration module based on a dual-interface SSD, which uses a hybrid interface SSD architecture for dynamic I / O redirection, device-side LSM write cache, multi-level cache, host device collaborative consistency, and rollback according to load adjustment; a software instruction reorganization and hardware partial reconfiguration module, which is used to perform software instruction reorganization and hardware partial reconfiguration in the event of a failure; a multi-tenant support module, which is used to achieve multi-tenant isolation. Through the above method, data reliability can be improved and system performance can be optimized, which effectively solves the challenges faced by traditional eMMC / SSD in data integrity protection and provides a more reliable, efficient and secure solution for new storage devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them: Figure 1 It is a structural diagram of an embodiment of an enhanced eMMC / SSD data integrity protection framework provided by the present application; Figure 2 It is a structural diagram of an embodiment of a multi-level adaptive ECC architecture module provided by the present application; Figure 3 It is a structural diagram of an embodiment of an AI-assisted error management module provided by the present application; Figure 4 It is a structural diagram of an embodiment of a distributed authentication mechanism module provided by the present application; Figure 5 It is a structural diagram of an embodiment of a write acceleration module based on a dual-interface SSD provided by the present application; Figure 6 It is a structural diagram of an embodiment of a software instruction reorganization and hardware partial reconfiguration module provided by the present application. DETAILED DESCRIPTION

[0022] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some but not all structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0023] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0024] See also Figure 1 , Figure 1 1 is a schematic diagram of an embodiment of an enhanced eMMC / SSD data integrity protection framework provided by the present application. The enhanced eMMC / SSD data integrity protection framework 100 includes: a multi-level adaptive ECC architecture module 10, an AI-assisted error management module 20, a distributed verification mechanism module 30, a dual-interface SSD-based write acceleration module 40, a software instruction reorganization and hardware partial reconfiguration module 50, and a multi-tenant support module 60.

[0025] In some embodiments, the multi-level adaptive ECC architecture module 10 is used to use multiple ECC algorithms to perform at least one of dynamic protection operations, adaptive error correction operations, incremental check operations, and variable code rate ECC operations.

[0026] See also Figure 2 The multi-level adaptive ECC architecture module 10 includes: a dynamic protection unit 11, an adaptive error correction unit 12, an incremental check unit 13 and a variable code rate ECC unit 14.

[0027] In some embodiments, the multi-level adaptive ECC architecture module 10 combines different ECC algorithms to adapt to different error types and occurrence frequencies. For example, for random errors, Hamming code or BCH code is used, and for burst errors, Reed-Solomon code or LDPC code is used to achieve effective coverage of multiple error types.

[0028] The dynamic protection unit 11 is used to dynamically adjust the ECC protection strength according to the importance of the data. For important data, a high-intensity ECC algorithm (such as LDPC code or BCH code) is used to provide stronger error correction capability and a larger check codeword length. For non-important data, a low-intensity ECC algorithm (such as Hamming code) is used to reduce the check overhead and shorten the delay.

[0029] The adaptive error correction unit 12 is used to automatically select the most appropriate error correction strategy according to the real-time error situation and channel conditions, and adopts the adaptive decoding technology based on deep learning and the neural network bit flipping (NBF) decoding technology.

[0030] The adaptive decoding technology based on deep learning mainly uses deep learning technology to train the neural network (NN) decoder. The decoder can dynamically adjust the decoding complexity according to the channel conditions (such as the bit error rate BER) and select the appropriate decoder and NN layer number. For example, when the channel conditions are good, a lower complexity decoder and fewer NN layers are selected to reduce latency and energy consumption; when the channel conditions are poor, a higher complexity decoder and more NN layers are selected to ensure error correction performance.

[0031] The neural network bit flipping (NBF) decoding technology mainly uses the NBF algorithm to implement the bit flipping algorithm. Using trellis representation, it can share the same structure with the neural belief propagation (NBP) and neural offset minimum sum (NOMS) algorithms, can be flexibly switched, and has the characteristics of low latency and low power consumption.

[0032] The incremental verification unit 13 is used to verify only the modified data blocks. Incremental verification is supported, and only the modified data blocks are verified to reduce the verification overhead. The modification of the data blocks is tracked using a hash algorithm or a timestamp, and only the changed data blocks are verified.

[0033] The variable rate ECC unit 14 is used to dynamically adjust the ECC rate according to the real-time channel conditions and data importance, so as to optimize the performance and protection effect in different scenarios. The rate adjustment can be based on the prediction and real-time detection of the error occurrence rate.

[0034] In some embodiments, the AI-assisted error management module 20 is used to perform real-time error feature analysis, error propagation prediction, and intelligent recovery strategy selection on the storage device, and uses a time-to-digital converter sensor to monitor the voltage fluctuation and signal delay changes of the chip in real time.

[0035] In some embodiments, there is an error mode database in the AI-assisted error management module 20. For example, a database containing various known error modes is established, including resistance shifts caused by temperature changes and process differences, single-particle effects (SEE) caused by radiation, and failure modes caused by hardware aging. The error modes in the database include not only the type of error, but also the frequency, intensity, and scope of the error.

[0036] See also Figure 3 The AI-assisted error management module 20 includes: an error feature analysis unit 21, an error propagation prediction unit 22, an intelligent recovery strategy unit 23 and a time-to-digital converter sensor unit 24.

[0037] The error feature analysis unit 21 is used to analyze the errors generated by the storage device in real time and extract features. For example, the errors generated by the storage device are analyzed in real time and features are extracted so as to timely discover and handle abnormal situations. The error features are analyzed using a machine learning algorithm to classify the errors, thereby selecting the best recovery strategy. For example, ECC error correction is used for soft errors, and partial reconfiguration is used for hard errors.

[0038] The error propagation prediction unit 22 is used to predict potential error propagation paths using an AI model. For example, by analyzing the network and pattern of error propagation, the area that the error may affect is predicted, and preventive measures are taken in advance. Graph neural networks are used for prediction and analysis.

[0039] The intelligent recovery strategy unit 23 is used to intelligently select the best recovery strategy according to the error type and the prediction result. For example, for soft errors, ECC error correction and data rereading are used; for hard errors, partial reconfiguration or software instruction reorganization is used.

[0040] The time-to-digital converter sensor unit 24 is used to use a TDC sensor to monitor the voltage fluctuation and signal delay changes of the chip in real time, so as to detect potential faults and attacks in advance. The TDC sensor detects abnormal voltage fluctuations by measuring the propagation delay of the signal in the chip, and these abnormal fluctuations are often a precursor to faults or attacks.

[0041] In some embodiments, the distributed verification mechanism module 30 is used to perform cross-device verification, build multiple protection layers, optimize verification performance, and use incremental synchronization technology to perform data recovery.

[0042] See also Figure 4 The distributed verification mechanism module 30 includes: a cross-device verification unit 31, a multiple protection layer unit 32, a verification performance optimization unit 33 and a fast recovery unit 34.

[0043] The cross-device verification unit 31 is used to perform data verification between multiple storage devices. Among them, performing data verification between multiple storage devices can improve reliability, especially in large-scale storage systems. Distributed hash tables (DHT) or other distributed data structures are used to store verification information and perform fast verification operations.

[0044] The multi-layer protection unit 32 is used to establish a multi-layer data protection mechanism. By establishing a multi-layer data protection mechanism, data integrity can be guaranteed even if some devices fail. For example, RAID technology (such as RAID5 or RAID6) is used for data backup and redundancy, and erasure coding technology is used to improve data storage efficiency.

[0045] The verification performance optimization unit 33 is used to optimize the performance by using a parallel verification algorithm and a hardware accelerator. For example, by optimizing the algorithm and hardware, the performance overhead caused by the distributed verification is reduced. For example, a parallel verification algorithm and a hardware accelerator are used, such as using an FPGA or an ASIC to implement hardware acceleration, thereby reducing the occupancy of CPU resources.

[0046] The fast recovery unit 34 is used to synchronize only the changed data by using the incremental synchronization technology. The fast recovery unit 34 supports fast data recovery and shortens the system downtime caused by errors. By using the incremental synchronization technology, only the changed data is synchronized, thereby reducing the time for data recovery.

[0047] In some embodiments, the dual-interface SSD-based write acceleration module 40 uses a hybrid interface SSD architecture for dynamic I / O redirection, device-side LSM write cache, multi-level cache, host device collaborative consistency, and load-adjusted rollback.

[0048] The dual-interface SSD write acceleration module 40 adopts a hybrid interface SSD architecture, which divides the logical NAND flash address space of the SSD into two areas: one for the traditional block interface, which is managed by the LSM-KVS on the host side; and the other for the key value (KV) interface, which serves as a temporary write buffer. This architecture allows both block access and KV access to be supported on the same device.

[0049] See also Figure 5 The write acceleration module 40 based on the dual-interface SSD includes: a dynamic I / O redirection unit 41, a device-side LSM write cache unit 42, a multi-level cache unit 43, a host device collaborative consistency unit 44 and a rollback unit 45.

[0050] The dynamic I / O redirection unit 41 is used to utilize the characteristics of the dual-interface SSD to redirect the write request to the KV interface on the device side as a temporary write buffer when a write pause occurs in the host-side LSM-KVS. Utilizing the characteristics of the dual-interface SSD, when a write pause occurs in the host-side LSM-KVS (for example, during compaction), the write request is redirected to the KV interface on the device side as a temporary write buffer, thereby avoiding the write pause. When the host-side write operation is resumed, the data in the temporary write buffer can be synchronized back to the host side.

[0051] The device-side LSM write cache unit 42 is used to build a write buffer using the LSM tree structure on the device side, and implement a range scan mechanism through an iterator to synchronize the cached data back to the host-side LSM. The device-side LSM tree can be a lightweight implementation that reduces resource usage and focuses on fast write and scan operations.

[0052] The multi-level cache unit 43 is used to adopt a multi-level cache strategy in the device-side LSM write cache.

[0053] To further improve performance, a multi-level caching strategy can be adopted in the device-side LSM write cache, such as using DRAM as a faster cache layer to reduce access to NAND flash memory.

[0054] The host device coordination consistency unit 44 is used to ensure the consistency of data on the host side and the device side through a range scan rollback mechanism. The range scan rollback mechanism ensures the consistency of data on the host side and the device side and optimizes the flow of data between the two interfaces. The rollback mechanism includes the following steps: Detecting write stalls: The write stall detector on the host side monitors the status of LSM-KVS and triggers I / O redirection when a write stall is detected.

[0055] Write to KV interface: Write write requests to the KV interface on the device side. These write requests will be cached in the LSM tree structure on the device side.

[0056] Range scan: When the host-side write operation is resumed, the iterator is used to perform a range scan on the LSM tree on the device side to read the cached data back to the host side.

[0057] Data merging: Merge the cached data on the device into the LSM tree on the host. During the data merging process, deduplication and sorting operations are required.

[0058] The rollback unit 45 is used to dynamically adjust the rollback strategy according to the load type.

[0059] For read-intensive workloads, an active rollback strategy is used to synchronize data back to the host in a timely manner to improve read performance. For write-intensive workloads, a delayed rollback strategy is used to reduce the frequency of rollback operations to improve write performance.

[0060] The write acceleration module 40 based on the dual-interface SSD further includes: a dynamic namespace allocation unit, which dynamically divides and manages the address space of the block interface and the key-value interface through the NVMe namespace technology, and supports isolation and access control of different tenants.

[0061] In some embodiments, the host-side LSM-KVS uses a two-layer perfect hash table, perfect hash technology, smart partitioning technology, fragment file structure, and data organization technology.

[0062] The first layer of the two-layer perfect hash table is configured with a fragment file, and the second layer is configured with a local hash table; each fragment file is used to store a portion of key-value pair data, and each fragment file contains a local hash table.

[0063] Among them, the perfect hashing technology divides the hash bucket into dense buckets and sparse buckets. When constructing a new hash bucket, it reuses the existing hash parameters and uses SIMD instructions (parallel calculation of the key hash value and index position.

[0064] The host-side LSM-KVS management mainly includes the following contents: 1. Two-layer perfect hash table (TPH).

[0065] Problems with traditional databases: In traditional databases, data is usually stored in SSTable (SortedStringTable) files. When reading, it may be necessary to search in multiple SSTables, resulting in high read latency. When updating data, the compaction process of the LSM tree may require rewriting a large amount of data, resulting in write amplification problems.

[0066] This application LSM-KVS management innovation: a special data structure called "two-layer perfect hash table" (TPH) is used.

[0067] First layer: Piece Files: The data of the entire database is divided into multiple piece files. Each piece file is similar to the SS Table in the traditional database and stores a part of the key-value pair data.

[0068] Second layer: Local Hash Table: Each fragment file contains a local hash table. This hash table maps the key to the specific location (offset) of the key-value pair data in the fragment file.

[0069] Global Perfect Hash Table: There is a global perfect hash table. This hash table maps the hash value of a key to an index slot. The index slot stores the signature of the key and a pointer to the fragment file where the key is located.

[0070] The advantage is that it can be quickly searched and updated easily.

[0071] Fast search is mainly reflected in the key search. First, the index slot is calculated through the global perfect hash table, and then the fragment file is directly located based on the information in the slot (key signature and fragment file pointer). Then, in the fragment file, the exact location of the key-value pair data is found through the local hash table. The whole process only requires two memory searches and one disk I / O (O(1) search complexity).

[0072] The convenience of updating is mainly reflected in that when updating data, only a new segment file needs to be created to store the updated key-value pairs. The old, unmodified segment files remain unchanged. This avoids the rewriting of a large amount of data during the LSM tree compaction process (reducing write amplification).

[0073] 2. Super index: perfect hash (CP Hash).

[0074] Problems with regular indexes: Regular hash indexes may encounter hash conflicts: different keys are mapped to the same index position. Resolving conflicts requires additional overhead.

[0075] Super index of this application: Super index uses "perfect hashing" technology to ensure that each key has a unique index position, completely avoiding hash conflicts. CP Hash algorithm: This algorithm minimizes the space occupied by the index itself while ensuring no conflicts, and the speed of building the index is also very fast.

[0076] Dense and Sparse Buckets: CP Hash divides hash buckets into "dense buckets" and "sparse buckets", similar to the PTHash algorithm (fast perfect hashing algorithm based on bucket grouping) to balance performance and space occupancy.

[0077] Parameter Reuse: When CPHash constructs a new hash bucket, it tries to reuse existing hash parameters to reduce the space required to store the hash function.

[0078] SIMD Parallelization: CPHash uses SIMD (Single Instruction Multiple Data) instructions (such as Intel AVX2) to parallelize the calculation of key hash values ​​and index positions, which increases the speed of building indexes.

[0079] Segmentation: For very large datasets, CPHash divides the fragment files into multiple segments. Keys are first mapped to segments, and then perfect hash indexes are built within the segments. The construction process is done in parallel between segments.

[0080] 3. Smart Partitioning: Hash Range.

[0081] More detailed management: Compass DB not only divides data into fragment files, but also organizes fragment files into different ranges based on the hash value of the key ("search key").

[0082] Function of Hash Range: Search Key: Calculate a 32-bit hash value for each key, called the "search key".

[0083] Partition by search key: According to the key's "search key", the key-value pairs are assigned to the ranges of different fragment files. This can form multiple independent "subtrees".

[0084] Faster compression (Compaction): When performing a compression operation, only a few relevant ranges of fragment files need to be processed instead of scanning the entire data set, which improves compression efficiency.

[0085] 4. Piece File Structure.

[0086] Piece File: Each piece file is an independent file.

[0087] Read-Only: Once a segment file is written, it becomes immutable (read-only).

[0088] Segments and Blocks: Each fragment file is divided into multiple segments, and each segment contains multiple blocks. Blocks are the basic unit of data reading. This allows for more efficient management and reading of data.

[0089] 5. Data compaction (Compaction).

[0090] Smarter sorting: Separation of old and new data: When sorting, old data is located in one TPH (base TPH), and new or updated data is located in another TPH (delta TPH). Only necessary data is moved: only the key-value pairs that have changed are moved to the new fragment file, and the unchanged key-value pairs remain in the original fragment file.

[0091] Automatic cleanup (garbage collection): If all key-value pairs in an old segment file are obsolete, the segment file will be automatically deleted (garbage collection).

[0092] In summary, the following key technologies are used to achieve the goals of fast search, efficient update and space saving: Two-layer perfect hash table (TPH): provides O(1) complexity search and efficient update mechanism.

[0093] CPHash algorithm: Generates efficient perfect hash index.

[0094] Hash Range: It realizes intelligent partitioning of data and improves compression efficiency.

[0095] Piece File Structure: Makes data storage, management and reading more efficient.

[0096] Intelligent compression method: reduces unnecessary data movement.

[0097] These technologies enable this application to have higher performance advantages than traditional LSM tree-based databases when processing large amounts of data.

[0098] In some embodiments, the software instruction reorganization and hardware partial reconfiguration module 50 is used to perform software instruction reorganization and hardware partial reconfiguration when a fault occurs.

[0099] See also Figure 6 The software instruction reorganization and hardware partial reconfiguration module 50 includes: a software instruction reorganization unit 51 and a hardware partial reconfiguration unit 52 .

[0100] The software instruction reorganization unit 51 is used to replace the affected instructions with other available ALU instructions or instruction sequences through software instruction reorganization when an arithmetic logic unit component failure is detected. For example, when an arithmetic logic unit (ALU) component failure is detected, the affected instructions are replaced with other available ALU instructions or instruction sequences through software instruction reorganization to achieve fault recovery. For example, if the multiplier is damaged, shift and addition instructions can be used to simulate multiplication operations. Instruction reorganization can dynamically adjust the code to avoid using faulty hardware units.

[0101] The hardware partial reconfiguration unit 52 is used to replace the damaged processor component to an undamaged area on the chip through dynamic partial reconfiguration when the hardware fails. For example, when the hardware failure is extensive or serious, the damaged processor component is replaced to an undamaged area on the chip through dynamic partial reconfiguration (PR) to ensure the continuous operation of the system. Partial reconfiguration allows a part of the FPGA logic to be dynamically modified while the system is running without restarting the entire system. The partial reconfiguration function of the FPGA is used to achieve replacement and rerouting of the faulty unit.

[0102] In some embodiments, the multi-tenant support module 60 is used to implement multi-tenant isolation.

[0103] The multi-tenant support module 60 includes a namespace isolation unit, which is used to implement multi-tenant isolation on the block interface and the key-value interface respectively by using the namespace technology of the NVMe standard. Each tenant can have an independent namespace, thereby achieving data isolation and secure access.

[0104] The technical solution proposed in this application aims to achieve the following technical effects: Data reliability is significantly improved: Through multi-level ECC architecture, AI-assisted error management and distributed verification mechanism, data reliability is improved to more than 99.999%, significantly reducing the risk of data corruption.

[0105] Greatly improved error recovery success rate: Through intelligent recovery strategies and partial reconfiguration, the error recovery success rate is increased by more than 40%, ensuring that the system can recover quickly when an error occurs.

[0106] Verification overhead is significantly reduced: Through technologies such as adaptive error correction and incremental verification, verification overhead is reduced by more than 30%, reducing resource consumption and performance loss.

[0107] The silent error detection rate is significantly improved: Through AI-assisted error management, the silent error detection rate is increased by more than 50%, and potential data corruption is discovered and handled in a timely manner.

[0108] Significantly improved system availability: Through partial hardware reconfiguration and software instruction reorganization, system availability is increased by more than 35%, reducing system downtime caused by errors.

[0109] Data recovery time is significantly shortened: Through the fast recovery mechanism and incremental synchronization technology, data recovery time is shortened by more than 60%, reducing the impact of failures on system operations.

[0110] Significantly reduced write latency: The write acceleration of dual-interface SSDs significantly reduces write pauses caused by compaction operations, thereby maintaining the high throughput of LSM-KVS. Under write-intensive workloads, throughput can be increased by more than 17%.

[0111] Enhanced adaptability: Through deep learning and adaptive decoding, the system can dynamically adjust parameters according to different channel conditions and load conditions to achieve better performance and reliability.

[0112] Improved resource utilization: Through technologies such as hardware partial reconfiguration and software instruction reorganization, the available resources can be maximized to ensure the continuous operation of the system when hardware failure occurs.

[0113] Overall performance improvement: Under various workloads, including write-intensive and mixed read-write workloads, system throughput and latency are significantly optimized, and CPU efficiency is also improved. For example, under write-intensive workloads, the throughput is increased by 17% compared to ADOC, and the CPU efficiency is also significantly improved.

[0114] Multi-tenant isolation and resource management: Through NVMe namespace and dynamic resource allocation, secure isolation and fair resource allocation in a multi-tenant environment are achieved.

[0115] The innovation of this application lies in the organic integration of multiple independent technical solutions into a whole. It is not a simple superposition of technologies, but through in-depth analysis, the various technical solutions are coordinated to form an efficient closed-loop system, which is non-obvious: Adaptive neural decoding and TDC sensor collaboration: Adaptive neural decoding works with TDC sensors to achieve more accurate error detection and more efficient error recovery. The TDC sensor monitors the chip's operating status in real time. Once an abnormality is detected, adaptive neural decoding can select the optimal decoding strategy based on the detected error pattern and channel conditions, thereby quickly and accurately recovering data. This synergy can improve the overall reliability and response speed of the system.

[0116] The combination of dual-interface SSD write acceleration, adaptive error correction and hardware fault recovery: The combination of dual-interface SSD write acceleration, adaptive error correction and hardware fault recovery constitutes a more stable and reliable storage system. The dual-interface SSD acts as a temporary cache, avoiding the performance degradation caused by write pauses in traditional methods.

[0117] Adaptive error correction can reduce the risk of data errors caused by temporary write cache, and the hardware fault recovery mechanism ensures the continuous operation of the system in the event of hardware failure. This combination allows the system to maintain efficient operation under various abnormal conditions.

[0118] Overall synergy: The technical modules work together. For example, when the TDC sensor detects a fault, it not only triggers partial hardware reconfiguration, but also notifies the adaptive ECC to adjust the decoding strategy. The dual-interface SSD also adjusts the rollback operation according to the system load.

[0119] This overall synergy forms an efficient error management and recovery system, allowing the system to respond dynamically to various challenges rather than each module working in isolation.

[0120] The enhanced eMMC / SSD data integrity protection framework 100 proposed in this application effectively solves the challenges faced by traditional eMMC / SSD in data integrity protection by integrating multi-level adaptive ECC architecture, AI-assisted error management, distributed verification mechanism, dual-interface SSD write acceleration, software instruction reorganization, hardware partial reconfiguration, and multi-tenant support. It provides a more reliable, efficient, and secure solution for new storage devices. This solution not only improves data reliability, but also optimizes system performance, provides strong support for high-performance storage applications, and has strong commercial value. The innovative methods adopted in this application and the synergy of multiple technical solutions make it significantly non-obvious.

[0121] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only illustrative, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0122] If the integrated units in the above other embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0123] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An enhanced eMMC / SSD data integrity protection framework, characterized in that: include: A multi-level adaptive ECC architecture module for using multiple ECC algorithms to perform at least one of dynamic protection operation, adaptive error correction operation, incremental check operation and variable bit rate ECC operation; AI-assisted error management module for real-time error feature analysis of storage devices, error propagation prediction, and selection of intelligent recovery strategies, as well as real-time monitoring of chip voltage fluctuations and signal delay changes using time-to-digital converter sensors; Distributed verification mechanism module, used for cross-device verification, building multiple protection layers, optimizing verification performance, and using incremental synchronization technology for data recovery; The dual-interface SSD write acceleration module uses a hybrid interface SSD architecture for dynamic I / O redirection, device-side LSM write cache, multi-level cache, host-device collaborative consistency, and load-based rollback adjustment; A software instruction reorganization and hardware partial reconfiguration module, used for performing software instruction reorganization and hardware partial reconfiguration in case of failure; Multi-tenant support module, used to implement multi-tenant isolation.

2. The enhanced eMMC / SSD data integrity protection framework according to claim 1, characterized in that: The multi-level adaptive ECC architecture module includes: Dynamic protection unit, used to dynamically adjust the ECC protection strength according to the importance of the data; Adaptive error correction unit, used to automatically select the most appropriate error correction strategy based on real-time error conditions and channel conditions; An incremental verification unit, used to verify only the modified data blocks; The variable rate ECC unit is used to dynamically adjust the ECC rate according to the real-time channel conditions and data importance.

3. The enhanced eMMC / SSD data integrity protection framework according to claim 1, characterized in that: The AI-assisted error management module includes: An error feature analysis unit, used to analyze the errors generated by the storage device in real time and extract features; Error propagation prediction unit, used to predict potential error propagation paths using AI models; Intelligent recovery strategy unit, used to intelligently select the best recovery strategy based on the error type and prediction result; The time-to-digital converter sensor unit is used to monitor the voltage fluctuation and signal delay change of the chip in real time using a TDC sensor.

4. The enhanced eMMC / SSD data integrity protection framework according to claim 1, characterized in that: The distributed verification mechanism module includes: A cross-device verification unit, used for performing data verification between multiple storage devices; Multiple protection layer unit, used to establish a multi-layer data protection mechanism; A verification performance optimization unit, used to optimize performance using a parallel verification algorithm and a hardware accelerator; The fast recovery unit is used to synchronize only the changed data using the incremental synchronization technology.

5. The enhanced eMMC / SSD data integrity protection framework according to claim 1, characterized in that: The write acceleration module based on the dual-interface SSD includes: Dynamic I / O redirection unit, which is used to take advantage of the characteristics of dual-interface SSDs and redirect write requests to the KV interface on the device side as a temporary write buffer when a write pause occurs in the host-side LSM-KVS; The device-side LSM write cache unit is used to build a write buffer using the LSM tree structure on the device side, and implement a range scan mechanism through an iterator to synchronize cached data back to the host-side LSM; A multi-level cache unit is used to adopt a multi-level cache strategy in the device-side LSM write cache; The host-device collaborative consistency unit is used to ensure the consistency of data between the host and device through a range scan rollback mechanism; The rollback unit is used to dynamically adjust the rollback strategy according to the load type.

6. The enhanced eMMC / SSD data integrity protection framework according to claim 5, characterized in that: The host-side LSM-KVS adopts a two-layer perfect hash table, perfect hash technology, intelligent partitioning technology, fragment file structure and data sorting technology.

7. The enhanced eMMC / SSD data integrity protection framework according to claim 6, characterized in that: The first layer of the two-layer perfect hash table is configured with a fragment file, and the second layer is configured with a local hash table; wherein each of the fragment files is used to store a portion of key-value pair data, and each of the fragment files contains a local hash table.

8. The enhanced eMMC / SSD data integrity protection framework according to claim 6, characterized in that: The perfect hashing technology divides hash buckets into dense buckets and sparse buckets. When constructing new hash buckets, existing hash parameters are reused, and SIMD instructions are used to calculate the hash value and index position of the key in parallel.

9. The enhanced eMMC / SSD data integrity protection framework according to claim 1, characterized in that: The software instruction reorganization and hardware partial reconfiguration module includes: A software instruction reorganization unit, for replacing affected instructions with other available ALU instructions or instruction sequences through software instruction reorganization when an arithmetic logic unit component failure is detected; The hardware partial reconfiguration unit is used to replace the damaged processor component to an undamaged area on the chip through dynamic partial reconfiguration when hardware fails.

10. The enhanced eMMC / SSD data integrity protection framework according to claim 1, characterized in that: The multi-tenant support module includes: The namespace isolation unit is used to use the namespace technology of the NVMe standard to implement multi-tenant isolation on the block interface and the key-value interface respectively.

Images