Enhanced emmc / ssd data integrity protection framework

Abstract

The application discloses an enhanced eMMC / SSD data integrity protection framework, comprising: a multi-level adaptive ECC architecture module for at least one of dynamic protection operation, adaptive error correction operation, incremental check operation and variable code rate ECC operation by using multiple ECC algorithms; an AI assisted error management module for real-time error feature analysis and error propagation prediction of a storage device; a distributed verification mechanism module for cross-device check, construction of multiple protection layers and check performance optimization; a write acceleration module based on a dual-interface SSD for dynamic I / O redirection, device-side log-structured merge tree write cache, multi-level cache and host device cooperative consistency by using a hybrid interface SSD architecture; a software instruction reorganization and hardware partial reconfiguration module for software instruction reorganization and hardware partial reconfiguration in case of failure; and a multi-tenant support module for realizing multi-tenant isolation. In the above manner, data reliability can be improved, and system performance can be optimized.

Description

Technical Field

[0001] The present application relates to the field of eMMC / SSD technology, and in particular to an enhanced eMMC / SSD data integrity protection framework. Background Art

[0002] Currently, eMMC / SSD storage devices face multiple challenges in data integrity. Traditional error correction code (ECC) technology has limitations when dealing with complex error patterns, especially in the following aspects:

[0003] Complex Error Patterns: Traditional ECC technology struggles to effectively detect and recover from complex error patterns caused by a variety of factors, such as process variations, temperature changes, radiation, and electromagnetic interference. These error patterns include random errors, sudden errors, and permanent errors caused by hardware aging or damage.

[0004] Performance overhead: Performing data integrity checks (such as ECC and data comparison) introduces additional performance overhead, reducing the overall efficiency of the storage system. Frequent verification operations consume CPU resources and increase I / O latency of storage devices.

[0005] Error recovery capability: Existing technologies have limited error recovery capabilities, making it difficult to ensure data reliability under various abnormal conditions. For example, traditional ECC may not be able to recover errors that exceed its error correction capability.

[0006] Silent data corruption: Silent data corruption is difficult to prevent and detect. This refers to errors that occur when data is not being used. These errors are often difficult to detect and may result in data loss or corruption.

[0007] New storage requirements: New storage devices (such as high-density eMMC / SSD and STT-MRAM) place higher demands on data protection and performance. For example, STT-MRAM is significantly affected by temperature fluctuations and requires adaptive error correction strategies.

[0008] Hardware failures: Factors such as space radiation and malicious attacks can cause hardware failures, requiring a mechanism that can quickly detect, recover, and ensure continuous system operation. Traditional fault-tolerance methods, such as triple modular redundancy (TMR), can handle hardware failures but incur significant overhead.

[0009] Write latency: Key-value stores based on log-structured merge trees experience write latency when performing compaction operations (reading data from disk, reorganizing the data structure and indexes, and then writing it back to disk, which can exponentially improve read performance). This impacts system performance. Traditional I / O scheduling methods or multi-threaded compaction can alleviate this problem, but they cannot eliminate it. Summary of the Invention

[0010] The enhanced eMMC / SSD data integrity protection framework provided in this application can improve data reliability and optimize system performance.

[0011] In the first aspect, the present application provides an enhanced eMMC / SSD data integrity protection framework, including: a multi-level adaptive ECC architecture module, which is used to use multiple ECC algorithms to perform at least one of dynamic protection operations, adaptive error correction operations, incremental verification operations and variable bit rate ECC operations; an AI-assisted error management module, which is used to perform real-time error feature analysis, error propagation prediction, and intelligent recovery strategy selection on storage devices, and use time-to-digital converter sensors to monitor the voltage fluctuations and signal delay changes of the chip in real time; a distributed verification mechanism module, which is used to perform cross-device verification, build multiple protection layers, optimize verification performance, and use incremental synchronization technology for data recovery; a write acceleration module based on dual-interface SSD, which uses a hybrid interface SSD architecture for dynamic I / O redirection, device-side log structure merge tree write cache, multi-level cache, host device collaborative consistency, and load-adjusted rollback; a software instruction reorganization and hardware partial reconfiguration module, which is used to perform software instruction reorganization and hardware partial reconfiguration in the event of a failure; and a multi-tenant support module for achieving multi-tenant isolation.

[0012] Among them, the multi-level adaptive ECC architecture module includes: a dynamic protection unit, which is used to dynamically adjust the ECC protection strength according to the importance of the data; an adaptive error correction unit, which is used to automatically select the most appropriate error correction strategy according to the real-time error situation and channel conditions; an incremental check unit, which is used to check only the modified data blocks; and a variable code rate ECC unit, which is used to dynamically adjust the ECC code rate according to the real-time channel conditions and data importance.

[0013] Among them, the AI-assisted error management module includes: an error feature analysis unit, which is used to perform real-time analysis of errors generated by storage devices and extract features; an error propagation prediction unit, which is used to use AI models to predict potential error propagation paths; an intelligent recovery strategy unit, which is used to intelligently select the best recovery strategy based on the error type and prediction results; a time-to-digital converter sensor unit, which uses a TDC sensor to monitor the chip's voltage fluctuations and signal delay changes in real time.

[0014] Among them, the distributed verification mechanism module includes: a cross-device verification unit, which is used to perform data verification between multiple storage devices; a multiple protection layer unit, which is used to establish a multi-layer data protection mechanism; a verification performance optimization unit, which is used to use parallel verification algorithms and hardware accelerators for performance optimization; and a fast recovery unit, which is used to use incremental synchronization technology to synchronize only the data that has changed.

[0015] Among them, the write acceleration module based on dual-interface SSD includes: a dynamic I / O redirection unit, which is used to utilize the characteristics of dual-interface SSD to redirect the write request to the key-value interface on the device side as a temporary write buffer when a write pause occurs in the host-side log structure merge tree-key value storage; a device-side log structure merge tree write cache unit, which is used to use the log structure merge tree structure on the device side to build a write buffer, and implement a range scanning mechanism through an iterator to synchronize the cached data back to the host-side log structure merge tree; a multi-level cache unit, which is used to adopt a multi-level cache strategy in the device-side log structure merge tree write cache; a host-device collaborative consistency unit, which is used to ensure the consistency of host-side and device-side data through a range scanning rollback mechanism; and a rollback unit, which is used to dynamically adjust the rollback strategy according to the load type.

[0016] Among them, the host-side log structure merge tree-key value storage adopts two-layer perfect hash table, perfect hash technology, intelligent partitioning technology, fragment file structure and data organization technology.

[0017] The first layer of the two-layer perfect hash table is configured with a fragment file, and the second layer is configured with a local hash table; each fragment file is used to store a portion of key-value pair data, and each fragment file contains a local hash table.

[0018] Among them, the perfect hashing technology divides the hash bucket into dense buckets and sparse buckets. When constructing a new hash bucket, it reuses the existing hash parameters and uses SIMD instructions (parallel calculation of the key hash value and index position).

[0019] Among them, the software instruction reorganization and hardware partial reconfiguration module includes: a software instruction reorganization unit, which is used to replace the affected instructions with other available ALU instructions or instruction sequences through software instruction reorganization when an arithmetic logic unit component failure is detected; a hardware partial reconfiguration unit, which is used to replace the damaged processor component with an undamaged area on the chip through dynamic partial reconfiguration when the hardware fails.

[0020] Among them, the multi-tenant support module includes: a namespace isolation unit, which is used to use the NVMe standard namespace technology to implement multi-tenant isolation on the block interface and key-value interface respectively.

[0021] The beneficial effects of the present application are: different from the existing technology, the enhanced eMMC / SSD data integrity protection framework provided by the present application includes: a multi-level adaptive ECC architecture module, which is used to use multiple ECC algorithms to perform at least one of dynamic protection operations, adaptive error correction operations, incremental verification operations and variable bit rate ECC operations; an AI-assisted error management module, which is used to perform real-time error feature analysis, error propagation prediction, and intelligent recovery strategy selection on storage devices, and use time-to-digital converter sensors to monitor chip voltage fluctuations and signal delay changes in real time; a distributed verification mechanism module, which is used to perform cross-device verification, build multiple protection layers, optimize verification performance, and use incremental synchronization technology for data recovery; a write acceleration module based on dual-interface SSD, which uses a hybrid interface SSD architecture for dynamic I / O redirection, device-side log structure merge tree write cache, multi-level cache, host device collaborative consistency, and load-based rollback adjustment; a software instruction reorganization and hardware partial reconfiguration module, which is used to perform software instruction reorganization and hardware partial reconfiguration in the event of a failure; and a multi-tenant support module for achieving multi-tenant isolation. Through the above method, data reliability can be improved and system performance can be optimized, effectively solving the challenges faced by traditional eMMC / SSD in data integrity protection, and providing a more reliable, efficient and secure solution for new storage devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without inventive efforts. Among them:

[0023] Figure 1 This is a structural diagram of an embodiment of an enhanced eMMC / SSD data integrity protection framework provided by the present application;

[0024] Figure 2 This is a schematic diagram of the structure of an embodiment of a multi-level adaptive ECC architecture module provided by the present application;

[0025] Figure 3 This is a structural diagram of an embodiment of the AI-assisted error management module provided by this application;

[0026] Figure 4 This is a schematic diagram of the structure of an embodiment of a distributed authentication mechanism module provided by this application;

[0027] Figure 5 This is a structural diagram of an embodiment of a write acceleration module based on a dual-interface SSD provided by the present application;

[0028] Figure 6 It is a structural diagram of an embodiment of a software instruction reorganization and hardware partial reconfiguration module provided by this application. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It will be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some, rather than all, structures related to the present application are shown in the drawings. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0030] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0031] See Figure 1 , Figure 1 This is a schematic diagram of an embodiment of an enhanced eMMC / SSD data integrity protection framework provided herein. The enhanced eMMC / SSD data integrity protection framework 100 includes a multi-level adaptive ECC architecture module 10, an AI-assisted error management module 20, a distributed verification mechanism module 30, a dual-interface SSD-based write acceleration module 40, a software instruction reassembly and hardware partial reconfiguration module 50, and a multi-tenant support module 60.

[0032] In some embodiments, the multi-level adaptive ECC architecture module 10 is configured to employ multiple ECC algorithms to perform at least one of dynamic protection operations, adaptive error correction operations, incremental check operations, and variable rate ECC operations.

[0033] See Figure 2 The multi-level adaptive ECC architecture module 10 includes: a dynamic protection unit 11, an adaptive error correction unit 12, an incremental check unit 13 and a variable code rate ECC unit 14.

[0034] In some embodiments, the multi-level adaptive ECC architecture module 10 combines different ECC algorithms to adapt to different error types and frequencies. For example, Hamming codes or BCH codes are used for random errors, and Reed-Solomon codes or LDPC codes are used for burst errors, thereby effectively covering multiple error types.

[0035] The dynamic protection unit 11 is used to dynamically adjust the ECC protection strength based on the importance of the data. For important data, a high-strength ECC algorithm (such as LDPC code or BCH code) is used to provide stronger error correction capabilities and a larger check codeword length. For less important data, a low-strength ECC algorithm (such as Hamming code) is used to reduce checksum overhead and shorten latency.

[0036] The adaptive error correction unit 12 is used to automatically select the most appropriate error correction strategy based on the real-time error situation and channel conditions, and adopts adaptive decoding technology based on deep learning and neural network bit flipping (NBF) decoding technology.

[0037] Deep learning-based adaptive decoding technology primarily utilizes deep learning techniques to train a neural network (NN) decoder. This decoder dynamically adjusts decoding complexity based on channel conditions (e.g., bit error rate (BER)) and selects an appropriate decoder and number of NN layers. For example, when channel conditions are favorable, a lower-complexity decoder and fewer NN layers are selected to reduce latency and energy consumption. When channel conditions are poor, a higher-complexity decoder and more NN layers are selected to ensure error correction performance.

[0038] The neural network bit flipping (NBF) decoding technology mainly uses the NBF algorithm to implement the bit flipping algorithm. It uses trellis representation and can share the same structure with the neural belief propagation (NBP) and neural offset minimum sum (NOMS) algorithms. It can be flexibly switched and has the characteristics of low latency and low power consumption.

[0039] The incremental verification unit 13 is used to verify only the modified data blocks. It supports incremental verification and only verifies the modified data blocks to reduce verification overhead. It uses a hash algorithm or timestamp to track data block modifications and only verifies the data blocks that have changed.

[0040] The variable rate ECC unit 14 is used to dynamically adjust the ECC rate based on real-time channel conditions and data importance, thereby optimizing performance and protection in different scenarios. The rate adjustment can be based on the prediction and real-time detection of error rates.

[0041] In some embodiments, the AI-assisted error management module 20 is used to perform real-time error feature analysis, error propagation prediction, and intelligent recovery strategy selection on the storage device, and uses a time-to-digital converter sensor to monitor the voltage fluctuation and signal delay changes of the chip in real time.

[0042] In some embodiments, the AI-assisted error management module 20 includes an error pattern database. For example, a database containing various known error patterns can be established, including resistance shifts caused by temperature variations and process variations, single event effects (SEEs) caused by radiation, and failure modes caused by hardware aging. The error patterns in the database include not only the type of error, but also the frequency, intensity, and scope of the error.

[0043] See Figure 3 The AI-assisted error management module 20 includes: an error feature analysis unit 21, an error propagation prediction unit 22, an intelligent recovery strategy unit 23 and a time-to-digital converter sensor unit 24.

[0044] The error signature analysis unit 21 is used to perform real-time analysis and feature extraction of errors generated by the storage device. For example, this allows for timely detection and handling of abnormalities. Machine learning algorithms are used to analyze error signatures and classify errors, thereby selecting the optimal recovery strategy. For example, ECC correction may be used for soft errors, while partial reconfiguration may be employed for hard errors.

[0045] The error propagation prediction unit 22 is used to predict potential error propagation paths using AI models. For example, by analyzing the networks and patterns of error propagation, it predicts the areas that the error may affect and allows for proactive preventive measures. This prediction and analysis is performed using a graph neural network.

[0046] The intelligent recovery strategy unit 23 is used to intelligently select the best recovery strategy based on the error type and prediction result. For example, for soft errors, ECC error correction and data rereading are used; for hard errors, partial reconfiguration or software instruction reassembly are used.

[0047] The time-to-digital converter sensor unit 24 uses a TDC sensor to monitor chip voltage fluctuations and signal delay changes in real time, thereby detecting potential faults and attacks in advance. The TDC sensor detects abnormal voltage fluctuations by measuring the propagation delay of signals within the chip, which are often precursors to faults or attacks.

[0048] In some embodiments, the distributed verification mechanism module 30 is used to perform cross-device verification, build multiple protection layers, optimize verification performance, and use incremental synchronization technology to perform data recovery.

[0049] See Figure 4The distributed verification mechanism module 30 includes: a cross-device verification unit 31, a multiple protection layer unit 32, a verification performance optimization unit 33 and a fast recovery unit 34.

[0050] The cross-device verification unit 31 is used to perform data verification across multiple storage devices. Performing data verification across multiple storage devices can improve reliability, especially in large-scale storage systems. A distributed hash table (DHT) or other distributed data structure is used to store verification information and perform fast verification operations.

[0051] The multi-layer protection unit 32 is used to establish a multi-layer data protection mechanism. This multi-layer data protection mechanism ensures data integrity even if some devices fail. For example, RAID technology (such as RAID 5 or RAID 6) is used for data backup and redundancy, and erasure coding technology is used to improve data storage efficiency.

[0052] The verification performance optimization unit 33 is used to optimize performance using parallel verification algorithms and hardware accelerators. For example, through algorithm and hardware optimization, the performance overhead of distributed verification can be reduced. For example, parallel verification algorithms and hardware accelerators, such as FPGAs or ASICs, can be used to implement hardware acceleration, thereby reducing CPU resource usage.

[0053] The fast recovery unit 34 is used to synchronize only the changed data using incremental synchronization technology. This allows for rapid data recovery, shortening system downtime caused by errors. By using incremental synchronization technology, only the changed data is synchronized, thus reducing data recovery time.

[0054] In some embodiments, the dual-interface SSD-based write acceleration module 40 uses a hybrid interface SSD architecture for dynamic I / O redirection, device-side log structure merge tree write cache, multi-level cache, host device collaborative consistency, and load-adjusted rollback.

[0055] The dual-interface SSD-based write acceleration module 40 uses a hybrid interface SSD architecture, dividing the SSD's logical NAND flash address space into two areas: one for the traditional block interface, managed by the host's log-structured merge tree-key-value storage; and the other for the key-value interface, which serves as a temporary write buffer. This architecture allows for simultaneous support of block and key-value access on the same device.

[0056] See Figure 5 The write acceleration module 40 based on the dual-interface SSD includes: a dynamic I / O redirection unit 41, a device-side log structure merge tree write cache unit 42, a multi-level cache unit 43, a host device collaborative consistency unit 44 and a rollback unit 45.

[0057] The dynamic I / O redirection unit 41 is used to utilize the characteristics of the dual-interface SSD to redirect write requests to the key-value interface on the device side as a temporary write buffer when a write pause occurs in the host-side log-structured merge tree-key-value storage. Utilizing the characteristics of the dual-interface SSD, when a write pause occurs in the host-side log-structured merge tree-key-value storage (for example, during compaction), the write request is redirected to the key-value interface on the device side as a temporary write buffer, thereby avoiding the write pause. When the host-side write operation resumes, the data in the temporary write buffer can be synchronized back to the host side.

[0058] The device-side log-structured merge tree write cache unit 42 is used to construct a write buffer using a log-structured merge tree structure on the device side, and implement a range scan mechanism through an iterator to synchronize the cached data back to the host-side log-structured merge tree. The device-side log-structured merge tree structure is used to construct a write buffer, and implement a range scan mechanism through an iterator to quickly synchronize the cached data back to the host-side log-structured merge tree. The device-side log-structured merge tree can be a lightweight implementation that reduces resource usage and focuses on fast write and scan operations.

[0059] The multi-level cache unit 43 is used to adopt a multi-level cache strategy in the device-side log structure merge tree write cache.

[0060] To further improve performance, a multi-level caching strategy can be adopted in the device-side log-structured merge-tree write cache, such as using DRAM as a faster cache layer to reduce access to NAND flash memory.

[0061] The host-device collaborative consistency unit 44 is used to ensure the consistency of data between the host and device through a range scan rollback mechanism. The range scan rollback mechanism ensures the consistency of data between the host and device and optimizes the flow of data between the two interfaces. The rollback mechanism includes the following steps:

[0062] Detecting write stalls: A host-side write stall detector monitors the state of the log-structured merge tree key-value store and triggers I / O redirection when a write stall is detected.

[0063] Write key-value interface: Write write requests to the key-value interface on the device side. These write requests will be cached in the log structure merge tree structure on the device side.

[0064] Range Scan: When the host-side write operation resumes, the iterator is used to perform a range scan on the log-structured merge tree on the device side to read the cached data back to the host side.

[0065] Data merging: Merge the cached data on the device into the log structure merge tree on the host. During the data merging process, deduplication and sorting operations are required.

[0066] The rollback unit 45 is used to dynamically adjust the rollback strategy according to the load type.

[0067] For read-intensive workloads, an active rollback strategy is used to synchronize data back to the host in a timely manner to improve read performance. For write-intensive workloads, a delayed rollback strategy is used to reduce the frequency of rollback operations to improve write performance.

[0068] The write acceleration module 40 based on the dual-interface SSD further includes: a dynamic namespace allocation unit, which dynamically divides and manages the address space of the block interface and the key-value interface through NVMe namespace technology, and supports isolation and access control of different tenants.

[0069] In some embodiments, the host-side log structure merge tree-key value storage adopts a two-layer perfect hash table, perfect hash technology, smart partitioning technology, fragment file structure and data sorting technology.

[0070] The first layer of the two-layer perfect hash table is configured with a fragment file, and the second layer is configured with a local hash table; each fragment file is used to store a portion of key-value pair data, and each fragment file contains a local hash table.

[0071] Among them, the perfect hashing technology divides the hash bucket into dense buckets and sparse buckets. When constructing a new hash bucket, it reuses the existing hash parameters and uses SIMD instructions (parallel calculation of the key hash value and index position).

[0072] The host-side log structure merge tree-key value storage management mainly includes the following:

[0073] 1. Two-layer perfect hash table (TPH).

[0074] Problems with Traditional Databases: In traditional databases, data is typically stored in SSTable (Sorted String Table) files. Reading data may require searching across multiple SSTables, resulting in high read latency. When updating data, the log-structured tree-merge compaction process may require rewriting large amounts of data, leading to write amplification.

[0075] This application's log-structured merge tree-key-value storage management innovation uses a special data structure called a "two-level perfect hash table" (TPH).

[0076] First layer: Piece Files: The data of the entire database is divided into multiple piece files. Each piece file is similar to the SS Table in a traditional database, storing a part of the key-value pair data.

[0077] Second layer: Local Hash Table: Each fragment file contains a local hash table. This hash table maps the key to the specific location (offset) of the key-value pair data in the fragment file.

[0078] Global Perfect Hash Table: There is a global perfect hash table. This hash table maps the hash value of a key to an index slot. The index slot stores the key's signature and a pointer to the segment file where the key is located.

[0079] The advantage is that it can be quickly searched and updated easily.

[0080] Fast lookups are primarily achieved by first calculating an index slot using a global perfect hash table during key lookups. The information in the slot (key signature and fragment file pointer) is then used to directly locate the fragment file. Within the fragment file, the local hash table is then used to find the precise location of the key-value pair. This entire process requires only two memory lookups and one disk I / O (O(1) lookup complexity).

[0081] The ease of updating is primarily reflected in the fact that when updating data, only a new segment file needs to be created to store the updated key-value pairs. The old, unmodified segment files remain unchanged. This avoids the need to rewrite large amounts of data during the log-structured merge tree compaction process (reducing write amplification).

[0082] 2. Super index: perfect hash (CP Hash).

[0083] Problems with regular indexes: Regular hash indexes can experience hash conflicts: different keys are mapped to the same index position. Resolving conflicts requires additional overhead.

[0084] This application's Super Index uses "perfect hashing" technology to ensure that each key has a unique index location, completely avoiding hash conflicts. CP Hash algorithm: This algorithm minimizes the space occupied by the index itself while ensuring no conflicts, and it also builds the index very quickly.

[0085] Dense and Sparse Buckets: CP Hash divides hash buckets into "dense buckets" and "sparse buckets", similar to the PTHash algorithm (fast perfect hashing algorithm based on bucket grouping) to balance performance and space occupancy.

[0086] Parameter Reuse: When constructing a new hash bucket, CPHash will try to reuse existing hash parameters to reduce the space required to store the hash function.

[0087] SIMD Parallelization: CPHash uses SIMD (Single Instruction Multiple Data) instructions (such as Intel AVX2) to parallelize the calculation of key hash values ​​and index positions, which improves the speed of index construction.

[0088] Segmentation: For very large datasets, CPHash divides the fragment file into multiple segments. Keys are first mapped to segments, and then a perfect hash index is built within the segments. This construction process is performed in parallel across the segments.

[0089] 3. Smart Partitioning: Hash Range.

[0090] More detailed management: Compass DB not only splits data into fragment files, but also organizes fragment files into different ranges based on the hash value of the key ("search key").

[0091] Function of Hash Range: Search Key: Calculate a 32-bit hash value for each key, called the "search key".

[0092] Partitioning by search key: Assign key-value pairs to different fragment file ranges based on the key's "search key". This can form multiple independent "subtrees".

[0093] Faster compression (Compaction): When performing a compression operation, only a few relevant ranges of fragment files need to be processed instead of scanning the entire data set, which improves compression efficiency.

[0094] 4. Piece File Structure

[0095] Piece File: Each piece file is an independent file.

[0096] Read-Only: Once a segment file is written, it becomes immutable (read-only).

[0097] Segments and Blocks: Each fragment file is divided into multiple segments, and each segment contains multiple blocks. Blocks are the basic unit of data reading. This allows for more efficient data management and reading.

[0098] 5. Data compaction (Compaction).

[0099] Smarter organization: Separation of old and new data: During organization, old data is placed in one TPH (base TPH), and new or updated data is placed in another TPH (delta TPH).

[0100] Only necessary data is moved: only the key-value pairs that have changed are moved to the new fragment file, and the unchanged key-value pairs remain in the original fragment file.

[0101] Automatic cleanup (garbage collection): If all key-value pairs in an old segment file are outdated, the segment file will be automatically deleted (garbage collection).

[0102] In summary, the goals of fast search, efficient update, and space saving are achieved through the following key technologies:

[0103] Two-level perfect hash table (TPH): provides O(1) complexity search and efficient update mechanism.

[0104] CPHash algorithm: Generates efficient perfect hash index.

[0105] Hash Range: This enables intelligent data partitioning and improves compression efficiency.

[0106] Piece File Structure: Makes data storage, management and reading more efficient.

[0107] Intelligent compression method: reduces unnecessary data movement.

[0108] These technologies enable this application to have higher performance advantages than traditional log-structured merge tree-based databases when processing large amounts of data.

[0109] In some embodiments, the software instruction reorganization and hardware partial reconfiguration module 50 is used to perform software instruction reorganization and hardware partial reconfiguration when a fault occurs.

[0110] See Figure 6 The software instruction reorganization and hardware partial reconfiguration module 50 includes: a software instruction reorganization unit 51 and a hardware partial reconfiguration unit 52 .

[0111] The software instruction reorganization unit 51 is used to, when an arithmetic logic unit (ALU) component failure is detected, reorganize the software instructions to replace the affected instructions with other available ALU instructions or instruction sequences. For example, when an arithmetic logic unit (ALU) component failure is detected, software instruction reorganization is used to replace the affected instructions with other available ALU instructions or instruction sequences to achieve fault recovery. For example, if a multiplier is damaged, shift and add instructions can be used to simulate multiplication operations. Instruction reorganization can dynamically adjust the code to avoid using the faulty hardware unit.

[0112] The hardware partial reconfiguration unit 52 is used to replace damaged processor components with undamaged areas on the chip through dynamic partial reconfiguration when a hardware failure occurs. For example, when a hardware failure is widespread or severe, dynamic partial reconfiguration (PR) can be used to replace damaged processor components with undamaged areas on the chip, ensuring continued system operation. Partial reconfiguration allows for the dynamic modification of a portion of the FPGA logic during system operation without requiring a full system restart. Leveraging the FPGA's partial reconfiguration capabilities, faulty components can be replaced and rerouted.

[0113] In some embodiments, the multi-tenant support module 60 is used to implement multi-tenant isolation.

[0114] The multi-tenant support module 60 includes a namespace isolation unit that uses the NVMe standard's namespace technology to implement multi-tenant isolation on both the block interface and the key-value interface. Each tenant can have an independent namespace, thereby achieving data isolation and secure access.

[0115] The technical solution proposed in this application aims to achieve the following technical effects:

[0116] Significantly improved data reliability: Through a multi-level ECC architecture, AI-assisted error management, and distributed verification mechanisms, data reliability is increased to over 99.999%, significantly reducing the risk of data corruption.

[0117] Significantly improved error recovery success rate: Through intelligent recovery strategies and partial reconfiguration, the error recovery success rate is increased by more than 40%, ensuring that the system can recover quickly when an error occurs.

[0118] Verification overhead is significantly reduced: Through technologies such as adaptive error correction and incremental verification, verification overhead is reduced by more than 30%, reducing resource consumption and performance loss.

[0119] Significantly improved silent error detection rate: Through AI-assisted error management, the silent error detection rate is increased by more than 50%, allowing potential data corruption to be discovered and addressed in a timely manner.

[0120] Significantly improved system availability: Through partial hardware reconfiguration and software instruction reorganization, system availability is increased by more than 35%, reducing system downtime caused by errors.

[0121] Data recovery time is significantly shortened: Through the rapid recovery mechanism and incremental synchronization technology, data recovery time is shortened by more than 60%, reducing the impact of failures on system operations.

[0122] Significantly Reduced Write Latency: Dual-interface SSD write acceleration significantly reduces write pauses caused by compaction operations, maintaining high throughput for log-structured merge tree key-value storage. Under write-intensive workloads, throughput can be increased by over 17%.

[0123] Enhanced adaptability: Through deep learning and adaptive decoding, the system can dynamically adjust parameters according to different channel conditions and load conditions to achieve better performance and reliability.

[0124] Improved resource utilization: Through technologies such as hardware partial reconfiguration and software instruction reorganization, available resources are maximized to ensure continuous system operation in the event of hardware failure.

[0125] Overall performance improvement: Across various workloads, including write-intensive and mixed read-write workloads, system throughput and latency are significantly optimized, and CPU efficiency is also improved. For example, under write-intensive workloads, throughput is increased by 17% compared to ADOC, and CPU efficiency is also significantly improved.

[0126] Multi-tenant isolation and resource management: NVMe namespaces and dynamic resource allocation enable secure isolation and fair resource allocation in a multi-tenant environment.

[0127] The innovation of this application lies in the organic integration of multiple independent technical solutions into a whole. It is not a simple superposition of technologies, but through in-depth analysis, the various technical solutions are coordinated to form an efficient closed-loop system. It is non-obvious:

[0128] Adaptive neural decoding and TDC sensor synergy: Adaptive neural decoding and TDC sensors work together to achieve more accurate error detection and more efficient error recovery. The TDC sensor monitors the chip's operating status in real time. Once an anomaly is detected, adaptive neural decoding selects the optimal decoding strategy based on the detected error pattern and channel conditions, allowing for rapid and accurate data recovery. This synergy improves overall system reliability and responsiveness.

[0129] Dual-interface SSD write acceleration combined with adaptive error correction and hardware fault recovery: This combination of dual-interface SSD write acceleration, adaptive error correction, and hardware fault recovery creates a more stable and reliable storage system. The dual-interface SSD acts as a temporary cache, avoiding the performance degradation caused by write stalls in traditional methods.

[0130] Adaptive error correction reduces the risk of data errors caused by temporary write caching, while hardware failure recovery ensures continued system operation in the event of a hardware failure. This combination allows the system to maintain efficient operation under various abnormal conditions.

[0131] Overall synergy: The various technical modules work together. For example, when the TDC sensor detects a fault, it not only triggers partial hardware reconfiguration but also notifies the adaptive ECC to adjust the decoding strategy. The dual-interface SSD also adjusts the rollback operation based on the system load.

[0132] This overall synergy forms an efficient error management and recovery system, enabling the system to dynamically respond to various challenges rather than each module working in isolation.

[0133] The enhanced eMMC / SSD data integrity protection framework 100 proposed in this application effectively solves the challenges faced by traditional eMMC / SSD in data integrity protection by integrating technologies such as multi-level adaptive ECC architecture, AI-assisted error management, distributed verification mechanism, dual-interface SSD write acceleration, software instruction reorganization, hardware partial reconfiguration, and multi-tenant support, providing a more reliable, efficient, and secure solution for new storage devices. This solution not only improves data reliability, but also optimizes system performance, provides strong support for high-performance storage applications, and has strong commercial value. The innovative methods adopted in this application and the synergy of multiple technical solutions make it significantly non-obvious.

[0134] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules or units is merely a logical functional division. In actual implementation, other division methods may be used, such as combining or integrating multiple units or components into another system, or ignoring or not implementing certain features.

[0135] If the integrated units in the above other embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0136] The above description is only an implementation method of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An enhanced eMMC / SSD data integrity protection framework, characterized in that: include: Multi-level adaptive ECC architecture module for dynamic protection, adaptive error correction, incremental checksum, and variable bit rate ECC operations using multiple ECC algorithms; An AI-assisted error management module performs real-time error feature analysis, error propagation prediction, and intelligent recovery strategy selection on storage devices. It also uses a time-to-digital converter sensor to monitor chip voltage fluctuations and signal delay changes in real time. A distributed verification mechanism module, used for cross-device verification, building multiple protection layers, optimizing verification performance, and utilizing incremental synchronization technology for data recovery; A dual-interface SSD-based write acceleration module uses a hybrid-interface SSD architecture for dynamic I / O redirection, device-side log-structured merge-tree write caching, multi-level caching, host-device collaborative consistency, and load-adjusted rollback. A software instruction reorganization and hardware partial reconfiguration module, used for performing software instruction reorganization and hardware partial reconfiguration in case of failure; Multi-tenant support module, used to achieve multi-tenant isolation; The multi-level adaptive ECC architecture module includes: Dynamic protection unit, used to dynamically adjust ECC protection strength according to the importance of data; Adaptive error correction unit, which automatically selects the most appropriate error correction strategy based on real-time error conditions and channel conditions; An incremental check unit, used to check only the modified data blocks; Variable-rate ECC unit, used to dynamically adjust the ECC rate based on real-time channel conditions and data importance; The AI-assisted error management module includes: an error feature analysis unit, configured to analyze errors generated by the storage device in real time and extract features; Error propagation prediction unit, used to predict potential error propagation paths using AI models; Intelligent recovery strategy unit, used to intelligently select the best recovery strategy based on error type and prediction results; A time-to-digital converter sensor unit, used to monitor the chip's voltage fluctuations and signal delay changes in real time; The distributed verification mechanism module includes: A cross-device verification unit, used to perform data verification between multiple storage devices; Multiple protection layer unit, used to establish a multi-layer data protection mechanism; Verification performance optimization unit, used to optimize performance using parallel verification algorithms and hardware accelerators; A fast recovery unit is used to synchronize only the changed data using incremental synchronization technology; The dual-interface SSD-based write acceleration module includes: Dynamic I / O redirection unit, which is used to take advantage of the characteristics of dual-interface SSDs and redirect write requests to the key-value interface on the device side as a temporary write buffer when a write pause occurs in the host-side log structure merge tree-key-value store; The device-side log-structured merge-tree write cache unit is used to build a write buffer using the log-structured merge-tree tree structure on the device side, and implement a range scan mechanism through an iterator to synchronize the cached data back to the host-side log-structured merge-tree; A multi-level cache unit for adopting a multi-level cache strategy in the device-side log structure merge tree write cache; The host-device collaborative consistency unit is used to ensure the consistency of data between the host and the device through a range scan rollback mechanism; The rollback unit is used to dynamically adjust the rollback strategy based on the load type.

2. The enhanced eMMC / SSD data integrity protection framework according to claim 1, wherein: The host-side log structure merge tree-key value storage adopts a two-layer perfect hash table, perfect hash technology, intelligent partitioning technology, fragment file structure and data sorting technology.

3. The enhanced eMMC / SSD data integrity protection framework according to claim 2, wherein: The first layer of the two-layer perfect hash table is configured with a segment file, and the second layer is configured with a local hash table; wherein each segment file is used to store a portion of key-value pair data, and each segment file contains a local hash table.

4. The enhanced eMMC / SSD data integrity protection framework according to claim 2, wherein: The perfect hashing technology divides hash buckets into dense buckets and sparse buckets. When constructing a new hash bucket, it reuses existing hash parameters and uses SIMD instructions to calculate the hash value and index position of the key in parallel.

5. The enhanced eMMC / SSD data integrity protection framework according to claim 1, wherein: The software instruction reorganization and hardware partial reconfiguration module includes: A software instruction reorganization unit, configured to replace affected instructions with other available ALU instructions or instruction sequences through software instruction reorganization when an arithmetic logic unit component failure is detected; The hardware partial reconfiguration unit is used to replace the damaged processor component to an undamaged area on the chip through dynamic partial reconfiguration when the hardware fails.

6. The enhanced eMMC / SSD data integrity protection framework according to claim 1, wherein: The multi-tenant support module includes: The namespace isolation unit is used to implement multi-tenant isolation on the block interface and key-value interface using the NVMe standard namespace technology.

Images