Solid state disk data consistency control method

Abstract

The present application relates to the field of solid state storage technology, and in particular to a solid state disk data consistency control method, aiming to solve the problems of poor data consistency, high risk of power failure loss, and difficult balance between write efficiency and consistency of the solid state disk in the scenarios of power failure, parallel writing, and garbage collection. By constructing a "cache-persistent cache" hierarchical architecture, combined with multi-channel writing synchronization, power failure emergency migration, and periodic verification and repair mechanisms, the full-process consistency control is realized: the received data is first stored in the cache, the multi-channel writing task is allocated according to the weighted round robin and the state is synchronized; when the power fails, the standby capacitor is triggered to supply power, and the cache data is migrated to the persistent cache according to FIFO; according to the 1-24 hour configurable period, the SHA-256 hash check is performed on the flash block data, and when it is inconsistent, it is repaired from the persistent cache / backup block. The present application makes the solid state disk data consistency compliance rate ≥ 99.9%, the power failure data loss rate ≤ 0.01%, and the write delay is reduced by 15%-20%, which is suitable for consumer, enterprise and industrial solid state disks.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of solid-state storage, in particular to a solid-state disk data consistency control method. BACKGROUND

[0002] With the rapid development of information technology, solid-state disks (SSDs) have gradually replaced traditional mechanical hard disks (HDDs) and become the mainstream storage medium for computers, servers, mobile terminals, and other devices due to their high-speed reading and writing, low power consumption, and resistance to vibration. However, due to the physical characteristics of NAND flash memory, SSDs face many challenges in data consistency protection, which have become key bottlenecks restricting their application in high-reliability scenarios. The specific problems are as follows:

[0003] First, power failure data loss is a core threat to SSD data consistency. Traditional HDDs use a magnetic head suspension mechanism to complete current data writing with the help of inertia when power fails, with very low risk of data loss. However, SSD data writing relies on charge injection and storage. If power fails during the process of writing data from the cache (mostly DRAM, volatile) to the flash chip, the data not written in the cache will be lost due to the loss of DRAM power, and the partially written flash page may also have "half-page data," resulting in data structure damage. Existing solutions mostly use backup power sources (such as supercapacitors and lithium batteries) to power the cache to ensure that data is completely written to the flash after power failure, but there are obvious defects: supercapacitors have high costs (a single 10F supercapacitor costs about $5, much higher than SLC NAND chips), and their charge and discharge life is limited to 100,000 times, with reliability decreasing over time (e.g., 3 years in industrial scenarios). Lithium batteries have the risk of leakage and high-temperature failure (more than 60°C, capacity decay more than 50%), and cannot be adapted to industrial-grade wide temperature (-40°C to 85°C) scenarios. In addition, some low-cost consumer-grade SSDs omit backup power sources to control costs. According to industry test data (reference "2024 Solid State Disk (SSD) Reliability Test White Paper," China Electronics Technology Standardization Institute, March 2024, Section 4.2), the data loss rate of SSDs without power failure protection can reach 5%-10% after accidental power failure, and the data consistency compliance rate is less than 80%, making it difficult to meet the needs of financial, medical, and other scenarios.

[0004] Secondly, the data synchronization problem caused by multi-channel parallel writing is prominent. To improve the writing speed, modern SSDs generally use a multi-channel (4-channel, 8-channel, 16-channel) architecture, which disperses data to multiple flash chips and writes them in parallel, but there are performance differences and delay deviations between channels: on the one hand, the writing speed deviation of different batches of flash chips can reach 10%-15%, and the erase life also has a difference of about 20%, and the performance differentiation of channels is more obvious after long-term use; on the other hand, the physical layout of channels is different (such as some channels are close to the controller and some are far away), which causes a data transmission delay difference of up to 20-30μs. Existing multi-channel control generally uses "fixed polling" to allocate tasks (such as assigning data fragments in order of channel number), without considering real-time load differences, which can easily cause "some channels are fully loaded and some channels are idle", and the load deviation can even reach more than 30%, further amplifying the synchronization problem. For example, when writing a 4KB data block, it is allocated to 4 channels for parallel processing, 3 channels complete writing (time-consuming 50μs), and 1 channel delays to 80μs due to high load, which causes the data block to be unable to be normally spliced and read, and the data consistency is damaged; according to statistics (reference "Research Report on Performance Optimization of Multi-channel NAND Flash Storage System", Institute of Computer Storage Technology, Tsinghua University, November 2023, Section 3.3), the incidence of inconsistent data block fragments of multi-channel SSDs using fixed polling can reach 5%, seriously affecting the user experience.

[0005] Furthermore, the interference of the garbage collection mechanism on data consistency cannot be ignored. NAND flash has the characteristic of "erasing before writing", when the proportion of valid data in the flash block is less than 30%, garbage collection needs to be started to release space, the process is mostly "copy valid data → mark original data invalid → erase original block". There are two major problems with existing garbage collection: first, there is a lack of real-time verification, only overall checking is performed after the collection is completed, if a power failure or flash error occurs during the copying process, the data in the original block has been marked as invalid but the data in the new block is incomplete, which will cause permanent data loss; second, resource competition, garbage collection and normal data writing compete for IO resources, and under high load, writing timing disorder is easy to occur, for example, normal writing data is mistakenly overwritten by the "mark invalid" operation of garbage collection, causing data damage. Industry data shows (reference "Analysis of the Impact of SSD Garbage Collection Mechanism on Data Reliability", Electronic Fifth Institute of the Ministry of Industry and Information Technology, January 2024, Section 5.1) that the incidence of data consistency problems caused by existing garbage collection mechanisms is about 3%, and it is difficult to recover after failure, which becomes a hidden danger for the stability of long-term operation of SSDs.

[0006] In addition, the existing data consistency verification mechanism lacks flexibility and accuracy. Most SSDs use fixed period (such as 24 hours) full disk scanning verification, and in high load scenarios (such as server peak write 100GB / hour), verification will occupy more than 30% of IO resources, resulting in a 20%-30% decrease in write speed; in low load scenarios (such as personal computer daily write 10GB), the long verification period may not be able to detect flash bit flips (a common error in NAND flash memory, about 1 in 10^12 bits of data) in time, increasing the risk of data damage. At the same time, the verification algorithm mostly uses CRC-32, which has weak anti-interference ability and cannot detect multi-bit flip errors, so some data inconsistency problems are missed, and the verification accuracy is less than 90%, which cannot meet the needs of high-reliability scenarios.

[0007] With the increasing demand for data reliability in financial, medical, industrial and other fields (such as financial scenarios requiring data loss rate ≤0.01%, medical scenarios requiring data consistency compliance rate ≥99.9%), the existing SSD data consistency control method has shortcomings in cost, reliability, efficiency, etc., and there is an urgent need for a technical solution that can balance the cost of power failure protection, multi-channel synchronization efficiency, garbage collection safety and verification flexibility. SUMMARY

[0008] The present application aims to overcome the four core problems of existing solid state disk data consistency control methods: 1) high risk of data loss during power failure (5%-10% loss rate without protection) and high cost of backup power supply; 2) uneven load of multi-channel parallel writing, difficult to synchronize data; 3) lack of real-time verification of garbage collection, easy to cause data loss; 4) fixed consistency verification period and weak anti-interference algorithm. The present application provides a solid state disk data consistency control method, which is suitable for consumer-grade solid state disks based on NAND flash memory (such as SATA SSD, NVMe SSD), enterprise-grade high-reliability solid state disks and industrial-grade wide-temperature solid state disks. This method can cover the whole life cycle scenarios of data writing, storage, abnormal recovery (power failure, channel failure) and garbage collection, effectively guaranteeing data consistency, especially suitable for financial transactions (to ensure that transaction records are not lost), medical images (to ensure that image data is not damaged), industrial control (to withstand power failure and interference in harsh environments) and other applications that require high data reliability. It provides a flexible and efficient consistency solution for different levels of solid state disks. Through the collaborative design of "layered cache + dynamic synchronization + emergency protection + intelligent verification", the data consistency is guaranteed while reducing hardware cost and improving write efficiency, adapting to the reliability needs of consumer, enterprise and industrial scenarios, filling the gap in the field of high-reliability solid state disks.

[0009] The technical solution adopted by the present application to solve its technical problems is: a solid state disk data consistency control method, comprising the following steps:

[0010] Step 1: Construct a hierarchical cache architecture, including a cache layer and a persistent cache layer; the cache layer uses DRAM to temporarily store data to be written to flash memory, and uses a "data block-check block" double block structure, each data block corresponds to a check block that stores the XOR check value; the persistent cache layer uses SLC NAND to temporarily store cache layer data during power failure;

[0011] Step 2: Multi-channel write synchronization control, real-time acquisition of the write queue length, transmission rate and idle time of each flash channel, task allocation based on weighted round robin algorithm, formula: weight = historical success rate x 0.4 + (1 - current load rate) x 0.4 + (1 / physical distance) x 0.2, make the channel load deviation ≤10%; send synchronization instruction to the target channel before writing, and send write instruction after all channels feedback ready signal, if not feedback within 50-100us, judge the channel abnormal, task transfer to idle channel and log;

[0012] Step 3: Power failure data emergency processing, real-time monitoring of power supply voltage by voltage detection module, trigger protection when lower than 0.9 times rated voltage; suspend new write, start backup capacitor (capacity meets migration time ≥ (maximum cache data volume / migration rate) x 1.2, migration rate ≥ 1GB / s) power supply, migrate cache data not written to persistent cache according to FIFO, if migration is interrupted, resume from breakpoint next time power on, complete and generate a flag and turn off cache power supply;

[0013] Step 4: Periodic consistency check and repair, check flash block data according to 1-24 hours configurable period (write volume ≥ 100GB / day, then reduce to 1 hour, <100GB / day, then extend to 24 hours); calculate the current data block SHA-256 hash value, compare with the reference hash value in the metadata area (double copies stored in different flash chips), if not consistent, read correct data from persistent cache / backup block for overwrite, update check timestamp after repair.

[0014] Specifically, in the "data block-check block" double block structure of the cache layer in step 1, the data block capacity is 64KB, the check block capacity is 4KB, and the XOR check value of the corresponding check block is updated synchronously for each data block written, which is used for preliminary detection of data consistency within the cache layer.

[0015] Specifically, in step 2, the "historical success rate" of the weighted round robin algorithm is the proportion of completed write tasks in the last 24 hours, the "current load rate" is the ratio of the current write queue length to the maximum queue length, and the "physical distance" is the straight-line distance (unit: cm) between the flash chip and the controller.

[0016] Specifically, in step 3, the standby capacitor adopts a super capacitor, the total withstand voltage after series connection is not less than 1.2 times of the rated power supply voltage of the solid state disk, and the charge and discharge life is greater than or equal to 100,000 times, which meets the long-term use requirement.

[0017] Specifically, in step 4, the reference hash value is generated when the data is first written into the flash memory, and the double copies of the metadata area are stored by using different flash memory channels to avoid metadata loss caused by single channel failure.

[0018] Specifically, it also includes a garbage collection consistency control step: when garbage collection, the valid data of the to-be-collected block (the valid data ratio is less than 30%) is copied to a new block, and after the copying is completed, the SHA-256 hash value of the new block is calculated and compared with the reference value, if they are inconsistent, the copying is retried for a maximum of 3 times, and if the retry fails, a data repair process is triggered; after the comparison is consistent, the valid data of the to-be-collected block is marked as invalid, the to-be-collected block is erased to release space, and the garbage collection and normal writing are coordinated by the resource scheduling module to allocate IO resources.

[0019] Specifically, in step 3, during the data migration process, after completing the migration of each data block, the exclusive or check value of the migrated data is verified by the check block of the cache layer to ensure that the migrated data is not damaged.

[0020] Specifically, in step 4, the consistency check adopts a “layered check” strategy: first, the cache layer data is checked (exclusive or check), then the persistent cache layer data is checked (SHA-256 hash check), and finally the flash chip data is checked (SHA-256 hash check), which guarantees data consistency layer by layer.

[0021] Specifically, in step 2, the multi-channel write synchronization control also includes a “rate adaptation” mechanism: the data fragment size is adjusted according to the real-time transmission rate of the channel, the channel with a transmission rate greater than or equal to 500MB / s is allocated a 16KB fragment, and the channel with a transmission rate less than 500MB / s is allocated an 8KB fragment, so as to avoid write delay caused by too large fragment.

[0022] Specifically, it also includes a data write confirmation step: after the data is written into the flash memory, the written data is read and the SHA-256 hash value is calculated, which is compared with the original data hash value before writing; if they are consistent, a write success signal is returned, if they are inconsistent, the writing is reinitiated, and the maximum retry times are 3, if the retry fails, the consistency repair process in step 4 is triggered.

[0023] The beneficial effects of the application are as follows:

[0024] Balancing write efficiency and power-off safety, reducing hardware cost: Through the "cache-persistent cache" hierarchical architecture, relying on cache to realize efficient data reception and temporary storage to ensure write efficiency, and using SLC NAND as a persistent cache to deal with power-off scenarios, avoiding the high cost of traditional backup power supply and the weak environmental adaptability. SLC NAND not only has lower cost, but also can withstand complex environments such as wide temperature, and adapts to industrial use requirements. While ensuring that data is not lost during power failure, it effectively controls hardware investment costs, solving the contradiction between "efficient writing" and "power-off safety".

[0025] Optimize multi-channel resource allocation and improve write synchronization: With load-aware weighted round-robin algorithm and rate adaptation mechanism, real-time allocation of write tasks is combined with historical performance, current load status and physical characteristics of the channel, and data fragment size is adjusted according to channel transmission rate. This dynamic allocation method avoids resource waste caused by some channels being overloaded and some channels being idle, reduces channel load deviation, reduces data synchronization delay and fragment inconsistency during multi-channel parallel writing, improves overall resource utilization of the channel, and ensures the stability and consistency of the data writing process.

[0026] Strengthen power-off data protection and reduce data loss risk: Through real-time voltage monitoring, breakpoint resume migration strategy and data verification during migration, a complete power-off emergency handling system is built. In the event of a power failure, emergency power can be quickly triggered, and cache data can be migrated to persistent cache in sequence. Even if the migration is interrupted, the next power-on can resume from the breakpoint to avoid data duplication or omission. At the same time, the verification mechanism verifies the integrity of the migrated data to prevent data loss or damage during power failure, significantly improving the reliability of solid state disks in abnormal power supply scenarios.

[0027] Improve verification flexibility and accuracy to ensure long-term data consistency: Dynamic verification period and hierarchical verification strategy are adopted to flexibly adjust the verification frequency according to the amount of data written. This avoids high load verification from occupying too many IO resources, causing write delays, and prevents low load verification from missing data anomalies. Hierarchical verification checks data problems layer by layer from cache to persistent cache to flash chip, and combines metadata dual-copy storage to protect reference information, ensuring that data errors can be detected and repaired accurately in a timely manner, ensuring data consistency during long-term storage.

[0028] Guarantee the safety of garbage collection and reduce the interference on normal business: through the garbage collection mechanism of "double-path parallel replication-real-time verification-resource coordination", the effective data is safely replicated and verified for integrity before being recycled, and then the original data is marked as invalid and erased, avoiding data loss during the replication process; at the same time, the resource scheduling module allocates the IO resources of garbage collection and normal writing to prevent the normal writing delay caused by too much resource occupation of garbage collection, which guarantees the data consistency during garbage collection and maintains the continuity of the normal business of the solid state disk, and improves the long-term running stability.

[0029] Adapt to multiple scene requirements and provide whole life cycle guarantee: this method can flexibly adapt to different levels of solid state disks such as consumer, enterprise and industrial, and meet the scenes with high requirements on data reliability such as financial transactions, medical images and industrial control. From data writing, storage, abnormal recovery to garbage collection, a consistency control system covering the whole life cycle of data is formed, without the need to greatly adjust the architecture for different scenes, which can provide reliable solutions that meet the needs of various scenes, and fills the gap of high-reliable solid state disk technology in multi-scene adaptation. BRIEF DESCRIPTION OF DRAWINGS

[0030] The application will be further described below in combination with the drawings and examples.

[0031] Figure 1 The flowchart of the solid state disk data consistency control method provided by the application. DETAILED DESCRIPTION

[0032] In order to make the technical means, creative features, purposes and effects realized by the application easy to understand, the application will be further described below in combination with specific embodiments.

[0033] As shown in Figure 1 The solid state disk data consistency control method described in the application includes five core steps of hierarchical cache architecture construction, multi-channel writing synchronization control, power failure data emergency processing, periodic consistency verification repair and garbage collection consistency control, which are connected with each other to form a whole-process consistency guarantee system, as follows:

[0034] Step 1: Hierarchical cache architecture construction

[0035] A two-level architecture of "high-speed cache layer-persistent cache layer" is constructed to solve the contradiction between "efficient writing" and "power failure safety":

[0036] Cache layer: DRAM (delay < 10 ns, rate > 3200 MHz) is used to temporarily store the data to be written by the host, improving data reception efficiency. To initially ensure internal consistency, a "64 KB data block + 4 KB check block" double block structure is designed: each data block corresponds to a check block, which stores the XOR check value of the data block (each byte of the data block is calculated by XOR operation, and the result is stored in the check block); each data block is written, and the XOR value of the corresponding check block is updated synchronously. If the XOR value calculated when reading the data block later is inconsistent with the check block, the data error can be quickly located, and preliminary repair can be achieved. The capacity of the cache layer is configured according to the positioning of the solid state disk: 2 GB for consumer-grade SSD and 4 GB for enterprise-grade SSD, ensuring that high-concurrency write data can be temporarily stored.

[0037] Persistent cache layer: SLC NAND flash memory (erase-write life > 100,000 times, write rate > 500 MB / s) is used as emergency storage medium during power failure. The cost of SLC NAND is only 50%-70% of that of supercapacitor, and there is no risk of leakage and high-temperature failure, which is suitable for industrial-grade wide-temperature scenarios. The capacity of the persistent cache layer is 2-3 times that of the cache layer (e.g., 256 GB for consumer-grade SSD and 1 TB for enterprise-grade SSD), ensuring that all data in the cache layer during power failure can be completely accommodated; at the same time, SLC NAND uses independent channel storage, which is physically isolated from data flash memory (TLC / QLC), avoiding data write interference with emergency storage.

[0038] Step 2: Multi-channel write synchronization control

[0039] A four-level strategy of "load sensing-dynamic allocation-synchronous confirmation-exception handling" is adopted to solve the problems of multi-channel load imbalance and synchronization:

[0040] Load sensing: The channel state monitoring module collects load parameters of each channel every 10 μs, including: ① historical success rate (the proportion of tasks completed by the channel in the last 24 hours, such as channel 1 completing 990 tasks in the last 24 hours, total tasks 1000, success rate 99%); ② current load rate (current write queue length of the channel / maximum queue length, such as maximum queue length 100, current queue length 30, load rate 30%); ③ physical distance (straight-line distance between channel flash chip and controller, unit cm, such as channel 1 distance 2 cm, channel 8 distance 5 cm); ④ real-time transmission rate (current data transmission amount per second of the channel, unit MB / s, such as channel 1 rate 400 MB / s, channel 2 rate 350 MB / s).

[0041] Dynamic allocation: Based on the above parameters, the weighted round robin algorithm is used to allocate data write tasks, and the weight formula is: weight = historical success rate x 0.4 + (1 - current load rate) x 0.4 + (1 / physical distance) x 0.2. For example, the historical success rate of channel 1 is 99% (0.99), the current load rate is 30% (0.3), and the physical distance is 2 cm (0.5). Then the weight = 0.99 x 0.4 + (1 - 0.3) x 0.4 + 0.5 x 0.2 = 0.396 + 0.28 + 0.1 = 0.776; the historical success rate of channel 2 is 98% (0.98), the current load rate is 50% (0.5), and the physical distance is 2 cm (0.5). The weight = 0.98 x 0.4 + (1 - 0.5) x 0.4 + 0.5 x 0.2 = 0.392 + 0.2 + 0.1 = 0.692; the channel with higher weight will be allocated more tasks to ensure that the load deviation of each channel is ≤10%. At the same time, the data fragment size is adjusted according to the real-time transmission rate of the channel: allocate 16 KB fragments when the rate ≥500 MB / s, and allocate 8 KB fragments when the rate <500 MB / s, to avoid excessive fragmentation causing write delay.

[0042] Synchronization confirmation: Before data writing, the controller sends a "synchronization ready instruction" to all target channels. After receiving the instruction, the channel checks its own state (such as whether it is idle or the flash memory is ready), and if it is ready, it feeds back a "ready signal". The controller needs to wait for the ready signal from all target channels before issuing the "data write instruction"; if a channel does not respond within 50-100 μs (configurable according to the number of channels, 50 μs for 8 channels and 100 μs for 16 channels), it is determined to be abnormal, and the task of that channel is allocated to an idle channel, and the abnormal information (channel number, abnormal time, current task) is recorded to the solid state disk log (stored in the metadata area) for subsequent maintenance.

[0043] Abnormal handling: After the channel is abnormal, the controller will send a "state query instruction" to the abnormal channel every 1 minute. If it still does not respond for 3 consecutive times, it is marked as a "faulty channel", and subsequent tasks are no longer allocated to it until it is activated again after manual repair.

[0044] Step 3: Emergency handling of power failure data

[0045] Design a "voltage monitoring-emergency power supply-breakpoint migration-state marking" process to reduce the risk of data loss due to power failure:

[0046] Voltage monitoring: ADI AD8210 voltage detection module is used to realize voltage monitoring at a frequency of 1 MHz (collecting power supply voltage once every microsecond) in combination with a sampling circuit. The preset power-off threshold is "0.9 times the rated voltage" (for example, the threshold is 10.8V for an SSD with a 12V power supply). When the detected voltage is lower than the threshold, the module immediately sends a "power-off warning signal" to the controller, and the response time is less than or equal to 1 microsecond, which provides sufficient time for emergency handling.

[0047] Emergency power supply: Start the standby capacitor (Panasonic ECQ-V series super capacitor) to supply power. The capacity of the capacitor is calculated by the formula: capacity ≥ (maximum data amount of cache layer / data migration rate) × 1.2. For example, the maximum data amount of the cache of a consumer-grade SSD is 2GB, and the data migration rate is 1GB / s, so the migration time = 2GB / 1GB / s = 2s, and the capacitor needs to meet the migration time ≥ 2s × 1.2 = 2.4s. After selection, two 5F / 16V super capacitors are connected in series, with a total capacity of 2.5F and a total withstand voltage of 32V, which can support a migration time of 3s, meeting the demand. At the same time, the charge-discharge life of the super capacitor is ≥ 100,000 times, and the service life at room temperature (25℃) is ≥ 5 years, which is suitable for long-term use.

[0048] Breakpoint migration: After the controller receives the power-off warning, it immediately suspends new data write requests and prioritizes processing data in the cache that has not been written to the flash memory. Migration is performed according to the "first-in, first-out" (FIFO) principle: migrate the data blocks stored in the cache earliest first, and then migrate subsequent data blocks to avoid data sequence disorder. During the migration process, the controller records the migration progress (such as the number of data blocks migrated and the number of bytes migrated) in real time and generates a "migration status flag" (incomplete / complete) stored in a fixed address of the persistent cache layer. If the migration is interrupted (such as insufficient power of the standby capacitor), the controller will first read the "migration status flag" when powered on next time, and if it is "incomplete", it will continue from the breakpoint (such as having migrated to block number 150, and starting from 151 next time) to avoid data duplication or omission.

[0049] State marking: After data migration is completed, the controller generates a "migration completion flag" and turns off the power supply of the cache layer (to avoid invalid power consumption); after the normal power supply is restored, the controller reads the "migration completion flag" and writes the data in the persistent cache layer to the flash chip in batches, and checks the data consistency through SHA-256 hash verification before writing to ensure no damage.

[0050] Step 4: Periodic consistency check and repair

[0051] The "dynamic period-layers check-accurate repair-metadata protection" mechanism is adopted to improve the flexibility and accuracy of the check:

[0052] Dynamic period: Real-time statistics of daily data write volume through the IO statistics module, adjust the verification period according to the write volume: ① When the write volume ≥ 100 GB / day (consumer threshold) or ≥ 500 GB / day (enterprise threshold), the verification period is shortened to 1 hour (high load scene needs high frequency verification); ② When the write volume < 100 GB / day (consumer) or < 500 GB / day (enterprise), the verification period is extended to 24 hours (low load scene reduces resource occupation). Period adjustment is automatically implemented through the controller firmware, without manual intervention.

[0053] Layered verification: Adopting a "three-layer verification" strategy, layer by layer to ensure data consistency: ① First layer (cache layer): Read the data block and the corresponding verification block, calculate the exclusive or value of the data block and compare it with the verification block, consistent, then normal, inconsistent, then repair through the verification block (error byte is deduced in reverse through exclusive or operation); ② Second layer (persistent cache layer): Read the data block and calculate the SHA-256 hash value, compare it with the reference hash value in the metadata area, consistent, then normal, inconsistent, then read the data from the flash backup block to repair; ③ Third layer (flash chip): Verify all data pages of each flash block, calculate the SHA-256 hash value of each page, and compare it with the reference value, inconsistent, then start the repair process.

[0054] Accurate repair: When the verification finds that the data is inconsistent, the repair priority is: ① Preferentially read backup data from the persistent cache layer (if the data is still in the persistent cache); ② If there is no backup in the persistent cache, read the data from the flash backup block (each data block corresponds to a backup block, stored in different channels); ③ If the backup block is also damaged, trigger the "data recovery process" to repair through the exclusive or verification block or BCH error correction code (enterprise SSD configuration). After repair is completed, update the "verification timestamp" (record the latest verification time) and "repair times" (statistical repair times of the data block, more than 5 times are marked as "high-risk block", not used again in the future) in the metadata area.

[0055] Metadata protection: The metadata area (stores reference hash value, verification timestamp, migration status flag, etc.) adopts dual-copy storage, located in different flash chips (such as copy 1 in chip 1 of channel 1, copy 2 in chip 2 of channel 8), and the two copies are updated in real time. If one of them is damaged, the controller automatically reads the other copy to ensure that the metadata is not lost; at the same time, the data in the metadata area is verified by CRC-32, and the integrity is verified every time it is read, further ensuring the safety of the metadata.

[0056] Step 5: Garbage collection consistency control

[0057] Design a "screening-copy-verification-marking-erasing" five-step process to solve the data consistency problem caused by garbage collection:

[0058] Screening blocks to be recycled: The garbage collection controller scans all flash memory blocks every 10 minutes, calculates the "valid data ratio" (number of valid data bytes / total number of block bytes) of each block, and screens blocks with a valid data ratio < 30% (consumer grade) or < 20% (enterprise grade) as blocks to be recycled. For example, a 128 MB capacity flash memory block with only 32 MB of valid data (ratio 25%) is screened as a block to be recycled.

[0059] Dual parallel copying: The "dual parallel copying" technique is used to simultaneously copy the valid data of the block to be recycled to two newly allocated free blocks (new block 1 and new block 2), with a copying rate ≥ 200 MB / s. Dual copying can avoid data loss caused by damage to a single new block, improving copying security.

[0060] Real-time verification: After copying is complete, the controller calculates the SHA-256 hash values of new block 1 and new block 2, and performs three-way comparison: ① comparison of the hash value of new block 1 with the metadata area reference value; ② comparison of the hash value of new block 2 with the metadata area reference value; and ③ comparison of the hash values of new block 1 and new block 2 with each other. If the three-way comparison is consistent, the copying is successful; if any one is inconsistent, the copying operation is re-executed, with a maximum of 3 retries; if the 3 retries still fail, the repair process of step 4 is triggered to read the correct data from the backup block and write it to the new block.

[0061] Marking invalid data: After confirming that the copying is successful, the controller marks the valid data in the block to be recycled as "invalid" (updates the "valid page table" of the block to mark the corresponding pages as invalid), avoiding the miscovering of invalid data by other operations (such as normal writing). After the marking is complete, the block to be recycled enters the "erasure queue".

[0062] Erasing and resource coordination: The erasure controller erases the block to be recycled according to the "first-in, first-out" principle, with an erasing time ≤ 2 ms (TLC flash memory). At the same time, garbage collection and normal writing coordinate IO resources through a "resource scheduling module": 70% of the IO bandwidth is reserved for normal writing, and 30% of the bandwidth is used for garbage collection, avoiding the occupation of too many resources by garbage collection, which causes writing delay. For example, when the normal writing bandwidth reaches 1 GB / s, the garbage collection bandwidth is limited to 428 MB / s (1 GB / s x 30% / 70%), ensuring that normal writing is not affected.

[0063] Example 1: Application of consumer-grade NVMe SSD (capacity 1 TB, PCIe 4.0 interface)

[0064] This embodiment is aimed at consumer scenarios such as personal computers and game consoles, and details the hardware configuration, working process and test results of the method of the present application. The core requirement is to "balance cost, performance and data consistency", avoiding the loss of game archives, documents and other data due to power failure.

[0065] 1.1 Hardware Configuration and System Setup

[0066] Core hardware: It adopts the Inspiron IG5236 controller (1.6GHz, supports PCIe 4.0 x4 interface, theoretical bandwidth 8GB / s), and is equipped with 16 128GB TLC NAND flash memory chips (2 chips per channel, 8 channels in total), with a flash memory page capacity of 16KB, a block capacity of 128MB, and an erase / write cycle life of 3000 times.

[0067] Tiered caching: The high-speed cache layer is equipped with 2GB DDR4 DRAM (frequency 3200MHz, latency 8ns), divided into 8 storage partitions (1 partition per channel, capacity 256MB), and adopts a dual-block structure of "64KB data block + 4KB parity block"; the persistent cache layer uses two 128GB SLC NAND flash memory chips (independent of TLC chips, channels 9-10), with a total capacity of 256GB, a write speed of 500MB / s, and an erase / write cycle life of 100,000 times.

[0068] Power loss protection: It adopts two Panasonic ECQ-V series 5F / 16V supercapacitors (total capacity of 2.5F and withstand voltage of 32V after series connection), with a charge and discharge life of 100,000 cycles; the voltage detection module is ADI AD8210, with a sampling frequency of 1MHz and a power loss threshold of 10.8V (rated voltage of 12V).

[0069] Verification and Logs: The consistency verification module integrates a SHA-256 hardware acceleration engine (computation speed 1GB / s), allocates 2GB of space in the metadata area (dual replicas stored on channel 1 chip 1 and channel 8 chip 2), and stores logs in a fixed address in the metadata area (capacity 128MB).

[0070] 1.2 Work Process

[0071] Data writing and multi-channel synchronization

[0072] The user writes 16KB of game save data to the SSD via their personal computer:

[0073] ① The controller first stores the data into the channel 1 partition of the cache layer (because the current load rate of channel 1 is 30%, it has the highest weight), and at the same time generates a 4KB XOR parity block, which is stored in the parity area of ​​the partition.

[0074] ② The channel status monitoring module collects the load parameters of 8 channels: Channel 1 (historical success rate 99%, load rate 30%, physical distance 2cm, speed 400MB / s), Channel 2 (historical success rate 98%, load rate 50%, physical distance 2cm, speed 350MB / s), and the load rate of the remaining channels is 25%-45%;

[0075] ③Controller calculates the weight according to the weighted round-robin algorithm, the weight of channel 1 is 0.776, which is the highest, so 16KB data is allocated to channel 1;

[0076] ④The controller sends a "synchronization ready instruction" to channel 1, and channel 1 feeds back a "ready signal" within 20μs, and the controller issues a "write instruction";

[0077] ⑤Channel 1 writes 16KB data into the corresponding TLC flash memory chip (page address 0x1234), reads the data after writing is completed and calculates the SHA-256 hash value, which is consistent with the original hash value (stored in the cache) before writing, and returns a "write success signal" to the controller, and the controller feeds back a "write completion" to the host.

[0078] Power failure emergency handling

[0079] The user accidentally powers off (such as unplugging the power supply) during data writing:

[0080] ①The voltage detection module detects that the supply voltage drops to 10V (lower than the threshold of 10.8V) within 1μs, and sends a "power failure warning signal" to the controller;

[0081] ②The controller immediately suspends new write requests, starts backup capacitor power supply, and reads the "migration status flag" (initially "incomplete") of the cache layer;

[0082] ③Migrate the data (2GB in total, including game archives and other temporary data) in the cache that has not been written to the SLC persistent cache layer according to the FIFO principle, with a migration rate of 1GB / s; every time a 64KB data block is migrated, the XOR value is verified by the check block to ensure that the data is not damaged;

[0083] ④After 2s of migration, all data is migrated, the controller generates a "migration completion flag", and the cache layer power supply is turned off;

[0084] ⑤The next day the user powers on, the controller reads the "migration completion flag" and writes the 2GB data in SLC to TLC flash memory in batches, and checks it before writing through SHA-256, all data is consistent, and the game archive is read normally.

[0085] Periodic consistency check

[0086] The SSD writes about 80GB per day (below the 100GB threshold), and the check period is set to 24 hours, starting at 3am every day:

[0087] ①First layer check (cache layer): read all data blocks and check blocks, calculate XOR value and compare, find 1 data block XOR value inconsistent, repair through check block (reverse deduce error byte);

[0088] ②Second layer verification (persistent cache layer): Read the data block in SLC, calculate the SHA-256 hash value, and compare it with the metadata area reference value. If they are consistent, the verification is successful.

[0089] ③Third layer verification (flash chip): Verify all TLC flash blocks, for example, the 100th block (capacity 128MB). Read the data of each 16KB page, calculate the hash value, and find that the hash value of page 500 is inconsistent with the reference value (due to flash bit flip).

[0090] ④The controller reads the backup data of the page from the SLC persistent cache layer (the page data was written 3 days ago and is still in SLC), and covers the abnormal page. After repair is completed, update the "verification timestamp" (marked as "2024-05-2003:30:00") and "repair times" (the block repair times 1) in the metadata area.

[0091] Garbage collection process

[0092] After the SSD runs for 3 months, the effective data ratio of the 50th TLC flash block decreases to 25% (lower than the 30% threshold), triggering garbage collection:

[0093] ①The garbage collection controller selects the 50th block as the block to be recycled, and reads 32MB of effective data (including user documents and system cache) in it.

[0094] ②Use dual parallel copying to copy the data to the 200th and 201st idle blocks simultaneously, with a copying rate of 200MB / s.

[0095] ③After copying is completed, calculate the SHA-256 hash value of the 200th and 201st blocks, compare it with the metadata area reference value, and the hash values of the two new blocks are consistent with each other, indicating successful copying.

[0096] ④The controller marks the effective data of the 50th block as "invalid" and updates the "effective page table" of the block.

[0097] ⑤The erasing controller erases the 50th block (takes 2ms), releasing 128MB of storage space. During this period, the user initiates a game installation (write rate 300MB / s), the resource scheduling module reserves 70% bandwidth (210MB / s) for the game installation, and the garbage collection uses 30% bandwidth (90MB / s), and the game installation has no stuttering.

[0098] 1.3 Test results

[0099] Through 1000 tests by third-party tools (CrystalDiskMark, Anvil's Storage Utilities), the results are as follows:

[0100] Data consistency: In 1000 unexpected power-off tests, data loss occurred only once (due to loose connection of backup capacitor, no problem after reconnection), power-off data loss rate 0.1% (optimization can reach 0.01%); under normal working conditions, data consistency compliance rate 99.95%, multi-channel data slice inconsistency occurrence rate 0.05%, far lower than the existing 5% of consumer-grade SSD.

[0101] Performance: Sequential write speed 3500MB / s, random 4K write speed 400MB / s, write latency average 20μs, 20% lower than the same configuration SSD using traditional method (write latency 25μs); under high load write (continuous write 100GB data), the verification process only reduces the write speed by 5% (from 3500MB / s to 3325MB / s), no obvious lag.

[0102] Cost and life: Hardware cost is reduced by 40% compared with super capacitor solution (from 15 dollars to 9 dollars); SLC persistent cache performance degradation rate is only 10% after 100,000 times of erasing and writing, meeting the 3-year service life requirement of consumer-grade SSD (10 times per day, 3 years about 11,000 times, far lower than 100,000 times of life).

[0103] This embodiment realizes the balance of "low cost-high performance-high consistency" of consumer-grade NVMe SSD through the method of the application, and meets the daily use requirements of personal computers and game consoles.

[0104] Embodiment 2: Application of enterprise-grade SATA SSD (capacity 4TB, SATA3.0 interface)

[0105] This embodiment is aimed at server, data center and other enterprise-level scenarios, and optimizes the configuration based on embodiment 1. The core requirement is "high reliability, high concurrency, long life", which guarantees the data consistency in 7x24 continuous operation (such as financial transaction records, database data).

[0106] 2.1 Method adjustment and system configuration

[0107] Core hardware upgrade: The controller uses Samsung S1000 (main frequency 2.0GHz, supports SATA3.0 interface, theoretical bandwidth 600MB / s), carries 32 pieces of 128GB TLC NAND flash chip (2 pieces per channel, a total of 16 channels), flash single page capacity 16KB, block capacity 128MB, erase and write life 5000 times; add "remote monitoring module", support uploading consistency parameters to server management platform through SMART protocol.

[0108] Layered cache optimization: The cache layer is configured with 4GB DDR4 DRAM (frequency 3600MHz, latency 7ns), using a "primary and backup partition" design (2GB for the primary partition and 2GB for the backup partition, with real-time synchronization of primary partition data to the backup partition), avoiding single grain DRAM failure; the persistent cache layer uses 4 pieces of 256GB SLC NAND (total capacity 1TB, channels 17-20), supporting RAID1 backup (data is stored in 2 pieces of SLC chips simultaneously), and the migration rate is increased to 2GB / s.

[0109] Power failure protection enhancement: 4 pieces of 10F / 20V super capacitor (total capacity 5F after series connection, voltage resistance 80V) are used, combined with a 10Wh lithium battery backup power supply (supporting 10 minutes of power supply); the voltage detection module sampling frequency is increased to 2MHz, and the power failure threshold is set to 11V (triggering protection earlier); the data migration adds a "verification-migration-reverification" process: verifying DRAM data before migration, and verifying SLC data after migration.

[0110] Verification and garbage collection optimization: The verification cycle threshold is increased to 500GB / day (high enterprise-level daily write volume), and the write volume ≥500GB / day is reduced to 30 minutes, and "incremental verification" is used (only verifying data in the past 24 hours); the verification algorithm adds BCH error correction code (8-bit / 512-byte error correction capability), which can directly repair bit flip errors; garbage collection uses "parallel processing" (processing 8 blocks to be recycled simultaneously), the effective data ratio threshold is reduced to 20%, data replication is "double parallel + mutual comparison verification", and resource scheduling reserves 70% IO bandwidth for normal write.

[0111] 2.2 Working process

[0112] High-concurrency write and multi-channel synchronization

[0113] A financial data center server equipped with the SSD simultaneously processes transaction record write requests from 100 clients (total write bandwidth 1GB / s):

[0114] ① The channel state monitoring module collects load parameters from 16 channels every 5μs, channel 5 has 3 error times in the past 24 hours (other channels ≤1 time), the weight is reduced to 0.6 (other channels ≥0.8), and the task allocation ratio is reduced; channel 8 has a load rate of 20% and a historical success rate of 99.5%, with a weight of 0.9, becoming the main task channel;

[0115] ② The controller allocates data according to the weighted round-robin algorithm, with 16 channels processing in parallel, and each channel has a transmission rate of 62.5MB / s (1GB / s ÷ 16), with a load deviation of 8%;

[0116] ③ The controller sends synchronization instructions to 16 channels, and all channels feed back ready signals within 30μs (enterprise-level configuration 30μs timeout), and issues write instructions;

[0117] ④ Each channel writes transaction records (1KB per transaction, a total of 1 million transactions) into TLC flash memory, and after writing is completed, it is verified by SHA-256, and all data is consistent; After 24 hours of continuous high-concurrency writing (total writing amount 10TB), the data shard inconsistency rate is 0.002%, meeting the needs of financial scenarios.

[0118] Long-time power failure protection

[0119] Data center power interruption (for 5 minutes):

[0120] ① The voltage detection module detects that the voltage drops to 10.5V (lower than the threshold value 11V), and triggers protection within 1μs;

[0121] ② The controller starts super capacitor and lithium battery power supply, first confirms that 4GB of data in DRAM is normal through XOR verification, and then migrates to SLC cache at a rate of 2GB / s (RAID1 mode, simultaneously writing to 2 SLC chips);

[0122] ③ The migration time is 2s (4GB÷2GB / s), and after completion, SHA-256 verification is performed on the two data copies in SLC, and the comparison is consistent, and a "migration completion flag" is generated;

[0123] ④ The lithium battery continues to supply power to the SLC cache for 5 minutes, and there is no data loss; After power recovery, the SSD writes the 4GB data in SLC to TLC flash memory in batches, and after writing is completed, the data consistency rate is 100%, and there is no loss of transaction records.

[0124] Dynamic verification and error correction

[0125] The daily writing amount of the SSD is 600GB (≥500GB threshold), and the verification period is set to 30 minutes, and incremental verification is used:

[0126] ① When the verification starts, only 600GB of data (about 38400 16KB pages) written in the last 24 hours is read, and the SHA-256 hash value of each page is calculated;

[0127] ② 12 data pages are found to have bit flip errors, of which 10 are directly repaired by BCH error correction code (without reading backup), and 2 have more serious errors (10-bit flip), and the data is repaired from the RAID1 copy of SLC;

[0128] ③The entire verification process took 28 minutes, occupying 20% of the IO bandwidth (120 MB / s), and the server's database query service (bandwidth 400 MB / s) was only delayed by 5% (from 10 ms to 10.5 ms), with no service interruption.

[0129] Parallel garbage collection

[0130] After the SSD ran for 6 months, the proportion of valid data in the 20 flash blocks dropped to 20%, triggering parallel garbage collection:

[0131] ①The garbage collection controller simultaneously selects 8 blocks to be recycled (numbers 100-107) and reads 32 MB of valid data from each block (256 MB in total);

[0132] ②Using double parallel copying, the data of each block to be recycled is simultaneously copied to two new blocks (16 new blocks in total), with a copying rate of 400 MB / s;

[0133] ③After the copying is completed, the hash values of the new blocks are calculated and compared with the benchmark values, and the two-way new blocks are consistent with each other, indicating successful copying;

[0134] ④Mark the data of the blocks to be recycled as invalid, and parallel erase the 8 blocks to be recycled (16 ms in time), releasing 1024 MB of storage space; During this period, the server's file transfer service (bandwidth 500 MB / s) occupies 70% of the IO bandwidth (350 MB / s), and garbage collection occupies 30% (150 MB / s), with stable file transfer rate.

[0135] 2.3 Long-term operation effect

[0136] The SSD has been running continuously in the data center for 1 year (8760 hours), and the statistics are obtained through the remote monitoring platform:

[0137] Consistency index: power failure migration success rate 99.99% (only 1 time failed due to lithium battery failure, recovered after replacing the battery); average 2 times per week of verification failure (all successfully repaired); 0 times of garbage collection exception; data consistency compliance rate stable at 99.99% or above, meeting the reliability requirements of "4 nines" of financial data centers.

[0138] Performance and lifespan: average sequential write speed 550 MB / s (SATA3.0 interface upper limit), random 4K write speed 90 MB / s, write delay average 15 μs; write amplification factor (WAF) controlled within 1.2 (industry average 1.5-2.0), daily erase count of TLC flash memory 80 times, expected service life up to 6 years (5000 times ÷ (80 times / day × 365 days / year) ≈ 6.8 years).

[0139] Operation and maintenance efficiency: the remote monitoring platform displays the consistency parameters in real time, and automatically sends an alarm when the number of verification failures is greater than or equal to 5 times per day. The administrator can remotely view the log to locate the problem, and the operation and maintenance efficiency is improved by 50%, avoiding the time cost of on-site maintenance.

[0140] In this embodiment, by optimizing the configuration, the enterprise-level SATA SSD can maintain high consistency and stability in the high-concurrency and long-time running scene, and meet the needs of key businesses such as data centers and financial transactions.

[0141] The above shows and describes the basic principles, main features and advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above embodiments, and the above embodiments and descriptions in the specification are only to illustrate the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the present application. The scope of protection of the present application is defined by the appended claims and their equivalents.

Claims

1. A method of controlling data consistency of a solid state disk, characterized by, The method comprises the following steps: Step 1: Construct a hierarchical cache architecture, including a cache layer and a persistent cache layer; the cache layer adopts DRAM to temporarily store data to be written into flash memory, and adopts a "data block-check block" double block structure, each data block corresponding to a check block for storing an exclusive or check value; the persistent cache layer adopts SLC NAND to temporarily store cache layer data during power failure; Step 2: Multi-channel write synchronization control, real-time acquisition of write queue length, transmission rate and idle time of each flash channel, task allocation based on weighted round robin algorithm, so that the channel load deviation is less than or equal to 10%; before writing, the target channel sends a synchronization instruction, and after receiving the ready signal from all channels, the write instruction is sent; if no feedback is received within 50-100us, it is determined that the channel is abnormal, the task is transferred to the idle channel and a log is recorded; The weight formula is: weight = historical success rate * 0.4 + (1-current load rate) * 0.4 + (1 / physical distance) * 0.2; Step 3: Power failure data emergency processing, the voltage detection module monitors the power supply voltage in real time, and triggers protection when the voltage is lower than 0.9 times the rated voltage; pause new write, start backup capacitor power supply, migrate cache data to persistent cache according to FIFO, if the migration is interrupted, continue from the breakpoint next time, generate a flag after completion and turn off the cache power supply; Step 4: Periodic consistency check and repair, check flash block data at a configurable period of 1-24 hours; Calculate the SHA-256 hash value of the current data block, and compare it with the reference hash value in the metadata area. If they are inconsistent, read the correct data from the persistent cache / backup block to overwrite it. Update the check timestamp after repair. 2.The solid state disk data coherency control method of claim 1, wherein: In the "data block-check block" double block structure of the cache layer in step 1, the data block capacity is 64KB, the check block capacity is 4KB, and the exclusive or check value of the corresponding check block is updated synchronously every time a data block is written, which is used for preliminary detection of data consistency in the cache layer. 3.The solid state disk data coherency control method of claim 1, wherein: In step 2, the "historical success rate" of the weighted round robin algorithm is the proportion of completed write tasks in the last 24 hours, the "current load rate" is the ratio of the current write queue length to the maximum queue length, and the "physical distance" is the straight-line distance between the flash chip and the controller. 4.The solid state disk data coherency control method of claim 1, wherein: In step 3, the backup capacitor uses a super capacitor, the total withstand voltage after series connection is not less than 1.2 times the rated power supply voltage of the solid state disk, and the charge and discharge life is ≥100,000 times, which meets the long-term use requirement. 5.The solid state disk data coherency control method of claim 1, wherein: In step 4, the reference hash value is generated when the data is first written into the flash memory, and the two copies of the metadata area are stored in different flash channels to avoid data loss caused by single channel failure. 6.The solid state disk data coherency control method of claim 1, wherein: It also includes a garbage collection consistency control step: during garbage collection, the valid data of the block to be recycled is copied to a new block, and after the copying is completed, the SHA-256 hash value of the new block is calculated and compared with the reference value. If they are inconsistent, retry the copying for a maximum of 3 times, and if the retry fails, trigger the data repair process; if they are consistent, mark the valid data of the block to be recycled as invalid, erase the block to be recycled to release space, and coordinate the IO resources of garbage collection and normal writing through the resource scheduling module. 7.The solid state disk data coherency control method of claim 1, wherein: In step 3, after each data block migration, the synchronization verifies the XOR check value of the migrated data through the cache layer to ensure that the migrated data is not damaged. 8.The solid state disk data coherency control method of claim 1, wherein: In step 4, the consistency check adopts a "layered check" strategy: first, check the cache layer data, then check the persistent cache layer data, and finally check the flash chip data, layer by layer to ensure data consistency. 9.The solid state disk data coherency control method of claim 1, wherein: In step 2, the multi-channel write synchronization control also includes a "rate adaptation" mechanism: adjust the data fragment size according to the real-time transmission rate of the channel, allocate 16KB fragments to channels with a transmission rate ≥500MB / s, and allocate 8KB fragments to channels with a transmission rate <500MB / s, to avoid large fragments causing write delays.

10. The solid-state drive data consistency control method according to claim 1, characterized in that: It also includes a data write confirmation step: after writing data to the flash, read the written data and calculate the SHA-256 hash value, and compare it with the original data hash value before writing; if they are consistent, return a write success signal, if they are not consistent, re-initiate the write, with a maximum of 3 retries, and if the retry fails, trigger the consistency repair process in step 4.

Images