NAND performance optimization method based on static data identification and independent MCU compressed storage

Abstract

The invention relates to the technical field of NAND Flash storage, and discloses an NAND performance optimization method based on static data identification and independent MCU compression storage, and the method comprises the steps: carrying out the statistics of host write events of a compression migration unit through a main control processing unit, and screening candidate units meeting a dual-threshold condition; the independent MCU calculates a unit-level fingerprint under read consistency protection, and divides the unit-level fingerprint into a strong static level and a weak static level after re-checking of a second time window; a full-amount compression strategy or a main body compression and differential log strategy is adopted according to grades, and mapping atom updating is achieved through two-stage submission; meanwhile, the original physical block is marked as delayed recovery, and a preset window is reserved. According to the method, static data can be accurately identified, write amplification and tail delay are reduced, data consistency after power failure is guaranteed, and the storage utilization rate and the system performance stability are improved on the premise that the host burden is not increased.

Description

Technical Field

[0001] This application relates to the field of NAND Flash storage technology, specifically a NAND performance optimization method based on static data identification and independent MCU compressed storage. Background Technology

[0002] NAND Flash storage systems are widely used in various storage devices such as embedded multimedia cards, general-purpose flash memory, and solid-state drives due to their high storage density, excellent read / write performance, and low power consumption. With the continuous growth of data volume, improving storage utilization and system performance through static data compression storage without increasing the host processing burden has become an important research direction in this field.

[0003] Existing static data compression schemes face numerous technical bottlenecks in engineering practice: First, the reliability of static data identification is insufficient. Mechanisms such as garbage collection, wear leveling, or copy-on-write can trigger data migration. Simply relying on the number of physical writes or simple read / write frequency statistics can easily lead to misjudgment of static data, causing data that should remain unchanged for a long time to be frequently compressed or migrated, or missing truly static data. Second, write amplification and tail latency issues are prominent. For data areas that remain unchanged for a long time but are occasionally updated slightly, recompressing them as a whole will cause unnecessary performance loss. Third, data consistency is difficult to guarantee after a power outage. When the independent microcontroller performs compression migration in the background, it needs to update the logic to the physical mapping. If the mapping switch lacks atomicity and rollback mechanism, mapping tearing is very likely to occur, leading to unrecoverable data. These problems seriously affect the stability, reliability, and performance of NAND Flash storage systems. Summary of the Invention

[0004] The purpose of this application is to provide a NAND performance optimization method based on static data identification and independent MCU compressed storage, so as to solve the problems mentioned in the background art.

[0005] According to one aspect of this application, a NAND performance optimization method based on static data identification and independent MCU compressed storage is provided, comprising the following steps: The host write events of the statistical compression migration unit include only write commands initiated by the host; When the triggering condition is met, the compression migration units that have a host write event count of no more than a first threshold within a preset first time window and a time interval of no less than the cooling duration threshold are selected as candidate units. The independent MCU reads candidate cell data under read consistency protection, calculates the page-level fingerprint of each physical page, and aggregates them to generate the first cell-level fingerprint; After a preset second time window, the secondary unit-level fingerprint is calculated and compared with the first unit-level fingerprint to determine the static strength level of the data. Compressed data blocks and block index tables are generated based on the static strength level of the data, and target physical space is allocated; among them, full compression is used for strong static data, and main compression plus differential log is used for weak static data. Write protection is set for candidate cells, compressed data blocks, block index tables, and shadow mapping entries are written to non-volatile memory, and preparation and commit records are written to the mapping log and forced to disk; the mapping entries of candidate cells in the main mapping table are atomically updated to indirect descriptors pointing to shadow mapping entries. The original physical block of the candidate unit is marked as delayed recycling, and the data of the original physical block is retained within the preset delayed recycling window.

[0006] Preferably, the statistics of host write events adopt a sliding window counting method. For each host write event, a corresponding timestamp is recorded. When the interval between the timestamp of a host write event and the current time exceeds the first time window, the host write event is removed from the write event count.

[0007] Preferably, during read consistency protection, if the main control processing unit receives a host write request for a protected candidate unit, it temporarily stores the host write request in the pending write queue and marks the candidate unit as a candidate failure pending determination in the candidate set; during subsequent fingerprint comparison, if a pending write request is detected, the candidate unit is directly determined to be failure and removed from the candidate set.

[0008] Preferably, the page-level fingerprint is the CRC32C check value calculated for each physical page of data; the initial unit-level fingerprint is generated by sequentially aggregating the page-level fingerprint sequence through a rolling aggregation function, and the operation rules of the rolling aggregation function are fixed rules preset by the system firmware.

[0009] Preferably, after the independent MCU writes the compressed data block into the non-volatile memory, it performs a sampling readback verification: randomly selects the written compressed data block according to a preset ratio for readback and verifies its integrity; if the verification fails, a rollback process is triggered.

[0010] Preferably, within the delayed recycling window, if the main control processing unit receives a host write request for the migrated compressed migration unit, or if a decompression verification failure occurs during the read operation of the unit, a rollback process is triggered: the main mapping table is restored to the original mapping relationship of the unit, an abort record is written to the mapping log and forced to disk, and the written compressed state metadata is marked as invalid.

[0011] Preferably, the process further includes a power-off recovery step: After the device is powered on, the main control processing unit scans the mapping log and determines the final state of each compressed migration unit based on the combination of preparation records, commit records, and abort records. If only preparation records exist but no corresponding commit or abort records exist, a rollback operation is performed to clean up the compressed state metadata. If commit records exist but no abort records exist, the main mapping table is restored to an indirect descriptor pointing to the shadow mapping item. If abort records exist, a rollback operation is performed to restore the original mapping relationship. The integrity of the shadow mapping item is verified for all units in the compressed state. If the verification fails and the original physical block is still within the delayed recycling window, a rollback is triggered; otherwise, the data is marked as abnormal.

[0012] In another aspect, this application also provides a NAND performance optimization device based on static data identification and independent MCU compressed storage, comprising: The event statistics module is used to count host write events of the compression migration unit, and the host write events only include write commands initiated by the host; The candidate filtering module is used to filter out compression migration units that, when the triggering conditions are met, have a host write event count within a preset first time window that does not exceed a first threshold and a time interval between the most recent host write event and a cooling duration threshold, as candidate units. The fingerprint generation module is used to read candidate cell data through an independent MCU under read consistency protection, calculate the page-level fingerprint of each physical page, and aggregate to generate the first cell-level fingerprint; The static strength determination module is used to calculate the secondary unit-level fingerprint, compare it with the primary unit-level fingerprint, and determine the static strength level of the data. The compression processing module is used to generate compressed data blocks and block index tables based on the data static strength level and allocate target physical space; among them, full compression is selected for strong static data and main compression plus differential log is selected for weak static data. The atomic update module is used to set write protection for candidate cells, write compressed data blocks, block index tables, and shadow mapping entries to non-volatile memory, and write preparation and commit records to the mapping log and force disk write; the mapping entries of candidate cells in the atomic update master mapping table are indirect descriptors pointing to shadow mapping entries. The delayed recycling module is used to mark the original physical blocks of candidate units as delayed recycling units through the main control processing unit, and retain the data of the original physical blocks within a preset delayed recycling window.

[0013] This application also provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the NAND performance optimization method based on static data identification and independent MCU compressed storage as described above.

[0014] In another aspect, this application provides a storage medium having computer program instructions stored thereon, which can be executed by a processor to implement the NAND performance optimization method based on static data identification and independent MCU compressed storage as described above.

[0015] This application employs a dual-threshold screening and content fingerprint verification mechanism to accurately identify static data, effectively avoiding misjudgments caused by data migration within the system. It utilizes differentiated compression strategies for different static levels: full compression for strong static data and main compression plus differential logging for weak static data, significantly reducing write amplification and tail latency in small-scale data update areas. Based on a two-phase commit, delayed reclamation, and power-off recovery mechanism, it achieves atomic switching of mapping relationships and rollback capability in abnormal scenarios, ensuring data consistency and integrity after power failure. The entire process is executed collaboratively by the main control processing unit and an independent MCU, without increasing the host's burden. This enhances the system's stability and reliability while improving storage space utilization and overall I / O performance. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram illustrating a NAND performance optimization method based on static data identification and independent MCU compressed storage, provided in an embodiment of this application.

[0018] Figure 2 This is a schematic diagram of the static data intensity division process provided in an embodiment of this disclosure.

[0019] Figure 3 This is a schematic diagram illustrating the compressed data migration, writing, and mapping update process provided in an embodiment of this application.

[0020] Figure 4 This is a schematic diagram of a NAND performance optimization device based on static data recognition and independent MCU compressed storage, provided as an embodiment of this application.

[0021] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0022] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] This method is applicable to NAND Flash storage systems, covering embedded multimedia cards (eMMC), universal flash storage (UFS), solid-state drives (SSDs), and other storage devices equipped with storage controllers. It is executed collaboratively by the main control processing unit of the storage controller and an independent microcontroller unit (MCU). The independent MCU has independent input / output (I / O) access capabilities and can directly access NAND Flash or intermediate buffers without increasing the processing burden on the host. Storage systems implementing this method must have a pre-running basic flash translation layer (FTL) with normal logical-to-physical (L2P) address mapping capabilities, and have established various statistical data tables and metadata storage areas with power-down persistence. The method's triggering conditions include three categories: periodic triggering, space triggering, and load triggering. All threshold parameters can be preset by the controller firmware or calibrated offline based on the device's total capacity, write intensity, and garbage collection (GC) frequency, and then solidified. The overall management granularity is based on the compression migration unit, completing the identification, compression storage, and full-process anomaly recovery operations for static data.

[0024] The following detailed description, in conjunction with specific embodiments, illustrates the implementation process of the NAND performance optimization method based on static data identification and independent MCU compressed storage described in this application. It should be noted that this embodiment is merely for explaining this application and not for limiting the scope of protection of this application. Any conventional adjustments or substitutions made by those skilled in the art to the steps without departing from the concept of this application should be included within the scope of protection of this application.

[0025] like Figure 1 As shown in the figure, this application discloses a schematic diagram of a NAND performance optimization method based on static data identification and independent MCU compressed storage, including the following method steps: S1, the main control processing unit counts the host write events of the compression migration unit, and the host write events only include write commands initiated by the host; S2, when the triggering condition is met, select the compression migration units that do not exceed the first threshold in the preset first time window and whose time interval from the most recent host write event is not less than the cooling time threshold, as candidate units; S3, the independent MCU reads candidate cell data under read consistency protection, calculates the page-level fingerprint of each physical page, and aggregates to generate the first cell-level fingerprint for persistent storage; S4. After the preset second time window, the independent MCU reads the candidate unit data again to calculate the secondary unit-level fingerprint and compares it with the first unit-level fingerprint. If they match, it is determined to be strong static data; if they do not match, the page-level fingerprint is compared page by page to obtain the difference page ratio. If the difference page ratio does not exceed the second threshold, it is determined to be weak static data; otherwise, it is determined to be non-static data and removed from the candidate set. S5, the independent MCU compresses data according to the selected strategy, generates compressed data blocks and block index tables, and allocates target physical space; among them, full compression is selected for strong static compression, and main compression plus differential log is selected for weak static compression. S6, the main control processing unit sets write protection for the candidate unit. The independent MCU first writes the compressed data block, block index table and shadow mapping item to the non-volatile memory, and writes the preparation record in the mapping log and forces it to be written to disk. The main control processing unit writes the commit record in the mapping log and forces it to be written to disk. It atomically updates the mapping entry of the candidate unit in the main mapping table to an indirect descriptor pointing to the shadow mapping item. S7, the main control processing unit marks the original physical block of the candidate unit as delayed recycling and retains the data of the original physical block within the preset delayed recycling window.

[0026] In some embodiments, for step S1, the main control processing unit counts the host write events of the compression migration unit, and the host write events only include write commands initiated by the host.

[0027] For example, the logical address space is divided into multiple compression migration units. Each compression migration unit is a continuous range of logical addresses, serving as the basic granularity for static data identification, write event statistics, and compression migration. Its length can be configured to a fixed value, such as 256KB, 1MB, or 4MB. This parameter value is pre-set by the system firmware based on the device's storage capacity and I / O access characteristics, requiring no dynamic updates. The compression migration units are divided by performing a rounding operation on the logical block address, ensuring that each logical block address uniquely belongs to one compression migration unit. There is no address overlap or omission of logical block addresses between units. After division, the storage system will perform all subsequent related operations using this unit as the basic granularity, including host write event statistics, content fingerprint calculation, and compression strategy selection.

[0028] In some embodiments, for step S2, when the triggering condition is met, compression migration units that do not exceed a first threshold in the preset first time window and whose time interval from the most recent host write event is not less than a cooling duration threshold are selected as candidate units.

[0029] According to embodiments of this disclosure, the main control processing unit statistically analyzes host write events of each compression migration unit. A host write event is defined as only including write commands initiated by the host. Data migration operations performed by the storage system due to mechanisms such as garbage collection, wear leveling, or copy-on-write (COW) are not included in the statistical scope of host write events. This definition avoids erroneous statistics of write events caused by non-host operations, improving the accuracy of static data identification from the statistical source. The statistics of host write events adopt a sliding window counting method, where the time range of the sliding window is a preset first time window. The value range of the first time window is from hours to days, and its parameter values ​​are obtained from performance testing and calibration in actual application scenarios before the device leaves the factory. It does not require dynamic updates after being pre-configured in the firmware.

[0030] Specifically, when the main control processing unit receives any host write command, it first calculates the compression migration unit to which the logical block address corresponding to the write command belongs. Then, it performs an update operation on the entry for the corresponding compression migration unit in the write event statistics table. This involves incrementing the write event count by 1 and recording the timestamp corresponding to the host write event. For each count value in the write event statistics table, the main control processing unit verifies each write event timestamp in real time. When the interval between the timestamp of a write event and the current time exceeds the first time window, the write event is removed from the count value to ensure that the write event count value always represents the number of valid host write events for the compression migration unit within the first time window. The write event statistics table is deployed in the controller memory. To prevent the loss of statistical data due to device restarts, the full data of the write event statistics table can be persisted to a reserved metadata block in the NAND Flash after each statistical cycle. The update frequency of the statistical data is consistent with the reception frequency of host write commands to ensure the real-time nature of the statistical results.

[0031] In actual implementation, when the preset trigger conditions are met, the main control processing unit performs a candidate unit screening operation on all compression migration units. There are three types of trigger conditions, and screening can be started when any one of them is met: periodic trigger, which is triggered once every preset period duration; space trigger, which is triggered when the proportion of free blocks in NAND Flash is lower than a preset threshold or when the high-performance storage area needs to be expanded; and load trigger, which is triggered when the I / O of the storage system is at a low point and the queue depth is lower than a preset threshold. The thresholds for the three types of trigger conditions are preset by the firmware and can take effect independently or in combination according to the actual operating status of the device.

[0032] Candidate units must meet two threshold conditions simultaneously. The first condition is that the number of valid host write events of the compressed migration unit within the first time window does not exceed a first threshold. The first threshold is a non-negative integer, and its parameter value is preset by the system firmware based on the writing characteristics of static data, such as 0 or 1, to filter out compressed migration units with very few host write events. The second condition is that the time interval between the compressed migration unit and the most recent host write event is not less than a cooling time threshold. The cooling time threshold ranges from days to weeks, and its parameter value is preset in combination with the erase and write characteristics of NAND Flash and the life cycle characteristics of static data, to filter out compressed migration units with no host writes for a long time.

[0033] This dual-threshold screening mechanism can solve the problem of static data misjudgment caused by simply relying on the number of physical writes or simple read and write frequency statistics in the existing technology. By constraining the number of writes and the write interval, it eliminates false data changes caused by internal system mechanisms and only adds the compression migration units that truly meet the low write characteristics to the candidate table as candidate units. The candidate table records the identification information of the candidate units and the timestamp of entering the candidate set.

[0034] In some embodiments, for step S3, the independent MCU reads candidate cell data under read consistency protection, calculates the page-level fingerprint of each physical page, aggregates and generates the first cell-level fingerprint, and persists it for storage.

[0035] After the candidate unit selection is completed, the independent MCU reads the candidate unit data and calculates the content fingerprint under read consistency protection. The read consistency protection is implemented by the independent MCU sending a protection request to the main control processing unit. After receiving the request, the main control processing unit applies a short-term write block to the logical block address range corresponding to the candidate unit. That is, during the fingerprint calculation, any host write command for the candidate unit is prohibited from entering the submission path, ensuring the uniqueness and stability of the data source during the fingerprint calculation process. This mechanism can solve the problem of content changes caused by data writing during the fingerprint calculation process and ensure the accuracy of the fingerprint calculation results.

[0036] For example, during read consistency protection, if the master control processing unit receives a host write request for the protected candidate unit, it will not directly reject or submit the request, but will temporarily store it in the pending write queue. At the same time, it will mark the candidate unit as a candidate failure pending judgment in the candidate table. This status will be checked first in the subsequent fingerprint verification steps. If there is a pending write request, the candidate unit will be directly judged as failure and removed from the candidate set.

[0037] After applying read consistency protection, the independent MCU begins reading candidate cell data and calculating content fingerprints. First, it reads all physical page data within the candidate cell page by page in page order into an intermediate buffer. The size of a physical page is configurable to 4KB, a parameter value that matches the physical storage characteristics of NAND Flash and is preset by the firmware. For each physical page data read into the intermediate buffer, the independent MCU calculates its page-level fingerprint. The page-level fingerprint is the CRC32C checksum calculated from the physical page data. The CRC32C checksum algorithm requires no additional configuration; only consistency in the calculation process needs to be ensured.

[0038] After completing the page-level fingerprint calculation for all physical pages, the independent MCU aggregates the page-level fingerprint sequence sequentially through a rolling aggregation function to generate the first cell-level fingerprint of the candidate cell. The operation rules of the rolling aggregation function are fixed rules preset by the system firmware, such as a combination of bitwise rotation XOR and multiplication. The operation rules remain unchanged during device operation to ensure that the aggregation results of the same data sequence at different times are unique and repeatable.

[0039] The rolling aggregation operation begins by iterating from the first element of the page-level fingerprint sequence. The initial hash value is a fixed constant preset by the firmware. Each iteration combines the result of the previous iteration with the current page-level fingerprint using the rolling aggregation function. The final iteration result is the first cell-level fingerprint. The independent MCU organizes the calculated first cell-level fingerprint and the page-level fingerprint sequence into a fingerprint record, writes it to the fingerprint table, and performs a forced disk write operation to achieve power-off persistence of the fingerprint data. The fingerprint table also records the timestamp of the first fingerprint calculation. After the fingerprint data persistence is complete, the independent MCU sends a notification to the main control processing unit to release the read consistency protection for the candidate cell and restore its normal write command submission path.

[0040] Understandably, thereafter, the main control processing unit continues to monitor the host write events of the candidate unit through the existing write event statistics table (using a sliding window mechanism). When fingerprint verification is triggered after the preset second time window, the main control processing unit first queries the candidate unit's valid host write event count in the write event statistics table since the first fingerprint calculation timestamp. If the count is greater than 0, it indicates that a host write occurred during this period, and the candidate unit is directly determined to be invalid and removed from the candidate set. There is no need to start an independent MCU for data reading and fingerprint verification. Otherwise, the candidate unit is added to the review queue. The review queue processes the devices in batches according to their idle status. When the queue depth is low and the idle ratio of the NAND Flash channels is higher than the preset threshold, the main control processing unit triggers the independent MCU to perform fingerprint review. Otherwise, the processing is delayed to avoid the fingerprint review operation affecting the normal I / O performance of the storage system.

[0041] In some embodiments, for step S4, after a preset second time window, the independent MCU reads the candidate cell data again to calculate the secondary cell-level fingerprint and compares it with the first cell-level fingerprint. If they match, it is determined to be strong static data; if they do not match, the page-level fingerprint is compared page by page to obtain the difference page ratio. If the difference page ratio does not exceed the second threshold, it is determined to be weak static data; otherwise, it is determined to be non-static data and removed from the candidate set.

[0042] Please see Figure 2 , Figure 2 This is a schematic diagram of the static data strength partitioning process provided in an embodiment of this disclosure. In S201, the main control processing unit checks whether there are any pending write requests or new host write events for the candidate unit since the first fingerprint calculation. When the independent MCU performs fingerprint verification on the candidate units in the verification queue, the main control processing unit first checks whether there are any pending write requests or new host write events for the candidate unit since the first fingerprint calculation. If so, the candidate unit is directly determined to be invalid, removed from the candidate set, and the corresponding record in the fingerprint table is marked as expired. If not, the independent MCU reads the physical page data of the candidate unit again, calculates the page-level fingerprint in a manner completely consistent with the first fingerprint calculation, and aggregates it to generate a secondary unit-level fingerprint, ensuring that the two fingerprint calculation processes are identical and guaranteeing the validity of the comparison results.

[0043] In S202, the independent MCU compares the secondary unit-level fingerprint with the initial unit-level fingerprint to determine the static data strength level. If they are completely identical, the candidate unit is determined to be strongly static, meaning that the data content of the candidate unit has not changed at all within the second time window, exhibiting extremely strong static properties. If the secondary unit-level fingerprint is inconsistent with the initial unit-level fingerprint, the independent MCU performs a page-by-page comparison of the page-level fingerprint sequences obtained from the two calculations to locate the pages where the content has changed. Simultaneously, it records a difference page bitmap. This difference page bitmap serves as identification information for marking the physical page locations where the page-level fingerprint within the compression migration unit has changed, providing accurate information for subsequent compression migration strategy selection.

[0044] After locating the difference page, the independent MCU calculates the difference page ratio. The formula for calculating the difference page ratio is as follows: ,in To reduce the number of difference pages within a migration unit, To compress the total number of physical pages within the migration unit, the independent MCU compares the calculated percentage of differing pages with two preset thresholds to determine the static level of the candidate unit. If the percentage of differing pages does not exceed the first difference threshold, it is considered weakly static data; otherwise, it is considered non-static data and directly removed from the candidate set. The first difference threshold can range from 0 to 10%, and its parameter value is obtained from statistical analysis of a large number of static data samples. It does not need to be dynamically updated after being preset in the firmware. Weakly static data can be processed using differentiated compression migration strategies.

[0045] This fingerprint verification and static level determination mechanism can solve the problem of misjudging data areas that remain unchanged for a long time but are occasionally updated slightly in the existing technology. Through accurate content comparison and level division, it matches appropriate compression strategies for units with different static attributes, avoiding indiscriminate full compression operations.

[0046] In some embodiments, for step S5, the independent MCU compresses the data according to the selected strategy, generates compressed data blocks and block index tables, and allocates target physical space; wherein, full compression is selected for strong static compression, and main compression plus differential logging is selected for weak static compression.

[0047] According to embodiments of this disclosure, the storage system selects a corresponding compression migration strategy based on the static level of the candidate unit. If the static level of the candidate unit is strong static, a full compression migration strategy is adopted to perform full compression operations on all physical page data within the candidate unit. If the static level of the candidate unit is weak static, a main compression plus differential log strategy is adopted. This strategy can solve the write amplification and tail latency problems caused by overall recompression of small-updated data areas. By separating stable data and differential data, it reduces invalid compression and migration operations and improves the performance of the storage system.

[0048] The specific implementation of the main compression plus differential log strategy is as follows: First, the compression migration unit is divided into multiple compression blocks of fixed size. The compression block is a logical block composed of consecutive physical pages, and its size can be configured to 64KB. The firmware presets the size based on the efficiency of the compression algorithm and the erase / write granularity of the NAND Flash. An independent MCU performs differential page detection on each compression block. If a compression block does not contain any differential pages, the compression block is included in the main data and compression is performed. If a compression block contains differential pages, the compression block is not compressed, and the differential page data inside is recorded through a differential log. The differential log records the physical location and data content of the differential pages, providing a basis for subsequent read redirection and data recovery.

[0049] The independent MCU performs compression operations on the data of the candidate units according to the selected compression migration strategy, generating compressed data blocks. For each generated compressed data block, the independent MCU calculates its block-level checksum using the CRC32C checksum algorithm to ensure the integrity of the compressed data block. After generating all compressed data blocks, the independent MCU generates a block index table. The block index table records the offset address, data length, and block-level checksum of each compressed data block in the physical write area, as well as the sequence number of the compressed block and the identification information of the compression migration unit to which it belongs. The block index table provides accurate indexing for subsequent compressed data reading and verification.

[0050] After the block index table is generated, the main control processing unit allocates target physical space for the compressed data from the free block pool of NAND Flash. The target physical space can be a contiguous range of physical pages or a linked list of physical pages. The allocation rules follow the physical address management rules of FTL to ensure no address conflicts or overlaps. At the same time, the main control processing unit also allocates dedicated metadata physical space for the block index table and shadow mapping entries. The shadow mapping entries are metadata structures that record the physical layout and verification information of the compressed migration units after compression. Their fields include the starting logical address, length, compression algorithm identifier, physical address of the block index table, physical address of the compressed data, unit-level fingerprint, version number, etc. All field information is obtained from the calculation and allocation results of the previous steps to ensure the accuracy and completeness of the information.

[0051] In some embodiments, for step S6, the main control processing unit sets write protection for the candidate unit, the independent MCU first writes the compressed data block, block index table and shadow mapping item to non-volatile memory, and writes a preparation record in the mapping log and forces it to be written to disk; the main control processing unit writes a commit record in the mapping log and forces it to be written to disk, and atomically updates the mapping entry of the candidate unit in the main mapping table to an indirect descriptor pointing to the shadow mapping item.

[0052] Please see Figure 3 , Figure 3 This is a schematic diagram of the compressed data migration and mapping update process provided in an embodiment of this application. In S301, after completing the generation of compressed data blocks and the allocation of target physical space, the main control processing unit sets write protection for the candidate unit. Specifically, the main control processing unit sets the state of the candidate unit to the migration write protection state. In this state, the host write request received by the main control processing unit for the candidate unit is not directly submitted to the NAND Flash, but is temporarily stored in the write queue to be processed. At the same time, the read request initiated by the host still points to the data of the original physical block according to the original L2P mapping relationship, ensuring the consistency of host read and write semantics during the compression migration process and avoiding host I / O access abnormalities caused by the compression migration operation.

[0053] In S302, a two-phase commit operation is executed collaboratively by an independent MCU and the main control processing unit. The two-phase commit mechanism addresses the risk of power-off inconsistency caused by the lack of atomicity in mapping switching during the compression migration process in the independent MCU background. Its principle is to divide the writing of compressed data and the updating of mapping relationships into two independent phases, and the compression migration only officially takes effect after the second phase is completed. This ensures that the mapping switching operation either succeeds completely or fails completely, without any intermediate half-switching state, fundamentally avoiding the problems of mapping tearing and data irrecoverability.

[0054] Specifically, the first stage of the two-stage submission is the preparation stage, in which the independent MCU performs the specific operations. First, the independent MCU writes the compressed data block, block index table, and shadow mapping item into the NAND Flash non-volatile memory according to the target physical space allocated by the main control processing unit. During the writing of the compressed data block, the independent MCU records the actual physical page range and block-level check value for each compressed data block written, which is convenient for subsequent verification and reading.

[0055] After all compressed data blocks have been written, the independent MCU performs a sampling readback verification operation. The sampling ratio for the sampling readback verification is a preset value, ranging from 1 / 32 to 1 / 8. This parameter value is preset by the firmware after balancing data verification accuracy and verification efficiency. The independent MCU randomly selects the written compressed data blocks according to this sampling ratio, reads them back from the NAND Flash, and recalculates their block-level checksums. It then compares these checksums with the block-level checksums recorded during the writing process. If the checksums do not match, the compressed data block writing is deemed to have failed, and a rollback process is immediately triggered, with no further commit operations performed. If the checksums match, the compressed data block writing is deemed successful, and subsequent steps continue.

[0056] After the compressed data block, block index table, and shadow mapping item are all successfully written, the independent MCU writes a preparation record to the mapping log. The mapping log is a sequential log of persistent metadata updates, deployed in a reserved metadata block of NAND Flash, and has power-loss persistence characteristics. The preparation record contains information such as record type, starting logical address of the compressed migration unit, version number, physical address of the shadow mapping item, and record-level checksum. All information is used for subsequent power-loss recovery determination. After the preparation record is written, the independent MCU immediately performs a forced disk write operation to ensure that the record is persisted to NAND Flash. Even if the device loses power at this time, the subsequent recovery process can determine the status of the compressed migration through this record.

[0057] In this embodiment, the commit phase is initiated only after all operations in the preparation phase are completed and the sampling readback verification passes. The main control processing unit first writes a commit record to the mapping log. The commit record includes information such as the record type, the starting logical address of the compressed migration unit, the version number, the indirect descriptor pointing to the shadow mapping item, and the record-level checksum. The indirect descriptor contains the compressed state identifier and the physical address of the shadow mapping item, used by the main mapping table to identify the compressed migration unit as compressed and accurately locate the shadow mapping item. After the commit record is written, the main control processing unit immediately performs a forced disk write operation. Successful forced disk write of the commit record serves as the sole criterion for the compressed migration to take effect externally. This criterion ensures that the compressed migration only takes effect after the commit record is persisted. If a power outage occurs before the commit record is written to disk, the subsequent power outage recovery process will determine that the compressed migration has not taken effect, avoiding abnormal updates to the mapping relationship.

[0058] In S303, after the commit record is successfully written to disk, the master control processing unit performs an atomic update operation on the master mapping table. Specifically, the mapping entry corresponding to the candidate unit in the master mapping table is updated from the original page-level mapping pointer to an indirect descriptor pointing to the shadow mapping entry. The implementation of the atomic update operation ensures that the update process is either completely completed or not executed at all, with no intermediate state of partial update. After the memory update of the master mapping table is completed, the master control processing unit persists the update to the NAND Flash in the form of metadata refresh. This persistence operation can be appropriately delayed, but it must be ensured that the commit record has been written to disk, ensuring that even if a power outage occurs before the master mapping table is persisted, the subsequent power outage recovery process can reconstruct the mapping relationship of the master mapping table through the commit record in the mapping log.

[0059] It should be noted that, to ensure mutually exclusive access to the shared resource of the mapped log, the main control unit and the independent MCU synchronize via a hardware mailbox or atomic flags in shared memory. Before the independent MCU begins writing the preparation record, it must acquire and hold the log lock for the corresponding candidate unit. After writing is complete and forced to disk, the lock is released, and the main control unit is notified via an interrupt or polling flag. Upon receiving the notification, the main control unit re-acquires the lock before writing the commit record. Throughout the two-phase commit process, the independent MCU will not perform any other operations on the candidate unit until the migration process is completely completed or rolled back.

[0060] In some embodiments, for step S7, the main control processing unit marks the original physical block of the candidate unit as delayed recycling and retains the data of the original physical block within a preset delayed recycling window.

[0061] After the two-stage commit operation of the compression migration is completed, the main control processing unit marks the original physical block of the candidate unit as delayed recycling. At the same time, it retains the data of the original physical block within the preset delayed recycling window. The delayed recycling window is a preset time window with a value range from hours to days. Its parameter value is preset by the firmware in combination with the device's garbage collection frequency and the probability of misjudgment of static data. This mechanism is the core means to solve the problem of misjudgment of static data and unrecoverable media abnormalities caused by the lack of a rollback mechanism after compression migration. By retaining the data of the original physical block, it provides a rollbackable physical data foundation for various abnormal scenarios.

[0062] Specifically, the main control processing unit adds a delayed reclamation flag to the original physical block. This flag excludes the original physical block from the reclaimable set for garbage collection, ensuring that the original physical block will not be erased by garbage collection operations within the delayed reclamation window. Within the delayed reclamation window, if the main control processing unit receives a host write request for the migrated and compressed migration unit, or if a decompression verification failure occurs during a read operation of the unit, a rollback process will be triggered immediately.

[0063] The rollback process is implemented as follows: First, the main control unit sets the state of the compression migration unit to rollback, suspends its new read path, restores the mapping entry of the unit in the main mapping table to the original mapping relationship, writes an abort record to the mapping log, and performs a forced disk write operation. The abort record contains information such as record type, starting logical address of the compression migration unit, version number, and rollback reason, providing a basis for subsequent power outage recovery. After completing the restoration of the main mapping table and the disk write of the abort record, the main control unit marks the written compression state metadata, including shadow mapping entries and block index tables, as invalid. The compressed data blocks are not immediately erased, but only marked as recyclable garbage data, which is then handled by the garbage collection mechanism according to the normal process. Finally, the pending write queue is released, and the temporarily stored host write requests are submitted according to the original mapping relationship to ensure that the host write requests are effective.

[0064] Within the delayed reclamation window, if no rollback process is triggered, when the interval between the current time and the compaction migration commit time exceeds the delayed reclamation window, the main control processing unit will clear the delayed reclamation mark of the original physical block, include it in the reclaimable set for garbage collection, and have the FTL perform erasure and reclamation operations according to normal garbage collection rules to release physical storage space. At the same time, the compaction migration unit can be marked as statically locked, indicating that its static attributes have been verified as stable and will not be re-evaluated in the short term, reducing system performance jitter caused by repeated evaluations.

[0065] In some embodiments, a power failure recovery step is also included, which is used to resolve the problems of inconsistent mapping relationships and unrecoverable data caused by abnormal power failure when the device is powered on and initialized, and to ensure that the mapping status and metadata status of all compression migration units are consistent.

[0066] Specifically, after the device is powered on, the main control processing unit first checks the device's power-off status. If an abnormal power outage is detected during the last operation, or if an unclosed record sequence is detected in the mapping log, the power outage recovery process is initiated. The main control processing unit scans all records in the mapping log in ascending order of sequence number, and groups the scanned preparation records, commit records, and abort records according to the identifier and version number of the compression migration unit, ensuring that each compression migration unit corresponds to a unique set of record combinations.

[0067] The main control processing unit performs status determination and processing on the corresponding compression migration unit according to the combination of each group of records. If a compression migration unit only has a preparation record and no corresponding commit or abort record, it means that the power failure occurred after the preparation phase was completed and before the commit phase was executed. At this time, the main control processing unit performs a rollback operation, directly clears the compression state metadata of the unit, and marks the shadow mapping item and block index table as invalid. If a compression migration unit has a commit record and no corresponding abort record, it means that the power failure occurred after the commit phase was completed and the compression migration has officially taken effect. At this time, the main control processing unit reconstructs the entry of the unit in the main mapping table as an indirect descriptor pointing to the shadow mapping item and sets the status of the unit to compression effective. If a compression migration unit has an abort record, regardless of whether there is a preparation record or a commit record, the abort record shall be used as the standard, and a rollback operation shall be performed to restore the original mapping relationship of the unit.

[0068] After completing the status determination and processing of all compression migration units, the main control processing unit performs shadow mapping item integrity verification on all units with a compression effective status. It recalculates the verification value of the shadow mapping item and the verification value of the block index table, and compares them with the stored verification value. If the verification matches, the shadow mapping item is determined to be complete and valid. If the verification does not match, and the original physical block of the unit is still within the delayed recycling window, the rollback process is immediately triggered to restore the original mapping relationship to ensure the access availability of the host. If the original physical block has exceeded the delayed recycling window and has been garbage collected, the physical block is marked as a bad block, and the upper-layer error handling mechanism is triggered to report the data access error to the host to prevent the host from reading the corrupted compressed data.

[0069] Therefore, this method achieves accurate identification of static data through a dual-threshold screening and content fingerprint verification mechanism, solving the problem of misjudgment of static data in existing technologies. By selecting differentiated compression migration strategies based on static level, it effectively avoids write amplification and tail latency caused by overall recompression of small-update data areas. The atomic switching of mapping relationships is achieved through a two-phase commit mechanism, combined with delayed recycling and rollback mechanisms, solving the risk of power failure inconsistency caused by the lack of atomicity and rollback mechanism during compression migration. Through the power failure recovery process, the consistency between the mapping state and the metadata state after abnormal power failure of the device is ensured. The entire process is executed collaboratively by the main control processing unit and the independent MCU, without increasing the processing burden of the host. It achieves efficient compression and storage of static data in the NAND Flash storage system, effectively improving the storage space utilization and overall I / O performance of the storage system, and eliminating problems such as performance jitter, write amplification, and unrecoverable power failure data during static data compression.

[0070] It should be noted that although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the steps depicted in the flowchart can be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0071] Please see Figure 4 , Figure 4 This application provides a NAND performance optimization device based on static data identification and independent MCU compressed storage. This device embodiment is similar to... Figure 1 Corresponding to the illustrated method embodiment, this device can be specifically applied to various electronic devices. The device specifically includes: The event statistics module 401 is used to count the host write events of the compression migration unit through the main control processing unit. The host write events only include write commands initiated by the host. The candidate filtering module 402 is used to filter out compression migration units that, when the triggering condition is met, have a host write event count within a preset first time window that does not exceed a first threshold and a time interval between the most recent host write event and a cooling duration threshold, as candidate units. The fingerprint generation module 403 is used to read candidate cell data under read consistency protection through an independent MCU, calculate the page-level fingerprint of each physical page, aggregate and generate the first cell-level fingerprint, and persistently store it. The static strength determination module 404 is used to read the candidate unit data again through an independent MCU after a preset second time window to calculate the secondary unit-level fingerprint and compare it with the first unit-level fingerprint. If they are consistent, it is determined to be strong static data; if they are inconsistent, the page-level fingerprint is compared page by page to obtain the difference page ratio. If the difference page ratio does not exceed the second threshold, it is determined to be weak static data; otherwise, it is determined to be non-static data and removed from the candidate set. The compression processing module 405 is used to compress data according to the selected strategy through an independent MCU, generate compressed data blocks and block index tables, and allocate target physical space; among them, full compression is selected for strong static compression, and main compression plus differential log is selected for weak static compression. The atomic update module 406 is used to set write protection for candidate units through the main control processing unit. The independent MCU first writes the compressed data block, block index table and shadow mapping item into the non-volatile memory, and writes a preparation record into the mapping log and forces it to be written to disk. The main control processing unit writes a commit record into the mapping log and forces it to be written to disk. The mapping entry of the candidate unit in the main mapping table is an indirect descriptor pointing to the shadow mapping item. The delayed recycling module 407 is used to mark the original physical block of the candidate unit as delayed recycling through the main control processing unit, and retain the data of the original physical block within the preset delayed recycling window.

[0072] Those skilled in the art will clearly understand that the technical solutions of the embodiments of this application can be implemented by means of software and / or hardware. In this specification, "unit" and "module" refer to software and / or hardware that can independently complete or cooperate with other components to complete a specific function, wherein the hardware may be, for example, a field-programmable gate array (FPGA), an integrated circuit (IC), etc.

[0073] Each processing unit and / or module in the embodiments of this application can be implemented by an analog circuit that implements the functions described in the embodiments of this application, or by software that executes the functions described in the embodiments of this application.

[0074] Based on the same inventive concept, this application also provides an electronic device. The method corresponding to the electronic device can be the method in the foregoing embodiments, and its problem-solving principle is similar to that method. The electronic device provided in this application includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the methods and / or technical solutions of the foregoing embodiments of this application.

[0075] Figure 5 The diagram illustrates the structure of an electronic device suitable for implementing the methods and / or technical solutions in the embodiments of this application. The electronic device includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) 502 or a program loaded from a storage section 508 into a random access memory (RAM) 503. The RAM 503 also stores various programs and data required for system operation. The CPU 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input section 506, an output section 507, a communication section 509, and an input / output (I / O) interface 505 are also connected to the bus 504.

[0076] In particular, the methods and / or embodiments in this application can be implemented as computer software programs. For example, the embodiments disclosed in this application include a computer program product comprising a computer program carried on a storage medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 501, it performs the functions defined in the methods of this application.

[0077] Another embodiment of this application provides a computer-readable storage medium having computer program instructions stored thereon, which can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of this application described above.

[0078] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of electronic devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0079] The above description represents the preferred embodiments of the present invention. It should be noted that, for those skilled in the art, various improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A NAND performance optimization method based on static data identification and independent MCU compressed storage, characterized in that, include: The host write events of the statistical compression migration unit include only write commands initiated by the host; When the triggering condition is met, the compression migration units that have a host write event count of no more than a first threshold within a preset first time window and a time interval of no less than the cooling duration threshold are selected as candidate units. Under read consistency protection, candidate cell data is read, page-level fingerprints of each physical page are calculated, and aggregated to generate the first cell-level fingerprint; After a preset second time window, the secondary unit-level fingerprint is calculated and compared with the first unit-level fingerprint to determine the static strength level of the data. Compressed data blocks and block index tables are generated based on the static strength level of the data, and target physical space is allocated; among them, full compression is used for strong static data, and main compression plus differential log is used for weak static data. Write protection is set for candidate cells, compressed data blocks, block index tables, and shadow mapping entries are written to non-volatile memory, and preparation and commit records are written to the mapping log and forced to disk; the mapping entries of candidate cells in the main mapping table are atomically updated to indirect descriptors pointing to shadow mapping entries. The original physical block of the candidate unit is marked as delayed recycling, and the data of the original physical block is retained within the preset delayed recycling window.

2. The NAND performance optimization method based on static data identification and independent MCU compressed storage according to claim 1, characterized in that, The host write events are counted using a sliding window counting method. For each host write event, a corresponding timestamp is recorded. When the interval between the timestamp of a host write event and the current time exceeds the first time window, the host write event is removed from the write event count.

3. The NAND performance optimization method based on static data identification and independent MCU compressed storage according to claim 1, characterized in that, During read consistency protection, if the main control processing unit receives a host write request for a protected candidate unit, it temporarily stores the host write request in the pending write queue and marks the candidate unit as a candidate failure pending determination in the candidate set; during subsequent fingerprint comparison, if a pending write request is detected, the candidate unit is directly determined to be failure and removed from the candidate set.

4. The NAND performance optimization method based on static data identification and independent MCU compressed storage according to claim 1, characterized in that, The page-level fingerprint is the CRC32C check value calculated for each physical page of data; the initial unit-level fingerprint is generated by sequentially aggregating the page-level fingerprint sequence through a rolling aggregation function, and the operation rules of the rolling aggregation function are fixed rules preset by the system firmware.

5. The NAND performance optimization method based on static data identification and independent MCU compressed storage according to claim 1, characterized in that, in, After the independent MCU writes the compressed data block into the non-volatile memory, it performs a sampling readback verification: it randomly selects the written compressed data block according to a preset ratio to read back and verify its integrity. If the verification fails, it triggers a rollback process.

6. The NAND performance optimization method based on static data identification and independent MCU compressed storage according to claim 1, characterized in that, Within the delayed recycling window, if the main control processing unit receives a host write request for the migrated compressed migration unit, or if a decompression verification failure occurs during the read operation of the unit, a rollback process is triggered: the main mapping table is restored to the original mapping relationship of the unit, an abort record is written to the mapping log and forced to disk, and the written compressed state metadata is marked as invalid.

7. The NAND performance optimization method based on static data identification and independent MCU compressed storage according to claim 1, characterized in that, It also includes a power outage recovery step: After the device is powered on, the main control processing unit scans the mapping log and determines its final status based on the combination of the preparation record, submission record and abort record corresponding to each compression migration unit. If only a preparation record exists but no corresponding commit or abort record exists, then a rollback operation is performed to clean up the compressed metadata. If there is a commit record and no abort record, the main mapping table is restored to an indirect descriptor pointing to the shadow mapping item; if there is an abort record, a rollback operation is performed to restore the original mapping relationship; and the integrity of the shadow mapping item is checked for all units in the compressed state. If the check fails and the original physical block is still within the delayed recycling window, a rollback is triggered; otherwise, the data is marked as abnormal.

8. A NAND performance optimization device based on static data recognition and independent MCU compressed storage, characterized in that, include: The event statistics module is used to count host write events of the compression migration unit, and the host write events only include write commands initiated by the host; The candidate filtering module is used to filter out compression migration units that, when the triggering conditions are met, have a host write event count within a preset first time window that does not exceed a first threshold and a time interval between the most recent host write event and a cooling duration threshold, as candidate units. The fingerprint generation module is used to read candidate cell data through an independent MCU under read consistency protection, calculate the page-level fingerprint of each physical page, and aggregate to generate the first cell-level fingerprint; The static strength determination module is used to calculate the secondary unit-level fingerprint after a preset second time window, compare it with the first unit-level fingerprint, and determine the static strength level of the data. The compression processing module is used to generate compressed data blocks and block index tables based on the data static strength level, and allocate target physical space; among them, strong static data uses full compression, and weak static data uses main compression plus differential logs. The atomic update module is used to set write protection for candidate cells, write compressed data blocks, block index tables, and shadow mapping entries to non-volatile memory, and write preparation and commit records to the mapping log and force disk write; the mapping entries of candidate cells in the atomic update master mapping table are indirect descriptors pointing to shadow mapping entries. The delayed recycling module is used to mark the original physical block of the candidate unit as delayed recycling and retain the data of the original physical block within a preset delayed recycling window.

9. An electronic device, characterized in that, include: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

10. A storage medium storing computer program instructions thereon, characterized in that, The computer program instructions can be executed by a processor to implement the method as described in any one of claims 1-7.

Images