Data processing method, chip, storage medium and computer program product

Abstract

The embodiments of the present application provide a data processing method, chip, storage medium and computer program product, which relate to the field of data processing technology. The method is applied to a file system for managing NAND flash memory, and the method includes: obtaining a write instruction; dynamically determining the current working mode based on the real-time task scenario and / or the data characteristics of the data to be stored; the current working mode includes one of a high-performance mode and a high-life mode; based on the current working mode and / or the compression strategy, dynamically determining the compression ratio used when storing the data to be stored; the compression ratio corresponding to the high-life mode is greater than the compression ratio for the high-performance mode; determining the target storage block for storing the data to be stored in the NAND flash memory; based on the capacity of the storage page, compressing the data to be stored according to the determined compression ratio and storing it in the target storage block. The adoption of this solution improves storage efficiency, storage performance and extends the service life of the NAND flash memory.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, chip, storage medium and computer program product. Background Art

[0002] With the advancement of NAND flash (not-and-gate) storage technology, the manufacturing process for single-level cell (SLC) NAND Flash has been gradually reduced, from the traditional 38nm and 32nm processes to today's 24nm and even 16nm. This progress has brought higher storage density and certain performance improvements, but it has also brought significant problems. Due to the reduction in transistor size, the program / erase (P / E) cycle endurance has been significantly reduced. Traditional large-scale NAND Flash processes (such as 38nm and 32nm) can support more than 100,000 P / E cycles, while newer 24nm and 16nm processes can only support approximately 50,000 to 80,000 P / E cycles.

[0003] Therefore, how to extend the service life of NAND Flash is a technical problem that needs to be solved urgently. Summary of the Invention

[0004] The present application provides a data processing method, chip, storage medium, and computer program product, which can extend the service life of NAND Flash.

[0005] In a first aspect, an embodiment of the present application provides a data processing method, which is applied to a file system that manages a NAND flash memory for a NAND gate. The data processing method includes: obtaining a write instruction to store data to be stored in the NAND flash memory; dynamically determining a current working mode based on a real-time task scenario and / or data characteristics of the data to be stored; the data characteristics include: one of cold data and hot data; the current working mode includes one of a high-performance mode and a high-life mode; based on the current working mode and / or the compression strategy, dynamically determining a compression ratio to be used when storing the data to be stored; wherein the compression ratio corresponding to the high-life mode is greater than the compression ratio corresponding to the high-performance mode; determining a target storage block for storing the data to be stored in the NAND flash memory; based on the capacity of the storage page, compressing the data to be stored according to the determined compression ratio and storing it in the target storage block.

[0006] Since data is written in units of storage pages, the data to be stored is compressed according to a determined compression ratio based on the capacity of the storage page and then stored in the target storage block. In specific implementation, the following methods can be used: (1) The data to be stored is compressed according to the determined compression ratio. If the compressed size is not larger than the capacity of the storage page, it is directly stored in the storage page; (2) If the compressed size is larger than the capacity of the storage page, the compressed data is split into multiple data blocks within the storage page capacity according to the capacity of the storage page, and then the split data blocks are respectively stored in the data pages of the target storage block; (3) When the data to be stored is large, in addition to splitting after compression, it can also be split before compression. Specifically, based on the determined compression ratio, the maximum storage capacity corresponding to the storage page (i.e., the capacity before compression, such as R) is estimated, and the data to be stored is split into data blocks not larger than R in size. The split data blocks are then compressed according to the determined compression ratio, and the compressed data are respectively stored in the data pages of the target storage block.

[0007] This solution significantly improves the storage efficiency and performance of file systems managing NAND flash memory. This method first obtains write instructions to the NAND flash memory and dynamically selects either a high-performance mode or a high-lifespan mode based on the real-time task scenario and / or data characteristics (cold or hot data). In high-performance mode, the system prioritizes data processing speed, making it suitable for hot data that requires rapid access. In high-lifespan mode, the system maximizes storage space by increasing the compression ratio, making it particularly suitable for long-term cold data. This solution not only allows for timely adjustment of the operating mode, but also allows the compression ratio to vary based on the current operating mode and / or compression strategy, ensuring a consistently optimized storage strategy. Because data is compressed at a predetermined compression ratio before being stored, the high compression ratio achieved in high-lifespan mode effectively reduces the NAND flash memory space occupied by the data being stored, thereby reducing the frequency of writes and erases to the flash memory cells and significantly extending the lifespan of the NAND flash memory.

[0008] In combination with the first aspect, in a possible implementation of the first aspect, determining a target storage block for storing the data to be stored in the NAND flash memory includes: determining a target storage block for storing the data to be stored in the NAND flash memory based on bad block prediction and wear leveling strategy.

[0009] This embodiment improves NAND flash memory space utilization and reliability by introducing bad block prediction and a wear-leveling strategy to determine target storage blocks. Bad block prediction effectively avoids storing data in potentially bad blocks, reducing the risk of data loss and storage errors. Furthermore, wear-leveling evenly distributes write operations, extending the overall lifespan of the NAND flash memory and reducing the likelihood of performance degradation or data loss due to excessive wear of some storage blocks.

[0010] In combination with the first aspect, in a possible implementation of the first aspect, when there are multiple alternative storage blocks, a target storage block for storing data to be stored is determined in the NAND flash memory, including: determining a storage block with the largest weight among the multiple alternative storage blocks as the target storage block; wherein the weight of the storage block = α×P / E count weight + β×write frequency weight + γ×reliability rating weight, wherein α, β, and γ are weight coefficients and are constants; the P / E count weight is a remaining proportion of life based on the P / E cycle count, and the P / E count weight = 1-(current P / E count / maximum P / E cycle limit); the write frequency weight = the most recent write time interval / the longest write time interval; and the reliability rating weight is a value corresponding to the reliability level of the storage block.

[0011] This embodiment comprehensively considers multiple factors, including P / E count, write frequency, and reliability rating, when determining target storage blocks, providing a scientific and comprehensive weighted evaluation system for storage block selection. This solution ensures that data is stored on storage blocks with long lifespans, low write frequencies, and high reliability. Furthermore, in some possible implementations, the importance of different factors can be flexibly configured by adjusting the weight coefficients. This helps further optimize storage resource allocation and improve the overall performance and stability of the storage system.

[0012] In combination with the first aspect, in a possible implementation of the first aspect, the data to be stored is compressed according to a determined compression ratio and then stored in a target storage block, including: when the data to be stored is smaller than the capacity of the storage page and the set with at least one adjacent data to be stored is smaller than a preset value, the data to be stored and at least one adjacent data to be stored are merged, and then compressed according to the determined compression ratio and stored in a storage page in the target storage block.

[0013] This solution effectively reduces storage fragmentation and wasted storage space by merging small data blocks and compressing them for storage. When the data to be stored is smaller than the storage page capacity, it is merged with adjacent small data blocks to fully utilize the storage page space and improve storage efficiency. Furthermore, compressed storage further reduces the physical space occupied by data, helping to extend the life of the NAND flash memory and improve overall storage system performance.

[0014] In combination with the first aspect, in a possible implementation of the first aspect, the compression strategy includes: determining the compression ratio of the data to be stored based on one or more of the current load condition of the NAND flash memory, the wear status of the available storage blocks, and the data characteristics of the data to be stored.

[0015] This solution dynamically determines the compression ratio based on factors such as the current NAND flash load, the wear status of available storage blocks, and the characteristics of the data to be stored, achieving flexible storage strategies. This approach can meet the needs of different application scenarios and help improve storage resource utilization.

[0016] In combination with the first aspect, in a possible implementation of the first aspect, the data processing method further includes: when obtaining a read instruction, reading target data from a target data page; and when the target data is compressed data, decompressing the target data.

[0017] This solution decompresses data as needed during data reading, ensuring data accuracy and readability. When compressed data is stored, decompression can be used to restore the original data, satisfying data access requirements. This design not only saves space through compressed storage, but also ensures that data can be correctly read and used when needed.

[0018] In combination with the first aspect, in a possible implementation of the first aspect, decompressing the target data includes: dynamically allocating decompression resources to decompress the target data during a peak reading scenario.

[0019] This solution effectively addresses the high concurrency demands of data reads by dynamically allocating decompression resources during peak read times. Flexible adjustments to the allocation of decompression resources during peak read times ensure timely and accurate data reads, avoiding read delays or errors caused by insufficient resources. This dynamic resource allocation method improves the storage system's responsiveness and stability, providing a better user experience.

[0020] In a second aspect, an embodiment of the present application provides a chip comprising a processor and a communication interface, wherein the communication interface is used to receive signals and transmit the signals to the processor, and the processor processes the signals and executes the data processing method provided in the first aspect and any possible implementation thereof.

[0021] In a third aspect, an embodiment of the present application provides a computer-readable storage medium comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the data processing method provided in the first aspect and any possible implementation thereof.

[0022] In a fourth aspect, an embodiment of the present application provides a computer program product, which includes: computer program code, which, when executed, implements the data processing method provided in the first aspect and any possible implementation thereof.

[0023] It can be understood that the beneficial effects that can be achieved by the chip of the second aspect, the computer-readable storage medium of the third aspect, and the computer program product of the fourth aspect provided above can refer to the beneficial effects of the first aspect and any possible implementation thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a flow chart of the data processing method provided in Example 1 of the present application;

[0025] Figure 2 This is a flow chart of the data processing method provided in Example 2 of the present application;

[0026] Figure 3A This is a flow chart of the data processing method provided in Example 3 of the present application;

[0027] Figure 3B Schematic diagram of data reliability ratings corresponding to storage blocks provided in the third embodiment of the present application;

[0028] Figure 3C Schematic diagram of the P / E counting weights corresponding to each storage block provided in the third embodiment of the present application;

[0029] Figure 3D Schematic diagram of the write frequency weights corresponding to each storage block provided in the third embodiment of the present application;

[0030] Figure 3E Schematic diagram of reliability rating weights corresponding to each storage block provided in the third embodiment of the present application;

[0031] Figure 3F Schematic diagram of the weights of storage blocks corresponding to each storage block provided in the third embodiment of the present application;

[0032] Figure 3G This is a schematic diagram of determining the ranking of each storage block according to the weight score provided in the third embodiment of the present application;

[0033] Figure 4 This is a flow chart of the data processing method provided in Example 4 of the present application;

[0034] Figure 5 This is a flow chart of the data processing method provided in Example 5 of the present application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0036] In this application, the terms "first" and "second" are used to distinguish different objects, or to distinguish different processing of the same object, rather than to describe a specific order of objects. For example, the terms "first operation" and "second operation" are used to distinguish different operations, rather than to describe a specific order of operations. In the embodiments of this application, "plurality" refers to two or more.

[0037] Some nouns or terms involved in this application are explained below.

[0038] 1. NAND Flash, storage blocks (Blocks) and pages (Pages)

[0039] NAND Flash, Block, and Page are three key concepts in NAND Flash memory, which have a clear hierarchical structure and functional relationship.

[0040] NAND Flash is a non-volatile memory technology that retains data after a power outage. NAND Flash is widely used in various electronic devices, such as smartphones, tablets, and solid-state drives (SSDs), to store large amounts of data.

[0041] NAND Flash consists of multiple blocks, each of which is composed of multiple pages. In NAND Flash, erase operations are performed on a block-by-block basis, meaning the entire block must be erased before new data can be written. Data read and write operations are performed on a page-by-page basis, allowing the system to operate on data in smaller units and improving data access efficiency.

[0042] Therefore, NAND Flash, storage block (Block) and page are three closely related concepts in NAND Flash memory, which together constitute the data organization and management structure of NAND Flash memory.

[0043] 2. UBIFS file system

[0044] The Unsorted Block Image File System (UBIFS) is a file system designed specifically for NAND Flash. It organizes, manages, and stores file data on NAND Flash, effectively managing block and page erase and write operations. The UBIFS file system optimizes NAND Flash features, such as bad block management and wear leveling, to ensure data integrity, reliability, and efficient access.

[0045] 3. Hot Data & Cold Data

[0046] Hot data and cold data are terms used in the storage field to describe data access frequency. Hot data refers to data that is frequently accessed and modified. This type of data typically requires fast read and write speeds to meet high access demands. Examples include system log files, frequently traded financial data, and frequently accessed database records. Because hot data is frequently accessed and modified, it places high demands on the I / O performance of the storage system. Cold data refers to data that is rarely accessed or modified after being written. This type of data typically does not require frequent read and write operations, such as backup data, archived information, or old files that are rarely viewed. Cold data has relatively low access speed requirements for the storage system, but is more sensitive to storage costs and long-term preservation capabilities.

[0047] 4. Zstandard

[0048] Zstandard is a compression algorithm that enables fast, real-time data compression with high compression ratios and speeds. It is widely used in various data storage, transmission, and processing scenarios.

[0049] The present invention provides a data processing method, chip, storage medium, and computer program product, relating to the field of data processing technology. The data processing method compresses the data to be stored according to a predetermined compression ratio before storing the data, which helps reduce the amount of physical data written, improves storage efficiency, and extends the service life of NAND flash memory.

[0050] Example 1

[0051] like Figure 1 As shown, this embodiment provides a data processing method, which is applied to a file system for managing NAND Flash. The file system may be a UBIFS file system. The data processing method may include the following steps: 101 to 105.

[0052] 101. Obtain a write instruction for storing data to be stored in a NAND flash memory.

[0053] For example, when a user takes a photo and chooses to save it, the file system obtains a write instruction, which instructs: store the binary data of the photo into the built-in NAND flash memory.

[0054] 102. Dynamically determine the current working mode based on the real-time task scenario and / or data characteristics of the data to be stored.

[0055] Among them, real-time task scenarios may be high-frequency interactive or high-real-time task scenarios, such as video stream processing, game loading, etc. There may also be scenarios with low real-time requirements: such as backup data and other scenarios.

[0056] The data characteristics of the data to be stored include: one of cold data and hot data; the current working mode includes: one of high-performance mode and long-life mode.

[0057] In NAND Flash storage technology, managing hot and cold data is particularly important. Because each physical NAND Flash storage block has a limited number of erase and write cycles, frequent erases and writes accelerate block wear, shortening the lifespan of the storage device. Therefore, a sound data management strategy can extend the lifespan of storage devices.

[0058] In some possible implementations, using different compression ratios for hot and cold data can optimize storage performance and costs. For example, hot data can be stored at a low compression ratio to ensure fast access, while cold data can be compressed at a high compression ratio to conserve storage space. This strategy balances performance and capacity, improving system efficiency and reducing storage costs.

[0059] A specific scenario corresponding to the high-life mode is, for example, the persistent data archiving scenario of industrial equipment: in this scenario, the equipment is required to be able to operate stably for up to 5 years or more, during which time a large amount of data writing tasks need to be processed, but the real-time processing speed of the data is not strictly required. By enabling the high-life mode, it can be ensured that the equipment can still maintain the integrity and reliability of the data under long-term high-intensity operation, while effectively extending the life of the storage medium and reducing maintenance costs. For example, when industrial equipment enters the high-life mode, the file system automatically switches to a high compression level (such as Zstandard compression levels 9-12), and all written data is stored after being efficiently compressed.

[0060] It should be noted that the current working mode is not fixed and can be switched dynamically between different states. The switching conditions can be set as needed. For example, the working mode can be switched based on the task type. For example, when the data write priority is high, the current working mode is switched to the high-performance mode. When archiving large quantities of historical data, the current working mode is switched to the high-life mode. For example, industrial monitoring equipment records real-time monitoring data in high-performance mode during the day, and can switch to the high-life mode at night (for example, after 12 o'clock) to compress and archive the storage.

[0061] 103. Based on the current working mode and / or compression strategy, dynamically determine a compression ratio to be used when storing the data to be stored.

[0062] The compression ratio corresponding to High Life mode is greater than that of High Performance mode. As you can see, High Life mode employs a higher compression ratio, significantly extending the life of the storage media and reducing long-term storage costs and energy consumption compared to High Performance mode. While this may sacrifice some immediate performance, for data that requires long-term storage, this strategy optimizes storage resource utilization and reduces overall operating costs, achieving an effective balance between storage efficiency and cost.

[0063] In some possible implementations, the compression ratio used for the data to be stored can be determined based solely on the current working mode, or based solely on a preset compression strategy. Alternatively, the final compression ratio can be determined based on the compression ratios determined by both. The preset compression strategy can be set based on experience or actual needs. It is understood that when the compression ratio determined based on the working mode differs from the compression ratio determined by the current working mode, either one of them can be used as the standard, or the compression ratio actually used to compress the data to be stored can be determined based on a preset weight ratio or other method, both of which are feasible.

[0064] 104. Determine a target storage block for storing the data to be stored in the NAND flash memory.

[0065] 105. Based on the capacity of the storage page, compress the data to be stored according to a determined compression ratio and then store it in the target storage block.

[0066] Since data is written in units of storage pages, the data to be stored is compressed according to a determined compression ratio based on the capacity of the storage page and then stored in the target storage block. In specific implementation, the following methods can be used: (1) The data to be stored is compressed according to the determined compression ratio. If the compressed size is not larger than the capacity of the storage page, it is directly stored in the storage page; (2) If the compressed size is larger than the capacity of the storage page, the compressed data is split into multiple data blocks within the storage page capacity according to the capacity of the storage page, and then the split data blocks are respectively stored in the data pages of the target storage block; (3) When the data to be stored is large, in addition to splitting after compression, it can also be split before compression. Specifically, based on the determined compression ratio, the maximum storage capacity corresponding to the storage page (i.e., the capacity before compression, such as R) is estimated, and the data to be stored is split into data blocks not larger than R in size. The split data blocks are then compressed according to the determined compression ratio, and the compressed data are respectively stored in the data pages of the target storage block.

[0067] As you can understand, a high compression ratio helps reduce the amount of writes: the compression algorithm reduces the amount of data written and also reduces the number of P / E operations on the NAND Flash. For example, if the compression ratio is 50%, the physical write volume is reduced by half, significantly delaying the wear of the NAND Flash.

[0068] For example, in some possible implementations, compression levels may include low, medium, and high. Low compression levels are suitable for scenarios such as real-time logging systems for embedded devices. The corresponding data characteristics include frequent updates, high write rates, and short log file lifecycles. A low compression ratio (for example, a Zstandard compression level of 1-3) can be used to ensure write rates while minimizing the impact of compression operations on real-time performance. Medium compression levels are suitable for scenarios such as incremental backups of embedded database files. The corresponding data characteristics include moderate update frequency and mixed data with certain update characteristics (such as some cold data and some hot data). A medium compression ratio (for example, a Zstandard compression level of 5-7) can be used to balance compression efficiency and write speed while reducing storage space usage. High compression levels are suitable for scenarios such as firmware images or configuration files stored in embedded systems. The corresponding data characteristics include infrequently accessed or read-only cold data, such as system backup files or application configuration files. A high compression ratio (for example, a Zstandard compression level of 9-12) can be used to minimize storage usage and write volume, thereby extending the life of the memory.

[0069] Using the method provided in this embodiment, a file system managing NAND flash memory achieves significant improvements in storage efficiency and performance. This method first obtains write instructions for storing data in the NAND flash memory and dynamically selects an operating mode: high-performance mode or high-life mode, based on the real-time task scenario and / or data characteristics (cold data or hot data). In high-performance mode, the system prioritizes data processing speed, making it suitable for hot data that requires rapid access. In high-life mode, the system maximizes storage space by increasing the compression ratio, making it particularly suitable for long-term cold data. This solution not only allows for timely adjustment of the operating mode, but also allows the compression ratio to vary with the current operating mode and / or compression strategy, thus ensuring a consistently optimized storage strategy. Because data to be stored is compressed according to a predetermined compression ratio before being stored, the high compression ratio in high-life mode effectively reduces the NAND flash memory space occupied by the data to be stored, thereby reducing the frequency of writes and erases to the flash memory cells and significantly extending the lifespan of the NAND flash memory.

[0070] Example 2

[0071] like Figure 2 As shown, this embodiment provides a data processing method applied to a file system that manages NAND Flash, which may be a UBIFS file system. Differences from the first embodiment include: when determining the target storage block, this embodiment uses bad block prediction and wear leveling strategies. The data processing method provided in this embodiment includes the following steps: 201 to 205.

[0072] 201. Obtain a write instruction for storing data to be stored in a NAND flash memory.

[0073] 202. Dynamically determine a current working mode based on a real-time task scenario and / or data characteristics of the data to be stored.

[0074] Among them, real-time task scenarios may be high-frequency interactive or high-real-time task scenarios, such as video stream processing, game loading, etc. There may also be scenarios with low real-time requirements: such as backup data and other scenarios.

[0075] The data characteristics of the data to be stored include: one of cold data and hot data; the current working mode includes: one of high-performance mode and long-life mode.

[0076] 203. Based on the current working mode and / or compression strategy, dynamically determine a compression ratio to be used when storing the data to be stored.

[0077] The compression ratio corresponding to High Life mode is greater than that of High Performance mode. As you can see, High Life mode employs a higher compression ratio, significantly extending the life of the storage media and reducing long-term storage costs and energy consumption compared to High Performance mode. While this may sacrifice some immediate performance, for data that requires long-term storage, this strategy optimizes storage resource utilization and reduces overall operating costs, achieving an effective balance between storage efficiency and cost.

[0078] High-performance mode is suitable for scenarios with high real-time requirements. High-performance mode can adopt a low compression ratio or a direct non-compression strategy to reduce compression and decompression time and improve real-time response speed.

[0079] 204. Based on the bad block prediction and the wear leveling strategy, determine a target storage block for storing the data to be stored in the NAND flash memory.

[0080] Among them, the bad block prediction algorithm includes: when it is detected that the physical block is about to reach the P / E cycle limit, the system will give priority to writing compressed data to physical blocks with longer lifespans, thereby delaying the further use of high-wear blocks and ensuring the long-term reliability of data.

[0081] For example, before each write operation, the file system scans the P / E cycle counts of available NAND Flash blocks. If a block's wear is detected nearing its lifespan (e.g., 90%), data is written to newer blocks first. Blocks with high wear are marked as potential bad blocks to prevent further use.

[0082] In some possible implementations, the storage block with the least current P / E cycle count may be selected as the target storage block.

[0083] In some possible implementations, when there are multiple storage blocks that meet the bad block prediction algorithm, further consideration can be given to the following: Temperature hotspots of the storage chip. For example, when adjacent and intervening storage blocks are available as candidates, the intervening storage blocks are preferentially selected as the target storage blocks. This helps avoid concentrated use of storage blocks and reduces local aging. In some possible implementations, the target storage block can also be determined based on usage frequency. For example, based on historical usage records, the storage block with the lowest recent write frequency can be selected as the target storage block.

[0084] As you can understand, by combining data importance grading, important data (such as system configuration files) can be stored in more reliable storage blocks. These more reliable storage blocks can be combined with a bad block prediction algorithm to select the storage block with the lowest number of current P / E cycles as the more reliable storage block.

[0085] In practice, the file system can scan and record the P / E cycle counts, write frequencies, and current status of all candidate storage blocks to establish a priority queue. Before writing data, the file system selects the block with the longest lifespan and lowest current load from the candidate blocks based on priority for data storage.

[0086] Wear leveling is a key storage optimization strategy. Its goal is to evenly distribute wear across storage blocks in a storage device by properly distributing data writes, thereby extending the lifespan of the entire storage system. In long-life mode, the system achieves balanced wear distribution by combining a bad block prediction mechanism with a data write strategy. The system selects write targets based not only on the number of P / E cycles and write frequency of the physical block, but also on thermal hotspots of the storage chip and the location of the storage block to avoid localized aging.

[0087] Furthermore, the file system regularly assesses the overall wear status of the NAND Flash and dynamically adjusts storage policies based on the assessment results, concentrating new data writes on physical blocks with significantly lower wear than the average. This dynamic adjustment not only further optimizes wear leveling but also ensures the long-term stability of the storage device, meeting the needs of various application scenarios.

[0088] 205. Based on the capacity of the storage page, compress the data to be stored according to the determined compression ratio and then store it in the target storage block.

[0089] This embodiment improves NAND flash memory space utilization and reliability by introducing bad block prediction and a wear-leveling strategy to determine target storage blocks. Bad block prediction effectively avoids storing data in potentially bad blocks, reducing the risk of data loss and storage errors. Furthermore, wear-leveling evenly distributes write operations, extending the overall lifespan of the NAND flash memory and reducing the likelihood of performance degradation or data loss due to excessive wear of some storage blocks.

[0090] Example 3

[0091] like Figure 3A As shown, this embodiment provides a data processing method applied to a file system that manages NAND Flash, which file system can be a UBIFS file system. The difference from the first embodiment is that when determining the target storage block, when there are multiple storage blocks available, the storage block with the largest weight among the multiple alternative storage blocks is determined as the target storage block. The data processing method provided by this embodiment includes the following steps: 301 to 305.

[0092] 301. Obtain a write instruction for storing data to be stored in a NAND flash memory.

[0093] 302. Dynamically determine a current working mode based on a real-time task scenario and / or data characteristics of the data to be stored.

[0094] Among them, real-time task scenarios may be high-frequency interactive or high-real-time task scenarios, such as video stream processing, game loading, etc. There may also be scenarios with low real-time requirements: such as backup data and other scenarios.

[0095] The data characteristics of the data to be stored include: one of cold data and hot data; the current working mode includes: one of high-performance mode and long-life mode.

[0096] 303. Based on the current working mode and / or compression strategy, dynamically determine a compression ratio to be used when storing the data to be stored.

[0097] The compression ratio corresponding to High Life mode is greater than that of High Performance mode. As you can see, High Life mode employs a higher compression ratio, significantly extending the life of the storage media and reducing long-term storage costs and energy consumption compared to High Performance mode. While this may sacrifice some immediate performance, for data that requires long-term storage, this strategy optimizes storage resource utilization and reduces overall operating costs, achieving an effective balance between storage efficiency and cost.

[0098] 304. When there are multiple candidate storage blocks, determine a storage block with the largest weight among the multiple candidate storage blocks as a target storage block.

[0099] For example, if there are five candidate storage blocks in the system, namely Block A, Block B, Block C, Block D and Block E, the goal is to select one storage block from these five storage blocks to store the compressed data.

[0100] The number of P / E cycles, the last write interval (i.e., how long has it been since data was written) and the reliability rating of each storage block are as follows: Figure 3B shown.

[0101] The target storage blocks are screened out according to a preset priority strategy. Specifically, the weight of each storage block is calculated according to a weight formula.

[0102] The weight of the storage block = α × P / E count weight + β × write frequency weight + γ × reliability rating weight, where the P / E count weight is the remaining life proportion based on the P / E cycle count. Specifically, the P / E count weight = 1-(current P / E count / maximum P / E cycle limit), and the current P / E count is Figure 3B The P / E value in the table is 50000, and the maximum P / E cycle limit is assumed to be 50000. The P / E count weight of each storage block is as follows: Figure 3C shown.

[0103] The write frequency weight reflects the frequency of use based on the most recent write interval. The longer the interval, the higher the weight. Specifically, the write frequency weight = most recent write interval / longest write interval. Assuming the longest write interval is 100 minutes, the write frequency weight of each storage block is as follows: Figure 3D shown.

[0104] The reliability rating weight is a reliability grade for storage blocks, which is directly assigned a value (e.g., high reliability = 1.0, medium reliability = 0.8, low reliability = 0.6). High reliability means: no error checking and correcting (ECC) failure records, uniform P / E cycles, and no surrounding bad blocks. Medium reliability means: occasional ECC failure records, or a bad block nearby, but still able to work normally. Low reliability means: frequent ECC errors, or multiple bad blocks around, with a significant risk of failure. The reliability rating weights of each storage block are as follows: Figure 3E shown.

[0105] Weight coefficients: α, β, and γ control the influence of the three weights respectively. For example,

[0106] α=0.5: Prioritize P / E counting life.

[0107] β=0.3: moderate focus on write frequency.

[0108] γ=0.2: Reliability rating is considered but given slightly lower priority.

[0109] According to the weight formula of storage blocks: Storage block weight = α×P / E count weight + β×write frequency weight + γ×reliability rating weight, the weight of each storage block is as follows: Figure 3F shown.

[0110] According to the weight of the storage block, the priority of the storage block is determined as follows Figure 3G As shown, the storage block corresponding to Block C with the highest weight score can be selected as the target storage block.

[0111] 305. Based on the capacity of the storage page, compress the data to be stored according to the determined compression ratio and then store it in the target storage block.

[0112] This embodiment comprehensively considers multiple factors, including P / E count, write frequency, and reliability rating, when determining target storage blocks, providing a scientific and comprehensive weighted evaluation system for storage block selection. This solution ensures that data is stored on storage blocks with long lifespans, low write frequencies, and high reliability. Furthermore, in some possible implementations, the importance of different factors can be flexibly configured by adjusting the weight coefficients. This helps further optimize storage resource allocation and improve the overall performance and stability of the storage system.

[0113] Example 4

[0114] like Figure 4 As shown, this embodiment provides a data processing method applied to a file system for managing NAND Flash, which may be a UBIFS file system. The difference from the first embodiment is that when the data to be stored is smaller than the capacity of a storage page and the set of at least one adjacent data to be stored is smaller than a preset value, the data to be stored and the at least one adjacent data to be stored are merged and then compressed according to a determined compression ratio and stored in a storage page of a target storage block. The data processing method provided in this embodiment includes the following steps: 401 to 405.

[0115] 401. Obtain a write instruction for storing data to be stored in a NAND flash memory.

[0116] 402. Dynamically determine a current working mode based on the real-time task scenario and / or data characteristics of the data to be stored.

[0117] Among them, real-time task scenarios may be high-frequency interactive or high-real-time task scenarios, such as video stream processing, game loading, etc. There may also be scenarios with low real-time requirements: such as backup data and other scenarios.

[0118] The data characteristics of the data to be stored include: one of cold data and hot data; the current working mode includes: one of high-performance mode and long-life mode.

[0119] 403. Based on the current working mode and / or compression strategy, dynamically determine a compression ratio to be used when storing the data to be stored.

[0120] The compression ratio corresponding to High Life mode is greater than that of High Performance mode. As you can see, High Life mode employs a higher compression ratio, significantly extending the life of the storage media and reducing long-term storage costs and energy consumption compared to High Performance mode. While this may sacrifice some immediate performance, for data that requires long-term storage, this strategy optimizes storage resource utilization and reduces overall operating costs, achieving an effective balance between storage efficiency and cost.

[0121] 404. Determine a target storage block for storing the data to be stored in the NAND flash memory.

[0122] In some possible implementations, a target storage block for storing the data to be stored may be determined in the NAND flash memory based on bad block prediction and a wear leveling strategy.

[0123] 405. When the data to be stored is smaller than the capacity of the storage page and the set of at least one adjacent data to be stored is smaller than a preset value, the data to be stored and the at least one adjacent data to be stored are merged, and then compressed according to a determined compression ratio and stored in the storage page of the target storage block.

[0124] For example, if a storage page is 2048 bytes in size and the data to be stored is 500 bytes (smaller than the page size), the typical data storage method is to write the 500 bytes of data directly into the page, occupying a small portion of the page and wasting the remaining storage space. It should be noted that frequently storing data in this manner and frequently writing data into unfilled pages can increase wear on the NAND Flash. As an improvement, this embodiment merges and compresses small data before storing it, reducing the number of operations on the NAND Flash pages and helping to improve the lifespan of the NAND Flash. For example, if the data to be stored is data block A, which is 500 bytes in size, the three data blocks to be stored adjacent to data block A are data blocks B, C, and D, with sizes of 700 bytes, 400 bytes, and 600 bytes, respectively. These four data blocks are merged to form a combined data packet E of approximately 2KB in size. The capacity of the combined data packet E is: A+B+C+D=2,200 bytes. Data packet E is then compressed. Assuming the compression algorithm achieves a 30% compression rate, the final data size is 1,540 bytes. The compressed 1,540-byte data is then stored in a single page. As can be seen, the above operation eliminates the need for scattered writes, avoiding page waste. This significantly reduces the write frequency by reducing the number of writes: merging four independent write operations into one. This also improves storage efficiency. After compression, the data occupies less physical space, avoiding page-level waste, and extending the life of the NAND Flash. By reducing the number of page erases and writes, better wear leveling is also achieved.

[0125] This solution effectively reduces storage fragmentation and wasted storage space by merging small data blocks and compressing them for storage. When the data to be stored is smaller than the storage page capacity, it is merged with adjacent small data blocks to fully utilize the storage page space and improve storage efficiency. Furthermore, compressed storage further reduces the physical space occupied by data, helping to extend the life of the NAND flash memory and improve overall storage system performance.

[0126] Example 5

[0127] like Figure 5 As shown, this embodiment provides a data processing method applied to a file system that manages NAND Flash, which may be a UBIFS file system. Differences from the first embodiment include: upon obtaining a read instruction, reading target data from a target data page; and, if the target data is compressed, decompressing the target data. The data processing method provided in this embodiment includes the following steps: 501 to 507.

[0128] 501. Obtain a write instruction for storing data to be stored in a NAND flash memory.

[0129] 502. Dynamically determine the current working mode based on the real-time task scenario and / or data characteristics of the data to be stored.

[0130] Among them, real-time task scenarios may be high-frequency interactive or high-real-time task scenarios, such as video stream processing, game loading, etc. There may also be scenarios with low real-time requirements: such as backup data and other scenarios.

[0131] The data characteristics of the data to be stored include: one of cold data and hot data; the current working mode includes: one of high-performance mode and long-life mode.

[0132] 503. Based on the current working mode and / or compression strategy, dynamically determine a compression ratio to be used when storing the data to be stored.

[0133] The compression ratio corresponding to High Life mode is greater than that of High Performance mode. As you can see, High Life mode employs a higher compression ratio, significantly extending the life of the storage media and reducing long-term storage costs and energy consumption compared to High Performance mode. While this may sacrifice some immediate performance, for data that requires long-term storage, this strategy optimizes storage resource utilization and reduces overall operating costs, achieving an effective balance between storage efficiency and cost.

[0134] 504. Determine a target storage block for storing the data to be stored in the NAND flash memory.

[0135] In some possible implementations, a target storage block for storing the data to be stored may be determined in the NAND flash memory based on bad block prediction and a wear leveling strategy.

[0136] 505. Based on the capacity of the storage page, compress the data to be stored according to a determined compression ratio and then store it in the target storage block.

[0137] In some possible implementations, when the data to be stored is smaller than the capacity of the storage page and the set of at least one adjacent data to be stored is smaller than a preset value, the data to be stored and at least one adjacent data to be stored are merged, and then compressed according to a determined compression ratio and stored in the storage page of the target storage block.

[0138] 506. When a read instruction is obtained, target data is read from the target data page.

[0139] 507. When the target data is compressed data, decompress the target data.

[0140] For example, a user requests to read a specific data item (such as a document, a record, or a set of values) stored in a compressed data page. After receiving the read instruction, the file system locates the compressed data page containing the document and reads it from the hard disk into memory. At this point, the data in memory is compressed and cannot be directly read or processed. The file system can determine that the data is compressed using the ZIP compression algorithm by examining the data page's metadata. The file system then calls the ZIP decompression function to decompress the compressed data into its original document format. Finally, the file system extracts the requested document content from the decompressed document and displays it to the user or returns it to the caller.

[0141] This solution decompresses data as needed during data reading, ensuring data accuracy and readability. When compressed data is stored, decompression can be used to restore the original data, satisfying data access requirements. This design not only saves space through compressed storage, but also ensures that data can be correctly read and used when needed.

[0142] In some possible implementations, during a reading peak scenario, decompression resources may be dynamically allocated to decompress the target data.

[0143] It is understandable that in peak reading scenarios, dynamic allocation of decompression resources can ensure efficient operation of the system. For example, suppose that during a peak reading period, the system detects that the number of read requests for video data has surged by 3 times. In order to cope with this change, the dynamic allocation of decompression resources mechanism can be activated: for example, the number of CPU cores used for decompression can be increased from the original 4 cores to 12 cores. At the same time, more memory resources can be allocated to the decompression task to ensure that the decompression process can be carried out efficiently. Furthermore, through the load balancing strategy, the read requests and decompression tasks can be distributed to multiple nodes for processing, avoiding overload of a single node.

[0144] This solution effectively addresses the high concurrency demands of data reads by dynamically allocating decompression resources during peak read times. Flexible adjustments to the allocation of decompression resources during peak read times ensure timely and accurate data reads, avoiding read delays or errors caused by insufficient resources. This dynamic resource allocation method improves the storage system's responsiveness and stability, providing a better user experience.

[0145] In some possible implementations, to provide additional protection for data stored in NAND Flash, in addition to data compression during storage, redundancy can be implemented to address small-scale bad block issues and enable data recovery in the event of a single block failure. Furthermore, error correction code (ECC) mechanisms can be incorporated to implement data verification and recovery functions, further enhancing system reliability.

[0146] For example, an efficient compression algorithm (such as Zstandard or LZ4) can be used to compress the original data. The compressed data block size is aligned with the Flash page size (e.g., 2KB) to optimize storage and read efficiency. A data block checksum algorithm (such as CRC32) or a redundant code (such as Reed-Solomon encoding) is then used to calculate redundant information for each compressed data block. The redundant bits are used to record integrity check and error recovery information for the compressed data block, enabling recovery in the event of data errors. Storage can be centralized or decentralized. Centralized storage, for example, involves storing the compressed data block and its redundant bits in the same page. This facilitates fast read and verification, but carries the risk of simultaneous data and redundant bit corruption. Decentralized storage, storing the redundant bits in separate pages, reduces the possibility of simultaneous data and redundant bit corruption. However, this increases the complexity of read and verification. The specific choice can be determined based on specific needs, selecting the appropriate storage method. The corresponding decompression and recovery process includes reading data and compressing data. Reading data involves reading the compressed data block and its redundant bits from the page. Compressed data blocks are verified using a checksum algorithm or redundant code. If data errors are detected, data recovery can be attempted using redundant bits and the ECC mechanism. It should be noted that the ECC mechanism can correct error bits within a certain range, improving the success rate of data recovery. Data decompression involves decompressing the repaired compressed data block to recover the original data. The decompressed data is then provided to the application layer for further processing.

[0147] By reducing data block size through compression technology and combining redundancy with ECC mechanisms to provide multiple protections, this approach enables efficient and reliable data storage and recovery during data processing. In the event of a single block failure or data error, redundant bits and ECC mechanisms can be used to quickly recover data, ensuring data integrity and system stability.

[0148] An embodiment of the present application also provides a chip, which includes a processor and a communication interface. The communication interface is used to receive signals and transmit the signals to the processor. The processor processes the signals and executes the data processing method provided by any of the above method embodiments.

[0149] An embodiment of the present application further provides a computer-readable storage medium comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the data processing method provided by any of the above method embodiments.

[0150] An embodiment of the present application further provides a computer program product, which includes: computer program code, which, when executed, implements the data processing method provided by any of the above method embodiments.

[0151] An embodiment of the present application also provides an electronic device, comprising: a memory and one or more processors; the memory is coupled to the processor; wherein the memory includes a NAND flash memory; the NAND flash memory includes a flash memory controller, the flash memory controller stores computer program code, and the computer program code includes computer instructions. When the computer instructions are executed by the flash memory controller, the flash memory controller executes the data processing method provided by any of the above method embodiments.

[0152] It can be understood that the beneficial effects that can be achieved by the above-mentioned chip, computer-readable storage medium, computer program product and electronic device can refer to the beneficial effects described in the previous method embodiments, and will not be repeated here.

[0153] In the embodiments provided in this application, it should be understood that the disclosed data processing method, chip, storage medium and computer program product can be implemented in other ways. The embodiments of the data processing method, chip, storage medium and computer program product described above are merely illustrative.

[0154] Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments.

[0155] In the above description, specific details such as specific system structures and technologies are provided for illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary details.

[0156] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0157] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the sentences "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways. The order of the steps of the method involved in the embodiments of the present application is not fixed and can be changed according to the specific circumstances.

[0158] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A data processing method, characterized in that: A file system for managing NAND flash memory, wherein the data processing method includes: Obtaining a write instruction for storing data to be stored in the NAND flash memory; Based on the real-time task scenario and the data characteristics of the data to be stored, the high-performance mode or the high-life mode is dynamically selected as the current working mode; when the real-time task scenario is a high-frequency interaction or high-real-time task scenario, and the data to be stored is hot data, the high-performance mode is selected; when the real-time task scenario is a scenario with low real-time requirements and the data to be stored is cold data, the high-life mode is selected; Based on the compression strategy, in high-performance mode, a first compression ratio is used to store hot data; in high-life mode, a second compression ratio is used to store cold data; wherein the first compression ratio is less than the second compression ratio; the compression strategy includes: determining the compression ratio of the data to be stored based on one or more of the current load of the NAND flash memory, the wear state of the available storage blocks, and the data characteristics of the data to be stored; When there are multiple storage blocks available for selection, a target storage block for storing the data to be stored is determined in the NAND flash memory according to the service life, write frequency and reliability rating of the storage blocks, combined with bad block prediction and wear leveling strategy; including: determining the storage block with the largest weight among the multiple optional storage blocks as the target storage block; wherein the weight of the storage block = α×P / E count weight+β×write frequency weight+γ×reliability rating weight, wherein α, β, and γ are weight coefficients and are constants; the P / E count weight is a remaining proportion of life based on the P / E cycle count, and the P / E count weight = 1-(current P / E count / maximum P / E cycle limit); the write frequency weight = the most recent write time interval / the longest write time interval; and the reliability rating weight is a value corresponding to the reliability level of the storage block; Based on the capacity of the storage page, the data to be stored is compressed according to the determined compression ratio and then stored in the target storage block; including: in the high life mode, if the data to be stored is smaller than the storage page capacity and merging with at least one adjacent data to be stored can more effectively utilize the storage page space, the data to be stored and the at least one adjacent data to be stored are merged, and then compressed according to the determined compression ratio and stored in the storage page of the target storage block.

2. The data processing method according to claim 1, wherein: The data processing method further includes: When a read instruction is obtained, target data is read from the target data page; When the target data is compressed data, the target data is decompressed.

3. The data processing method according to claim 2, characterized in that: The decompressing the target data comprises: During peak reading scenarios, decompression resources are dynamically allocated to decompress the target data.

4. A chip, characterized in that: The chip includes a processor and a communication interface, wherein the communication interface is used to receive a signal and transmit the signal to the processor, and the processor processes the signal and executes the data processing method according to any one of claims 1 to 3.

5. A computer-readable storage medium, characterized in that The method comprises computer instructions, which, when executed on an electronic device, enable the electronic device to execute the data processing method according to any one of claims 1 to 3.

6. A computer program product, characterized in that The computer program product comprises: a computer program code, and when the computer program code is executed, the data processing method according to any one of claims 1 to 3 is implemented.

Images