Data processing method, chip, storage medium and computer program product

Abstract

The embodiment of the invention provides a data processing method, a chip, a storage medium and a computer program product, and relates to the technical field of data processing. The method is applied to a file system for managing an NAND flash memory, and comprises the following steps: acquiring a write-in instruction; dynamically determining a current working mode based on the real-time task scene and / or the data characteristics of the to-be-stored data; the current working mode comprises one of a high-performance mode and a long-life mode; dynamically determining a compression ratio adopted when the to-be-stored data is stored based on the current working mode and / or the compression strategy; the compression ratio corresponding to the long-life mode is greater than the compression ratio corresponding to the high-performance mode; determining a target storage block used for storing the to-be-stored data in the NAND flash memory; and based on the capacity of the storage page, compressing the to-be-stored data according to the determined compression ratio, and storing the compressed to-be-stored data into the target storage block. By adopting the scheme, the storage efficiency and the storage performance are improved, and the service life of the NAND flash memory is prolonged.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, chip, storage medium and computer program product. Background Art

[0002] With the development of NAND Flash (not and flash, NAND Flash) storage technology, the manufacturing process of single-level (SLC) NAND Flash has gradually shrunk from the traditional 38nm and 32nm processes to today's 24nm or even 16nm. This progress has brought higher storage density and certain performance improvements, but it has also brought significant problems. Due to the reduction in transistor size, the program / erase (P / E) cycle endurance has been significantly reduced. Traditional large-scale process (such as 38nm, 32nm) NAND Flash can support more than 100,000 P / E cycles, while the new 24nm and 16nm process NAND Flash can only support about 50,000 to 80,000 P / E cycles.

[0003] Therefore, how to extend the service life of NAND Flash is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present application provides a data processing method, a chip, a storage medium and a computer program product, which can extend the service life of NAND Flash.

[0005] In a first aspect, an embodiment of the present application provides a data processing method, which is applied to a file system that manages a NAND flash memory. The data processing method includes: obtaining a write instruction to store data to be stored in the NAND flash memory; dynamically determining a current working mode based on a real-time task scenario and / or data characteristics of the data to be stored; the data characteristics include: one of cold data and hot data; the current working mode includes one of a high-performance mode and a high-life mode; based on the current working mode and / or a compression strategy, dynamically determining a compression ratio to be used when storing the data to be stored; wherein the compression ratio corresponding to the high-life mode is greater than the compression ratio for the high-performance mode; determining a target storage block for storing the data to be stored in the NAND flash memory; based on the capacity of the storage page, compressing the data to be stored according to the determined compression ratio and storing it in the target storage block.

[0006] Since data is written in units of storage pages, the data to be stored is compressed according to a determined compression ratio based on the capacity of the storage page and then stored in the target storage block. In specific implementation, the following methods can be used. (1) The data to be stored is compressed according to a determined compression ratio. If the compressed size is not larger than the capacity of the storage page, it is directly stored in the storage page; (2) If the compressed size is larger than the capacity of the storage page, the compressed data is split into multiple data blocks within the storage page capacity according to the capacity of the storage page, and then the split data blocks are respectively stored in the data pages of the target storage block; (3) When the data to be stored is large, in addition to being split after compression, it can also be split before compression. Specifically, in combination with the determined compression ratio, the maximum storage capacity corresponding to the storage page (i.e., the capacity before compression, such as R) is estimated, and then the data to be stored is split into data blocks not larger than R in size, and then the split data blocks are compressed according to the determined compression ratio, and the compressed data are respectively stored in the data pages of the target storage block.

[0007] By adopting this scheme, the file system that manages NAND flash memory achieves a significant improvement in storage efficiency and performance. The method first obtains the write instruction for storing data in the NAND flash memory, and dynamically selects the working mode: high-performance mode or high-life mode according to the real-time task scenario and / or data characteristics (cold data or hot data). In the high-performance mode, the system gives priority to data processing speed, which is suitable for hot data that needs to be accessed quickly; in the high-life mode, the storage space is maximized by increasing the compression ratio, which is particularly suitable for long-term cold data. In this scheme, not only the working mode can be adjusted in time, but the compression ratio can also change with the current working mode and / or compression strategy, which is conducive to making the storage strategy always optimal. Since the data to be stored is compressed according to the determined compression ratio before being stored, especially the high compression ratio in the high-life mode effectively reduces the space occupied by the data to be stored in the NAND flash memory, thereby reducing the writing and erasing frequency of the flash memory unit, and significantly extending the service life of the NAND flash memory.

[0008] In combination with the first aspect, in a possible implementation of the first aspect, determining a target storage block for storing the data to be stored in a NAND flash memory includes: based on bad block prediction and wear leveling strategy, determining a target storage block for storing the data to be stored in the NAND flash memory.

[0009] By adopting this embodiment, the target storage block is determined by introducing bad block prediction and wear leveling strategy, thereby improving the space utilization and reliability of the NAND flash memory. Bad block prediction can effectively avoid storing data on potential bad blocks, thereby reducing the risk of data loss and storage errors. At the same time, the wear leveling strategy prolongs the overall service life of the NAND flash memory by evenly distributing write operations, and reduces the possibility of performance degradation or data loss caused by excessive wear of some storage blocks.

[0010] In combination with the first aspect, in a possible implementation of the first aspect, when there are multiple storage blocks as alternatives, determining a target storage block for storing data to be stored in a NAND flash memory includes: determining a storage block with the largest weight among the multiple alternative storage blocks as the target storage block; wherein the weight of the storage block = α×P / E count weight+β×write frequency weight+γ×reliability rating weight, wherein α, β, γ are weight coefficients, which are constants; the P / E count weight is a remaining proportion of life based on the P / E cycle count, and the P / E count weight = 1-(current P / E count / maximum P / E cycle limit); the write frequency weight = the most recent write time interval / the longest write time interval; the reliability rating weight is a value corresponding to the reliability level of the storage block.

[0011] With this embodiment, when determining the target storage block, multiple factors such as P / E count, write frequency and reliability rating are comprehensively considered, providing a scientific and comprehensive weight evaluation system for the selection of storage blocks. This solution can ensure that data is stored on storage blocks with long life, low write frequency and high reliability. In addition, in some possible implementations, the importance of different factors can be flexibly configured by adjusting the weight coefficient. This helps to further optimize the allocation of storage resources and improve the overall performance and stability of the storage system.

[0012] In combination with the first aspect, in a possible implementation of the first aspect, the data to be stored is compressed according to a determined compression ratio and then stored in a target storage block, including: when the data to be stored is smaller than the capacity of a storage page and a set with at least one adjacent data to be stored is smaller than a preset value, the data to be stored and at least one adjacent data to be stored are merged, and then compressed according to the determined compression ratio and stored in a storage page in the target storage block.

[0013] This solution effectively reduces storage fragmentation and storage space waste by merging small data blocks and compressing them for storage. When the data to be stored is smaller than the storage page capacity, it is merged with adjacent small data blocks to fully utilize the storage page space and improve storage efficiency. At the same time, compressed storage further reduces the physical space occupied by data, which helps to extend the life of NAND flash memory and improve the overall performance of the storage system.

[0014] In combination with the first aspect, in a possible implementation of the first aspect, the compression strategy includes: determining the compression ratio of the data to be stored based on one or more of the current load condition of the NAND flash memory, the wear state of the available storage blocks, and the data characteristics of the data to be stored.

[0015] This solution can dynamically determine the compression ratio based on factors such as the current load of the NAND flash memory, the wear status of the available storage blocks, and the data characteristics of the data to be stored, thus achieving the flexibility of the storage strategy. It can meet the needs of different application scenarios and help improve the utilization of storage resources.

[0016] In combination with the first aspect, in a possible implementation manner of the first aspect, the data processing method further includes: when obtaining a read instruction, reading target data from a target data page; and when the target data is compressed data, decompressing the target data.

[0017] With this solution, data can be decompressed as needed when it is read, ensuring the correctness and readability of the data. When the stored data is compressed data, the original data can be restored through decompression to meet the needs of data reading. This design not only ensures the space saving brought by the compressed storage of data, but also ensures that the data can be correctly read and used when needed.

[0018] In combination with the first aspect, in a possible implementation manner of the first aspect, decompressing the target data includes: dynamically allocating decompression resources to decompress the target data during a reading peak scenario.

[0019] This solution effectively copes with the high concurrency demand for data reading by dynamically allocating decompression resources during peak reading scenarios. During peak reading periods, the timeliness and accuracy of data reading can be ensured by flexibly adjusting the allocation of decompression resources, avoiding reading delays or errors caused by insufficient resources. This method of dynamically allocating resources improves the response speed and stability of the storage system and provides users with a better user experience.

[0020] In a second aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, the communication interface is used to receive signals and transmit the signals to the processor, the processor processes the signals and executes the data processing method provided in the first aspect and any possible implementation thereof.

[0021] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, including computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes a data processing method as provided in the first aspect and any possible implementation thereof.

[0022] In a fourth aspect, an embodiment of the present application provides a computer program product, the computer program product comprising: a computer program code, which, when executed, implements the data processing method provided in the first aspect and any possible implementation thereof.

[0023] It can be understood that the beneficial effects that can be achieved by the chip of the second aspect, the computer-readable storage medium of the third aspect, and the computer program product of the fourth aspect provided above can refer to the beneficial effects of the first aspect and any possible implementation manner thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flowchart of the data processing method provided in Example 1 of the present application; Figure 2 It is a flowchart of the data processing method provided in Example 2 of the present application; Figure 3A It is a flowchart of the data processing method provided in Example 3 of the present application; Figure 3B is a schematic diagram of data reliability ratings corresponding to each storage block provided in Embodiment 3 of the present application; Figure 3C is a schematic diagram of the P / E counting weights corresponding to each storage block provided in the third embodiment of the present application; Figure 3D is a schematic diagram of the write frequency weights corresponding to each storage block provided in the third embodiment of the present application; Figure 3E is a schematic diagram of reliability rating weights corresponding to each storage block provided in Embodiment 3 of the present application; Figure 3F is a schematic diagram of the weights of storage blocks corresponding to each storage block provided in Embodiment 3 of the present application; Figure 3G It is a schematic diagram of determining the ranking of each storage block according to the weight score provided in the third embodiment of the present application; Figure 4 It is a flowchart of the data processing method provided in Example 4 of the present application; Figure 5 It is a flowchart of the data processing method provided in Example 5 of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0026] The terms "first" and "second" in this application are used to distinguish different objects, or to distinguish different processing of the same object, rather than to describe a specific order of objects. For example, a first operation and a second operation are used to distinguish different operations, rather than to describe a specific order of operations. In the embodiments of this application, "plurality" refers to two or more.

[0027] Some nouns or terms involved in this application are explained below.

[0028] 1. NAND Flash, storage blocks (Blocks) and pages (Pages) NAND Flash, Block and Page are three key concepts in NAND Flash memory, and they have clear hierarchical structures and functional relationships.

[0029] NAND Flash is a non-volatile memory technology that can keep data intact after power failure. NAND Flash is widely used in various electronic devices, such as smartphones, tablets, solid state drives (SSDs), etc., to store large amounts of data.

[0030] NAND Flash is composed of multiple blocks, and each block is composed of multiple pages. In NAND Flash, the erase operation is performed in blocks, which means that the entire block must be erased before writing new data. Data read and write operations are performed in pages, which allows the system to perform data operations in smaller units, improving data access efficiency.

[0031] Therefore, NAND Flash, storage block (Block) and Page are three closely related concepts in NAND Flash memory, which together constitute the data organization and management structure of NAND Flash memory.

[0032] 2. UBIFS file system Unsorted Block Image File System (UBIFS) is a file system designed for NAND Flash. It is responsible for organizing, managing, and storing file data on NAND Flash, and can effectively manage the erase and write operations of blocks and pages. The UBIFS file system provides optimized support for NAND Flash features, such as bad block management and wear leveling, to ensure data integrity, reliability, and efficient access.

[0033] 3. Hot Data & Cold Data Hot data and cold data are two terms used in the storage field to describe the frequency of data access. Hot data refers to data that is frequently accessed and modified. This type of data usually requires fast read and write speeds to meet high access requirements. For example, system log files, frequently traded financial data, or frequently accessed database records are all hot data. Since hot data is frequently accessed and modified, they have high I / O performance requirements for storage systems. Cold data refers to data that is rarely accessed or modified after being written. This type of data usually does not require frequent read and write operations, such as backup data, archived information, or old files that are not frequently viewed. Cold data has relatively low access speed requirements for storage systems, but is more sensitive to storage costs and long-term preservation capabilities.

[0034] 4. Zstandard Zstandard is a compression algorithm that can quickly and in real time compress data with a higher compression ratio and compression speed. It is widely used in various data storage, transmission and processing scenarios.

[0035] The embodiments of the present application provide a data processing method, a chip, a storage medium and a computer program product, which relate to the field of data processing technology. In the data processing method, before storing the data, the data to be stored is compressed according to a determined compression ratio, which is conducive to reducing the amount of physical data written, improving storage efficiency and extending the service life of NAND flash memory.

[0036] Embodiment 1 like Figure 1 As shown, this embodiment provides a data processing method, which is applied to a file system for managing NAND Flash. The file system may be a UBIFS file system. The data processing method may include the following steps: 101 to 105.

[0037] 101. Obtain a write instruction for storing data to be stored in a NAND flash memory.

[0038] Exemplarily, when a user takes a photo and chooses to save it, the file system obtains a write instruction, which indicates: store the binary data of the photo into the built-in NAND flash memory.

[0039] 102. Dynamically determine the current working mode based on the real-time task scenario and / or data characteristics of the data to be stored.

[0040] Among them, real-time task scenarios may be high-frequency interactions or high-real-time task scenarios, such as video stream processing, game loading, etc. There may be scenarios with low real-time requirements: such as backup data and other scenarios.

[0041] The data characteristics of the data to be stored include: one of cold data and hot data; the current working mode includes one of a high performance mode and a high life mode.

[0042] In NAND Flash storage technology, the management of hot data and cold data is particularly important. Because each physical storage block of NAND Flash has a limited number of erase and write times, frequent erase and write will accelerate the wear of the block, thus affecting the life of the storage device. Therefore, a reasonable data management strategy can extend the service life of the storage device.

[0043] In some possible implementations, using different compression ratios for hot data and cold data can optimize storage performance and cost. For example, hot data is stored at a low compression ratio to ensure fast access, while cold data is stored at a high compression ratio to save storage space. This strategy balances performance and capacity, which is conducive to improving system efficiency and reducing storage costs.

[0044] The specific scenario corresponding to the high-life mode, for example, can be a persistent data archiving scenario for industrial equipment: in this scenario, the equipment is required to be able to operate stably and continuously for up to 5 years or more, during which a large number of data writing tasks need to be processed, but the real-time processing speed of the data is not strict. By enabling the high-life mode, it can be ensured that the equipment can still maintain data integrity and reliability under long-term high-intensity operation, while effectively extending the life of the storage medium and reducing maintenance costs. For example, when industrial equipment enters the high-life mode, the file system automatically switches to a high compression level (such as Zstandard compression level 9-12), and all written data is stored after being efficiently compressed.

[0045] It should be noted that the current working mode is not fixed and can be switched dynamically between different states. The switching conditions can be set as needed, for example, the working mode can be switched based on the task type. For example, when the data write priority is high, the current working mode is switched to the high-performance mode, and when archiving large quantities of historical data, the current working mode is switched to the high-life mode. For example, industrial monitoring equipment is in high-performance mode during the day to record real-time monitoring data, and can be switched to high-life mode at night (such as after 12 o'clock) to compress and archive the storage.

[0046] 103. Based on the current working mode and / or compression strategy, dynamically determine a compression ratio to be used when storing the data to be stored.

[0047] Among them, the compression ratio corresponding to the high-life mode is greater than that of the high-performance mode; it is understandable that the high-life mode uses a higher compression ratio, which can significantly extend the life of the storage medium and reduce long-term storage costs and energy consumption compared to the high-performance mode. Although some immediate performance may be sacrificed, for data that needs to be stored for a long time, this strategy optimizes storage resource utilization and reduces overall operating costs, which is an effective balance between storage efficiency and cost.

[0048] In some possible implementations, the compression ratio used for the data to be stored can be determined only according to the current working mode, or only according to the preset compression strategy, or the final compression ratio can be determined based on the compression ratios determined by the two. Among them, the preset compression strategy can be set according to experience or actual needs. It can be understood that when the compression ratio determined according to the working mode is different from the compression ratio determined by the current working mode, one of them can be used as the standard, or the compression ratio actually used to compress the data to be stored can be determined according to a preset weight ratio or other methods, which are both feasible.

[0049] 104. Determine a target storage block for storing the data to be stored in the NAND flash memory.

[0050] 105. Based on the capacity of the storage page, compress the data to be stored according to a determined compression ratio and store it in the target storage block.

[0051] Since data is written in units of storage pages, the data to be stored is compressed according to a determined compression ratio based on the capacity of the storage page and then stored in the target storage block. In specific implementation, the following methods can be used. (1) The data to be stored is compressed according to a determined compression ratio. If the compressed size is not larger than the capacity of the storage page, it is directly stored in the storage page; (2) If the compressed size is larger than the capacity of the storage page, the compressed data is split into multiple data blocks within the storage page capacity according to the capacity of the storage page, and then the split data blocks are respectively stored in the data pages of the target storage block; (3) When the data to be stored is large, in addition to being split after compression, it can also be split before compression. Specifically, in combination with the determined compression ratio, the maximum storage capacity corresponding to the storage page (i.e., the capacity before compression, such as R) is estimated, and then the data to be stored is split into data blocks not larger than R in size, and then the split data blocks are compressed according to the determined compression ratio, and the compressed data are respectively stored in the data pages of the target storage block.

[0052] It is understandable that a high compression ratio helps reduce the amount of writes: the compression algorithm reduces the amount of data written while reducing the number of P / E operations of the NAND Flash. For example, if the compression ratio is 50%, the physical write amount is reduced by half, significantly delaying the wear of the NAND Flash.

[0053] Exemplarily, in some possible implementations, the compression level may include three situations: low, medium, and high. Among them, the low compression level is for scenarios including: real-time log system of embedded devices. The characteristics of the corresponding data include: high frequency update, high write rate, and short life cycle of log files. A low compression ratio (for example, the compression level of Zstandard is set to 1-3) can be used to ensure the write rate while reducing the impact of compression operation on real-time performance. The scenarios corresponding to the medium compression level include: incremental backup of embedded database files. The characteristics of the corresponding data include: moderate update frequency, mixed data with certain update characteristics (such as some cold data and some hot data). A medium compression ratio (for example, the compression level of Zstandard is set to 5-7) can be used to balance compression efficiency and write speed while reducing the storage space occupied. The scenarios corresponding to the high compression level include: firmware image or configuration file storage of embedded systems. The characteristics of the corresponding data include: low-frequency access or read-only cold data, such as system backup files or application configuration files. A high compression ratio (for example, the compression level of Zstandard is set to 9-12) can be used to minimize storage occupancy and write volume, thereby extending the life of the memory.

[0054] By adopting the method provided in this embodiment, the file system for managing NAND flash memory achieves a significant improvement in storage efficiency and performance. The method first obtains a write instruction for storing data in the NAND flash memory, and dynamically selects a working mode according to the real-time task scenario and / or data characteristics (cold data or hot data): a high-performance mode or a high-life mode. In the high-performance mode, the system gives priority to data processing speed, which is suitable for hot data that needs to be accessed quickly; in the high-life mode, the storage space is maximized by increasing the compression ratio, which is particularly suitable for long-term cold data. In this scheme, not only the working mode can be adjusted in time, but the compression ratio can also change with the current working mode and / or compression strategy, which is conducive to making the storage strategy always optimal. Since the data to be stored is compressed according to a determined compression ratio before being stored, especially the high compression ratio in the high-life mode effectively reduces the occupation of the NAND flash memory space by the data to be stored, thereby reducing the writing and erasing frequency of the flash memory unit, and significantly extending the service life of the NAND flash memory.

[0055] Embodiment 2 like Figure 2As shown, this embodiment provides a data processing method, which is applied to a file system for managing NAND Flash, and the file system may be a UBIFS file system. The difference from the first embodiment includes: when determining the target storage block, this embodiment determines it based on bad block prediction and wear leveling strategy. The data processing method provided by this embodiment includes the following steps: 201 to 205.

[0056] 201. Obtain a write instruction for storing data to be stored in a NAND flash memory.

[0057] 202. Dynamically determine a current working mode based on the real-time task scenario and / or data characteristics of the data to be stored.

[0058] Among them, real-time task scenarios may be high-frequency interactions or high-real-time task scenarios, such as video stream processing, game loading, etc. There may be scenarios with low real-time requirements: such as backup data and other scenarios.

[0059] The data characteristics of the data to be stored include: one of cold data and hot data; the current working mode includes one of a high performance mode and a high life mode.

[0060] 203. Based on the current working mode and / or compression strategy, dynamically determine a compression ratio to be used when storing the data to be stored.

[0061] Among them, the compression ratio corresponding to the high-life mode is greater than that of the high-performance mode; it is understandable that the high-life mode uses a higher compression ratio, which can significantly extend the life of the storage medium and reduce long-term storage costs and energy consumption compared to the high-performance mode. Although some immediate performance may be sacrificed, for data that needs to be stored for a long time, this strategy optimizes storage resource utilization and reduces overall operating costs, which is an effective balance between storage efficiency and cost.

[0062] The high-performance mode is used for scenarios with high real-time requirements. The high-performance mode can adopt a low compression ratio or a direct non-compression strategy to reduce compression and decompression time and improve real-time response speed.

[0063] 204. Based on bad block prediction and wear leveling strategy, determine a target storage block for storing the data to be stored in the NAND flash memory.

[0064] Among them, the bad block prediction algorithm includes: when it is detected that the physical block is about to reach the P / E cycle limit, the system will give priority to writing compressed data to physical blocks with longer lifespans, thereby delaying the further use of high-wear blocks and ensuring the long-term reliability of data.

[0065] For example, before each write operation, the file system scans the P / E cycle count of available storage blocks in the NAND Flash. If it detects that the wear of a storage block is close to the upper limit of its life (such as 90%), data is written to newer blocks first. For blocks with high wear, they are marked as prepared bad blocks to avoid reuse.

[0066] In some possible implementations, a storage block with the least current P / E cycle number may be selected as the target storage block.

[0067] In some possible implementations, when there are multiple storage blocks that meet the bad block prediction algorithm, further consideration can be given to: the temperature hotspot of the storage chip. For example, when there are adjacent and spaced storage blocks as candidates, the spaced storage blocks are preferentially selected as the target storage blocks. This helps to avoid concentrated use of storage blocks and reduce local aging. In some possible implementations, the target storage block can also be determined in combination with the frequency of use. For example, in combination with historical usage records, the storage block with the lowest recent write frequency is selected as the target storage block.

[0068] It is understandable that, combined with data importance classification, important data (such as system configuration files) can be stored in storage blocks with higher reliability. Storage blocks with higher reliability can be combined with bad block prediction algorithms to use storage blocks with the least current P / E cycles as storage blocks with higher reliability.

[0069] In specific implementation, the following method can be adopted: the file system scans and records the P / E cycle times, write frequencies and current states of all candidate storage blocks, and establishes a priority queue. Before data is written, a storage block with the longest life and the lowest current load is selected from the candidate storage blocks according to the priority for data storage.

[0070] Wear leveling is an important storage optimization strategy. Its goal is to distribute data writes reasonably so that each storage block in the storage device can wear evenly, thereby extending the service life of the entire storage system. In high life mode, the system achieves balanced distribution of wear through the combination of bad block prediction mechanism and data write strategy. The system not only selects the write target based on the number of P / E cycles and write frequency of the physical block, but also considers the temperature hotspot of the storage chip and the location factor of the storage block to avoid local aging.

[0071] In addition, the file system can also regularly evaluate the overall wear status of the NAND Flash and dynamically adjust the storage strategy based on the evaluation results, writing new data to physical blocks with significantly lower wear than the average level. This dynamic adjustment not only further optimizes the wear leveling effect, but also ensures that the storage device can operate stably for a long time and meet the needs of various application scenarios.

[0072] 205. Based on the capacity of the storage page, compress the data to be stored according to the determined compression ratio and store the compressed data in the target storage block.

[0073] By adopting this embodiment, the target storage block is determined by introducing bad block prediction and wear leveling strategy, thereby improving the space utilization and reliability of the NAND flash memory. Bad block prediction can effectively avoid storing data on potential bad blocks, thereby reducing the risk of data loss and storage errors. At the same time, the wear leveling strategy prolongs the overall service life of the NAND flash memory by evenly distributing write operations, and reduces the possibility of performance degradation or data loss caused by excessive wear of some storage blocks.

[0074] Embodiment 3 like Figure 3A As shown, this embodiment provides a data processing method, which is applied to a file system for managing NAND Flash, and the file system may be a UBIFS file system. The difference from the first embodiment includes: when determining the target storage block, when there are multiple storage blocks available, the storage block with the largest weight among the multiple storage blocks available is determined as the target storage block. The data processing method provided by this embodiment includes the following steps: 301 to 305.

[0075] 301. Obtain a write instruction for storing data to be stored in a NAND flash memory.

[0076] 302. Dynamically determine a current working mode based on the real-time task scenario and / or data characteristics of the data to be stored.

[0077] Among them, real-time task scenarios may be high-frequency interactions or high-real-time task scenarios, such as video stream processing, game loading, etc. There may be scenarios with low real-time requirements: such as backup data and other scenarios.

[0078] The data characteristics of the data to be stored include: one of cold data and hot data; the current working mode includes one of a high performance mode and a high life mode.

[0079] 303. Based on the current working mode and / or compression strategy, dynamically determine a compression ratio to be used when storing the data to be stored.

[0080] Among them, the compression ratio corresponding to the high-life mode is greater than that of the high-performance mode; it is understandable that the high-life mode uses a higher compression ratio, which can significantly extend the life of the storage medium and reduce long-term storage costs and energy consumption compared to the high-performance mode. Although some immediate performance may be sacrificed, for data that needs to be stored for a long time, this strategy optimizes storage resource utilization and reduces overall operating costs, which is an effective balance between storage efficiency and cost.

[0081] 304. When there are multiple storage blocks to be selected, determine a storage block with the largest storage block weight among the multiple storage blocks to be selected as a target storage block.

[0082] For example, if there are five candidate storage blocks in the system, namely Block A, Block B, Block C, Block D and Block E, the goal is to select one storage block from these five storage blocks to store the compressed data.

[0083] The P / E cycle count, the last write interval (i.e., how long has it been since data was written) and the reliability rating of each storage block are as follows: Figure 3B shown.

[0084] The target storage block is screened out according to a preset priority strategy. Specifically, the weight of each storage block is calculated according to a weight formula.

[0085] The weight of the storage block = α × P / E count weight + β × write frequency weight + γ × reliability rating weight, where the P / E count weight is the remaining life proportion based on the P / E cycle count. Specifically, the P / E count weight = 1-(current P / E count / maximum P / E cycle limit), and the current P / E count is Figure 3B The P / E value in the example, the maximum P / E cycle limit is assumed to be 50000, and the P / E count weight of each storage block is as follows: Figure 3C shown.

[0086] The write frequency weight reflects the usage frequency based on the most recent write time interval. The longer the interval, the higher the weight. Specifically, the write frequency weight = most recent write time interval / longest write time interval. Assuming that the longest write time interval is 100 minutes, the write frequency weight of each storage block is as follows: Figure 3D shown.

[0087] The reliability rating weight is a reliability rating for storage blocks, which is directly assigned values ​​(e.g., high reliability = 1.0, medium reliability = 0.8, low reliability = 0.6). High reliability means: no error checking and correction (ECC) verification failure records, uniform P / E cycles, and no surrounding bad blocks. Medium reliability means: occasional ECC verification failure records, or a bad block nearby, but it can still work normally. Low reliability means: frequent ECC errors, or multiple bad blocks around, with a significant risk of failure. The reliability rating weights of each storage block are as follows: Figure 3E shown.

[0088] Weight coefficients: α, β, and γ control the influence of the three weights respectively. For example, α=0.5: Prioritize P / E counting life.

[0089] β=0.3: moderately concerned about write frequency.

[0090] γ=0.2: The reliability rating is considered but given a slightly lower priority.

[0091] According to the weight formula of storage block: storage block weight = α×P / E count weight + β×write frequency weight + γ×reliability rating weight, the weight of each storage block is as follows: Figure 3F shown.

[0092] According to the weight of the storage block, the priority of the storage block is determined as follows: Figure 3G As shown, the storage block corresponding to Block C with the highest weight score can be selected as the target storage block.

[0093] 305. Based on the capacity of the storage page, compress the data to be stored according to the determined compression ratio and store the compressed data in the target storage block.

[0094] With this embodiment, when determining the target storage block, multiple factors such as P / E count, write frequency and reliability rating are comprehensively considered, providing a scientific and comprehensive weight evaluation system for the selection of storage blocks. This solution can ensure that data is stored on storage blocks with long life, low write frequency and high reliability. In addition, in some possible implementations, the importance of different factors can be flexibly configured by adjusting the weight coefficient. This helps to further optimize the allocation of storage resources and improve the overall performance and stability of the storage system.

[0095] Embodiment 4 like Figure 4 As shown, this embodiment provides a data processing method, which is applied to a file system for managing NAND Flash, and the file system may be a UBIFS file system. The difference from the first embodiment includes: when the data to be stored is smaller than the capacity of the storage page, and the set with at least one adjacent data to be stored is smaller than a preset value, the data to be stored and at least one adjacent data to be stored are merged, and then compressed according to a determined compression ratio and stored in the storage page of the target storage block. The data processing method provided by this embodiment includes the following steps: 401 to 405.

[0096] 401. Obtain a write instruction for storing data to be stored in a NAND flash memory.

[0097] 402. Dynamically determine the current working mode based on the real-time task scenario and / or data characteristics of the data to be stored.

[0098] Among them, real-time task scenarios may be high-frequency interactions or high-real-time task scenarios, such as video stream processing, game loading, etc. There may be scenarios with low real-time requirements: such as backup data and other scenarios.

[0099] The data characteristics of the data to be stored include: one of cold data and hot data; the current working mode includes one of a high performance mode and a high life mode.

[0100] 403. Based on the current working mode and / or compression strategy, dynamically determine a compression ratio to be used when storing the data to be stored.

[0101] Among them, the compression ratio corresponding to the high-life mode is greater than that of the high-performance mode; it is understandable that the high-life mode uses a higher compression ratio, which can significantly extend the life of the storage medium and reduce long-term storage costs and energy consumption compared to the high-performance mode. Although some immediate performance may be sacrificed, for data that needs to be stored for a long time, this strategy optimizes storage resource utilization and reduces overall operating costs, which is an effective balance between storage efficiency and cost.

[0102] 404. Determine a target storage block for storing the data to be stored in the NAND flash memory.

[0103] In some possible implementations, a target storage block for storing the data to be stored may be determined in the NAND flash memory based on bad block prediction and wear leveling strategy.

[0104] 405. When the data to be stored is smaller than the capacity of the storage page and the set of at least one adjacent data to be stored is smaller than a preset value, the data to be stored and at least one adjacent data to be stored are merged, and then compressed according to a determined compression ratio and stored in the storage page of the target storage block.

[0105] For example, if the size of a storage page (Page) is: 2048Byte, the size of the data to be stored is: 500Byte (smaller than the Page size), the usual data storage method is: 500Byte of data to be written is directly written to the Page, occupying a small part of the Page space, wasting the remaining storage space. It should be noted that if data is stored frequently in this way, frequently writing data to unfilled Pages will increase the wear of NAND Flash. As an improvement, this embodiment merges and compresses small data before storing it, reducing the number of operations on the Page in NAND Flash, which is beneficial to improving the service life of NAND Flash. For example, if the data to be stored is data block A, the size is 500Byte, and the three data to be stored adjacent to data block A are data block B, data block C and data block D, with sizes of 700Byte, 400Byte and 600Byte respectively. The four data blocks are merged to form a merged data packet E with a size of nearly 2KB. The capacity of the merged data packet E is: A+B+C+D=2,200Byte. Then, data packet E is compressed. Assuming that the compression rate of the compression algorithm is 30%, the final data size is 1,540Byte. Then, the compressed 1,540-byte data is stored in a Page. It can be seen that through the above operations, there is no need to disperse the write to avoid Page waste. Due to the reduction in the number of writes: 4 independent write operations are merged into 1 write operation, which significantly reduces the write frequency. At the same time, the storage efficiency is improved. After compression, the physical space occupied by the data is smaller, and the waste at the Page level is avoided, which prolongs the life of the NAND Flash. By reducing the number of Page erases and writes, better wear leveling is also achieved.

[0106] This solution effectively reduces storage fragmentation and storage space waste by merging small data blocks and compressing them for storage. When the data to be stored is smaller than the storage page capacity, it is merged with adjacent small data blocks to fully utilize the storage page space and improve storage efficiency. At the same time, compressed storage further reduces the physical space occupied by data, which helps to extend the life of NAND flash memory and improve the overall performance of the storage system.

[0107] Embodiment 5 like Figure 5 As shown, this embodiment provides a data processing method, which is applied to a file system for managing NAND Flash, and the file system may be a UBIFS file system. The difference from the first embodiment includes that this embodiment further includes: when obtaining a read instruction, reading target data from a target data page; when the target data is compressed data, decompressing the target data. The data processing method provided by this embodiment includes the following steps: 501 to 507.

[0108] 501. Obtain a write instruction for storing data to be stored in a NAND flash memory.

[0109] 502. Dynamically determine the current working mode based on the real-time task scenario and / or data characteristics of the data to be stored.

[0110] Among them, real-time task scenarios may be high-frequency interactions or high-real-time task scenarios, such as video stream processing, game loading, etc. There may be scenarios with low real-time requirements: such as backup data and other scenarios.

[0111] The data characteristics of the data to be stored include: one of cold data and hot data; the current working mode includes one of a high performance mode and a high life mode.

[0112] 503. Based on the current working mode and / or compression strategy, dynamically determine a compression ratio to be used when storing the data to be stored.

[0113] Among them, the compression ratio corresponding to the high-life mode is greater than that of the high-performance mode; it is understandable that the high-life mode uses a higher compression ratio, which can significantly extend the life of the storage medium and reduce long-term storage costs and energy consumption compared to the high-performance mode. Although some immediate performance may be sacrificed, for data that needs to be stored for a long time, this strategy optimizes storage resource utilization and reduces overall operating costs, which is an effective balance between storage efficiency and cost.

[0114] 504. Determine a target storage block for storing the data to be stored in the NAND flash memory.

[0115] In some possible implementations, a target storage block for storing the data to be stored may be determined in the NAND flash memory based on bad block prediction and wear leveling strategy.

[0116] 505. Based on the capacity of the storage page, compress the data to be stored according to a determined compression ratio and store the compressed data in the target storage block.

[0117] In some possible implementations, when the data to be stored is smaller than the capacity of a storage page and a set of at least one adjacent data to be stored is smaller than a preset value, the data to be stored and at least one adjacent data to be stored are merged, and then compressed according to a determined compression ratio and stored in a storage page in a target storage block.

[0118] 506. When a read instruction is obtained, target data is read from a target data page.

[0119] 507. When the target data is compressed data, decompress the target data.

[0120] For example, a user requests to read a specific data item (e.g., a document, a record, or a set of values) stored in a compressed data page. After the file system obtains the read instruction, it locates the compressed data page containing the document and reads it from the hard disk into the memory. At this point, the data in the memory is in compressed form and cannot be read or processed directly. The file system can determine that the data is compressed and uses the ZIP compression algorithm by checking the metadata of the data page. The file system then calls the ZIP decompression function to decompress the compressed data into the original document format. Finally, the file system extracts the document content requested by the user from the decompressed document and displays it to the user or returns it to the caller.

[0121] With this solution, data can be decompressed as needed when it is read, ensuring the correctness and readability of the data. When the stored data is compressed data, the original data can be restored through decompression to meet the needs of data reading. This design not only ensures the space saving brought by the compressed storage of data, but also ensures that the data can be correctly read and used when needed.

[0122] In some possible implementations, during a peak reading scenario, decompression resources may be dynamically allocated to decompress the target data.

[0123] It is understandable that in peak reading scenarios, dynamic allocation of decompression resources can ensure efficient operation of the system. For example, suppose that during a peak reading period, the system detects that the number of read requests for video data has surged by 3 times. In order to cope with this change, the dynamic allocation of decompression resources can be started: for example, increase the number of CPU cores used for decompression from the original 4 cores to 12 cores. At the same time, more memory resources can be allocated to the decompression task to ensure that the decompression process can be carried out efficiently. Furthermore, through the load balancing strategy, read requests and decompression tasks can be distributed to multiple nodes for processing, avoiding overload of a single node.

[0124] This solution effectively copes with the high concurrency demand for data reading by dynamically allocating decompression resources during peak reading scenarios. During peak reading periods, the timeliness and accuracy of data reading can be ensured by flexibly adjusting the allocation of decompression resources, avoiding reading delays or errors caused by insufficient resources. This method of dynamically allocating resources improves the response speed and stability of the storage system and provides users with a better user experience.

[0125] In some possible implementations, in order to provide additional protection for data stored in NAND Flash, in addition to compressing the data during storage, a redundancy design can also be provided to deal with small-scale bad block problems and to recover data when a single block fails. At the same time, the Error Correction Code (ECC) mechanism can also be combined to implement data verification and recovery functions to further enhance system reliability.

[0126] Exemplarily, an efficient compression algorithm (such as Zstandard or LZ4) can be used to compress the original data. The size of the compressed data block is set to align with the Page size of Flash (such as 2KB) to optimize storage and reading efficiency. Then, a data block check algorithm (such as CRC32) or a redundant code (such as Reed-Solomon encoding) is used to calculate the redundant information of each compressed data block. The redundant bits are used to record the integrity check information and error recovery information of the compressed data block so as to recover when the data is wrong. When storing, it can be stored in a centralized manner or in a decentralized manner. Centralized storage, for example, is stored in the same Page, that is, the compressed data block and its redundant bits are stored in the same Page, so that the operation is convenient for fast reading and verification. However, there is a risk of data and redundant bits being damaged at the same time. Decentralized storage, that is, redundant bits are stored in independent Pages, reducing the possibility of data and redundant bits being damaged at the same time. However, this increases the complexity of reading and verification. The specific choice can be determined according to the needs, and a suitable storage method is selected. The corresponding decompression and recovery process includes: reading data and squeezing data. Among them, reading data means reading compressed data blocks and their redundant bits from Pages. Use a verification algorithm or redundant code to verify the compressed data block. If data errors are found, redundant bits and the ECC mechanism can be used to try to recover the data. It should be noted that the ECC mechanism can correct error bits within a certain range and improve the success rate of data recovery. Decompress the data, that is, decompress the repaired compressed data block to restore the original data. The decompressed data is provided to the application layer for further processing.

[0127] By reducing the size of data blocks through compression technology and combining redundancy design and ECC mechanism to provide multiple protections for data, this implementation method can achieve efficient and reliable data storage and recovery in data processing. When a single block fails or data is wrong, the redundant bits and ECC mechanism can be used to quickly recover data to ensure data integrity and system stability.

[0128] An embodiment of the present application also provides a chip, which includes a processor and a communication interface, the communication interface is used to receive signals and transmit the signals to the processor, the processor processes the signals and executes the data processing method provided by any of the above method embodiments.

[0129] An embodiment of the present application also provides a computer-readable storage medium, including computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the data processing method provided by any of the above method embodiments.

[0130] An embodiment of the present application further provides a computer program product, which includes: a computer program code, and when the computer program code is executed, the data processing method provided by any of the above method embodiments is implemented.

[0131] An embodiment of the present application also provides an electronic device, comprising: a memory and one or more processors; the memory is coupled to the processor; wherein the memory comprises a NAND flash memory; the NAND flash memory comprises a flash memory controller, the flash memory controller stores a computer program code, the computer program code comprises computer instructions, and when the computer instructions are executed by the flash memory controller, the flash memory controller executes the data processing method provided by any of the above method embodiments.

[0132] It can be understood that the beneficial effects that can be achieved by the above-provided chip, computer-readable storage medium, computer program product and electronic device can refer to the beneficial effects described in the previous method embodiments, and will not be repeated here.

[0133] In the embodiments provided in this application, it should be understood that the disclosed data processing method, chip, storage medium and computer program product can be implemented in other ways. The embodiments of the data processing method, chip, storage medium and computer program product described above are only illustrative.

[0134] Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment methods, which can be completed by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiments can be implemented.

[0135] In the above description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0136] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0137] References to "one embodiment" or "some embodiments" etc. described in the specification of the present application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways. The order of the steps of the method involved in the embodiments of the present application is not fixed and can be changed according to the specific circumstances.

[0138] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A data processing method, characterized in that: A file system for managing NAND flash memory, wherein the data processing method comprises: Obtaining a write instruction to store data to be stored in the NAND flash memory; Dynamically determine the current working mode based on the real-time task scenario and / or the data characteristics of the data to be stored; the data characteristics include: one of cold data and hot data; the current working mode includes one of a high performance mode and a long life mode; Based on the current working mode and / or compression strategy, dynamically determine the compression ratio used when storing the data to be stored; wherein the compression ratio corresponding to the high life mode is greater than the compression ratio for the high performance mode; Determining a target storage block for storing the data to be stored in a NAND flash memory; Based on the capacity of the storage page, the data to be stored is compressed according to the determined compression ratio and then stored in the target storage block.

2. The data processing method according to claim 1, characterized in that: The determining of a target storage block for storing the data to be stored in the NAND flash memory includes: Based on bad block prediction and wear leveling strategy, a target storage block for storing the data to be stored is determined in the NAND flash memory.

3. The data processing method according to claim 2, characterized in that: When there are multiple storage blocks to be selected, determining a target storage block for storing the data to be stored in the NAND flash memory includes: The storage block with the largest weight among the multiple candidate storage blocks is determined as the target storage block; wherein, The weight of the storage block = α × P / E count weight + β × write frequency weight + γ × reliability rating weight, Wherein, α, β, and γ are weight coefficients and are constants; The P / E count weight is the remaining life proportion based on the P / E cycle count, P / E count weight=1-(current P / E count / maximum P / E cycle limit); The write frequency weight = the most recent write time interval / the longest write time interval; The reliability rating weight is a value corresponding to the reliability level of the storage block.

4. The data processing method according to any one of claims 1 to 3, characterized in that: The compressing the data to be stored according to the determined compression ratio and storing it in the target storage block comprises: When the data to be stored is smaller than the capacity of a storage page and a set of at least one adjacent data to be stored is smaller than a preset value, the data to be stored and the at least one adjacent data to be stored are merged, and then compressed according to the determined compression ratio and stored in the storage page of the target storage block.

5. The data processing method according to claim 1, characterized in that: The compression strategy includes: determining the compression ratio of the data to be stored according to one or more of the current load condition of the NAND flash memory, the wear state of the available storage blocks, and the data characteristics of the data to be stored.

6. The data processing method according to any one of claims 1 to 3, characterized in that: The data processing method further includes: When a read instruction is obtained, target data is read from a target data page; When the target data is compressed data, the target data is decompressed.

7. The data processing method according to claim 6, characterized in that: The decompressing the target data comprises: During a peak reading scenario, decompression resources are dynamically allocated to decompress the target data.

8. A chip, characterized in that: The chip includes a processor and a communication interface, wherein the communication interface is used to receive a signal and transmit the signal to the processor, and the processor processes the signal and executes the data processing method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, enable the electronic device to execute the data processing method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The computer program product comprises: a computer program code, and when the computer program code is executed, the data processing method according to any one of claims 1 to 7 is implemented.

Images