Data processing method and device, electronic equipment and readable storage medium

Abstract

The invention discloses a data processing method and device, electronic equipment and a readable storage medium, and relates to the technical field of data read-write.The method comprises the steps that after target data is determined, the data type and the operation frequency of the target data are obtained; determining a target data block according to the data type and the operation frequency, and distributing the target data block to a corresponding performance area in the target hard disk in advance according to the access frequency of the target data block; the data processing operation is executed based on the target data and the target data block, so that the problem that a write amplification effect exists in the data processing process can be solved, the data processing is executed based on the performance advantages of the target data block, the frequently written data is placed in an area with a smaller write amplification effect, and the data processing efficiency is improved. Therefore, the technical effects of reducing the write amplification effect based on flexible placement of the data, improving the data processing efficiency and prolonging the service life of the SSD are achieved.

Description

Technical Field

[0001] The present application relates to the field of data reading and writing technology, and in particular to a data processing method, device, electronic device and readable storage medium. Background Art

[0002] Solid State Drives (SSDs), as storage devices based on flash memory technology, primarily consist of flash memory chips, a controller, and interface circuits. Flash memory chips utilize NAND flash technology, offering non-volatility and high-density storage. However, NAND flash technology also has inherent characteristics, such as requiring an erase before writing and a limited number of erase and write cycles. These characteristics present SSDs with a series of challenges in data management, particularly the Write Amplification Factor (WAF), which significantly impacts SSD performance and lifespan.

[0003] To address the write amplification problem, various data management technologies have been proposed, such as multi-stream and Zoned Namespace (ZNS). These technologies have improved SSD data management efficiency to a certain extent, but they still have some limitations. For example, multi-stream technology requires close coordination between the host and storage devices, and requires high support from host software. While ZNS technology offers a certain degree of flexibility in data placement, in practice, its strict data partitioning management can lead to wasted storage space and increased management complexity. Therefore, these technologies cannot effectively address the write amplification problem. Summary of the Invention

[0004] The present application provides a data processing method, device, electronic device and readable storage medium to at least solve the problem of write amplification effect in related technologies.

[0005] This application provides a data processing method, including:

[0006] After the target data is determined, the data type and operation frequency of the target data are obtained;

[0007] Determine the target data block according to the data type and the operation frequency, wherein the target data block is pre-allocated to the corresponding performance area in the target hard disk according to the access frequency of the target data block;

[0008] A data processing operation is performed based on the target data and the target data block.

[0009] The present application also provides a data processing device, comprising:

[0010] The data processing module is configured to, after determining the target data, acquire a data type and an operation frequency of the target data, wherein the data type comprises write data and read data.

[0011] The data optimization module is configured to determine a target data block according to the data type and the operation frequency, wherein the target data block is pre-allocated to a corresponding performance area in the target hard disk according to an access frequency of the target data block.

[0012] The operation execution module is configured to perform a write operation based on the write data and the target data block, or perform a read operation based on the read data and the target data block.

[0013] The present application also provides an electronic device, comprising a memory configured to store a computer program, and a processor configured to execute the computer program to implement the steps of any of the data processing methods.

[0014] The present application also provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of any of the data processing methods.

[0015] The present application also provides a computer program product, comprising a computer program, and the computer program is executed by a processor to implement the steps of any of the data processing methods.

[0016] According to the present application, the data type and the operation frequency of the target data are acquired, the target data block pre-allocated according to the functional area is determined according to the data type and the operation frequency, and the data processing operation is executed based on the target data and the target data block, so that the problem of write amplification effect in the data processing process can be solved, the performance advantage of the target data block is used to execute the data processing, the data frequently written is placed in the area with small write amplification effect, the flexible placement based on the data is used to reduce the write amplification effect, the processing efficiency of the data is improved, and the service life of the SSD is prolonged. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0018] Figure 1 A flowchart of a data processing method provided by an embodiment of the present application is shown in the figure.

[0019] Figure 2 A storage system structure composition diagram of a data processing method provided by an embodiment of the present application is shown in the figure.

[0020] Figure 3 A flowchart of another data processing method provided by the embodiment of the present application is shown in FIG. 4;

[0021] Figure 4 A flowchart of another data processing method provided by the embodiment of the present application is shown in FIG. 4;

[0022] Figure 5 A flowchart of another data processing method provided by the embodiment of the present application is shown in FIG. 4;

[0023] Figure 6 A structural block diagram of a data processing device provided by the embodiment of the present application is shown in FIG. 5;

[0024] Figure 7 A structural block diagram of a data processing device provided by the embodiment of the present application is shown in FIG. 5; DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be clearly and completely described with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0026] It should be noted that, in the description of the present application, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, article or device. The terms "first", "second" and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0027] With the rapid development of information technology, the demand for data storage has exploded, placing increasingly higher demands on the performance and efficiency of storage devices. The NVMe (Non-Volatile Memory Express) protocol, a high-performance storage interface protocol designed specifically for non-volatile storage media, is a technology for accessing non-volatile memory (NAND flash) connected via the PCI Express (PCIe) interface, offering significant performance advantages. Through a streamlined command set, optimized data transfer mechanisms, and efficient support for multiple queues, the NVMe protocol significantly improves storage device IOPS (input and output operations per second) and reduces latency. This allows SSDs (solid-state drives) to fully realize their potential and meet the high-speed storage needs of modern data centers, high-performance computing, and various end devices. An SSD is a hard disk drive that uses flash memory technology as its data storage medium. Compared to traditional mechanical hard drives, SSDs offer faster read and write speeds, lower latency, and greater durability and shock resistance.

[0028] However, in existing SSD data management, due to the write and erase characteristics of NAND flash memory, when new data is written to a target location where old data already exists, the controller must first erase the block containing the old data before continuing the write operation. This process may involve data movement and merging, resulting in the actual amount of data written being greater than the amount requested by the user, thus causing write amplification. For example, NAND uses a "page" (4KB-16KB) as the minimum write unit and a "block" (128KB-1MB) as the minimum erase unit. If the host modifies 4KB of data, the SSD must first erase the entire block (assuming 128KB) and then merge the old data (124KB) and the new data (4KB) into a new block, resulting in an actual write of 128KB. Write amplification results in additional NAND write operations, which consumes channel bandwidth and reduces SSD write performance. Furthermore, NAND has a limited number of erase and write cycles (e.g., approximately 1000 for TLC). Higher write amplification results in more NAND erase and write cycles for the same host write volume, accelerating wear on the flash memory chips and shortening the SSD's lifespan. Therefore, how to effectively reduce the write amplification effect and improve the performance and reliability of SSDs has become a technical difficulty in the storage field. The data processing method provided in the embodiments of this application determines the target data block based on the data type and operation frequency, thereby improving data processing efficiency based on the physical performance of the hard disk and reducing the write amplification effect.

[0029] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0030] In conjunction with the specific application environment architecture or specific hardware architecture on which the execution of the data processing method depends, the specific application environment architecture or specific hardware architecture is described here.

[0031] The embodiment of the present application provides a data processing method that can be used in a pre-built storage system. Figure 1 is a flow chart of a data processing method according to an embodiment of the present application, such as Figure 1 As shown, the process includes the following steps:

[0032] Step S101: After the target data is determined, the data type and operation frequency of the target data are obtained.

[0033] Specifically, in the embodiment of the present application, the basic storage unit of the NAND flash memory is a floating gate transistor, which is composed of a control gate, a floating gate and a substrate. The floating gate is located between the control gate and the substrate and is an insulating electron capture layer for storing charge (electrons). The control gate controls the charge state of the floating gate by applying a voltage. The amount of charge stored in the floating gate determines the conduction state of the transistor, corresponding to the binary data "0" or "1". The NAND flash memory adopts a two-dimensional (2D) or three-dimensional (3D) array structure, connecting a large number of floating gate transistors in series into a "string", multiple strings form a "page", and multiple pages form a "block".

[0034] In some optional embodiments, the data operations of NAND flash memory include write operations and read operations. Among them, the write operation is in units of pages, and is written bit by bit. However, since NAND can only be written from "1" to "0" and cannot be directly overwritten, it is necessary to erase the block before writing. Erasing is in units of blocks, resetting all cells to "1", that is, the transistor is turned on, and then writing "0" on a page basis (charging the target cell to cut off and store "0"). The erase speed is faster than writing, but frequent erasing will accelerate the wear of the floating gate insulation layer. Reading applies a medium voltage (about 5 to 8V) to the control gate to detect whether the floating gate charge affects the conduction of the transistor. If the floating gate has charge (high threshold voltage), the transistor is not turned on and reads "0". If there is no charge (low threshold voltage), the transistor is turned on and reads "1".

[0035] In some optional implementations, in order to improve data processing efficiency, the present application pre-builds a storage system, such as Figure 2As shown, it includes a host layer and an NVMe SSD. The host layer includes applications that interact with users and a flexible data placement system that processes and optimizes data. The flexible data placement system includes a data processing module, a data optimization module, a garbage collection module, and an NVMe driver layer. When the user writes or reads data on the application side, the data processing module of the flexible data placement system needs to perform targeted processing on the write and read operations. Therefore, in the early stage of processing, after determining the target data, the target data needs to be analyzed to determine whether the target data is write data or read data. Among them, the user inputs write data or read data to the SSD through the host. The write data needs to be written to the SSD, and the read data is the basis for reading and is used to retrieve the corresponding data from the SSD.

[0036] Step S102 : determining a target data block according to the data type and the operation frequency, wherein the target data block is pre-allocated to a corresponding performance area in the target hard disk according to the access frequency of the target data block.

[0037] Specifically, in the embodiment of the present application, the performance areas of the SSD are determined in advance according to its physical characteristics, including: high-performance areas, low-performance areas, and over-buffered areas. The high-performance area refers to a storage area in the SSD with high read and write speeds and low latency characteristics. It is mainly used to store data that is frequently accessed or sensitive to response time. In order to give full play to the performance advantages of the SSD, SLC (single-layer cell) or MLC (double-layer cell) flash memory particles are mostly used. Each cell of SLC flash memory stores only 1 bit of data and has the characteristics of long erase and write life (up to 100,000 times or more), fast write speed (about 3-5 times that of TLC), and low read latency (10-20μs); each cell of MLC flash memory stores 2 bits of data and has the second-best performance. The low-performance area is the storage part of the SSD with relatively slow read and write speeds and high latency. It is mainly used to store data that is infrequently accessed and sensitive to storage costs. It maximizes the storage capacity utilization of the SSD while ensuring a certain performance. It mainly uses TLC (three-level cell) or QLC (quadruple-level cell) flash memory particles. TLC flash memory stores 3 bits of data per cell, which is low in cost but has a short erase and write life (about 1000-300 times); QLC flash memory stores 4 bits of data per cell, has a higher capacity density, but has the slowest write speed and the shortest erase and write life (about 100-300 times).

[0038] In some optional implementations, in order to improve data processing efficiency, the present application distinguishes data based on data access frequency. Data blocks that are accessed frequently will be placed in the high-performance area of ​​the SSD to ensure access speed; data blocks that are accessed infrequently will be placed in the low-performance area to optimize storage space and resources. Secondly, the high-performance area is further divided into a write amplification optimization area and a read optimization area according to the write amplification effect and the read rate. When a write operation or a read operation needs to be performed, the data optimization module of the flexible data placement system distinguishes the data according to the data type. For data that is written frequently, the data blocks are placed in an area with a smaller write amplification effect, i.e., the write amplification optimization area; for data that is read frequently, they are preferentially stored in an area with a higher read speed, i.e., the read optimization area.

[0039] Step S103: performing a data processing operation based on the target data and the target data block.

[0040] Specifically, in an embodiment of the present application, after determining the target data block, the corresponding data processing operation is performed. For write operations, when writing data to an SSD, the system writes to the area corresponding to the target data block in the SSD based on the data's access pattern. The NVMe command issued by the system specifies a handle to the SSD area. The SSD contains multiple handles for system use, and different handles are assigned to storage areas with different performance. After the command arrives at the SSD, the firmware passes the command to the designated processing unit based on the handle information specified in the command. The processing unit is a hardware module. The processing unit manages the handle resource pool in the SSD. During the SSD power-up phase, the data in the handle resource pool is initialized. The firmware parses the command and transmits the configuration descriptor to the processing unit. The processing unit then calls and checks the relevant handle based on the descriptor information and writes it to the relevant storage area. When the system generates a large number of commands, a queue structure is designed in the processing unit, allowing different processing units to process their own descriptors, thereby achieving concurrency. Through this process, frequently accessed data is written to the high-performance storage area, and multiple storage channels are used for parallel access, thereby improving data processing speed.

[0041] In some optional embodiments, an SSD is typically composed of multiple NAND flash memory chips, multiple controllers, and multiple storage channels. Each channel can process the read and write operations of multiple data blocks at the same time, so the solid-state drive itself has powerful parallel processing capabilities. The present invention fully utilizes this parallel processing feature of the SSD hardware architecture to achieve parallel processing of multiple data blocks during data storage and reading. Data blocks are allocated to different storage channels according to flexible data placement mapping rules to ensure that each channel can independently and parallelly process the read and write tasks of multiple data blocks. Different flash memory chips can perform read and write operations at the same time, and parallel data processing is performed in each channel, thereby greatly improving data throughput. The present application adopts a parallel data processing mechanism based on the SSD hardware architecture, making full use of the parallel processing capabilities of multi-channels, multi-chips, and multi-controllers of modern SSD memories, so that multiple data blocks can be read and written simultaneously during data storage and access, thereby greatly improving the data processing speed. Compared with the traditional serial processing method, it greatly improves the data throughput and the responsiveness of the system.

[0042] In some optional implementations, in addition to hardware-level parallel processing, the present application further improves performance through software-level parallel processing. Multi-threading or multi-process technology is used to enable multiple data blocks to be read, written, or processed in parallel. Each thread / process is responsible for processing a different data block, maximizing the utilization of multi-core CPU and storage system resources. Multiple threads are started at the software level, each thread is responsible for operating on a different data block, thereby fully utilizing the processing power of the multi-core CPU.

[0043] This application divides data into multiple data blocks and uses the parallel processing capabilities of SSD hardware to perform read and write operations on multiple data blocks at the same time, which can achieve parallel processing of data and significantly improve data processing speed. The flexible data placement function flexibly maps data blocks to different storage areas of the SSD according to the access frequency and characteristics of the data, achieving efficient storage and fast access to data, reducing unnecessary data write operations, and reducing the write amplification factor, thereby extending the service life of the SSD. The flexible data placement function allows data storage and processing strategies to be dynamically adjusted according to actual needs, adapting to different application scenarios and load changes, effectively managing the garbage collection process of the SSD, reducing additional write operations caused by garbage collection, and further reducing the write amplification effect. By reducing the write amplification effect, the number of writes to the SSD can be reduced, thereby reducing energy consumption.

[0044] Through the present application, since the data type and operation frequency of the target data are obtained, the target data blocks pre-allocated according to the functional areas are determined according to the data type and operation frequency, and data processing operations are performed based on the target data and the target data blocks. Therefore, the problem of write amplification effect in the data processing process can be solved, and data processing can be performed based on the performance advantages of the target data blocks, and data that is frequently written can be placed in an area with a smaller write amplification effect, thereby reducing the write amplification effect based on flexible data placement, improving data processing efficiency, and extending the service life of the SSD.

[0045] The embodiment of the present application also provides a data processing method that can be used in a pre-built storage system. Figure 3 is a flow chart of a data processing method according to an embodiment of the present application, such as Figure 3 As shown, the process includes the following steps:

[0046] Step S301: After the target data is determined, the data type and operation frequency of the target data are obtained. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0047] Step S302 : determining a target data block according to the data type and the operation frequency, wherein the target data block is pre-allocated to a corresponding performance area in the target hard disk according to the access frequency of the target data block.

[0048] Specifically, the above step S302 includes:

[0049] Step S3021: If the data type is write data and the operation frequency exceeds a third frequency threshold, the write amplification optimized area is used as a target data block.

[0050] Specifically, in an embodiment of the present application, the system monitors the data access pattern in real time and determines the operation frequency of the target data by analyzing the access log. If the target data is write data, the write frequency of the write data is determined. The write frequency is compared with a predetermined frequency threshold to determine whether the data is frequently written. Taking the scenario where a large e-commerce platform processes a large amount of order data as an example, the system needs to store product information, user data, order history, etc. Moreover, with the development of the business, the order volume increases rapidly, especially during promotional activities, the access frequency of some products increases sharply. At this time, the system needs to efficiently process data that is frequently read or written, such as information on hot-selling products. For example, during promotional periods such as Double Eleven and 618, the frequency of inventory updates, price adjustments, and order creation for hot-selling products surges, and the read and write request volume for some products can reach 10-100 times that of daily requests. During non-promotional periods, traffic is dispersed, and the number of visits to unpopular products is low, but historical order data continues to accumulate.

[0051] In some optional implementations, taking the inventory deduction of hot-selling product A (such as a discounted mobile phone) (inventory -1 is triggered each time an order is placed) as an example, due to the temporary adjustment of the promotional price (such as the original price of 3,999 yuan is adjusted to the promotional price of 2,999 yuan), the number of consumer users increases, and the sales volume continues to rise, so the database needs to be updated in real time, and the product comment area needs to be updated in real time according to the comments written by users. If the system frequency threshold is 500 write operations per second, after the promotion starts, the inventory update frequency of product A reaches 800 times per second, exceeding the threshold, triggering the write amplification optimization process. Among them, the inventory data of product A is stored in the data block LBA-1001, the price data is in LBA-1002, and the comment data is in LBA-1003-1005. Monitoring shows: LBA-1001 is written 800 times per second (inventory deduction), LBA-1002 is written 100 times per second (price adjustment), and LBA-1003 is written 300 times per second (new comments). At this time, LBA-1001 to LBA-1005 are aggregated into the "Product A Promotion Data Optimized Data Block", covering inventory, price, and comment-related data blocks, and the write amplification optimized area in the SSD is used as the storage location of the "Product A Promotion Data Optimized Data Block". The write amplification optimized area is a high-performance area of ​​the SSD, so it can meet high-frequency data writing while reducing the write amplification effect.

[0052] Step S3022: If the data type is read data and the operation frequency exceeds the fourth frequency threshold, the read optimization area is used as the target data block.

[0053] Specifically, in an embodiment of the present application, in the above scenario, during the promotion period, certain commodities (such as discounted commodities) are frequently accessed and become hot-selling commodities. Users frequently query the inventory, price, comments and other information of these commodities, resulting in a surge in the number of read requests for these data. During the non-promotion period, the access frequency of other commodities decreases, a large number of users begin to pay attention to discounted commodities, and the number of views of unpopular commodities decreases. Therefore, the system also needs to flexibly handle read data, such as information on hot-selling commodities, so the information on hot-selling commodities is placed in a high-performance area in advance to meet the needs of frequent reading, while the commodity data with low frequency access can be placed in a lower performance storage area. At the same time, for commodity data whose reading frequency exceeds a preset threshold, the data blocks corresponding to the commodity data are preferentially placed in the read optimization area to ensure the reading rate.

[0054] In some optional implementations, the present application employs dynamic storage optimization technology by monitoring the usage of SSD storage areas and data block access patterns in real time. As the data access patterns change during system operation, the division of storage areas and the location of data blocks are automatically adjusted. For example, when the access frequency of certain data blocks increases, the system will migrate them from low-performance areas to high-performance areas to ensure high-speed access to frequently accessed data. The specific process includes:

[0055] Step a1: update the access frequency according to the cumulative number of writes or the cumulative number of reads, and determine whether the access frequency has changed.

[0056] Step a2: if the access frequency rises from below the second frequency threshold to above the first frequency threshold, data in the target data block is migrated from the low-performance area to the high-performance area.

[0057] Step a3: if the access frequency drops from above the first frequency threshold to below the second frequency threshold, the data in the target data block is migrated from the high-performance area to the low-performance area.

[0058] In some optional implementations, taking the scenario of a large e-commerce platform processing a large amount of order data as an example, the system monitors data access patterns in real time and identifies which products are frequently queried by analyzing the access logs. The access patterns of these data begin to change, and the originally unpopular products become hot-selling products. Based on real-time monitoring and data analysis, the system can dynamically adjust the data storage location. For example, the relevant data of hot-selling products (such as product information, comments, inventory, etc.) are migrated from the TLC area with lower storage performance to the SLC area with higher storage performance to ensure that users can get a quick response when querying hot-selling products. For unpopular products, their data can be migrated to lower performance areas (such as TLC or QLC areas) to save storage space and improve resource utilization. This application constructs a dynamic storage optimization and adaptive scheduling mechanism, which can dynamically adjust the storage location and processing strategy of data blocks during operation based on the usage of SSD and data access patterns. The adaptive scheduling algorithm optimizes storage resource allocation in real time under different workloads by analyzing factors such as system load and data access trends, maximizes storage efficiency and ensures stable system operation. Dynamic storage optimization and adaptive scheduling mechanisms can significantly improve the storage system's responsiveness and processing efficiency in high-load, high-concurrency scenarios, innovatively reduce unnecessary write operations, and mitigate the write amplification effect. By precisely controlling data storage locations and write strategies, the write amplification effect is effectively reduced, extending the life of the SSD.

[0059] Step S303: Perform data processing operations based on the target data and target data blocks. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0060] Through the present application, since the data type and operation frequency of the target data are obtained, the target data blocks pre-allocated according to the functional areas are determined according to the data type and operation frequency, and data processing operations are performed based on the target data and the target data blocks. Therefore, the problem of write amplification effect in the data processing process can be solved, and data processing can be performed based on the performance advantages of the target data blocks, and data that is frequently written can be placed in an area with a smaller write amplification effect, thereby reducing the write amplification effect based on flexible data placement, improving data processing efficiency, and extending the service life of the SSD.

[0061] The embodiment of the present application also provides a data processing method that can be used in a pre-built storage system. Figure 4 is a flow chart of a data processing method according to an embodiment of the present application, such as Figure 4 As shown, the process includes the following steps:

[0062] Step S401: After the target data is determined, the data type and operation frequency of the target data are obtained. Figure 3 Step S301 of the illustrated embodiment will not be described in detail here.

[0063] Step S402: Determine the target data block based on the data type and the operation frequency, wherein the target data block is allocated to the corresponding performance area in the target hard disk in advance according to the access frequency of the target data block. Figure 3 Step S302 of the illustrated embodiment will not be described in detail here.

[0064] Step S403: performing a data processing operation based on the target data and the target data block.

[0065] Specifically, the above step S403 includes:

[0066] Step S4031: Obtain a write command including corresponding handle information generated according to the write data and the target data block.

[0067] Specifically, in the embodiment of the present application, SSD is a high-speed storage device that realizes data reading and writing through the collaboration of hardware components and firmware logic. Figure 5As shown in the figure, an SSD consists of a command reception module, firmware, a processing unit, a storage medium, DMA, and a command return module. The command reception module is the entry hub for external commands and serves as the communication interface between the host (e.g., a computer or server) and the SSD. It is responsible for receiving and parsing storage commands (e.g., read / write / erase commands) sent by the host. The firmware acts as the SSD's "operating system," managing the SSD's overall operational logic, coordinating hardware components, and implementing underlying optimization strategies. The processing unit, the "brain" of the SSD, is typically the SSD's main control chip, integrating a CPU core, cache, and various control logic. The storage medium is the physical carrier of data and is based on NAND flash memory chips (e.g., 3D TLC / QLC) and consists of multiple storage cells (cells) organized into blocks and pages. DMA (Direct Memory Access) is the "highway" for data transmission, bypassing the CPU and transferring data directly between the storage medium and the host memory, reducing CPU load and improving transmission efficiency. The command return module is the exit for processing results, packaging the SSD's operation results (e.g., command execution success / failure status, return data) into a response packet and returning it to the host via the interface protocol. Therefore, after the host layer determines the target data block according to the target data and the operating frequency, it can perform data processing based on the target data and the target data block, including data writing and data reading.

[0068] In some optional implementations, taking writing data as an example, the host layer generates a write command containing corresponding handle information based on the write data and the target data block, and sends it to the command receiving module of the SSD. For example, the logical block address LBA-1001 (corresponding to the inventory data block of the product) is the address of the target data block, and the handle information is the metadata that uniquely identifies the write operation, including the transaction ID (such as TID-20250625-001), command priority (order writes during the promotion period are set to high priority), and data verification parameters (CRC32 checksum). The content requested by the user to be written is the value after inventory deduction: 999, and then a write command is generated based on the handle information. Step S4032, the write command is sent to the firmware of the target hard disk, and the firmware generates a command descriptor based on the handle information.

[0069] Specifically, in an embodiment of the present application, the command acceptance module receives a write command and sends the write command to the firmware of the SSD. The physical link is host memory → PCIe bus → command acceptance module of the SSD controller → firmware. The firmware converts the host command into an underlying instruction set executable inside the SSD, namely a command descriptor, which includes: physical address mapping, converting LBA-1001 to physical address PBA-2001 through FTL (Flash Translation Layer); operation type, which is clearly a "page programming" operation (the basic unit of NAND flash memory write); handle metadata pass-through, retaining transaction ID, priority and other information for processing unit scheduling. In actual operation, if the amount of write data associated with the handle is small (such as a 4B inventory value), the firmware may wait for other write commands in the same area (such as price updates) and merge them to generate a command descriptor (reducing the number of physical writes).

[0070] Step S4033: Send the command descriptor to the processing unit of the target hard disk, and the processing unit writes the write data into the target data block based on the command descriptor.

[0071] Specifically, in an embodiment of the present application, the processing unit is a hardware module. The processing unit manages the handle resource pool of the SSD. During the power-on phase of the SSD, the data in the handle resource pool will be initialized, the firmware will parse the command, and the configuration descriptor will be passed to the processing unit. The processing unit will call and check the relevant handle based on the descriptor information and then write it to the relevant storage area. When the system generates a large number of commands, a queue structure is designed in the processing unit, and different processing units process their respective descriptors to achieve concurrency. Frequently accessed data should be placed in an area with high storage performance, and multiple storage channels should be used for parallel access to increase data processing speed.

[0072] In some optional implementations, after the firmware sends the command descriptor to the command queue of the main control chip (i.e., the processing unit) via an internal bus (e.g., an AHB bus), the main control inserts the command into the high-priority execution queue based on the priority field (e.g., High) in the descriptor, skipping the normal command wait. If there are 100 write command descriptors at the same time, the main control will schedule according to the following rules: high-priority commands (e.g., flash sale orders) are executed first; write commands in the same area (e.g., inventory and price of product A) are merged for execution; NAND flash controller channels are allocated (e.g., 4-channel parallel processing) to improve throughput. The main control stores the write data (e.g., 4B inventory value) in the onboard DRAM cache and calculates the ECC check code (e.g., Hamming code) to ensure data integrity. At the same time, it checks whether the target physical block PBA-2001 has been erased (must be erased before NAND writing). If not, a block erase is triggered first (taking about 1ms); data is written to Page-10 through voltage programming, and the ECC code is stored in the spare area.

[0073] In some optional implementations, the data processing process further includes a data reading operation, and the steps are as follows:

[0074] Step b1: Obtain a read command containing corresponding handle information generated according to the write data and the target data block.

[0075] Step b2: Send the read command to the firmware of the target hard disk, and the firmware generates a command descriptor according to the handle information.

[0076] Step b3: Send the command descriptor to the processing unit of the target hard disk, and the processing unit will read data from the target data block based on the descriptor.

[0077] Specifically, in an embodiment of the present invention, the processing paths of read operations and write operations in the SSD are similar, but the core logic focuses on data reading and verification. The overall process can be divided into five layers: "command reception-firmware parsing-data reading-cache scheduling-result return". First, the command acceptance module obtains the read command generated by the host layer. The read command contains the target data block address (such as the inventory data of commodity A corresponding to the logical block address LBA-1001), handle information (transaction ID, priority, data length requirement) and protocol identifier. The command acceptance module sends the read command to the firmware, and the firmware inserts the read command into the high priority queue according to the handle priority (such as High), which takes precedence over ordinary command processing. At the same time, the firmware generates a command descriptor and sends it to the SSD controller (processing unit) via the AHB bus. The core contents of the descriptor include: physical address mapping, which converts LBA-1001 to physical address PBA-2001 (corresponding to Page-10 of NAND flash block 50) via the FTL (Flash Translation Layer); the operation type, marked as "Page Read," which includes the read page number and data offset (e.g., reading 4 bytes starting from byte 0 of Page-10); and cache control, which determines whether to store the data in the DRAM cache based on the handle priority (high-priority data is forced to be cached). The controller allocates hardware resources based on the priority field in the descriptor (e.g., "High"). For example, during a promotional event, the controller will dedicate 80% of the NAND channels to processing high-priority read commands, ensuring inventory query response times of less than 50ms. The controller sends a voltage signal to the NAND flash memory, activating Page-10 corresponding to PBA-2001 and converting the stored charge signal into digital data. When reading data, the ECC code (Hamming code) stored in the spare area of ​​Page 10 is simultaneously retrieved to verify data integrity. If an error is found, error correction or rereading is triggered. If the descriptor is marked as "High Priority," the master controller stores the data in DRAM cache (such as LPDDR4) for direct access by subsequent similar requests. When a user continuously queries the inventory of product A, the master controller returns the data directly from the DRAM cache (taking <100ns), without accessing the NAND flash memory (taking ~100μs).

[0078] In some optional embodiments, such as Figure 2 As shown, the garbage collection module in the flexible data placement system of the present application is used to clean up invalid data and reorganize the storage based on the garbage collection mechanism when the storage area in the SSD is fragmented. Through dynamic optimization strategies, space utilization can be improved, unnecessary write operations can be reduced, and the write amplification effect can be reduced.

[0079] Through the present application, since the data type and operation frequency of the target data are obtained, the target data blocks pre-allocated according to the functional areas are determined according to the data type and operation frequency, and data processing operations are performed based on the target data and the target data blocks. Therefore, the problem of write amplification effect in the data processing process can be solved, and data processing can be performed based on the performance advantages of the target data blocks, and data that is frequently written can be placed in an area with a smaller write amplification effect, thereby reducing the write amplification effect based on flexible data placement, improving data processing efficiency, and extending the service life of the SSD.

[0080] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0081] The embodiments of the present application further provide a data processing device for implementing the above-mentioned embodiments and preferred embodiments, which have been described and will not be repeated here. As used below, the term "module" may be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0082] This embodiment provides a data processing device, such as Figure 6 Shown, including:

[0083] The data processing module 601 is used to obtain the data type and operation frequency of the target data after determining the target data. The data type includes: write data and read data.

[0084] The data optimization module 602 is configured to determine target data blocks according to data types and operation frequencies, wherein the target data blocks are pre-allocated to corresponding performance areas in the target hard disk according to the access frequencies of the target data blocks.

[0085] The execution operation module 603 is configured to execute a write operation based on the write data and the target data block, or execute a read operation based on the read data and the target data block.

[0086] In some optional embodiments, the apparatus further includes: a storage area division module, the storage area division module including:

[0087] The frequency determination units are configured to record the cumulative number of write operations or the cumulative number of read operations when performing a write operation or a read operation based on the target data block.

[0088] The frequency comparison unit is used to calculate the access frequency of the target data block according to the cumulative number of writes or reads within a preset time interval, and determine whether the access frequency exceeds a first frequency threshold or is lower than a second frequency threshold.

[0089] The area division unit is used to allocate the target data block to a high-performance area in the target hard disk if the frequency exceeds a first frequency threshold, and to allocate the target data block to a low-performance area in the target hard disk if the frequency is lower than a second frequency threshold.

[0090] In some optional implementations, the area division unit further includes:

[0091] The write area division subunit is used to obtain the actual write data volume and the requested write data volume when the high-performance area performs a write operation, determine the write amplification coefficient based on the actual write data volume and the requested write data volume, and determine the write amplification optimization area in the high-performance area.

[0092] The read region division subunit is used to obtain a read rate when the high-performance region performs a read operation, and determine a read optimization region in the high-performance region according to the read rate.

[0093] In some optional implementations, the storage area division module further includes:

[0094] The frequency change monitoring unit is used to update the access frequency according to the cumulative number of writes or the cumulative number of reads, and to determine whether the access frequency has changed.

[0095] The first data migration unit is configured to migrate data in the target data block from the low-performance area to the high-performance area if the access frequency increases from below the second frequency threshold to above the first frequency threshold.

[0096] The second data migration unit is configured to migrate data in the target data block from the high-performance area to the low-performance area if the access frequency drops from above the first frequency threshold to below the second frequency threshold.

[0097] In some optional implementations, the data optimization module 602 includes:

[0098] The write region determining unit is configured to use the write amplification optimized region as a target data block if the data type is write data and the operation frequency exceeds a third frequency threshold.

[0099] The read area determination unit is configured to use the read optimization area as a target data block if the data type is read data and the operation frequency exceeds a fourth frequency threshold.

[0100] In some optional implementations, the execution operation module 603 includes: a write operation unit and a read operation unit;

[0101] In some optional implementations, the write operation unit includes:

[0102] The first command acquisition subunit is used to acquire a write command including corresponding handle information generated according to the write data and the target data block.

[0103] The first command conversion subunit is used to send the write command to the firmware of the target hard disk, and the firmware generates a command descriptor according to the handle information.

[0104] The write operation execution subunit is used to send the command descriptor to the processing unit of the target hard disk, and the processing unit writes the write data into the target data block based on the command descriptor.

[0105] In some optional implementations, the read operation unit includes:

[0106] The second command acquisition subunit is used to acquire a read command containing corresponding handle information generated according to the write data and the target data block.

[0107] The second command conversion subunit is used to send the read command to the firmware of the target hard disk, and the firmware generates a command descriptor according to the handle information.

[0108] The read operation execution subunit is used to send the command descriptor to the processing unit of the target hard disk, and the processing unit reads the data from the target data block based on the descriptor.

[0109] In some optional implementations, the device further includes: a garbage collection module, configured to determine whether the target data block is fragmented, and if fragmented, to clear invalid data in the target data block.

[0110] For the description of the features in the embodiments corresponding to the data processing device, reference can be made to the relevant description of the embodiments corresponding to the data processing method, and no further details will be given here.

[0111] The embodiment of the present application also provides an electronic device, such as Figure 7 As shown, it includes a processor 10 and a memory 20, wherein the memory 20 stores a computer program, and the processor 10 is configured to run the computer program to execute the steps in any of the above data processing method embodiments.

[0112] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned data processing method embodiments when run.

[0113] In an example embodiment, the computer readable storage medium described above can include, but is not limited to, a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0114] Embodiments of the present application also provide a computer program product, which includes a computer program, and the computer program, when executed by a processor, implements the steps in any of the data processing method embodiments described above.

[0115] Embodiments of the present application also provide another computer program product, which includes a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program, and the computer program, when executed by a processor, implements the steps in any of the data processing method embodiments described above.

[0116] The skilled person can further realize that the units and algorithm steps of the examples described in connection with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0117] The above describes in detail a data processing method, device, electronic equipment and readable storage medium provided by the present application. The principles and implementation modes of the present application are described by applying specific examples. The above example descriptions are only used to help understand the method and core idea of the present application. It should be pointed out that, for ordinary skilled persons in the technical field, without departing from the principles of the present application, the present application can be improved and modified in several ways. These improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: After the target data is determined, the data type and operation frequency of the target data are obtained; Determining a target data block according to the data type and the operation frequency, wherein the target data block is pre-allocated to a corresponding performance area in the target hard disk according to the access frequency of the target data block; A data processing operation is performed based on the target data and the target data block.

2. The method according to claim 1, characterized in that The step of allocating the target data block to a corresponding performance area in the target hard disk in advance according to the access frequency of the target data block includes: When performing a write operation or a read operation based on the target data block, recording a cumulative number of writes or a cumulative number of reads; Calculating an access frequency of the target data block according to the cumulative number of writes or the cumulative number of reads within a preset time interval, and determining whether the access frequency exceeds a first frequency threshold or is lower than a second frequency threshold; If the frequency exceeds the first frequency threshold, the target data block is allocated to a high-performance area in the target hard disk; if the frequency is lower than the second frequency threshold, the target data block is allocated to a low-performance area in the target hard disk.

3. The method according to claim 2, characterized in that After allocating the target data block to the high-performance area of ​​the target hard disk, the method further includes: Obtaining an actual amount of written data and a requested amount of written data when the high-performance area performs a write operation, determining a write amplification factor according to the actual amount of written data and the requested amount of written data, and determining a write amplification optimization area in the high-performance area based on the actual amount of written data and the requested amount of written data; A read rate when the high-performance area performs a read operation is acquired, and a read-optimized area is determined in the high-performance area according to the read rate.

4. The method according to claim 2, characterized in that After allocating the target data block to a corresponding performance area in the target hard disk according to the access frequency of the target data block, the method further includes: updating the access frequency according to the cumulative number of writes or the cumulative number of reads, and determining whether the access frequency changes; If the access frequency increases from below the second frequency threshold to above the first frequency threshold, migrating the data in the target data block from the low-performance area to the high-performance area; If the access frequency drops from above the first frequency threshold to below the second frequency threshold, the data in the target data block is migrated from the high-performance area to the low-performance area.

5. The method according to claim 3, characterized in that The data type includes: write data and read data, and determining the target data block according to the data type and the operation frequency includes: If the data type is write data and the operation frequency exceeds a third frequency threshold, using the write amplification optimized area as the target data block; If the data type is read data and the operating frequency exceeds a fourth frequency threshold, the read optimization area is used as the target data block.

6. The method according to claim 1, characterized in that The data types include: write data and read data, and the performing of a data processing operation based on the target data and the target data block includes: Obtaining a write command including corresponding handle information generated according to the write data and the target data block; Sending the write command to the firmware of the target hard disk, and having the firmware generate a command descriptor according to the handle information; Sending the command descriptor to a processing unit of the target hard disk, and having the processing unit write the write data into the target data block based on the command descriptor; Or, obtaining a read command including corresponding handle information generated according to the write data and the target data block; Sending the read command to the firmware of the target hard disk, and having the firmware generate a command descriptor according to the handle information; The command descriptor is sent to a processing unit of the target hard disk, and the processing unit reads the read data from the target data block based on the descriptor.

7. The method according to claim 1, characterized in that Also includes: Determine whether the target data block is fragmented, and if so, clear invalid data in the target data block.

8. A data processing device, characterized in that: The device comprises: A data processing module is used to obtain the data type and operation frequency of the target data after determining the target data, wherein the data type includes: write data and read data; a data optimization module, configured to determine a target data block according to the data type and the operation frequency, wherein the target data block is pre-allocated to a corresponding performance area in a target hard disk according to the access frequency of the target data block; An execution operation module is configured to execute a write operation based on the write data and the target data block, or execute a read operation based on the read data and the target data block.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the data processing method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data processing method according to any one of claims 1 to 7.

Images