Data processing method, storage system, device, storage medium and program product

Abstract

The present application discloses a data processing method and system, equipment, storage medium and program product, which relate to the field of data technology, including identifying whether a data writing operation is a sequential writing operation. When the writing operation is a sequential writing operation, the order of the data to be written to the disk in the temporary storage module is controlled to match its initial writing order, that is, the data written by the sequential writing operation can be allocated relatively continuous storage resources, so that when performing a sequential reading operation, reading the data written to the disk in this way can reduce the cumbersome steps such as additional sorting, so that the sequential reading efficiency can be improved by adjusting the form of the distribution of the data written to the disk, solving the technical problem of poor sequential reading performance, and achieving the technical effect of facilitating the matching of the data order after the data written by the sequential writing operation is written to the disk with its initial writing order, so as to improve the sequential reading performance.

Description

Technical Field

[0001] The present application relates to the field of data technology, and in particular to a data processing method, a storage system, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Sequential reading and writing are one of the important functions of SSDs. Sequential reading performance is mainly reflected in the efficiency of SSDs in completing tasks such as large multimedia file access, transcoding, large game loading, video reading and editing, and can be used to measure the ability of SSDs to read data in continuous space.

[0003] However, in a multi-core architecture, when data is written through a sequential write operation, the data is transmitted separately by multiple cores of the multi-core processor. However, since the transmission efficiency of each core may vary, the order of data is usually chaotic when it is transferred to the memory of the solid-state drive. This can easily lead to the order of data being chaotic when it is written to the solid-state drive, so that it takes a long time to obtain data that matches the initial write order during sequential reading, resulting in poor sequential read performance. Summary of the invention

[0004] The present application provides a data processing method, a data processing device, a storage system, an electronic device, a computer-readable storage medium, and a computer program product to at least solve the problem of poor sequential read performance in the related art.

[0005] The present application provides a data processing method, which includes: a plurality of data management modules respectively acquire data to be written to disk; the data management modules sequentially allocate storage addresses of write temporary storage modules to the data to be written to disk according to the acquisition order of the data to be written to disk, and write the data to the temporary storage modules; the data management modules determine whether the write operation of the data to be written to disk is a sequential write operation; in response to determining that the write operation is a sequential write operation, the data management module writes the data to be written to disk into the temporary storage module, and controls the order of the data to be written to disk in the temporary storage module to match its initial write order; wherein the initial write order indicates the order in which the data to be written to disk is transmitted to the storage system; the write management module controls the data in the temporary storage module to be written to the solid-state hard disk.

[0006] The present application also provides a storage system, which includes: a temporary storage module, a solid-state drive and a multi-core processor; the multi-core processor is connected to the temporary storage module and the solid-state drive; the multi-core processor includes a write management module and multiple data management modules, and the write management module is connected to the multiple data management modules; the multiple data management modules are used to respectively obtain data to be written to the disk; according to the order in which the data to be written to the disk is obtained, the storage address of the temporary storage module is allocated to the data to be written to the disk in turn, and the data is written to the temporary storage module; it is determined whether the write operation of the data to be written to the disk is a sequential write operation; in response to determining that the write operation is a sequential write operation, the data to be written to the temporary storage module is written, and the order of the data to be written to the disk in the temporary storage module is controlled to match its initial write order; wherein the initial write order represents the order in which the data to be written to the disk is transmitted to the storage system; the write management module is used to control the data in the temporary storage module to be written to the solid-state drive.

[0007] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned data processing methods when executing the computer program.

[0008] The present application also provides a computer-readable storage medium, in which a computer program is stored, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned data processing methods are implemented.

[0009] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned data processing methods when executed by a processor.

[0010] Through the present application, when a write operation is identified as a sequential write operation, the order of the data to be written to the disk in the temporary storage module will be controlled to match its initial write order, that is, the data written by the sequential write operation can be allocated relatively continuous storage resources, so that when performing a sequential read operation, reading the data written to the disk in this way can reduce cumbersome steps such as additional sorting, thereby improving the sequential read efficiency by adjusting the distribution of the data written to the disk. Therefore, the technical problem of poor sequential read performance can be solved, and the technical effect of facilitating the matching of the data order after the data written by the sequential write operation is written to the disk with its initial write order is achieved to improve the sequential read performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0012] Figure 1It is a structural diagram of an embodiment of a storage system of the present application;

[0013] Figure 2 It is a structural schematic diagram of an embodiment of a data processing device of the present application;

[0014] Figure 3 This is a flowchart of an embodiment of the working principle of the storage system of the present application;

[0015] Figure 4 It is a flowchart of an embodiment of the data processing method of the present application;

[0016] Figure 5 It is a flowchart of another embodiment of the data processing method of the present application;

[0017] Figure 6 It is a flowchart of another embodiment of the data processing method of the present application;

[0018] Figure 7 yes Figure 6 A schematic diagram of a flow chart of an embodiment of a data processing method in which a temporary storage module stores data;

[0019] Figure 8 It is a flowchart of an embodiment of a method for storing data in a logical drive of the present application;

[0020] Fig. 9 yes Figure 8 A schematic structural diagram of an embodiment of a method for storing data. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0022] It should be noted that, in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0023] Sequential reading and writing means that the data is operated in the order in which it is stored in the file during the reading and writing process. Specifically, sequential reading and writing includes sequential writing and sequential reading. Sequential writing means that the data will be written in sequence according to the storage order in the file, and sequential reading means that the data in the file will be read in sequence according to the storage order in the file.

[0024] In order to solve the technical problem of poor sequential read performance, the present application provides a data processing method, a data processing device, a storage system, an electronic device, a computer-readable storage medium, and a computer program product. In order to enable those skilled in the art to better understand the present application scheme, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0025] In conjunction with the specific application environment architecture or the specific hardware architecture on which the execution of the data processing method depends, the specific application environment architecture or the specific hardware architecture is described herein.

[0026] See also Figure 1 , Figure 1 It is a structural diagram of an embodiment of the storage system of the present application.

[0027] In one embodiment, the storage system may include a temporary storage module 10 , a solid state drive 20 , and a multi-core processor 30 .

[0028] As the name implies, the temporary storage module 10 can temporarily store data, that is, it can store write data and write the data to, for example, the solid state drive 20 after a period of time. For example, the temporary storage module 10 can be, for example, a CU (Cache Unit), and the CU can represent a DRAM (Dynamic Random Access Memory) area inside the SSD (Solid State Disk or Solid State Drive, solid state drive 20) for temporarily storing data.

[0029] The solid state drive 20 refers to a hard disk made of a solid-state electronic storage chip array, which is usually composed of a control unit, a storage unit, and a cache unit, etc., and can be used to read and write data.

[0030] A multi-core processor 30 is a type of processor. A multi-core processor 30 means that at least two complete computing engines are concentrated in one processor. The computing engine can be considered as a core, that is, two or more complete computing engines are concentrated in the multi-core processor 30. As such, the multi-core processor 30 can support multiple processors on the system bus, and there is a bus controller that provides all bus control signals and command signals.

[0031] In this embodiment, the multi-core processor 30 may include a data processing device, which may process the data to be written to the disk and write the processed data to the solid state drive 20. The data processing device may include a write management module 32 and multiple data management modules 31. The number of the multiple data management modules 31 may be associated with the number of cores of the multi-core processor 30.

[0032] Please refer to Figure 1 as well as Figure 2 , Figure 2 It is a structural diagram of an embodiment of a data processing device of the present application.

[0033] In one embodiment, a plurality of data management modules 31 are used to respectively acquire data to be written to the disk; in accordance with the order in which the data to be written to the disk is acquired, the storage addresses of the write temporary storage module 10 are allocated to the data to be written to the disk in sequence, and the data to be written to the temporary storage module 10 are written; it is determined whether the write operation of the data to be written to the disk is a sequential write operation; in response to determining that the write operation is a sequential write operation, the data to be written to the disk is written to the temporary storage module 10, and the order of the data to be written to the disk in the temporary storage module 10 is controlled to match its initial write order; wherein, the initial write order represents the order in which the data to be written to the disk is transmitted to the storage system.

[0034] The write management module 32 is used to control the writing of data in the temporary storage module 10 to the solid state drive 20 .

[0035] The embodiment of the present application provides a data processing method, as briefly described in the data processing device part, and the working principle of the storage system is described in detail in combination with the execution flow of the data processing method.

[0036] Please refer to Figures 1 to 4 , Figure 3 This is a flowchart of an embodiment of the working principle of the storage system of the present application. Figure 4 It is a flow chart of an embodiment of the data processing method of the present application.

[0037] In one embodiment, the host can be considered as a container for placing a motherboard and other major components in a computer hardware system. The host can generally include a central processing unit, memory, hard disk, optical drive, power supply, other input and output controllers and interfaces, etc.

[0038] The host can send data to the storage system as data to be written to the disk. The storage system can be, for example, an SSD, and can further be, for example, Figure 3 Non-volatile memories such as NVMe (Non Volatile Memory Express) are shown as examples in the article.

[0039] The storage system can adaptively form a write command indicator and feed it back to multiple data management modules, so that the data management modules can respond to the write command indicator and obtain the data to be written to the disk input by the host. Figure 3 The example of write cmd (write Command Prompt) is shown in the figure. The command indicator is a working prompt that prompts for command input in the operating system.

[0040] Figure 3 The example shows that the data processing device includes three data management modules (Data Manager, DM), namely DM0, DM1 and DM2, and the three data management modules can all participate in the data transmission of the write operation.

[0041] In this case, the write manager (WM) can process the data to be written to the disk obtained by the data management module, and adaptively write the data to be written to the disk to the solid-state drive, for example, writing the data to be written to the disk to the solid-state drive. Figure 3 NAND (a type of non-volatile memory) shown as an example.

[0042] In this embodiment, the number of write management modules is less than the number of data management modules. Figure 3 As shown in the example, the number of write management modules can be one, which is conducive to improving the data processing synchronization of the write management modules and reducing the risk of data processing problems caused by performance errors and communication between multiple write management modules. The steps implemented by the multi-core processor in this process can be described as follows.

[0043] S101: Determine whether the write operation is a sequential write operation.

[0044] In this embodiment, when it is determined that the write operation is a sequential write operation, step S102 may be executed; when it is determined that the write operation is not a sequential write operation, step S103 may be executed.

[0045] S102: Allocate the written data to be written to the disk to the storage addresses in the storage module that match the initial writing order.

[0046] In this embodiment, in response to the write operation being a sequential write operation, the disk placement strategy of the data to be placed on the disk can be adjusted to change the data distribution form of the data to be placed on the disk. That is, in this embodiment, by allocating the written data to be placed on the disk to storage addresses in the storage module that match the initial write order, relatively continuous data to be placed on the disk can be located at relatively continuous storage addresses, that is, the order in which the data to be placed on the disk is placed can match the initial write order.

[0047] S103: Allocate the written data to be written to the disk to the free storage address of the storage module.

[0048] In this embodiment, in response to the write operation not being a sequential write operation, there is no need to pay extra attention to the initial write order of the data to be written to the disk. If so, a random idle storage address can be allocated to the written data to be written to the disk to improve data writing efficiency, or the data to be written to the disk can be allocated to the idle storage address according to the acquisition order, or other storage address allocation methods in the related art can be used, which will not be described in detail here.

[0049] S104: Assembling the data to be written to the disk from the host.

[0050] In this embodiment, the acquired data to be stored in the disk is assembled. The specific implementation method will be described in detail below with examples, and will not be described in detail here.

[0051] S105: Write the data to be written to the solid state drive.

[0052] In this embodiment, in response to completing the assembly of the data to be written to the disk, the data to be written to the disk can be written to the solid state drive. Specifically, writing the data to be written to the disk to the solid state drive can be represented by writing the data to be written to the non-volatile memory of the solid state drive such as the NAND mentioned above, which is not limited here.

[0053] See also Figure 5 , Figure 5 It is a flowchart of another embodiment of the data processing method of the present application.

[0054] S201: Multiple data management modules respectively obtain data to be written to disk.

[0055] In this embodiment, the data management module refers to a module that can obtain the data to be written to the disk and perform processing such as transmission and management. As described above, multiple data management modules can be provided in a multi-core processor. Furthermore, the number of data management modules can be associated with the number of cores of the multi-core processor. For example, the number of data management modules can be the same as the number of cores of the multi-core processor, or in a proportional relationship, etc., which is not limited here. Among them, there may be differences in data processing speed between data management modules.

[0056] Multiple data management modules can acquire the data to be written to the disk transmitted from the host in parallel to improve the transmission efficiency of the data to be written to the disk.

[0057] S202: The data management module allocates storage addresses for writing the data to the temporary storage module in turn according to the order in which the data to be written to the disk is obtained, and writes the data to the temporary storage module.

[0058] In this embodiment, before verifying whether the write operation is a sequential write operation, and / or when it has been determined that the write operation is not a sequential write operation, it can be considered that the order of the data to be written to the disk may not have a strong correlation, so there is no need to pay attention to the distribution order of the data to be written to the disk in the solid-state hard disk. In this case, the data management module can allocate the storage address of the temporary storage module to the data to be written to the disk in sequence according to the acquisition order of the data to be written to the disk. In other words, the acquisition order can affect the distribution order of the data to be written to the disk.

[0059] For example, the data management module can assign random temporary storage module storage addresses to the data to be written to the disk in turn, so as to reduce the efficiency of allocating storage addresses, while at the same time improving the efficiency of writing the data to be written to the temporary storage module; or, the data management module can control the data to be written to the temporary storage module to match the order in which it is acquired. Other achievable implementation methods will not be described here.

[0060] S203: The data management module determines whether the write operation of the data to be written to the disk is a sequential write operation.

[0061] In this embodiment, the data management module can identify the write operation of the data to be written to the disk to determine whether the write operation of the data to be written to the disk is a sequential write operation. If so, when the write operation of the data to be written to the disk is a sequential write operation, the strategy of the write temporary storage module of the data to be written to the disk can be adjusted in time, so as to optimize the sequential read performance by adjusting the data distribution form in the solid-state hard disk.

[0062] S204: In response to determining that the write operation is a sequential write operation, the data management module writes the data to be written to the disk into the temporary storage module, and controls the order of the data to be written to the disk in the temporary storage module to match its initial write order; wherein the initial write order indicates the order in which the data to be written to the disk are transmitted to the storage system.

[0063] In this embodiment, in response to determining that the write operation of writing the data to be written to the disk is a sequential write operation, it can be considered that the read operation when reading the data to be written to the disk is also a sequential read operation. Therefore, in order to optimize the sequential read performance, the data management module can intervene in the distribution form of the data to be written to the disk in the temporary storage module and after being written to the disk. After the data management module obtains the data to be written to the disk, it can obtain the initial write order of the data to be written to the disk, and allocate the storage address of the temporary storage module that matches the initial write order of the data to be written to the disk to the data to be written to the disk, so as to control the order of the data to be written to the disk in the temporary storage module to match its initial write order.

[0064] S205: The write management module controls the data in the temporary storage module to be written to the solid state hard disk.

[0065] In this embodiment, the write management module refers to a module that can control the storage of acquired data. For example, it can integrate and assemble the data to be stored in the disk acquired from multiple data management modules to form data that can be stored in the solid state drive.

[0066] The write management module can write the data in the temporary storage module to the solid state drive.

[0067] It can be seen that in this embodiment, since it is possible to identify whether the data write operation is a sequential write operation, the order of data falling to disk can be adaptively and dynamically adjusted. When the write operation is a sequential write operation, the order of the data to be dropped to disk in the temporary storage module will be controlled to match its initial write order. That is, the data written by the sequential write operation can be allocated relatively continuous storage resources. When performing a sequential read operation, because the storage order of the data dropped to disk in this way in the solid-state hard disk matches its initial write order, the sequential read operation can be performed in the order in which the data in the solid-state hard disk falls to disk, so that the additional cumbersome steps such as sorting can be reduced, so that the sequential read efficiency can be improved by adjusting the distribution of the dropped data. In this way, the technical problem of poor sequential read performance can be solved, and the technical effect of facilitating the matching of the data order after the data written by the sequential write operation falls to disk with its initial write order is achieved to improve the sequential read performance.

[0068] Please refer to Figure 6 as well as Figure 7 , Figure 6 It is a flowchart of another embodiment of the data processing method of the present application. Figure 7 yes Figure 6 The data processing method is a schematic diagram of a flow chart of an embodiment of storing data in a temporary storage module.

[0069] S301: Multiple data management modules respectively obtain data to be written to disk.

[0070] In this embodiment, the data management module can receive the data to be written to the disk and store the received data to be written to the disk. For example, the data management module can obtain 4KB (kilobytes) of data to be written to the disk. In layman's terms, the data management module can move 4KB of data to be written to the disk from the host to the temporary storage module of the solid-state drive each time.

[0071] S302: The data management module writes the data to be written to the disk into the temporary storage unit of the temporary storage module in sequence.

[0072] In this embodiment, the temporary storage module may include multiple temporary storage units. Figure 7 As shown in the example, the temporary storage module may include 31 temporary storage units, namely temporary storage unit CU_0 to temporary storage unit CU_31.

[0073] The data management module can allocate storage addresses of the temporary storage module for the data to be written to the disk in the order in which the data to be written to the disk is obtained, and write the data to the temporary storage module. As described in the example above, when it is not initially determined whether the current write operation is a sequential write operation, the strategy of allocating memory by the data management module can be to obtain the temporary storage unit if there is an idle temporary storage unit in the temporary storage module, so as to allocate the data to be written to the idle temporary storage unit.

[0074] For example, in response to acquiring the data to be written to the disk, the data management module can acquire information about the idle temporary storage units in the temporary storage module. The acquired data to be written to the disk can be randomly assigned to the idle temporary storage units. The design in this embodiment can reduce the tedious work of sequentially identifying whether each temporary storage unit is idle and controlling the sequential writing of the data to be written to the disk. The random assignment of idle temporary storage units can effectively reduce the assignment time, so that when it is not clear whether the data to be written to the disk is written through a sequential write operation, it can be helpful to improve the efficiency of writing the data to be written to the temporary storage module, thereby improving the efficiency of data writing to the disk, so as to optimize the performance of the data processing device and the storage system.

[0075] like Figure 7 As shown in the example, the data 0 to 9 to be written to the disk can be randomly allocated to temporary storage units CU_0 to CU_31.

[0076] In layman's terms, each core is randomly allocated memory at the beginning and put it all into the WM assembly NAND. Possible distribution scenarios are as follows: Figure 7 As shown in the example, the number lba_num in the table represents the stored data to be written to the disk. Among them, lba is the logical block address, which can represent the data unit of the host and the solid-state drive, generally 4KB. In this embodiment, it can also be considered to represent the data to be written to the disk, and lba_num represents the initial writing order of the data to be written to the disk. It can be seen that Figure 7 The SSD receives consecutive lba, that is, data 0 to 9 to be written to the disk, but stores them in discontinuous DRAM addresses.

[0077] S303: The data management module determines whether the write operation of the data to be written to the disk is a sequential write operation.

[0078] In this embodiment, when the data management module determines that the write operation of the data to be written to the disk is a sequential write operation, step S303 is executed; when the data management module determines that the write operation of the data to be written to the disk is not a sequential write operation, step S302 and / or step S303 is executed.

[0079] The steps of determining whether the write operation is a sequential write operation will be described in detail with examples later, and will not be repeated here.

[0080] S304: The data management module continues to write the acquired data to be written to the disk into the idle temporary storage unit until the temporary storage unit meets the conditions for writing to the disk.

[0081] In this embodiment, in response to determining that the write operation is a sequential write operation, considering that some data to be written to disk may already exist in the current temporary storage module, it can be considered that the order of the data to be written to disk in the current temporary storage module may not match its initial write order, so the data to be written to disk can be written to the idle temporary storage module first, and the temporary storage module is cleared after the data to be written to disk in the temporary storage module is written to the solid-state hard disk, so as to facilitate the analysis of the subsequent temporary storage unit allocated to the data to be written to disk.

[0082] Specifically, the data management module sequentially queries whether the temporary storage units of the temporary storage module are idle. The data management module places the data to be written into the idle temporary storage units that match their initial write order. In layman's terms, the newly acquired data to be written into the idle temporary storage units can be written into the idle temporary storage units according to their initial write order. From the perspective of the idle temporary storage units independently, the data to be written into the idle temporary storage units can be matched with their initial write order, that is, the data to be written into the relatively continuous idle temporary storage units are also relatively continuous.

[0083] In other words, when it is detected that the current sequential write is in progress, the data to be transmitted is first allocated starting from CU_0. If the current CU is already occupied, it will be allocated backwards in turn until all N CUs are allocated (e.g. Figure 7 The example shown in FIG. 1 shows 32 temporary storage units).

[0084] like Figure 7 As shown in the example, after the data to be written to disk 0-9 have been randomly allocated to temporary storage units CU_0-temporary storage unit CU_31, the data to be written to disk 11-temporary storage unit 31 can be written to other idle temporary storage units according to the initial writing order.

[0085] Furthermore, in response to the existence of idle temporary storage units in the temporary storage module and the fulfillment of the filling condition, the data management module can fill the idle temporary storage units with preset virtual data to make the temporary storage module meet the disk-writing condition. The data in the temporary storage module can be written to the solid-state hard disk, thereby reducing the risk of long-term storage of the data to be written in the temporary storage module, improving the efficiency of writing the data to be written to the disk, and helping to ensure the integrity of the data to be written to the disk, further improving the reliability of the storage system.

[0086] For example, the disk-push condition may include that the amount of data temporarily stored in the temporary storage module reaches a first data threshold; and / or that the temporary storage module has not written new data to be flushed to the disk for a target period of time.

[0087] Specifically, the write management module can monitor the amount of data temporarily stored in the temporary storage module.

[0088] In response to the amount of temporarily stored data reaching a first data threshold, the data to be written to the disk in the temporary storage module can be written to the solid state drive. Automatically writing the data to be written to the disk can improve the efficiency of data writing while ensuring data security.

[0089] For example, the write management module may write the data in the temporary storage module to the solid state drive in response to data being written to each temporary storage unit of the temporary storage module.

[0090] Alternatively, the data management module may sequentially write the data to be written to the disk into temporary storage units that match the acquisition order.

[0091] In response to the current batch of data to be written to the temporary storage module having been written and there being idle temporary storage units in the temporary storage module, the data management module fills the idle temporary storage units with preset virtual data to make the temporary storage module meet the conditions for writing to the disk.

[0092] S305: The write management module controls the data in the temporary storage module to be written to the solid state hard disk, and clears the temporary storage module.

[0093] In this embodiment, in response to the temporary storage module meeting the disk write condition, the write management module can write the data in the temporary storage module to the solid state drive and clear the temporary storage module so that the temporary storage module can store new data to be written in the initial write order.

[0094] Optionally, the write management module can determine whether the disk placement condition is met and the data is placed on the disk based on the temporary storage module as a unit. Alternatively, the write management module can determine whether the disk placement condition is met based on the temporary storage unit group, and place the data on the disk based on the temporary storage unit group as a unit. That is, the temporary storage module can include multiple temporary storage unit groups. Figure 7 As shown in the example, the temporary storage module may include eight temporary storage unit groups, namely temporary storage unit groups (1) to (8), and each temporary storage unit group may include four temporary storage units. When the temporary storage unit group is full of data, the data in the temporary storage unit group is written to the solid state drive.

[0095] S306: The data management module identifies the starting data of the target data.

[0096] In this embodiment, the data management module identifies the starting data of the target data according to the initial writing order; wherein the target data includes the data to be written to the disk of the current batch to be written to the temporary storage module.

[0097] S307: The data management module obtains the sequence difference between the current data and the starting data.

[0098] In this embodiment, the data management module obtains the current data and can analyze the order difference between the current data and the starting data. The current data represents the data to be written into the temporary storage module. When the order difference between the current data and the starting data is obtained, the current data can be written into the storage address of the temporary storage module that matches the order difference. In this embodiment, the design can pay attention to the difference between the initial writing order of the current data and the starting data when the current data is written to the disk, so that the storage address to be written to each current data can be quickly located, thereby improving data processing efficiency.

[0099] In an alternative embodiment, the acquired data to be written to the disk may be sorted according to the initial writing order, and then written into consecutive storage addresses in the sorted order.

[0100] If so, in response to determining that the write operation is a sequential write operation, the data management module obtains the data to be written to the disk to write to the temporary storage module, and controls the order of the data to be written to the disk in the temporary storage module to match its initial write order; wherein the initial write order represents the order in which the data to be written to the disk are transmitted to the storage system.

[0101] Further, in response to the current data being the starting data, the data management module may write the current data into the first temporary storage unit; otherwise, the data management module may write the current data into the temporary storage unit of the order difference bit.

[0102] For example, when it is detected that the current operation is a sequential write operation, the starting data seq_start_lba is recorded. The strategy for allocating memory by the data management module is changed. The calculation formula for the memory address of the current data to be written to the disk, that is, the current data cur_lba, can be as follows:

[0103] cuID=(cur_lab–seq_start_lba)%cuNUM Formula 1-1

[0104] Among them, cuID indicates that the cuIDth temporary storage unit is used to store current data; cuNUM indicates the maximum number of cu supported inside the SSD, that is, the number of temporary storage units.

[0105] In this way, the memory for obtaining the data to be written to the disk continuously can be basically considered to be continuous.

[0106] For example, if Seq_start_lba=100, the current data of Lba=100 corresponds to cuID=0, and the current data of Lba=101 corresponds to cuID=1. Seq_start_lba=100 indicates that the initial writing order of the starting data is 100.

[0107] Furthermore, considering that some of the initial temporary storage units in the temporary storage module may have stored data, Formula 1-1 can be further changed to the following formula:

[0108] cuID=((cur_lab–seq_start_lba)%cuNUM+start_seq_cuID)%cuNUM Equation 1-2

[0109] Wherein, start_seq_cuID indicates the sequence number of the starting idle temporary storage unit.

[0110] In this embodiment, the data management module may identify whether the current operation is a sequential write operation. Alternatively, the firmware may identify whether the current operation is a sequential write operation. It may also be considered that the data management module may also include firmware, etc., which is not limited here.

[0111] Furthermore, when multiple cuID applications fail and / or the host data corresponding to the cuID does not meet the requirements of the data distribution strategy, for example, the data with lba=100 fails to apply for the memory with cuID=0 for 10 consecutive times, but the cuID is occupied by lba=99, it can be considered that the current sequential write pattern (syntax description sentence) has changed, and the sequential write processing logic can be exited, and data processing can be performed according to conventional data write operations such as step S202 and step S302. When the write operation syntax description sentence of the external host changes, it can be considered that the current write operation is not a sequential write operation, and the data management module can also exit the sequential write operation data processing strategy.

[0112] The specific judgment logic of whether to exit the sequential write operation will be explained with examples in the following text and will not be repeated here.

[0113] In an alternative embodiment, the identification may also be performed by a write management module, which is not limited here.

[0114] S308: The write management module controls the data in the temporary storage module to be written to the solid state drive.

[0115] In this embodiment, the write management module can write the data in the temporary storage module to a plurality of logical units (LUNs). In response to the amount of data stored in the logical unit reaching a second data threshold, the write management module can write the data in the logical unit to the solid state drive.

[0116] like Figure 8 as well as Fig. 9 As shown in the example, Figure 8 This is a flow chart of an embodiment of a method for storing data in a logical drive of the present application. Fig. 9 yes Figure 8A schematic structural diagram of an embodiment of a method for storing data.

[0117] For example, when assembling data on the WM side, continuous temporary storage units can be arranged to form a tlc page (storage unit of flash memory) and then arranged as follows: Figure 8 The NAND operation is performed in the order shown in the example. The more logical drives there are, the better the sequential read performance will be, but the occupied resources will also increase, which may easily lead to less recyclable resources. Therefore, actual adjustments can be made based on the DRAM resources of the disk.

[0118] The recyclable resource may be a full logical drive. The data in the full logical drive may be written to the NAND disk. After the writing is completed, the logical drive is cleared and released to participate in the new data writing process, so as to realize resource recycling.

[0119] Optionally, taking the matching of the capacity of each logical storage unit in the logical drive with the capacity of the temporary storage unit as an example, the number of temporary storage units in the temporary storage module for storing data to be written to the disk can be identified. The number of logical storage units in a single row of a single logical drive can also be obtained. The ratio of the number of temporary storage units to the number of logical storage units is obtained, and the number of logical drives is selected to be less than the ratio. In this way, when the data of the temporary storage module can be controlled to fall into the logical drive, at least one logical drive can accommodate relatively more data, so that it can fill up the data faster and write the data to the disk to the solid-state hard disk, thereby being able to dynamically adjust the number of logical drives, thereby being able to ensure recyclable resources while ensuring sequential read performance, and improving data processing performance.

[0120] Furthermore, the data writing efficiency of the current write operation can also be evaluated. When the data writing efficiency reaches the efficiency threshold, it can be considered that the amount of data written by the current sequential write operation is large, so the number of logical drives can be allowed to be equal to or slightly larger than the above ratio, and it is believed that the logical drives can be quickly filled up in this way.

[0121] It should be noted that, regardless of whether the write operation is a sequential write operation, when the temporary storage module meets the disk write conditions, the write management module can write the data in the temporary storage module to the solid-state drive. For example, the write management module can save continuous memory data to NAND in the form of multiplane.

[0122] The following is an example of the detailed working principle of the present application for determining whether the writing operation of the data to be written to the disk is a sequential writing operation.

[0123] Each data management module can respectively record the data information of the current batch of data to be stored in the disk.

[0124] The data management module obtains data information and determines whether the data to be written to the disk obtained by each data management module is a sequential write operation. In response to the target number of data management modules obtaining the data to be written to the disk being a sequential write operation, the write operation is determined to be a sequential write operation and the global mark is currently a sequential write operation.

[0125] Specifically, when determining whether the data to be written to the disk acquired by each data management module is a sequential write operation, the data management module can determine whether the first write order is on the downstream side of the second write order based on the data information. The first write order represents the initial write order of the current batch of data to be written to the disk, and the second write order represents the initial write order of the adjacent previous batch of data to be written to the disk.

[0126] Optionally, the first writing order and the second writing order may be directly obtained and compared, or other information may be obtained to infer whether the first writing order is on the timing downstream side of the second writing order.

[0127] In response to the first write order being downstream of the second write order, the data management module accumulates sequential write counts until the sequential write counts reach a preset count, and determines that the data to be written to the disk acquired by the data management module is a sequential write operation.

[0128] In response to the first write order not being on the timing downstream side of the second write order, it may be determined that the current write operation is not a sequential write operation.

[0129] For example, each core can record several sets of slba and len of the current write cmd, where slba indicates the initial write order of the data to be written to the disk, and len indicates the length of the data to be written to the disk.

[0130] If cur_slba>= record_last_slba+record_len of the current write cmd and cur_len ==record_len, then record_seq_num++. The above conditions and conclusions can indicate that if the initial write order of the starting data of the data to be written to the disk transmitted by the current write cmd is greater than or equal to the sum of the initial write order of the starting data of the adjacent previous data to be written to the disk and the length of the recorded data to be written to the disk, then the sequential write count (i.e., record_seq_num) is increased by one.

[0131] When the initial write order in the continuous write cmd is getting larger and larger, when record_seq_num>= 100, it is considered that the current write command is a sequential write, and record_seq_flag = true (marking that the data to be written to the disk obtained by the current data management module is written through a sequential write operation). Among them, 100 is an example of the preset count in this embodiment. The preset count can be 30, 50, 75, 100, etc., which is not limited here.

[0132] When all cores satisfy record_seq_flag == true, the global mark seq_write_flag = true, that is, the global mark is currently a sequential write operation.

[0133] Furthermore, after determining that the write operation is a sequential write operation as described in the above embodiment, the data management module can also determine whether the write operation of the data to be written to the disk is a non-sequential write operation. If the write operation is changed to another write operation, it can be considered that the data to be written to the disk does not need to be strictly required to be read in the initial write order when reading, and it is not necessary to control the data to be written to the disk in the initial write order, so as to change the writing strategy of the data to be written to the disk into a more efficient conventional writing method, thereby reducing the computing burden of the storage system and improving the efficiency of data writing.

[0134] Specifically, the data management module determines whether each data management module meets the verification condition.

[0135] Among them, the verification conditions include that the first write order is not on the timing downstream side of the second write order; the first write order represents the initial write order of the current batch of data to be written to the disk, and the second write order represents the initial write order of the adjacent previous data to be written to the disk; and / or, no write operation command is received within a preset time length; and / or, the number of failures in applying for the storage address of the temporary storage module reaches a preset number of times.

[0136] In response to the write operation being a non-sequential write operation, the data management module stops controlling the order of the data to be written to the disk in the temporary storage module to match the initial write order.

[0137] In response to the write operation being a sequential write operation, the data management module may continue to control the data to be written to the disk in the initial write order according to the implementation described in the above steps S102, S204, and S304 to S307.

[0138] For example, in the current core, record_seq_flag == true, cur_slba<= record_last_slba or cur_len != record_len in the current write_cmd, it is considered that disturbance_cnt++, when disturbance_cnt>30 (empirical value), it is considered that the current write cmd pattern has changed, and record_seq_flag is controlled to be false (marking that the data to be written to the disk obtained by the current data management module is not written through sequential write operations).

[0139] For the current core, when record_seq_flag == true, if no write cmd is generated within a preset time such as 1ms, it is considered that the current write cmd pattern has changed, and record_seq_flag = false.

[0140] As described above, when controlling the data to be written to the disk in the initial writing order, if multiple applications for cuID fail and the host_lba corresponding to the cuID does not meet the requirements of strategy 2, control record_seq_flag = false.

[0141] When record_seq_flag == false in any core, the global mark seq_write_flag = false, that is, cancel the global mark that the current sequential write operation is in progress.

[0142] It can be seen that, based on the multi-core architecture, the present application can add intelligent identification of sequential write scenarios, so as to dynamically adjust the memory allocation strategy according to different scenarios, optimize the data distribution form when sequential write operations write data to disk, and thus improve sequential read performance.

[0143] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0144] For the description of the features in the embodiments corresponding to the data processing device and the storage system, reference can be made to the relevant description of the embodiments corresponding to the data processing method, which will not be repeated here.

[0145] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above data processing method embodiments.

[0146] For example, the electronic device may be a data center, a server, a mobile phone, a smart watch, etc., which is not limited here.

[0147] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned data processing method embodiments when running.

[0148] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0149] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above data processing method embodiments are implemented.

[0150] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned data processing method embodiments are implemented.

[0151] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0152] The above is a detailed introduction to a data processing method, a data processing device, a storage system, an electronic device, a computer-readable storage medium, and a computer program product provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application, and the description of the above embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A data processing method, characterized in that: The data processing method comprises: Multiple data management modules respectively obtain the data to be stored on disk; The data management module sequentially allocates storage addresses of the temporary storage module for the data to be written into the disk according to the order in which the data to be written into the disk is obtained, and writes the storage addresses into the temporary storage module; The data management module determines whether the writing operation of the data to be written to the disk is a sequential writing operation; In response to determining that the write operation is the sequential write operation, the data management module writes the data to be written to the disk into the temporary storage module, and controls the order of the data to be written to the disk in the temporary storage module to match the initial write order; wherein the initial write order indicates the order in which the data to be written to the disk are transmitted to the storage system; The write management module controls the data in the temporary storage module to be written to the solid state hard disk.

2. The data processing method according to claim 1, characterized in that: The controlling the order of the data to be written to the disk in the temporary storage module to match the initial writing order includes: The data management module identifies the starting data of the target data according to the initial writing order; wherein the target data includes the current batch of data to be written to the temporary storage module; The data management module acquires current data, analyzes the sequence difference between the current data and the starting data, and writes the current data into a storage address of the temporary storage module that matches the sequence difference.

3. The data processing method according to claim 2, characterized in that: The temporary storage module includes a plurality of temporary storage units; the step of writing the current data into a storage address of the temporary storage module that matches the sequence difference includes: In response to the current data being the starting data, the data management module writes the current data into a first temporary storage unit; Otherwise, the data management module writes the current data into a temporary storage unit of the order difference bit.

4. The data processing method according to claim 1, characterized in that: Before controlling the order of the data to be written to the disk in the temporary storage module to match the initial writing order, the method further includes: In response to the temporary storage module currently having an idle temporary storage unit, the data management module writes the acquired data to be written to the disk into the idle temporary storage unit until the temporary storage module meets the writing condition; The data management module controls the data in the temporary storage module to be stored in the solid state drive and clears the temporary storage module.

5. The data processing method according to claim 4, characterized in that: The data management module writes the acquired data to be written to the disk into the idle temporary storage unit, including: The data management module sequentially queries whether the temporary storage units of the temporary storage module are idle; The data management module places the data to be written to the disk into an idle temporary storage unit that matches the initial writing order.

6. The data processing method according to claim 5, characterized in that: The data management module places the data to be written into the disk into an idle temporary storage unit that matches the initial writing order, and further includes: In response to the temporary storage module having an idle temporary storage unit and meeting a filling condition, the data management module fills the idle temporary storage unit with preset virtual data to make the temporary storage module meet the disk-writing condition.

7. The data processing method according to claim 4, characterized in that: The data management module writes the acquired data to be written to the disk into the idle temporary storage unit, including: The data management module sequentially writes the data to be written to the disk into a temporary storage unit matching the acquisition order; In response to the current batch of data to be written to the temporary storage module having been written and there being idle temporary storage units in the temporary storage module, the data management module fills the idle temporary storage units with preset virtual data to make the temporary storage module meet the disk writing condition.

8. The data processing method according to claim 1, characterized in that: The step of sequentially allocating storage addresses for the data to be written into the temporary storage module comprises: The data management module acquires information of idle temporary storage units in the temporary storage module in response to acquiring the data to be stored on the disk; The data management module randomly distributes the acquired data to be written to the disk to the idle temporary storage unit.

9. The data processing method according to claim 1, characterized in that: The data management module determines whether the writing operation of the data to be written to the disk is a sequential writing operation, including: Each of the data management modules respectively records the data information of the current batch of data to be stored in disks; The data management module obtains the data information, and determines respectively whether the data to be written to the disk obtained by each data management module is the sequential write operation; In response to the target number of data management modules acquiring the data to be written to the disk being the sequential write operation, it is determined that the write operation is the sequential write operation, and the global mark is currently a sequential write operation.

10. The data processing method according to claim 9, characterized in that: The determining whether the data to be written to the disk acquired by each of the data management modules is the sequential write operation includes: The data management module determines, based on the data information, whether the first write order is on the time sequence downstream side of the second write order; wherein the first write order represents the initial write order of the current batch of data to be written to the disk, and the second write order represents the initial write order of the adjacent previous batch of data to be written to the disk; In response to the first write order being on the timing downstream side of the second write order, the data management module accumulates sequential write counts until the sequential write counts reach a preset count, and determines that the data to be written to the disk obtained by the data management module is the sequential write operation.

11. The data processing method according to claim 1, characterized in that: The step of responding to determining that the write operation is the sequential write operation further includes: The data management module determines whether the write operation of the new data to be written to the disk is a non-sequential write operation; In response to the write operation being a non-sequential write operation, the data management module stops controlling the order of the data to be written to the disk in the temporary storage module to match the initial write order.

12. The data processing method according to claim 11, characterized in that: The data management module determines whether the write operation of the data to be written to the disk is a non-sequential write operation, including: The data management module determines whether each of the data management modules meets the verification conditions; Among them, the verification conditions include that the first write order is not on the timing downstream side of the second write order; the first write order represents the initial write order of the current batch of data to be written to the disk, and the second write order represents the initial write order of the adjacent previous data to be written to the disk; and / or, no write operation command is received within a preset time length; and / or, the number of failures in applying for the storage address of the temporary storage module reaches a preset number of times.

13. The data processing method according to claim 1, characterized in that: The writing management module writes the data in the temporary storage module to the solid state hard disk, including: The write management module monitors the amount of data temporarily stored in the temporary storage module; In response to the amount of temporarily stored data reaching a first data threshold, the data to be written to the disk in the temporary storage module is written to the solid state drive.

14. A storage system, characterized in that: The storage system comprises: Temporary storage module and solid state drive; A multi-core processor connected to the temporary storage module and the solid state drive; the multi-core processor includes a write management module and a plurality of data management modules, and the write management module is connected to the plurality of data management modules; The plurality of data management modules are used to respectively obtain the data to be written to the disk; in accordance with the order in which the data to be written to the disk is obtained, the storage addresses of the temporary storage modules are sequentially allocated to the data to be written to the disk, and the data is written to the temporary storage module; it is determined whether the writing operation of the data to be written to the disk is a sequential writing operation; in response to determining that the writing operation is the sequential writing operation, the data to be written to the disk is written to the temporary storage module, and the order of the data to be written to the disk in the temporary storage module is controlled to match the initial writing order; wherein the initial writing order represents the order in which the data to be written to the disk is transmitted to the storage system; The write management module is used to control the data in the temporary storage module to be written to the solid state hard disk.

15. An electronic device, characterized in that: The electronic device comprises: Memory for storing computer programs; A processor, configured to implement the steps of the data processing method according to any one of claims 1 to 13 when executing the computer program.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data processing method according to any one of claims 1 to 13.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 13 are implemented.

Images