Data storage method and related apparatus

Abstract

The present disclosure provides a data storage method and related device. The data storage method is applied to a metadata server of a NAS architecture, and includes: based on a received write request, storing small block data to be written in a to-be-written region of a user file in a memory; and in a case where a preset condition is met, storing the memory data after arrangement to a disk of a file system; wherein for the to-be-written region, the memory data after arrangement only includes data existing after a latest operation of the user when the preset condition is met. The present disclosure optimizes the layout of data on the disk, and achieves the purpose of improving the read performance of a client.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of storage, and in particular, to a data storage method and related device. BACKGROUND

[0002] In a network attached storage (NAS) architecture, one possible storage solution is to directly write data to disk, i.e., a front-end machine directly stores data received from a client in a file system disk and then updates a metadata server. The disadvantage of this storage solution is that: 1) in the case of a large amount of write data being written to disk at a time, since each write request needs to be written to disk twice (i.e., writing data to a file system disk and writing metadata of the data to a metadata server), the latency is serious; 2) in an append-only storage mode, the overwrite operation of the client will generate a large amount of useless disk write, and the subsequent read performance of the client will be affected.

[0003] At present, in order to solve the problem of latency, small blocks of data are sent to a metadata server (meta server) to be saved on disk. However, in this improved solution, the disk layout of the small blocks of data does not change, and the read performance of the client will be relatively low due to the influence of useless data fragments on the disk. SUMMARY

[0004] Therefore, the present disclosure aims to improve the data storage solution in the NAS architecture, so that the disk layout of the data is optimized, thereby improving the read performance of the client.

[0005] According to a first aspect of the present disclosure, a data storage method is provided, applied to a metadata server of a NAS architecture, comprising:

[0006] Based on the received write request, storing small blocks of data to be written in a user file in a memory;

[0007] In the case where a preset condition is met, transferring the sorted memory data to a disk of a file system;

[0008] For the to-be-written region, the sorted memory data only includes data existing in the to-be-written region after the latest operation of the user when the preset condition is met.

[0009] Optionally, the latest operation of the user on the to-be-written region when the preset condition is met is a write operation.

[0010] For the to-be-written region, the sorted memory data only includes data to be written in the to-be-written region by the latest write operation of the user when the preset condition is met.

[0011] Optionally, the latest operation of the user on the to-be-written region is an erasing operation when the preset condition is met.

[0012] For the to-be-written region, the arranged memory data no longer includes data written into the to-be-written region before the preset condition is met.

[0013] Optionally, the arranged memory data is obtained by arranging data stored in the memory before the preset condition is met, and the arranged memory data is still stored in the memory.

[0014] The arranged memory data is transferred to the disk of the file system, including: reading the arranged memory data from the memory and storing the read arranged memory data into the disk of the file system.

[0015] Optionally, the arranged memory data is transferred to the disk of the file system, including:

[0016] The data is read from the memory, and it is judged whether the read data is the to-be-discarded data.

[0017] In the case that the read data is the to-be-discarded data, the read data is no longer transferred to the disk of the file system.

[0018] Wherein, if the latest operation of the user on the to-be-written region is a writing operation when the preset condition is met, the to-be-discarded data is data written into the to-be-written region before the latest writing operation; if the latest operation of the user on the to-be-written region is an erasing operation when the preset condition is met, the to-be-discarded data is data written into the to-be-written region before the preset condition is met.

[0019] Optionally, the arranged data includes a plurality of sub-data, and the arranged memory data is transferred to the disk of the file system, including: transferring the plurality of sub-data so that the plurality of sub-data are stored in the disk of the file system in a preset order.

[0020] Optionally, the arranged data includes at least one sub-data, and the arranged memory data is transferred to the disk of the file system, including: setting index information for the sub-data, the index information being used to indicate the position of the sub-data in the disk of the file system and stored in the metadata server.

[0021] Optionally, the preset condition is that the size of the used space in the memory is not less than a preset value.

[0022] Alternatively, the preset condition is that the time length from the last time of transferring data to the disk of the file system is not less than a preset time length.

[0023] Optionally, based on the received write request, the small block data to be written in the to-be-written region of the user file is stored in the memory, comprising:

[0024] determining whether the current received write request satisfies the back pressure condition, the back pressure condition being that the number of times of continuously performing small block writing by the user is not less than a number threshold;

[0025] in the case that the current received write request does not satisfy the back pressure condition, only storing the small block data to be written in the to-be-written region of the user file in the memory;

[0026] in the case that the current received write request satisfies the back pressure condition, reading out the data in the region around the to-be-written region of the user file and storing the read-out data and the small block data to be written in the to-be-written region in the memory.

[0027] According to a second aspect of the present disclosure, a metadata server of a NAS architecture is provided, comprising:

[0028] a memory for storing computer executable code;

[0029] a processor for executing the computer executable code to implement any of the data storage methods of the first aspect.

[0030] According to a third aspect of the present disclosure, a storage platform is provided, comprising:

[0031] a file system;

[0032] and the metadata server of the second aspect.

[0033] According to a fourth aspect of the present disclosure, a computer readable medium is provided, comprising computer executable code, which, when executed by a processor, implements any of the data storage methods of the first aspect.

[0034] In the embodiments of the present disclosure, the metadata server of the NAS architecture first stores the small block data to be written in the to-be-written region of the user file in the memory based on the received write request; and then in the case that a preset condition is satisfied, the sorted memory data is stored to the disk of the file system. For the to-be-written region, the sorted memory data only includes the data existing after the latest operation of the to-be-written region when the preset condition is satisfied, thus avoiding a large number of useless disk writes of the to-be-written region on the disk of the file system, which optimizes the layout of the data of the to-be-written region on the disk, thereby achieving the purpose of improving the read performance of the client. BRIEF DESCRIPTION OF DRAWINGS

[0035] The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of the embodiments of the present disclosure, taken in conjunction with the accompanying drawings, in which:

[0036] Figure 1 An application diagram of the NAS architecture is shown;

[0037] Figure 2 Another application diagram of the NAS architecture is shown;

[0038] Figure 3 A flow chart of a data storage method according to an embodiment of the present disclosure is shown;

[0039] Figure 4 An application diagram of a metadata server in the NAS architecture according to an embodiment of the present disclosure is shown;

[0040] Figure 5 A flow chart of storing data to be written in a to-be-written region in a user file in the memory based on a received write request according to an embodiment of the present disclosure is shown;

[0041] Figure 6 A structural diagram of a computer device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0042] The present disclosure is described below based on embodiments, but the present disclosure is not limited to only these embodiments. In the following detailed description of the present disclosure, some specific details are described in detail. The present disclosure can also be fully understood without the description of these specific details by those skilled in the art. In order to avoid confusion of the essence of the present disclosure, the well-known methods, processes, and flows are not described in detail. In addition, the drawings are not necessarily drawn to scale.

[0043] The embodiments of the present disclosure provide a storage platform, which comprises a file system and a metadata server, wherein a disk of the file system is used to store data written in a user file, and the data written in the user file is stored in the disk as part of a disk file; the metadata server is a server of a NAS architecture, and mainly functions to store positions of region data in the user file in the disk file.

[0044] Figure 1 An application diagram of the NAS architecture is shown. Referring to Figure 1The NAS architecture includes a front-end server and the aforementioned metadata server. The main function of the front-end server is to parse storage requests and forward data, thus acting as a gateway within the NAS architecture. The front-end server connects to both the client and the file system, and also to the metadata server. When data needs to be written to a write area in a user file, the user sends a storage request to the front-end server through the client. The front-end server parses the storage request to extract the data to be written to the write area, sends the parsed data to the file system for disk storage, and then sends the data location information returned by the file system (i.e., the location of the data within the file system's disk file) to the metadata server for disk storage. In this storage scheme, the latency is relatively high because the front-end machine needs to wait for the data to be written to the file system before it can obtain the data location information, and then it needs to wait for the front-end machine to send the obtained data location information to the metadata server before the data location information can be written to the metadata server. Applications that write small blocks of data (i.e., data smaller than the preset value) are generally more sensitive to latency. This is because small blocks of data are more likely to be metadata or some kind of control structure inside the file. Reading and writing small blocks of data is often synchronous, meaning that the application will wait for the input / output of small blocks of data to be completed before proceeding to the next step. Therefore, the write latency of small blocks of data will affect the overall performance of the application, and the latency impact is even more severe when a large number of small blocks of data are written to disk at once.

[0045] To address latency issues, small blocks of data are sent to a metadata server and persisted to disk. For example... Figure 2 The diagram illustrates a NAS architecture where the front-end server connects to both the client and the metadata server, which in turn connects to the file system. When a user needs to write a small block of data to a write area in their file, the user sends a storage request to the front-end server via the client. The front-end server parses the storage request to extract the small block of data to be written and then sends it to the metadata server. The metadata server then sends the received data to the file system for disk storage. When the file system returns data location information, it can directly save the data location information without forwarding it through the front-end server, thus reducing data storage latency to some extent.

[0046] For a file system that stores data, the append-only interface is an optional data input interface. When the file system chooses to use the append-only interface, the file system only allows the user file to newly write data to the disk file, but does not allow the existing row of the disk file to be updated or deleted, so that the data written in the same area of the user file at different times will be different parts in the disk file. Here, the data written in the same area of the user file at different times is embodied as a modification of the data written at an earlier time in the user file, which can be regarded as an overwrite operation of the user on the same area of the file, but in the disk file, it is embodied as multiple repeated writings of the same area of the user file. Since the disk file is stored on the disk, the overwrite operation of the user will generate a large amount of useless disk writes. Based on Figure 2 The storage scheme of the NAS architecture shown in FIG. 1 makes some progress in solving the problem of latency, but since it does not improve the layout of data on the disk, it cannot solve the technical problem of a large number of useless disk writes like the storage scheme of the NAS architecture shown in FIG. 2. Figure 1

[0047] It is worth noting that a large number of useless disk writes not only leads to the consumption of disk resources in the file system, but also wastes resources at multiple levels such as the metadata server in the file system and the metadata server in the NAS architecture. In the case where the to-be-written area of the user file is small, multiple repeated writings of the to-be-written area will cause a large number of useless data fragments in the NAS architecture and the underlying file system, which not only requires additional metadata to track and manage the fragments, but also needs to avoid the interference of useless data fragments when reading the to-be-written area in the future, thereby causing greater pressure on the entire system and reducing the user's perception of data read performance. Although background garbage collection can effectively reduce the fragmentation rate, a large number of useless fragments will cause garbage collection pressure to increase and other problems, thereby affecting system stability, and the performance of the client reading data is still low in the case where garbage collection is not timely.

[0048] In view of the above reasons, the embodiments of the present disclosure are based on Figure 2 ​The NAS architecture shown is further improved, and the improvement is in the metadata server in the NAS architecture, mainly in the process of the metadata server writing small block data of a to-be-written region in a user file to a file system. Specifically, the metadata server in the NAS architecture no longer eagerly persists the small block data of the to-be-written region in the user file to be written to a disk of the file system, but writes the small block data of the to-be-written region in the user file to be written to the memory of the metadata server first, and then writes the sorted memory data to the disk of the file system when a preset condition is met. For the to-be-written region, the sorted memory data only includes data existing after the to-be-written region is operated by the user last time when the preset condition is met, so that a large amount of useless data fragments of the to-be-written region on the disk of the file system are avoided, which optimizes the layout of the data of the to-be-written region on the disk, and thus the purpose of improving the reading performance of the client is achieved.

[0049] Figure 3 Fig. 1 is a flowchart of a data storage method provided by an embodiment of the present disclosure. Referring to Fig. 1, Figure 3 the data storage method comprises the following steps.

[0050] In step S110, based on the received write request, small block data of a to-be-written region in a user file is stored in the memory.

[0051] In step S120, when a preset condition is met, the sorted memory data is written to the disk of the file system.

[0052] The data storage method provided by the embodiment of the present disclosure is executed by the metadata server in the NAS architecture. The metadata server in the NAS architecture writes the sorted memory data to the disk of the file system once when a preset condition is met each time, and the sorted memory data written is data obtained by sorting the memory data of a current period, and the memory no longer stores the memory data of the current period and the sorted memory data after the writing. The current period is a period between a first time and a second time, and the memory data of the current period is small block data stored in the memory based on each received write request between the first time and the second time, wherein the first time is the time when the last preset condition is met, and the second time is the time when the current preset condition is met. In the following description, the current period, the first time and the second time are interpreted as above unless otherwise specified.

[0053] Figure 4 Fig. 2 is an application schematic diagram of the metadata server in the NAS architecture, and the following Figure 3 and Figure 4 the steps executed by the metadata server in the NAS architecture in the data storage process are described in detail.

[0054] Referring to Figure 4In some examples, the metadata server stores the small block data in a write-ahead-log (WAL) manner, i.e., first records the write request of the user in a log file, and then a background thread executes the steps S110 and S120 according to the write request in the log file, so as to realize the storage of the rearranged memory data to the file system disk. Once the write request of the user is successfully written into the log, the metadata server will feedback success. The central idea of the write-ahead-log is that the modification of the memory data file can only occur after the modification has been recorded in the log, and any data that has not been written into the memory data file will be first redone from the pre-written log record (this is called forward roll recovery, also called REDO), so that even if the data is not written into the memory data file due to unexpected system exit or unexpected link disconnection after receiving the write request, the pre-written log can be used to recover the data, i.e., to ensure that the feedback success write request can completely write the data into the memory data file, thereby ensuring the reliability of the write data.

[0055] The write request can be sent by the front-end machine based on the storage request received from the client, i.e., the front-end machine receives the storage request from the client, analyzes the storage request to obtain the small block data to be written into the to-be-written region of the user file, encapsulates the small block data to be written into the to-be-written region of the user file into a write request, and sends the encapsulated write request to the metadata server. Thus, the metadata server can obtain the small block data to be written into the to-be-written region of the user file after receiving the write request in step S110.

[0056] It should be understood that the to-be-written region of the user file is specified by the user and can be encapsulated in the write request together with the small block data to be written, so that the metadata server can determine the to-be-written region after receiving the write request in step S110. One write request corresponds to one to-be-written region, and multiple write requests can correspond to the same or different to-be-written regions. The to-be-written region in step S110 is to write small block data, which is a smaller region.

[0057] The small block data to be written in each to-be-written region is stored in the metadata server with corresponding metadata. Metadata, also known as intermediate data or relay data, is data describing data, mainly information describing data attributes, so it not only includes the position of the data to be written in the disk file, but also includes the historical data in the to-be-written region and the range of the to-be-written region in the user file, etc., which supports the function of finding data in the to-be-written region from the memory and the file system disk, thereby realizing the subsequent reading of the data in the to-be-written region.

[0058] Generally, the to-be-written region is a content part of the user file having a specific function. For example, the user file is a service configuration file, the service configuration file includes a service object configuration part for indicating the object of the service, and the to-be-written region is the service object configuration part in the service configuration file. In this example, the to-be-written region is a small region, and the to-be-written region is a small block write when data is written into the to-be-written region. When the service configuration file is stored for the first time, an initial service object is written into the to-be-written region of the service object configuration part; when the subsequent service object is changed, the user needs to modify the service object configuration part, which is embodied in the service configuration file as modifying the content of the service object configuration part or deleting the content of the service object configuration part to make the service object configuration part default, i.e., the service is publicized.

[0059] It should be noted that, for the to-be-written region, the memory data after the arrangement only includes data existing in the to-be-written region after the latest operation of the user, wherein the latest operation is an operation of the user on the to-be-written region closest to the time when the preset condition is met.

[0060] In some examples, the latest operation of the user on the to-be-written region when the preset condition is met in step S120 is a write operation; for the to-be-written region, the memory data after the arrangement only includes data to be written into the to-be-written region by the latest write operation of the user. For data written into the to-be-written region before the latest write operation in the current period, the data is no longer included in the memory data after the arrangement, so that only useful data of the to-be-written region in the current period is stored in the disk of the file system, and the useless data of the to-be-written region in the current period is no longer stored in the disk of the file system, i.e., the data of the to-be-written region is optimized in layout in the disk of the file system. For the case that the user modifies or repeatedly writes a small region in the file for a short time, the memory data after the arrangement in this example greatly saves the storage space of the disk in the file system.

[0061] In other examples, the latest operation of the user on the to-be-written region when the preset condition is met is an erase operation; for the to-be-written region, the memory data after the arrangement no longer includes data written into the to-be-written region before the preset condition is met, so that data written into the to-be-written region in the current period is no longer stored in the disk of the file system, and the data of the to-be-written region is optimized in layout in the disk of the file system. For the case that the user creates and writes some small temporary files and then deletes them soon, the memory data after the arrangement in this example makes these temporary files no longer persistent to the disk, i.e., data to be written into the disk by a write request is all digested in the memory without being persistent to the disk except for writing the pre-write log, which not only saves the storage space of the disk but also improves the data write efficiency.

[0062] It should be understood that the memory of the metadata server is a storage unit set in the metadata server itself, and the disk of the file system is a storage unit outside the metadata server, so the read-write speed of the metadata server to its own memory is much higher than that to the disk of the file system. In addition, as shown in Figure 4 When reading data, both the memory of the metadata server itself and the disk of the file system can be accessed, but the priority of the memory of the metadata server itself is higher than that of the disk of the file system, that is, the metadata server will access its own memory based on the received read request to determine whether the data to be read has been stored in the memory, if so, the memory hits, and the data to be read is directly called from the memory; otherwise, the disk of the file system is accessed to find the data to be read in the disk of the file system.

[0063] Based on the above, for the case of reading the user data together after being written in small blocks, if the data written by multiple small blocks has not been persisted to the disk, the data written by multiple small blocks can be directly called by accessing the memory. In the storage scheme based on the NAS architecture as shown in Figure 2 In the storage scheme based on the NAS architecture as shown in Figure 2 Compared with the storage scheme based on the NAS architecture as shown in

[0064] In an optional embodiment, the arranged memory data is obtained by arranging the data stored in the memory before the preset condition is met, and the arranged memory data is still stored in the memory; in step S120, the arranged memory data is transferred to the disk of the file system, including: reading the arranged memory data from the memory and storing the read arranged memory data in the disk of the file system.

[0065] Specifically, for a to-be-written region, the data stored in the memory is arranged before the preset condition is met, which can be deleting the data written in the memory by the previous write request of the to-be-written region from the memory after receiving a new write request of the to-be-written region, and deleting the data written in the memory by the previous write request of the to-be-written region from the memory if a delete operation request of the to-be-written region is received, so that only the data existing in the to-be-written region after the latest operation is in the memory when the preset condition is met.

[0066] Further, in the case that the data to be written in the to-be-written region has a fixed length, after receiving each write request, if the to-be-written region corresponding to the write request does not occupy a storage space in the memory, a storage space is allocated for it in the memory; if the to-be-written region corresponding to the write request has occupied a storage space in the memory, the data written in the memory by the previous write request of the to-be-written region can be deleted from the memory, and the data to be written by the current write request is written in the storage space occupied by the previous write request, so that the data stored in the memory is rearranged in space, and the memory space is not wasted.

[0067] The rearranged memory data read above can be stored in the disk of the file system in sequence, i.e., the read data is sequentially stored in the disk of the file system according to the order of reading.

[0068] The rearranged memory data obtained by rearranging the data stored in the memory in advance in the embodiment of the present disclosure not only saves the time consumed in the data storing process, but also releases the memory space occupied by the useless data early.

[0069] In an optional embodiment, the step S120 of storing the rearranged memory data in the disk of the file system comprises: reading the data from the memory and judging whether the read data is the to-be-discarded data; in the case that the read data is the to-be-discarded data, the read data is not stored in the disk of the file system, but can be directly deleted from the memory.

[0070] Specifically, if the latest operation of the user on the to-be-written region is a write operation when the preset condition is met, the to-be-discarded data is the data written in the to-be-written region before the latest write operation; if the latest operation of the user on the to-be-written region is an erase operation when the preset condition is met, the to-be-discarded data is the data written in the to-be-written region before the preset condition is met. In the data storing process, whether the read data is the to-be-discarded data can be judged according to the metadata recorded for the data.

[0071] The embodiment of the present disclosure rearranges the data in the data storing process, and the data storing and rearranging can be simultaneously realized by running one thread, so that the processor of the metadata server does not need to allocate an additional thread for the execution of the step S120 and the step S130, and the processor resource of the metadata server is less occupied.

[0072] Further, the arranged data includes a plurality of sub-data, and the step S120 of storing the arranged memory data into the disk of the file system includes: storing the plurality of sub-data so that the plurality of sub-data are stored in the disk of the file system in a preset order, thereby realizing the sorting of the plurality of sub-data in the disk. Specifically, the preset order is the arrangement order of the plurality of sub-data, and each sub-data can be read from the memory in the preset order, and then the read sub-data can be stored into the disk in the order of reading. Here, the arrangement order of the plurality of sub-data can be the order of the arrangement of the plurality of sub-data when the plurality of sub-data are read together in a common application. Since the user reads data from the disk by generally giving the initial sector position of the disk and the number of continuous sectors to be read from the initial sector, in the scenario that the plurality of sub-data need to be read together, the plurality of sub-data are stored in the disk in sequence and continuously, which is beneficial to efficiently read the plurality of sub-data from the disk.

[0073] Further, the arranged data includes at least one sub-data, and the step of storing the arranged memory data into the disk of the file system further includes: setting index information for the sub-data, the index information being used to indicate the position of the sub-data in the disk of the file system and being stored in the metadata server. It should be noted that the metadata stored in the metadata server is used to indicate the position of the data to be written in the disk file of the user file, but not the position of the data to be written in the disk of the user file, and the index information is used to indicate the position of the sub-data to be written in the disk of the file system. The metadata server of the file system in the prior art stores the position of each data in the disk of the file system, and thus the index information can be obtained from the feedback information of the file system after the sub-data is stored. The setting of the index information can effectively help to efficiently read the sub-data subsequently.

[0074] The data written into the memory in the current period is generally a plurality of groups of data written based on a plurality of write requests, Figure 4 One of the memory data files (MemDataFile) is a group of data written based on one write request. After the data is stored into the disk of the file system according to the last preset condition, the memory data files in the memory are continuously increased until the size of the used space in the memory is not less than a preset value or the time length from the last time of storing data into the disk of the file system is not less than a preset time length, and then the current storing of data into the disk of the file system is performed, that is, the arranged memory data is stored into the file system. Figure 4The data file shown in the figure is a disk file stored on a disk in the file system, and the disk file stores the sorted memory data. The disk file can be located on one disk or multiple disks according to a distributed storage technology. Here, the size of the used space in the memory is not less than a preset value, which is an optional preset condition, that is, whether the data needs to be stored in the disk of the file system is determined by monitoring the use of the memory in real time. The preset condition can ensure that the operation of the metadata server itself is not affected by the storage of the memory data file by reasonably setting the preset value. The time length since the last storage of data in the disk of the file system is not less than a preset time length, which is another optional preset condition, that is, the time is counted from the first time, and when the time length reaches the preset time length, the data is stored in the disk of the file system. Whether the preset condition is met can be easily determined in practice.

[0075] The data to be written in the to-be-written region of the user file is cached in the memory in advance, and the data stored in the file system subsequently is the sorted memory data, that is, the small piece of data to be written in the to-be-written region of the user file is subjected to garbage collection once before being stored in the disk, and no disk read / write is involved, so it is relatively efficient and reduces the pressure of garbage collection on the disk subsequently. In addition, the memory data file in the memory plays a role of read cache within a certain time, effectively reducing the pressure of data reading on the file system.

[0076] However, in some scenarios, the user may initiate continuous small piece writing, for example, the user continuously modifies a small piece of region in the user file. For this case, although the above data storage process can solve the data fragmentation caused by small piece writing, continuous small piece writing will always cause a large amount of resource occupation, and thus is not friendly to any file system, especially to the file system implemented in the append-only storage mode. For the user of continuous small piece writing, a flow limiting mode (that is, limiting the number of write requests sent by the user) can be adopted to reduce the number of small piece writings of the user, so as to solve the problem of deep read / write queue depth (IO depth) used by the user or even malicious large-scale parallel access. In addition, the data storage process can be improved to reduce the number of small piece writings of the user, that is, the data storage process is improved to Figure 5 The flow shown in the figure performs the above step S110.

[0077] Referring to Figure 5 Step S110, based on the received write request, storing the small piece of data to be written in the to-be-written region of the user file in the memory, including:

[0078] Step S111, determining whether the current received write request satisfies the back pressure condition, and the back pressure condition is that the number of continuous small piece writings of the user is not less than a number threshold.

[0079] For example, if the number threshold is 20, if the current received write request is the 20th write request in the small block write, the current received write request satisfies the back pressure condition; if the current received write request is the 21st write request in the small block write, the current received write request also satisfies the back pressure condition; if the current received write request is the 19th write request in the small block write, the current received write request does not satisfy the back pressure condition.

[0080] In step S112, if the current received write request does not satisfy the back pressure condition, only the small block data to be written in the to-be-written region in the user file is stored in the memory.

[0081] In step S113, if the current received write request satisfies the back pressure condition, the data in the region around the to-be-written region in the user file is read out and stored in the memory together with the small block data to be written in the to-be-written region.

[0082] Specifically, the current received write request can be the first write data to the to-be-written region in the user file or a modification.

[0083] If the current received write request is the first write data to the to-be-written region in the user file, in the case of writing the user file from front to back, there is no data of the region after the current region in the user file stored in the memory or the disk of the file system, and the region around the to-be-written region is only the region before the to-be-written region in the user file. In this case, step S113 is to read out the data in the region before the to-be-written region from the memory or the disk of the file system, and store the read data and the small block data to be written in the to-be-written region in the memory according to the order in the user file.

[0084] If the current received write request is a modification to the to-be-written region in the user file, in the case of writing the user file from front to back, there is data of the region before and after the current region in the user file stored in the memory or the disk of the file system, and the region around the to-be-written region can be the region before and / or after the to-be-written region in the user file. In this case, step S113 is to read out the data in the region before and / or after the to-be-written region from the memory or the disk of the file system, and store the read data and the small block data to be written in the to-be-written region in the memory according to the order in the user file.

[0085] The process of performing step S113 is to perform a read-modify-write process, wherein the modification is from the to-be-written region to the to-be-written region and the surrounding region of the to-be-written region. The process of step S113 leads to an increase in the overall write process latency, which is equivalent to back pressure to the user and reduces the write rate in the case of the same read-write queue depth. Compared with only writing the to-be-written region, the read-modify-write process increases the system consumption at the moment, but has the advantage of eliminating the user's small block write. In some cases, a plurality of adjacent fragments can also be connected into a large data block, which is equivalent to a local micro version of garbage collection, reducing the risk of fragmentation in the file system.

[0086] It should be noted that for the process performed in step S113, the read region in the user file can have a simultaneous user write performed by two processes run by the same user, that is, the surrounding region of the to-be-written region needs to be read out by one process and stored together with the data in the to-be-written region, but the surrounding region of the to-be-written region simultaneously performs a user write by another process. In this case, the two processes need to be coordinated so that the surrounding region of the to-be-written region first performs a user write by one process and then performs a read by another process, so as to prevent the newly written data in the surrounding region of the to-be-written region from being overwritten by the old data.

[0087] Commercial value of the present disclosure

[0088] The embodiment of the present disclosure is applied to a data write path, and optimizes the presentation of small block data in a file system and the disk layout by improving a data storage scheme, reduces the pressure of garbage collection on the disk, and to some extent, plays a role of read cache, effectively reduces the pressure of data reading on the file system. For the user's small block write, the number of continuous small block writes of the user can also be reduced, and the stability of the file system is protected. The entire data storage scheme has extremely high commercial value due to its outstanding advantages.

[0089] The data storage method according to one embodiment of the present disclosure can be implemented by Figure 6 The computer device 800. When implementing the data storage method according to one embodiment of the present disclosure, the computer device 800 is a metadata server in a NAS architecture. The computer device 800 according to one embodiment of the present disclosure will be described below with reference to Figure 6 Figure 6 The computer device 800 shown is merely an example and should not impose any limitation on the functions and use range of the embodiments of the present disclosure.

[0090] As Figure 6 ​As shown, the computer device 800 is in the form of a general-purpose computing device. The components of the computer device 800 can include, but are not limited to, the at least one processing unit 810 described above, the at least one memory unit 820 described above, and a bus 830 that connects the various system components, including the memory unit 820 and the processing unit 810.

[0091] The memory unit stores program code that can be executed by the processing unit 810 such that the processing unit 810 performs the steps of the various exemplary embodiments of the present disclosure described in the description of the exemplary methods described above. For example, the processing unit 810 can perform the various steps illustrated in the flowchart of FIG. 8, such as Figure 3 or Figure 5 as illustrated in the flowchart of FIG. 9.

[0092] The memory unit 820 can include a readable medium in the form of volatile memory units, such as a random access memory (RAM) 8201 and / or a cache memory unit 8202, and can further include a read-only memory (ROM) 8203.

[0093] The memory unit 820 can further include program / utility 8204 having a set of programs / modules 8205, including but not limited to, an operating system, one or more application programs, other program modules, and program data, each or a combination thereof possibly including implementation of a network environment.

[0094] The bus 830 can be representative of one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration bus, a processing bus, or a local bus using any of a variety of bus architectures.

[0095] Computer device 800 can also communicate with one or more external devices 700 (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more terminal devices that enable users to interact with computer device 800, and / or any device that enables computer device 800 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 850. Furthermore, computer device 800 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 860. As shown, network adapter 860 communicates with other modules of platform server 800 via bus 830. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with computer device 800, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0096] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. Therefore, this disclosure also provides a computer-readable medium storing computer-executable code, which, when executed by a processor, implements the various processes of the embodiments of the above data storage methods. The computer-readable medium includes various media capable of storing computer-executable code, such as a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0097] Since the code stored in this readable medium can execute the steps of any of the data storage methods provided in the embodiments of this disclosure, the beneficial effects achievable by any of the data storage methods provided in the embodiments of this disclosure can be realized, as detailed in the preceding embodiments, and will not be repeated here. The specific implementation of each of the above operations can be found in the preceding embodiments, and will not be repeated here.

[0098] It should be understood that the above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. For those skilled in the art, there are many variations of the embodiments in this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the protection scope of this disclosure.

[0099] It is to be understood that the various embodiments described herein are described in connection with what is presently considered to be the most practical and preferred implementations, however, as disclosed herein, the

[0100] It is to be understood that the various embodiments described herein are described in connection with what is presently considered to be the most practical and preferred implementations, however, as disclosed herein, the

[0101] It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It is further understood that the terms "comprising," "including," containing," and "having" and variations thereof, as used herein, are intended to be equivalent to the term "consisting of." Furthermore, it is to be understood that the use of "including," "comprising," "containing," "having," "consisting of," "consisting essentially of," or grammatical variants thereof, is not to be construed as limiting.

[0102] It is to be understood that the terms and expressions used herein are used as terms of description and not of limitation. There is no intention, therefore, of using the terms or expressions to exclude any equivalents of the features shown and described or portions thereof. It is recognized that various modifications are possible within the scope of the claims and it is intended to claim such modifications as fall within the scope of the claims. Other modifications, changes, and substitutions are also possible. Accordingly, the application is not to be limited by what has been particularly shown and described.

Claims

1. A data storage method applied to a metadata server of a NAS architecture, comprising: storing, based on a received write request, small pieces of data to be written into a to-be-written region of a user file in a memory; in a case where a preset condition is met, transferring, to a disk of a file system, memory data that has been arranged; wherein, for the to-be-written region, the memory data that has been arranged only includes data existing in the to-be-written region after a latest operation of a user when the preset condition is met; wherein, the transferring, to the disk of the file system, of the memory data that has been arranged comprises: reading data from the memory and judging whether the read data is to-be-discarded data; in a case where the read data is the to-be-discarded data, the read data is no longer transferred to the disk of the file system; wherein, if the latest operation of the user on the to-be-written region when the preset condition is met is a write operation, the to-be-discarded data is data written into the to-be-written region before the latest write operation; if the latest operation of the user on the to-be-written region when the preset condition is met is an erase operation, the to-be-discarded data is data written into the to-be-written region before the preset condition is met. 2.The method of claim 1, wherein: the latest operation of the user on the to-be-written region when the preset condition is met is a write operation; for the to-be-written region, the memory data that has been arranged only includes data to be written into the to-be-written region by the latest write operation of the user when the preset condition is met. 3.The method of claim 1, wherein: the latest operation of the user on the to-be-written region when the preset condition is met is an erase operation; for the to-be-written region, the memory data that has been arranged no longer includes data written into the to-be-written region before the preset condition is met. 4.The method of claim 1, wherein: the memory data that has been arranged is obtained by arranging data stored in the memory before the preset condition is met, and the memory data that has been arranged is still stored in the memory; the transferring, to the disk of the file system, of the memory data that has been arranged comprises: reading the memory data that has been arranged from the memory and storing the read memory data that has been arranged into the disk of the file system.

5. The method of claim 1, wherein, the data that has been arranged includes a plurality of sub-data, and the transferring, to the disk of the file system, of the memory data that has been arranged comprises: transferring the plurality of sub-data so that the plurality of sub-data are stored in the disk of the file system in a preset order.

6. The method of claim 1, wherein, the data that has been arranged includes at least one sub-data, and the transferring, to the disk of the file system, of the memory data that has been arranged comprises: setting index information for the sub-data, the index information being used to indicate a position of the sub-data in the disk of the file system and being stored in the metadata server. 7.The method of claim 1, wherein: the preset condition is that a size of a used space in the memory is not less than a preset value; or the preset condition is that a time length from a last time of transferring data to the disk of the file system is not less than a preset time length. ​ 8. The method of claim 1, wherein storing the small chunk data to be written into the to-be-written region of the user file in the memory based on the received write request comprises: determining whether the current received write request satisfies a back pressure condition, the back pressure condition being that a number of times of continuous small chunk writing performed by the user is not less than a threshold number of times; in a case where the current received write request does not satisfy the back pressure condition, storing only the small chunk data to be written into the to-be-written region of the user file in the memory; and in a case where the current received write request satisfies the back pressure condition, reading out data in a region surrounding the to-be-written region of the user file and storing the read out data together with the small chunk data to be written into the to-be-written region in the memory.

9. A metadata server of a NAS architecture, comprising: a memory configured to store computer executable code; and a processor configured to execute the computer executable code to implement the data storage method of any one of claims 1-8.

10. A storage platform, comprising: a file system; and the metadata server of claim 9.

11. A computer readable medium comprising computer executable code that, when executed by a processor, implements the data storage method of any one of claims 1-8. ​ ​ ​ ​ ​ ​ ​

Images